2024-04-19T23:02:34.902 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-19T23:02:34.902 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001 --owner scheduled_teuthology@teuthology --verbose --name teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi} -- /home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001/orig.config.yaml 2024-04-19T23:02:34.941 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-19T23:02:35.046 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001 branch: reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi} email: ceph-qa@ceph.com first_in_suite: false job_id: '7664001' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: rhel os_version: '8.6' overrides: admin_socket: branch: reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: reef sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 3659 sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d sleep_before_teardown: 0 subset: 0/1 suite: rgw suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d targets: smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIcvqhGL0N9TBJPBbHqO/Z9Admuo3j1MrM0n8mryoZRaN5gAbxH+wX247Y6GP/hxWVAgp2QfcTB35iT/FQ1H4a4= smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLPGn4JYphtVVEoq1g7lxNACnaZmW1uujquDNRq/8bGYczh4L4ye1SfOdTGMj2UeOhZiDkldB65GKI6v8+/ZkUU= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-19_22:40:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-19T23:02:35.046 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa; will attempt to use it 2024-04-19T23:02:35.047 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks 2024-04-19T23:02:35.047 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-19T23:02:35.049 INFO:teuthology.task.internal:Checking packages... 2024-04-19T23:02:35.073 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '646b8c7d293d8ec31a40c69f7b3237b19c1be24d' 2024-04-19T23:02:35.073 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-19T23:02:35.073 INFO:teuthology.packaging:ref: None 2024-04-19T23:02:35.073 INFO:teuthology.packaging:tag: None 2024-04-19T23:02:35.073 INFO:teuthology.packaging:branch: reef 2024-04-19T23:02:35.073 INFO:teuthology.packaging:sha1: 646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-19T23:02:35.073 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef 2024-04-19T23:02:35.437 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-0 2024-04-19T23:02:35.438 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-19T23:02:35.445 INFO:teuthology.task.internal:no buildpackages task found 2024-04-19T23:02:35.445 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-19T23:02:35.493 INFO:teuthology.task.internal:Saving configuration 2024-04-19T23:02:35.505 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-19T23:02:35.513 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-19T23:02:35.533 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 22:57:05.745051', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIcvqhGL0N9TBJPBbHqO/Z9Admuo3j1MrM0n8mryoZRaN5gAbxH+wX247Y6GP/hxWVAgp2QfcTB35iT/FQ1H4a4='} 2024-04-19T23:02:35.553 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 22:57:05.746628', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLPGn4JYphtVVEoq1g7lxNACnaZmW1uujquDNRq/8bGYczh4L4ye1SfOdTGMj2UeOhZiDkldB65GKI6v8+/ZkUU='} 2024-04-19T23:02:35.553 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-19T23:02:35.559 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2024-04-19T23:02:35.559 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2024-04-19T23:02:35.559 INFO:teuthology.run_tasks:Running task console_log... 2024-04-19T23:02:35.634 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd2072fc3a0>, signals=[15]) 2024-04-19T23:02:35.634 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-19T23:02:35.641 INFO:teuthology.task.internal:Opening connections... 2024-04-19T23:02:35.641 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2024-04-19T23:02:35.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T23:02:35.717 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-04-19T23:02:35.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T23:02:35.791 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-19T23:02:35.797 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2024-04-19T23:02:35.819 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2024-04-19T23:02:35.819 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2024-04-19T23:02:35.880 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="Red Hat Enterprise Linux" 2024-04-19T23:02:35.880 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="8.6 (Ootpa)" 2024-04-19T23:02:35.880 INFO:teuthology.orchestra.run.smithi019.stdout:ID="rhel" 2024-04-19T23:02:35.880 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="fedora" 2024-04-19T23:02:35.880 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="8.6" 2024-04-19T23:02:35.880 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el8" 2024-04-19T23:02:35.880 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-19T23:02:35.880 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2024-04-19T23:02:35.880 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-19T23:02:35.880 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://www.redhat.com/" 2024-04-19T23:02:35.880 INFO:teuthology.orchestra.run.smithi019.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-19T23:02:35.881 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-19T23:02:35.881 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:02:35.881 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-19T23:02:35.881 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-19T23:02:35.881 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-19T23:02:35.881 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-19T23:02:35.881 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2024-04-19T23:02:35.915 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-04-19T23:02:35.936 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-04-19T23:02:35.937 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-04-19T23:02:35.998 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Red Hat Enterprise Linux" 2024-04-19T23:02:35.999 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8.6 (Ootpa)" 2024-04-19T23:02:35.999 INFO:teuthology.orchestra.run.smithi031.stdout:ID="rhel" 2024-04-19T23:02:35.999 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="fedora" 2024-04-19T23:02:35.999 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8.6" 2024-04-19T23:02:35.999 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8" 2024-04-19T23:02:35.999 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-19T23:02:35.999 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-04-19T23:02:35.999 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-19T23:02:35.999 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.redhat.com/" 2024-04-19T23:02:35.999 INFO:teuthology.orchestra.run.smithi031.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-19T23:02:35.999 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-19T23:02:35.999 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:02:36.000 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-19T23:02:36.000 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-19T23:02:36.000 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-19T23:02:36.000 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-19T23:02:36.000 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-04-19T23:02:36.014 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-19T23:02:36.022 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-19T23:02:36.027 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-19T23:02:36.028 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2024-04-19T23:02:36.030 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-04-19T23:02:36.059 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-19T23:02:36.065 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-19T23:02:36.065 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2024-04-19T23:02:36.090 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-04-19T23:02:36.128 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-19T23:02:36.177 INFO:teuthology.run_tasks:Running task kernel... 2024-04-19T23:02:36.191 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-19T23:02:36.191 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-19T23:02:36.192 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T23:02:36.192 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T23:02:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T23:02:36.212 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-04-19T23:02:36.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T23:02:36.213 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-04-19T23:02:36.274 INFO:teuthology.orchestra.run.smithi019.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-19T23:02:36.274 INFO:teuthology.task.kernel:Running kernel on smithi019: 4.18.0-513.9.1.el8_9.x86_64 2024-04-19T23:02:36.274 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum install -y kernel 2024-04-19T23:02:36.275 INFO:teuthology.orchestra.run.smithi031.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-19T23:02:36.275 INFO:teuthology.task.kernel:Running kernel on smithi031: 4.18.0-513.9.1.el8_9.x86_64 2024-04-19T23:02:36.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2024-04-19T23:02:37.791 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:02:37.792 INFO:teuthology.orchestra.run.smithi019.stdout:Unable to read consumer identity 2024-04-19T23:02:37.815 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:02:37.815 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2024-04-19T23:02:37.912 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:02:37.913 INFO:teuthology.orchestra.run.smithi019.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-19T23:02:37.913 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:02:37.941 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:02:37.941 INFO:teuthology.orchestra.run.smithi031.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-19T23:02:37.941 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:02:39.608 INFO:teuthology.orchestra.run.smithi019.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-04-19T23:02:39.611 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-04-19T23:02:40.678 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2024-04-19T23:02:41.062 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-19T23:02:44.711 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 424 kB/s | 24 kB 00:00 2024-04-19T23:02:45.109 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 424 kB/s | 24 kB 00:00 2024-04-19T23:02:45.514 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-19T23:02:45.514 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-19T23:02:45.595 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:02:45.597 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-19T23:02:45.597 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:02:45.737 DEBUG:teuthology.orchestra.run.smithi031:> echo no | sudo yum reinstall kernel || true 2024-04-19T23:02:45.929 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-19T23:02:45.929 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-19T23:02:45.999 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:02:46.001 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-04-19T23:02:46.001 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:02:46.134 DEBUG:teuthology.orchestra.run.smithi019:> echo no | sudo yum reinstall kernel || true 2024-04-19T23:02:46.168 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:02:46.168 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2024-04-19T23:02:46.228 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:02:46.228 INFO:teuthology.orchestra.run.smithi031.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-19T23:02:46.229 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:02:46.523 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:02:46.523 INFO:teuthology.orchestra.run.smithi019.stdout:Unable to read consumer identity 2024-04-19T23:02:46.578 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:02:46.578 INFO:teuthology.orchestra.run.smithi019.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-19T23:02:46.578 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:02:46.591 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:02 ago on Fri 19 Apr 2024 11:02:44 PM UTC. 2024-04-19T23:02:46.686 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-19T23:02:46.686 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-19T23:02:46.727 INFO:teuthology.orchestra.run.smithi031.stderr:Error: No packages marked for reinstall. 2024-04-19T23:02:46.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum reinstall -y kernel || true 2024-04-19T23:02:46.940 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:01 ago on Fri 19 Apr 2024 11:02:45 PM UTC. 2024-04-19T23:02:47.022 INFO:teuthology.orchestra.run.smithi019.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-19T23:02:47.023 INFO:teuthology.orchestra.run.smithi019.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-19T23:02:47.059 INFO:teuthology.orchestra.run.smithi019.stderr:Error: No packages marked for reinstall. 2024-04-19T23:02:47.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum reinstall -y kernel || true 2024-04-19T23:02:47.166 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:02:47.167 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2024-04-19T23:02:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:02:47.228 INFO:teuthology.orchestra.run.smithi031.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-19T23:02:47.228 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:02:47.500 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:02:47.500 INFO:teuthology.orchestra.run.smithi019.stdout:Unable to read consumer identity 2024-04-19T23:02:47.553 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:02:47.553 INFO:teuthology.orchestra.run.smithi019.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-19T23:02:47.554 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:02:47.581 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:03 ago on Fri 19 Apr 2024 11:02:44 PM UTC. 2024-04-19T23:02:47.651 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-19T23:02:47.651 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-19T23:02:47.680 INFO:teuthology.orchestra.run.smithi031.stderr:Error: No packages marked for reinstall. 2024-04-19T23:02:47.735 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-04-19T23:02:47.901 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-19T23:02:47.902 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-19T23:02:47.902 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-19T23:02:47.902 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-04-19T23:02:47.903 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T23:02:47.910 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:02 ago on Fri 19 Apr 2024 11:02:45 PM UTC. 2024-04-19T23:02:47.936 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2024-04-19T23:02:47.948 DEBUG:teuthology.parallel:result is None 2024-04-19T23:02:47.981 INFO:teuthology.orchestra.run.smithi019.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-19T23:02:47.981 INFO:teuthology.orchestra.run.smithi019.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-19T23:02:48.009 INFO:teuthology.orchestra.run.smithi019.stderr:Error: No packages marked for reinstall. 2024-04-19T23:02:48.063 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-04-19T23:02:48.241 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-19T23:02:48.241 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-19T23:02:48.242 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-19T23:02:48.242 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-04-19T23:02:48.242 DEBUG:teuthology.orchestra.run.smithi019:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T23:02:48.278 INFO:teuthology.orchestra.run.smithi019.stdout:ttyS1 2024-04-19T23:02:48.296 DEBUG:teuthology.parallel:result is None 2024-04-19T23:02:48.296 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-19T23:02:48.304 INFO:teuthology.task.internal:Creating test directory... 2024-04-19T23:02:48.304 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T23:02:48.338 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T23:02:48.359 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-19T23:02:48.366 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-19T23:02:48.372 INFO:teuthology.task.internal:Creating archive directory... 2024-04-19T23:02:48.372 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T23:02:48.398 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T23:02:48.437 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-19T23:02:48.444 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-19T23:02:48.444 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T23:02:48.476 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T23:02:48.511 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T23:02:48.514 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T23:02:48.524 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T23:02:48.528 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T23:02:48.529 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-19T23:02:48.537 INFO:teuthology.task.internal:Configuring sudo... 2024-04-19T23:02:48.537 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T23:02:48.569 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T23:02:48.604 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-19T23:02:48.613 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-19T23:02:48.613 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T23:02:48.643 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T23:02:48.666 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T23:02:48.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T23:02:48.816 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T23:02:48.858 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T23:02:48.914 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:02:48.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T23:02:48.987 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T23:02:49.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T23:02:49.095 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T23:02:49.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T23:02:49.233 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:02:49.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T23:02:49.307 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2024-04-19T23:02:49.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-04-19T23:02:49.360 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-19T23:02:49.406 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-19T23:02:49.809 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-19T23:02:49.819 INFO:teuthology.task.internal:Starting timer... 2024-04-19T23:02:49.819 INFO:teuthology.run_tasks:Running task pcp... 2024-04-19T23:02:49.828 INFO:teuthology.run_tasks:Running task selinux... 2024-04-19T23:02:49.838 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-19T23:02:49.838 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2024-04-19T23:02:49.915 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2024-04-19T23:02:49.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-04-19T23:02:49.990 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-04-19T23:02:49.992 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-19T23:02:49.993 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2024-04-19T23:02:50.023 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2024-04-19T23:02:50.024 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-04-19T23:02:50.050 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-04-19T23:02:50.050 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi019.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi031.front.sepia.ceph.com': 'permissive'} 2024-04-19T23:02:50.050 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-19T23:02:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T23:02:50.097 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-19T23:02:50.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T23:02:50.130 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-19T23:02:50.131 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2024-04-19T23:02:50.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-04-19T23:02:50.208 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-19T23:02:50.218 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-19T23:02:50.235 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-19T23:02:50.236 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi019.front.sepia.ceph.com,smithi031.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-19T23:11:32.983 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi019.front.sepia.ceph.com'), Remote(name='ubuntu@smithi031.front.sepia.ceph.com')] 2024-04-19T23:11:32.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-04-19T23:11:32.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T23:11:33.064 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-04-19T23:11:33.140 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-04-19T23:11:33.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-19T23:11:33.141 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T23:11:33.219 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-04-19T23:11:33.291 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-19T23:11:33.292 INFO:teuthology.run_tasks:Running task clock... 2024-04-19T23:11:33.306 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-19T23:11:33.307 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T23:11:33.307 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T23:11:33.310 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T23:11:33.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T23:11:33.348 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-19T23:11:33.366 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-19T23:11:33.373 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-19T23:11:33.390 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-19T23:11:33.457 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2024-04-19T23:11:33.471 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2024-04-19T23:11:33.475 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-04-19T23:11:33.488 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-19T23:11:33.489 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-04-19T23:11:33.505 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-19T23:11:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-19T23:11:33.523 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-19T23:11:33.579 INFO:teuthology.orchestra.run.smithi019.stderr:bash: ntpq: command not found 2024-04-19T23:11:33.584 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-19T23:11:33.584 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-04-19T23:11:33.584 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-19T23:11:33.584 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-19T23:11:33.584 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-19T23:11:33.584 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-19T23:11:33.612 INFO:teuthology.orchestra.run.smithi031.stderr:bash: ntpq: command not found 2024-04-19T23:11:33.617 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-19T23:11:33.617 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-04-19T23:11:33.617 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-19T23:11:33.617 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-19T23:11:33.617 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-19T23:11:33.617 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-19T23:11:33.618 INFO:teuthology.run_tasks:Running task install... 2024-04-19T23:11:33.627 DEBUG:teuthology.task.install:project ceph 2024-04-19T23:11:33.627 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'}} 2024-04-19T23:11:33.627 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'} 2024-04-19T23:11:33.627 INFO:teuthology.task.install:Using flavor: default 2024-04-19T23:11:33.633 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-19T23:11:33.633 INFO:teuthology.task.install:extra packages: [] 2024-04-19T23:11:33.633 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'tag': None, 'wait_for_package': False} 2024-04-19T23:11:33.634 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-19T23:11:33.636 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'tag': None, 'wait_for_package': False} 2024-04-19T23:11:33.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-19T23:11:33.784 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/8/flavors/default/ 2024-04-19T23:11:33.784 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1237.g646b8c7d 2024-04-19T23:11:33.789 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/8/flavors/default/ 2024-04-19T23:11:33.789 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1237.g646b8c7d 2024-04-19T23:11:33.895 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-19T23:11:33.895 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:11:33.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-19T23:11:33.932 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-19T23:11:33.932 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-19T23:11:33.958 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-19T23:11:33.958 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:11:33.958 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-19T23:11:33.994 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-19T23:11:33.994 DEBUG:teuthology.orchestra.run.smithi019:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/646b8c7d293d8ec31a40c69f7b3237b19c1be24d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-19T23:11:34.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-19T23:11:34.074 DEBUG:teuthology.orchestra.run.smithi019:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-19T23:11:34.095 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-19T23:11:34.127 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-04-19T23:11:34.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-04-19T23:11:34.163 DEBUG:teuthology.orchestra.run.smithi019:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-19T23:11:34.195 INFO:teuthology.orchestra.run.smithi019.stdout:check_obsoletes = 1 2024-04-19T23:11:34.197 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum clean all 2024-04-19T23:11:34.560 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:11:34.635 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:11:34.985 INFO:teuthology.orchestra.run.smithi031.stdout:63 files removed 2024-04-19T23:11:34.992 INFO:teuthology.orchestra.run.smithi019.stdout:63 files removed 2024-04-19T23:11:35.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw 2024-04-19T23:11:35.048 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw 2024-04-19T23:11:35.425 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:11:35.435 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:11:36.357 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 215 kB/s | 80 kB 00:00 2024-04-19T23:11:36.468 INFO:teuthology.orchestra.run.smithi019.stdout:ceph packages for x86_64 170 kB/s | 80 kB 00:00 2024-04-19T23:11:36.824 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 27 kB/s | 11 kB 00:00 2024-04-19T23:11:36.843 INFO:teuthology.orchestra.run.smithi019.stdout:ceph noarch packages 34 kB/s | 11 kB 00:00 2024-04-19T23:11:37.216 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 4.1 kB/s | 1.5 kB 00:00 2024-04-19T23:11:37.219 INFO:teuthology.orchestra.run.smithi019.stdout:ceph source packages 4.3 kB/s | 1.5 kB 00:00 2024-04-19T23:11:37.461 INFO:teuthology.orchestra.run.smithi019.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-04-19T23:11:37.475 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-04-19T23:11:38.455 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2024-04-19T23:11:38.870 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-04-19T23:11:42.438 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 373 kB/s | 24 kB 00:00 2024-04-19T23:11:42.859 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 401 kB/s | 24 kB 00:00 2024-04-19T23:11:43.590 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 56 MB/s | 60 MB 00:01 2024-04-19T23:11:44.045 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 MB/s | 67 MB 00:01 2024-04-19T23:11:55.820 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 56 MB/s | 60 MB 00:01 2024-04-19T23:11:56.032 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 60 MB/s | 67 MB 00:01 2024-04-19T23:12:07.395 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 29 MB/s | 8.8 MB 00:00 2024-04-19T23:12:07.782 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 26 MB/s | 8.8 MB 00:00 2024-04-19T23:12:15.372 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:12:15.375 INFO:teuthology.orchestra.run.smithi031.stdout:==================================================================================================== 2024-04-19T23:12:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-19T23:12:15.375 INFO:teuthology.orchestra.run.smithi031.stdout:==================================================================================================== 2024-04-19T23:12:15.375 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:12:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 13 M 2024-04-19T23:12:15.375 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 3.7 M 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 3.6 M 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 5.5 M 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 22 M 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 24 k 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 780 k 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 474 k 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 6.6 M 2024-04-19T23:12:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 45 k 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 129 k 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 197 k 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 371 k 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 364 k 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 106 k 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout:==================================================================================================== 2024-04-19T23:12:15.377 INFO:teuthology.orchestra.run.smithi031.stdout:Install 21 Packages 2024-04-19T23:12:15.378 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 2 Packages 2024-04-19T23:12:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:12:15.378 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 59 M 2024-04-19T23:12:15.378 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:12:15.786 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:12:15.789 INFO:teuthology.orchestra.run.smithi019.stdout:==================================================================================================== 2024-04-19T23:12:15.789 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-04-19T23:12:15.789 INFO:teuthology.orchestra.run.smithi019.stdout:==================================================================================================== 2024-04-19T23:12:15.789 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:12:15.789 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 13 M 2024-04-19T23:12:15.789 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2024-04-19T23:12:15.789 INFO:teuthology.orchestra.run.smithi019.stdout: librados2 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 3.7 M 2024-04-19T23:12:15.789 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 3.6 M 2024-04-19T23:12:15.789 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-04-19T23:12:15.789 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 5.5 M 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 22 M 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 24 k 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 780 k 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 474 k 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2 x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 6.6 M 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 45 k 2024-04-19T23:12:15.790 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 129 k 2024-04-19T23:12:15.791 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 197 k 2024-04-19T23:12:15.791 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-19T23:12:15.791 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 371 k 2024-04-19T23:12:15.791 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 364 k 2024-04-19T23:12:15.791 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 106 k 2024-04-19T23:12:15.791 INFO:teuthology.orchestra.run.smithi019.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-19T23:12:15.791 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:12:15.791 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:12:15.791 INFO:teuthology.orchestra.run.smithi019.stdout:==================================================================================================== 2024-04-19T23:12:15.791 INFO:teuthology.orchestra.run.smithi019.stdout:Install 21 Packages 2024-04-19T23:12:15.791 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 2 Packages 2024-04-19T23:12:15.791 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:12:15.792 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 59 M 2024-04-19T23:12:15.792 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:12:16.612 INFO:teuthology.orchestra.run.smithi031.stdout:(1/23): ceph-base-18.2.2-1237.g646b8c7d.el8.x86 5.4 MB/s | 5.5 MB 00:01 2024-04-19T23:12:16.679 INFO:teuthology.orchestra.run.smithi031.stdout:(2/23): ceph-selinux-18.2.2-1237.g646b8c7d.el8. 365 kB/s | 24 kB 00:00 2024-04-19T23:12:16.871 INFO:teuthology.orchestra.run.smithi031.stdout:(3/23): libcephfs2-18.2.2-1237.g646b8c7d.el8.x8 4.0 MB/s | 780 kB 00:00 2024-04-19T23:12:17.012 INFO:teuthology.orchestra.run.smithi031.stdout:(4/23): libradosstriper1-18.2.2-1237.g646b8c7d. 3.3 MB/s | 474 kB 00:00 2024-04-19T23:12:18.158 INFO:teuthology.orchestra.run.smithi019.stdout:(1/23): ceph-base-18.2.2-1237.g646b8c7d.el8.x86 2.6 MB/s | 5.5 MB 00:02 2024-04-19T23:12:18.233 INFO:teuthology.orchestra.run.smithi019.stdout:(2/23): ceph-selinux-18.2.2-1237.g646b8c7d.el8. 325 kB/s | 24 kB 00:00 2024-04-19T23:12:18.388 INFO:teuthology.orchestra.run.smithi031.stdout:(5/23): ceph-common-18.2.2-1237.g646b8c7d.el8.x 7.8 MB/s | 22 MB 00:02 2024-04-19T23:12:18.454 INFO:teuthology.orchestra.run.smithi031.stdout:(6/23): python3-ceph-argparse-18.2.2-1237.g646b 673 kB/s | 45 kB 00:00 2024-04-19T23:12:18.467 INFO:teuthology.orchestra.run.smithi019.stdout:(3/23): libcephfs2-18.2.2-1237.g646b8c7d.el8.x8 3.3 MB/s | 780 kB 00:00 2024-04-19T23:12:18.538 INFO:teuthology.orchestra.run.smithi031.stdout:(7/23): librgw2-18.2.2-1237.g646b8c7d.el8.x86_6 4.3 MB/s | 6.6 MB 00:01 2024-04-19T23:12:18.563 INFO:teuthology.orchestra.run.smithi031.stdout:(8/23): python3-ceph-common-18.2.2-1237.g646b8c 1.2 MB/s | 129 kB 00:00 2024-04-19T23:12:18.597 INFO:teuthology.orchestra.run.smithi031.stdout:(9/23): python3-cephfs-18.2.2-1237.g646b8c7d.el 3.3 MB/s | 197 kB 00:00 2024-04-19T23:12:18.617 INFO:teuthology.orchestra.run.smithi019.stdout:(4/23): libradosstriper1-18.2.2-1237.g646b8c7d. 3.1 MB/s | 474 kB 00:00 2024-04-19T23:12:18.656 INFO:teuthology.orchestra.run.smithi031.stdout:(10/23): python3-rados-18.2.2-1237.g646b8c7d.el 3.9 MB/s | 371 kB 00:00 2024-04-19T23:12:18.715 INFO:teuthology.orchestra.run.smithi031.stdout:(11/23): python3-rbd-18.2.2-1237.g646b8c7d.el8. 3.0 MB/s | 364 kB 00:00 2024-04-19T23:12:18.739 INFO:teuthology.orchestra.run.smithi031.stdout:(12/23): python3-rgw-18.2.2-1237.g646b8c7d.el8. 1.2 MB/s | 106 kB 00:00 2024-04-19T23:12:18.898 INFO:teuthology.orchestra.run.smithi031.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 371 kB/s | 59 kB 00:00 2024-04-19T23:12:18.982 INFO:teuthology.orchestra.run.smithi031.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 903 kB/s | 75 kB 00:00 2024-04-19T23:12:19.010 INFO:teuthology.orchestra.run.smithi031.stdout:(15/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2024-04-19T23:12:19.074 INFO:teuthology.orchestra.run.smithi031.stdout:(16/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 707 kB/s | 47 kB 00:00 2024-04-19T23:12:19.149 INFO:teuthology.orchestra.run.smithi031.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 587 kB/s | 44 kB 00:00 2024-04-19T23:12:19.200 INFO:teuthology.orchestra.run.smithi031.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2024-04-19T23:12:19.258 INFO:teuthology.orchestra.run.smithi031.stdout:(19/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2024-04-19T23:12:19.609 INFO:teuthology.orchestra.run.smithi031.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 2.8 MB/s | 1.7 MB 00:00 2024-04-19T23:12:20.134 INFO:teuthology.orchestra.run.smithi031.stdout:(21/23): librados2-18.2.2-1237.g646b8c7d.el8.x8 4.2 MB/s | 3.7 MB 00:00 2024-04-19T23:12:20.192 INFO:teuthology.orchestra.run.smithi031.stdout:(22/23): librbd1-18.2.2-1237.g646b8c7d.el8.x86_ 6.1 MB/s | 3.6 MB 00:00 2024-04-19T23:12:20.376 INFO:teuthology.orchestra.run.smithi019.stdout:(5/23): ceph-common-18.2.2-1237.g646b8c7d.el8.x 5.0 MB/s | 22 MB 00:04 2024-04-19T23:12:20.476 INFO:teuthology.orchestra.run.smithi019.stdout:(6/23): librgw2-18.2.2-1237.g646b8c7d.el8.x86_6 3.5 MB/s | 6.6 MB 00:01 2024-04-19T23:12:20.501 INFO:teuthology.orchestra.run.smithi019.stdout:(7/23): python3-ceph-argparse-18.2.2-1237.g646b 357 kB/s | 45 kB 00:00 2024-04-19T23:12:20.518 INFO:teuthology.orchestra.run.smithi019.stdout:(8/23): python3-ceph-common-18.2.2-1237.g646b8c 3.0 MB/s | 129 kB 00:00 2024-04-19T23:12:20.568 INFO:teuthology.orchestra.run.smithi019.stdout:(9/23): python3-cephfs-18.2.2-1237.g646b8c7d.el 2.9 MB/s | 197 kB 00:00 2024-04-19T23:12:20.618 INFO:teuthology.orchestra.run.smithi019.stdout:(10/23): python3-rados-18.2.2-1237.g646b8c7d.el 3.6 MB/s | 371 kB 00:00 2024-04-19T23:12:20.669 INFO:teuthology.orchestra.run.smithi019.stdout:(11/23): python3-rbd-18.2.2-1237.g646b8c7d.el8. 3.6 MB/s | 364 kB 00:00 2024-04-19T23:12:20.694 INFO:teuthology.orchestra.run.smithi019.stdout:(12/23): python3-rgw-18.2.2-1237.g646b8c7d.el8. 1.4 MB/s | 106 kB 00:00 2024-04-19T23:12:20.851 INFO:teuthology.orchestra.run.smithi031.stdout:(23/23): ceph-radosgw-18.2.2-1237.g646b8c7d.el8 2.5 MB/s | 13 MB 00:05 2024-04-19T23:12:20.856 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:12:20.856 INFO:teuthology.orchestra.run.smithi031.stdout:Total 11 MB/s | 59 MB 00:05 2024-04-19T23:12:20.907 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:12:20.958 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:12:20.958 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:12:21.036 INFO:teuthology.orchestra.run.smithi019.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 172 kB/s | 59 kB 00:00 2024-04-19T23:12:21.195 INFO:teuthology.orchestra.run.smithi019.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 583 kB/s | 306 kB 00:00 2024-04-19T23:12:21.326 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:12:21.335 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:12:21.370 INFO:teuthology.orchestra.run.smithi019.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 225 kB/s | 75 kB 00:00 2024-04-19T23:12:21.429 INFO:teuthology.orchestra.run.smithi019.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 754 kB/s | 44 kB 00:00 2024-04-19T23:12:21.471 INFO:teuthology.orchestra.run.smithi019.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2024-04-19T23:12:21.512 INFO:teuthology.orchestra.run.smithi019.stdout:(18/23): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2024-04-19T23:12:21.554 INFO:teuthology.orchestra.run.smithi019.stdout:(19/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-19T23:12:22.142 INFO:teuthology.orchestra.run.smithi019.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 1.8 MB/s | 1.7 MB 00:00 2024-04-19T23:12:22.205 INFO:teuthology.orchestra.run.smithi019.stdout:(21/23): librados2-18.2.2-1237.g646b8c7d.el8.x8 5.7 MB/s | 3.7 MB 00:00 2024-04-19T23:12:22.613 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:12:22.663 INFO:teuthology.orchestra.run.smithi019.stdout:(22/23): librbd1-18.2.2-1237.g646b8c7d.el8.x86_ 6.8 MB/s | 3.6 MB 00:00 2024-04-19T23:12:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-19T23:12:22.950 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-19T23:12:23.014 INFO:teuthology.orchestra.run.smithi019.stdout:(23/23): ceph-radosgw-18.2.2-1237.g646b8c7d.el8 1.9 MB/s | 13 MB 00:06 2024-04-19T23:12:23.017 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:12:23.017 INFO:teuthology.orchestra.run.smithi019.stdout:Total 8.2 MB/s | 59 MB 00:07 2024-04-19T23:12:23.069 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:12:23.120 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:12:23.120 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:12:23.384 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-19T23:12:23.487 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:12:23.497 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:12:23.567 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/25 2024-04-19T23:12:23.697 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/25 2024-04-19T23:12:23.857 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/25 2024-04-19T23:12:24.097 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-19T23:12:24.266 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 6/25 2024-04-19T23:12:24.702 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 6/25 2024-04-19T23:12:24.754 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:12:25.071 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-19T23:12:25.116 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 7/25 2024-04-19T23:12:25.213 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-19T23:12:25.221 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 7/25 2024-04-19T23:12:25.408 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-19T23:12:25.459 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-19T23:12:25.616 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 9/25 2024-04-19T23:12:25.664 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-19T23:12:25.746 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:18.2.2-1237.g646b8c7d.el8. 10/25 2024-04-19T23:12:25.880 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/25 2024-04-19T23:12:25.948 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 11/25 2024-04-19T23:12:25.985 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/25 2024-04-19T23:12:26.154 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/25 2024-04-19T23:12:26.189 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 12/25 2024-04-19T23:12:26.366 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_6 13/25 2024-04-19T23:12:26.402 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-19T23:12:26.442 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_6 13/25 2024-04-19T23:12:26.585 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-19T23:12:26.597 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 6/25 2024-04-19T23:12:26.763 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-19T23:12:27.030 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 6/25 2024-04-19T23:12:27.230 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 7/25 2024-04-19T23:12:27.404 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 7/25 2024-04-19T23:12:27.591 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 8/25 2024-04-19T23:12:27.596 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-19T23:12:27.741 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 17/25 2024-04-19T23:12:27.755 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2024-04-19T23:12:27.816 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2024-04-19T23:12:27.822 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 17/25 2024-04-19T23:12:28.002 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-argparse-2:18.2.2-1237.g646b8c7d.el8. 10/25 2024-04-19T23:12:28.043 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 18/25 2024-04-19T23:12:28.183 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:18.2.2-1237.g646b8c7d.el8.x8 19/25 2024-04-19T23:12:28.228 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 11/25 2024-04-19T23:12:28.461 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 12/25 2024-04-19T23:12:28.622 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_6 13/25 2024-04-19T23:12:28.696 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_6 13/25 2024-04-19T23:12:28.874 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-19T23:12:29.068 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-19T23:12:29.916 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-19T23:12:30.063 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 17/25 2024-04-19T23:12:30.144 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 17/25 2024-04-19T23:12:30.632 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 18/25 2024-04-19T23:12:30.930 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-common-2:18.2.2-1237.g646b8c7d.el8.x8 19/25 2024-04-19T23:12:32.256 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 20/25 2024-04-19T23:12:32.482 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 20/25 2024-04-19T23:12:33.146 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 20/25 2024-04-19T23:12:33.393 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 21/25 2024-04-19T23:12:33.512 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 21/25 2024-04-19T23:12:33.656 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 22/25 2024-04-19T23:12:35.019 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 20/25 2024-04-19T23:12:35.238 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 20/25 2024-04-19T23:12:35.899 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 20/25 2024-04-19T23:12:36.386 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 21/25 2024-04-19T23:12:36.475 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 21/25 2024-04-19T23:12:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 22/25 2024-04-19T23:12:56.198 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 22/25 2024-04-19T23:12:56.198 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-04-19T23:12:56.199 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-04-19T23:12:56.199 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /dev 2024-04-19T23:12:56.199 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /run 2024-04-19T23:12:56.199 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-04-19T23:12:56.199 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-04-19T23:12:56.199 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-04-19T23:12:56.199 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-04-19T23:12:56.199 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /dev 2024-04-19T23:12:56.199 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:12:57.655 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 23/25 2024-04-19T23:12:57.683 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 23/25 2024-04-19T23:12:57.683 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-19T23:12:57.922 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-19T23:12:57.922 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-19T23:12:58.658 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-19T23:12:58.658 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/25 2024-04-19T23:12:58.659 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/25 2024-04-19T23:12:58.659 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/25 2024-04-19T23:12:58.659 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/25 2024-04-19T23:12:58.659 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/25 2024-04-19T23:12:58.659 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_6 6/25 2024-04-19T23:12:58.659 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 7/25 2024-04-19T23:12:58.659 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1237.g646b8c7d.el8. 8/25 2024-04-19T23:12:58.659 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:18.2.2-1237.g646b8c7d.el8.x8 9/25 2024-04-19T23:12:58.659 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 10/25 2024-04-19T23:12:58.659 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 11/25 2024-04-19T23:12:58.659 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 12/25 2024-04-19T23:12:58.659 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 13/25 2024-04-19T23:12:58.660 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-19T23:12:58.660 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-19T23:12:58.660 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-19T23:12:58.660 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-19T23:12:58.660 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 18/25 2024-04-19T23:12:58.660 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2024-04-19T23:12:58.660 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2024-04-19T23:12:58.660 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2024-04-19T23:12:58.660 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 22/25 2024-04-19T23:12:58.660 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-19T23:12:58.660 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 24/25 2024-04-19T23:12:59.144 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 22/25 2024-04-19T23:12:59.145 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /sys 2024-04-19T23:12:59.145 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /proc 2024-04-19T23:12:59.145 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /dev 2024-04-19T23:12:59.145 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /run 2024-04-19T23:12:59.145 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /mnt 2024-04-19T23:12:59.145 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /var/tmp 2024-04-19T23:12:59.145 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /home 2024-04-19T23:12:59.145 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /tmp 2024-04-19T23:12:59.145 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /dev 2024-04-19T23:12:59.145 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:13:00.608 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 23/25 2024-04-19T23:13:00.634 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 23/25 2024-04-19T23:13:00.635 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-19T23:13:00.837 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-19T23:13:00.837 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-19T23:13:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-19T23:13:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/25 2024-04-19T23:13:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/25 2024-04-19T23:13:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/25 2024-04-19T23:13:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/25 2024-04-19T23:13:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/25 2024-04-19T23:13:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_6 6/25 2024-04-19T23:13:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 7/25 2024-04-19T23:13:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1237.g646b8c7d.el8. 8/25 2024-04-19T23:13:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-common-2:18.2.2-1237.g646b8c7d.el8.x8 9/25 2024-04-19T23:13:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 10/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 11/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 12/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 13/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 20/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 21/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 22/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-19T23:13:01.575 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 24/25 2024-04-19T23:13:04.150 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-19T23:13:04.150 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:13:04.771 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:13:04.771 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-04-19T23:13:04.771 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.771 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-19T23:13:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-19T23:13:04.773 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.773 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.773 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.773 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-19T23:13:04.773 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.773 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.773 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:04.773 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-19T23:13:04.773 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:13:04.773 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:13:04.985 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-test 2024-04-19T23:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 100 kB/s | 4.1 kB 00:00 2024-04-19T23:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 121 kB/s | 4.5 kB 00:00 2024-04-19T23:13:07.060 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-19T23:13:07.060 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:13:07.667 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:13:07.667 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2024-04-19T23:13:07.667 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.667 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-19T23:13:07.668 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-19T23:13:07.669 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.669 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.669 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.669 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-19T23:13:07.669 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.669 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.669 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:07.669 INFO:teuthology.orchestra.run.smithi019.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-19T23:13:07.669 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:13:07.669 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:13:07.864 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-test 2024-04-19T23:13:08.269 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:13:09.065 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 122 kB/s | 4.5 kB 00:00 2024-04-19T23:13:09.521 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:13:09.522 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2024-04-19T23:13:09.522 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-19T23:13:09.522 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2024-04-19T23:13:09.522 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:13:09.522 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 47 M 2024-04-19T23:13:09.522 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-19T23:13:09.522 INFO:teuthology.orchestra.run.smithi031.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-19T23:13:09.522 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 142 k 2024-04-19T23:13:09.523 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-19T23:13:09.523 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-19T23:13:09.523 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-19T23:13:09.523 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:13:09.523 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:13:09.523 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2024-04-19T23:13:09.523 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2024-04-19T23:13:09.523 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:13:09.524 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 48 M 2024-04-19T23:13:09.524 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 213 M 2024-04-19T23:13:09.524 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:13:09.579 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.3 MB/s | 70 kB 00:00 2024-04-19T23:13:09.671 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.6 MB/s | 323 kB 00:00 2024-04-19T23:13:09.696 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 95 kB/s | 4.1 kB 00:00 2024-04-19T23:13:09.730 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 3.4 MB/s | 202 kB 00:00 2024-04-19T23:13:09.788 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-04-19T23:13:10.163 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): libcephsqlite-18.2.2-1237.g646b8c7d.el8. 222 kB/s | 142 kB 00:00 2024-04-19T23:13:12.260 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:13:12.261 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2024-04-19T23:13:12.261 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-04-19T23:13:12.261 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2024-04-19T23:13:12.261 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:13:12.261 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 47 M 2024-04-19T23:13:12.261 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-04-19T23:13:12.262 INFO:teuthology.orchestra.run.smithi019.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-19T23:13:12.262 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 142 k 2024-04-19T23:13:12.262 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-19T23:13:12.262 INFO:teuthology.orchestra.run.smithi019.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-19T23:13:12.262 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-19T23:13:12.262 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:13:12.262 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:13:12.262 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2024-04-19T23:13:12.262 INFO:teuthology.orchestra.run.smithi019.stdout:Install 6 Packages 2024-04-19T23:13:12.262 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:13:12.263 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 48 M 2024-04-19T23:13:12.263 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 213 M 2024-04-19T23:13:12.263 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:13:12.294 INFO:teuthology.orchestra.run.smithi019.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.3 MB/s | 70 kB 00:00 2024-04-19T23:13:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2024-04-19T23:13:12.412 INFO:teuthology.orchestra.run.smithi019.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2024-04-19T23:13:12.471 INFO:teuthology.orchestra.run.smithi019.stdout:(4/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-04-19T23:13:12.629 INFO:teuthology.orchestra.run.smithi019.stdout:(5/6): libcephsqlite-18.2.2-1237.g646b8c7d.el8. 389 kB/s | 142 kB 00:00 2024-04-19T23:13:16.996 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): ceph-test-18.2.2-1237.g646b8c7d.el8.x86_ 6.3 MB/s | 47 MB 00:07 2024-04-19T23:13:16.998 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:13:16.998 INFO:teuthology.orchestra.run.smithi031.stdout:Total 6.4 MB/s | 48 MB 00:07 2024-04-19T23:13:17.040 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:13:17.064 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:13:17.064 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:13:17.362 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:13:17.372 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:13:17.858 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:13:17.990 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-19T23:13:18.077 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-19T23:13:18.284 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-19T23:13:18.477 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-19T23:13:18.630 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-19T23:13:18.791 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/6 2024-04-19T23:13:23.759 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/6 2024-04-19T23:13:23.997 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 6/6 2024-04-19T23:13:24.283 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 6/6 2024-04-19T23:13:24.283 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/6 2024-04-19T23:13:24.283 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/6 2024-04-19T23:13:24.284 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-19T23:13:24.284 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-19T23:13:24.284 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-19T23:13:26.028 INFO:teuthology.orchestra.run.smithi019.stdout:(6/6): ceph-test-18.2.2-1237.g646b8c7d.el8.x86_ 3.4 MB/s | 47 MB 00:13 2024-04-19T23:13:26.030 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:13:26.030 INFO:teuthology.orchestra.run.smithi019.stdout:Total 3.5 MB/s | 48 MB 00:13 2024-04-19T23:13:26.072 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:13:26.096 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:13:26.096 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:13:26.393 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:13:26.403 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:13:26.891 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:13:27.064 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-19T23:13:27.151 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-19T23:13:27.348 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-19T23:13:27.533 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-19T23:13:27.736 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-19T23:13:27.897 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/6 2024-04-19T23:13:29.451 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-19T23:13:29.451 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:13:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:13:30.056 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:13:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: jq-1.6-7.el8.x86_64 2024-04-19T23:13:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:30.056 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-19T23:13:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-19T23:13:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-19T23:13:30.057 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:13:30.057 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:13:30.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph 2024-04-19T23:13:30.649 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:13:31.687 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 104 kB/s | 4.1 kB 00:00 2024-04-19T23:13:31.981 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 139 kB/s | 4.5 kB 00:00 2024-04-19T23:13:32.893 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/6 2024-04-19T23:13:33.137 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 6/6 2024-04-19T23:13:33.434 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 6/6 2024-04-19T23:13:33.434 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/6 2024-04-19T23:13:33.435 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/6 2024-04-19T23:13:33.435 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-19T23:13:33.435 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-19T23:13:33.435 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-19T23:13:35.028 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:13:35.032 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================= 2024-04-19T23:13:35.032 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-19T23:13:35.032 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================= 2024-04-19T23:13:35.032 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:13:35.032 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 6.4 k 2024-04-19T23:13:35.032 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 2.3 M 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 1.6 M 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 245 k 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 4.8 M 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 18 M 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-19T23:13:35.033 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-19T23:13:35.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-19T23:13:35.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-19T23:13:35.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-19T23:13:35.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-19T23:13:35.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-19T23:13:35.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-19T23:13:35.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-19T23:13:35.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-19T23:13:35.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-19T23:13:35.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-19T23:13:35.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-19T23:13:35.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-19T23:13:35.035 INFO:teuthology.orchestra.run.smithi031.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-19T23:13:35.035 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-19T23:13:35.035 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-19T23:13:35.035 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-19T23:13:35.035 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-19T23:13:35.035 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:13:35.035 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:13:35.035 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================= 2024-04-19T23:13:35.035 INFO:teuthology.orchestra.run.smithi031.stdout:Install 32 Packages 2024-04-19T23:13:35.035 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:13:35.036 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 31 M 2024-04-19T23:13:35.036 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 124 M 2024-04-19T23:13:35.036 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:13:35.306 INFO:teuthology.orchestra.run.smithi031.stdout:(1/32): ceph-18.2.2-1237.g646b8c7d.el8.x86_64.r 34 kB/s | 6.4 kB 00:00 2024-04-19T23:13:35.432 INFO:teuthology.orchestra.run.smithi031.stdout:(2/32): ceph-mgr-18.2.2-1237.g646b8c7d.el8.x86_ 5.0 MB/s | 1.6 MB 00:00 2024-04-19T23:13:35.524 INFO:teuthology.orchestra.run.smithi031.stdout:(3/32): ceph-mds-18.2.2-1237.g646b8c7d.el8.x86_ 5.6 MB/s | 2.3 MB 00:00 2024-04-19T23:13:35.582 INFO:teuthology.orchestra.run.smithi031.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1237.g646b 4.1 MB/s | 245 kB 00:00 2024-04-19T23:13:35.682 INFO:teuthology.orchestra.run.smithi031.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 438 kB/s | 44 kB 00:00 2024-04-19T23:13:35.774 INFO:teuthology.orchestra.run.smithi031.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 2.0 MB/s | 185 kB 00:00 2024-04-19T23:13:35.850 INFO:teuthology.orchestra.run.smithi031.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.3 MB/s | 173 kB 00:00 2024-04-19T23:13:35.975 INFO:teuthology.orchestra.run.smithi031.stdout:(8/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 3.0 MB/s | 384 kB 00:00 2024-04-19T23:13:36.092 INFO:teuthology.orchestra.run.smithi031.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2024-04-19T23:13:36.192 INFO:teuthology.orchestra.run.smithi031.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 176 kB/s | 18 kB 00:00 2024-04-19T23:13:36.276 INFO:teuthology.orchestra.run.smithi031.stdout:(11/32): python3-logutils-0.3.5-11.el8.noarch.r 586 kB/s | 49 kB 00:00 2024-04-19T23:13:36.335 INFO:teuthology.orchestra.run.smithi031.stdout:(12/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2024-04-19T23:13:36.643 INFO:teuthology.orchestra.run.smithi031.stdout:(13/32): ceph-osd-18.2.2-1237.g646b8c7d.el8.x86 15 MB/s | 18 MB 00:01 2024-04-19T23:13:36.668 INFO:teuthology.orchestra.run.smithi031.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 848 kB/s | 283 kB 00:00 2024-04-19T23:13:36.727 INFO:teuthology.orchestra.run.smithi031.stdout:(15/32): python3-simplegeneric-0.8.1-17.el8.noa 321 kB/s | 19 kB 00:00 2024-04-19T23:13:36.752 INFO:teuthology.orchestra.run.smithi031.stdout:(16/32): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2024-04-19T23:13:36.786 INFO:teuthology.orchestra.run.smithi031.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 406 kB/s | 24 kB 00:00 2024-04-19T23:13:36.811 INFO:teuthology.orchestra.run.smithi031.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 497 kB/s | 29 kB 00:00 2024-04-19T23:13:36.878 INFO:teuthology.orchestra.run.smithi031.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 300 kB/s | 27 kB 00:00 2024-04-19T23:13:36.954 INFO:teuthology.orchestra.run.smithi031.stdout:(20/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.8 MB/s | 255 kB 00:00 2024-04-19T23:13:36.979 INFO:teuthology.orchestra.run.smithi031.stdout:(21/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.5 MB/s | 251 kB 00:00 2024-04-19T23:13:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2024-04-19T23:13:37.030 INFO:teuthology.orchestra.run.smithi031.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 456 kB/s | 23 kB 00:00 2024-04-19T23:13:37.230 INFO:teuthology.orchestra.run.smithi031.stdout:(24/32): python3-pycparser-2.14-14.el8.noarch.r 485 kB/s | 109 kB 00:00 2024-04-19T23:13:37.287 INFO:teuthology.orchestra.run.smithi031.stdout:(25/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 925 kB/s | 238 kB 00:00 2024-04-19T23:13:37.322 INFO:teuthology.orchestra.run.smithi031.stdout:(26/32): python3-cryptography-3.2.1-7.el8_9.x86 5.9 MB/s | 559 kB 00:00 2024-04-19T23:13:37.356 INFO:teuthology.orchestra.run.smithi031.stdout:(27/32): python3-werkzeug-0.12.2-4.el8.noarch.r 6.5 MB/s | 457 kB 00:00 2024-04-19T23:13:37.381 INFO:teuthology.orchestra.run.smithi031.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 665 kB/s | 39 kB 00:00 2024-04-19T23:13:37.406 INFO:teuthology.orchestra.run.smithi031.stdout:(29/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 2.0 MB/s | 103 kB 00:00 2024-04-19T23:13:37.432 INFO:teuthology.orchestra.run.smithi031.stdout:(30/32): python3-mako-1.0.6-14.el8.noarch.rpm 3.0 MB/s | 157 kB 00:00 2024-04-19T23:13:37.457 INFO:teuthology.orchestra.run.smithi031.stdout:(31/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2024-04-19T23:13:37.691 INFO:teuthology.orchestra.run.smithi031.stdout:(32/32): ceph-mon-18.2.2-1237.g646b8c7d.el8.x86 2.0 MB/s | 4.8 MB 00:02 2024-04-19T23:13:37.693 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:13:37.693 INFO:teuthology.orchestra.run.smithi031.stdout:Total 12 MB/s | 31 MB 00:02 2024-04-19T23:13:37.797 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:13:37.839 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:13:37.839 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:13:38.094 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:13:38.104 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:13:38.802 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-19T23:13:38.802 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:13:39.230 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:13:39.421 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:13:39.421 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:13:39.421 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:39.421 INFO:teuthology.orchestra.run.smithi019.stdout: jq-1.6-7.el8.x86_64 2024-04-19T23:13:39.421 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:39.421 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-19T23:13:39.421 INFO:teuthology.orchestra.run.smithi019.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-19T23:13:39.421 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-19T23:13:39.422 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:13:39.422 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:13:39.435 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-19T23:13:39.629 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-19T23:13:39.747 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph 2024-04-19T23:13:39.768 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 3/32 2024-04-19T23:13:39.984 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2024-04-19T23:13:40.134 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:13:40.263 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 5/32 2024-04-19T23:13:40.455 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/32 2024-04-19T23:13:40.639 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2024-04-19T23:13:40.979 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 112 kB/s | 4.5 kB 00:00 2024-04-19T23:13:41.065 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 8/32 2024-04-19T23:13:41.360 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 9/32 2024-04-19T23:13:41.530 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2024-04-19T23:13:41.622 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 98 kB/s | 4.1 kB 00:00 2024-04-19T23:13:41.681 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2024-04-19T23:13:41.910 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2024-04-19T23:13:42.133 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-19T23:13:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-19T23:13:42.450 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-19T23:13:42.625 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-19T23:13:42.823 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-19T23:13:42.985 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-19T23:13:43.196 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-19T23:13:43.392 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-19T23:13:43.601 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-19T23:13:43.840 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-19T23:13:44.056 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-19T23:13:44.258 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-19T23:13:44.322 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:13:44.326 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================================= 2024-04-19T23:13:44.326 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-04-19T23:13:44.326 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================================= 2024-04-19T23:13:44.326 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:13:44.326 INFO:teuthology.orchestra.run.smithi019.stdout: ceph x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 6.4 k 2024-04-19T23:13:44.326 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-04-19T23:13:44.326 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 2.3 M 2024-04-19T23:13:44.326 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 1.6 M 2024-04-19T23:13:44.326 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 245 k 2024-04-19T23:13:44.326 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 4.8 M 2024-04-19T23:13:44.326 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 18 M 2024-04-19T23:13:44.326 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-19T23:13:44.327 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-19T23:13:44.328 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-19T23:13:44.328 INFO:teuthology.orchestra.run.smithi019.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-19T23:13:44.328 INFO:teuthology.orchestra.run.smithi019.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-19T23:13:44.328 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-19T23:13:44.328 INFO:teuthology.orchestra.run.smithi019.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-19T23:13:44.328 INFO:teuthology.orchestra.run.smithi019.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-19T23:13:44.328 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-19T23:13:44.328 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-19T23:13:44.328 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-19T23:13:44.328 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-19T23:13:44.328 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:13:44.328 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:13:44.329 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================================= 2024-04-19T23:13:44.329 INFO:teuthology.orchestra.run.smithi019.stdout:Install 32 Packages 2024-04-19T23:13:44.329 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:13:44.329 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 31 M 2024-04-19T23:13:44.329 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 124 M 2024-04-19T23:13:44.329 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:13:44.584 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-19T23:13:44.830 INFO:teuthology.orchestra.run.smithi019.stdout:(1/32): ceph-18.2.2-1237.g646b8c7d.el8.x86_64.r 16 kB/s | 6.4 kB 00:00 2024-04-19T23:13:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-19T23:13:45.293 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1237.g646b8c7d.el8. 27/32 2024-04-19T23:13:45.497 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 28/32 2024-04-19T23:13:45.905 INFO:teuthology.orchestra.run.smithi019.stdout:(2/32): ceph-mgr-18.2.2-1237.g646b8c7d.el8.x86_ 1.1 MB/s | 1.6 MB 00:01 2024-04-19T23:13:45.956 INFO:teuthology.orchestra.run.smithi019.stdout:(3/32): ceph-mds-18.2.2-1237.g646b8c7d.el8.x86_ 1.5 MB/s | 2.3 MB 00:01 2024-04-19T23:13:46.089 INFO:teuthology.orchestra.run.smithi019.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1237.g646b 1.8 MB/s | 245 kB 00:00 2024-04-19T23:13:46.240 INFO:teuthology.orchestra.run.smithi019.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 291 kB/s | 44 kB 00:00 2024-04-19T23:13:46.340 INFO:teuthology.orchestra.run.smithi019.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 1.8 MB/s | 185 kB 00:00 2024-04-19T23:13:46.407 INFO:teuthology.orchestra.run.smithi019.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.5 MB/s | 173 kB 00:00 2024-04-19T23:13:46.499 INFO:teuthology.orchestra.run.smithi019.stdout:(8/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 4.1 MB/s | 384 kB 00:00 2024-04-19T23:13:46.549 INFO:teuthology.orchestra.run.smithi019.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 223 kB/s | 11 kB 00:00 2024-04-19T23:13:46.625 INFO:teuthology.orchestra.run.smithi019.stdout:(10/32): ceph-mon-18.2.2-1237.g646b8c7d.el8.x86 2.7 MB/s | 4.8 MB 00:01 2024-04-19T23:13:46.650 INFO:teuthology.orchestra.run.smithi019.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2024-04-19T23:13:46.709 INFO:teuthology.orchestra.run.smithi019.stdout:(12/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2024-04-19T23:13:46.809 INFO:teuthology.orchestra.run.smithi019.stdout:(13/32): python3-logutils-0.3.5-11.el8.noarch.r 265 kB/s | 49 kB 00:00 2024-04-19T23:13:46.834 INFO:teuthology.orchestra.run.smithi019.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.2 MB/s | 283 kB 00:00 2024-04-19T23:13:46.868 INFO:teuthology.orchestra.run.smithi019.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2024-04-19T23:13:46.893 INFO:teuthology.orchestra.run.smithi019.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 319 kB/s | 19 kB 00:00 2024-04-19T23:13:46.927 INFO:teuthology.orchestra.run.smithi019.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 408 kB/s | 24 kB 00:00 2024-04-19T23:13:46.952 INFO:teuthology.orchestra.run.smithi019.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 500 kB/s | 29 kB 00:00 2024-04-19T23:13:46.986 INFO:teuthology.orchestra.run.smithi019.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 471 kB/s | 27 kB 00:00 2024-04-19T23:13:47.045 INFO:teuthology.orchestra.run.smithi019.stdout:(20/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.2 MB/s | 251 kB 00:00 2024-04-19T23:13:47.095 INFO:teuthology.orchestra.run.smithi019.stdout:(21/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.8 MB/s | 255 kB 00:00 2024-04-19T23:13:47.137 INFO:teuthology.orchestra.run.smithi019.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 930 kB/s | 85 kB 00:00 2024-04-19T23:13:47.162 INFO:teuthology.orchestra.run.smithi019.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 344 kB/s | 23 kB 00:00 2024-04-19T23:13:47.229 INFO:teuthology.orchestra.run.smithi019.stdout:(24/32): python3-markupsafe-0.23-19.el8.x86_64. 584 kB/s | 39 kB 00:00 2024-04-19T23:13:47.254 INFO:teuthology.orchestra.run.smithi019.stdout:(25/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2024-04-19T23:13:47.280 INFO:teuthology.orchestra.run.smithi019.stdout:(26/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 2.0 MB/s | 103 kB 00:00 2024-04-19T23:13:47.305 INFO:teuthology.orchestra.run.smithi019.stdout:(27/32): python3-mako-1.0.6-14.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2024-04-19T23:13:47.314 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 28/32 2024-04-19T23:13:47.330 INFO:teuthology.orchestra.run.smithi019.stdout:(28/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2024-04-19T23:13:47.355 INFO:teuthology.orchestra.run.smithi019.stdout:(29/32): python3-pycparser-2.14-14.el8.noarch.r 2.1 MB/s | 109 kB 00:00 2024-04-19T23:13:47.389 INFO:teuthology.orchestra.run.smithi019.stdout:(30/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2024-04-19T23:13:47.423 INFO:teuthology.orchestra.run.smithi019.stdout:(31/32): python3-cryptography-3.2.1-7.el8_9.x86 8.1 MB/s | 559 kB 00:00 2024-04-19T23:13:47.497 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/32 2024-04-19T23:13:48.029 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/32 2024-04-19T23:13:48.177 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 30/32 2024-04-19T23:13:48.340 INFO:teuthology.orchestra.run.smithi019.stdout:(32/32): ceph-osd-18.2.2-1237.g646b8c7d.el8.x86 7.6 MB/s | 18 MB 00:02 2024-04-19T23:13:48.342 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:13:48.343 INFO:teuthology.orchestra.run.smithi019.stdout:Total 7.7 MB/s | 31 MB 00:04 2024-04-19T23:13:48.416 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 30/32 2024-04-19T23:13:48.445 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:13:48.486 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:13:48.486 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:13:48.569 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 31/32 2024-04-19T23:13:48.591 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 31/32 2024-04-19T23:13:48.702 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 32/32 2024-04-19T23:13:48.740 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:13:48.750 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:13:49.126 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 32/32 2024-04-19T23:13:49.126 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/32 2024-04-19T23:13:49.126 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/32 2024-04-19T23:13:49.126 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/32 2024-04-19T23:13:49.126 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/32 2024-04-19T23:13:49.126 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/32 2024-04-19T23:13:49.126 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1237.g646b8c7d.el8. 6/32 2024-04-19T23:13:49.126 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-19T23:13:49.126 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-19T23:13:49.126 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-19T23:13:49.127 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-19T23:13:49.128 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2024-04-19T23:13:49.128 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 26/32 2024-04-19T23:13:49.128 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 27/32 2024-04-19T23:13:49.128 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2024-04-19T23:13:49.129 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2024-04-19T23:13:49.129 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 30/32 2024-04-19T23:13:49.129 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 31/32 2024-04-19T23:13:49.888 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:13:50.152 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-19T23:13:50.363 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-19T23:13:50.587 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2024-04-19T23:13:50.969 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 4/32 2024-04-19T23:13:51.200 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 5/32 2024-04-19T23:13:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2024-04-19T23:13:51.521 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2024-04-19T23:13:51.744 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2024-04-19T23:13:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 9/32 2024-04-19T23:13:52.157 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 10/32 2024-04-19T23:13:52.514 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 11/32 2024-04-19T23:13:52.762 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 12/32 2024-04-19T23:13:53.047 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-19T23:13:53.243 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-19T23:13:53.407 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-19T23:13:53.606 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-19T23:13:53.807 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-19T23:13:54.027 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-19T23:13:54.252 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-19T23:13:54.473 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-19T23:13:54.682 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-19T23:13:54.689 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 32/32 2024-04-19T23:13:54.689 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:13:54.964 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-19T23:13:55.212 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-19T23:13:55.298 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:13:55.298 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:13:55.298 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:55.298 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:55.298 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:55.298 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:13:55.298 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:55.298 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:13:55.299 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-19T23:13:55.299 INFO:teuthology.orchestra.run.smithi031.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-19T23:13:55.299 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-19T23:13:55.299 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-19T23:13:55.299 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-19T23:13:55.299 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-19T23:13:55.299 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-19T23:13:55.299 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-19T23:13:55.299 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-19T23:13:55.299 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-19T23:13:55.299 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-19T23:13:55.299 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-2.6-1.el8.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-19T23:13:55.300 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-19T23:13:55.301 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-19T23:13:55.301 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:13:55.301 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:13:55.456 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-19T23:13:55.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-base 2024-04-19T23:13:55.790 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-19T23:13:55.898 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:13:56.118 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-19T23:13:56.529 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1237.g646b8c7d.el8. 27/32 2024-04-19T23:13:56.725 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 28/32 2024-04-19T23:13:56.770 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 125 kB/s | 4.1 kB 00:00 2024-04-19T23:13:57.044 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 116 kB/s | 4.5 kB 00:00 2024-04-19T23:13:58.549 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 28/32 2024-04-19T23:13:58.908 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/32 2024-04-19T23:13:59.445 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 29/32 2024-04-19T23:13:59.651 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 30/32 2024-04-19T23:13:59.837 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:13:59.890 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 30/32 2024-04-19T23:14:00.048 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-19T23:14:00.049 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:14:00.184 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 31/32 2024-04-19T23:14:00.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install cephadm 2024-04-19T23:14:00.209 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 31/32 2024-04-19T23:14:00.334 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 32/32 2024-04-19T23:14:00.588 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:14:01.331 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 32/32 2024-04-19T23:14:01.331 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/32 2024-04-19T23:14:01.331 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/32 2024-04-19T23:14:01.331 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 3/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 4/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 5/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1237.g646b8c7d.el8. 6/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-19T23:14:01.332 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-19T23:14:01.333 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-19T23:14:01.333 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-19T23:14:01.333 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-19T23:14:01.333 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-19T23:14:01.333 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-19T23:14:01.333 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-19T23:14:01.333 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-19T23:14:01.333 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/32 2024-04-19T23:14:01.333 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/32 2024-04-19T23:14:01.333 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 27/32 2024-04-19T23:14:01.333 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 28/32 2024-04-19T23:14:01.333 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 29/32 2024-04-19T23:14:01.334 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 30/32 2024-04-19T23:14:01.334 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 31/32 2024-04-19T23:14:01.477 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 105 kB/s | 4.1 kB 00:00 2024-04-19T23:14:01.759 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 123 kB/s | 4.5 kB 00:00 2024-04-19T23:14:04.771 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:14:04.771 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:14:04.771 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-19T23:14:04.771 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:14:04.771 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:14:04.771 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 170 k 2024-04-19T23:14:04.771 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:04.771 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:14:04.771 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:14:04.772 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-19T23:14:04.772 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:04.772 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 170 k 2024-04-19T23:14:04.772 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 692 k 2024-04-19T23:14:04.772 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:14:05.138 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm-18.2.2-1237.g646b8c7d.el8.noarch.rpm 464 kB/s | 170 kB 00:00 2024-04-19T23:14:05.139 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:14:05.139 INFO:teuthology.orchestra.run.smithi031.stdout:Total 462 kB/s | 170 kB 00:00 2024-04-19T23:14:05.139 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:14:05.150 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:14:05.150 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:14:05.153 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:14:05.163 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:14:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:14:07.157 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-19T23:14:07.323 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-19T23:14:07.546 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 32/32 2024-04-19T23:14:07.546 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:14:07.558 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-19T23:14:08.343 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:08.343 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:14:08.343 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:14:08.343 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-19T23:14:08.344 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend-2.6-1.el8.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-19T23:14:08.345 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-19T23:14:08.346 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-19T23:14:08.346 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:08.346 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:14:08.544 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base 2024-04-19T23:14:08.948 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:14:09.708 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 127 kB/s | 4.5 kB 00:00 2024-04-19T23:14:10.336 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 109 kB/s | 4.1 kB 00:00 2024-04-19T23:14:12.705 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-19T23:14:12.705 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:14:12.759 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-base-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:14:12.967 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:14:12.968 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-04-19T23:14:12.969 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:14:13.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install cephadm 2024-04-19T23:14:13.319 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:13.319 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:14:13.319 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:14:13.319 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:13.319 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:14:13.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-immutable-object-cache 2024-04-19T23:14:13.516 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:14:13.907 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:14:14.373 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 111 kB/s | 4.5 kB 00:00 2024-04-19T23:14:14.803 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 102 kB/s | 4.1 kB 00:00 2024-04-19T23:14:14.990 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 100 kB/s | 4.1 kB 00:00 2024-04-19T23:14:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 113 kB/s | 4.5 kB 00:00 2024-04-19T23:14:17.694 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:14:17.694 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:14:17.695 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-04-19T23:14:17.695 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:14:17.695 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:14:17.695 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 170 k 2024-04-19T23:14:17.695 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:17.695 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:14:17.695 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:14:17.695 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-04-19T23:14:17.695 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:17.696 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 170 k 2024-04-19T23:14:17.696 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 692 k 2024-04-19T23:14:17.696 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:14:17.882 INFO:teuthology.orchestra.run.smithi019.stdout:cephadm-18.2.2-1237.g646b8c7d.el8.noarch.rpm 916 kB/s | 170 kB 00:00 2024-04-19T23:14:17.883 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:14:17.883 INFO:teuthology.orchestra.run.smithi019.stdout:Total 909 kB/s | 170 kB 00:00 2024-04-19T23:14:17.883 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:14:17.893 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:14:17.893 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:14:17.895 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:14:17.905 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:14:17.959 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:14:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:14:18.114 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:14:18.114 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-04-19T23:14:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:14:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:14:18.115 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 166 k 2024-04-19T23:14:18.115 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:14:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:14:18.115 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-19T23:14:18.115 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 166 k 2024-04-19T23:14:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 503 k 2024-04-19T23:14:18.116 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:14:18.305 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-immutable-object-cache-18.2.2-1237.g646b8c 879 kB/s | 166 kB 00:00 2024-04-19T23:14:18.305 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:14:18.305 INFO:teuthology.orchestra.run.smithi031.stdout:Total 873 kB/s | 166 kB 00:00 2024-04-19T23:14:18.306 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:14:18.320 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:14:18.320 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:14:18.382 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:14:18.392 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:14:18.518 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:14:18.689 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d. 1/1 2024-04-19T23:14:19.148 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d. 1/1 2024-04-19T23:14:19.973 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-19T23:14:20.159 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-19T23:14:20.389 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-19T23:14:24.757 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d. 1/1 2024-04-19T23:14:24.757 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:14:25.382 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:25.382 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:14:25.382 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:14:25.382 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:25.383 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:14:25.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr 2024-04-19T23:14:25.848 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/1 2024-04-19T23:14:25.848 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:14:25.949 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:14:26.465 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:26.466 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:14:26.466 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:14:26.466 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:26.466 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:14:26.642 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-immutable-object-cache 2024-04-19T23:14:26.853 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 95 kB/s | 4.1 kB 00:00 2024-04-19T23:14:27.058 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:14:27.152 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 106 kB/s | 4.5 kB 00:00 2024-04-19T23:14:27.882 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 113 kB/s | 4.5 kB 00:00 2024-04-19T23:14:28.518 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 96 kB/s | 4.1 kB 00:00 2024-04-19T23:14:29.962 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:14:30.174 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:14:30.175 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-19T23:14:30.175 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:14:30.477 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-dashboard 2024-04-19T23:14:30.877 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:14:31.253 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:14:31.253 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:14:31.253 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-04-19T23:14:31.254 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:14:31.254 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:14:31.254 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 166 k 2024-04-19T23:14:31.254 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:31.254 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:14:31.254 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:14:31.254 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-04-19T23:14:31.254 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:31.255 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 166 k 2024-04-19T23:14:31.255 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 503 k 2024-04-19T23:14:31.255 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:14:31.443 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-immutable-object-cache-18.2.2-1237.g646b8c 882 kB/s | 166 kB 00:00 2024-04-19T23:14:31.444 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:14:31.444 INFO:teuthology.orchestra.run.smithi019.stdout:Total 875 kB/s | 166 kB 00:00 2024-04-19T23:14:31.444 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:14:31.458 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:14:31.459 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:14:31.522 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:14:31.532 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:14:31.653 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:14:31.776 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 100 kB/s | 4.1 kB 00:00 2024-04-19T23:14:31.796 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d. 1/1 2024-04-19T23:14:32.062 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 105 kB/s | 4.5 kB 00:00 2024-04-19T23:14:32.241 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d. 1/1 2024-04-19T23:14:35.059 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:14:35.059 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 3.5 M 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 24 k 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 15 k 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:14:35.060 INFO:teuthology.orchestra.run.smithi031.stdout:Install 5 Packages 2024-04-19T23:14:35.061 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.8 M 2024-04-19T23:14:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 87 M 2024-04-19T23:14:35.061 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:14:35.627 INFO:teuthology.orchestra.run.smithi031.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1237.g646 109 kB/s | 24 kB 00:00 2024-04-19T23:14:35.652 INFO:teuthology.orchestra.run.smithi031.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1237.g646b 62 kB/s | 15 kB 00:00 2024-04-19T23:14:35.778 INFO:teuthology.orchestra.run.smithi031.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 224 kB/s | 34 kB 00:00 2024-04-19T23:14:35.836 INFO:teuthology.orchestra.run.smithi031.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2024-04-19T23:14:36.078 INFO:teuthology.orchestra.run.smithi031.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1237.g646b8c7d 5.2 MB/s | 3.5 MB 00:00 2024-04-19T23:14:36.081 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:14:36.081 INFO:teuthology.orchestra.run.smithi031.stdout:Total 3.7 MB/s | 3.8 MB 00:01 2024-04-19T23:14:36.081 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:14:36.091 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:14:36.091 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:14:36.174 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:14:36.184 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:14:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:14:36.655 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-19T23:14:36.831 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-19T23:14:36.944 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1237.g646b8c7d.el8.n 3/5 2024-04-19T23:14:37.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d. 1/1 2024-04-19T23:14:37.623 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:14:38.187 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1237.g646b8c7d.el8. 4/5 2024-04-19T23:14:38.243 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:38.243 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:14:38.243 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:14:38.243 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:38.244 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:14:38.413 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarc 5/5 2024-04-19T23:14:38.421 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr 2024-04-19T23:14:38.707 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarc 5/5 2024-04-19T23:14:38.707 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1237.g646b8c7d.el8. 1/5 2024-04-19T23:14:38.708 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarc 2/5 2024-04-19T23:14:38.708 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1237.g646b8c7d.el8.n 3/5 2024-04-19T23:14:38.708 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-19T23:14:38.827 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:14:40.732 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-19T23:14:41.411 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 kB/s | 4.1 kB 00:00 2024-04-19T23:14:43.913 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-mgr-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:14:44.124 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:14:44.126 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-04-19T23:14:44.126 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:14:44.405 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-19T23:14:44.405 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:14:44.413 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-dashboard 2024-04-19T23:14:44.811 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:14:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:45.038 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:14:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:14:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:14:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:14:45.038 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-19T23:14:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-19T23:14:45.039 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:45.039 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:14:45.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-19T23:14:45.631 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:14:45.672 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 108 kB/s | 4.5 kB 00:00 2024-04-19T23:14:46.290 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 118 kB/s | 4.1 kB 00:00 2024-04-19T23:14:46.548 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 108 kB/s | 4.1 kB 00:00 2024-04-19T23:14:46.833 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 115 kB/s | 4.5 kB 00:00 2024-04-19T23:14:48.959 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:14:48.960 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:14:48.960 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-04-19T23:14:48.960 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:14:48.961 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:14:48.961 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 3.5 M 2024-04-19T23:14:48.961 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-04-19T23:14:48.961 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 24 k 2024-04-19T23:14:48.961 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 15 k 2024-04-19T23:14:48.961 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-19T23:14:48.961 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-19T23:14:48.961 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:48.961 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:14:48.961 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:14:48.961 INFO:teuthology.orchestra.run.smithi019.stdout:Install 5 Packages 2024-04-19T23:14:48.961 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:48.962 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 3.8 M 2024-04-19T23:14:48.962 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 87 M 2024-04-19T23:14:48.962 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:14:49.189 INFO:teuthology.orchestra.run.smithi019.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1237.g646b 108 kB/s | 15 kB 00:00 2024-04-19T23:14:49.281 INFO:teuthology.orchestra.run.smithi019.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1237.g646 104 kB/s | 24 kB 00:00 2024-04-19T23:14:49.423 INFO:teuthology.orchestra.run.smithi019.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 144 kB/s | 34 kB 00:00 2024-04-19T23:14:49.515 INFO:teuthology.orchestra.run.smithi019.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 859 kB/s | 196 kB 00:00 2024-04-19T23:14:49.833 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:14:49.835 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================================= 2024-04-19T23:14:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-19T23:14:49.835 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================================= 2024-04-19T23:14:49.835 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:14:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 7.4 M 2024-04-19T23:14:49.835 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-19T23:14:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-19T23:14:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-19T23:14:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-19T23:14:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-19T23:14:49.835 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-19T23:14:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-19T23:14:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-19T23:14:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:49.836 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:14:49.836 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================================= 2024-04-19T23:14:49.836 INFO:teuthology.orchestra.run.smithi031.stdout:Install 8 Packages 2024-04-19T23:14:49.836 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:14:49.837 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 36 M 2024-04-19T23:14:49.837 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 207 M 2024-04-19T23:14:49.837 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:14:49.943 INFO:teuthology.orchestra.run.smithi031.stdout:(1/8): libquadmath-8.5.0-20.el8.x86_64.rpm 1.6 MB/s | 172 kB 00:00 2024-04-19T23:14:49.977 INFO:teuthology.orchestra.run.smithi031.stdout:(2/8): libgfortran-8.5.0-20.el8.x86_64.rpm 4.5 MB/s | 645 kB 00:00 2024-04-19T23:14:50.036 INFO:teuthology.orchestra.run.smithi031.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 3.8 MB/s | 225 kB 00:00 2024-04-19T23:14:50.103 INFO:teuthology.orchestra.run.smithi031.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 23 MB/s | 3.7 MB 00:00 2024-04-19T23:14:50.245 INFO:teuthology.orchestra.run.smithi031.stdout:(5/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 25 MB/s | 5.2 MB 00:00 2024-04-19T23:14:50.345 INFO:teuthology.orchestra.run.smithi031.stdout:(6/8): openblas-0.3.15-6.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-04-19T23:14:50.440 INFO:teuthology.orchestra.run.smithi019.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1237.g646b8c7d 2.5 MB/s | 3.5 MB 00:01 2024-04-19T23:14:50.444 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:14:50.444 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.6 MB/s | 3.8 MB 00:01 2024-04-19T23:14:50.444 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:14:50.454 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:14:50.454 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:14:50.537 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:14:50.547 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:14:50.612 INFO:teuthology.orchestra.run.smithi031.stdout:(7/8): python3-scipy-1.0.0-21.module+el8.9.0+20 37 MB/s | 14 MB 00:00 2024-04-19T23:14:50.789 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:14:51.037 INFO:teuthology.orchestra.run.smithi031.stdout:(8/8): ceph-mgr-diskprediction-local-18.2.2-123 6.2 MB/s | 7.4 MB 00:01 2024-04-19T23:14:51.038 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:14:51.038 INFO:teuthology.orchestra.run.smithi031.stdout:Total 30 MB/s | 36 MB 00:01 2024-04-19T23:14:51.039 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-19T23:14:51.239 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-19T23:14:51.308 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:14:51.329 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:14:51.330 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:14:51.402 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1237.g646b8c7d.el8.n 3/5 2024-04-19T23:14:51.675 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:14:51.684 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:14:52.285 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:14:52.424 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-19T23:14:52.572 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-19T23:14:52.705 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1237.g646b8c7d.el8. 4/5 2024-04-19T23:14:52.755 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-19T23:14:52.917 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarc 5/5 2024-04-19T23:14:53.231 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarc 5/5 2024-04-19T23:14:53.231 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1237.g646b8c7d.el8. 1/5 2024-04-19T23:14:53.231 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarc 2/5 2024-04-19T23:14:53.231 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1237.g646b8c7d.el8.n 3/5 2024-04-19T23:14:53.231 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-19T23:14:53.452 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-19T23:14:53.633 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-19T23:14:54.288 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-19T23:14:54.499 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-19T23:14:55.258 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-19T23:14:55.574 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-19T23:14:55.741 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-19T23:14:57.708 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-19T23:14:58.468 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-19T23:14:58.468 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:14:59.085 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-19T23:14:59.092 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:59.092 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:14:59.092 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:14:59.092 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:14:59.092 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:14:59.092 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-19T23:14:59.092 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-19T23:14:59.092 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:14:59.093 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:14:59.290 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8c7 8/8 2024-04-19T23:14:59.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-19T23:14:59.623 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8c7 8/8 2024-04-19T23:14:59.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8c7 1/8 2024-04-19T23:14:59.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-19T23:14:59.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 3/8 2024-04-19T23:14:59.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-19T23:14:59.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-19T23:14:59.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 6/8 2024-04-19T23:14:59.623 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 7/8 2024-04-19T23:14:59.738 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:15:00.596 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 111 kB/s | 4.5 kB 00:00 2024-04-19T23:15:01.222 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 123 kB/s | 4.1 kB 00:00 2024-04-19T23:15:03.909 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:15:03.910 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================================================= 2024-04-19T23:15:03.910 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================================================= 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 7.4 M 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:15:03.911 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================================================= 2024-04-19T23:15:03.912 INFO:teuthology.orchestra.run.smithi019.stdout:Install 8 Packages 2024-04-19T23:15:03.912 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:15:03.912 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 36 M 2024-04-19T23:15:03.912 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 207 M 2024-04-19T23:15:03.912 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:15:04.005 INFO:teuthology.orchestra.run.smithi019.stdout:(1/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-04-19T23:15:04.113 INFO:teuthology.orchestra.run.smithi019.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2024-04-19T23:15:04.222 INFO:teuthology.orchestra.run.smithi019.stdout:(3/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 24 MB/s | 5.2 MB 00:00 2024-04-19T23:15:04.339 INFO:teuthology.orchestra.run.smithi019.stdout:(4/8): openblas-0.3.15-6.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-04-19T23:15:04.398 INFO:teuthology.orchestra.run.smithi019.stdout:(5/8): libgfortran-8.5.0-20.el8.x86_64.rpm 11 MB/s | 645 kB 00:00 2024-04-19T23:15:04.440 INFO:teuthology.orchestra.run.smithi019.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 4.0 MB/s | 172 kB 00:00 2024-04-19T23:15:04.648 INFO:teuthology.orchestra.run.smithi019.stdout:(7/8): python3-scipy-1.0.0-21.module+el8.9.0+20 32 MB/s | 14 MB 00:00 2024-04-19T23:15:04.832 INFO:teuthology.orchestra.run.smithi019.stdout:(8/8): ceph-mgr-diskprediction-local-18.2.2-123 8.1 MB/s | 7.4 MB 00:00 2024-04-19T23:15:04.833 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:15:04.833 INFO:teuthology.orchestra.run.smithi019.stdout:Total 39 MB/s | 36 MB 00:00 2024-04-19T23:15:05.012 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 8/8 2024-04-19T23:15:05.012 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:15:05.103 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:15:05.123 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:15:05.124 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:15:05.468 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:15:05.477 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:15:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:05.644 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:15:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:15:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-19T23:15:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-19T23:15:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-19T23:15:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-19T23:15:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-19T23:15:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-19T23:15:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-19T23:15:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:05.645 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:15:05.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-rook 2024-04-19T23:15:06.061 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:15:06.208 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-19T23:15:06.229 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:15:06.349 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-19T23:15:06.525 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-19T23:15:07.223 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-19T23:15:07.274 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 36 kB/s | 4.1 kB 00:00 2024-04-19T23:15:07.391 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-19T23:15:07.570 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 4.5 kB 00:00 2024-04-19T23:15:08.068 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-19T23:15:08.258 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-19T23:15:09.024 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-19T23:15:09.351 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-19T23:15:09.550 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-19T23:15:10.691 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:15:10.693 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2024-04-19T23:15:10.693 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-19T23:15:10.693 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2024-04-19T23:15:10.693 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:15:10.694 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 49 k 2024-04-19T23:15:10.694 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-19T23:15:10.694 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-19T23:15:10.694 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-19T23:15:10.694 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-19T23:15:10.694 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-19T23:15:10.694 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-19T23:15:10.694 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-19T23:15:10.694 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-19T23:15:10.694 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-19T23:15:10.694 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-19T23:15:10.694 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-19T23:15:10.695 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-19T23:15:10.695 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-19T23:15:10.695 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-19T23:15:10.695 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:10.695 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:15:10.695 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2024-04-19T23:15:10.695 INFO:teuthology.orchestra.run.smithi031.stdout:Install 14 Packages 2024-04-19T23:15:10.695 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:10.696 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.0 M 2024-04-19T23:15:10.696 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 25 M 2024-04-19T23:15:10.696 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:15:11.062 INFO:teuthology.orchestra.run.smithi031.stdout:(1/14): python3-certifi-2018.10.15-7.el8.noarch 89 kB/s | 16 kB 00:00 2024-04-19T23:15:11.087 INFO:teuthology.orchestra.run.smithi031.stdout:(2/14): python3-cachetools-3.1.1-4.el8.noarch.r 159 kB/s | 33 kB 00:00 2024-04-19T23:15:11.154 INFO:teuthology.orchestra.run.smithi031.stdout:(3/14): python3-google-auth-1.1.1-10.el8.noarch 885 kB/s | 81 kB 00:00 2024-04-19T23:15:11.196 INFO:teuthology.orchestra.run.smithi031.stdout:(4/14): ceph-mgr-rook-18.2.2-1237.g646b8c7d.el8 155 kB/s | 49 kB 00:00 2024-04-19T23:15:11.221 INFO:teuthology.orchestra.run.smithi031.stdout:(5/14): python3-rsa-4.9-2.el8.noarch.rpm 962 kB/s | 64 kB 00:00 2024-04-19T23:15:11.297 INFO:teuthology.orchestra.run.smithi031.stdout:(6/14): python3-kubernetes-11.0.0-6.el8.noarch. 5.6 MB/s | 1.2 MB 00:00 2024-04-19T23:15:11.355 INFO:teuthology.orchestra.run.smithi031.stdout:(7/14): python3-websocket-client-0.56.0-5.el8.n 386 kB/s | 61 kB 00:00 2024-04-19T23:15:11.439 INFO:teuthology.orchestra.run.smithi031.stdout:(8/14): python3-jwt-1.6.1-2.el8.noarch.rpm 200 kB/s | 43 kB 00:00 2024-04-19T23:15:11.542 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-19T23:15:11.589 INFO:teuthology.orchestra.run.smithi031.stdout:(9/14): python3-oauthlib-2.1.0-1.el8.noarch.rpm 530 kB/s | 155 kB 00:00 2024-04-19T23:15:11.714 INFO:teuthology.orchestra.run.smithi031.stdout:(10/14): python3-requests-oauthlib-1.0.0-1.el8. 119 kB/s | 43 kB 00:00 2024-04-19T23:15:11.881 INFO:teuthology.orchestra.run.smithi031.stdout:(11/14): python3-jsonpointer-1.10-11.el8.noarch 45 kB/s | 20 kB 00:00 2024-04-19T23:15:11.906 INFO:teuthology.orchestra.run.smithi031.stdout:(12/14): python3-pyasn1-modules-0.3.7-6.el8.noa 347 kB/s | 110 kB 00:00 2024-04-19T23:15:11.998 INFO:teuthology.orchestra.run.smithi031.stdout:(13/14): python3-jsonpatch-1.21-2.el8.noarch.rp 94 kB/s | 27 kB 00:00 2024-04-19T23:15:12.131 INFO:teuthology.orchestra.run.smithi031.stdout:(14/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 505 kB/s | 126 kB 00:00 2024-04-19T23:15:12.135 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:15:12.135 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-04-19T23:15:12.221 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:15:12.240 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:15:12.240 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:15:12.335 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:15:12.346 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:15:12.887 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-19T23:15:12.895 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:15:13.066 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8c7 8/8 2024-04-19T23:15:13.102 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-19T23:15:13.365 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-19T23:15:13.400 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8c7 8/8 2024-04-19T23:15:13.400 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8c7 1/8 2024-04-19T23:15:13.400 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 2/8 2024-04-19T23:15:13.400 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 3/8 2024-04-19T23:15:13.400 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 4/8 2024-04-19T23:15:13.400 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 5/8 2024-04-19T23:15:13.401 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 6/8 2024-04-19T23:15:13.401 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2024-04-19T23:15:13.537 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-19T23:15:13.688 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/14 2024-04-19T23:15:13.852 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/14 2024-04-19T23:15:14.080 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-04-19T23:15:14.372 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-04-19T23:15:14.586 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-04-19T23:15:14.796 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-19T23:15:15.004 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-19T23:15:15.266 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-19T23:15:16.447 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-19T23:15:16.669 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-19T23:15:16.879 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 14/14 2024-04-19T23:15:17.179 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 14/14 2024-04-19T23:15:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 1/14 2024-04-19T23:15:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-19T23:15:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-19T23:15:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-19T23:15:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-19T23:15:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-19T23:15:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-19T23:15:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-04-19T23:15:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-04-19T23:15:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-04-19T23:15:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/14 2024-04-19T23:15:17.180 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/14 2024-04-19T23:15:17.181 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-04-19T23:15:19.047 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2024-04-19T23:15:19.048 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:15:19.686 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:15:19.686 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:15:19.686 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:15:19.687 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-19T23:15:19.687 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-19T23:15:19.687 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-19T23:15:19.687 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-19T23:15:19.687 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-19T23:15:19.687 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-19T23:15:19.687 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-19T23:15:19.687 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:15:19.687 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:15:19.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-rook 2024-04-19T23:15:20.279 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:15:21.134 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 4.5 kB 00:00 2024-04-19T23:15:21.862 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-19T23:15:23.003 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 14/14 2024-04-19T23:15:23.004 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:15:23.626 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:23.627 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:15:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:15:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-19T23:15:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-19T23:15:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-19T23:15:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-19T23:15:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-19T23:15:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-19T23:15:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-19T23:15:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-19T23:15:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-19T23:15:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-19T23:15:23.628 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-19T23:15:23.628 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-19T23:15:23.628 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-19T23:15:23.628 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:23.628 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:15:23.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-cephadm 2024-04-19T23:15:24.343 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:15:24.633 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:15:24.635 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================================== 2024-04-19T23:15:24.635 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================================== 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 49 k 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-19T23:15:24.636 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-19T23:15:24.637 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-19T23:15:24.637 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-19T23:15:24.637 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-19T23:15:24.637 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:15:24.637 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:15:24.637 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================================== 2024-04-19T23:15:24.637 INFO:teuthology.orchestra.run.smithi019.stdout:Install 14 Packages 2024-04-19T23:15:24.637 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:15:24.638 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.0 M 2024-04-19T23:15:24.638 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 25 M 2024-04-19T23:15:24.638 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:15:25.130 INFO:teuthology.orchestra.run.smithi019.stdout:(1/14): python3-certifi-2018.10.15-7.el8.noarch 156 kB/s | 16 kB 00:00 2024-04-19T23:15:25.155 INFO:teuthology.orchestra.run.smithi019.stdout:(2/14): python3-cachetools-3.1.1-4.el8.noarch.r 254 kB/s | 33 kB 00:00 2024-04-19T23:15:25.197 INFO:teuthology.orchestra.run.smithi019.stdout:(3/14): python3-google-auth-1.1.1-10.el8.noarch 1.2 MB/s | 81 kB 00:00 2024-04-19T23:15:25.239 INFO:teuthology.orchestra.run.smithi019.stdout:(4/14): python3-rsa-4.9-2.el8.noarch.rpm 1.5 MB/s | 64 kB 00:00 2024-04-19T23:15:25.258 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 94 kB/s | 4.1 kB 00:00 2024-04-19T23:15:25.281 INFO:teuthology.orchestra.run.smithi019.stdout:(5/14): python3-websocket-client-0.56.0-5.el8.n 1.4 MB/s | 61 kB 00:00 2024-04-19T23:15:25.315 INFO:teuthology.orchestra.run.smithi019.stdout:(6/14): ceph-mgr-rook-18.2.2-1237.g646b8c7d.el8 170 kB/s | 49 kB 00:00 2024-04-19T23:15:25.349 INFO:teuthology.orchestra.run.smithi019.stdout:(7/14): python3-kubernetes-11.0.0-6.el8.noarch. 6.0 MB/s | 1.2 MB 00:00 2024-04-19T23:15:25.415 INFO:teuthology.orchestra.run.smithi019.stdout:(8/14): python3-jsonpointer-1.10-11.el8.noarch. 149 kB/s | 20 kB 00:00 2024-04-19T23:15:25.441 INFO:teuthology.orchestra.run.smithi019.stdout:(9/14): python3-jsonpatch-1.21-2.el8.noarch.rpm 357 kB/s | 27 kB 00:00 2024-04-19T23:15:25.466 INFO:teuthology.orchestra.run.smithi019.stdout:(10/14): python3-pyasn1-modules-0.3.7-6.el8.noa 731 kB/s | 110 kB 00:00 2024-04-19T23:15:25.491 INFO:teuthology.orchestra.run.smithi019.stdout:(11/14): python3-jwt-1.6.1-2.el8.noarch.rpm 851 kB/s | 43 kB 00:00 2024-04-19T23:15:25.517 INFO:teuthology.orchestra.run.smithi019.stdout:(12/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2024-04-19T23:15:25.541 INFO:teuthology.orchestra.run.smithi019.stdout:(13/14): python3-requests-oauthlib-1.0.0-1.el8. 855 kB/s | 43 kB 00:00 2024-04-19T23:15:25.547 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 112 kB/s | 4.5 kB 00:00 2024-04-19T23:15:25.567 INFO:teuthology.orchestra.run.smithi019.stdout:(14/14): python3-oauthlib-2.1.0-1.el8.noarch.rp 1.5 MB/s | 155 kB 00:00 2024-04-19T23:15:25.568 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:15:25.568 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-04-19T23:15:25.656 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:15:25.673 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:15:25.674 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:15:25.768 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:15:25.778 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:15:26.328 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:15:27.263 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-19T23:15:27.525 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-19T23:15:27.729 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-19T23:15:27.973 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 4/14 2024-04-19T23:15:28.240 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 5/14 2024-04-19T23:15:28.455 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 6/14 2024-04-19T23:15:28.591 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:15:28.593 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================================== 2024-04-19T23:15:28.593 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-19T23:15:28.593 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================================== 2024-04-19T23:15:28.593 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:15:28.593 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 138 k 2024-04-19T23:15:28.593 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-19T23:15:28.593 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-19T23:15:28.593 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-19T23:15:28.593 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-19T23:15:28.593 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-19T23:15:28.593 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:28.594 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:15:28.594 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================================== 2024-04-19T23:15:28.594 INFO:teuthology.orchestra.run.smithi031.stdout:Install 5 Packages 2024-04-19T23:15:28.594 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:28.594 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 6.0 M 2024-04-19T23:15:28.594 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 26 M 2024-04-19T23:15:28.594 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:15:28.639 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 7/14 2024-04-19T23:15:28.821 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 8/14 2024-04-19T23:15:29.064 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-19T23:15:29.289 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-19T23:15:29.542 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-19T23:15:29.583 INFO:teuthology.orchestra.run.smithi031.stdout:(1/5): ceph-mgr-cephadm-18.2.2-1237.g646b8c7d.e 202 kB/s | 138 kB 00:00 2024-04-19T23:15:29.955 INFO:teuthology.orchestra.run.smithi031.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 13 MB/s | 4.8 MB 00:00 2024-04-19T23:15:30.268 INFO:teuthology.orchestra.run.smithi031.stdout:(3/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.7 MB/s | 538 kB 00:00 2024-04-19T23:15:30.527 INFO:teuthology.orchestra.run.smithi031.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 37 kB/s | 60 kB 00:01 2024-04-19T23:15:30.724 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-19T23:15:30.995 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-19T23:15:31.136 INFO:teuthology.orchestra.run.smithi031.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 213 kB/s | 476 kB 00:02 2024-04-19T23:15:31.139 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:15:31.139 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.3 MB/s | 6.0 MB 00:02 2024-04-19T23:15:31.180 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 14/14 2024-04-19T23:15:31.200 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:15:31.214 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:15:31.214 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:15:31.315 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:15:31.325 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:15:31.499 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 14/14 2024-04-19T23:15:31.499 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 1/14 2024-04-19T23:15:31.499 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-19T23:15:31.499 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-19T23:15:31.499 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-19T23:15:31.499 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-19T23:15:31.499 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-19T23:15:31.499 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-19T23:15:31.499 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 8/14 2024-04-19T23:15:31.499 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/14 2024-04-19T23:15:31.499 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/14 2024-04-19T23:15:31.500 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/14 2024-04-19T23:15:31.500 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 12/14 2024-04-19T23:15:31.500 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 13/14 2024-04-19T23:15:32.407 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:15:32.757 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-19T23:15:32.991 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-19T23:15:33.318 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-19T23:15:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-19T23:15:33.742 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 5/5 2024-04-19T23:15:34.060 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 5/5 2024-04-19T23:15:34.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/5 2024-04-19T23:15:34.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-19T23:15:34.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-19T23:15:34.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-19T23:15:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 14/14 2024-04-19T23:15:37.021 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:15:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:15:37.662 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:15:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:15:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-19T23:15:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:15:37.663 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:15:38.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-cephadm 2024-04-19T23:15:38.788 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:15:39.606 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-19T23:15:39.606 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:15:39.742 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 112 kB/s | 4.5 kB 00:00 2024-04-19T23:15:40.226 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:40.226 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:15:40.226 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:15:40.227 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-19T23:15:40.227 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-19T23:15:40.227 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-19T23:15:40.227 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-19T23:15:40.227 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:40.227 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:15:40.393 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 4.1 kB 00:00 2024-04-19T23:15:40.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-fuse 2024-04-19T23:15:40.809 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:15:41.749 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 100 kB/s | 4.1 kB 00:00 2024-04-19T23:15:42.040 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 113 kB/s | 4.5 kB 00:00 2024-04-19T23:15:43.124 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:15:43.125 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================================== 2024-04-19T23:15:43.125 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-04-19T23:15:43.125 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================================== 2024-04-19T23:15:43.125 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:15:43.125 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 138 k 2024-04-19T23:15:43.126 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-04-19T23:15:43.126 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-19T23:15:43.126 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-19T23:15:43.126 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-19T23:15:43.126 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-19T23:15:43.126 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:15:43.126 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:15:43.126 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================================== 2024-04-19T23:15:43.126 INFO:teuthology.orchestra.run.smithi019.stdout:Install 5 Packages 2024-04-19T23:15:43.126 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:15:43.127 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 6.0 M 2024-04-19T23:15:43.127 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 26 M 2024-04-19T23:15:43.127 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:15:43.846 INFO:teuthology.orchestra.run.smithi019.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 111 kB/s | 60 kB 00:00 2024-04-19T23:15:44.063 INFO:teuthology.orchestra.run.smithi019.stdout:(2/5): ceph-mgr-cephadm-18.2.2-1237.g646b8c7d.e 181 kB/s | 138 kB 00:00 2024-04-19T23:15:44.356 INFO:teuthology.orchestra.run.smithi019.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 451 kB/s | 476 kB 00:01 2024-04-19T23:15:44.615 INFO:teuthology.orchestra.run.smithi019.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 992 kB/s | 538 kB 00:00 2024-04-19T23:15:45.015 INFO:teuthology.orchestra.run.smithi019.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 4.1 MB/s | 4.8 MB 00:01 2024-04-19T23:15:45.016 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:15:45.016 INFO:teuthology.orchestra.run.smithi019.stdout:Total 3.2 MB/s | 6.0 MB 00:01 2024-04-19T23:15:45.078 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:15:45.084 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:15:45.085 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:15:45.085 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-19T23:15:45.085 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:15:45.086 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:15:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse 2024-04-19T23:15:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 859 k 2024-04-19T23:15:45.086 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-19T23:15:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-19T23:15:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:45.086 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:15:45.086 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:15:45.086 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2024-04-19T23:15:45.086 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:45.087 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 942 k 2024-04-19T23:15:45.087 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 3.0 M 2024-04-19T23:15:45.087 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:15:45.091 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:15:45.092 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:15:45.192 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:15:45.202 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:15:45.316 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 364 kB/s | 83 kB 00:00 2024-04-19T23:15:45.557 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-fuse-18.2.2-1237.g646b8c7d.el8.x86_ 1.8 MB/s | 859 kB 00:00 2024-04-19T23:15:45.558 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:15:45.559 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.9 MB/s | 942 kB 00:00 2024-04-19T23:15:45.573 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:15:45.600 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:15:45.600 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:15:45.701 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:15:45.711 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:15:45.916 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:15:46.294 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:15:46.426 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-19T23:15:46.649 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/2 2024-04-19T23:15:46.668 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-19T23:15:46.917 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-19T23:15:47.090 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/2 2024-04-19T23:15:47.090 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/2 2024-04-19T23:15:47.269 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-19T23:15:47.543 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-19T23:15:47.728 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 5/5 2024-04-19T23:15:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 5/5 2024-04-19T23:15:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 1/5 2024-04-19T23:15:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-19T23:15:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-19T23:15:48.044 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-19T23:15:57.436 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-19T23:15:57.436 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:15:58.076 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:15:58.076 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:15:58.076 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:15:58.076 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-19T23:15:58.076 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-19T23:15:58.076 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-19T23:15:58.076 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-19T23:15:58.077 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:15:58.077 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:15:58.254 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-fuse 2024-04-19T23:15:58.302 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-19T23:15:58.302 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:15:58.665 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:15:59.038 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:59.038 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:15:59.038 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-19T23:15:59.038 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:15:59.038 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:15:59.210 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-volume 2024-04-19T23:15:59.626 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:16:00.561 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-19T23:16:01.307 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-19T23:16:01.322 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 kB/s | 4.1 kB 00:00 2024-04-19T23:16:01.695 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-04-19T23:16:04.040 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:16:04.041 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:16:04.041 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-04-19T23:16:04.041 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:16:04.041 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:16:04.041 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse 2024-04-19T23:16:04.041 INFO:teuthology.orchestra.run.smithi019.stdout: x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 859 k 2024-04-19T23:16:04.041 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-04-19T23:16:04.041 INFO:teuthology.orchestra.run.smithi019.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-19T23:16:04.042 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:04.042 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:16:04.042 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:16:04.042 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-04-19T23:16:04.042 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:04.042 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 942 k 2024-04-19T23:16:04.042 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 3.0 M 2024-04-19T23:16:04.043 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:16:04.173 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 638 kB/s | 83 kB 00:00 2024-04-19T23:16:04.473 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-fuse-18.2.2-1237.g646b8c7d.el8.x86_ 1.9 MB/s | 859 kB 00:00 2024-04-19T23:16:04.474 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:16:04.474 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.1 MB/s | 942 kB 00:00 2024-04-19T23:16:04.489 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:16:04.515 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:16:04.515 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:16:04.616 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:16:04.626 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:16:04.731 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 263 k 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 311 k 2024-04-19T23:16:04.733 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.4 M 2024-04-19T23:16:04.734 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:16:04.803 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:16:04.980 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 196 kB/s | 48 kB 00:00 2024-04-19T23:16:05.076 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-19T23:16:05.172 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-volume-18.2.2-1237.g646b8c7d.el8.no 601 kB/s | 263 kB 00:00 2024-04-19T23:16:05.173 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:16:05.173 INFO:teuthology.orchestra.run.smithi031.stdout:Total 707 kB/s | 311 kB 00:00 2024-04-19T23:16:05.188 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:16:05.202 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:16:05.202 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:16:05.226 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/2 2024-04-19T23:16:05.249 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:16:05.259 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:16:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:16:05.686 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2/2 2024-04-19T23:16:05.686 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/2 2024-04-19T23:16:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-19T23:16:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 2/2 2024-04-19T23:16:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 2/2 2024-04-19T23:16:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 1/2 2024-04-19T23:16:11.666 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-19T23:16:11.666 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:16:12.300 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:12.300 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:16:12.300 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-19T23:16:12.300 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:12.300 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:16:12.442 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-19T23:16:12.442 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:16:12.480 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-volume 2024-04-19T23:16:12.897 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:16:13.239 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:16:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:16:13.240 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:16:13.240 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-19T23:16:13.240 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:16:13.240 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:16:13.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados-devel 2024-04-19T23:16:13.829 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:16:14.051 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-19T23:16:14.793 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-19T23:16:15.813 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 37 kB/s | 4.1 kB 00:00 2024-04-19T23:16:16.215 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-19T23:16:17.502 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:16:17.503 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================================== 2024-04-19T23:16:17.503 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-04-19T23:16:17.503 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================================== 2024-04-19T23:16:17.503 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:16:17.503 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume noarch 2:18.2.2-1237.g646b8c7d.el8 ceph-noarch 263 k 2024-04-19T23:16:17.503 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-04-19T23:16:17.503 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-19T23:16:17.503 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:17.504 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:16:17.504 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================================== 2024-04-19T23:16:17.504 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-04-19T23:16:17.504 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:17.504 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 311 k 2024-04-19T23:16:17.504 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.4 M 2024-04-19T23:16:17.504 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:16:17.678 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 278 kB/s | 48 kB 00:00 2024-04-19T23:16:17.729 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-volume-18.2.2-1237.g646b8c7d.el8.no 1.1 MB/s | 263 kB 00:00 2024-04-19T23:16:17.729 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:16:17.730 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.3 MB/s | 311 kB 00:00 2024-04-19T23:16:17.746 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:16:17.760 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:16:17.760 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:16:17.807 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:16:17.817 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:16:17.994 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:16:18.341 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-19T23:16:18.566 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 2/2 2024-04-19T23:16:19.022 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 2/2 2024-04-19T23:16:19.022 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 1/2 2024-04-19T23:16:19.240 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:16:19.241 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:16:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-04-19T23:16:19.241 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:16:19.241 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:16:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 139 k 2024-04-19T23:16:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:16:19.242 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:16:19.242 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:16:19.242 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-19T23:16:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:16:19.242 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 139 k 2024-04-19T23:16:19.243 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 513 k 2024-04-19T23:16:19.243 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:16:19.761 INFO:teuthology.orchestra.run.smithi031.stdout:librados-devel-18.2.2-1237.g646b8c7d.el8.x86_64 268 kB/s | 139 kB 00:00 2024-04-19T23:16:19.762 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:16:19.762 INFO:teuthology.orchestra.run.smithi031.stdout:Total 267 kB/s | 139 kB 00:00 2024-04-19T23:16:19.762 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:16:19.770 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:16:19.770 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:16:19.839 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:16:19.848 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:16:19.987 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:16:20.151 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:16:20.445 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:16:28.980 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-19T23:16:28.980 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:16:29.593 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:29.593 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:16:29.593 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume-2:18.2.2-1237.g646b8c7d.el8.noarch 2024-04-19T23:16:29.593 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-19T23:16:29.593 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:29.593 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:16:29.740 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados-devel 2024-04-19T23:16:30.158 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:16:31.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:16:31.107 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:16:31.552 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 92 kB/s | 4.5 kB 00:00 2024-04-19T23:16:31.729 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:16:31.729 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:16:31.729 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:16:31.729 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:16:31.730 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:16:31.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs2 2024-04-19T23:16:32.279 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-19T23:16:32.301 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:16:34.130 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 98 kB/s | 4.1 kB 00:00 2024-04-19T23:16:34.412 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 144 kB/s | 4.5 kB 00:00 2024-04-19T23:16:34.998 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:16:34.999 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:16:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-04-19T23:16:34.999 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:16:34.999 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:16:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 139 k 2024-04-19T23:16:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:34.999 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:16:34.999 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:16:34.999 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-04-19T23:16:34.999 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:35.000 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 139 k 2024-04-19T23:16:35.000 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 513 k 2024-04-19T23:16:35.000 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:16:35.178 INFO:teuthology.orchestra.run.smithi019.stdout:librados-devel-18.2.2-1237.g646b8c7d.el8.x86_64 783 kB/s | 139 kB 00:00 2024-04-19T23:16:35.178 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:16:35.179 INFO:teuthology.orchestra.run.smithi019.stdout:Total 777 kB/s | 139 kB 00:00 2024-04-19T23:16:35.179 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:16:35.186 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:16:35.186 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:16:35.254 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:16:35.264 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:16:35.410 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:16:35.595 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:16:35.898 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:16:37.260 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:16:37.475 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:16:37.476 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-19T23:16:37.477 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:16:37.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs-devel 2024-04-19T23:16:38.032 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:16:39.758 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-19T23:16:40.183 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-19T23:16:43.216 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:16:43.217 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:16:43.217 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-04-19T23:16:43.217 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:16:43.217 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:16:43.217 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 31 k 2024-04-19T23:16:43.217 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:16:43.217 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:16:43.217 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:16:43.217 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-19T23:16:43.217 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:16:43.218 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 31 k 2024-04-19T23:16:43.218 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 139 k 2024-04-19T23:16:43.218 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:16:43.364 INFO:teuthology.orchestra.run.smithi031.stdout:libcephfs-devel-18.2.2-1237.g646b8c7d.el8.x86_6 212 kB/s | 31 kB 00:00 2024-04-19T23:16:43.365 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:16:43.365 INFO:teuthology.orchestra.run.smithi031.stdout:Total 210 kB/s | 31 kB 00:00 2024-04-19T23:16:43.365 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:16:43.370 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:16:43.370 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:16:43.387 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:16:43.397 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:16:43.460 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:16:43.630 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:16:43.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:16:43.630 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:16:43.906 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:16:44.270 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:44.271 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:16:44.271 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:16:44.271 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:44.271 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:16:44.436 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs2 2024-04-19T23:16:44.850 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:16:46.879 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-19T23:16:47.619 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-19T23:16:50.124 INFO:teuthology.orchestra.run.smithi019.stdout:Package libcephfs2-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:16:50.338 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:16:50.339 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-04-19T23:16:50.339 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:16:50.492 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs-devel 2024-04-19T23:16:50.910 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:16:52.796 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-19T23:16:53.478 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 47 kB/s | 4.1 kB 00:00 2024-04-19T23:16:53.493 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:16:53.494 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:16:54.128 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:16:54.128 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:16:54.128 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:16:54.128 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:16:54.128 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:16:54.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados2 2024-04-19T23:16:54.746 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:16:56.164 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:16:56.165 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:16:56.165 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-04-19T23:16:56.165 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:16:56.165 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:16:56.165 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 31 k 2024-04-19T23:16:56.165 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:56.165 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:16:56.165 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:16:56.165 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-04-19T23:16:56.165 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:16:56.166 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 31 k 2024-04-19T23:16:56.166 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 139 k 2024-04-19T23:16:56.166 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:16:56.524 INFO:teuthology.orchestra.run.smithi019.stdout:libcephfs-devel-18.2.2-1237.g646b8c7d.el8.x86_6 86 kB/s | 31 kB 00:00 2024-04-19T23:16:56.524 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:16:56.525 INFO:teuthology.orchestra.run.smithi019.stdout:Total 86 kB/s | 31 kB 00:00 2024-04-19T23:16:56.525 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:16:56.530 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:16:56.530 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:16:56.548 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:16:56.558 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:16:56.625 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:16:56.625 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-19T23:16:56.809 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:16:57.072 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 24 kB/s | 4.5 kB 00:00 2024-04-19T23:16:57.093 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:16:59.985 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:17:00.197 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:17:00.199 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-19T23:17:00.199 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:17:00.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librbd1 2024-04-19T23:17:00.769 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:02.797 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-19T23:17:03.165 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 kB/s | 4.5 kB 00:00 2024-04-19T23:17:04.150 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:17:04.150 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:17:04.787 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:17:04.787 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:17:04.787 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:17:04.787 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:17:04.788 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:17:04.957 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados2 2024-04-19T23:17:05.374 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:05.984 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:17:06.196 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:17:06.197 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-19T23:17:06.197 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:17:06.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rados 2024-04-19T23:17:06.604 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 53 kB/s | 4.5 kB 00:00 2024-04-19T23:17:06.767 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:07.319 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-19T23:17:08.744 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 22 kB/s | 4.1 kB 00:00 2024-04-19T23:17:09.130 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-04-19T23:17:09.802 INFO:teuthology.orchestra.run.smithi019.stdout:Package librados2-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:17:10.016 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:17:10.017 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-04-19T23:17:10.017 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:17:10.172 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librbd1 2024-04-19T23:17:10.584 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:11.952 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:17:12.163 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:17:12.165 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-19T23:17:12.165 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:17:12.314 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rgw 2024-04-19T23:17:12.654 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-04-19T23:17:12.732 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:13.394 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-19T23:17:14.637 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-04-19T23:17:15.015 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 kB/s | 4.5 kB 00:00 2024-04-19T23:17:15.895 INFO:teuthology.orchestra.run.smithi019.stdout:Package librbd1-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:17:16.112 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:17:16.113 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-04-19T23:17:16.113 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:17:16.264 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rados 2024-04-19T23:17:16.676 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:17.848 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:17:18.058 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:17:18.059 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-19T23:17:18.059 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:17:18.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-cephfs 2024-04-19T23:17:18.590 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 24 kB/s | 4.5 kB 00:00 2024-04-19T23:17:18.634 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:19.365 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-19T23:17:20.433 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 4.1 kB 00:00 2024-04-19T23:17:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 106 kB/s | 4.5 kB 00:00 2024-04-19T23:17:21.858 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rados-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:17:22.080 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:17:22.082 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-04-19T23:17:22.082 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:17:22.264 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rgw 2024-04-19T23:17:22.676 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:23.562 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:17:23.774 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:17:23.776 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-19T23:17:23.776 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:17:23.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rbd 2024-04-19T23:17:24.338 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:24.467 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-19T23:17:25.215 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-19T23:17:26.164 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-19T23:17:26.512 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 53 kB/s | 4.5 kB 00:00 2024-04-19T23:17:27.662 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rgw-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:17:27.874 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:17:27.875 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-04-19T23:17:27.875 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:17:28.015 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-cephfs 2024-04-19T23:17:28.429 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:29.265 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:17:29.478 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:17:29.479 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-19T23:17:29.479 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:17:29.626 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-fuse 2024-04-19T23:17:30.039 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:30.208 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-19T23:17:30.953 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-19T23:17:31.833 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 102 kB/s | 4.1 kB 00:00 2024-04-19T23:17:32.135 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 109 kB/s | 4.5 kB 00:00 2024-04-19T23:17:33.441 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-cephfs-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:17:33.651 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:17:33.652 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-04-19T23:17:33.652 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:17:33.798 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rbd 2024-04-19T23:17:34.212 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:35.237 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:17:35.238 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:17:35.238 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-04-19T23:17:35.238 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:17:35.238 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:17:35.238 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 95 k 2024-04-19T23:17:35.238 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:17:35.238 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:17:35.238 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:17:35.238 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-19T23:17:35.238 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:17:35.239 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 95 k 2024-04-19T23:17:35.239 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 280 k 2024-04-19T23:17:35.239 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:17:35.989 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-04-19T23:17:36.109 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-fuse-18.2.2-1237.g646b8c7d.el8.x86_64.rpm 109 kB/s | 95 kB 00:00 2024-04-19T23:17:36.110 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:17:36.110 INFO:teuthology.orchestra.run.smithi031.stdout:Total 109 kB/s | 95 kB 00:00 2024-04-19T23:17:36.111 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:17:36.124 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:17:36.125 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:17:36.194 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:17:36.204 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:17:36.335 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:17:36.487 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:17:36.729 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-19T23:17:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:17:39.213 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 is already installed. 2024-04-19T23:17:39.423 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:17:39.424 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-04-19T23:17:39.424 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:17:39.572 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-fuse 2024-04-19T23:17:39.985 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:41.887 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 kB/s | 4.5 kB 00:00 2024-04-19T23:17:42.646 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-19T23:17:44.303 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:17:44.304 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:17:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:17:44.924 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:17:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:17:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:17:44.924 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:17:45.111 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2024-04-19T23:17:45.351 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:17:45.351 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:17:45.351 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-04-19T23:17:45.351 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:17:45.351 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:17:45.351 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 95 k 2024-04-19T23:17:45.352 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:17:45.352 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:17:45.352 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:17:45.352 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-04-19T23:17:45.352 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:17:45.352 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 95 k 2024-04-19T23:17:45.353 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 280 k 2024-04-19T23:17:45.353 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:17:45.525 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:45.538 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-fuse-18.2.2-1237.g646b8c7d.el8.x86_64.rpm 514 kB/s | 95 kB 00:00 2024-04-19T23:17:45.539 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:17:45.539 INFO:teuthology.orchestra.run.smithi019.stdout:Total 510 kB/s | 95 kB 00:00 2024-04-19T23:17:45.539 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:17:45.552 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:17:45.552 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:17:45.618 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:17:45.628 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:17:45.752 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:17:45.911 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:17:46.221 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:17:46.950 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 22 kB/s | 4.1 kB 00:00 2024-04-19T23:17:47.364 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-19T23:17:50.381 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:17:50.382 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:17:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2024-04-19T23:17:50.382 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:17:50.382 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:17:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 3.5 M 2024-04-19T23:17:50.382 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:17:50.382 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:17:50.383 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:17:50.383 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-19T23:17:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:17:50.383 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.5 M 2024-04-19T23:17:50.383 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 16 M 2024-04-19T23:17:50.383 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:17:51.087 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-mirror-18.2.2-1237.g646b8c7d.el8.x86_64.rpm 4.9 MB/s | 3.5 MB 00:00 2024-04-19T23:17:51.088 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:17:51.088 INFO:teuthology.orchestra.run.smithi031.stdout:Total 4.9 MB/s | 3.5 MB 00:00 2024-04-19T23:17:51.088 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:17:51.104 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:17:51.104 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:17:51.185 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:17:51.195 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:17:51.697 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:17:51.697 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:17:51.699 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:17:51.853 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:17:52.313 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:17:52.334 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:17:52.335 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:17:52.335 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:17:52.335 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:17:52.335 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:17:52.516 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-mirror 2024-04-19T23:17:52.935 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:53.724 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 138 kB/s | 4.5 kB 00:00 2024-04-19T23:17:54.354 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 108 kB/s | 4.1 kB 00:00 2024-04-19T23:17:57.046 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:17:57.047 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:17:57.047 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-04-19T23:17:57.047 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:17:57.047 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:17:57.047 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 3.5 M 2024-04-19T23:17:57.047 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:17:57.047 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:17:57.047 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:17:57.047 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-04-19T23:17:57.048 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:17:57.048 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 3.5 M 2024-04-19T23:17:57.048 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 16 M 2024-04-19T23:17:57.048 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:17:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:17:57.727 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:17:58.345 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:17:58.345 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:17:58.345 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:17:58.345 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:17:58.345 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:17:58.541 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-mirror-18.2.2-1237.g646b8c7d.el8.x86_64.rpm 2.3 MB/s | 3.5 MB 00:01 2024-04-19T23:17:58.542 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:17:58.542 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.3 MB/s | 3.5 MB 00:01 2024-04-19T23:17:58.542 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:17:58.554 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-nbd 2024-04-19T23:17:58.558 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:17:58.558 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:17:58.639 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:17:58.648 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:17:58.969 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2024-04-19T23:17:59.134 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:17:59.310 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:17:59.769 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:17:59.873 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 100 kB/s | 4.1 kB 00:00 2024-04-19T23:18:00.164 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 113 kB/s | 4.5 kB 00:00 2024-04-19T23:18:03.237 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-19T23:18:03.238 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:18:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-04-19T23:18:03.239 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:18:03.239 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-19T23:18:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 181 k 2024-04-19T23:18:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:03.239 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-19T23:18:03.239 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-19T23:18:03.239 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2024-04-19T23:18:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:03.239 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 181 k 2024-04-19T23:18:03.239 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 572 k 2024-04-19T23:18:03.239 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-19T23:18:04.107 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-nbd-18.2.2-1237.g646b8c7d.el8.x86_64.rpm 209 kB/s | 181 kB 00:00 2024-04-19T23:18:04.107 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:18:04.108 INFO:teuthology.orchestra.run.smithi031.stdout:Total 208 kB/s | 181 kB 00:00 2024-04-19T23:18:04.108 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-19T23:18:04.123 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-19T23:18:04.123 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-19T23:18:04.192 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-19T23:18:04.202 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-19T23:18:04.332 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-19T23:18:04.475 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:18:04.782 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:18:05.186 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:18:05.186 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:18:05.836 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:05.836 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:18:05.837 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:18:05.837 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:05.837 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:18:06.050 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-nbd 2024-04-19T23:18:06.477 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-04-19T23:18:07.398 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 85 kB/s | 4.5 kB 00:00 2024-04-19T23:18:08.172 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-19T23:18:10.912 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-04-19T23:18:10.914 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:18:10.914 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-04-19T23:18:10.914 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:18:10.914 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-04-19T23:18:10.915 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd x86_64 2:18.2.2-1237.g646b8c7d.el8 ceph 181 k 2024-04-19T23:18:10.915 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:10.915 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-04-19T23:18:10.915 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-04-19T23:18:10.915 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-04-19T23:18:10.915 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:10.915 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 181 k 2024-04-19T23:18:10.915 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 572 k 2024-04-19T23:18:10.916 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-04-19T23:18:11.295 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-nbd-18.2.2-1237.g646b8c7d.el8.x86_64.rpm 476 kB/s | 181 kB 00:00 2024-04-19T23:18:11.295 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-04-19T23:18:11.296 INFO:teuthology.orchestra.run.smithi019.stdout:Total 474 kB/s | 181 kB 00:00 2024-04-19T23:18:11.296 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-04-19T23:18:11.311 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-04-19T23:18:11.311 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-04-19T23:18:11.380 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-04-19T23:18:11.390 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-04-19T23:18:11.526 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-04-19T23:18:11.699 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:18:12.000 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:18:13.631 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2024-04-19T23:18:14.272 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:14.272 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-19T23:18:14.272 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:18:14.273 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:14.273 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-19T23:18:14.449 DEBUG:teuthology.parallel:result is None 2024-04-19T23:18:18.266 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 1/1 2024-04-19T23:18:18.267 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-04-19T23:18:19.048 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:19.048 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-04-19T23:18:19.049 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-2:18.2.2-1237.g646b8c7d.el8.x86_64 2024-04-19T23:18:19.049 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:19.049 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-04-19T23:18:19.231 DEBUG:teuthology.parallel:result is None 2024-04-19T23:18:19.231 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-19T23:18:19.515 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-19T23:18:19.544 INFO:teuthology.orchestra.run.smithi019.stdout:18.2.2-1237.g646b8c7d.el8 2024-04-19T23:18:19.545 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1237.g646b8c7d.el8 2024-04-19T23:18:19.545 INFO:teuthology.task.install:The correct ceph version 18.2.2-1237.g646b8c7d is installed. 2024-04-19T23:18:19.547 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=646b8c7d293d8ec31a40c69f7b3237b19c1be24d 2024-04-19T23:18:19.692 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-19T23:18:19.721 INFO:teuthology.orchestra.run.smithi031.stdout:18.2.2-1237.g646b8c7d.el8 2024-04-19T23:18:19.721 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1237.g646b8c7d.el8 2024-04-19T23:18:19.721 INFO:teuthology.task.install:The correct ceph version 18.2.2-1237.g646b8c7d is installed. 2024-04-19T23:18:19.723 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-19T23:18:19.723 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:19.723 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T23:18:19.761 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:19.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T23:18:19.800 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-19T23:18:19.800 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:19.800 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T23:18:19.837 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T23:18:19.912 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:19.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T23:18:19.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T23:18:20.021 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-19T23:18:20.021 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:20.021 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T23:18:20.056 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T23:18:20.131 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:20.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T23:18:20.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T23:18:20.239 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-19T23:18:20.240 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:20.240 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T23:18:20.274 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T23:18:20.349 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:20.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T23:18:20.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T23:18:20.458 INFO:teuthology.run_tasks:Running task ceph... 2024-04-19T23:18:20.538 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-19T23:18:20.539 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /var/log/ceph 2024-04-19T23:18:20.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2024-04-19T23:18:20.575 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-19T23:18:20.575 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T23:18:20.616 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T23:18:20.651 INFO:tasks.ceph:Creating extra log directories... 2024-04-19T23:18:20.652 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T23:18:20.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T23:18:20.736 INFO:tasks.ceph:Creating ceph cluster c1... 2024-04-19T23:18:20.736 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-19T23:18:20.736 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001', 'branch': 'reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7664001', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef', 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 3659, 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'rgw', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'targets': {'smithi019.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIcvqhGL0N9TBJPBbHqO/Z9Admuo3j1MrM0n8mryoZRaN5gAbxH+wX247Y6GP/hxWVAgp2QfcTB35iT/FQ1H4a4=', 'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLPGn4JYphtVVEoq1g7lxNACnaZmW1uujquDNRq/8bGYczh4L4ye1SfOdTGMj2UeOhZiDkldB65GKI6v8+/ZkUU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-19_22:40:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-19T23:18:20.737 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-19T23:18:20.767 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-19T23:18:20.788 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T23:18:20.829 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T23:18:20.863 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:20.863 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T23:18:20.924 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T23:18:20.924 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2024-04-19T23:18:20.987 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T23:18:20.987 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T23:18:20.987 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 6h/6d Inode: 178764 Links: 1 2024-04-19T23:18:20.987 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T23:18:20.987 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-04-19T23:18:20.987 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-04-19 23:17:59.511586226 +0000 2024-04-19T23:18:20.988 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-04-19 23:10:52.940536869 +0000 2024-04-19T23:18:20.988 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-04-19 23:10:52.940536869 +0000 2024-04-19T23:18:20.988 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2024-04-19T23:18:20.988 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T23:18:21.061 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-04-19T23:18:21.061 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-04-19T23:18:21.061 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000222595 s, 2.3 MB/s 2024-04-19T23:18:21.062 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T23:18:21.126 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2024-04-19T23:18:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T23:18:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T23:18:21.188 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 6h/6d Inode: 176828 Links: 1 2024-04-19T23:18:21.189 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T23:18:21.189 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-04-19T23:18:21.189 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-04-19 23:17:59.512586216 +0000 2024-04-19T23:18:21.189 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-04-19 23:10:53.316533585 +0000 2024-04-19T23:18:21.189 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-04-19 23:10:53.316533585 +0000 2024-04-19T23:18:21.189 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2024-04-19T23:18:21.189 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T23:18:21.261 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-04-19T23:18:21.261 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-04-19T23:18:21.261 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000219047 s, 2.3 MB/s 2024-04-19T23:18:21.262 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T23:18:21.325 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2024-04-19T23:18:21.387 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T23:18:21.388 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T23:18:21.388 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 6h/6d Inode: 180558 Links: 1 2024-04-19T23:18:21.388 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T23:18:21.388 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-04-19T23:18:21.388 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-04-19 23:17:59.512586216 +0000 2024-04-19T23:18:21.388 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-04-19 23:10:53.690530319 +0000 2024-04-19T23:18:21.388 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-04-19 23:10:53.690530319 +0000 2024-04-19T23:18:21.388 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2024-04-19T23:18:21.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T23:18:21.460 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-04-19T23:18:21.460 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-04-19T23:18:21.460 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000225053 s, 2.3 MB/s 2024-04-19T23:18:21.461 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T23:18:21.526 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2024-04-19T23:18:21.588 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T23:18:21.588 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T23:18:21.589 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 6h/6d Inode: 178895 Links: 1 2024-04-19T23:18:21.589 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T23:18:21.589 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-04-19T23:18:21.589 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-04-19 23:17:59.512586216 +0000 2024-04-19T23:18:21.589 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-04-19 23:10:54.064527053 +0000 2024-04-19T23:18:21.589 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-04-19 23:10:54.064527053 +0000 2024-04-19T23:18:21.589 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2024-04-19T23:18:21.589 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T23:18:21.661 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-04-19T23:18:21.661 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-04-19T23:18:21.661 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000220465 s, 2.3 MB/s 2024-04-19T23:18:21.662 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T23:18:21.726 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T23:18:21.727 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-19T23:18:21.727 INFO:tasks.ceph:Generating config... 2024-04-19T23:18:21.728 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-19T23:18:21.728 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-04-19T23:18:21.728 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-19T23:18:21.728 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-19T23:18:21.728 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-19T23:18:21.728 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-04-19T23:18:21.728 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-04-19T23:18:21.728 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-04-19T23:18:21.729 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-04-19T23:18:21.729 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-04-19T23:18:21.729 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-04-19T23:18:21.729 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-04-19T23:18:21.729 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-19T23:18:21.729 INFO:tasks.ceph:[client] setuser = ceph 2024-04-19T23:18:21.729 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-04-19T23:18:21.729 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-19T23:18:21.729 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-19T23:18:21.730 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-19T23:18:21.730 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-19T23:18:21.730 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-19T23:18:21.730 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-19T23:18:21.730 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-19T23:18:21.730 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-04-19T23:18:21.730 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-04-19T23:18:21.730 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-04-19T23:18:21.730 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-04-19T23:18:21.730 INFO:tasks.ceph:Setting up c1.mon.a... 2024-04-19T23:18:21.730 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-04-19T23:18:21.826 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/c1.keyring 2024-04-19T23:18:21.830 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-04-19T23:18:21.883 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-19T23:18:21.914 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.19')] 2024-04-19T23:18:21.914 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.19'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-04-19T23:18:21.915 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:21.915 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-19T23:18:21.976 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.19 --print /home/ubuntu/cephtest/c1.monmap 2024-04-19T23:18:22.067 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:18:22.067 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:18:22.068 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-04-19T23:18:22.068 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: generated fsid 212ff2d0-184e-4238-bed0-226e23e84773 2024-04-19T23:18:22.069 INFO:teuthology.orchestra.run.smithi019.stdout:setting min_mon_release = pacific 2024-04-19T23:18:22.069 INFO:teuthology.orchestra.run.smithi019.stdout:epoch 0 2024-04-19T23:18:22.069 INFO:teuthology.orchestra.run.smithi019.stdout:fsid 212ff2d0-184e-4238-bed0-226e23e84773 2024-04-19T23:18:22.069 INFO:teuthology.orchestra.run.smithi019.stdout:last_changed 2024-04-19T23:18:22.072112+0000 2024-04-19T23:18:22.069 INFO:teuthology.orchestra.run.smithi019.stdout:created 2024-04-19T23:18:22.072112+0000 2024-04-19T23:18:22.069 INFO:teuthology.orchestra.run.smithi019.stdout:min_mon_release 16 (pacific) 2024-04-19T23:18:22.070 INFO:teuthology.orchestra.run.smithi019.stdout:election_strategy: 1 2024-04-19T23:18:22.070 INFO:teuthology.orchestra.run.smithi019.stdout:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-04-19T23:18:22.070 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-04-19T23:18:22.072 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-19T23:18:22.091 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 212ff2d0-184e-4238-bed0-226e23e84773... 2024-04-19T23:18:22.092 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-19T23:18:22.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: chdir = "" 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: auth supported = cephx 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: filestore xattr use omap = true 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: mon clock drift allowed = 1.000 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: osd crush chooseleaf type = 0 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: auth debug = true 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on old message = true 2024-04-19T23:18:22.191 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on bug = true 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: mon pg warn max object skew = 0 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default size = 2 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow primary affinity = true 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow pg remap = true 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on legacy crush tunables = false 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on crush straw calc version zero = false 2024-04-19T23:18:22.192 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on no sortbitwise = false 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on osd down out interval zero = false 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on too few osds = false 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: mon_allow_pool_size_one = true 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: osd default data pool replay window = 5 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: mon allow pool delete = true 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: mon cluster log file level = debug 2024-04-19T23:18:22.193 INFO:teuthology.orchestra.run.smithi019.stdout: debug asserts on shutdown = true 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: mon health detail to clog = false 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: mon host = 172.21.15.19 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = 212ff2d0-184e-4238-bed0-226e23e84773 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout:[osd] 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: osd journal size = 100 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub load threshold = 5.0 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub max interval = 600 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock profile = high_recovery_ops 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: osd recover clone overlap = true 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: osd recovery max chunk = 1048576 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug shutdown = true 2024-04-19T23:18:22.194 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug op order = true 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify stray on activate = true 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: osd open classes on start = true 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug pg log writeout = true 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: osd deep scrub update digest min age = 30 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: osd map max advance = 10 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: journal zero on create = true 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: filestore ondisk finisher threads = 3 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: filestore apply finisher threads = 3 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: bdev debug aio = true 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug misdirected ops = true 2024-04-19T23:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 20 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: rocksdb delete range threshold = 0 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr] 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 20 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: mgr/telemetry/nag = false 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout:[mon] 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-04-19T23:18:22.196 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 20 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: mon data avail warn = 5 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: mon mgr mkfs grace = 240 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd reporter subtree level = osd 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd prime pg temp = true 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.197 INFO:teuthology.orchestra.run.smithi019.stdout: # 1m isn't quite enough 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout: mon_down_mkfs_grace = 2m 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_filestore_osds = false 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout:[client] 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout: rgw cache enabled = true 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable ops log = true 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable usage log = true 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout: debug rgw = 20 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout: debug rgw sync = 20 2024-04-19T23:18:22.198 INFO:teuthology.orchestra.run.smithi019.stdout: rgw crypt require ssl = False 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: rgw crypt s3 kms backend = testing 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: rgw curl low speed time = 300 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: rgw data log num shards = 4 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: rgw md log max shards = 4 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: rgw sync data inject err probability = 0.1 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: rgw sync log trim interval = 0 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: rgw sync meta inject err probability = 0.1 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: rgw sync obj etag verify = True 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: setgroup = ceph 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: setuser = ceph 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.a] 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: rgw data notify interval msec = 0 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.0] 2024-04-19T23:18:22.199 INFO:teuthology.orchestra.run.smithi019.stdout: osd_max_omap_entries_per_request = 10 2024-04-19T23:18:22.200 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.1] 2024-04-19T23:18:22.200 INFO:teuthology.orchestra.run.smithi019.stdout: osd_max_omap_entries_per_request = 1000 2024-04-19T23:18:22.200 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.2] 2024-04-19T23:18:22.200 INFO:teuthology.orchestra.run.smithi019.stdout: osd_max_omap_entries_per_request = 10000 2024-04-19T23:18:22.200 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-04-19T23:18:22.200 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2024-04-19T23:18:22.200 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T23:18:22.200 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2024-04-19T23:18:22.200 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.200 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2024-04-19T23:18:22.200 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2024-04-19T23:18:22.201 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2024-04-19T23:18:22.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = 172.21.15.19 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 212ff2d0-184e-4238-bed0-226e23e84773 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2024-04-19T23:18:22.203 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2024-04-19T23:18:22.204 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: rocksdb delete range threshold = 0 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2024-04-19T23:18:22.205 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.206 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T23:18:22.207 INFO:teuthology.orchestra.run.smithi031.stdout: debug rgw = 20 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: debug rgw sync = 20 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: rgw crypt require ssl = False 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: rgw crypt s3 kms backend = testing 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: rgw curl low speed time = 300 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: rgw data log num shards = 4 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: rgw md log max shards = 4 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: rgw sync data inject err probability = 0.1 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: rgw sync log trim interval = 0 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: rgw sync meta inject err probability = 0.1 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: rgw sync obj etag verify = True 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: setgroup = ceph 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout: setuser = ceph 2024-04-19T23:18:22.208 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2024-04-19T23:18:22.209 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-04-19T23:18:22.209 INFO:teuthology.orchestra.run.smithi031.stdout: rgw data notify interval msec = 0 2024-04-19T23:18:22.209 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2024-04-19T23:18:22.209 INFO:teuthology.orchestra.run.smithi031.stdout: osd_max_omap_entries_per_request = 10 2024-04-19T23:18:22.209 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2024-04-19T23:18:22.209 INFO:teuthology.orchestra.run.smithi031.stdout: osd_max_omap_entries_per_request = 1000 2024-04-19T23:18:22.209 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2024-04-19T23:18:22.209 INFO:teuthology.orchestra.run.smithi031.stdout: osd_max_omap_entries_per_request = 10000 2024-04-19T23:18:22.210 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-04-19T23:18:22.211 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-04-19T23:18:22.277 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-19T23:18:22.277 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:22.277 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-04-19T23:18:22.297 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:22.297 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-04-19T23:18:22.358 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi019.front.sepia.ceph.com 2024-04-19T23:18:22.358 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:22.358 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/c1.keyring 2024-04-19T23:18:22.358 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-19T23:18:22.445 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:22.446 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-19T23:18:22.508 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2024-04-19T23:18:22.508 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:22.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/c1.keyring 2024-04-19T23:18:22.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-19T23:18:22.556 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:22.556 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-19T23:18:22.618 INFO:tasks.ceph:Setting up mon nodes... 2024-04-19T23:18:22.619 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-19T23:18:22.619 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-04-19T23:18:22.687 INFO:teuthology.orchestra.run.smithi019.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-04-19T23:18:22.690 INFO:tasks.ceph:Setting up mds nodes... 2024-04-19T23:18:22.690 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-19T23:18:22.690 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-04-19T23:18:22.743 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/c1.client.0.keyring 2024-04-19T23:18:22.759 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-04-19T23:18:22.811 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/c1.client.1.keyring 2024-04-19T23:18:22.826 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-19T23:18:22.827 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-19T23:18:22.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-04-19T23:18:22.858 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T23:18:22.858 INFO:tasks.ceph:role: c1.osd.0 2024-04-19T23:18:22.859 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com 2024-04-19T23:18:22.859 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T23:18:22.939 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T23:18:22.939 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T23:18:22.939 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T23:18:22.941 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T23:18:22.941 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T23:18:22.941 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-04-19T23:18:22.941 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T23:18:22.941 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T23:18:22.942 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T23:18:22.942 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T23:18:23.364 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-04-19T23:18:23.366 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-04-19T23:18:23.366 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-04-19T23:18:23.541 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-04-19T23:18:23.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-04-19T23:18:23.656 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T23:18:23.657 INFO:tasks.ceph:role: c1.osd.1 2024-04-19T23:18:23.657 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com 2024-04-19T23:18:23.657 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T23:18:23.740 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T23:18:23.740 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T23:18:23.740 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T23:18:23.740 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T23:18:23.740 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T23:18:23.740 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-04-19T23:18:23.741 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T23:18:23.741 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T23:18:23.741 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T23:18:23.741 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T23:18:24.161 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-04-19T23:18:24.167 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-04-19T23:18:24.167 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-04-19T23:18:24.278 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-04-19T23:18:24.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-04-19T23:18:24.392 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T23:18:24.392 INFO:tasks.ceph:role: c1.osd.2 2024-04-19T23:18:24.392 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com 2024-04-19T23:18:24.392 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T23:18:24.474 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T23:18:24.474 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T23:18:24.474 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T23:18:24.474 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T23:18:24.474 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T23:18:24.475 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-04-19T23:18:24.475 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T23:18:24.475 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T23:18:24.475 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T23:18:24.475 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T23:18:24.926 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-04-19T23:18:24.928 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-04-19T23:18:24.928 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-04-19T23:18:25.088 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-04-19T23:18:25.130 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-19T23:18:25.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:25.232+0000 7fb1bc6fa7c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-04-19T23:18:25.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:25.235+0000 7fb1bc6fa7c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-04-19T23:18:25.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:25.235+0000 7fb1bc6fa7c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-19T23:18:25.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:25.235+0000 7fb1bc6fa7c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-19T23:18:25.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:25.235+0000 7fb1bc6fa7c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-19T23:18:25.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:25.243+0000 7fb1bc6fa7c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-04-19T23:18:27.319 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-04-19T23:18:27.354 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-19T23:18:27.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:27.457+0000 7fcf345767c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-04-19T23:18:27.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:27.457+0000 7fcf345767c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-04-19T23:18:27.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:27.457+0000 7fcf345767c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-19T23:18:27.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:27.457+0000 7fcf345767c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-19T23:18:27.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:27.457+0000 7fcf345767c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-19T23:18:27.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:27.458+0000 7fcf345767c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-04-19T23:18:29.550 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-04-19T23:18:29.585 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-19T23:18:29.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:29.688+0000 7f3e58b377c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-04-19T23:18:29.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:29.688+0000 7f3e58b377c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-04-19T23:18:29.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:29.688+0000 7f3e58b377c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-19T23:18:29.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:29.688+0000 7f3e58b377c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-19T23:18:29.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:29.688+0000 7f3e58b377c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-19T23:18:29.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:18:29.689+0000 7f3e58b377c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-04-19T23:18:31.778 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-04-19T23:18:31.812 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-19T23:18:31.813 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:31.813 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-04-19T23:18:31.888 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:31.888 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-04-19T23:18:31.963 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:31.963 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-04-19T23:18:32.038 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:32.038 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-04-19T23:18:32.113 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:32.113 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-04-19T23:18:32.175 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:32.175 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-04-19T23:18:32.236 INFO:tasks.ceph:Adding keys to all mons... 2024-04-19T23:18:32.236 DEBUG:teuthology.orchestra.run.smithi019:> sudo tee -a /etc/ceph/c1.keyring 2024-04-19T23:18:32.308 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.x] 2024-04-19T23:18:32.309 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQC++yJmK241KRAA7jzfoM0j3ZfXolzdp1r0og== 2024-04-19T23:18:32.309 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.0] 2024-04-19T23:18:32.309 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDB+yJmFp7pDRAAzQ8B1QscUAzz/ZhFJtAYjw== 2024-04-19T23:18:32.309 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.1] 2024-04-19T23:18:32.309 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDD+yJmd4BSGxAAuBisJ2hfq7+RzD5YIaCXBA== 2024-04-19T23:18:32.309 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.2] 2024-04-19T23:18:32.309 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDF+yJm6aoWKRAA0YLQ3q4rDXDFHnWRlTHdEA== 2024-04-19T23:18:32.309 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2024-04-19T23:18:32.309 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQC++yJmw4SJLBAAQIhara9m1RlYxiGcr72lRQ== 2024-04-19T23:18:32.309 INFO:teuthology.orchestra.run.smithi019.stdout:[client.1] 2024-04-19T23:18:32.309 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQC++yJmj8mWMBAALB98Oi9Sdu4bj0B1c1apWw== 2024-04-19T23:18:32.310 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T23:18:32.416 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T23:18:32.477 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T23:18:32.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T23:18:32.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T23:18:32.661 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T23:18:32.724 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-19T23:18:32.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-04-19T23:18:32.755 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-04-19T23:18:32.864 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-04-19T23:18:32.895 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-04-19T23:18:32.956 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-04-19T23:18:32.956 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-04-19T23:18:32.956 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-04-19T23:18:32.998 INFO:tasks.ceph.c1.mon.a:Started 2024-04-19T23:18:32.999 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-04-19T23:18:32.999 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-04-19T23:18:32.999 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-04-19T23:18:33.001 INFO:tasks.ceph.c1.mgr.x:Started 2024-04-19T23:18:33.001 INFO:tasks.ceph:Setting crush tunables to default 2024-04-19T23:18:33.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster c1 osd crush tunables default 2024-04-19T23:18:33.255 INFO:teuthology.orchestra.run.smithi019.stderr:adjusted tunables profile to default 2024-04-19T23:18:33.267 INFO:tasks.ceph:check_enable_crimson: False 2024-04-19T23:18:33.267 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-04-19T23:18:33.268 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:33.268 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-04-19T23:18:33.298 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:33.298 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-04-19T23:18:33.368 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:33.368 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-04-19T23:18:33.439 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster c1 osd new 888199a9-3921-44c7-9b6f-10aeeaee30ec 0 2024-04-19T23:18:33.715 INFO:teuthology.orchestra.run.smithi019.stdout:0 2024-04-19T23:18:33.729 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster c1 osd new ec8f0240-9fc8-4663-9827-b53f589c87c0 1 2024-04-19T23:18:33.774 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:33.778+0000 7f253deee200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-19T23:18:33.883 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:33.887+0000 7f253deee200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-19T23:18:33.975 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:33.978+0000 7f253deee200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-19T23:18:33.979 INFO:teuthology.orchestra.run.smithi019.stdout:1 2024-04-19T23:18:33.993 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster c1 osd new 531feae5-cba7-45bb-abb6-e4f62076bc02 2 2024-04-19T23:18:34.059 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:34.062+0000 7f253deee200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-19T23:18:34.230 INFO:teuthology.orchestra.run.smithi019.stdout:2 2024-04-19T23:18:34.245 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-04-19T23:18:34.245 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-04-19T23:18:34.247 INFO:tasks.ceph.c1.osd.0:Started 2024-04-19T23:18:34.247 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-04-19T23:18:34.247 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-04-19T23:18:34.249 INFO:tasks.ceph.c1.osd.1:Started 2024-04-19T23:18:34.249 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-04-19T23:18:34.249 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-04-19T23:18:34.251 INFO:tasks.ceph.c1.osd.2:Started 2024-04-19T23:18:34.251 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-19T23:18:34.266 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:34.269+0000 7f253deee200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-19T23:18:34.486 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:34.486 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":5,"fsid":"212ff2d0-184e-4238-bed0-226e23e84773","created":"2024-04-19T23:18:33.149175+0000","modified":"2024-04-19T23:18:34.231875+0000","last_up_change":"0.000000","last_in_change":"2024-04-19T23:18:34.231875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"888199a9-3921-44c7-9b6f-10aeeaee30ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ec8f0240-9fc8-4663-9827-b53f589c87c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"531feae5-cba7-45bb-abb6-e4f62076bc02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T23:18:34.494 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:34.498+0000 7f253deee200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-19T23:18:34.500 INFO:tasks.ceph.ceph_manager.c1:[] 2024-04-19T23:18:34.501 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-19T23:18:34.622 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:34.626+0000 7f253deee200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-19T23:18:34.730 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:34.734+0000 7f253deee200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-19T23:18:34.803 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-04-19T23:18:34.816 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:34.820+0000 7f253deee200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-19T23:18:34.868 INFO:tasks.ceph.c1.osd.1.smithi019.stderr:2024-04-19T23:18:34.872+0000 7feecfd4a7c0 -1 Falling back to public interface 2024-04-19T23:18:34.869 INFO:tasks.ceph.c1.osd.2.smithi019.stderr:2024-04-19T23:18:34.873+0000 7f89e94d77c0 -1 Falling back to public interface 2024-04-19T23:18:34.887 INFO:tasks.ceph.c1.osd.0.smithi019.stderr:2024-04-19T23:18:34.891+0000 7fab50efa7c0 -1 Falling back to public interface 2024-04-19T23:18:35.055 INFO:teuthology.misc.health.smithi019.stdout: 2024-04-19T23:18:35.055 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":5,"fsid":"212ff2d0-184e-4238-bed0-226e23e84773","created":"2024-04-19T23:18:33.149175+0000","modified":"2024-04-19T23:18:34.231875+0000","last_up_change":"0.000000","last_in_change":"2024-04-19T23:18:34.231875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"888199a9-3921-44c7-9b6f-10aeeaee30ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ec8f0240-9fc8-4663-9827-b53f589c87c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"531feae5-cba7-45bb-abb6-e4f62076bc02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T23:18:35.073 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-19T23:18:35.086 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:35.089+0000 7f253deee200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-19T23:18:35.168 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:35.171+0000 7f253deee200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-19T23:18:35.599 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:35.603+0000 7f253deee200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-19T23:18:35.969 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:35.973+0000 7f253deee200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-19T23:18:36.151 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:36.154+0000 7f253deee200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-19T23:18:36.241 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:36.245+0000 7f253deee200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-19T23:18:37.037 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:37.041+0000 7f253deee200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-19T23:18:37.133 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:37.137+0000 7f253deee200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-19T23:18:37.284 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:37.287+0000 7f253deee200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-19T23:18:37.571 INFO:tasks.ceph.c1.osd.1.smithi019.stderr:2024-04-19T23:18:37.575+0000 7feecfd4a7c0 -1 osd.1 0 log_to_monitors true 2024-04-19T23:18:37.615 INFO:tasks.ceph.c1.osd.2.smithi019.stderr:2024-04-19T23:18:37.619+0000 7f89e94d77c0 -1 osd.2 0 log_to_monitors true 2024-04-19T23:18:37.650 INFO:tasks.ceph.c1.osd.0.smithi019.stderr:2024-04-19T23:18:37.654+0000 7fab50efa7c0 -1 osd.0 0 log_to_monitors true 2024-04-19T23:18:37.937 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:37.940+0000 7f253deee200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-19T23:18:38.135 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:38.139+0000 7f253deee200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-19T23:18:38.324 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:38.328+0000 7f253deee200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-19T23:18:39.339 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:39.343+0000 7f253deee200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-19T23:18:39.573 INFO:tasks.ceph.c1.osd.1.smithi019.stderr:2024-04-19T23:18:39.576+0000 7feec4102700 -1 osd.1 0 waiting for initial osdmap 2024-04-19T23:18:39.603 INFO:tasks.ceph.c1.osd.1.smithi019.stderr:2024-04-19T23:18:39.607+0000 7feebbcdc700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T23:18:39.640 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:39.644+0000 7f253deee200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-19T23:18:39.725 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:39.729+0000 7f253deee200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-19T23:18:40.104 INFO:tasks.ceph.c1.osd.2.smithi019.stderr:2024-04-19T23:18:40.108+0000 7f89dd88f700 -1 osd.2 0 waiting for initial osdmap 2024-04-19T23:18:40.122 INFO:tasks.ceph.c1.osd.2.smithi019.stderr:2024-04-19T23:18:40.126+0000 7f89d5469700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T23:18:40.202 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:40.206+0000 7f253deee200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-19T23:18:40.333 INFO:tasks.ceph.c1.osd.0.smithi019.stderr:2024-04-19T23:18:40.337+0000 7fab45ab3700 -1 osd.0 0 waiting for initial osdmap 2024-04-19T23:18:40.351 INFO:tasks.ceph.c1.osd.0.smithi019.stderr:2024-04-19T23:18:40.355+0000 7fab3ce8c700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T23:18:41.227 INFO:tasks.ceph.c1.mgr.x.smithi019.stderr:2024-04-19T23:18:41.231+0000 7f24f9168700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-19T23:18:41.377 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-04-19T23:18:41.735 INFO:teuthology.misc.health.smithi019.stdout: 2024-04-19T23:18:41.735 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":9,"fsid":"212ff2d0-184e-4238-bed0-226e23e84773","created":"2024-04-19T23:18:33.149175+0000","modified":"2024-04-19T23:18:41.110838+0000","last_up_change":"2024-04-19T23:18:41.110838+0000","last_in_change":"2024-04-19T23:18:34.231875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"888199a9-3921-44c7-9b6f-10aeeaee30ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":40878},{"type":"v1","addr":"172.21.15.19:6817","nonce":40878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":40878},{"type":"v1","addr":"172.21.15.19:6819","nonce":40878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":40878},{"type":"v1","addr":"172.21.15.19:6823","nonce":40878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":40878},{"type":"v1","addr":"172.21.15.19:6821","nonce":40878}]},"public_addr":"172.21.15.19:6817/40878","cluster_addr":"172.21.15.19:6819/40878","heartbeat_back_addr":"172.21.15.19:6823/40878","heartbeat_front_addr":"172.21.15.19:6821/40878","state":["exists","up"]},{"osd":1,"uuid":"ec8f0240-9fc8-4663-9827-b53f589c87c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":40860},{"type":"v1","addr":"172.21.15.19:6801","nonce":40860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":40860},{"type":"v1","addr":"172.21.15.19:6803","nonce":40860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":40860},{"type":"v1","addr":"172.21.15.19:6807","nonce":40860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":40860},{"type":"v1","addr":"172.21.15.19:6805","nonce":40860}]},"public_addr":"172.21.15.19:6801/40860","cluster_addr":"172.21.15.19:6803/40860","heartbeat_back_addr":"172.21.15.19:6807/40860","heartbeat_front_addr":"172.21.15.19:6805/40860","state":["exists","up"]},{"osd":2,"uuid":"531feae5-cba7-45bb-abb6-e4f62076bc02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":40861},{"type":"v1","addr":"172.21.15.19:6809","nonce":40861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":40861},{"type":"v1","addr":"172.21.15.19:6811","nonce":40861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":40861},{"type":"v1","addr":"172.21.15.19:6815","nonce":40861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":40861},{"type":"v1","addr":"172.21.15.19:6813","nonce":40861}]},"public_addr":"172.21.15.19:6809/40861","cluster_addr":"172.21.15.19:6811/40861","heartbeat_back_addr":"172.21.15.19:6815/40861","heartbeat_front_addr":"172.21.15.19:6813/40861","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:18:38.608085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T23:18:41.750 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-19T23:18:41.750 INFO:tasks.ceph:Creating RBD pool 2024-04-19T23:18:41.750 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster c1 osd pool create rbd 8 2024-04-19T23:18:42.235 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'rbd' created 2024-04-19T23:18:42.248 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-19T23:18:43.243 INFO:teuthology.orchestra.run.smithi019.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-19T23:18:43.257 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-04-19T23:18:43.257 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-04-19T23:18:43.257 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-04-19T23:18:43.258 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-04-19T23:18:43.258 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-19T23:18:43.692 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:43.707 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":40481},{"type":"v1","addr":"172.21.15.19:6825","nonce":40481}]},"active_addr":"172.21.15.19:6825/40481","active_change":"2024-04-19T23:18:40.219287+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":19097579}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":363106343}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":11376156}]}]} 2024-04-19T23:18:43.709 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-04-19T23:18:43.709 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-04-19T23:18:43.709 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-19T23:18:44.076 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:44.076 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":11,"fsid":"212ff2d0-184e-4238-bed0-226e23e84773","created":"2024-04-19T23:18:33.149175+0000","modified":"2024-04-19T23:18:43.233217+0000","last_up_change":"2024-04-19T23:18:41.110838+0000","last_in_change":"2024-04-19T23:18:34.231875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-19T23:18:42.150338+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-19T23:18:42.285463+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"888199a9-3921-44c7-9b6f-10aeeaee30ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":40878},{"type":"v1","addr":"172.21.15.19:6817","nonce":40878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":40878},{"type":"v1","addr":"172.21.15.19:6819","nonce":40878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":40878},{"type":"v1","addr":"172.21.15.19:6823","nonce":40878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":40878},{"type":"v1","addr":"172.21.15.19:6821","nonce":40878}]},"public_addr":"172.21.15.19:6817/40878","cluster_addr":"172.21.15.19:6819/40878","heartbeat_back_addr":"172.21.15.19:6823/40878","heartbeat_front_addr":"172.21.15.19:6821/40878","state":["exists","up"]},{"osd":1,"uuid":"ec8f0240-9fc8-4663-9827-b53f589c87c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":40860},{"type":"v1","addr":"172.21.15.19:6801","nonce":40860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":40860},{"type":"v1","addr":"172.21.15.19:6803","nonce":40860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":40860},{"type":"v1","addr":"172.21.15.19:6807","nonce":40860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":40860},{"type":"v1","addr":"172.21.15.19:6805","nonce":40860}]},"public_addr":"172.21.15.19:6801/40860","cluster_addr":"172.21.15.19:6803/40860","heartbeat_back_addr":"172.21.15.19:6807/40860","heartbeat_front_addr":"172.21.15.19:6805/40860","state":["exists","up"]},{"osd":2,"uuid":"531feae5-cba7-45bb-abb6-e4f62076bc02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":40861},{"type":"v1","addr":"172.21.15.19:6809","nonce":40861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":40861},{"type":"v1","addr":"172.21.15.19:6811","nonce":40861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":40861},{"type":"v1","addr":"172.21.15.19:6815","nonce":40861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":40861},{"type":"v1","addr":"172.21.15.19:6813","nonce":40861}]},"public_addr":"172.21.15.19:6809/40861","cluster_addr":"172.21.15.19:6811/40861","heartbeat_back_addr":"172.21.15.19:6815/40861","heartbeat_front_addr":"172.21.15.19:6813/40861","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:18:38.630993+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:18:38.608085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:18:40.108624+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T23:18:44.091 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-04-19T23:18:44.092 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-19T23:18:44.454 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:44.455 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":12,"fsid":"212ff2d0-184e-4238-bed0-226e23e84773","created":"2024-04-19T23:18:33.149175+0000","modified":"2024-04-19T23:18:44.234126+0000","last_up_change":"2024-04-19T23:18:41.110838+0000","last_in_change":"2024-04-19T23:18:34.231875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-19T23:18:42.150338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-19T23:18:42.285463+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"888199a9-3921-44c7-9b6f-10aeeaee30ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":40878},{"type":"v1","addr":"172.21.15.19:6817","nonce":40878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":40878},{"type":"v1","addr":"172.21.15.19:6819","nonce":40878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":40878},{"type":"v1","addr":"172.21.15.19:6823","nonce":40878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":40878},{"type":"v1","addr":"172.21.15.19:6821","nonce":40878}]},"public_addr":"172.21.15.19:6817/40878","cluster_addr":"172.21.15.19:6819/40878","heartbeat_back_addr":"172.21.15.19:6823/40878","heartbeat_front_addr":"172.21.15.19:6821/40878","state":["exists","up"]},{"osd":1,"uuid":"ec8f0240-9fc8-4663-9827-b53f589c87c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":40860},{"type":"v1","addr":"172.21.15.19:6801","nonce":40860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":40860},{"type":"v1","addr":"172.21.15.19:6803","nonce":40860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":40860},{"type":"v1","addr":"172.21.15.19:6807","nonce":40860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":40860},{"type":"v1","addr":"172.21.15.19:6805","nonce":40860}]},"public_addr":"172.21.15.19:6801/40860","cluster_addr":"172.21.15.19:6803/40860","heartbeat_back_addr":"172.21.15.19:6807/40860","heartbeat_front_addr":"172.21.15.19:6805/40860","state":["exists","up"]},{"osd":2,"uuid":"531feae5-cba7-45bb-abb6-e4f62076bc02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":40861},{"type":"v1","addr":"172.21.15.19:6809","nonce":40861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":40861},{"type":"v1","addr":"172.21.15.19:6811","nonce":40861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":40861},{"type":"v1","addr":"172.21.15.19:6815","nonce":40861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":40861},{"type":"v1","addr":"172.21.15.19:6813","nonce":40861}]},"public_addr":"172.21.15.19:6809/40861","cluster_addr":"172.21.15.19:6811/40861","heartbeat_back_addr":"172.21.15.19:6815/40861","heartbeat_front_addr":"172.21.15.19:6813/40861","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:18:38.630993+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:18:38.608085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:18:40.108624+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T23:18:44.471 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-04-19T23:18:44.472 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-04-19T23:18:44.472 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-04-19T23:18:44.664 INFO:teuthology.orchestra.run.smithi019.stdout:34359738370 2024-04-19T23:18:44.664 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-19T23:18:44.676 INFO:teuthology.orchestra.run.smithi019.stdout:38654705666 2024-04-19T23:18:44.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-19T23:18:44.680 INFO:teuthology.orchestra.run.smithi019.stdout:38654705666 2024-04-19T23:18:44.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-04-19T23:18:45.091 INFO:teuthology.orchestra.run.smithi019.stdout:34359738369 2024-04-19T23:18:45.176 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2024-04-19T23:18:45.246 INFO:teuthology.orchestra.run.smithi019.stdout:38654705665 2024-04-19T23:18:45.246 INFO:teuthology.orchestra.run.smithi019.stdout:38654705665 2024-04-19T23:18:45.269 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705665 for osd.2 2024-04-19T23:18:45.270 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705665 for osd.0 2024-04-19T23:18:46.178 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-19T23:18:46.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-04-19T23:18:46.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-19T23:18:46.573 INFO:teuthology.orchestra.run.smithi019.stdout:34359738370 2024-04-19T23:18:46.590 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.1 2024-04-19T23:18:46.590 DEBUG:teuthology.parallel:result is None 2024-04-19T23:18:46.651 INFO:teuthology.orchestra.run.smithi019.stdout:38654705666 2024-04-19T23:18:46.663 INFO:teuthology.orchestra.run.smithi019.stdout:38654705666 2024-04-19T23:18:46.671 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705666 for osd.0 2024-04-19T23:18:46.671 DEBUG:teuthology.parallel:result is None 2024-04-19T23:18:46.681 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705666 for osd.2 2024-04-19T23:18:46.682 DEBUG:teuthology.parallel:result is None 2024-04-19T23:18:46.682 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-04-19T23:18:46.682 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-19T23:18:47.086 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:47.088 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-04-19T23:18:47.101 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-19T23:18:46.226367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":512,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":17,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81368,"kb_used_data":296,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314491432,"statfs":{"total":322122547200,"available":322039226368,"internally_reserved":0,"allocated":303104,"data_stored":76558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.961911"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.642400+0000","last_change":"2024-04-19T23:18:43.243301+0000","last_active":"2024-04-19T23:18:44.642400+0000","last_peered":"2024-04-19T23:18:44.642400+0000","last_clean":"2024-04-19T23:18:44.642400+0000","last_became_active":"2024-04-19T23:18:43.243031+0000","last_became_peered":"2024-04-19T23:18:43.243031+0000","last_unstale":"2024-04-19T23:18:44.642400+0000","last_undegraded":"2024-04-19T23:18:44.642400+0000","last_fullsized":"2024-04-19T23:18:44.642400+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T23:54:36.382684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.642241+0000","last_change":"2024-04-19T23:18:43.241961+0000","last_active":"2024-04-19T23:18:44.642241+0000","last_peered":"2024-04-19T23:18:44.642241+0000","last_clean":"2024-04-19T23:18:44.642241+0000","last_became_active":"2024-04-19T23:18:43.241699+0000","last_became_peered":"2024-04-19T23:18:43.241699+0000","last_unstale":"2024-04-19T23:18:44.642241+0000","last_undegraded":"2024-04-19T23:18:44.642241+0000","last_fullsized":"2024-04-19T23:18:44.642241+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T08:07:13.823986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.236211+0000","last_change":"2024-04-19T23:18:43.243415+0000","last_active":"2024-04-19T23:18:44.236211+0000","last_peered":"2024-04-19T23:18:44.236211+0000","last_clean":"2024-04-19T23:18:44.236211+0000","last_became_active":"2024-04-19T23:18:43.243091+0000","last_became_peered":"2024-04-19T23:18:43.243091+0000","last_unstale":"2024-04-19T23:18:44.236211+0000","last_undegraded":"2024-04-19T23:18:44.236211+0000","last_fullsized":"2024-04-19T23:18:44.236211+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T09:37:04.771413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.642249+0000","last_change":"2024-04-19T23:18:43.241900+0000","last_active":"2024-04-19T23:18:44.642249+0000","last_peered":"2024-04-19T23:18:44.642249+0000","last_clean":"2024-04-19T23:18:44.642249+0000","last_became_active":"2024-04-19T23:18:43.241595+0000","last_became_peered":"2024-04-19T23:18:43.241595+0000","last_unstale":"2024-04-19T23:18:44.642249+0000","last_undegraded":"2024-04-19T23:18:44.642249+0000","last_fullsized":"2024-04-19T23:18:44.642249+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T10:00:20.536383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:18:43.250007+0000","last_change":"2024-04-19T23:18:43.247130+0000","last_active":"2024-04-19T23:18:43.250007+0000","last_peered":"2024-04-19T23:18:43.250007+0000","last_clean":"2024-04-19T23:18:43.250007+0000","last_became_active":"2024-04-19T23:18:43.246787+0000","last_became_peered":"2024-04-19T23:18:43.246787+0000","last_unstale":"2024-04-19T23:18:43.250007+0000","last_undegraded":"2024-04-19T23:18:43.250007+0000","last_fullsized":"2024-04-19T23:18:43.250007+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T06:50:28.834317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"12'8","reported_seq":21,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.663256+0000","last_change":"2024-04-19T23:18:44.645572+0000","last_active":"2024-04-19T23:18:44.663256+0000","last_peered":"2024-04-19T23:18:44.663256+0000","last_clean":"2024-04-19T23:18:44.663256+0000","last_became_active":"2024-04-19T23:18:44.644651+0000","last_became_peered":"2024-04-19T23:18:44.644651+0000","last_unstale":"2024-04-19T23:18:44.663256+0000","last_undegraded":"2024-04-19T23:18:44.663256+0000","last_fullsized":"2024-04-19T23:18:44.663256+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:43.233217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:43.233217+0000","last_clean_scrub_stamp":"2024-04-19T23:18:43.233217+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T10:00:21.537333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":17,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.642344+0000","last_change":"2024-04-19T23:18:43.246070+0000","last_active":"2024-04-19T23:18:44.642344+0000","last_peered":"2024-04-19T23:18:44.642344+0000","last_clean":"2024-04-19T23:18:44.642344+0000","last_became_active":"2024-04-19T23:18:43.245751+0000","last_became_peered":"2024-04-19T23:18:43.245751+0000","last_unstale":"2024-04-19T23:18:44.642344+0000","last_undegraded":"2024-04-19T23:18:44.642344+0000","last_fullsized":"2024-04-19T23:18:44.642344+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T07:22:38.206512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.642197+0000","last_change":"2024-04-19T23:18:43.245204+0000","last_active":"2024-04-19T23:18:44.642197+0000","last_peered":"2024-04-19T23:18:44.642197+0000","last_clean":"2024-04-19T23:18:44.642197+0000","last_became_active":"2024-04-19T23:18:43.244882+0000","last_became_peered":"2024-04-19T23:18:43.244882+0000","last_unstale":"2024-04-19T23:18:44.642197+0000","last_undegraded":"2024-04-19T23:18:44.642197+0000","last_fullsized":"2024-04-19T23:18:44.642197+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T05:52:11.630190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.236216+0000","last_change":"2024-04-19T23:18:43.243374+0000","last_active":"2024-04-19T23:18:44.236216+0000","last_peered":"2024-04-19T23:18:44.236216+0000","last_clean":"2024-04-19T23:18:44.236216+0000","last_became_active":"2024-04-19T23:18:43.243051+0000","last_became_peered":"2024-04-19T23:18:43.243051+0000","last_unstale":"2024-04-19T23:18:44.236216+0000","last_undegraded":"2024-04-19T23:18:44.236216+0000","last_fullsized":"2024-04-19T23:18:44.236216+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T07:35:24.555283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":512,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":17,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":106496,"data_stored":28090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":24234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":98304,"data_stored":24234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T23:18:47.102 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-19T23:18:47.458 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:47.461 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-04-19T23:18:47.474 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-19T23:18:46.226367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":512,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":17,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81368,"kb_used_data":296,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314491432,"statfs":{"total":322122547200,"available":322039226368,"internally_reserved":0,"allocated":303104,"data_stored":76558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.961911"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.642400+0000","last_change":"2024-04-19T23:18:43.243301+0000","last_active":"2024-04-19T23:18:44.642400+0000","last_peered":"2024-04-19T23:18:44.642400+0000","last_clean":"2024-04-19T23:18:44.642400+0000","last_became_active":"2024-04-19T23:18:43.243031+0000","last_became_peered":"2024-04-19T23:18:43.243031+0000","last_unstale":"2024-04-19T23:18:44.642400+0000","last_undegraded":"2024-04-19T23:18:44.642400+0000","last_fullsized":"2024-04-19T23:18:44.642400+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T23:54:36.382684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.642241+0000","last_change":"2024-04-19T23:18:43.241961+0000","last_active":"2024-04-19T23:18:44.642241+0000","last_peered":"2024-04-19T23:18:44.642241+0000","last_clean":"2024-04-19T23:18:44.642241+0000","last_became_active":"2024-04-19T23:18:43.241699+0000","last_became_peered":"2024-04-19T23:18:43.241699+0000","last_unstale":"2024-04-19T23:18:44.642241+0000","last_undegraded":"2024-04-19T23:18:44.642241+0000","last_fullsized":"2024-04-19T23:18:44.642241+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T08:07:13.823986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.236211+0000","last_change":"2024-04-19T23:18:43.243415+0000","last_active":"2024-04-19T23:18:44.236211+0000","last_peered":"2024-04-19T23:18:44.236211+0000","last_clean":"2024-04-19T23:18:44.236211+0000","last_became_active":"2024-04-19T23:18:43.243091+0000","last_became_peered":"2024-04-19T23:18:43.243091+0000","last_unstale":"2024-04-19T23:18:44.236211+0000","last_undegraded":"2024-04-19T23:18:44.236211+0000","last_fullsized":"2024-04-19T23:18:44.236211+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T09:37:04.771413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.642249+0000","last_change":"2024-04-19T23:18:43.241900+0000","last_active":"2024-04-19T23:18:44.642249+0000","last_peered":"2024-04-19T23:18:44.642249+0000","last_clean":"2024-04-19T23:18:44.642249+0000","last_became_active":"2024-04-19T23:18:43.241595+0000","last_became_peered":"2024-04-19T23:18:43.241595+0000","last_unstale":"2024-04-19T23:18:44.642249+0000","last_undegraded":"2024-04-19T23:18:44.642249+0000","last_fullsized":"2024-04-19T23:18:44.642249+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T10:00:20.536383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:18:43.250007+0000","last_change":"2024-04-19T23:18:43.247130+0000","last_active":"2024-04-19T23:18:43.250007+0000","last_peered":"2024-04-19T23:18:43.250007+0000","last_clean":"2024-04-19T23:18:43.250007+0000","last_became_active":"2024-04-19T23:18:43.246787+0000","last_became_peered":"2024-04-19T23:18:43.246787+0000","last_unstale":"2024-04-19T23:18:43.250007+0000","last_undegraded":"2024-04-19T23:18:43.250007+0000","last_fullsized":"2024-04-19T23:18:43.250007+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T06:50:28.834317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"12'8","reported_seq":21,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.663256+0000","last_change":"2024-04-19T23:18:44.645572+0000","last_active":"2024-04-19T23:18:44.663256+0000","last_peered":"2024-04-19T23:18:44.663256+0000","last_clean":"2024-04-19T23:18:44.663256+0000","last_became_active":"2024-04-19T23:18:44.644651+0000","last_became_peered":"2024-04-19T23:18:44.644651+0000","last_unstale":"2024-04-19T23:18:44.663256+0000","last_undegraded":"2024-04-19T23:18:44.663256+0000","last_fullsized":"2024-04-19T23:18:44.663256+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:43.233217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:43.233217+0000","last_clean_scrub_stamp":"2024-04-19T23:18:43.233217+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T10:00:21.537333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":17,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.642344+0000","last_change":"2024-04-19T23:18:43.246070+0000","last_active":"2024-04-19T23:18:44.642344+0000","last_peered":"2024-04-19T23:18:44.642344+0000","last_clean":"2024-04-19T23:18:44.642344+0000","last_became_active":"2024-04-19T23:18:43.245751+0000","last_became_peered":"2024-04-19T23:18:43.245751+0000","last_unstale":"2024-04-19T23:18:44.642344+0000","last_undegraded":"2024-04-19T23:18:44.642344+0000","last_fullsized":"2024-04-19T23:18:44.642344+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T07:22:38.206512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.642197+0000","last_change":"2024-04-19T23:18:43.245204+0000","last_active":"2024-04-19T23:18:44.642197+0000","last_peered":"2024-04-19T23:18:44.642197+0000","last_clean":"2024-04-19T23:18:44.642197+0000","last_became_active":"2024-04-19T23:18:43.244882+0000","last_became_peered":"2024-04-19T23:18:43.244882+0000","last_unstale":"2024-04-19T23:18:44.642197+0000","last_undegraded":"2024-04-19T23:18:44.642197+0000","last_fullsized":"2024-04-19T23:18:44.642197+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T05:52:11.630190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-19T23:18:44.236216+0000","last_change":"2024-04-19T23:18:43.243374+0000","last_active":"2024-04-19T23:18:44.236216+0000","last_peered":"2024-04-19T23:18:44.236216+0000","last_clean":"2024-04-19T23:18:44.236216+0000","last_became_active":"2024-04-19T23:18:43.243051+0000","last_became_peered":"2024-04-19T23:18:43.243051+0000","last_unstale":"2024-04-19T23:18:44.236216+0000","last_undegraded":"2024-04-19T23:18:44.236216+0000","last_fullsized":"2024-04-19T23:18:44.236216+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:18:42.232267+0000","last_clean_scrub_stamp":"2024-04-19T23:18:42.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T07:35:24.555283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":512,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":17,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":106496,"data_stored":28090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":24234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":98304,"data_stored":24234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T23:18:47.475 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-04-19T23:18:47.475 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-04-19T23:18:47.475 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-04-19T23:18:47.475 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-04-19T23:18:47.893 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:47.894 INFO:teuthology.orchestra.run.smithi019.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-19T23:18:47.909 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-04-19T23:18:47.910 INFO:teuthology.run_tasks:Running task ceph... 2024-04-19T23:18:47.921 INFO:tasks.ceph:Creating ceph cluster c2... 2024-04-19T23:18:47.921 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-19T23:18:47.921 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001', 'branch': 'reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7664001', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef', 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 3659, 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'rgw', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'targets': {'smithi019.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIcvqhGL0N9TBJPBbHqO/Z9Admuo3j1MrM0n8mryoZRaN5gAbxH+wX247Y6GP/hxWVAgp2QfcTB35iT/FQ1H4a4=', 'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLPGn4JYphtVVEoq1g7lxNACnaZmW1uujquDNRq/8bGYczh4L4ye1SfOdTGMj2UeOhZiDkldB65GKI6v8+/ZkUU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '646b8c7d293d8ec31a40c69f7b3237b19c1be24d', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-19_22:40:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-19T23:18:47.921 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-19T23:18:47.924 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-19T23:18:47.947 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T23:18:47.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T23:18:48.021 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:48.021 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T23:18:48.084 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T23:18:48.084 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-04-19T23:18:48.146 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T23:18:48.146 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T23:18:48.146 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 178938 Links: 1 2024-04-19T23:18:48.146 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T23:18:48.146 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-19T23:18:48.146 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-19 23:17:52.052707822 +0000 2024-04-19T23:18:48.146 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-19 23:10:52.955968435 +0000 2024-04-19T23:18:48.146 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-19 23:10:52.955968435 +0000 2024-04-19T23:18:48.147 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2024-04-19T23:18:48.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T23:18:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-19T23:18:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-19T23:18:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000305342 s, 1.7 MB/s 2024-04-19T23:18:48.220 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T23:18:48.283 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-04-19T23:18:48.346 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T23:18:48.346 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T23:18:48.346 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 182593 Links: 1 2024-04-19T23:18:48.346 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T23:18:48.346 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-19T23:18:48.346 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-19 23:17:52.052707822 +0000 2024-04-19T23:18:48.346 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-19 23:10:53.316962771 +0000 2024-04-19T23:18:48.346 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-19 23:10:53.316962771 +0000 2024-04-19T23:18:48.346 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2024-04-19T23:18:48.347 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T23:18:48.418 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-19T23:18:48.419 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-19T23:18:48.419 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000241095 s, 2.1 MB/s 2024-04-19T23:18:48.420 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T23:18:48.484 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-04-19T23:18:48.547 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T23:18:48.548 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T23:18:48.548 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 183352 Links: 1 2024-04-19T23:18:48.548 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T23:18:48.548 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-19T23:18:48.548 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-19 23:17:52.052707822 +0000 2024-04-19T23:18:48.548 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-19 23:10:53.672957184 +0000 2024-04-19T23:18:48.548 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-19 23:10:53.672957184 +0000 2024-04-19T23:18:48.549 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2024-04-19T23:18:48.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T23:18:48.620 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-19T23:18:48.620 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-19T23:18:48.620 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000207869 s, 2.5 MB/s 2024-04-19T23:18:48.622 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T23:18:48.684 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-04-19T23:18:48.747 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T23:18:48.747 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T23:18:48.747 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 181770 Links: 1 2024-04-19T23:18:48.747 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T23:18:48.747 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-19T23:18:48.747 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-19 23:17:52.052707822 +0000 2024-04-19T23:18:48.747 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-19 23:10:54.041951394 +0000 2024-04-19T23:18:48.748 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-19 23:10:54.041951394 +0000 2024-04-19T23:18:48.748 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2024-04-19T23:18:48.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T23:18:48.820 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-19T23:18:48.820 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-19T23:18:48.820 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000262411 s, 2.0 MB/s 2024-04-19T23:18:48.822 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T23:18:48.885 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T23:18:48.885 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-19T23:18:48.885 INFO:tasks.ceph:Generating config... 2024-04-19T23:18:48.886 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-19T23:18:48.886 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-04-19T23:18:48.886 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-19T23:18:48.886 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-19T23:18:48.886 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-19T23:18:48.886 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-04-19T23:18:48.887 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-04-19T23:18:48.887 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-04-19T23:18:48.887 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-04-19T23:18:48.887 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-04-19T23:18:48.887 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-04-19T23:18:48.887 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-04-19T23:18:48.887 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-19T23:18:48.887 INFO:tasks.ceph:[client] setuser = ceph 2024-04-19T23:18:48.887 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-04-19T23:18:48.887 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-19T23:18:48.887 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-19T23:18:48.887 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-19T23:18:48.887 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-19T23:18:48.888 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-19T23:18:48.888 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-19T23:18:48.888 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-19T23:18:48.888 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-04-19T23:18:48.888 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-04-19T23:18:48.888 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-04-19T23:18:48.888 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-04-19T23:18:48.888 INFO:tasks.ceph:Setting up c2.mon.a... 2024-04-19T23:18:48.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-04-19T23:18:48.985 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/c2.keyring 2024-04-19T23:18:48.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-04-19T23:18:49.042 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-19T23:18:49.073 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.31')] 2024-04-19T23:18:49.074 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-04-19T23:18:49.075 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:49.075 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-19T23:18:49.135 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --print /home/ubuntu/cephtest/c2.monmap 2024-04-19T23:18:49.226 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:18:49.227 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:18:49.227 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-04-19T23:18:49.227 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid 681b77c7-0ee4-4b2e-ae4e-eba57b2133ac 2024-04-19T23:18:49.227 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = pacific 2024-04-19T23:18:49.228 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2024-04-19T23:18:49.228 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 681b77c7-0ee4-4b2e-ae4e-eba57b2133ac 2024-04-19T23:18:49.228 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2024-04-19T23:18:49.231513+0000 2024-04-19T23:18:49.228 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-04-19T23:18:49.231513+0000 2024-04-19T23:18:49.228 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 16 (pacific) 2024-04-19T23:18:49.228 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2024-04-19T23:18:49.228 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-19T23:18:49.228 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-04-19T23:18:49.229 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-19T23:18:49.249 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 681b77c7-0ee4-4b2e-ae4e-eba57b2133ac... 2024-04-19T23:18:49.250 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-19T23:18:49.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-19T23:18:49.309 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2024-04-19T23:18:49.309 INFO:teuthology.orchestra.run.smithi019.stdout: chdir = "" 2024-04-19T23:18:49.310 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T23:18:49.310 INFO:teuthology.orchestra.run.smithi019.stdout: auth supported = cephx 2024-04-19T23:18:49.310 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.310 INFO:teuthology.orchestra.run.smithi019.stdout: filestore xattr use omap = true 2024-04-19T23:18:49.310 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.310 INFO:teuthology.orchestra.run.smithi019.stdout: mon clock drift allowed = 1.000 2024-04-19T23:18:49.310 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.310 INFO:teuthology.orchestra.run.smithi019.stdout: osd crush chooseleaf type = 0 2024-04-19T23:18:49.310 INFO:teuthology.orchestra.run.smithi019.stdout: auth debug = true 2024-04-19T23:18:49.310 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.311 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on old message = true 2024-04-19T23:18:49.311 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on bug = true 2024-04-19T23:18:49.311 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.311 INFO:teuthology.orchestra.run.smithi019.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T23:18:49.311 INFO:teuthology.orchestra.run.smithi019.stdout: mon pg warn max object skew = 0 2024-04-19T23:18:49.311 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.311 INFO:teuthology.orchestra.run.smithi019.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T23:18:49.311 INFO:teuthology.orchestra.run.smithi019.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T23:18:49.311 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.311 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default size = 2 2024-04-19T23:18:49.311 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.311 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow primary affinity = true 2024-04-19T23:18:49.312 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow pg remap = true 2024-04-19T23:18:49.312 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on legacy crush tunables = false 2024-04-19T23:18:49.312 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on crush straw calc version zero = false 2024-04-19T23:18:49.312 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on no sortbitwise = false 2024-04-19T23:18:49.312 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on osd down out interval zero = false 2024-04-19T23:18:49.312 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on too few osds = false 2024-04-19T23:18:49.312 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T23:18:49.312 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T23:18:49.312 INFO:teuthology.orchestra.run.smithi019.stdout: mon_allow_pool_size_one = true 2024-04-19T23:18:49.312 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.312 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T23:18:49.312 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: osd default data pool replay window = 5 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: mon allow pool delete = true 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: mon cluster log file level = debug 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: debug asserts on shutdown = true 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: mon health detail to clog = false 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: mon host = 172.21.15.31 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = 681b77c7-0ee4-4b2e-ae4e-eba57b2133ac 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout:[osd] 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: osd journal size = 100 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.313 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub load threshold = 5.0 2024-04-19T23:18:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub max interval = 600 2024-04-19T23:18:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock profile = high_recovery_ops 2024-04-19T23:18:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: osd recover clone overlap = true 2024-04-19T23:18:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: osd recovery max chunk = 1048576 2024-04-19T23:18:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug shutdown = true 2024-04-19T23:18:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug op order = true 2024-04-19T23:18:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify stray on activate = true 2024-04-19T23:18:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: osd open classes on start = true 2024-04-19T23:18:49.314 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug pg log writeout = true 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: osd deep scrub update digest min age = 30 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: osd map max advance = 10 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: journal zero on create = true 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: filestore ondisk finisher threads = 3 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: filestore apply finisher threads = 3 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: bdev debug aio = true 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug misdirected ops = true 2024-04-19T23:18:49.315 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 20 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: rocksdb delete range threshold = 0 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr] 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 20 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: mgr/telemetry/nag = false 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout:[mon] 2024-04-19T23:18:49.316 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 20 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: mon data avail warn = 5 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: mon mgr mkfs grace = 240 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd reporter subtree level = osd 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd prime pg temp = true 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T23:18:49.317 INFO:teuthology.orchestra.run.smithi019.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T23:18:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T23:18:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T23:18:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T23:18:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: # 1m isn't quite enough 2024-04-19T23:18:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: mon_down_mkfs_grace = 2m 2024-04-19T23:18:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_filestore_osds = false 2024-04-19T23:18:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:18:49.318 INFO:teuthology.orchestra.run.smithi019.stdout:[client] 2024-04-19T23:18:49.318 INFO:teuthology.orchestra.run.smithi019.stdout: rgw cache enabled = true 2024-04-19T23:18:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable ops log = true 2024-04-19T23:18:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable usage log = true 2024-04-19T23:18:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T23:18:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T23:18:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: debug rgw = 20 2024-04-19T23:18:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: debug rgw sync = 20 2024-04-19T23:18:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: rgw crypt require ssl = False 2024-04-19T23:18:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: rgw crypt s3 kms backend = testing 2024-04-19T23:18:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-19T23:18:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: rgw curl low speed time = 300 2024-04-19T23:18:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: rgw data log num shards = 4 2024-04-19T23:18:49.319 INFO:teuthology.orchestra.run.smithi019.stdout: rgw md log max shards = 4 2024-04-19T23:18:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: rgw sync data inject err probability = 0.1 2024-04-19T23:18:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: rgw sync log trim interval = 0 2024-04-19T23:18:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: rgw sync meta inject err probability = 0.1 2024-04-19T23:18:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: rgw sync obj etag verify = True 2024-04-19T23:18:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: setgroup = ceph 2024-04-19T23:18:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: setuser = ceph 2024-04-19T23:18:49.320 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.a] 2024-04-19T23:18:49.320 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2024-04-19T23:18:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: rgw data notify interval msec = 0 2024-04-19T23:18:49.320 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.0] 2024-04-19T23:18:49.320 INFO:teuthology.orchestra.run.smithi019.stdout: osd_max_omap_entries_per_request = 10 2024-04-19T23:18:49.320 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.1] 2024-04-19T23:18:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: osd_max_omap_entries_per_request = 1000 2024-04-19T23:18:49.321 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.2] 2024-04-19T23:18:49.321 INFO:teuthology.orchestra.run.smithi019.stdout: osd_max_omap_entries_per_request = 10000 2024-04-19T23:18:49.348 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-04-19T23:18:49.348 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2024-04-19T23:18:49.348 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T23:18:49.348 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2024-04-19T23:18:49.348 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.348 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2024-04-19T23:18:49.348 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.348 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2024-04-19T23:18:49.348 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.349 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2024-04-19T23:18:49.349 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2024-04-19T23:18:49.349 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.349 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2024-04-19T23:18:49.349 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2024-04-19T23:18:49.349 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.349 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T23:18:49.349 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2024-04-19T23:18:49.349 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.349 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T23:18:49.349 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T23:18:49.349 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2024-04-19T23:18:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2024-04-19T23:18:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2024-04-19T23:18:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2024-04-19T23:18:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2024-04-19T23:18:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2024-04-19T23:18:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2024-04-19T23:18:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2024-04-19T23:18:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T23:18:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T23:18:49.350 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2024-04-19T23:18:49.351 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.351 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T23:18:49.351 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.351 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2024-04-19T23:18:49.351 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.351 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2024-04-19T23:18:49.351 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.351 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2024-04-19T23:18:49.351 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2024-04-19T23:18:49.351 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2024-04-19T23:18:49.351 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = 172.21.15.31 2024-04-19T23:18:49.351 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 681b77c7-0ee4-4b2e-ae4e-eba57b2133ac 2024-04-19T23:18:49.352 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.352 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2024-04-19T23:18:49.352 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2024-04-19T23:18:49.352 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.352 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2024-04-19T23:18:49.352 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2024-04-19T23:18:49.352 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2024-04-19T23:18:49.352 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.352 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2024-04-19T23:18:49.353 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2024-04-19T23:18:49.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.353 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2024-04-19T23:18:49.353 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2024-04-19T23:18:49.353 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2024-04-19T23:18:49.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.353 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2024-04-19T23:18:49.353 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2024-04-19T23:18:49.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.353 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2024-04-19T23:18:49.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.353 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout: rocksdb delete range threshold = 0 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.354 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2024-04-19T23:18:49.355 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-04-19T23:18:49.356 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2024-04-19T23:18:49.356 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2024-04-19T23:18:49.356 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-04-19T23:18:49.356 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2024-04-19T23:18:49.356 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2024-04-19T23:18:49.356 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-04-19T23:18:49.356 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.356 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T23:18:49.356 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T23:18:49.356 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T23:18:49.356 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.357 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T23:18:49.357 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T23:18:49.357 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T23:18:49.357 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.357 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2024-04-19T23:18:49.357 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2024-04-19T23:18:49.357 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.357 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2024-04-19T23:18:49.357 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:18:49.357 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2024-04-19T23:18:49.357 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2024-04-19T23:18:49.357 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2024-04-19T23:18:49.358 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2024-04-19T23:18:49.358 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T23:18:49.358 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T23:18:49.358 INFO:teuthology.orchestra.run.smithi031.stdout: debug rgw = 20 2024-04-19T23:18:49.358 INFO:teuthology.orchestra.run.smithi031.stdout: debug rgw sync = 20 2024-04-19T23:18:49.358 INFO:teuthology.orchestra.run.smithi031.stdout: rgw crypt require ssl = False 2024-04-19T23:18:49.358 INFO:teuthology.orchestra.run.smithi031.stdout: rgw crypt s3 kms backend = testing 2024-04-19T23:18:49.358 INFO:teuthology.orchestra.run.smithi031.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-19T23:18:49.358 INFO:teuthology.orchestra.run.smithi031.stdout: rgw curl low speed time = 300 2024-04-19T23:18:49.358 INFO:teuthology.orchestra.run.smithi031.stdout: rgw data log num shards = 4 2024-04-19T23:18:49.358 INFO:teuthology.orchestra.run.smithi031.stdout: rgw md log max shards = 4 2024-04-19T23:18:49.358 INFO:teuthology.orchestra.run.smithi031.stdout: rgw sync data inject err probability = 0.1 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout: rgw sync log trim interval = 0 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout: rgw sync meta inject err probability = 0.1 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout: rgw sync obj etag verify = True 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout: setgroup = ceph 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout: setuser = ceph 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout: rgw data notify interval msec = 0 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout: osd_max_omap_entries_per_request = 10 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout: osd_max_omap_entries_per_request = 1000 2024-04-19T23:18:49.359 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2024-04-19T23:18:49.360 INFO:teuthology.orchestra.run.smithi031.stdout: osd_max_omap_entries_per_request = 10000 2024-04-19T23:18:49.362 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-04-19T23:18:49.362 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-04-19T23:18:49.472 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-19T23:18:49.473 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:49.473 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-04-19T23:18:49.492 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:49.493 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-04-19T23:18:49.554 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi019.front.sepia.ceph.com 2024-04-19T23:18:49.554 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:49.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/c2.keyring 2024-04-19T23:18:49.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-19T23:18:49.601 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-04-19T23:18:49.601 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-19T23:18:49.663 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2024-04-19T23:18:49.664 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:49.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/c2.keyring 2024-04-19T23:18:49.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-19T23:18:49.711 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:49.711 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-19T23:18:49.773 INFO:tasks.ceph:Setting up mon nodes... 2024-04-19T23:18:49.774 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-19T23:18:49.774 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-04-19T23:18:49.883 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-04-19T23:18:49.886 INFO:tasks.ceph:Setting up mds nodes... 2024-04-19T23:18:49.887 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-19T23:18:49.887 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-04-19T23:18:49.938 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/c2.client.0.keyring 2024-04-19T23:18:49.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-04-19T23:18:50.005 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/c2.client.1.keyring 2024-04-19T23:18:50.021 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-19T23:18:50.021 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-19T23:18:50.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-04-19T23:18:50.052 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T23:18:50.052 INFO:tasks.ceph:role: c2.osd.0 2024-04-19T23:18:50.052 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2024-04-19T23:18:50.052 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T23:18:50.134 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T23:18:50.134 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T23:18:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T23:18:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T23:18:50.135 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T23:18:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-04-19T23:18:50.135 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T23:18:50.135 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T23:18:50.135 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T23:18:50.136 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T23:18:50.552 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-04-19T23:18:50.555 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-04-19T23:18:50.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-04-19T23:18:50.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-04-19T23:18:50.791 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-04-19T23:18:50.866 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T23:18:50.866 INFO:tasks.ceph:role: c2.osd.1 2024-04-19T23:18:50.866 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2024-04-19T23:18:50.866 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T23:18:50.948 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T23:18:50.948 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T23:18:50.948 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T23:18:50.948 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T23:18:50.948 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T23:18:50.948 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-04-19T23:18:50.948 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T23:18:50.948 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T23:18:50.948 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T23:18:50.948 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T23:18:51.373 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-04-19T23:18:51.376 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-04-19T23:18:51.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-04-19T23:18:51.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-04-19T23:18:51.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-04-19T23:18:51.610 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T23:18:51.610 INFO:tasks.ceph:role: c2.osd.2 2024-04-19T23:18:51.610 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2024-04-19T23:18:51.610 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T23:18:51.693 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T23:18:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T23:18:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T23:18:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T23:18:51.693 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T23:18:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-04-19T23:18:51.693 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T23:18:51.693 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T23:18:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T23:18:51.693 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T23:18:52.132 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-04-19T23:18:52.134 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-04-19T23:18:52.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-04-19T23:18:52.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-04-19T23:18:52.322 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-19T23:18:52.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:52.425+0000 7fdcd31b97c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-04-19T23:18:52.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:52.427+0000 7fdcd31b97c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-04-19T23:18:52.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:52.427+0000 7fdcd31b97c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-19T23:18:52.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:52.427+0000 7fdcd31b97c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-19T23:18:52.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:52.427+0000 7fdcd31b97c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-19T23:18:52.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:52.432+0000 7fdcd31b97c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-04-19T23:18:54.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-04-19T23:18:54.551 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-19T23:18:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:54.654+0000 7fa17ac647c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-04-19T23:18:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:54.654+0000 7fa17ac647c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-04-19T23:18:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:54.654+0000 7fa17ac647c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-19T23:18:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:54.654+0000 7fa17ac647c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-19T23:18:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:54.654+0000 7fa17ac647c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-19T23:18:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:54.655+0000 7fa17ac647c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-04-19T23:18:56.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-04-19T23:18:56.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-19T23:18:56.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:56.878+0000 7f25361d27c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-04-19T23:18:56.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:56.878+0000 7f25361d27c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-04-19T23:18:56.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:56.878+0000 7f25361d27c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-19T23:18:56.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:56.878+0000 7f25361d27c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-19T23:18:56.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:56.878+0000 7f25361d27c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-19T23:18:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:18:56.879+0000 7f25361d27c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-04-19T23:18:58.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-04-19T23:18:58.997 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-19T23:18:58.997 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:58.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-04-19T23:18:59.072 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:59.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-04-19T23:18:59.146 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:59.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-04-19T23:18:59.221 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:59.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-04-19T23:18:59.296 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:59.296 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-04-19T23:18:59.358 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:18:59.358 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-04-19T23:18:59.419 INFO:tasks.ceph:Adding keys to all mons... 2024-04-19T23:18:59.419 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/c2.keyring 2024-04-19T23:18:59.492 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2024-04-19T23:18:59.493 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDZ+yJmb9DkNBAAitBgKqyov498mV9d7uiTvg== 2024-04-19T23:18:59.493 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2024-04-19T23:18:59.493 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDc+yJmOdhoGRAAJFEqMN5tcR6QgYt4tbTZNg== 2024-04-19T23:18:59.493 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2024-04-19T23:18:59.493 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDe+yJmRNETJxAALUnqCN88FoRpJnUa553F4A== 2024-04-19T23:18:59.493 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2024-04-19T23:18:59.493 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDg+yJmwjdtNBAAkOpfFXo9Nz6eUBT4AM++HQ== 2024-04-19T23:18:59.493 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-04-19T23:18:59.493 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDZ+yJmNk4wOBAAi3K6C6MlDy+MsPU3bvOycw== 2024-04-19T23:18:59.494 INFO:teuthology.orchestra.run.smithi031.stdout:[client.1] 2024-04-19T23:18:59.494 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDa+yJmgPiNABAA2aK6FG3mC+frIBQUPB2R3Q== 2024-04-19T23:18:59.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T23:18:59.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T23:18:59.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T23:18:59.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T23:18:59.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T23:18:59.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T23:18:59.911 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-19T23:18:59.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-04-19T23:18:59.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-04-19T23:19:00.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-04-19T23:19:00.080 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-04-19T23:19:00.141 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-04-19T23:19:00.141 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-04-19T23:19:00.141 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-04-19T23:19:00.184 INFO:tasks.ceph.c2.mon.a:Started 2024-04-19T23:19:00.184 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-04-19T23:19:00.185 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-04-19T23:19:00.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-04-19T23:19:00.187 INFO:tasks.ceph.c2.mgr.x:Started 2024-04-19T23:19:00.187 INFO:tasks.ceph:Setting crush tunables to default 2024-04-19T23:19:00.187 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster c2 osd crush tunables default 2024-04-19T23:19:00.438 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-04-19T23:19:00.450 INFO:tasks.ceph:check_enable_crimson: False 2024-04-19T23:19:00.450 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-04-19T23:19:00.451 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:19:00.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-04-19T23:19:00.480 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:19:00.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-04-19T23:19:00.554 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-19T23:19:00.554 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-04-19T23:19:00.625 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster c2 osd new d1a6bb9b-ed91-414d-83e2-0f216f562f05 0 2024-04-19T23:19:00.900 INFO:teuthology.orchestra.run.smithi031.stdout:0 2024-04-19T23:19:00.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster c2 osd new 2ef96ed5-03a1-4867-ae88-056e71336634 1 2024-04-19T23:19:00.958 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:00.962+0000 7fafa58f6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-19T23:19:01.051 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:01.054+0000 7fafa58f6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-19T23:19:01.137 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:01.140+0000 7fafa58f6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-19T23:19:01.150 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-04-19T23:19:01.165 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster c2 osd new 4fa945b5-bedf-4090-b9de-e80fb8171b2a 2 2024-04-19T23:19:01.220 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:01.223+0000 7fafa58f6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-19T23:19:01.415 INFO:teuthology.orchestra.run.smithi031.stdout:2 2024-04-19T23:19:01.429 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-04-19T23:19:01.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-04-19T23:19:01.432 INFO:tasks.ceph.c2.osd.0:Started 2024-04-19T23:19:01.432 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-04-19T23:19:01.432 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-04-19T23:19:01.433 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:01.436+0000 7fafa58f6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-19T23:19:01.434 INFO:tasks.ceph.c2.osd.1:Started 2024-04-19T23:19:01.434 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-04-19T23:19:01.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-04-19T23:19:01.436 INFO:tasks.ceph.c2.osd.2:Started 2024-04-19T23:19:01.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-19T23:19:01.662 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:01.665+0000 7fafa58f6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-19T23:19:01.669 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:19:01.669 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"fsid":"681b77c7-0ee4-4b2e-ae4e-eba57b2133ac","created":"2024-04-19T23:19:00.336408+0000","modified":"2024-04-19T23:19:01.416501+0000","last_up_change":"0.000000","last_in_change":"2024-04-19T23:19:01.416501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d1a6bb9b-ed91-414d-83e2-0f216f562f05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2ef96ed5-03a1-4867-ae88-056e71336634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4fa945b5-bedf-4090-b9de-e80fb8171b2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T23:19:01.686 INFO:tasks.ceph.ceph_manager.c2:[] 2024-04-19T23:19:01.686 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-19T23:19:01.789 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:01.792+0000 7fafa58f6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-19T23:19:01.896 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:01.899+0000 7fafa58f6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-19T23:19:01.982 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:01.985+0000 7fafa58f6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-19T23:19:01.990 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-04-19T23:19:02.054 INFO:tasks.ceph.c2.osd.0.smithi031.stderr:2024-04-19T23:19:02.057+0000 7ff6dedc37c0 -1 Falling back to public interface 2024-04-19T23:19:02.054 INFO:tasks.ceph.c2.osd.2.smithi031.stderr:2024-04-19T23:19:02.057+0000 7f3ea37bc7c0 -1 Falling back to public interface 2024-04-19T23:19:02.054 INFO:tasks.ceph.c2.osd.0.smithi031.stderr:2024-04-19T23:19:02.057+0000 7ff6d6181700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6803/0: (98) Address already in use 2024-04-19T23:19:02.055 INFO:tasks.ceph.c2.osd.0.smithi031.stderr:2024-04-19T23:19:02.058+0000 7ff6d6181700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6813/0: (98) Address already in use 2024-04-19T23:19:02.071 INFO:tasks.ceph.c2.osd.1.smithi031.stderr:2024-04-19T23:19:02.074+0000 7fb39a71e7c0 -1 Falling back to public interface 2024-04-19T23:19:02.263 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:02.266+0000 7fafa58f6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-19T23:19:02.263 INFO:teuthology.misc.health.smithi031.stdout: 2024-04-19T23:19:02.264 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":5,"fsid":"681b77c7-0ee4-4b2e-ae4e-eba57b2133ac","created":"2024-04-19T23:19:00.336408+0000","modified":"2024-04-19T23:19:01.416501+0000","last_up_change":"0.000000","last_in_change":"2024-04-19T23:19:01.416501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d1a6bb9b-ed91-414d-83e2-0f216f562f05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2ef96ed5-03a1-4867-ae88-056e71336634","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4fa945b5-bedf-4090-b9de-e80fb8171b2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T23:19:02.277 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-19T23:19:02.345 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:02.348+0000 7fafa58f6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-19T23:19:02.779 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:02.782+0000 7fafa58f6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-19T23:19:03.146 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:03.149+0000 7fafa58f6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-19T23:19:03.327 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:03.331+0000 7fafa58f6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-19T23:19:03.418 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:03.421+0000 7fafa58f6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-19T23:19:04.236 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:04.239+0000 7fafa58f6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-19T23:19:04.329 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:04.332+0000 7fafa58f6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-19T23:19:04.479 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:04.482+0000 7fafa58f6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-19T23:19:04.788 INFO:tasks.ceph.c2.osd.2.smithi031.stderr:2024-04-19T23:19:04.791+0000 7f3ea37bc7c0 -1 osd.2 0 log_to_monitors true 2024-04-19T23:19:04.807 INFO:tasks.ceph.c2.osd.0.smithi031.stderr:2024-04-19T23:19:04.810+0000 7ff6dedc37c0 -1 osd.0 0 log_to_monitors true 2024-04-19T23:19:04.824 INFO:tasks.ceph.c2.osd.1.smithi031.stderr:2024-04-19T23:19:04.827+0000 7fb39a71e7c0 -1 osd.1 0 log_to_monitors true 2024-04-19T23:19:05.129 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:05.132+0000 7fafa58f6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-19T23:19:05.320 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:05.323+0000 7fafa58f6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-19T23:19:05.508 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:05.512+0000 7fafa58f6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-19T23:19:06.529 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:06.532+0000 7fafa58f6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-19T23:19:06.819 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:06.822+0000 7fafa58f6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-19T23:19:06.910 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:06.913+0000 7fafa58f6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-19T23:19:07.113 INFO:tasks.ceph.c2.osd.1.smithi031.stderr:2024-04-19T23:19:07.117+0000 7fb38ead6700 -1 osd.1 0 waiting for initial osdmap 2024-04-19T23:19:07.123 INFO:tasks.ceph.c2.osd.1.smithi031.stderr:2024-04-19T23:19:07.126+0000 7fb3866b0700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T23:19:07.141 INFO:tasks.ceph.c2.osd.0.smithi031.stderr:2024-04-19T23:19:07.144+0000 7ff6d317b700 -1 osd.0 0 waiting for initial osdmap 2024-04-19T23:19:07.142 INFO:tasks.ceph.c2.osd.2.smithi031.stderr:2024-04-19T23:19:07.146+0000 7f3e97b74700 -1 osd.2 0 waiting for initial osdmap 2024-04-19T23:19:07.152 INFO:tasks.ceph.c2.osd.0.smithi031.stderr:2024-04-19T23:19:07.155+0000 7ff6cad55700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T23:19:07.159 INFO:tasks.ceph.c2.osd.2.smithi031.stderr:2024-04-19T23:19:07.163+0000 7f3e8f74e700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T23:19:07.397 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:07.400+0000 7fafa58f6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-19T23:19:08.409 INFO:tasks.ceph.c2.mgr.x.smithi031.stderr:2024-04-19T23:19:08.412+0000 7faf60b70700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-19T23:19:08.580 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-04-19T23:19:08.939 INFO:teuthology.misc.health.smithi031.stdout: 2024-04-19T23:19:08.940 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":8,"fsid":"681b77c7-0ee4-4b2e-ae4e-eba57b2133ac","created":"2024-04-19T23:19:00.336408+0000","modified":"2024-04-19T23:19:08.118796+0000","last_up_change":"2024-04-19T23:19:08.118796+0000","last_in_change":"2024-04-19T23:19:01.416501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d1a6bb9b-ed91-414d-83e2-0f216f562f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":40953},{"type":"v1","addr":"172.21.15.31:6801","nonce":40953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":40953},{"type":"v1","addr":"172.21.15.31:6806","nonce":40953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":40953},{"type":"v1","addr":"172.21.15.31:6815","nonce":40953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":40953},{"type":"v1","addr":"172.21.15.31:6811","nonce":40953}]},"public_addr":"172.21.15.31:6801/40953","cluster_addr":"172.21.15.31:6806/40953","heartbeat_back_addr":"172.21.15.31:6815/40953","heartbeat_front_addr":"172.21.15.31:6811/40953","state":["exists","up"]},{"osd":1,"uuid":"2ef96ed5-03a1-4867-ae88-056e71336634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":40964},{"type":"v1","addr":"172.21.15.31:6817","nonce":40964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":40964},{"type":"v1","addr":"172.21.15.31:6819","nonce":40964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":40964},{"type":"v1","addr":"172.21.15.31:6823","nonce":40964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":40964},{"type":"v1","addr":"172.21.15.31:6821","nonce":40964}]},"public_addr":"172.21.15.31:6817/40964","cluster_addr":"172.21.15.31:6819/40964","heartbeat_back_addr":"172.21.15.31:6823/40964","heartbeat_front_addr":"172.21.15.31:6821/40964","state":["exists","up"]},{"osd":2,"uuid":"4fa945b5-bedf-4090-b9de-e80fb8171b2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":40954},{"type":"v1","addr":"172.21.15.31:6803","nonce":40954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":40954},{"type":"v1","addr":"172.21.15.31:6807","nonce":40954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":40954},{"type":"v1","addr":"172.21.15.31:6813","nonce":40954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":40954},{"type":"v1","addr":"172.21.15.31:6810","nonce":40954}]},"public_addr":"172.21.15.31:6803/40954","cluster_addr":"172.21.15.31:6807/40954","heartbeat_back_addr":"172.21.15.31:6813/40954","heartbeat_front_addr":"172.21.15.31:6810/40954","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T23:19:08.954 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-19T23:19:08.954 INFO:tasks.ceph:Creating RBD pool 2024-04-19T23:19:08.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster c2 osd pool create rbd 8 2024-04-19T23:19:09.417 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'rbd' created 2024-04-19T23:19:09.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-19T23:19:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-19T23:19:10.437 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-04-19T23:19:10.437 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-04-19T23:19:10.437 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-04-19T23:19:10.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-04-19T23:19:10.438 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-19T23:19:10.870 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:19:10.884 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":40574},{"type":"v1","addr":"172.21.15.31:6825","nonce":40574}]},"active_addr":"172.21.15.31:6825/40574","active_change":"2024-04-19T23:19:07.403055+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":204739862}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2949539420}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1757323026}]}]} 2024-04-19T23:19:10.888 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-04-19T23:19:10.888 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-04-19T23:19:10.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-19T23:19:11.252 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:19:11.252 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"fsid":"681b77c7-0ee4-4b2e-ae4e-eba57b2133ac","created":"2024-04-19T23:19:00.336408+0000","modified":"2024-04-19T23:19:10.415706+0000","last_up_change":"2024-04-19T23:19:08.118796+0000","last_in_change":"2024-04-19T23:19:01.416501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-19T23:19:09.348108+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-19T23:19:09.465506+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1a6bb9b-ed91-414d-83e2-0f216f562f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":40953},{"type":"v1","addr":"172.21.15.31:6801","nonce":40953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":40953},{"type":"v1","addr":"172.21.15.31:6806","nonce":40953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":40953},{"type":"v1","addr":"172.21.15.31:6815","nonce":40953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":40953},{"type":"v1","addr":"172.21.15.31:6811","nonce":40953}]},"public_addr":"172.21.15.31:6801/40953","cluster_addr":"172.21.15.31:6806/40953","heartbeat_back_addr":"172.21.15.31:6815/40953","heartbeat_front_addr":"172.21.15.31:6811/40953","state":["exists","up"]},{"osd":1,"uuid":"2ef96ed5-03a1-4867-ae88-056e71336634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":40964},{"type":"v1","addr":"172.21.15.31:6817","nonce":40964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":40964},{"type":"v1","addr":"172.21.15.31:6819","nonce":40964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":40964},{"type":"v1","addr":"172.21.15.31:6823","nonce":40964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":40964},{"type":"v1","addr":"172.21.15.31:6821","nonce":40964}]},"public_addr":"172.21.15.31:6817/40964","cluster_addr":"172.21.15.31:6819/40964","heartbeat_back_addr":"172.21.15.31:6823/40964","heartbeat_front_addr":"172.21.15.31:6821/40964","state":["exists","up"]},{"osd":2,"uuid":"4fa945b5-bedf-4090-b9de-e80fb8171b2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":40954},{"type":"v1","addr":"172.21.15.31:6803","nonce":40954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":40954},{"type":"v1","addr":"172.21.15.31:6807","nonce":40954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":40954},{"type":"v1","addr":"172.21.15.31:6813","nonce":40954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":40954},{"type":"v1","addr":"172.21.15.31:6810","nonce":40954}]},"public_addr":"172.21.15.31:6803/40954","cluster_addr":"172.21.15.31:6807/40954","heartbeat_back_addr":"172.21.15.31:6813/40954","heartbeat_front_addr":"172.21.15.31:6810/40954","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:19:05.790089+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:19:05.809829+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:19:05.758191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T23:19:11.267 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-04-19T23:19:11.268 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-19T23:19:11.655 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:19:11.655 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":11,"fsid":"681b77c7-0ee4-4b2e-ae4e-eba57b2133ac","created":"2024-04-19T23:19:00.336408+0000","modified":"2024-04-19T23:19:11.416554+0000","last_up_change":"2024-04-19T23:19:08.118796+0000","last_in_change":"2024-04-19T23:19:01.416501+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-19T23:19:09.348108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-19T23:19:09.465506+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d1a6bb9b-ed91-414d-83e2-0f216f562f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":40953},{"type":"v1","addr":"172.21.15.31:6801","nonce":40953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":40953},{"type":"v1","addr":"172.21.15.31:6806","nonce":40953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":40953},{"type":"v1","addr":"172.21.15.31:6815","nonce":40953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":40953},{"type":"v1","addr":"172.21.15.31:6811","nonce":40953}]},"public_addr":"172.21.15.31:6801/40953","cluster_addr":"172.21.15.31:6806/40953","heartbeat_back_addr":"172.21.15.31:6815/40953","heartbeat_front_addr":"172.21.15.31:6811/40953","state":["exists","up"]},{"osd":1,"uuid":"2ef96ed5-03a1-4867-ae88-056e71336634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":40964},{"type":"v1","addr":"172.21.15.31:6817","nonce":40964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":40964},{"type":"v1","addr":"172.21.15.31:6819","nonce":40964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":40964},{"type":"v1","addr":"172.21.15.31:6823","nonce":40964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":40964},{"type":"v1","addr":"172.21.15.31:6821","nonce":40964}]},"public_addr":"172.21.15.31:6817/40964","cluster_addr":"172.21.15.31:6819/40964","heartbeat_back_addr":"172.21.15.31:6823/40964","heartbeat_front_addr":"172.21.15.31:6821/40964","state":["exists","up"]},{"osd":2,"uuid":"4fa945b5-bedf-4090-b9de-e80fb8171b2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":40954},{"type":"v1","addr":"172.21.15.31:6803","nonce":40954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":40954},{"type":"v1","addr":"172.21.15.31:6807","nonce":40954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":40954},{"type":"v1","addr":"172.21.15.31:6813","nonce":40954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":40954},{"type":"v1","addr":"172.21.15.31:6810","nonce":40954}]},"public_addr":"172.21.15.31:6803/40954","cluster_addr":"172.21.15.31:6807/40954","heartbeat_back_addr":"172.21.15.31:6813/40954","heartbeat_front_addr":"172.21.15.31:6810/40954","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:19:05.790089+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:19:05.809829+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T23:19:05.758191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T23:19:11.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-04-19T23:19:11.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-04-19T23:19:11.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-04-19T23:19:11.890 INFO:teuthology.orchestra.run.smithi031.stdout:34359738370 2024-04-19T23:19:11.890 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-19T23:19:11.893 INFO:teuthology.orchestra.run.smithi031.stdout:34359738370 2024-04-19T23:19:11.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-19T23:19:11.946 INFO:teuthology.orchestra.run.smithi031.stdout:34359738370 2024-04-19T23:19:11.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-19T23:19:12.310 INFO:teuthology.orchestra.run.smithi031.stdout:34359738369 2024-04-19T23:19:12.321 INFO:teuthology.orchestra.run.smithi031.stdout:34359738369 2024-04-19T23:19:12.327 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2024-04-19T23:19:12.337 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.1 2024-04-19T23:19:12.395 INFO:teuthology.orchestra.run.smithi031.stdout:34359738369 2024-04-19T23:19:12.411 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2024-04-19T23:19:13.328 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-19T23:19:13.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-19T23:19:13.412 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-19T23:19:13.734 INFO:teuthology.orchestra.run.smithi031.stdout:34359738369 2024-04-19T23:19:13.750 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2024-04-19T23:19:13.756 INFO:teuthology.orchestra.run.smithi031.stdout:34359738369 2024-04-19T23:19:13.772 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.1 2024-04-19T23:19:13.810 INFO:teuthology.orchestra.run.smithi031.stdout:34359738369 2024-04-19T23:19:13.826 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2024-04-19T23:19:14.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-19T23:19:14.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-19T23:19:14.826 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-19T23:19:15.164 INFO:teuthology.orchestra.run.smithi031.stdout:34359738370 2024-04-19T23:19:15.187 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.0 2024-04-19T23:19:15.187 DEBUG:teuthology.parallel:result is None 2024-04-19T23:19:15.206 INFO:teuthology.orchestra.run.smithi031.stdout:34359738370 2024-04-19T23:19:15.222 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.1 2024-04-19T23:19:15.222 DEBUG:teuthology.parallel:result is None 2024-04-19T23:19:15.223 INFO:teuthology.orchestra.run.smithi031.stdout:34359738370 2024-04-19T23:19:15.239 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.2 2024-04-19T23:19:15.239 DEBUG:teuthology.parallel:result is None 2024-04-19T23:19:15.239 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-04-19T23:19:15.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-19T23:19:15.640 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:19:15.643 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-19T23:19:15.656 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-19T23:19:15.409639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82836,"kb_used_data":1456,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489964,"statfs":{"total":322122547200,"available":322037723136,"internally_reserved":0,"allocated":1490944,"data_stored":1261286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.989184"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.419966+0000","last_change":"2024-04-19T23:19:10.424009+0000","last_active":"2024-04-19T23:19:11.419966+0000","last_peered":"2024-04-19T23:19:11.419966+0000","last_clean":"2024-04-19T23:19:11.419966+0000","last_became_active":"2024-04-19T23:19:10.423720+0000","last_became_peered":"2024-04-19T23:19:10.423720+0000","last_unstale":"2024-04-19T23:19:11.419966+0000","last_undegraded":"2024-04-19T23:19:11.419966+0000","last_fullsized":"2024-04-19T23:19:11.419966+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T01:11:02.087940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.419886+0000","last_change":"2024-04-19T23:19:10.425857+0000","last_active":"2024-04-19T23:19:11.419886+0000","last_peered":"2024-04-19T23:19:11.419886+0000","last_clean":"2024-04-19T23:19:11.419886+0000","last_became_active":"2024-04-19T23:19:10.425604+0000","last_became_peered":"2024-04-19T23:19:10.425604+0000","last_unstale":"2024-04-19T23:19:11.419886+0000","last_undegraded":"2024-04-19T23:19:11.419886+0000","last_fullsized":"2024-04-19T23:19:11.419886+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T10:11:16.502904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.934341+0000","last_change":"2024-04-19T23:19:10.426475+0000","last_active":"2024-04-19T23:19:11.934341+0000","last_peered":"2024-04-19T23:19:11.934341+0000","last_clean":"2024-04-19T23:19:11.934341+0000","last_became_active":"2024-04-19T23:19:10.426197+0000","last_became_peered":"2024-04-19T23:19:10.426197+0000","last_unstale":"2024-04-19T23:19:11.934341+0000","last_undegraded":"2024-04-19T23:19:11.934341+0000","last_fullsized":"2024-04-19T23:19:11.934341+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T07:14:13.534971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.419850+0000","last_change":"2024-04-19T23:19:10.426589+0000","last_active":"2024-04-19T23:19:11.419850+0000","last_peered":"2024-04-19T23:19:11.419850+0000","last_clean":"2024-04-19T23:19:11.419850+0000","last_became_active":"2024-04-19T23:19:10.426283+0000","last_became_peered":"2024-04-19T23:19:10.426283+0000","last_unstale":"2024-04-19T23:19:11.419850+0000","last_undegraded":"2024-04-19T23:19:11.419850+0000","last_fullsized":"2024-04-19T23:19:11.419850+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T04:10:47.936483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.816074+0000","last_change":"2024-04-19T23:19:10.424627+0000","last_active":"2024-04-19T23:19:11.816074+0000","last_peered":"2024-04-19T23:19:11.816074+0000","last_clean":"2024-04-19T23:19:11.816074+0000","last_became_active":"2024-04-19T23:19:10.424202+0000","last_became_peered":"2024-04-19T23:19:10.424202+0000","last_unstale":"2024-04-19T23:19:11.816074+0000","last_undegraded":"2024-04-19T23:19:11.816074+0000","last_fullsized":"2024-04-19T23:19:11.816074+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T09:27:48.417291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:12.045961+0000","last_change":"2024-04-19T23:19:11.422830+0000","last_active":"2024-04-19T23:19:12.045961+0000","last_peered":"2024-04-19T23:19:12.045961+0000","last_clean":"2024-04-19T23:19:12.045961+0000","last_became_active":"2024-04-19T23:19:11.422581+0000","last_became_peered":"2024-04-19T23:19:11.422581+0000","last_unstale":"2024-04-19T23:19:12.045961+0000","last_undegraded":"2024-04-19T23:19:12.045961+0000","last_fullsized":"2024-04-19T23:19:12.045961+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:10.415706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:10.415706+0000","last_clean_scrub_stamp":"2024-04-19T23:19:10.415706+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T04:10:48.937338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.419850+0000","last_change":"2024-04-19T23:19:10.423959+0000","last_active":"2024-04-19T23:19:11.419850+0000","last_peered":"2024-04-19T23:19:11.419850+0000","last_clean":"2024-04-19T23:19:11.419850+0000","last_became_active":"2024-04-19T23:19:10.423688+0000","last_became_peered":"2024-04-19T23:19:10.423688+0000","last_unstale":"2024-04-19T23:19:11.419850+0000","last_undegraded":"2024-04-19T23:19:11.419850+0000","last_fullsized":"2024-04-19T23:19:11.419850+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T01:43:49.264437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.419861+0000","last_change":"2024-04-19T23:19:10.426011+0000","last_active":"2024-04-19T23:19:11.419861+0000","last_peered":"2024-04-19T23:19:11.419861+0000","last_clean":"2024-04-19T23:19:11.419861+0000","last_became_active":"2024-04-19T23:19:10.425718+0000","last_became_peered":"2024-04-19T23:19:10.425718+0000","last_unstale":"2024-04-19T23:19:11.419861+0000","last_undegraded":"2024-04-19T23:19:11.419861+0000","last_fullsized":"2024-04-19T23:19:11.419861+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T08:57:04.433066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.934369+0000","last_change":"2024-04-19T23:19:10.425984+0000","last_active":"2024-04-19T23:19:11.934369+0000","last_peered":"2024-04-19T23:19:11.934369+0000","last_clean":"2024-04-19T23:19:11.934369+0000","last_became_active":"2024-04-19T23:19:10.425663+0000","last_became_peered":"2024-04-19T23:19:10.425663+0000","last_unstale":"2024-04-19T23:19:11.934369+0000","last_undegraded":"2024-04-19T23:19:11.934369+0000","last_fullsized":"2024-04-19T23:19:11.934369+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T23:48:17.173247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27892,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829708,"statfs":{"total":107374182400,"available":107345620992,"internally_reserved":0,"allocated":700416,"data_stored":619558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":25680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27820,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":692224,"data_stored":616048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T23:19:15.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-19T23:19:16.012 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:19:16.014 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-19T23:19:16.027 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-19T23:19:15.409639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82836,"kb_used_data":1456,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489964,"statfs":{"total":322122547200,"available":322037723136,"internally_reserved":0,"allocated":1490944,"data_stored":1261286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.989184"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.419966+0000","last_change":"2024-04-19T23:19:10.424009+0000","last_active":"2024-04-19T23:19:11.419966+0000","last_peered":"2024-04-19T23:19:11.419966+0000","last_clean":"2024-04-19T23:19:11.419966+0000","last_became_active":"2024-04-19T23:19:10.423720+0000","last_became_peered":"2024-04-19T23:19:10.423720+0000","last_unstale":"2024-04-19T23:19:11.419966+0000","last_undegraded":"2024-04-19T23:19:11.419966+0000","last_fullsized":"2024-04-19T23:19:11.419966+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T01:11:02.087940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.419886+0000","last_change":"2024-04-19T23:19:10.425857+0000","last_active":"2024-04-19T23:19:11.419886+0000","last_peered":"2024-04-19T23:19:11.419886+0000","last_clean":"2024-04-19T23:19:11.419886+0000","last_became_active":"2024-04-19T23:19:10.425604+0000","last_became_peered":"2024-04-19T23:19:10.425604+0000","last_unstale":"2024-04-19T23:19:11.419886+0000","last_undegraded":"2024-04-19T23:19:11.419886+0000","last_fullsized":"2024-04-19T23:19:11.419886+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T10:11:16.502904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.934341+0000","last_change":"2024-04-19T23:19:10.426475+0000","last_active":"2024-04-19T23:19:11.934341+0000","last_peered":"2024-04-19T23:19:11.934341+0000","last_clean":"2024-04-19T23:19:11.934341+0000","last_became_active":"2024-04-19T23:19:10.426197+0000","last_became_peered":"2024-04-19T23:19:10.426197+0000","last_unstale":"2024-04-19T23:19:11.934341+0000","last_undegraded":"2024-04-19T23:19:11.934341+0000","last_fullsized":"2024-04-19T23:19:11.934341+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T07:14:13.534971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.419850+0000","last_change":"2024-04-19T23:19:10.426589+0000","last_active":"2024-04-19T23:19:11.419850+0000","last_peered":"2024-04-19T23:19:11.419850+0000","last_clean":"2024-04-19T23:19:11.419850+0000","last_became_active":"2024-04-19T23:19:10.426283+0000","last_became_peered":"2024-04-19T23:19:10.426283+0000","last_unstale":"2024-04-19T23:19:11.419850+0000","last_undegraded":"2024-04-19T23:19:11.419850+0000","last_fullsized":"2024-04-19T23:19:11.419850+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T04:10:47.936483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.816074+0000","last_change":"2024-04-19T23:19:10.424627+0000","last_active":"2024-04-19T23:19:11.816074+0000","last_peered":"2024-04-19T23:19:11.816074+0000","last_clean":"2024-04-19T23:19:11.816074+0000","last_became_active":"2024-04-19T23:19:10.424202+0000","last_became_peered":"2024-04-19T23:19:10.424202+0000","last_unstale":"2024-04-19T23:19:11.816074+0000","last_undegraded":"2024-04-19T23:19:11.816074+0000","last_fullsized":"2024-04-19T23:19:11.816074+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T09:27:48.417291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:12.045961+0000","last_change":"2024-04-19T23:19:11.422830+0000","last_active":"2024-04-19T23:19:12.045961+0000","last_peered":"2024-04-19T23:19:12.045961+0000","last_clean":"2024-04-19T23:19:12.045961+0000","last_became_active":"2024-04-19T23:19:11.422581+0000","last_became_peered":"2024-04-19T23:19:11.422581+0000","last_unstale":"2024-04-19T23:19:12.045961+0000","last_undegraded":"2024-04-19T23:19:12.045961+0000","last_fullsized":"2024-04-19T23:19:12.045961+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:10.415706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:10.415706+0000","last_clean_scrub_stamp":"2024-04-19T23:19:10.415706+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T04:10:48.937338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.419850+0000","last_change":"2024-04-19T23:19:10.423959+0000","last_active":"2024-04-19T23:19:11.419850+0000","last_peered":"2024-04-19T23:19:11.419850+0000","last_clean":"2024-04-19T23:19:11.419850+0000","last_became_active":"2024-04-19T23:19:10.423688+0000","last_became_peered":"2024-04-19T23:19:10.423688+0000","last_unstale":"2024-04-19T23:19:11.419850+0000","last_undegraded":"2024-04-19T23:19:11.419850+0000","last_fullsized":"2024-04-19T23:19:11.419850+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T01:43:49.264437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.419861+0000","last_change":"2024-04-19T23:19:10.426011+0000","last_active":"2024-04-19T23:19:11.419861+0000","last_peered":"2024-04-19T23:19:11.419861+0000","last_clean":"2024-04-19T23:19:11.419861+0000","last_became_active":"2024-04-19T23:19:10.425718+0000","last_became_peered":"2024-04-19T23:19:10.425718+0000","last_unstale":"2024-04-19T23:19:11.419861+0000","last_undegraded":"2024-04-19T23:19:11.419861+0000","last_fullsized":"2024-04-19T23:19:11.419861+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T08:57:04.433066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-19T23:19:11.934369+0000","last_change":"2024-04-19T23:19:10.425984+0000","last_active":"2024-04-19T23:19:11.934369+0000","last_peered":"2024-04-19T23:19:11.934369+0000","last_clean":"2024-04-19T23:19:11.934369+0000","last_became_active":"2024-04-19T23:19:10.425663+0000","last_became_peered":"2024-04-19T23:19:10.425663+0000","last_unstale":"2024-04-19T23:19:11.934369+0000","last_undegraded":"2024-04-19T23:19:11.934369+0000","last_fullsized":"2024-04-19T23:19:11.934369+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T23:19:09.414852+0000","last_clean_scrub_stamp":"2024-04-19T23:19:09.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T23:48:17.173247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27892,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829708,"statfs":{"total":107374182400,"available":107345620992,"internally_reserved":0,"allocated":700416,"data_stored":619558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":25680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27820,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":692224,"data_stored":616048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T23:19:16.028 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-04-19T23:19:16.028 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-04-19T23:19:16.028 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-04-19T23:19:16.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-04-19T23:19:16.436 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-19T23:19:16.436 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-19T23:19:16.452 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-04-19T23:19:16.452 INFO:teuthology.run_tasks:Running task rgw... 2024-04-19T23:19:16.468 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-04-19T23:19:16.468 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-04-19T23:19:16.468 INFO:tasks.rgw:Creating data pools 2024-04-19T23:19:16.469 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-04-19T23:19:16.469 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-04-19T23:19:17.297 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'default.rgw.buckets.data' created 2024-04-19T23:19:17.313 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-04-19T23:19:18.306 INFO:teuthology.orchestra.run.smithi019.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-19T23:19:18.327 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-04-19T23:19:19.303 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'default.rgw.buckets.index' created 2024-04-19T23:19:19.321 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-04-19T23:19:20.329 INFO:teuthology.orchestra.run.smithi019.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-19T23:19:20.341 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-04-19T23:19:20.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-04-19T23:19:20.706 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'default.rgw.buckets.data' already exists 2024-04-19T23:19:20.718 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-04-19T23:19:21.312 INFO:teuthology.orchestra.run.smithi019.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-19T23:19:21.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-04-19T23:19:21.694 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'default.rgw.buckets.index' already exists 2024-04-19T23:19:21.707 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-04-19T23:19:22.318 INFO:teuthology.orchestra.run.smithi019.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-19T23:19:22.333 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-04-19T23:19:22.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-04-19T23:19:23.474 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'default.rgw.buckets.data' created 2024-04-19T23:19:23.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-04-19T23:19:24.492 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-19T23:19:24.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-04-19T23:19:25.491 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'default.rgw.buckets.index' created 2024-04-19T23:19:25.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-04-19T23:19:26.511 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-19T23:19:26.522 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-04-19T23:19:26.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-04-19T23:19:26.888 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'default.rgw.buckets.data' already exists 2024-04-19T23:19:26.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-04-19T23:19:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-19T23:19:27.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-04-19T23:19:27.875 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'default.rgw.buckets.index' already exists 2024-04-19T23:19:27.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-04-19T23:19:28.492 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-19T23:19:28.506 DEBUG:tasks.rgw:Pools created 2024-04-19T23:19:28.507 INFO:tasks.rgw:Configuring compression type = random 2024-04-19T23:19:28.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-04-19T23:19:28.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-04-19T23:19:28.507 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-04-19T23:19:28.586 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:28.587 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:28.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:28.602+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.318+0000 7f36bc992780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:30.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.319+0000 7f36bc992780 20 realm 2024-04-19T23:19:30.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.319+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.319+0000 7f36bc992780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:30.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.319+0000 7f36bc992780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-19T23:19:30.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.319+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.322+0000 7f36bc992780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:30.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.322+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.323+0000 7f36bc992780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:30.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.323+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.323+0000 7f36bc992780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:30.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.323+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.324+0000 7f36bc992780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:30.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.324+0000 7f36bc992780 10 Creating default zonegroup 2024-04-19T23:19:30.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.324+0000 7f36bc992780 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-04-19T23:19:30.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.335+0000 7f36bc992780 10 WARNING: driver->list_zones() returned r=-2 2024-04-19T23:19:30.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.336+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.336+0000 7f36bc992780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:30.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.340+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.340+0000 7f36bc992780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:30.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.340+0000 7f36bc992780 10 could not read realm id: (2) No such file or directory 2024-04-19T23:19:30.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.340+0000 7f36bc992780 10 WARNING: failed to set zone as default, r=-22 2024-04-19T23:19:30.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.340+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.340+0000 7f36bc992780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.343+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.343+0000 7f36bc992780 20 rados_obj.operate() r=0 bl.length=874 2024-04-19T23:19:30.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.343+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.344+0000 7f36bc992780 20 rados_obj.operate() r=0 bl.length=420 2024-04-19T23:19:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.344+0000 7f36bc992780 10 Using default name default 2024-04-19T23:19:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.344+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.344+0000 7f36bc992780 20 rados_obj.operate() r=0 bl.length=46 2024-04-19T23:19:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.344+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.345+0000 7f36bc992780 20 rados_obj.operate() r=0 bl.length=874 2024-04-19T23:19:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.345+0000 7f36bc992780 20 zone default found 2024-04-19T23:19:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.345+0000 7f36bc992780 4 Realm: () 2024-04-19T23:19:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.345+0000 7f36bc992780 4 ZoneGroup: default (3eda5230-aa80-40a7-b066-ac92d2d2f97e) 2024-04-19T23:19:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.345+0000 7f36bc992780 4 Zone: default (346c37b6-11e2-4e32-87e3-fa2af0dd1589) 2024-04-19T23:19:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.345+0000 7f36bc992780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-19T23:19:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.345+0000 7f36bc992780 20 zonegroup default 2024-04-19T23:19:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.345+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:30.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.345+0000 7f36bc992780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:30.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:30.345+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:32.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:32.331+0000 7f36bc992780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:32.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:32.331+0000 7f36bc992780 20 rados->read ofs=0 len=0 2024-04-19T23:19:32.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:32.332+0000 7f36bc992780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:32.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:32.332+0000 7f36bc992780 20 started sync module instance, tier type = 2024-04-19T23:19:32.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:32.332+0000 7f36bc992780 20 started zone id=346c37b6-11e2-4e32-87e3-fa2af0dd1589 (name=default) with tier type = 2024-04-19T23:19:34.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.374+0000 7f36bc992780 20 add_watcher() i=0 2024-04-19T23:19:34.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.374+0000 7f36bc992780 20 add_watcher() i=1 2024-04-19T23:19:34.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.374+0000 7f36bc992780 20 add_watcher() i=2 2024-04-19T23:19:34.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.374+0000 7f36bc992780 20 add_watcher() i=3 2024-04-19T23:19:34.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.374+0000 7f36bc992780 20 add_watcher() i=4 2024-04-19T23:19:34.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.374+0000 7f36bc992780 20 add_watcher() i=5 2024-04-19T23:19:34.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.374+0000 7f36bc992780 20 add_watcher() i=6 2024-04-19T23:19:34.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.377+0000 7f36bc992780 20 add_watcher() i=7 2024-04-19T23:19:34.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.377+0000 7f36bc992780 2 all 8 watchers are set, enabling cache 2024-04-19T23:19:34.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.378+0000 7f36bc992780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-19T23:19:34.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.378+0000 7f36bc992780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-19T23:19:34.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.378+0000 7f36bc992780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-19T23:19:34.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.378+0000 7f36bc992780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-19T23:19:34.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.378+0000 7f36bc992780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-19T23:19:34.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.378+0000 7f36bc992780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-19T23:19:34.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.378+0000 7f36bc992780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-19T23:19:34.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.378+0000 7f36bc992780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-19T23:19:34.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.378+0000 7f36bc992780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-19T23:19:34.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.378+0000 7f36bc992780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-19T23:19:34.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.378+0000 7f36bc992780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-19T23:19:34.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.378+0000 7f36bc992780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-19T23:19:34.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.379+0000 7f36bc992780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-19T23:19:34.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.379+0000 7f36bc992780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-04-19T23:19:34.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.380+0000 7f36bc992780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-19T23:19:34.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.386+0000 7f36a190d700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-19T23:19:34.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.386+0000 7f36bc992780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-19T23:19:34.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:34.386+0000 7f36bc992780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-19T23:19:37.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.328+0000 7f36bc992780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-19T23:19:37.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.328+0000 7f36bc992780 5 note: GC not initialized 2024-04-19T23:19:37.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.329+0000 7f36bc992780 20 init_complete bucket index max shards: 11 2024-04-19T23:19:37.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.329+0000 7f36bc992780 10 rgw notify: Started notification manager with: 1 workers 2024-04-19T23:19:37.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.330+0000 7f369b100700 20 rgw notify: INFO: next queues processing will happen at: Fri Apr 19 23:20:07 2024 2024-04-19T23:19:37.327 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-04-19T23:19:37.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.330+0000 7f369a8de700 20 reqs_thread_entry: start 2024-04-19T23:19:37.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.330+0000 7f369d104700 20 reqs_thread_entry: start 2024-04-19T23:19:37.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.353+0000 7f36a190d700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-19T23:19:37.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.357+0000 7f36bc992780 20 remove_watcher() i=0 2024-04-19T23:19:37.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.357+0000 7f36bc992780 2 removed watcher, disabling cache 2024-04-19T23:19:37.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.360+0000 7f36bc992780 20 remove_watcher() i=1 2024-04-19T23:19:37.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.363+0000 7f36bc992780 20 remove_watcher() i=2 2024-04-19T23:19:37.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.366+0000 7f36bc992780 20 remove_watcher() i=3 2024-04-19T23:19:37.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.370+0000 7f36bc992780 20 remove_watcher() i=4 2024-04-19T23:19:37.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.373+0000 7f36bc992780 20 remove_watcher() i=5 2024-04-19T23:19:37.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.376+0000 7f36bc992780 20 remove_watcher() i=6 2024-04-19T23:19:37.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.377+0000 7f36bc992780 20 remove_watcher() i=7 2024-04-19T23:19:37.377 INFO:teuthology.orchestra.run.smithi019.stdout:[] 2024-04-19T23:19:37.380 DEBUG:tasks.util.rgw: json result: [] 2024-04-19T23:19:37.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-19T23:19:37.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-19T23:19:37.380 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-19T23:19:37.438 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:37.439 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:37.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.452+0000 7fa849980780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-19T23:19:37.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.452+0000 7fa849980780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-19T23:19:37.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.456+0000 7fa82b0f4700 20 reqs_thread_entry: start 2024-04-19T23:19:37.461 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:19:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "346c37b6-11e2-4e32-87e3-fa2af0dd1589", 2024-04-19T23:19:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default", 2024-04-19T23:19:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "domain_root": "default.rgw.meta:root", 2024-04-19T23:19:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "control_pool": "default.rgw.control", 2024-04-19T23:19:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-19T23:19:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-19T23:19:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "log_pool": "default.rgw.log", 2024-04-19T23:19:37.461 INFO:teuthology.orchestra.run.smithi019.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "otp_pool": "default.rgw.otp", 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "system_key": { 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "access_key": "", 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "secret_key": "" 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:37.462 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_pools": [ 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "default-placement", 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": { 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD": { 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "compression_type": "random" 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "index_type": 0, 2024-04-19T23:19:37.463 INFO:teuthology.orchestra.run.smithi019.stdout: "inline_data": true 2024-04-19T23:19:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:19:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "", 2024-04-19T23:19:37.464 INFO:teuthology.orchestra.run.smithi019.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-19T23:19:37.464 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:19:37.465 DEBUG:tasks.util.rgw: json result: {'id': '346c37b6-11e2-4e32-87e3-fa2af0dd1589', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-19T23:19:37.466 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-04-19T23:19:37.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-04-19T23:19:37.466 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-04-19T23:19:37.514 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:37.514 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:37.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.527+0000 7faf9f330780 20 rados->read ofs=0 len=0 2024-04-19T23:19:37.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.528+0000 7faf9f330780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:37.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.528+0000 7faf9f330780 20 realm 2024-04-19T23:19:37.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.528+0000 7faf9f330780 20 rados->read ofs=0 len=0 2024-04-19T23:19:37.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.528+0000 7faf9f330780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:37.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.528+0000 7faf9f330780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-19T23:19:37.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.528+0000 7faf9f330780 20 rados->read ofs=0 len=0 2024-04-19T23:19:37.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.528+0000 7faf9f330780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:37.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.529+0000 7faf9f330780 20 rados->read ofs=0 len=0 2024-04-19T23:19:37.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.529+0000 7faf9f330780 20 rados_obj.operate() r=0 bl.length=46 2024-04-19T23:19:37.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.529+0000 7faf9f330780 20 rados->read ofs=0 len=0 2024-04-19T23:19:37.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.529+0000 7faf9f330780 20 rados_obj.operate() r=0 bl.length=890 2024-04-19T23:19:37.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.530+0000 7faf9f330780 20 searching for the correct realm 2024-04-19T23:19:37.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.538+0000 7faf9f330780 20 RGWRados::pool_iterate: got zonegroup_info.3eda5230-aa80-40a7-b066-ac92d2d2f97e 2024-04-19T23:19:37.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.538+0000 7faf9f330780 20 RGWRados::pool_iterate: got zone_info.346c37b6-11e2-4e32-87e3-fa2af0dd1589 2024-04-19T23:19:37.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.538+0000 7faf9f330780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-19T23:19:37.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.538+0000 7faf9f330780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-19T23:19:37.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.538+0000 7faf9f330780 20 rados->read ofs=0 len=0 2024-04-19T23:19:37.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.538+0000 7faf9f330780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:37.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.538+0000 7faf9f330780 20 rados->read ofs=0 len=0 2024-04-19T23:19:37.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.538+0000 7faf9f330780 20 rados_obj.operate() r=0 bl.length=46 2024-04-19T23:19:37.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.538+0000 7faf9f330780 20 rados->read ofs=0 len=0 2024-04-19T23:19:37.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 20 rados_obj.operate() r=0 bl.length=420 2024-04-19T23:19:37.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 20 zone default found 2024-04-19T23:19:37.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 4 Realm: () 2024-04-19T23:19:37.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 4 ZoneGroup: default (3eda5230-aa80-40a7-b066-ac92d2d2f97e) 2024-04-19T23:19:37.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 4 Zone: default (346c37b6-11e2-4e32-87e3-fa2af0dd1589) 2024-04-19T23:19:37.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-19T23:19:37.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 20 zonegroup default 2024-04-19T23:19:37.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 20 rados->read ofs=0 len=0 2024-04-19T23:19:37.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:37.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 20 rados->read ofs=0 len=0 2024-04-19T23:19:37.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:37.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 20 rados->read ofs=0 len=0 2024-04-19T23:19:37.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:37.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 20 started sync module instance, tier type = 2024-04-19T23:19:37.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.539+0000 7faf9f330780 20 started zone id=346c37b6-11e2-4e32-87e3-fa2af0dd1589 (name=default) with tier type = 2024-04-19T23:19:37.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.564+0000 7faf9f330780 20 add_watcher() i=0 2024-04-19T23:19:37.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.564+0000 7faf9f330780 20 add_watcher() i=1 2024-04-19T23:19:37.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.564+0000 7faf9f330780 20 add_watcher() i=2 2024-04-19T23:19:37.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.564+0000 7faf9f330780 20 add_watcher() i=3 2024-04-19T23:19:37.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.564+0000 7faf9f330780 20 add_watcher() i=4 2024-04-19T23:19:37.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.564+0000 7faf9f330780 20 add_watcher() i=5 2024-04-19T23:19:37.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.564+0000 7faf9f330780 20 add_watcher() i=6 2024-04-19T23:19:37.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.565+0000 7faf9f330780 20 add_watcher() i=7 2024-04-19T23:19:37.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.565+0000 7faf9f330780 2 all 8 watchers are set, enabling cache 2024-04-19T23:19:37.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.567+0000 7faf842ab700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-19T23:19:37.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.567+0000 7faf9f330780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-19T23:19:37.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.567+0000 7faf9f330780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-19T23:19:37.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.571+0000 7faf9f330780 5 note: GC not initialized 2024-04-19T23:19:37.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.572+0000 7faf7faa2700 20 reqs_thread_entry: start 2024-04-19T23:19:37.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.572+0000 7faf9f330780 20 init_complete bucket index max shards: 11 2024-04-19T23:19:37.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.572+0000 7faf9f330780 10 rgw notify: Started notification manager with: 1 workers 2024-04-19T23:19:37.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.572+0000 7faf7da9e700 20 rgw notify: INFO: next queues processing will happen at: Fri Apr 19 23:20:07 2024 2024-04-19T23:19:37.570 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-04-19T23:19:37.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.573+0000 7faf7d27c700 20 reqs_thread_entry: start 2024-04-19T23:19:37.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.581+0000 7faf842ab700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-19T23:19:37.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.583+0000 7faf9f330780 20 remove_watcher() i=0 2024-04-19T23:19:37.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.583+0000 7faf9f330780 2 removed watcher, disabling cache 2024-04-19T23:19:37.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.585+0000 7faf9f330780 20 remove_watcher() i=1 2024-04-19T23:19:37.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.586+0000 7faf9f330780 20 remove_watcher() i=2 2024-04-19T23:19:37.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.588+0000 7faf9f330780 20 remove_watcher() i=3 2024-04-19T23:19:37.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.591+0000 7faf9f330780 20 remove_watcher() i=4 2024-04-19T23:19:37.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.592+0000 7faf9f330780 20 remove_watcher() i=5 2024-04-19T23:19:37.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.593+0000 7faf9f330780 20 remove_watcher() i=6 2024-04-19T23:19:37.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.594+0000 7faf9f330780 20 remove_watcher() i=7 2024-04-19T23:19:37.594 INFO:teuthology.orchestra.run.smithi019.stdout:[] 2024-04-19T23:19:37.596 DEBUG:tasks.util.rgw: json result: [] 2024-04-19T23:19:37.596 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-19T23:19:37.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-19T23:19:37.596 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-19T23:19:37.652 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:37.652 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:37.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.666+0000 7f3c3efbf780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-19T23:19:37.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.666+0000 7f3c3efbf780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-19T23:19:37.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:37.670+0000 7f3c20733700 20 reqs_thread_entry: start 2024-04-19T23:19:37.674 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:19:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "346c37b6-11e2-4e32-87e3-fa2af0dd1589", 2024-04-19T23:19:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default", 2024-04-19T23:19:37.674 INFO:teuthology.orchestra.run.smithi019.stdout: "domain_root": "default.rgw.meta:root", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "control_pool": "default.rgw.control", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "log_pool": "default.rgw.log", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "otp_pool": "default.rgw.otp", 2024-04-19T23:19:37.675 INFO:teuthology.orchestra.run.smithi019.stdout: "system_key": { 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "access_key": "", 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "secret_key": "" 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_pools": [ 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "default-placement", 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": { 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD": { 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: "compression_type": "random" 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-19T23:19:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "index_type": 0, 2024-04-19T23:19:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "inline_data": true 2024-04-19T23:19:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:19:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "", 2024-04-19T23:19:37.677 INFO:teuthology.orchestra.run.smithi019.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-19T23:19:37.677 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:19:37.678 DEBUG:tasks.util.rgw: json result: {'id': '346c37b6-11e2-4e32-87e3-fa2af0dd1589', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-19T23:19:37.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-04-19T23:19:37.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-04-19T23:19:37.678 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-04-19T23:19:37.760 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:37.760 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:37.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:37.775+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.507+0000 7ff6c56df780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:39.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.507+0000 7ff6c56df780 20 realm 2024-04-19T23:19:39.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.507+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.508+0000 7ff6c56df780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:39.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.508+0000 7ff6c56df780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-19T23:19:39.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.508+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.508+0000 7ff6c56df780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:39.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.508+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.508+0000 7ff6c56df780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:39.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.508+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.508+0000 7ff6c56df780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:39.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.508+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.509+0000 7ff6c56df780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:39.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.509+0000 7ff6c56df780 10 Creating default zonegroup 2024-04-19T23:19:39.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.509+0000 7ff6c56df780 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-04-19T23:19:39.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.519+0000 7ff6c56df780 10 WARNING: driver->list_zones() returned r=-2 2024-04-19T23:19:39.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.520+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.520+0000 7ff6c56df780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:39.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.524+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.524+0000 7ff6c56df780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:39.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.524+0000 7ff6c56df780 10 could not read realm id: (2) No such file or directory 2024-04-19T23:19:39.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.524+0000 7ff6c56df780 10 WARNING: failed to set zone as default, r=-22 2024-04-19T23:19:39.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.524+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.524+0000 7ff6c56df780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:39.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.527+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.527+0000 7ff6c56df780 20 rados_obj.operate() r=0 bl.length=874 2024-04-19T23:19:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.527+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.528+0000 7ff6c56df780 20 rados_obj.operate() r=0 bl.length=420 2024-04-19T23:19:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.528+0000 7ff6c56df780 10 Using default name default 2024-04-19T23:19:39.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.528+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.528+0000 7ff6c56df780 20 rados_obj.operate() r=0 bl.length=46 2024-04-19T23:19:39.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.528+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.529+0000 7ff6c56df780 20 rados_obj.operate() r=0 bl.length=874 2024-04-19T23:19:39.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.529+0000 7ff6c56df780 20 zone default found 2024-04-19T23:19:39.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.529+0000 7ff6c56df780 4 Realm: () 2024-04-19T23:19:39.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.529+0000 7ff6c56df780 4 ZoneGroup: default (b5eb58ff-205c-48b4-ab59-5c42171e118b) 2024-04-19T23:19:39.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.529+0000 7ff6c56df780 4 Zone: default (237275ee-c473-4285-96e6-3d73a554c4a2) 2024-04-19T23:19:39.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.529+0000 7ff6c56df780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-19T23:19:39.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.529+0000 7ff6c56df780 20 zonegroup default 2024-04-19T23:19:39.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.529+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:39.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.529+0000 7ff6c56df780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:39.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:39.529+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:41.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:41.514+0000 7ff6c56df780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:41.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:41.514+0000 7ff6c56df780 20 rados->read ofs=0 len=0 2024-04-19T23:19:41.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:41.514+0000 7ff6c56df780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:41.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:41.514+0000 7ff6c56df780 20 started sync module instance, tier type = 2024-04-19T23:19:41.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:41.514+0000 7ff6c56df780 20 started zone id=237275ee-c473-4285-96e6-3d73a554c4a2 (name=default) with tier type = 2024-04-19T23:19:43.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.541+0000 7ff6c56df780 20 add_watcher() i=0 2024-04-19T23:19:43.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.541+0000 7ff6c56df780 20 add_watcher() i=1 2024-04-19T23:19:43.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.541+0000 7ff6c56df780 20 add_watcher() i=2 2024-04-19T23:19:43.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.541+0000 7ff6c56df780 20 add_watcher() i=3 2024-04-19T23:19:43.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.541+0000 7ff6c56df780 20 add_watcher() i=4 2024-04-19T23:19:43.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.541+0000 7ff6c56df780 20 add_watcher() i=5 2024-04-19T23:19:43.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.541+0000 7ff6c56df780 20 add_watcher() i=6 2024-04-19T23:19:43.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.542+0000 7ff6c56df780 20 add_watcher() i=7 2024-04-19T23:19:43.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.542+0000 7ff6c56df780 2 all 8 watchers are set, enabling cache 2024-04-19T23:19:43.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.542+0000 7ff6c56df780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-19T23:19:43.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.542+0000 7ff6c56df780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-19T23:19:43.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.542+0000 7ff6c56df780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-19T23:19:43.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.542+0000 7ff6c56df780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-19T23:19:43.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.542+0000 7ff6c56df780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-19T23:19:43.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.542+0000 7ff6c56df780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-19T23:19:43.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.542+0000 7ff6c56df780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-19T23:19:43.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.543+0000 7ff6c56df780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-19T23:19:43.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.543+0000 7ff6c56df780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-19T23:19:43.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.543+0000 7ff6c56df780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-19T23:19:43.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.543+0000 7ff6c56df780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-19T23:19:43.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.543+0000 7ff6c56df780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-19T23:19:43.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.543+0000 7ff6c56df780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-19T23:19:43.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.543+0000 7ff6c56df780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-04-19T23:19:43.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.544+0000 7ff6c56df780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-19T23:19:43.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.551+0000 7ff6aa65a700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-19T23:19:43.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.551+0000 7ff6c56df780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-19T23:19:43.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:43.551+0000 7ff6c56df780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-19T23:19:46.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.509+0000 7ff6c56df780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-19T23:19:46.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.509+0000 7ff6c56df780 5 note: GC not initialized 2024-04-19T23:19:46.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.511+0000 7ff6a5e51700 20 reqs_thread_entry: start 2024-04-19T23:19:46.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.511+0000 7ff6c56df780 20 init_complete bucket index max shards: 11 2024-04-19T23:19:46.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.511+0000 7ff6c56df780 10 rgw notify: Started notification manager with: 1 workers 2024-04-19T23:19:46.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.511+0000 7ff6a3e4d700 20 rgw notify: INFO: next queues processing will happen at: Fri Apr 19 23:20:16 2024 2024-04-19T23:19:46.509 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-04-19T23:19:46.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.511+0000 7ff6a362b700 20 reqs_thread_entry: start 2024-04-19T23:19:46.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.533+0000 7ff6aa65a700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-19T23:19:46.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.535+0000 7ff6c56df780 20 remove_watcher() i=0 2024-04-19T23:19:46.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.535+0000 7ff6c56df780 2 removed watcher, disabling cache 2024-04-19T23:19:46.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.537+0000 7ff6c56df780 20 remove_watcher() i=1 2024-04-19T23:19:46.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.538+0000 7ff6c56df780 20 remove_watcher() i=2 2024-04-19T23:19:46.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.539+0000 7ff6c56df780 20 remove_watcher() i=3 2024-04-19T23:19:46.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.541+0000 7ff6c56df780 20 remove_watcher() i=4 2024-04-19T23:19:46.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.542+0000 7ff6c56df780 20 remove_watcher() i=5 2024-04-19T23:19:46.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.543+0000 7ff6c56df780 20 remove_watcher() i=6 2024-04-19T23:19:46.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.544+0000 7ff6c56df780 20 remove_watcher() i=7 2024-04-19T23:19:46.545 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-19T23:19:46.547 DEBUG:tasks.util.rgw: json result: [] 2024-04-19T23:19:46.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-19T23:19:46.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-19T23:19:46.548 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-19T23:19:46.605 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:46.606 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:46.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.618+0000 7fc3add86780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-19T23:19:46.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.618+0000 7fc3add86780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-19T23:19:46.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.622+0000 7fc38f4fa700 20 reqs_thread_entry: start 2024-04-19T23:19:46.628 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:19:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "237275ee-c473-4285-96e6-3d73a554c4a2", 2024-04-19T23:19:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default", 2024-04-19T23:19:46.628 INFO:teuthology.orchestra.run.smithi031.stdout: "domain_root": "default.rgw.meta:root", 2024-04-19T23:19:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "control_pool": "default.rgw.control", 2024-04-19T23:19:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-19T23:19:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-19T23:19:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "log_pool": "default.rgw.log", 2024-04-19T23:19:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-19T23:19:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-19T23:19:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-19T23:19:46.629 INFO:teuthology.orchestra.run.smithi031.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "otp_pool": "default.rgw.otp", 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "system_key": { 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "access_key": "", 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "secret_key": "" 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "placement_pools": [ 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "default-placement", 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:19:46.630 INFO:teuthology.orchestra.run.smithi031.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_classes": { 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "STANDARD": { 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "compression_type": "random" 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "index_type": 0, 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "inline_data": true 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_id": "", 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-19T23:19:46.631 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:19:46.632 DEBUG:tasks.util.rgw: json result: {'id': '237275ee-c473-4285-96e6-3d73a554c4a2', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-19T23:19:46.633 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-04-19T23:19:46.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-04-19T23:19:46.633 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-04-19T23:19:46.682 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:46.682 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:46.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.694+0000 7f90e32c3780 20 rados->read ofs=0 len=0 2024-04-19T23:19:46.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.695+0000 7f90e32c3780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:46.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.695+0000 7f90e32c3780 20 realm 2024-04-19T23:19:46.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.695+0000 7f90e32c3780 20 rados->read ofs=0 len=0 2024-04-19T23:19:46.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.696+0000 7f90e32c3780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:46.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.696+0000 7f90e32c3780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-19T23:19:46.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.696+0000 7f90e32c3780 20 rados->read ofs=0 len=0 2024-04-19T23:19:46.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.696+0000 7f90e32c3780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:46.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.696+0000 7f90e32c3780 20 rados->read ofs=0 len=0 2024-04-19T23:19:46.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.696+0000 7f90e32c3780 20 rados_obj.operate() r=0 bl.length=46 2024-04-19T23:19:46.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.696+0000 7f90e32c3780 20 rados->read ofs=0 len=0 2024-04-19T23:19:46.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.697+0000 7f90e32c3780 20 rados_obj.operate() r=0 bl.length=890 2024-04-19T23:19:46.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.697+0000 7f90e32c3780 20 searching for the correct realm 2024-04-19T23:19:46.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.705+0000 7f90e32c3780 20 RGWRados::pool_iterate: got zone_info.237275ee-c473-4285-96e6-3d73a554c4a2 2024-04-19T23:19:46.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.705+0000 7f90e32c3780 20 RGWRados::pool_iterate: got zonegroup_info.b5eb58ff-205c-48b4-ab59-5c42171e118b 2024-04-19T23:19:46.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.705+0000 7f90e32c3780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-19T23:19:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.705+0000 7f90e32c3780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-19T23:19:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.705+0000 7f90e32c3780 20 rados->read ofs=0 len=0 2024-04-19T23:19:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 20 rados->read ofs=0 len=0 2024-04-19T23:19:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 20 rados_obj.operate() r=0 bl.length=46 2024-04-19T23:19:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 20 rados->read ofs=0 len=0 2024-04-19T23:19:46.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 20 rados_obj.operate() r=0 bl.length=420 2024-04-19T23:19:46.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 20 zone default found 2024-04-19T23:19:46.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 4 Realm: () 2024-04-19T23:19:46.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 4 ZoneGroup: default (b5eb58ff-205c-48b4-ab59-5c42171e118b) 2024-04-19T23:19:46.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 4 Zone: default (237275ee-c473-4285-96e6-3d73a554c4a2) 2024-04-19T23:19:46.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-19T23:19:46.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 20 zonegroup default 2024-04-19T23:19:46.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 20 rados->read ofs=0 len=0 2024-04-19T23:19:46.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:46.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.706+0000 7f90e32c3780 20 rados->read ofs=0 len=0 2024-04-19T23:19:46.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.707+0000 7f90e32c3780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:46.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.707+0000 7f90e32c3780 20 rados->read ofs=0 len=0 2024-04-19T23:19:46.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.707+0000 7f90e32c3780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-19T23:19:46.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.707+0000 7f90e32c3780 20 started sync module instance, tier type = 2024-04-19T23:19:46.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.707+0000 7f90e32c3780 20 started zone id=237275ee-c473-4285-96e6-3d73a554c4a2 (name=default) with tier type = 2024-04-19T23:19:46.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.717+0000 7f90e32c3780 20 add_watcher() i=0 2024-04-19T23:19:46.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.717+0000 7f90e32c3780 20 add_watcher() i=1 2024-04-19T23:19:46.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.717+0000 7f90e32c3780 20 add_watcher() i=2 2024-04-19T23:19:46.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.717+0000 7f90e32c3780 20 add_watcher() i=3 2024-04-19T23:19:46.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.717+0000 7f90e32c3780 20 add_watcher() i=4 2024-04-19T23:19:46.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.717+0000 7f90e32c3780 20 add_watcher() i=5 2024-04-19T23:19:46.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.717+0000 7f90e32c3780 20 add_watcher() i=6 2024-04-19T23:19:46.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.718+0000 7f90e32c3780 20 add_watcher() i=7 2024-04-19T23:19:46.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.718+0000 7f90e32c3780 2 all 8 watchers are set, enabling cache 2024-04-19T23:19:46.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.720+0000 7f90c823e700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-19T23:19:46.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.720+0000 7f90e32c3780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-19T23:19:46.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.720+0000 7f90e32c3780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-19T23:19:46.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.724+0000 7f90e32c3780 5 note: GC not initialized 2024-04-19T23:19:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.725+0000 7f90c3a35700 20 reqs_thread_entry: start 2024-04-19T23:19:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.726+0000 7f90e32c3780 20 init_complete bucket index max shards: 11 2024-04-19T23:19:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.726+0000 7f90e32c3780 10 rgw notify: Started notification manager with: 1 workers 2024-04-19T23:19:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.726+0000 7f90c1a31700 20 rgw notify: INFO: next queues processing will happen at: Fri Apr 19 23:20:16 2024 2024-04-19T23:19:46.728 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-04-19T23:19:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.726+0000 7f90c120f700 20 reqs_thread_entry: start 2024-04-19T23:19:46.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.735+0000 7f90c823e700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-19T23:19:46.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.737+0000 7f90e32c3780 20 remove_watcher() i=0 2024-04-19T23:19:46.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.737+0000 7f90e32c3780 2 removed watcher, disabling cache 2024-04-19T23:19:46.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.738+0000 7f90e32c3780 20 remove_watcher() i=1 2024-04-19T23:19:46.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.739+0000 7f90e32c3780 20 remove_watcher() i=2 2024-04-19T23:19:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.742+0000 7f90e32c3780 20 remove_watcher() i=3 2024-04-19T23:19:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.744+0000 7f90e32c3780 20 remove_watcher() i=4 2024-04-19T23:19:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.746+0000 7f90e32c3780 20 remove_watcher() i=5 2024-04-19T23:19:46.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.747+0000 7f90e32c3780 20 remove_watcher() i=6 2024-04-19T23:19:46.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.750+0000 7f90e32c3780 20 remove_watcher() i=7 2024-04-19T23:19:46.750 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-19T23:19:46.752 DEBUG:tasks.util.rgw: json result: [] 2024-04-19T23:19:46.753 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-19T23:19:46.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-19T23:19:46.753 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-19T23:19:46.810 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:46.810 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:46.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.822+0000 7fac3b941780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-19T23:19:46.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.822+0000 7fac3b941780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-19T23:19:46.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:19:46.827+0000 7fac1d0b5700 20 reqs_thread_entry: start 2024-04-19T23:19:46.832 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:19:46.832 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "237275ee-c473-4285-96e6-3d73a554c4a2", 2024-04-19T23:19:46.832 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default", 2024-04-19T23:19:46.832 INFO:teuthology.orchestra.run.smithi031.stdout: "domain_root": "default.rgw.meta:root", 2024-04-19T23:19:46.832 INFO:teuthology.orchestra.run.smithi031.stdout: "control_pool": "default.rgw.control", 2024-04-19T23:19:46.832 INFO:teuthology.orchestra.run.smithi031.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-19T23:19:46.832 INFO:teuthology.orchestra.run.smithi031.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-19T23:19:46.833 INFO:teuthology.orchestra.run.smithi031.stdout: "log_pool": "default.rgw.log", 2024-04-19T23:19:46.833 INFO:teuthology.orchestra.run.smithi031.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-19T23:19:46.833 INFO:teuthology.orchestra.run.smithi031.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-19T23:19:46.833 INFO:teuthology.orchestra.run.smithi031.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-19T23:19:46.833 INFO:teuthology.orchestra.run.smithi031.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-19T23:19:46.833 INFO:teuthology.orchestra.run.smithi031.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-19T23:19:46.833 INFO:teuthology.orchestra.run.smithi031.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-19T23:19:46.834 INFO:teuthology.orchestra.run.smithi031.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-19T23:19:46.834 INFO:teuthology.orchestra.run.smithi031.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-19T23:19:46.834 INFO:teuthology.orchestra.run.smithi031.stdout: "otp_pool": "default.rgw.otp", 2024-04-19T23:19:46.834 INFO:teuthology.orchestra.run.smithi031.stdout: "system_key": { 2024-04-19T23:19:46.834 INFO:teuthology.orchestra.run.smithi031.stdout: "access_key": "", 2024-04-19T23:19:46.835 INFO:teuthology.orchestra.run.smithi031.stdout: "secret_key": "" 2024-04-19T23:19:46.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:19:46.835 INFO:teuthology.orchestra.run.smithi031.stdout: "placement_pools": [ 2024-04-19T23:19:46.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:19:46.835 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "default-placement", 2024-04-19T23:19:46.835 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:19:46.836 INFO:teuthology.orchestra.run.smithi031.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-19T23:19:46.836 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_classes": { 2024-04-19T23:19:46.836 INFO:teuthology.orchestra.run.smithi031.stdout: "STANDARD": { 2024-04-19T23:19:46.836 INFO:teuthology.orchestra.run.smithi031.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-19T23:19:46.836 INFO:teuthology.orchestra.run.smithi031.stdout: "compression_type": "random" 2024-04-19T23:19:46.836 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:19:46.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:19:46.837 INFO:teuthology.orchestra.run.smithi031.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-19T23:19:46.837 INFO:teuthology.orchestra.run.smithi031.stdout: "index_type": 0, 2024-04-19T23:19:46.837 INFO:teuthology.orchestra.run.smithi031.stdout: "inline_data": true 2024-04-19T23:19:46.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:19:46.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:19:46.837 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:19:46.837 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_id": "", 2024-04-19T23:19:46.837 INFO:teuthology.orchestra.run.smithi031.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-19T23:19:46.837 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:19:46.839 DEBUG:tasks.util.rgw: json result: {'id': '237275ee-c473-4285-96e6-3d73a554c4a2', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-19T23:19:46.839 INFO:tasks.rgw:Starting rgw... 2024-04-19T23:19:46.839 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-04-19T23:19:46.840 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-19T23:19:46.840 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-04-19T23:19:46.840 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-04-19T23:19:46.844 INFO:tasks.rgw.c1.client.0:Started 2024-04-19T23:19:46.844 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-04-19T23:19:46.844 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-19T23:19:46.844 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-19T23:19:46.844 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-19T23:19:46.847 INFO:tasks.rgw.c1.client.1:Started 2024-04-19T23:19:46.847 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-04-19T23:19:46.847 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-19T23:19:46.847 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-19T23:19:46.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-19T23:19:46.850 INFO:tasks.rgw.c2.client.0:Started 2024-04-19T23:19:46.850 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-04-19T23:19:46.850 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-19T23:19:46.850 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-04-19T23:19:46.850 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-04-19T23:19:46.853 INFO:tasks.rgw.c2.client.1:Started 2024-04-19T23:19:46.853 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi019.front.sepia.ceph.com:8000/ 2024-04-19T23:19:46.853 DEBUG:teuthology.orchestra.run.smithi019:> curl http://smithi019.front.sepia.ceph.com:8000/ 2024-04-19T23:19:46.891 INFO:teuthology.orchestra.run.smithi019.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-19T23:19:46.892 INFO:teuthology.orchestra.run.smithi019.stderr: Dload Upload Total Spent Left Speed 2024-04-19T23:19:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-19T23:19:46.894 INFO:teuthology.orchestra.run.smithi019.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi019.front.sepia.ceph.com port 8000: Connection refused 2024-04-19T23:19:47.895 DEBUG:teuthology.orchestra.run.smithi019:> curl http://smithi019.front.sepia.ceph.com:8000/ 2024-04-19T23:19:47.922 INFO:teuthology.orchestra.run.smithi019.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-19T23:19:47.923 INFO:teuthology.orchestra.run.smithi019.stderr: Dload Upload Total Spent Left Speed 2024-04-19T23:19:47.924 INFO:teuthology.orchestra.run.smithi019.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-19T23:19:47.925 INFO:teuthology.orchestra.run.smithi019.stdout:anonymous 2024-04-19T23:19:47.925 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi019.front.sepia.ceph.com:8001/ 2024-04-19T23:19:47.925 DEBUG:teuthology.orchestra.run.smithi019:> curl http://smithi019.front.sepia.ceph.com:8001/ 2024-04-19T23:19:47.994 INFO:teuthology.orchestra.run.smithi019.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-19T23:19:47.995 INFO:teuthology.orchestra.run.smithi019.stderr: Dload Upload Total Spent Left Speed 2024-04-19T23:19:47.996 INFO:teuthology.orchestra.run.smithi019.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-19T23:19:47.996 INFO:teuthology.orchestra.run.smithi019.stdout:anonymous 2024-04-19T23:19:47.997 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi031.front.sepia.ceph.com:8000/ 2024-04-19T23:19:47.997 DEBUG:teuthology.orchestra.run.smithi031:> curl http://smithi031.front.sepia.ceph.com:8000/ 2024-04-19T23:19:48.035 INFO:teuthology.orchestra.run.smithi031.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-19T23:19:48.035 INFO:teuthology.orchestra.run.smithi031.stderr: Dload Upload Total Spent Left Speed 2024-04-19T23:19:48.037 INFO:teuthology.orchestra.run.smithi031.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-19T23:19:48.038 INFO:teuthology.orchestra.run.smithi031.stdout:anonymous 2024-04-19T23:19:48.038 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi031.front.sepia.ceph.com:8001/ 2024-04-19T23:19:48.038 DEBUG:teuthology.orchestra.run.smithi031:> curl http://smithi031.front.sepia.ceph.com:8001/ 2024-04-19T23:19:48.105 INFO:teuthology.orchestra.run.smithi031.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-19T23:19:48.105 INFO:teuthology.orchestra.run.smithi031.stderr: Dload Upload Total Spent Left Speed 2024-04-19T23:19:48.106 INFO:teuthology.orchestra.run.smithi031.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-19T23:19:48.106 INFO:teuthology.orchestra.run.smithi031.stdout:anonymous 2024-04-19T23:19:48.107 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-04-19T23:19:48.418 INFO:tasks.rgw_multisite:creating realm.. 2024-04-19T23:19:48.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:19:48.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:19:48.419 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-19T23:19:48.474 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:48.474 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:48.505 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:19:48.505 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:19:48.505 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-realm", 2024-04-19T23:19:48.505 INFO:teuthology.orchestra.run.smithi019.stdout: "current_period": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-19T23:19:48.505 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1 2024-04-19T23:19:48.505 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:19:48.507 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-04-19T23:19:48.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:19:48.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:19:48.507 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-19T23:19:48.561 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:48.561 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:48.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:48.580+0000 7fd5cbc3b780 -1 old format 2024-04-19T23:19:48.586 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:19:48.586 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-19T23:19:48.586 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-19T23:19:48.586 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "", 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [], 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-19T23:19:48.587 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [] 2024-04-19T23:19:48.588 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:48.588 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:19:48.588 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-19T23:19:48.588 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:19:48.588 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-19T23:19:48.588 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [] 2024-04-19T23:19:48.588 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:48.588 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-19T23:19:48.588 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-19T23:19:48.588 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:19:48.588 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:19:48.589 INFO:tasks.rgw_multisite:creating master zone.. 2024-04-19T23:19:48.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'TWO98NBPIQJFTC6N', '--secret', 'WVtqgebnqURqV1qdDSut7FusWVqSrgr8', '--endpoints', 'http://smithi019.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:19:48.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'TWO98NBPIQJFTC6N', '--secret', 'WVtqgebnqURqV1qdDSut7FusWVqSrgr8', '--endpoints', 'http://smithi019.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:19:48.590 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key TWO98NBPIQJFTC6N --secret WVtqgebnqURqV1qdDSut7FusWVqSrgr8 --endpoints http://smithi019.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-19T23:19:48.693 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:48.693 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:48.711 INFO:teuthology.orchestra.run.smithi019.stderr:NOTICE: set zone's realm_id=0542eeed-b7db-454d-a499-a81b016de7e3 2024-04-19T23:19:48.729 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:19:48.730 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:19:48.730 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-19T23:19:48.730 INFO:teuthology.orchestra.run.smithi019.stdout: "domain_root": "test-zone1.rgw.meta:root", 2024-04-19T23:19:48.730 INFO:teuthology.orchestra.run.smithi019.stdout: "control_pool": "test-zone1.rgw.control", 2024-04-19T23:19:48.730 INFO:teuthology.orchestra.run.smithi019.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2024-04-19T23:19:48.730 INFO:teuthology.orchestra.run.smithi019.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2024-04-19T23:19:48.730 INFO:teuthology.orchestra.run.smithi019.stdout: "log_pool": "test-zone1.rgw.log", 2024-04-19T23:19:48.730 INFO:teuthology.orchestra.run.smithi019.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2024-04-19T23:19:48.730 INFO:teuthology.orchestra.run.smithi019.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2024-04-19T23:19:48.730 INFO:teuthology.orchestra.run.smithi019.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2024-04-19T23:19:48.730 INFO:teuthology.orchestra.run.smithi019.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2024-04-19T23:19:48.730 INFO:teuthology.orchestra.run.smithi019.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: "otp_pool": "test-zone1.rgw.otp", 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: "system_key": { 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: "access_key": "TWO98NBPIQJFTC6N", 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: "secret_key": "WVtqgebnqURqV1qdDSut7FusWVqSrgr8" 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_pools": [ 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "default-placement", 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2024-04-19T23:19:48.731 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": { 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD": { 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout: "index_type": 0, 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout: "inline_data": true 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2024-04-19T23:19:48.732 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:19:48.733 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2024-04-19T23:19:49.314 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'test-zone1.rgw.buckets.data' created 2024-04-19T23:19:49.332 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2024-04-19T23:19:50.333 INFO:teuthology.orchestra.run.smithi019.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2024-04-19T23:19:50.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:19:50.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:19:50.356 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-19T23:19:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:50.438 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:19:50.438 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:19:50.438 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-19T23:19:50.438 INFO:teuthology.orchestra.run.smithi019.stdout: "domain_root": "test-zone1.rgw.meta:root", 2024-04-19T23:19:50.438 INFO:teuthology.orchestra.run.smithi019.stdout: "control_pool": "test-zone1.rgw.control", 2024-04-19T23:19:50.438 INFO:teuthology.orchestra.run.smithi019.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2024-04-19T23:19:50.438 INFO:teuthology.orchestra.run.smithi019.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2024-04-19T23:19:50.438 INFO:teuthology.orchestra.run.smithi019.stdout: "log_pool": "test-zone1.rgw.log", 2024-04-19T23:19:50.439 INFO:teuthology.orchestra.run.smithi019.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2024-04-19T23:19:50.439 INFO:teuthology.orchestra.run.smithi019.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2024-04-19T23:19:50.439 INFO:teuthology.orchestra.run.smithi019.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2024-04-19T23:19:50.439 INFO:teuthology.orchestra.run.smithi019.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2024-04-19T23:19:50.439 INFO:teuthology.orchestra.run.smithi019.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2024-04-19T23:19:50.439 INFO:teuthology.orchestra.run.smithi019.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2024-04-19T23:19:50.439 INFO:teuthology.orchestra.run.smithi019.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2024-04-19T23:19:50.439 INFO:teuthology.orchestra.run.smithi019.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2024-04-19T23:19:50.439 INFO:teuthology.orchestra.run.smithi019.stdout: "otp_pool": "test-zone1.rgw.otp", 2024-04-19T23:19:50.439 INFO:teuthology.orchestra.run.smithi019.stdout: "system_key": { 2024-04-19T23:19:50.439 INFO:teuthology.orchestra.run.smithi019.stdout: "access_key": "TWO98NBPIQJFTC6N", 2024-04-19T23:19:50.439 INFO:teuthology.orchestra.run.smithi019.stdout: "secret_key": "WVtqgebnqURqV1qdDSut7FusWVqSrgr8" 2024-04-19T23:19:50.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:50.440 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_pools": [ 2024-04-19T23:19:50.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:19:50.440 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "default-placement", 2024-04-19T23:19:50.440 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:19:50.440 INFO:teuthology.orchestra.run.smithi019.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2024-04-19T23:19:50.440 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": { 2024-04-19T23:19:50.440 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD": { 2024-04-19T23:19:50.440 INFO:teuthology.orchestra.run.smithi019.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2024-04-19T23:19:50.440 INFO:teuthology.orchestra.run.smithi019.stdout: "compression_type": "random" 2024-04-19T23:19:50.440 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:50.441 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:50.441 INFO:teuthology.orchestra.run.smithi019.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2024-04-19T23:19:50.441 INFO:teuthology.orchestra.run.smithi019.stdout: "index_type": 0, 2024-04-19T23:19:50.441 INFO:teuthology.orchestra.run.smithi019.stdout: "inline_data": true 2024-04-19T23:19:50.441 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:50.441 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:50.441 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:19:50.441 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:19:50.441 INFO:teuthology.orchestra.run.smithi019.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2024-04-19T23:19:50.441 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:19:50.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:19:50.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:19:50.443 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-19T23:19:50.496 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:19:50.496 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:19:50.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:19:50.512+0000 7fa4b5c35780 0 period (958711cb-8a73-4fca-9bb6-5b0d38627b0b does not have zone 73f46f02-c7ab-4426-a267-2273af911da5 configured 2024-04-19T23:19:57.407 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:19:57.407 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:19:57.407 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 1, 2024-04-19T23:19:57.407 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-19T23:19:57.407 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-19T23:19:57.407 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-19T23:19:57.407 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:19:57.407 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-19T23:19:57.407 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:19:57.407 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-19T23:19:57.407 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-19T23:19:57.408 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": false, 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-19T23:19:57.409 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:19:57.410 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [] 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:57.411 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-19T23:19:57.412 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-19T23:19:57.413 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:19:57.414 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:19:57.415 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-19T23:19:57.415 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:19:57.415 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:19:57.416 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-04-19T23:19:57.416 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T23:19:57.417 INFO:tasks.rgw.c1.client.0.smithi019.stdout:2024-04-19T23:19:57.420+0000 7fe9d7a38700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 46200) UID: 0 2024-04-19T23:19:57.417 INFO:tasks.rgw.c1.client.0.smithi019.stdout:2024-04-19T23:19:57.420+0000 7fe9dcc0d7c0 -1 shutting down 2024-04-19T23:20:01.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-04-19T23:20:03.519 INFO:tasks.rgw.c1.client.0:Stopped 2024-04-19T23:20:03.519 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-04-19T23:20:03.519 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-04-19T23:20:03.522 INFO:tasks.rgw.c1.client.0:Started 2024-04-19T23:20:03.522 DEBUG:teuthology.orchestra.run.smithi019:> curl http://smithi019.front.sepia.ceph.com:8000 2024-04-19T23:20:03.547 INFO:teuthology.orchestra.run.smithi019.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-19T23:20:03.547 INFO:teuthology.orchestra.run.smithi019.stderr: Dload Upload Total Spent Left Speed 2024-04-19T23:20:03.549 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-19T23:20:03.549 INFO:teuthology.orchestra.run.smithi019.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi019.front.sepia.ceph.com port 8000: Connection refused 2024-04-19T23:20:04.550 DEBUG:teuthology.orchestra.run.smithi019:> curl http://smithi019.front.sepia.ceph.com:8000 2024-04-19T23:20:04.577 INFO:teuthology.orchestra.run.smithi019.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-19T23:20:04.577 INFO:teuthology.orchestra.run.smithi019.stderr: Dload Upload Total Spent Left Speed 2024-04-19T23:20:04.579 INFO:teuthology.orchestra.run.smithi019.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-19T23:20:04.579 INFO:teuthology.orchestra.run.smithi019.stdout:anonymous 2024-04-19T23:20:04.580 INFO:tasks.rgw_multisite:creating admin user.. 2024-04-19T23:20:04.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'TWO98NBPIQJFTC6N', '--secret', 'WVtqgebnqURqV1qdDSut7FusWVqSrgr8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:04.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'TWO98NBPIQJFTC6N', '--secret', 'WVtqgebnqURqV1qdDSut7FusWVqSrgr8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:04.580 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key TWO98NBPIQJFTC6N --secret WVtqgebnqURqV1qdDSut7FusWVqSrgr8 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-19T23:20:04.687 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:04.687 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:04.728 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: "user_id": "realm-admin", 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: "display_name": "Realm Admin", 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: "email": "", 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: "suspended": 0, 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: "max_buckets": 1000, 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: "subusers": [], 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: "keys": [ 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "realm-admin", 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: "access_key": "TWO98NBPIQJFTC6N", 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: "secret_key": "WVtqgebnqURqV1qdDSut7FusWVqSrgr8" 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:04.729 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:20:04.730 INFO:teuthology.orchestra.run.smithi019.stdout: "swift_keys": [], 2024-04-19T23:20:04.730 INFO:teuthology.orchestra.run.smithi019.stdout: "caps": [], 2024-04-19T23:20:04.730 INFO:teuthology.orchestra.run.smithi019.stdout: "op_mask": "read, write, delete", 2024-04-19T23:20:04.730 INFO:teuthology.orchestra.run.smithi019.stdout: "system": true, 2024-04-19T23:20:04.730 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "", 2024-04-19T23:20:04.730 INFO:teuthology.orchestra.run.smithi019.stdout: "default_storage_class": "", 2024-04-19T23:20:04.730 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_tags": [], 2024-04-19T23:20:04.730 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-19T23:20:04.730 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-19T23:20:04.730 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-19T23:20:04.730 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-19T23:20:04.730 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-19T23:20:04.731 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-19T23:20:04.731 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:04.731 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-19T23:20:04.731 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-19T23:20:04.731 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-19T23:20:04.731 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-19T23:20:04.731 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-19T23:20:04.731 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-19T23:20:04.731 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:04.731 INFO:teuthology.orchestra.run.smithi019.stdout: "temp_url_keys": [], 2024-04-19T23:20:04.731 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "rgw", 2024-04-19T23:20:04.732 INFO:teuthology.orchestra.run.smithi019.stdout: "mfa_ids": [] 2024-04-19T23:20:04.732 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:20:04.732 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:20:04.743 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-04-19T23:20:04.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi019.front.sepia.ceph.com:8000', '--access-key', 'TWO98NBPIQJFTC6N', '--secret', 'WVtqgebnqURqV1qdDSut7FusWVqSrgr8', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:04.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi019.front.sepia.ceph.com:8000', '--access-key', 'TWO98NBPIQJFTC6N', '--secret', 'WVtqgebnqURqV1qdDSut7FusWVqSrgr8', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:04.743 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi019.front.sepia.ceph.com:8000 --access-key TWO98NBPIQJFTC6N --secret WVtqgebnqURqV1qdDSut7FusWVqSrgr8 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-19T23:20:04.805 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:04.805 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:04.854 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:20:04.854 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:20:04.854 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-realm", 2024-04-19T23:20:04.854 INFO:teuthology.orchestra.run.smithi031.stdout: "current_period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:20:04.854 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 2 2024-04-19T23:20:04.854 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:20:04.855 INFO:tasks.rgw_multisite:creating zone.. 2024-04-19T23:20:04.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'TWO98NBPIQJFTC6N', '--secret', 'WVtqgebnqURqV1qdDSut7FusWVqSrgr8', '--endpoints', 'http://smithi031.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:04.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'TWO98NBPIQJFTC6N', '--secret', 'WVtqgebnqURqV1qdDSut7FusWVqSrgr8', '--endpoints', 'http://smithi031.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:04.856 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key TWO98NBPIQJFTC6N --secret WVtqgebnqURqV1qdDSut7FusWVqSrgr8 --endpoints http://smithi031.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-19T23:20:04.917 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:04.918 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:NOTICE: set zone's realm_id=0542eeed-b7db-454d-a499-a81b016de7e3 2024-04-19T23:20:04.952 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:20:04.952 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-19T23:20:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zone2", 2024-04-19T23:20:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-04-19T23:20:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: "control_pool": "test-zone2.rgw.control", 2024-04-19T23:20:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-04-19T23:20:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-04-19T23:20:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: "log_pool": "test-zone2.rgw.log", 2024-04-19T23:20:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-04-19T23:20:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-04-19T23:20:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-04-19T23:20:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-04-19T23:20:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-04-19T23:20:04.953 INFO:teuthology.orchestra.run.smithi031.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: "system_key": { 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: "access_key": "TWO98NBPIQJFTC6N", 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: "secret_key": "WVtqgebnqURqV1qdDSut7FusWVqSrgr8" 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: "placement_pools": [ 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "default-placement", 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-04-19T23:20:04.954 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_classes": { 2024-04-19T23:20:04.955 INFO:teuthology.orchestra.run.smithi031.stdout: "STANDARD": { 2024-04-19T23:20:04.955 INFO:teuthology.orchestra.run.smithi031.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2024-04-19T23:20:04.955 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:04.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:04.955 INFO:teuthology.orchestra.run.smithi031.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-04-19T23:20:04.955 INFO:teuthology.orchestra.run.smithi031.stdout: "index_type": 0, 2024-04-19T23:20:04.955 INFO:teuthology.orchestra.run.smithi031.stdout: "inline_data": true 2024-04-19T23:20:04.955 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:04.955 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:04.955 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:20:04.955 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:20:04.955 INFO:teuthology.orchestra.run.smithi031.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-04-19T23:20:04.956 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:20:04.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2024-04-19T23:20:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'test-zone2.rgw.buckets.data' created 2024-04-19T23:20:05.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2024-04-19T23:20:06.512 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2024-04-19T23:20:06.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:06.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:06.534 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-19T23:20:06.591 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:06.592 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:06.613 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:20:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-19T23:20:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zone2", 2024-04-19T23:20:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-04-19T23:20:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "control_pool": "test-zone2.rgw.control", 2024-04-19T23:20:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-04-19T23:20:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-04-19T23:20:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "log_pool": "test-zone2.rgw.log", 2024-04-19T23:20:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-04-19T23:20:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-04-19T23:20:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-04-19T23:20:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-04-19T23:20:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-04-19T23:20:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-04-19T23:20:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-04-19T23:20:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-04-19T23:20:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-04-19T23:20:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "system_key": { 2024-04-19T23:20:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "access_key": "TWO98NBPIQJFTC6N", 2024-04-19T23:20:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "secret_key": "WVtqgebnqURqV1qdDSut7FusWVqSrgr8" 2024-04-19T23:20:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "placement_pools": [ 2024-04-19T23:20:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "default-placement", 2024-04-19T23:20:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-04-19T23:20:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_classes": { 2024-04-19T23:20:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "STANDARD": { 2024-04-19T23:20:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2024-04-19T23:20:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "compression_type": "random" 2024-04-19T23:20:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-04-19T23:20:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "index_type": 0, 2024-04-19T23:20:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "inline_data": true 2024-04-19T23:20:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:20:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:20:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-04-19T23:20:06.620 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:20:06.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:06.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:06.622 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-19T23:20:06.675 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:06.675 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:06.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:20:06.692+0000 7f248285e780 0 period (f494ba41-e721-4411-8938-a297d95f7246 does not have zone 03463f3d-be9b-4035-b2ce-7cd216cfabda configured 2024-04-19T23:20:07.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-04-19T23:20:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:Sending period to new master zone 73f46f02-c7ab-4426-a267-2273af911da5 2024-04-19T23:20:14.592 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:20:14.592 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:20:14.592 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 2, 2024-04-19T23:20:14.592 INFO:teuthology.orchestra.run.smithi031.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-19T23:20:14.592 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": [], 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: "period_map": { 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: "zonegroups": [ 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "test-zonegroup", 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zonegroup", 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: "api_name": "", 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: "is_master": true, 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: "endpoints": [ 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: "hostnames": [], 2024-04-19T23:20:14.593 INFO:teuthology.orchestra.run.smithi031.stdout: "hostnames_s3website": [], 2024-04-19T23:20:14.594 INFO:teuthology.orchestra.run.smithi031.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:20:14.594 INFO:teuthology.orchestra.run.smithi031.stdout: "zones": [ 2024-04-19T23:20:14.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:14.594 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-19T23:20:14.594 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zone2", 2024-04-19T23:20:14.594 INFO:teuthology.orchestra.run.smithi031.stdout: "endpoints": [ 2024-04-19T23:20:14.594 INFO:teuthology.orchestra.run.smithi031.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-19T23:20:14.594 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:20:14.594 INFO:teuthology.orchestra.run.smithi031.stdout: "log_meta": false, 2024-04-19T23:20:14.594 INFO:teuthology.orchestra.run.smithi031.stdout: "log_data": true, 2024-04-19T23:20:14.594 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_index_max_shards": 11, 2024-04-19T23:20:14.595 INFO:teuthology.orchestra.run.smithi031.stdout: "read_only": false, 2024-04-19T23:20:14.595 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_type": "", 2024-04-19T23:20:14.595 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from_all": true, 2024-04-19T23:20:14.595 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from": [], 2024-04-19T23:20:14.595 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_zone": "", 2024-04-19T23:20:14.595 INFO:teuthology.orchestra.run.smithi031.stdout: "supported_features": [ 2024-04-19T23:20:14.595 INFO:teuthology.orchestra.run.smithi031.stdout: "compress-encrypted", 2024-04-19T23:20:14.595 INFO:teuthology.orchestra.run.smithi031.stdout: "resharding" 2024-04-19T23:20:14.595 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:20:14.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:14.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:14.595 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zone1", 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "endpoints": [ 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "log_meta": false, 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "log_data": true, 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_index_max_shards": 11, 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "read_only": false, 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_type": "", 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from_all": true, 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from": [], 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_zone": "", 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "supported_features": [ 2024-04-19T23:20:14.596 INFO:teuthology.orchestra.run.smithi031.stdout: "compress-encrypted", 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: "resharding" 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: "placement_targets": [ 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default-placement", 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_classes": [ 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: "STANDARD" 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:20:14.597 INFO:teuthology.orchestra.run.smithi031.stdout: "default_placement": "default-placement", 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_policy": { 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: "groups": [] 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled_features": [ 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: "resharding" 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: "short_zone_ids": [ 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-19T23:20:14.598 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 2010863401 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 1221149675 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: "master_zonegroup": "test-zonegroup", 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: "period_config": { 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_quota": { 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false, 2024-04-19T23:20:14.599 INFO:teuthology.orchestra.run.smithi031.stdout: "check_on_raw": false, 2024-04-19T23:20:14.600 INFO:teuthology.orchestra.run.smithi031.stdout: "max_size": -1, 2024-04-19T23:20:14.600 INFO:teuthology.orchestra.run.smithi031.stdout: "max_size_kb": 0, 2024-04-19T23:20:14.600 INFO:teuthology.orchestra.run.smithi031.stdout: "max_objects": -1 2024-04-19T23:20:14.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:14.600 INFO:teuthology.orchestra.run.smithi031.stdout: "user_quota": { 2024-04-19T23:20:14.600 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false, 2024-04-19T23:20:14.601 INFO:teuthology.orchestra.run.smithi031.stdout: "check_on_raw": false, 2024-04-19T23:20:14.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max_size": -1, 2024-04-19T23:20:14.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max_size_kb": 0, 2024-04-19T23:20:14.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max_objects": -1 2024-04-19T23:20:14.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:14.601 INFO:teuthology.orchestra.run.smithi031.stdout: "user_ratelimit": { 2024-04-19T23:20:14.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_ops": 0, 2024-04-19T23:20:14.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_ops": 0, 2024-04-19T23:20:14.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_bytes": 0, 2024-04-19T23:20:14.602 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_bytes": 0, 2024-04-19T23:20:14.602 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false 2024-04-19T23:20:14.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:14.602 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_ratelimit": { 2024-04-19T23:20:14.602 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_ops": 0, 2024-04-19T23:20:14.602 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_ops": 0, 2024-04-19T23:20:14.607 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_bytes": 0, 2024-04-19T23:20:14.607 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_bytes": 0, 2024-04-19T23:20:14.607 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false 2024-04-19T23:20:14.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:14.607 INFO:teuthology.orchestra.run.smithi031.stdout: "anonymous_ratelimit": { 2024-04-19T23:20:14.607 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_ops": 0, 2024-04-19T23:20:14.607 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_ops": 0, 2024-04-19T23:20:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_bytes": 0, 2024-04-19T23:20:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_bytes": 0, 2024-04-19T23:20:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false 2024-04-19T23:20:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:20:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_name": "test-realm", 2024-04-19T23:20:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:14.608 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:20:14.610 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-19T23:20:14.610 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T23:20:14.610 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-19T23:20:14.614+0000 7f42ed606700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 46723) UID: 0 2024-04-19T23:20:14.611 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-19T23:20:14.614+0000 7f42f27db7c0 -1 shutting down 2024-04-19T23:20:18.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-19T23:20:20.712 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-19T23:20:20.712 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-19T23:20:20.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-19T23:20:20.715 INFO:tasks.rgw.c2.client.0:Started 2024-04-19T23:20:20.716 DEBUG:teuthology.orchestra.run.smithi031:> curl http://smithi031.front.sepia.ceph.com:8000 2024-04-19T23:20:20.745 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-19T23:20:20.745 INFO:teuthology.orchestra.run.smithi031.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-19T23:20:20.745 INFO:teuthology.orchestra.run.smithi031.stderr: Dload Upload Total Spent Left Speed 2024-04-19T23:20:20.745 INFO:teuthology.orchestra.run.smithi031.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi031.front.sepia.ceph.com port 8000: Connection refused 2024-04-19T23:20:21.747 DEBUG:teuthology.orchestra.run.smithi031:> curl http://smithi031.front.sepia.ceph.com:8000 2024-04-19T23:20:21.774 INFO:teuthology.orchestra.run.smithi031.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-19T23:20:21.775 INFO:teuthology.orchestra.run.smithi031.stderr: Dload Upload Total Spent Left Speed 2024-04-19T23:20:21.776 INFO:teuthology.orchestra.run.smithi031.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-19T23:20:21.776 INFO:teuthology.orchestra.run.smithi031.stdout:anonymous 2024-04-19T23:20:21.777 INFO:tasks.rgw_multisite:creating zone.. 2024-04-19T23:20:21.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'TWO98NBPIQJFTC6N', '--secret', 'WVtqgebnqURqV1qdDSut7FusWVqSrgr8', '--endpoints', 'http://smithi019.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:21.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'TWO98NBPIQJFTC6N', '--secret', 'WVtqgebnqURqV1qdDSut7FusWVqSrgr8', '--endpoints', 'http://smithi019.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:21.777 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key TWO98NBPIQJFTC6N --secret WVtqgebnqURqV1qdDSut7FusWVqSrgr8 --endpoints http://smithi019.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-19T23:20:21.841 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:21.841 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:21.862 INFO:teuthology.orchestra.run.smithi019.stderr:NOTICE: set zone's realm_id=0542eeed-b7db-454d-a499-a81b016de7e3 2024-04-19T23:20:21.878 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:20:21.878 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-19T23:20:21.878 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-19T23:20:21.878 INFO:teuthology.orchestra.run.smithi019.stdout: "domain_root": "test-zone3.rgw.meta:root", 2024-04-19T23:20:21.879 INFO:teuthology.orchestra.run.smithi019.stdout: "control_pool": "test-zone3.rgw.control", 2024-04-19T23:20:21.879 INFO:teuthology.orchestra.run.smithi019.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2024-04-19T23:20:21.879 INFO:teuthology.orchestra.run.smithi019.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2024-04-19T23:20:21.879 INFO:teuthology.orchestra.run.smithi019.stdout: "log_pool": "test-zone3.rgw.log", 2024-04-19T23:20:21.879 INFO:teuthology.orchestra.run.smithi019.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2024-04-19T23:20:21.879 INFO:teuthology.orchestra.run.smithi019.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2024-04-19T23:20:21.879 INFO:teuthology.orchestra.run.smithi019.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2024-04-19T23:20:21.879 INFO:teuthology.orchestra.run.smithi019.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2024-04-19T23:20:21.879 INFO:teuthology.orchestra.run.smithi019.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2024-04-19T23:20:21.880 INFO:teuthology.orchestra.run.smithi019.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2024-04-19T23:20:21.880 INFO:teuthology.orchestra.run.smithi019.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2024-04-19T23:20:21.880 INFO:teuthology.orchestra.run.smithi019.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2024-04-19T23:20:21.880 INFO:teuthology.orchestra.run.smithi019.stdout: "otp_pool": "test-zone3.rgw.otp", 2024-04-19T23:20:21.880 INFO:teuthology.orchestra.run.smithi019.stdout: "system_key": { 2024-04-19T23:20:21.880 INFO:teuthology.orchestra.run.smithi019.stdout: "access_key": "TWO98NBPIQJFTC6N", 2024-04-19T23:20:21.880 INFO:teuthology.orchestra.run.smithi019.stdout: "secret_key": "WVtqgebnqURqV1qdDSut7FusWVqSrgr8" 2024-04-19T23:20:21.881 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:21.881 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_pools": [ 2024-04-19T23:20:21.881 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:21.881 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "default-placement", 2024-04-19T23:20:21.881 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:20:21.881 INFO:teuthology.orchestra.run.smithi019.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2024-04-19T23:20:21.881 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": { 2024-04-19T23:20:21.881 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD": { 2024-04-19T23:20:21.882 INFO:teuthology.orchestra.run.smithi019.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2024-04-19T23:20:21.882 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:21.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:21.882 INFO:teuthology.orchestra.run.smithi019.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2024-04-19T23:20:21.882 INFO:teuthology.orchestra.run.smithi019.stdout: "index_type": 0, 2024-04-19T23:20:21.882 INFO:teuthology.orchestra.run.smithi019.stdout: "inline_data": true 2024-04-19T23:20:21.882 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:21.882 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:21.883 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:20:21.883 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:20:21.883 INFO:teuthology.orchestra.run.smithi019.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2024-04-19T23:20:21.883 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:20:21.884 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2024-04-19T23:20:22.347 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'test-zone3.rgw.buckets.data' created 2024-04-19T23:20:22.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2024-04-19T23:20:23.351 INFO:teuthology.orchestra.run.smithi019.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2024-04-19T23:20:23.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:23.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:23.373 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-19T23:20:23.436 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:23.436 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:23.457 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:20:23.457 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-19T23:20:23.457 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-19T23:20:23.457 INFO:teuthology.orchestra.run.smithi019.stdout: "domain_root": "test-zone3.rgw.meta:root", 2024-04-19T23:20:23.457 INFO:teuthology.orchestra.run.smithi019.stdout: "control_pool": "test-zone3.rgw.control", 2024-04-19T23:20:23.457 INFO:teuthology.orchestra.run.smithi019.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2024-04-19T23:20:23.457 INFO:teuthology.orchestra.run.smithi019.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2024-04-19T23:20:23.457 INFO:teuthology.orchestra.run.smithi019.stdout: "log_pool": "test-zone3.rgw.log", 2024-04-19T23:20:23.457 INFO:teuthology.orchestra.run.smithi019.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2024-04-19T23:20:23.457 INFO:teuthology.orchestra.run.smithi019.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2024-04-19T23:20:23.457 INFO:teuthology.orchestra.run.smithi019.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: "otp_pool": "test-zone3.rgw.otp", 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: "system_key": { 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: "access_key": "TWO98NBPIQJFTC6N", 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: "secret_key": "WVtqgebnqURqV1qdDSut7FusWVqSrgr8" 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_pools": [ 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "default-placement", 2024-04-19T23:20:23.458 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": { 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD": { 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: "compression_type": "random" 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: "index_type": 0, 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: "inline_data": true 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:23.459 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:20:23.460 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:20:23.460 INFO:teuthology.orchestra.run.smithi019.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2024-04-19T23:20:23.460 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:20:23.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:23.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:23.461 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-19T23:20:23.513 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:23.513 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:23.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:20:23.530+0000 7fe167165780 0 period (f494ba41-e721-4411-8938-a297d95f7246 does not have zone bea5de7f-54a0-40b9-979f-37d0f728ff53 configured 2024-04-19T23:20:23.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-19T23:20:30.425 INFO:teuthology.orchestra.run.smithi019.stderr:Sending period to new master zone 73f46f02-c7ab-4426-a267-2273af911da5 2024-04-19T23:20:30.448 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:20:30.448 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:20:30.448 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 3, 2024-04-19T23:20:30.448 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-19T23:20:30.448 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-19T23:20:30.448 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-19T23:20:30.448 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:20:30.448 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-19T23:20:30.448 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:30.448 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-19T23:20:30.449 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-19T23:20:30.449 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-19T23:20:30.449 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-19T23:20:30.449 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-19T23:20:30.449 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-19T23:20:30.449 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:20:30.449 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-19T23:20:30.449 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-19T23:20:30.449 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:20:30.449 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-19T23:20:30.450 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:30.450 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-19T23:20:30.450 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-19T23:20:30.450 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-19T23:20:30.450 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-19T23:20:30.450 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:20:30.450 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-19T23:20:30.450 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-19T23:20:30.450 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-19T23:20:30.450 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-19T23:20:30.450 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-19T23:20:30.450 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-19T23:20:30.451 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-19T23:20:30.451 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-19T23:20:30.451 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-19T23:20:30.451 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-19T23:20:30.451 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-19T23:20:30.451 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:20:30.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:30.451 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:30.452 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:20:30.452 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-19T23:20:30.452 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-19T23:20:30.452 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-19T23:20:30.452 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:20:30.453 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-19T23:20:30.453 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-19T23:20:30.453 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-19T23:20:30.453 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-19T23:20:30.453 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-19T23:20:30.453 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-19T23:20:30.454 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-19T23:20:30.454 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-19T23:20:30.454 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-19T23:20:30.454 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-19T23:20:30.454 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-19T23:20:30.454 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:20:30.455 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:30.455 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:30.455 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-19T23:20:30.455 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-19T23:20:30.455 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-19T23:20:30.455 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-19T23:20:30.456 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:20:30.456 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-19T23:20:30.456 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-19T23:20:30.456 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-19T23:20:30.456 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-19T23:20:30.456 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-19T23:20:30.457 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-19T23:20:30.457 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-19T23:20:30.457 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-19T23:20:30.457 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-19T23:20:30.457 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-19T23:20:30.457 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-19T23:20:30.457 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:20:30.458 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:30.458 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:20:30.458 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-19T23:20:30.458 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:30.458 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-19T23:20:30.459 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-19T23:20:30.459 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-19T23:20:30.459 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-19T23:20:30.459 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:20:30.459 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:30.459 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:20:30.460 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-19T23:20:30.460 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:20:30.460 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-19T23:20:30.460 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [] 2024-04-19T23:20:30.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:30.460 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-19T23:20:30.461 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-19T23:20:30.461 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:20:30.461 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:30.461 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:20:30.461 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-19T23:20:30.461 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:30.461 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-19T23:20:30.462 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-19T23:20:30.462 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:30.462 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:30.462 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:20:30.463 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-19T23:20:30.465 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:30.465 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:30.465 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-19T23:20:30.465 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-19T23:20:30.466 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:30.466 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:20:30.466 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:30.466 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-19T23:20:30.466 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:20:30.466 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-19T23:20:30.467 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-19T23:20:30.467 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-19T23:20:30.467 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-19T23:20:30.467 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-19T23:20:30.467 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-19T23:20:30.467 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-19T23:20:30.468 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:30.468 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-19T23:20:30.468 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-19T23:20:30.468 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-19T23:20:30.468 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-19T23:20:30.468 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-19T23:20:30.469 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-19T23:20:30.469 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:30.469 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-19T23:20:30.469 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-19T23:20:30.469 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-19T23:20:30.469 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-19T23:20:30.470 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-19T23:20:30.470 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-19T23:20:30.470 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:30.470 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-19T23:20:30.470 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-19T23:20:30.470 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-19T23:20:30.470 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-19T23:20:30.471 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-19T23:20:30.471 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-19T23:20:30.471 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:30.471 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-19T23:20:30.471 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-19T23:20:30.471 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-19T23:20:30.472 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-19T23:20:30.472 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-19T23:20:30.472 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-19T23:20:30.472 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:30.472 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:30.472 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:20:30.472 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-19T23:20:30.473 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:20:30.473 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:20:30.474 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-19T23:20:30.474 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T23:20:30.475 INFO:tasks.rgw.c1.client.1.smithi019.stdout:2024-04-19T23:20:30.479+0000 7fb74f04b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 46202) UID: 0 2024-04-19T23:20:30.475 INFO:tasks.rgw.c1.client.1.smithi019.stdout:2024-04-19T23:20:30.479+0000 7fb7542207c0 -1 shutting down 2024-04-19T23:20:35.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-19T23:20:36.575 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-19T23:20:36.575 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-19T23:20:36.576 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-19T23:20:36.578 INFO:tasks.rgw.c1.client.1:Started 2024-04-19T23:20:36.579 DEBUG:teuthology.orchestra.run.smithi019:> curl http://smithi019.front.sepia.ceph.com:8001 2024-04-19T23:20:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-19T23:20:36.611 INFO:teuthology.orchestra.run.smithi019.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-19T23:20:36.611 INFO:teuthology.orchestra.run.smithi019.stderr: Dload Upload Total Spent Left Speed 2024-04-19T23:20:36.611 INFO:teuthology.orchestra.run.smithi019.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi019.front.sepia.ceph.com port 8001: Connection refused 2024-04-19T23:20:37.612 DEBUG:teuthology.orchestra.run.smithi019:> curl http://smithi019.front.sepia.ceph.com:8001 2024-04-19T23:20:37.639 INFO:teuthology.orchestra.run.smithi019.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-19T23:20:37.639 INFO:teuthology.orchestra.run.smithi019.stderr: Dload Upload Total Spent Left Speed 2024-04-19T23:20:37.640 INFO:teuthology.orchestra.run.smithi019.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-19T23:20:37.641 INFO:teuthology.orchestra.run.smithi019.stdout:anonymous 2024-04-19T23:20:37.642 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-04-19T23:20:37.642 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-04-19T23:20:37.671 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi 2024-04-19T23:20:37.732 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-04-19T23:20:37.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:37.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:20:37.732 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-19T23:20:37.797 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:37.798 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:37.838 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:20:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: "user_id": "rgw-multisite-test-user", 2024-04-19T23:20:37.838 INFO:teuthology.orchestra.run.smithi019.stdout: "display_name": "Multisite Test User", 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "email": "", 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "suspended": 0, 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "max_buckets": 1000, 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "subusers": [], 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "keys": [ 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "rgw-multisite-test-user", 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "access_key": "T4PF8H1DJQT67HZSKX5F", 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "secret_key": "wJn6WgZZWlEnedZEmgbCQRjn1iF0A31Zc1sKxtDX" 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "swift_keys": [], 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: "caps": [ 2024-04-19T23:20:37.839 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "roles", 2024-04-19T23:20:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "perm": "*" 2024-04-19T23:20:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-19T23:20:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "op_mask": "read, write, delete", 2024-04-19T23:20:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "", 2024-04-19T23:20:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "default_storage_class": "", 2024-04-19T23:20:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_tags": [], 2024-04-19T23:20:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-19T23:20:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-19T23:20:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-19T23:20:37.840 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "temp_url_keys": [], 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "rgw", 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout: "mfa_ids": [] 2024-04-19T23:20:37.841 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:20:37.842 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-19T23:20:37.853 INFO:rgw_multi.tests:meta checkpoint 2024-04-19T23:20:37.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:20:37.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:20:37.853 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:20:37.909 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:37.910 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:37.943 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:20:37.959 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:20:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:20:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:20:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:20:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:20:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:20:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:20:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:20:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:20:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:20:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:20:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:20:37.960 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:20:37.961 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:20:37.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:20:37.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:20:37.962 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:20:38.020 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:38.020 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:38.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:20:38.058+0000 7f42d8a68780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:20:38.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:20:38.059+0000 7f42d8a68780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:20:38.068 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:20:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:20:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:20:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:20:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:20:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:20:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:20:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:20:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:38.071 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:38.072 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:38.073 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:38.073 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:38.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:38.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:38.073 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:20:38.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:38.073 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:20:38.073 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:20:38.073 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:20:38.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:38.073 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:20:38Z" 2024-04-19T23:20:38.073 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:20:38.074 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:20:41.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-19T23:20:43.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:20:43.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:20:43.076 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:20:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:43.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:20:43.178+0000 7f467be18780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:20:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:20:43.179+0000 7f467be18780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:20:43.188 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:20:43.189 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:20:43.189 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:20:43.189 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:20:43.189 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:20:43.189 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:20:43.189 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:43.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:43.189 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:20:43.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:43.189 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:20:43.189 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:43.189 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:43.190 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:43.191 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:43.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:43.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:43.193 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:20:43.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:43.193 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:20:43.193 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:20:43.193 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:20:43.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:43.193 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:20:43Z" 2024-04-19T23:20:43.193 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:20:43.194 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:20:48.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:20:48.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:20:48.195 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:20:48.254 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:48.255 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:48.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:20:48.292+0000 7f1cd0e01780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:20:48.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:20:48.293+0000 7f1cd0e01780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:20:48.303 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:20:48.304 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:20:48.304 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:20:48.304 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:20:48.304 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:20:48.304 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:20:48.304 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:48.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:48.304 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:20:48.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:48.304 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:20:48.304 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:48.304 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:48.305 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:48.306 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:48.306 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:48.306 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:48.306 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:48.306 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:48.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:48.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:48.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:48.306 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:20:48.306 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:48.306 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:48.306 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:48.307 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:20:48Z" 2024-04-19T23:20:48.308 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:20:48.309 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:20:53.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:20:53.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:20:53.311 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:20:53.371 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:53.371 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:53.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:20:53.409+0000 7f84f59da780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:20:53.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:20:53.410+0000 7f84f59da780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:20:53.420 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:20:53.420 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:20:53.420 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:20:53.420 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:20:53.420 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:20:53.420 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:20:53.420 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:53.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:53.420 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:53.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:53.422 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:53.423 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:53.424 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:53.424 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:53.424 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:53.424 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:53.424 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:53.424 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:53.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:53.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:53.424 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:20:53.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:53.424 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:20:53.425 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:20:53.425 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:20:53.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:53.425 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:20:53Z" 2024-04-19T23:20:53.425 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:20:53.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:20:58.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:20:58.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:20:58.427 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:20:58.487 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:20:58.487 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:20:58.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:20:58.525+0000 7f58b6fb7780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:20:58.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:20:58.527+0000 7f58b6fb7780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:20:58.537 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:20:58.538 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:20:58.538 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:20:58.538 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:20:58.538 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:20:58.538 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:20:58.538 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:58.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:58.538 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:20:58.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:58.538 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:20:58.538 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:58.539 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:58.540 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:58.540 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:58.540 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:58.540 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:58.540 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:58.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:58.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:58.540 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:20:58.540 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:58.540 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:58.540 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:58.540 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:20:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:20:58.542 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:20:58.542 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:20:58.542 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:58.542 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:20:58.542 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:20:58.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:58.542 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:20:58.542 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:20:58.542 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:20:58.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:20:58.542 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:20:58Z" 2024-04-19T23:20:58.542 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:20:58.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:21:03.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:03.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:03.546 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:21:03.604 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:21:03.605 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:21:03.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:03.643+0000 7f17a9a8a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:03.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:03.644+0000 7f17a9a8a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:03.654 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:21:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:21:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:21:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:21:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:21:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:21:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:21:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:21:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:21:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:21:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:21:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:21:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:21:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:21:03Z" 2024-04-19T23:21:03.659 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:21:03.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:21:08.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:08.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:08.662 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:21:08.721 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:21:08.721 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:21:08.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:08.762+0000 7f1c5d3a2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:08.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:08.763+0000 7f1c5d3a2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:08.773 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:21:08.773 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:21:08.773 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:21:08.773 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:21:08.773 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:21:08.773 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:21:08.773 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:08.774 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:08.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:21:08.776 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:21:08.777 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:21:08.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:08.778 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:21:08Z" 2024-04-19T23:21:08.778 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:21:08.779 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:21:13.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:13.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:13.780 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:21:13.840 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:21:13.841 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:21:13.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:13.879+0000 7fa6f9016780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:13.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:13.880+0000 7fa6f9016780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:13.890 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:21:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:21:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:21:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:21:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:21:13.890 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:13.891 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:13.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:21:13.893 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:21:13.894 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:21:13.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:13.895 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:21:13Z" 2024-04-19T23:21:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:21:13.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:21:18.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:18.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:18.898 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:21:18.957 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:21:18.957 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:21:18.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:18.997+0000 7f285295c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:18.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:18.998+0000 7f285295c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:19.008 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:21:19.008 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:21:19.008 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:21:19.008 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:21:19.008 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:21:19.008 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:19.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:19.010 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:19.011 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:21:19.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:19.013 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:21:19Z" 2024-04-19T23:21:19.013 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:21:19.014 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:21:24.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:24.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:24.015 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:21:24.076 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:21:24.076 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:21:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:24.115+0000 7f81f62c0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:24.116+0000 7f81f62c0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:24.126 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:21:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:21:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:21:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:21:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:21:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:21:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:21:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:21:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:21:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:21:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:21:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:21:24Z" 2024-04-19T23:21:24.130 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:21:24.131 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:21:29.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:29.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:29.133 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:21:29.193 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:21:29.193 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:21:29.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:29.231+0000 7fa748a53780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:29.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:29.233+0000 7fa748a53780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:29.243 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:21:29.243 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:21:29.243 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:21:29.243 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:21:29.243 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:21:29.243 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:21:29.243 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:29.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:29.243 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:21:29.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:29.243 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:21:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:21:29Z" 2024-04-19T23:21:29.248 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:21:29.248 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:21:34.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:34.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:34.250 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:21:34.309 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:21:34.310 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:21:34.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:34.348+0000 7f0f3eb9a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:34.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:34.349+0000 7f0f3eb9a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:34.358 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:21:34.358 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:21:34.358 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:21:34.359 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:34.360 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:34.361 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:34.362 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:21:34.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:34.363 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:21:34.363 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:21:34.363 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:21:34.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:34.363 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:21:34Z" 2024-04-19T23:21:34.363 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:21:34.364 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:21:39.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:39.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:39.365 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:21:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:21:39.424 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:21:39.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:39.461+0000 7f2977282780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:39.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:39.463+0000 7f2977282780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:39.472 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:21:39.472 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:21:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:21:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:21:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:21:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:21:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:21:39Z" 2024-04-19T23:21:39.477 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:21:39.478 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:21:44.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:44.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:44.480 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:21:44.541 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:21:44.541 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:21:44.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:44.581+0000 7fe412d95780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:44.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:44.582+0000 7fe412d95780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:44.592 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:21:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:21:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:21:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:21:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:21:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:21:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:44.592 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:44.593 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:21:44.594 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:44.595 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:21:44.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:21:44Z" 2024-04-19T23:21:44.597 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:21:44.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:21:49.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:49.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:49.599 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:21:49.658 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:21:49.658 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:21:49.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:49.696+0000 7f302e53b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:49.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:49.697+0000 7f302e53b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:49.708 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:21:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:21:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:21:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:21:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:21:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:21:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:21:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:49.709 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:21:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:21:49.711 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:21:49.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:49.713 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:21:49Z" 2024-04-19T23:21:49.713 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:21:49.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:21:54.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:54.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:54.715 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:21:54.774 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:21:54.774 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:21:54.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:54.815+0000 7fe85aa7a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:54.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:54.816+0000 7fe85aa7a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:54.827 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:21:54.827 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:21:54.827 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:21:54.827 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:21:54.827 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:21:54.827 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:21:54.827 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:54.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:54.827 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:21:54.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:54.828 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:54.829 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:54.830 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:21:54Z" 2024-04-19T23:21:54.831 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:21:54.832 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:21:59.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:59.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:21:59.834 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:21:59.892 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:21:59.892 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:21:59.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:59.929+0000 7ff027150780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:59.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:21:59.931+0000 7ff027150780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:21:59.940 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:21:59.940 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:59.941 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:59.942 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:59.943 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:21:59.944 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:59.945 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:21:59.945 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:21:59.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:59.945 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:21:59.945 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:21:59.945 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:21:59.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:21:59.945 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:21:59Z" 2024-04-19T23:21:59.945 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:21:59.946 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:22:04.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:04.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:04.947 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:05.006 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:05.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:05.045+0000 7fd2b6bba780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:05.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:05.047+0000 7fd2b6bba780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:05.056 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:22:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:22:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:05.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:05.061 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:05.061 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:22:05.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:05.061 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:22:05.061 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:22:05.061 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:22:05.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:05.061 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:22:05Z" 2024-04-19T23:22:05.061 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:22:05.062 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:22:10.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:10.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:10.064 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:10.123 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:10.124 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:10.163+0000 7fe7547b5780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:10.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:10.165+0000 7fe7547b5780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:10.175 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:22:10.175 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:22:10.175 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:22:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:22:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:22:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:22:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:22:10Z" 2024-04-19T23:22:10.180 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:22:10.181 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:22:15.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:15.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:15.183 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:15.242 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:15.243 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:15.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:15.280+0000 7fa6426d1780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:15.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:15.281+0000 7fa6426d1780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:15.291 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:22:15.291 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:22:15.291 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:22:15.291 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:22:15.291 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:22:15.291 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:15.292 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:15.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:15.292 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:22:15.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:15.292 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:22:15.292 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:15.292 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:15.292 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:15.292 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:15.292 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:22:15.292 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:15.292 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:15.293 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:15.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:15.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:15.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:15.294 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:22:15.294 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:15.294 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:15.294 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:15.294 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:15.294 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:15.294 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:15.294 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:15.294 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:15.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:15.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:15.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:15.295 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:22:15.295 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:15.295 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:15.295 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:15.295 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:15.295 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:15.295 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:15.295 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:15.295 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:15.296 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:15.296 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:15.296 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:22:15.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:15.296 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:22:15.296 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:22:15.296 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:22:15.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:15.296 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:22:15Z" 2024-04-19T23:22:15.296 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:22:15.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:22:20.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:20.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:20.298 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:20.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:20.397+0000 7f2f7f4f0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:20.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:20.399+0000 7f2f7f4f0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:20.408 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:20.409 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:20.410 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:20.411 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:20.412 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:20.413 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:20.413 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:20.413 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:22:20.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:20.413 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:22:20.413 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:22:20.413 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:22:20.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:20.413 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:22:20Z" 2024-04-19T23:22:20.413 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:22:20.414 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:22:25.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:25.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:25.415 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:25.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:25.516+0000 7fbe8d86a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:25.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:25.517+0000 7fbe8d86a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:25.528 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:22:25.528 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:22:25.528 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:22:25.528 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:22:25.529 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:25.530 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:25.531 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:25.532 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:22:25.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:25.533 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:22:25.533 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:22:25.533 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:22:25.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:25.533 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:22:25Z" 2024-04-19T23:22:25.533 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:22:25.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:22:30.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:30.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:30.535 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:30.594 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:30.595 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:30.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:30.632+0000 7f148cdd2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:30.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:30.633+0000 7f148cdd2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:30.643 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:22:30.643 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:30.644 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:30.645 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:30.646 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:30.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:30.648 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:22:30.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:30.648 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:22:30.648 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:22:30.648 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:22:30.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:30.648 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:22:30Z" 2024-04-19T23:22:30.648 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:22:30.649 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1713568837.840174_14.1 target= 2024-04-19T23:22:35.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:35.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:35.650 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:35.709 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:35.709 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:35.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:35.747+0000 7f54f3966780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:35.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:35.748+0000 7f54f3966780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:35.758 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:35.759 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:35.760 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:35.761 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:35.763 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:35.763 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:22:35.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:35.763 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:22:35.763 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:22:35.763 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:22:35.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:35.763 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:22:35Z" 2024-04-19T23:22:35.763 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:22:35.764 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:22:35.765 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:22:35.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:35.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:35.765 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:35.824 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:35.825 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:35.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:22:35.864+0000 7f3a8dfc3780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:35.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:22:35.865+0000 7f3a8dfc3780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:35.874 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:35.875 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:35.876 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:35.877 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:22:35.878 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:35.879 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:35.879 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:35.879 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:22:35.879 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:35.879 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:22:35.879 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:22:35.879 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:22:35.879 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:35.879 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:22:35Z" 2024-04-19T23:22:35.879 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:22:35.880 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:22:35.880 INFO:rgw_multi.tests:meta checkpoint 2024-04-19T23:22:35.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:35.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:35.881 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:35.934 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:35.934 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:35.973 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:22:35.987 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:22:35.987 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:35.987 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:35.987 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:22:35.987 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:35.987 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:35.987 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:35.988 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:22:35.988 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:35.988 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:35.988 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:35.988 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:22:35.988 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:35.988 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:35.988 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:35.988 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:22:35.988 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:35.988 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:22:35.989 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:22:35.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:35.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:35.990 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:36.049 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:36.049 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:36.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:36.088+0000 7f93f712c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:36.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:36.089+0000 7f93f712c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:36.100 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:22:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:22:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:22:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:22:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:22:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:36.100 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:36.101 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:36.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:36.104 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:22:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:22:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:22:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:36.105 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:22:36Z" 2024-04-19T23:22:36.105 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:22:36.106 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:22:36.106 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:22:36.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:36.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:36.106 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:36.166 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:36.166 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:36.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:22:36.205+0000 7f7c93535780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:36.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:22:36.206+0000 7f7c93535780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:36.215 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:22:36.215 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:22:36.215 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:22:36.216 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:22:36.216 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:22:36.216 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:36.216 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:36.216 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:36.216 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:22:36.216 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:36.216 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:22:36.216 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:36.216 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:36.216 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:36.216 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:36.217 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:36.218 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:36.219 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:36.220 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:36.220 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:22:36.220 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:36.220 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:22:36.220 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:22:36.220 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:22:36.220 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:36.220 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:22:36Z" 2024-04-19T23:22:36.220 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:22:36.221 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:22:36.221 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-04-19T23:22:36.286 DEBUG:boto:Using access key provided by client. 2024-04-19T23:22:36.286 DEBUG:boto:Using secret key provided by client. 2024-04-19T23:22:36.286 DEBUG:boto:Using access key provided by client. 2024-04-19T23:22:36.287 DEBUG:boto:Using secret key provided by client. 2024-04-19T23:22:36.287 DEBUG:boto:Using access key provided by client. 2024-04-19T23:22:36.287 DEBUG:boto:Using secret key provided by client. 2024-04-19T23:22:36.288 DEBUG:boto:Using access key provided by client. 2024-04-19T23:22:36.288 DEBUG:boto:Using secret key provided by client. 2024-04-19T23:22:36.288 DEBUG:boto:Using access key provided by client. 2024-04-19T23:22:36.288 DEBUG:boto:Using secret key provided by client. 2024-04-19T23:22:36.289 DEBUG:boto:Using access key provided by client. 2024-04-19T23:22:36.289 DEBUG:boto:Using secret key provided by client. 2024-04-19T23:22:36.289 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-1 2024-04-19T23:22:36.290 DEBUG:boto:path=/aktlji-1/ 2024-04-19T23:22:36.290 DEBUG:boto:auth_path=/aktlji-1/ 2024-04-19T23:22:36.290 DEBUG:boto:Method: PUT 2024-04-19T23:22:36.290 DEBUG:boto:Path: /aktlji-1/ 2024-04-19T23:22:36.290 DEBUG:boto:Data: 2024-04-19T23:22:36.290 DEBUG:boto:Headers: {} 2024-04-19T23:22:36.290 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:22:36.290 DEBUG:boto:Port: 8000 2024-04-19T23:22:36.290 DEBUG:boto:Params: {} 2024-04-19T23:22:36.290 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:22:36.291 DEBUG:boto:Token: None 2024-04-19T23:22:36.291 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:22:36 GMT /aktlji-1/ 2024-04-19T23:22:36.291 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:G0vcmyAR5pQ2GlzCSPDngLMtUXU= 2024-04-19T23:22:36.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:G0vcmyAR5pQ2GlzCSPDngLMtUXU=', 'Content-Length': '0'} 2024-04-19T23:22:38.431 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d0bf5d244408843-006622fcbc-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:38.432 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-2 2024-04-19T23:22:38.432 DEBUG:boto:path=/aktlji-2/ 2024-04-19T23:22:38.432 DEBUG:boto:auth_path=/aktlji-2/ 2024-04-19T23:22:38.432 DEBUG:boto:Method: PUT 2024-04-19T23:22:38.432 DEBUG:boto:Path: /aktlji-2/ 2024-04-19T23:22:38.432 DEBUG:boto:Data: 2024-04-19T23:22:38.432 DEBUG:boto:Headers: {} 2024-04-19T23:22:38.432 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:22:38.433 DEBUG:boto:Port: 8000 2024-04-19T23:22:38.433 DEBUG:boto:Params: {} 2024-04-19T23:22:38.433 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:22:38.433 DEBUG:boto:Token: None 2024-04-19T23:22:38.433 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:22:38 GMT /aktlji-2/ 2024-04-19T23:22:38.433 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:791kMCO7j/9BZpmGIwGSygUZgOc= 2024-04-19T23:22:38.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:791kMCO7j/9BZpmGIwGSygUZgOc=', 'Content-Length': '0'} 2024-04-19T23:22:40.607 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003377165e50df8ae7-006622fcbe-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:40.607 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-3 2024-04-19T23:22:40.607 DEBUG:boto:path=/aktlji-3/ 2024-04-19T23:22:40.608 DEBUG:boto:auth_path=/aktlji-3/ 2024-04-19T23:22:40.608 DEBUG:boto:Method: PUT 2024-04-19T23:22:40.608 DEBUG:boto:Path: /aktlji-3/ 2024-04-19T23:22:40.608 DEBUG:boto:Data: 2024-04-19T23:22:40.608 DEBUG:boto:Headers: {} 2024-04-19T23:22:40.608 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:22:40.608 DEBUG:boto:Port: 8001 2024-04-19T23:22:40.608 DEBUG:boto:Params: {} 2024-04-19T23:22:40.608 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-19T23:22:40.609 DEBUG:boto:Token: None 2024-04-19T23:22:40.609 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:22:40 GMT /aktlji-3/ 2024-04-19T23:22:40.609 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2M8jK4KFHBUmeCdluoSWL+1+VA= 2024-04-19T23:22:40.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2M8jK4KFHBUmeCdluoSWL+1+VA=', 'Content-Length': '0'} 2024-04-19T23:22:40.649 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bf3fb831e3afa5d0-006622fcc0-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:40.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:40.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:40.650 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:40.707 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:40.707 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:40.746 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:22:40.760 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:22:40.760 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:40.761 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:40.761 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:22:40.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:40.761 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:40.761 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:40.762 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:22:40.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:40.762 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:40.762 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:40.763 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:22:40.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:40.763 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:40.763 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568960.637890_44.1", 2024-04-19T23:22:40.763 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:22:40.637890Z" 2024-04-19T23:22:40.764 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:40.764 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:22:40.765 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568960.637890_44.1'} 2024-04-19T23:22:40.766 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:22:40.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:40.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:40.767 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:40.835 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:40.835 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:40.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:40.880+0000 7ff86c3d6780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:40.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:40.881+0000 7ff86c3d6780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:22:40.891 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:22:40.891 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:22:40.891 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:22:40.891 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:22:40.891 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:40.891 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:40.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:40.892 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:22:40.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:40.892 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:22:40.892 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:40.892 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:40.892 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:40.892 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:40.892 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:22:40.892 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:40.892 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:22:40.892 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:40.893 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:40.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:40.893 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:40.893 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:22:40.893 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:40.893 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:40.893 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:40.893 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:40.893 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:40.893 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:40.894 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:40.894 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:40.894 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:40.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:40.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:40.894 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:22:40.894 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:40.894 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:40.894 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:40.894 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:40.895 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:40.895 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:40.895 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:40.895 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:40.895 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:40.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:40.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:40.895 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:22:40.895 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:40.895 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:40.895 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568960.637890_44.1", 2024-04-19T23:22:40.896 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:40.896 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:40.896 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:40.896 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:22:40.637890Z", 2024-04-19T23:22:40.896 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:40.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:40.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:40.896 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:22:40.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:40.896 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:22:40.897 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:22:40.897 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:22:40.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:40.897 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:22:40Z" 2024-04-19T23:22:40.897 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:22:40.898 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568960.637890_44.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:22:40.637890Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:22:40Z" } 2024-04-19T23:22:40.898 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568960.637890_44.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:22:40.637890Z', 'realm_epoch': 2}}] 2024-04-19T23:22:40.898 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568960.637890_44.1'} 2024-04-19T23:22:40.898 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568960.637890_44.1'} 2024-04-19T23:22:40.898 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:22:40.899 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:22:40.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:40.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:40.899 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:40.957 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:40.958 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:40.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:22:40.994+0000 7f6cbbfcc780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:40.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:22:40.995+0000 7f6cbbfcc780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:41.005 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:22:41.005 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:22:41.005 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:22:41.005 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:22:41.005 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:22:41.006 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:41.006 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:41.006 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:41.006 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:22:41.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:41.006 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:22:41.006 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:41.006 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:41.006 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:41.007 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:41.007 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:22:41.007 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:41.007 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:22:41.007 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:41.007 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:41.007 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:41.007 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:41.007 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:22:41.007 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:41.008 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:41.008 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:41.008 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:41.008 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:41.008 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:41.008 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:22:41.008 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:41.008 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:41.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:41.008 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:41.008 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:22:41.009 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:41.009 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:41.009 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:41.009 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:41.009 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:41.009 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:41.009 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:22:41.009 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:41.009 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:41.009 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:41.010 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:41.010 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:22:41.010 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:41.010 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:41.010 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568958.412954_33.1", 2024-04-19T23:22:41.010 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:41.010 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:41.010 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:41.010 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:22:38.412954Z", 2024-04-19T23:22:41.010 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:41.010 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:41.011 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:41.011 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:22:41.011 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:41.011 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:22:41.011 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:22:41.011 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:22:41.011 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:41.011 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:22:40Z" 2024-04-19T23:22:41.011 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:22:41.012 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568958.412954_33.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:22:38.412954Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:22:40Z" } 2024-04-19T23:22:41.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568958.412954_33.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:22:38.412954Z', 'realm_epoch': 2}}] 2024-04-19T23:22:41.012 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568960.637890_44.1'} 2024-04-19T23:22:41.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568958.412954_33.1'} 2024-04-19T23:22:41.013 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1713568960.637890_44.1 target=1_1713568958.412954_33.1 2024-04-19T23:22:46.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:46.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:46.014 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:46.073 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:46.073 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:46.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:22:46.110+0000 7f4d01d1e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:46.108 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:22:46.111+0000 7f4d01d1e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:46.121 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:22:46.121 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:22:46.122 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:22:46.122 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:22:46.122 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:22:46.122 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:46.122 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:46.122 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:46.122 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:22:46.122 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:46.122 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:22:46.122 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:46.122 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:46.123 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:46.123 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:46.123 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:22:46.123 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:46.123 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:22:46.123 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:46.123 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:46.123 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:46.123 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:46.123 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:22:46.124 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:46.124 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:46.124 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:46.124 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:46.124 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:46.124 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:46.124 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:22:46.124 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:46.124 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:46.124 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:46.124 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:46.125 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:22:46.125 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:46.125 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:46.125 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:46.125 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:46.125 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:46.125 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:46.125 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:22:46.125 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:46.125 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:46.126 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:46.126 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:46.126 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:22:46.126 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:46.126 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:46.126 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568960.637890_44.1", 2024-04-19T23:22:46.126 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:46.126 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:46.126 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:46.126 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:22:40.637890Z", 2024-04-19T23:22:46.126 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:46.127 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:46.127 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:46.127 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:22:46.127 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:46.127 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:22:46.127 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:22:46.127 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:22:46.127 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:46.127 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:22:46Z" 2024-04-19T23:22:46.127 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:22:46.129 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568960.637890_44.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:22:40.637890Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:22:46Z" } 2024-04-19T23:22:46.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568960.637890_44.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:22:40.637890Z', 'realm_epoch': 2}}] 2024-04-19T23:22:46.129 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568960.637890_44.1'} 2024-04-19T23:22:46.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568960.637890_44.1'} 2024-04-19T23:22:46.129 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:22:46.129 DEBUG:boto:path=/aktlji-1/ 2024-04-19T23:22:46.130 DEBUG:boto:auth_path=/aktlji-1/ 2024-04-19T23:22:46.130 DEBUG:boto:Method: HEAD 2024-04-19T23:22:46.130 DEBUG:boto:Path: /aktlji-1/ 2024-04-19T23:22:46.130 DEBUG:boto:Data: 2024-04-19T23:22:46.130 DEBUG:boto:Headers: {} 2024-04-19T23:22:46.130 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:22:46.130 DEBUG:boto:Port: 8000 2024-04-19T23:22:46.130 DEBUG:boto:Params: {} 2024-04-19T23:22:46.130 DEBUG:boto:Token: None 2024-04-19T23:22:46.131 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:22:46 GMT /aktlji-1/ 2024-04-19T23:22:46.132 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3t4M6z8Mn/1h+PLHZ1lEoZyw0fk= 2024-04-19T23:22:46.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3t4M6z8Mn/1h+PLHZ1lEoZyw0fk=', 'Content-Length': '0'} 2024-04-19T23:22:46.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd790e8a77728732-006622fcc6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:46.134 DEBUG:boto:path=/aktlji-2/ 2024-04-19T23:22:46.134 DEBUG:boto:auth_path=/aktlji-2/ 2024-04-19T23:22:46.135 DEBUG:boto:Method: HEAD 2024-04-19T23:22:46.135 DEBUG:boto:Path: /aktlji-2/ 2024-04-19T23:22:46.135 DEBUG:boto:Data: 2024-04-19T23:22:46.135 DEBUG:boto:Headers: {} 2024-04-19T23:22:46.135 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:22:46.135 DEBUG:boto:Port: 8000 2024-04-19T23:22:46.135 DEBUG:boto:Params: {} 2024-04-19T23:22:46.135 DEBUG:boto:Token: None 2024-04-19T23:22:46.135 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:22:46 GMT /aktlji-2/ 2024-04-19T23:22:46.136 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MHSuQWkBxecCwW0um0IgZe5LW28= 2024-04-19T23:22:46.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MHSuQWkBxecCwW0um0IgZe5LW28=', 'Content-Length': '0'} 2024-04-19T23:22:46.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da5ec2709ce77825-006622fcc6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:46.138 DEBUG:boto:path=/aktlji-3/ 2024-04-19T23:22:46.138 DEBUG:boto:auth_path=/aktlji-3/ 2024-04-19T23:22:46.139 DEBUG:boto:Method: HEAD 2024-04-19T23:22:46.139 DEBUG:boto:Path: /aktlji-3/ 2024-04-19T23:22:46.139 DEBUG:boto:Data: 2024-04-19T23:22:46.139 DEBUG:boto:Headers: {} 2024-04-19T23:22:46.139 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:22:46.139 DEBUG:boto:Port: 8000 2024-04-19T23:22:46.139 DEBUG:boto:Params: {} 2024-04-19T23:22:46.139 DEBUG:boto:Token: None 2024-04-19T23:22:46.139 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:22:46 GMT /aktlji-3/ 2024-04-19T23:22:46.139 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VWRnQbmsQUw+aaJ/XCBI0Wj1D2s= 2024-04-19T23:22:46.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VWRnQbmsQUw+aaJ/XCBI0Wj1D2s=', 'Content-Length': '0'} 2024-04-19T23:22:46.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b4de895028a3a32-006622fcc6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:46.142 DEBUG:boto:path=/aktlji-1/ 2024-04-19T23:22:46.142 DEBUG:boto:auth_path=/aktlji-1/ 2024-04-19T23:22:46.142 DEBUG:boto:Method: HEAD 2024-04-19T23:22:46.142 DEBUG:boto:Path: /aktlji-1/ 2024-04-19T23:22:46.142 DEBUG:boto:Data: 2024-04-19T23:22:46.142 DEBUG:boto:Headers: {} 2024-04-19T23:22:46.142 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:22:46.142 DEBUG:boto:Port: 8000 2024-04-19T23:22:46.142 DEBUG:boto:Params: {} 2024-04-19T23:22:46.143 DEBUG:boto:Token: None 2024-04-19T23:22:46.143 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:22:46 GMT /aktlji-1/ 2024-04-19T23:22:46.143 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3t4M6z8Mn/1h+PLHZ1lEoZyw0fk= 2024-04-19T23:22:46.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3t4M6z8Mn/1h+PLHZ1lEoZyw0fk=', 'Content-Length': '0'} 2024-04-19T23:22:46.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a3e074fb3ae24d2-006622fcc6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:46.146 DEBUG:boto:path=/aktlji-2/ 2024-04-19T23:22:46.146 DEBUG:boto:auth_path=/aktlji-2/ 2024-04-19T23:22:46.146 DEBUG:boto:Method: HEAD 2024-04-19T23:22:46.146 DEBUG:boto:Path: /aktlji-2/ 2024-04-19T23:22:46.146 DEBUG:boto:Data: 2024-04-19T23:22:46.146 DEBUG:boto:Headers: {} 2024-04-19T23:22:46.146 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:22:46.146 DEBUG:boto:Port: 8000 2024-04-19T23:22:46.146 DEBUG:boto:Params: {} 2024-04-19T23:22:46.147 DEBUG:boto:Token: None 2024-04-19T23:22:46.147 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:22:46 GMT /aktlji-2/ 2024-04-19T23:22:46.147 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MHSuQWkBxecCwW0um0IgZe5LW28= 2024-04-19T23:22:46.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MHSuQWkBxecCwW0um0IgZe5LW28=', 'Content-Length': '0'} 2024-04-19T23:22:46.149 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad5c5a70defcb6ad-006622fcc6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:46.149 DEBUG:boto:path=/aktlji-3/ 2024-04-19T23:22:46.149 DEBUG:boto:auth_path=/aktlji-3/ 2024-04-19T23:22:46.149 DEBUG:boto:Method: HEAD 2024-04-19T23:22:46.150 DEBUG:boto:Path: /aktlji-3/ 2024-04-19T23:22:46.150 DEBUG:boto:Data: 2024-04-19T23:22:46.150 DEBUG:boto:Headers: {} 2024-04-19T23:22:46.150 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:22:46.150 DEBUG:boto:Port: 8000 2024-04-19T23:22:46.150 DEBUG:boto:Params: {} 2024-04-19T23:22:46.150 DEBUG:boto:Token: None 2024-04-19T23:22:46.150 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:22:46 GMT /aktlji-3/ 2024-04-19T23:22:46.150 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VWRnQbmsQUw+aaJ/XCBI0Wj1D2s= 2024-04-19T23:22:46.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VWRnQbmsQUw+aaJ/XCBI0Wj1D2s=', 'Content-Length': '0'} 2024-04-19T23:22:46.153 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f59e619d0bd9c63-006622fcc6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:46.153 DEBUG:boto:path=/aktlji-1/ 2024-04-19T23:22:46.153 DEBUG:boto:auth_path=/aktlji-1/ 2024-04-19T23:22:46.153 DEBUG:boto:Method: HEAD 2024-04-19T23:22:46.153 DEBUG:boto:Path: /aktlji-1/ 2024-04-19T23:22:46.153 DEBUG:boto:Data: 2024-04-19T23:22:46.153 DEBUG:boto:Headers: {} 2024-04-19T23:22:46.153 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:22:46.153 DEBUG:boto:Port: 8001 2024-04-19T23:22:46.154 DEBUG:boto:Params: {} 2024-04-19T23:22:46.154 DEBUG:boto:Token: None 2024-04-19T23:22:46.154 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:22:46 GMT /aktlji-1/ 2024-04-19T23:22:46.154 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3t4M6z8Mn/1h+PLHZ1lEoZyw0fk= 2024-04-19T23:22:46.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3t4M6z8Mn/1h+PLHZ1lEoZyw0fk=', 'Content-Length': '0'} 2024-04-19T23:22:46.156 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053e2542b907ff3b6-006622fcc6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:46.156 DEBUG:boto:path=/aktlji-2/ 2024-04-19T23:22:46.156 DEBUG:boto:auth_path=/aktlji-2/ 2024-04-19T23:22:46.156 DEBUG:boto:Method: HEAD 2024-04-19T23:22:46.157 DEBUG:boto:Path: /aktlji-2/ 2024-04-19T23:22:46.157 DEBUG:boto:Data: 2024-04-19T23:22:46.157 DEBUG:boto:Headers: {} 2024-04-19T23:22:46.157 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:22:46.157 DEBUG:boto:Port: 8001 2024-04-19T23:22:46.157 DEBUG:boto:Params: {} 2024-04-19T23:22:46.157 DEBUG:boto:Token: None 2024-04-19T23:22:46.157 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:22:46 GMT /aktlji-2/ 2024-04-19T23:22:46.157 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MHSuQWkBxecCwW0um0IgZe5LW28= 2024-04-19T23:22:46.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MHSuQWkBxecCwW0um0IgZe5LW28=', 'Content-Length': '0'} 2024-04-19T23:22:46.160 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093838a5873d3a9ef-006622fcc6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:46.160 DEBUG:boto:path=/aktlji-3/ 2024-04-19T23:22:46.160 DEBUG:boto:auth_path=/aktlji-3/ 2024-04-19T23:22:46.160 DEBUG:boto:Method: HEAD 2024-04-19T23:22:46.160 DEBUG:boto:Path: /aktlji-3/ 2024-04-19T23:22:46.160 DEBUG:boto:Data: 2024-04-19T23:22:46.160 DEBUG:boto:Headers: {} 2024-04-19T23:22:46.161 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:22:46.161 DEBUG:boto:Port: 8001 2024-04-19T23:22:46.161 DEBUG:boto:Params: {} 2024-04-19T23:22:46.161 DEBUG:boto:Token: None 2024-04-19T23:22:46.161 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:22:46 GMT /aktlji-3/ 2024-04-19T23:22:46.161 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VWRnQbmsQUw+aaJ/XCBI0Wj1D2s= 2024-04-19T23:22:46.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VWRnQbmsQUw+aaJ/XCBI0Wj1D2s=', 'Content-Length': '0'} 2024-04-19T23:22:46.164 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee1d7c0e6b050e12-006622fcc6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:46.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-04-19T23:22:46.165 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-4 2024-04-19T23:22:46.165 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:22:46.165 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:22:46.165 DEBUG:boto:Method: PUT 2024-04-19T23:22:46.165 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:22:46.165 DEBUG:boto:Data: 2024-04-19T23:22:46.165 DEBUG:boto:Headers: {} 2024-04-19T23:22:46.165 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:22:46.165 DEBUG:boto:Port: 8000 2024-04-19T23:22:46.166 DEBUG:boto:Params: {} 2024-04-19T23:22:46.166 DEBUG:boto:Token: None 2024-04-19T23:22:46.166 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:22:46 GMT /aktlji-4/ 2024-04-19T23:22:46.166 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:96aLJZsfebQ5yU4AemKcO87qYNQ= 2024-04-19T23:22:46.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:96aLJZsfebQ5yU4AemKcO87qYNQ=', 'Content-Length': '0'} 2024-04-19T23:22:46.183 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000532b46152f605cf9-006622fcc6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:46.184 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-5 2024-04-19T23:22:46.184 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:22:46.184 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:22:46.184 DEBUG:boto:Method: PUT 2024-04-19T23:22:46.184 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:22:46.184 DEBUG:boto:Data: 2024-04-19T23:22:46.184 DEBUG:boto:Headers: {} 2024-04-19T23:22:46.184 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:22:46.184 DEBUG:boto:Port: 8000 2024-04-19T23:22:46.184 DEBUG:boto:Params: {} 2024-04-19T23:22:46.185 DEBUG:boto:Token: None 2024-04-19T23:22:46.185 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:22:46 GMT /aktlji-5/ 2024-04-19T23:22:46.185 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PRWB9yHrQhvIW9v6hgEjvQn57Y8= 2024-04-19T23:22:46.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PRWB9yHrQhvIW9v6hgEjvQn57Y8=', 'Content-Length': '0'} 2024-04-19T23:22:46.212 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000005379f3d7890c747-006622fcc6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:46.213 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-6 2024-04-19T23:22:46.213 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:22:46.213 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:22:46.213 DEBUG:boto:Method: PUT 2024-04-19T23:22:46.213 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:22:46.213 DEBUG:boto:Data: 2024-04-19T23:22:46.213 DEBUG:boto:Headers: {} 2024-04-19T23:22:46.213 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:22:46.213 DEBUG:boto:Port: 8001 2024-04-19T23:22:46.213 DEBUG:boto:Params: {} 2024-04-19T23:22:46.214 DEBUG:boto:Token: None 2024-04-19T23:22:46.214 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:22:46 GMT /aktlji-6/ 2024-04-19T23:22:46.214 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:n8B3CdCqvW+bfPZFYd1M5hanzxc= 2024-04-19T23:22:46.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:22:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:n8B3CdCqvW+bfPZFYd1M5hanzxc=', 'Content-Length': '0'} 2024-04-19T23:22:46.247 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000b95e78d8ed2a4ef-006622fcc6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:22:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:22:46.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:46.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:46.248 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:46.304 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:46.304 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:46.352 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:22:46.368 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:22:46.368 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:46.368 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:46.368 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:22:46.368 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568966.236233_56.1", 2024-04-19T23:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:22:46.236233Z" 2024-04-19T23:22:46.370 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:46.370 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:22:46.371 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:22:46.371 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:22:46.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:46.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:46.371 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:46.438 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:46.438 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:46.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:46.477+0000 7f2765924780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:46.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:46.478+0000 7f2765924780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:46.489 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:22:46.489 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:22:46.489 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:22:46.489 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:22:46.489 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:22:46.489 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:46.489 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:46.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:46.490 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:22:46.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:46.490 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:22:46.490 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:46.490 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:46.490 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:46.490 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:46.490 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:22:46.490 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:46.490 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:22:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:22:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:46.492 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:46.492 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:46.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:46.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:46.492 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:22:46.492 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:46.492 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:46.492 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:46.492 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:46.492 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:46.492 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:46.493 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:46.493 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:46.493 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:46.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:46.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:46.493 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:22:46.493 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:46.493 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:46.493 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568966.232259_54.1", 2024-04-19T23:22:46.493 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:46.493 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:46.494 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:46.494 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:22:46.232259Z", 2024-04-19T23:22:46.494 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:46.494 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:46.494 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:46.494 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:22:46.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:46.494 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:22:46.494 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:22:46.494 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:22:46.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:46.495 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:22:46Z" 2024-04-19T23:22:46.495 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:22:46.496 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568966.232259_54.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:22:46.232259Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:22:46Z" } 2024-04-19T23:22:46.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568966.232259_54.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:22:46.232259Z', 'realm_epoch': 2}}] 2024-04-19T23:22:46.496 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:22:46.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.232259_54.1'} 2024-04-19T23:22:46.496 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713568966.236233_56.1 target=1_1713568966.232259_54.1 2024-04-19T23:22:51.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:51.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:51.498 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:51.558 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:51.558 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:51.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:51.596+0000 7fcd64400780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:51.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:51.597+0000 7fcd64400780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:51.607 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:22:51.607 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:22:51.607 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:22:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:22:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:22:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:22:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:22:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:51.609 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:51.609 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:51.609 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:22:51.609 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:51.609 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:22:51.609 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:51.609 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:51.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:51.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:51.609 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:22:51.610 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:51.610 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:51.610 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:51.610 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:51.610 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:51.610 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:51.610 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:51.610 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:51.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:51.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:51.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:51.611 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:22:51.611 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:51.611 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:51.611 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:51.611 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:51.611 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:51.612 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:51.612 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:51.612 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:51.612 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:51.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:51.612 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:51.612 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:22:51.612 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:51.612 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:51.613 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568966.232259_54.1", 2024-04-19T23:22:51.613 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:51.613 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:51.613 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:51.613 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:22:46.232259Z", 2024-04-19T23:22:51.613 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:51.613 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:51.613 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:51.613 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:22:51.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:51.614 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:22:51.614 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:22:51.614 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:22:51.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:51.614 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:22:51Z" 2024-04-19T23:22:51.614 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:22:51.615 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568966.232259_54.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:22:46.232259Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:22:51Z" } 2024-04-19T23:22:51.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568966.232259_54.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:22:46.232259Z', 'realm_epoch': 2}}] 2024-04-19T23:22:51.616 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:22:51.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.232259_54.1'} 2024-04-19T23:22:51.616 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713568966.236233_56.1 target=1_1713568966.232259_54.1 2024-04-19T23:22:56.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:56.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:56.618 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:56.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:56.716+0000 7f5bd5fca780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:56.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:22:56.717+0000 7f5bd5fca780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:56.728 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:22:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:22:56.728 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:22:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:22:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:22:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:22:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:22:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:56.729 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:22:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:22:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:22:56.730 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:56.731 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:22:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:22:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:22:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:22:56.732 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:22:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:22:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:22:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568966.236233_56.1", 2024-04-19T23:22:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:22:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:22:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:22:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:22:46.236233Z", 2024-04-19T23:22:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:22:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:22:56.733 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:22:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:22:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:22:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:22:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:22:56.734 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:22:56Z" 2024-04-19T23:22:56.734 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:22:56.735 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568966.236233_56.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:22:46.236233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:22:56Z" } 2024-04-19T23:22:56.735 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568966.236233_56.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:22:46.236233Z', 'realm_epoch': 2}}] 2024-04-19T23:22:56.735 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:22:56.735 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:22:56.736 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:22:56.736 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:22:56.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:56.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:22:56.736 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:22:56.796 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:22:56.796 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:22:56.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:22:56.835+0000 7f214782b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:56.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:22:56.836+0000 7f214782b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:22:56.847 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:22:56.847 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:22:56.847 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:22:56.847 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:22:56.847 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:22:56.847 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:22:56.847 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:56.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:56.848 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:22:56.848 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:56.848 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:22:56.848 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:56.848 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:56.848 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:22:56.848 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:56.848 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:22:56.848 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:56.849 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:22:56.849 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:56.849 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:56.849 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:56.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:56.849 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:22:56.849 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:56.849 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:56.849 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:56.849 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:56.849 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:56.849 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:56.850 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:22:56.850 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:56.850 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:56.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:56.850 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:56.850 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:22:56.850 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:56.850 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:56.850 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:22:56.850 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:56.850 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:56.851 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:56.851 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:22:56.851 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:56.851 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:56.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:56.851 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:22:56.851 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:22:56.851 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:22:56.851 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:22:56.852 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568966.232259_54.1", 2024-04-19T23:22:56.852 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:22:56.852 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:22:56.852 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:22:56.852 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:22:46.232259Z", 2024-04-19T23:22:56.852 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:22:56.852 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:56.852 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:22:56.852 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:22:56.852 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:56.852 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:22:56.853 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:22:56.853 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:22:56.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:22:56.853 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:22:56Z" 2024-04-19T23:22:56.853 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:22:56.854 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568966.232259_54.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:22:46.232259Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:22:56Z" } 2024-04-19T23:22:56.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568966.232259_54.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:22:46.232259Z', 'realm_epoch': 2}}] 2024-04-19T23:22:56.854 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:22:56.854 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.232259_54.1'} 2024-04-19T23:22:56.854 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1713568966.236233_56.1 target=1_1713568966.232259_54.1 2024-04-19T23:23:01.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:01.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:01.856 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:01.915 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:01.915 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:01.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:01.953+0000 7f17a1324780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:01.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:01.954+0000 7f17a1324780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:01.965 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:23:01.966 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:23:01.966 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:23:01.966 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:23:01.966 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:23:01.966 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:23:01.966 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:01.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:01.966 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:23:01.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:01.966 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:23:01.966 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:01.967 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:01.967 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:01.967 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:01.967 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:23:01.967 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:01.967 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:23:01.967 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:01.967 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:01.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:01.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:01.967 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:23:01.968 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:01.968 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:01.968 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:01.968 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:01.968 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:01.968 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:01.968 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:23:01.968 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:01.968 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:01.968 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:01.968 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:01.969 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:23:01.969 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:01.969 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:01.969 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:01.969 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:01.969 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:01.969 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:01.969 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:23:01.969 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:01.969 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:01.969 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:01.970 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:01.970 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:23:01.970 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:01.970 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:01.970 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568966.236233_56.1", 2024-04-19T23:23:01.970 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:01.970 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:01.970 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:01.970 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:22:46.236233Z", 2024-04-19T23:23:01.970 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:01.971 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:01.971 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:01.971 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:23:01.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:01.971 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:23:01.971 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:23:01.971 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:23:01.971 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:01.971 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:23:01Z" 2024-04-19T23:23:01.971 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:23:01.972 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568966.236233_56.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:22:46.236233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:23:01Z" } 2024-04-19T23:23:01.973 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568966.236233_56.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:22:46.236233Z', 'realm_epoch': 2}}] 2024-04-19T23:23:01.973 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:23:01.973 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:23:01.973 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:23:01.973 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:23:01.973 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:23:01.973 DEBUG:boto:Method: HEAD 2024-04-19T23:23:01.974 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:23:01.974 DEBUG:boto:Data: 2024-04-19T23:23:01.974 DEBUG:boto:Headers: {} 2024-04-19T23:23:01.974 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:01.974 DEBUG:boto:Port: 8000 2024-04-19T23:23:01.974 DEBUG:boto:Params: {} 2024-04-19T23:23:01.974 DEBUG:boto:Token: None 2024-04-19T23:23:01.974 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:01 GMT /aktlji-4/ 2024-04-19T23:23:01.975 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ha+oqNhcO2sqAKcYy47w1R3sSBk= 2024-04-19T23:23:01.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ha+oqNhcO2sqAKcYy47w1R3sSBk=', 'Content-Length': '0'} 2024-04-19T23:23:01.977 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8590c9f5b360ee3-006622fcd5-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:01.977 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:23:01.977 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:23:01.977 DEBUG:boto:Method: HEAD 2024-04-19T23:23:01.977 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:23:01.978 DEBUG:boto:Data: 2024-04-19T23:23:01.978 DEBUG:boto:Headers: {} 2024-04-19T23:23:01.978 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:01.978 DEBUG:boto:Port: 8000 2024-04-19T23:23:01.978 DEBUG:boto:Params: {} 2024-04-19T23:23:01.978 DEBUG:boto:Token: None 2024-04-19T23:23:01.978 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:01 GMT /aktlji-5/ 2024-04-19T23:23:01.978 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ntTj2/3RSJUBngUFYFEYFbKJ4cQ= 2024-04-19T23:23:01.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ntTj2/3RSJUBngUFYFEYFbKJ4cQ=', 'Content-Length': '0'} 2024-04-19T23:23:01.981 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b13b52539649499e-006622fcd5-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:01.981 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:23:01.981 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:23:01.981 DEBUG:boto:Method: HEAD 2024-04-19T23:23:01.981 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:23:01.981 DEBUG:boto:Data: 2024-04-19T23:23:01.981 DEBUG:boto:Headers: {} 2024-04-19T23:23:01.981 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:01.981 DEBUG:boto:Port: 8000 2024-04-19T23:23:01.981 DEBUG:boto:Params: {} 2024-04-19T23:23:01.982 DEBUG:boto:Token: None 2024-04-19T23:23:01.982 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:01 GMT /aktlji-6/ 2024-04-19T23:23:01.982 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gfw9nHSditRZhlcu5FEgse2aRcM= 2024-04-19T23:23:01.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gfw9nHSditRZhlcu5FEgse2aRcM=', 'Content-Length': '0'} 2024-04-19T23:23:01.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000096b3555154834e1-006622fcd5-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:01.984 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:23:01.984 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:23:01.984 DEBUG:boto:Method: HEAD 2024-04-19T23:23:01.984 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:23:01.985 DEBUG:boto:Data: 2024-04-19T23:23:01.985 DEBUG:boto:Headers: {} 2024-04-19T23:23:01.985 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:01.985 DEBUG:boto:Port: 8000 2024-04-19T23:23:01.985 DEBUG:boto:Params: {} 2024-04-19T23:23:01.985 DEBUG:boto:Token: None 2024-04-19T23:23:01.985 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:01 GMT /aktlji-4/ 2024-04-19T23:23:01.985 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ha+oqNhcO2sqAKcYy47w1R3sSBk= 2024-04-19T23:23:01.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ha+oqNhcO2sqAKcYy47w1R3sSBk=', 'Content-Length': '0'} 2024-04-19T23:23:01.988 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060ee8d0597203fc2-006622fcd5-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:01.988 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:23:01.988 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:23:01.988 DEBUG:boto:Method: HEAD 2024-04-19T23:23:01.988 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:23:01.988 DEBUG:boto:Data: 2024-04-19T23:23:01.988 DEBUG:boto:Headers: {} 2024-04-19T23:23:01.989 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:01.989 DEBUG:boto:Port: 8000 2024-04-19T23:23:01.989 DEBUG:boto:Params: {} 2024-04-19T23:23:01.989 DEBUG:boto:Token: None 2024-04-19T23:23:01.989 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:01 GMT /aktlji-5/ 2024-04-19T23:23:01.989 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ntTj2/3RSJUBngUFYFEYFbKJ4cQ= 2024-04-19T23:23:01.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ntTj2/3RSJUBngUFYFEYFbKJ4cQ=', 'Content-Length': '0'} 2024-04-19T23:23:01.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052ca54d977e92ee3-006622fcd5-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:01.991 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:23:01.991 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:23:01.992 DEBUG:boto:Method: HEAD 2024-04-19T23:23:01.992 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:23:01.992 DEBUG:boto:Data: 2024-04-19T23:23:01.992 DEBUG:boto:Headers: {} 2024-04-19T23:23:01.992 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:01.992 DEBUG:boto:Port: 8000 2024-04-19T23:23:01.992 DEBUG:boto:Params: {} 2024-04-19T23:23:01.992 DEBUG:boto:Token: None 2024-04-19T23:23:01.992 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:01 GMT /aktlji-6/ 2024-04-19T23:23:01.992 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gfw9nHSditRZhlcu5FEgse2aRcM= 2024-04-19T23:23:01.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gfw9nHSditRZhlcu5FEgse2aRcM=', 'Content-Length': '0'} 2024-04-19T23:23:01.995 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bab77a19dc87663a-006622fcd5-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:01.995 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:23:01.995 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:23:01.995 DEBUG:boto:Method: HEAD 2024-04-19T23:23:01.995 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:23:01.996 DEBUG:boto:Data: 2024-04-19T23:23:01.996 DEBUG:boto:Headers: {} 2024-04-19T23:23:01.996 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:01.996 DEBUG:boto:Port: 8001 2024-04-19T23:23:01.996 DEBUG:boto:Params: {} 2024-04-19T23:23:01.996 DEBUG:boto:Token: None 2024-04-19T23:23:01.996 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:01 GMT /aktlji-4/ 2024-04-19T23:23:01.996 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ha+oqNhcO2sqAKcYy47w1R3sSBk= 2024-04-19T23:23:01.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ha+oqNhcO2sqAKcYy47w1R3sSBk=', 'Content-Length': '0'} 2024-04-19T23:23:01.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000497dbf8d175fd7bc-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:01.999 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:23:01.999 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:23:01.999 DEBUG:boto:Method: HEAD 2024-04-19T23:23:01.999 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:23:01.999 DEBUG:boto:Data: 2024-04-19T23:23:01.999 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.000 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:02.000 DEBUG:boto:Port: 8001 2024-04-19T23:23:02.000 DEBUG:boto:Params: {} 2024-04-19T23:23:02.000 DEBUG:boto:Token: None 2024-04-19T23:23:02.000 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-5/ 2024-04-19T23:23:02.000 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4= 2024-04-19T23:23:02.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4=', 'Content-Length': '0'} 2024-04-19T23:23:02.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e483d0f582147c0-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.002 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:23:02.002 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:23:02.002 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.002 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:23:02.002 DEBUG:boto:Data: 2024-04-19T23:23:02.003 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.003 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:02.003 DEBUG:boto:Port: 8001 2024-04-19T23:23:02.003 DEBUG:boto:Params: {} 2024-04-19T23:23:02.003 DEBUG:boto:Token: None 2024-04-19T23:23:02.003 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-6/ 2024-04-19T23:23:02.003 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM= 2024-04-19T23:23:02.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM=', 'Content-Length': '0'} 2024-04-19T23:23:02.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003808da503a47098b-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.005 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:23:02.005 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:23:02.006 DEBUG:boto:Method: PUT 2024-04-19T23:23:02.006 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:23:02.006 DEBUG:boto:Data: 2024-04-19T23:23:02.006 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.006 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:02.006 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.006 DEBUG:boto:Params: {} 2024-04-19T23:23:02.006 DEBUG:boto:Token: None 2024-04-19T23:23:02.006 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:02 GMT /aktlji-4/ 2024-04-19T23:23:02.006 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ez6E4uif63s7HAPxgKWYyUEHWmc= 2024-04-19T23:23:02.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ez6E4uif63s7HAPxgKWYyUEHWmc=', 'Content-Length': '0'} 2024-04-19T23:23:02.009 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bdfb4ac701ea4295-006622fcd6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.010 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:23:02.010 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:23:02.010 DEBUG:boto:Method: PUT 2024-04-19T23:23:02.010 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:23:02.010 DEBUG:boto:Data: 2024-04-19T23:23:02.010 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.010 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:02.010 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.010 DEBUG:boto:Params: {} 2024-04-19T23:23:02.011 DEBUG:boto:Token: None 2024-04-19T23:23:02.011 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:02 GMT /aktlji-5/ 2024-04-19T23:23:02.011 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/jzi0ORwdXfqBzEMZfVicvDCzqs= 2024-04-19T23:23:02.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/jzi0ORwdXfqBzEMZfVicvDCzqs=', 'Content-Length': '0'} 2024-04-19T23:23:02.013 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c6c9eefa4bac5ea8-006622fcd6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.014 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:23:02.014 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:23:02.014 DEBUG:boto:Method: PUT 2024-04-19T23:23:02.014 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:23:02.014 DEBUG:boto:Data: 2024-04-19T23:23:02.014 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.014 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:02.014 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.014 DEBUG:boto:Params: {} 2024-04-19T23:23:02.015 DEBUG:boto:Token: None 2024-04-19T23:23:02.015 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:02 GMT /aktlji-6/ 2024-04-19T23:23:02.015 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:twSTx4T0gH7wTqk46/a+igP0j+I= 2024-04-19T23:23:02.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:twSTx4T0gH7wTqk46/a+igP0j+I=', 'Content-Length': '0'} 2024-04-19T23:23:02.017 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000062cb546a9afc1dc-006622fcd6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.017 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:23:02.018 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:23:02.018 DEBUG:boto:Method: PUT 2024-04-19T23:23:02.018 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:23:02.018 DEBUG:boto:Data: 2024-04-19T23:23:02.018 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.018 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:02.018 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.018 DEBUG:boto:Params: {} 2024-04-19T23:23:02.018 DEBUG:boto:Token: None 2024-04-19T23:23:02.019 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:02 GMT /aktlji-4/ 2024-04-19T23:23:02.019 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ez6E4uif63s7HAPxgKWYyUEHWmc= 2024-04-19T23:23:02.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ez6E4uif63s7HAPxgKWYyUEHWmc=', 'Content-Length': '0'} 2024-04-19T23:23:02.025 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cdd1e832688259b4-006622fcd6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.025 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:23:02.025 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:23:02.025 DEBUG:boto:Method: PUT 2024-04-19T23:23:02.025 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:23:02.025 DEBUG:boto:Data: 2024-04-19T23:23:02.025 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.025 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:02.025 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.026 DEBUG:boto:Params: {} 2024-04-19T23:23:02.026 DEBUG:boto:Token: None 2024-04-19T23:23:02.026 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:02 GMT /aktlji-5/ 2024-04-19T23:23:02.026 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/jzi0ORwdXfqBzEMZfVicvDCzqs= 2024-04-19T23:23:02.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/jzi0ORwdXfqBzEMZfVicvDCzqs=', 'Content-Length': '0'} 2024-04-19T23:23:02.072 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002ff466076cdc0927-006622fcd6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.073 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:23:02.073 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:23:02.073 DEBUG:boto:Method: PUT 2024-04-19T23:23:02.073 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:23:02.073 DEBUG:boto:Data: 2024-04-19T23:23:02.073 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.073 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:02.073 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.073 DEBUG:boto:Params: {} 2024-04-19T23:23:02.073 DEBUG:boto:Token: None 2024-04-19T23:23:02.074 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:02 GMT /aktlji-6/ 2024-04-19T23:23:02.074 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:twSTx4T0gH7wTqk46/a+igP0j+I= 2024-04-19T23:23:02.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:twSTx4T0gH7wTqk46/a+igP0j+I=', 'Content-Length': '0'} 2024-04-19T23:23:02.120 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000702f14b185348b1f-006622fcd6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.120 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:23:02.121 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:23:02.121 DEBUG:boto:Method: PUT 2024-04-19T23:23:02.121 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:23:02.121 DEBUG:boto:Data: 2024-04-19T23:23:02.121 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.121 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:02.121 DEBUG:boto:Port: 8001 2024-04-19T23:23:02.121 DEBUG:boto:Params: {} 2024-04-19T23:23:02.121 DEBUG:boto:Token: None 2024-04-19T23:23:02.121 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:02 GMT /aktlji-4/ 2024-04-19T23:23:02.122 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ez6E4uif63s7HAPxgKWYyUEHWmc= 2024-04-19T23:23:02.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ez6E4uif63s7HAPxgKWYyUEHWmc=', 'Content-Length': '0'} 2024-04-19T23:23:02.128 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000840b48caf1853eaa-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.128 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:23:02.128 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:23:02.129 DEBUG:boto:Method: PUT 2024-04-19T23:23:02.129 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:23:02.129 DEBUG:boto:Data: 2024-04-19T23:23:02.129 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.129 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:02.129 DEBUG:boto:Port: 8001 2024-04-19T23:23:02.129 DEBUG:boto:Params: {} 2024-04-19T23:23:02.129 DEBUG:boto:Token: None 2024-04-19T23:23:02.129 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:02 GMT /aktlji-5/ 2024-04-19T23:23:02.129 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/jzi0ORwdXfqBzEMZfVicvDCzqs= 2024-04-19T23:23:02.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/jzi0ORwdXfqBzEMZfVicvDCzqs=', 'Content-Length': '0'} 2024-04-19T23:23:02.176 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000089b002545e1442b3-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.176 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:23:02.176 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:23:02.176 DEBUG:boto:Method: PUT 2024-04-19T23:23:02.176 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:23:02.176 DEBUG:boto:Data: 2024-04-19T23:23:02.176 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.176 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:02.177 DEBUG:boto:Port: 8001 2024-04-19T23:23:02.177 DEBUG:boto:Params: {} 2024-04-19T23:23:02.177 DEBUG:boto:Token: None 2024-04-19T23:23:02.177 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:02 GMT /aktlji-6/ 2024-04-19T23:23:02.177 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:twSTx4T0gH7wTqk46/a+igP0j+I= 2024-04-19T23:23:02.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:twSTx4T0gH7wTqk46/a+igP0j+I=', 'Content-Length': '0'} 2024-04-19T23:23:02.224 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c02abc4e6ab9ae25-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.224 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:23:02.224 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:23:02.224 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.224 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:23:02.224 DEBUG:boto:Data: 2024-04-19T23:23:02.224 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.224 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:02.225 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.225 DEBUG:boto:Params: {} 2024-04-19T23:23:02.225 DEBUG:boto:Token: None 2024-04-19T23:23:02.225 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-4/ 2024-04-19T23:23:02.225 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sx95RGLaHq548eN6lCq7GC7SDBQ= 2024-04-19T23:23:02.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sx95RGLaHq548eN6lCq7GC7SDBQ=', 'Content-Length': '0'} 2024-04-19T23:23:02.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8a4103533bb8f15-006622fcd6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.228 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:23:02.228 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:23:02.228 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.228 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:23:02.228 DEBUG:boto:Data: 2024-04-19T23:23:02.228 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.228 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:02.228 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.229 DEBUG:boto:Params: {} 2024-04-19T23:23:02.229 DEBUG:boto:Token: None 2024-04-19T23:23:02.229 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-5/ 2024-04-19T23:23:02.229 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4= 2024-04-19T23:23:02.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4=', 'Content-Length': '0'} 2024-04-19T23:23:02.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1b5eea8b7ba9dcd-006622fcd6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.232 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:23:02.232 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:23:02.232 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.232 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:23:02.232 DEBUG:boto:Data: 2024-04-19T23:23:02.232 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.232 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:02.232 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.232 DEBUG:boto:Params: {} 2024-04-19T23:23:02.233 DEBUG:boto:Token: None 2024-04-19T23:23:02.233 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-6/ 2024-04-19T23:23:02.233 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM= 2024-04-19T23:23:02.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM=', 'Content-Length': '0'} 2024-04-19T23:23:02.235 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b14aa336713f800-006622fcd6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.236 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:23:02.236 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:23:02.236 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.236 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:23:02.236 DEBUG:boto:Data: 2024-04-19T23:23:02.236 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.236 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:02.236 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.237 DEBUG:boto:Params: {} 2024-04-19T23:23:02.237 DEBUG:boto:Token: None 2024-04-19T23:23:02.237 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-4/ 2024-04-19T23:23:02.237 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sx95RGLaHq548eN6lCq7GC7SDBQ= 2024-04-19T23:23:02.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sx95RGLaHq548eN6lCq7GC7SDBQ=', 'Content-Length': '0'} 2024-04-19T23:23:02.239 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055ec9117e4a297ab-006622fcd6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.239 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:23:02.239 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:23:02.240 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.240 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:23:02.240 DEBUG:boto:Data: 2024-04-19T23:23:02.240 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.240 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:02.240 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.240 DEBUG:boto:Params: {} 2024-04-19T23:23:02.240 DEBUG:boto:Token: None 2024-04-19T23:23:02.240 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-5/ 2024-04-19T23:23:02.241 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4= 2024-04-19T23:23:02.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4=', 'Content-Length': '0'} 2024-04-19T23:23:02.243 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014bea4868d4cb94e-006622fcd6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.243 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:23:02.243 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:23:02.243 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.243 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:23:02.243 DEBUG:boto:Data: 2024-04-19T23:23:02.243 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.244 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:02.244 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.244 DEBUG:boto:Params: {} 2024-04-19T23:23:02.244 DEBUG:boto:Token: None 2024-04-19T23:23:02.244 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-6/ 2024-04-19T23:23:02.244 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM= 2024-04-19T23:23:02.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM=', 'Content-Length': '0'} 2024-04-19T23:23:02.246 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b20a58d4c2909f1-006622fcd6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.247 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:23:02.247 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:23:02.247 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.247 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:23:02.247 DEBUG:boto:Data: 2024-04-19T23:23:02.247 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.247 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:02.247 DEBUG:boto:Port: 8001 2024-04-19T23:23:02.247 DEBUG:boto:Params: {} 2024-04-19T23:23:02.248 DEBUG:boto:Token: None 2024-04-19T23:23:02.248 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-4/ 2024-04-19T23:23:02.248 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sx95RGLaHq548eN6lCq7GC7SDBQ= 2024-04-19T23:23:02.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sx95RGLaHq548eN6lCq7GC7SDBQ=', 'Content-Length': '0'} 2024-04-19T23:23:02.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045d9b8481a3d6b0e-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.251 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:23:02.251 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:23:02.251 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.251 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:23:02.251 DEBUG:boto:Data: 2024-04-19T23:23:02.251 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.251 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:02.251 DEBUG:boto:Port: 8001 2024-04-19T23:23:02.251 DEBUG:boto:Params: {} 2024-04-19T23:23:02.251 DEBUG:boto:Token: None 2024-04-19T23:23:02.252 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-5/ 2024-04-19T23:23:02.252 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4= 2024-04-19T23:23:02.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4=', 'Content-Length': '0'} 2024-04-19T23:23:02.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6789a9235f91254-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.254 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:23:02.254 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:23:02.254 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.255 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:23:02.255 DEBUG:boto:Data: 2024-04-19T23:23:02.255 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.255 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:02.255 DEBUG:boto:Port: 8001 2024-04-19T23:23:02.255 DEBUG:boto:Params: {} 2024-04-19T23:23:02.255 DEBUG:boto:Token: None 2024-04-19T23:23:02.255 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-6/ 2024-04-19T23:23:02.255 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM= 2024-04-19T23:23:02.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM=', 'Content-Length': '0'} 2024-04-19T23:23:02.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f6e511aa965bbb3-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:02.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:02.259 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:02.320 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:02.320 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:02.366 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:23:02.381 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:23:02.381 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:02.381 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:02.382 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:23:02.382 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:02.384 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:02.384 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:02.384 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:23:02.384 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:02.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:02.385 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:02.385 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:23:02.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:02.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:02.385 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568966.236233_56.1", 2024-04-19T23:23:02.385 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:22:46.236233Z" 2024-04-19T23:23:02.385 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:02.385 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:23:02.386 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:23:02.386 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:23:02.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:02.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:02.387 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:02.447 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:02.447 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:02.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:02.486+0000 7f8944e6d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:02.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:02.487+0000 7f8944e6d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:02.497 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:23:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:23:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:23:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:23:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:23:02.497 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:23:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:23:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:23:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:23:02.498 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:02.499 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:23:02.499 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:02.499 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:02.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:02.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:02.499 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:23:02.499 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:02.499 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:02.499 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:23:02.499 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:02.499 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:02.500 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:02.500 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:23:02.500 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:02.500 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:02.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:02.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:02.500 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:23:02.500 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:02.500 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:02.500 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:23:02.501 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:02.501 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:02.501 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:02.501 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:23:02.501 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:02.501 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:02.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:02.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:02.501 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:23:02.501 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:02.501 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:02.502 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568966.236233_56.1", 2024-04-19T23:23:02.502 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:02.502 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:02.502 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:02.502 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:22:46.236233Z", 2024-04-19T23:23:02.502 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:02.502 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:02.502 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:02.502 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:23:02.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:02.503 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:23:02.503 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:23:02.503 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:23:02.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:02.503 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:23:02Z" 2024-04-19T23:23:02.503 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:23:02.504 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568966.236233_56.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:22:46.236233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:23:02Z" } 2024-04-19T23:23:02.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568966.236233_56.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:22:46.236233Z', 'realm_epoch': 2}}] 2024-04-19T23:23:02.504 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:23:02.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:23:02.505 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:23:02.505 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:23:02.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:02.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:02.505 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:02.565 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:02.565 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:02.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:02.604+0000 7feef9aab780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:02.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:02.606+0000 7feef9aab780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:02.616 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:23:02.616 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:23:02.616 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:23:02.616 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:23:02.616 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:23:02.616 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:23:02.616 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:02.616 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:02.616 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:23:02.616 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:02.617 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:23:02.617 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:02.617 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:02.617 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:02.617 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:02.617 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:23:02.617 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:02.617 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:23:02.617 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:02.617 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:02.617 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:02.618 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:02.618 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:23:02.618 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:02.618 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:02.618 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:02.618 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:02.618 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:02.618 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:02.618 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:23:02.618 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:02.618 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:02.619 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:02.619 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:02.619 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:23:02.619 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:02.619 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:02.619 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:02.619 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:02.619 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:02.619 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:02.619 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:23:02.619 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:02.620 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:02.620 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:02.620 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:02.620 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:23:02.620 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:02.620 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:02.620 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568966.236233_56.1", 2024-04-19T23:23:02.620 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:02.620 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:02.620 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:02.620 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:22:46.236233Z", 2024-04-19T23:23:02.621 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:02.621 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:02.621 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:02.621 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:23:02.621 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:02.621 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:23:02.621 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:23:02.621 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:23:02.621 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:02.621 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:23:02Z" 2024-04-19T23:23:02.621 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:23:02.622 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568966.236233_56.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:22:46.236233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:23:02Z" } 2024-04-19T23:23:02.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568966.236233_56.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:22:46.236233Z', 'realm_epoch': 2}}] 2024-04-19T23:23:02.623 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:23:02.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568966.236233_56.1'} 2024-04-19T23:23:02.623 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:23:02.623 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:23:02.623 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:23:02.623 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.623 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:23:02.623 DEBUG:boto:Data: 2024-04-19T23:23:02.624 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.624 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:02.624 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.624 DEBUG:boto:Params: {} 2024-04-19T23:23:02.624 DEBUG:boto:Token: None 2024-04-19T23:23:02.624 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-4/ 2024-04-19T23:23:02.624 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sx95RGLaHq548eN6lCq7GC7SDBQ= 2024-04-19T23:23:02.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sx95RGLaHq548eN6lCq7GC7SDBQ=', 'Content-Length': '0'} 2024-04-19T23:23:02.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9efc966536224fb-006622fcd6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.627 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:23:02.627 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:23:02.627 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.627 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:23:02.627 DEBUG:boto:Data: 2024-04-19T23:23:02.627 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.627 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:02.627 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.627 DEBUG:boto:Params: {} 2024-04-19T23:23:02.628 DEBUG:boto:Token: None 2024-04-19T23:23:02.628 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-5/ 2024-04-19T23:23:02.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4= 2024-04-19T23:23:02.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4=', 'Content-Length': '0'} 2024-04-19T23:23:02.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000758ef1fe2128ae3f-006622fcd6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.630 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:23:02.630 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:23:02.630 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.630 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:23:02.630 DEBUG:boto:Data: 2024-04-19T23:23:02.631 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.631 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:02.631 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.631 DEBUG:boto:Params: {} 2024-04-19T23:23:02.631 DEBUG:boto:Token: None 2024-04-19T23:23:02.631 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-6/ 2024-04-19T23:23:02.631 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM= 2024-04-19T23:23:02.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM=', 'Content-Length': '0'} 2024-04-19T23:23:02.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008de801fcbe172596-006622fcd6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.634 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:23:02.634 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:23:02.634 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.634 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:23:02.634 DEBUG:boto:Data: 2024-04-19T23:23:02.634 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.634 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:02.634 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.634 DEBUG:boto:Params: {} 2024-04-19T23:23:02.634 DEBUG:boto:Token: None 2024-04-19T23:23:02.635 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-4/ 2024-04-19T23:23:02.635 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sx95RGLaHq548eN6lCq7GC7SDBQ= 2024-04-19T23:23:02.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sx95RGLaHq548eN6lCq7GC7SDBQ=', 'Content-Length': '0'} 2024-04-19T23:23:02.637 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a644707d157b829-006622fcd6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.637 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:23:02.638 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:23:02.638 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.638 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:23:02.638 DEBUG:boto:Data: 2024-04-19T23:23:02.638 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.638 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:02.638 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.638 DEBUG:boto:Params: {} 2024-04-19T23:23:02.639 DEBUG:boto:Token: None 2024-04-19T23:23:02.639 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-5/ 2024-04-19T23:23:02.639 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4= 2024-04-19T23:23:02.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4=', 'Content-Length': '0'} 2024-04-19T23:23:02.641 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000719c88f1f15528bb-006622fcd6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.642 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:23:02.642 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:23:02.642 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.642 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:23:02.642 DEBUG:boto:Data: 2024-04-19T23:23:02.642 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.642 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:02.642 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.642 DEBUG:boto:Params: {} 2024-04-19T23:23:02.643 DEBUG:boto:Token: None 2024-04-19T23:23:02.643 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-6/ 2024-04-19T23:23:02.643 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM= 2024-04-19T23:23:02.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM=', 'Content-Length': '0'} 2024-04-19T23:23:02.645 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b40b25aa3f81e8a7-006622fcd6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.645 DEBUG:boto:path=/aktlji-4/ 2024-04-19T23:23:02.645 DEBUG:boto:auth_path=/aktlji-4/ 2024-04-19T23:23:02.645 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.646 DEBUG:boto:Path: /aktlji-4/ 2024-04-19T23:23:02.646 DEBUG:boto:Data: 2024-04-19T23:23:02.646 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.646 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:02.646 DEBUG:boto:Port: 8001 2024-04-19T23:23:02.646 DEBUG:boto:Params: {} 2024-04-19T23:23:02.646 DEBUG:boto:Token: None 2024-04-19T23:23:02.646 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-4/ 2024-04-19T23:23:02.646 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sx95RGLaHq548eN6lCq7GC7SDBQ= 2024-04-19T23:23:02.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sx95RGLaHq548eN6lCq7GC7SDBQ=', 'Content-Length': '0'} 2024-04-19T23:23:02.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b24ad4fa54a083fb-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.649 DEBUG:boto:path=/aktlji-5/ 2024-04-19T23:23:02.649 DEBUG:boto:auth_path=/aktlji-5/ 2024-04-19T23:23:02.649 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.649 DEBUG:boto:Path: /aktlji-5/ 2024-04-19T23:23:02.649 DEBUG:boto:Data: 2024-04-19T23:23:02.649 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.649 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:02.649 DEBUG:boto:Port: 8001 2024-04-19T23:23:02.649 DEBUG:boto:Params: {} 2024-04-19T23:23:02.649 DEBUG:boto:Token: None 2024-04-19T23:23:02.650 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-5/ 2024-04-19T23:23:02.650 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4= 2024-04-19T23:23:02.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:78t3yecgAiouKhu3xWQVZcKMam4=', 'Content-Length': '0'} 2024-04-19T23:23:02.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ae976e6df5649ae-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.652 DEBUG:boto:path=/aktlji-6/ 2024-04-19T23:23:02.652 DEBUG:boto:auth_path=/aktlji-6/ 2024-04-19T23:23:02.652 DEBUG:boto:Method: HEAD 2024-04-19T23:23:02.652 DEBUG:boto:Path: /aktlji-6/ 2024-04-19T23:23:02.652 DEBUG:boto:Data: 2024-04-19T23:23:02.653 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.653 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:02.653 DEBUG:boto:Port: 8001 2024-04-19T23:23:02.653 DEBUG:boto:Params: {} 2024-04-19T23:23:02.653 DEBUG:boto:Token: None 2024-04-19T23:23:02.653 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:02 GMT /aktlji-6/ 2024-04-19T23:23:02.653 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM= 2024-04-19T23:23:02.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:91G10sv/FoMokD2R/Kj1byRoEvM=', 'Content-Length': '0'} 2024-04-19T23:23:02.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060729fcf08eaca02-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-04-19T23:23:02.656 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-7 2024-04-19T23:23:02.656 DEBUG:boto:path=/aktlji-7/ 2024-04-19T23:23:02.656 DEBUG:boto:auth_path=/aktlji-7/ 2024-04-19T23:23:02.656 DEBUG:boto:Method: PUT 2024-04-19T23:23:02.656 DEBUG:boto:Path: /aktlji-7/ 2024-04-19T23:23:02.657 DEBUG:boto:Data: 2024-04-19T23:23:02.657 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.657 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:02.657 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.657 DEBUG:boto:Params: {} 2024-04-19T23:23:02.657 DEBUG:boto:Token: None 2024-04-19T23:23:02.657 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:02 GMT /aktlji-7/ 2024-04-19T23:23:02.657 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QCzVzIAu/0r4nfbpstk25Wok1uc= 2024-04-19T23:23:02.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QCzVzIAu/0r4nfbpstk25Wok1uc=', 'Content-Length': '0'} 2024-04-19T23:23:02.672 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000026edfccb09279ee8-006622fcd6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.672 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-8 2024-04-19T23:23:02.672 DEBUG:boto:path=/aktlji-8/ 2024-04-19T23:23:02.672 DEBUG:boto:auth_path=/aktlji-8/ 2024-04-19T23:23:02.673 DEBUG:boto:Method: PUT 2024-04-19T23:23:02.673 DEBUG:boto:Path: /aktlji-8/ 2024-04-19T23:23:02.673 DEBUG:boto:Data: 2024-04-19T23:23:02.673 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.673 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:02.673 DEBUG:boto:Port: 8000 2024-04-19T23:23:02.673 DEBUG:boto:Params: {} 2024-04-19T23:23:02.673 DEBUG:boto:Token: None 2024-04-19T23:23:02.673 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:02 GMT /aktlji-8/ 2024-04-19T23:23:02.674 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Tlx/WQbktYUHErykzbzQkikHQ/A= 2024-04-19T23:23:02.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Tlx/WQbktYUHErykzbzQkikHQ/A=', 'Content-Length': '0'} 2024-04-19T23:23:02.702 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fe4a73809aa43d43-006622fcd6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.703 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-9 2024-04-19T23:23:02.703 DEBUG:boto:path=/aktlji-9/ 2024-04-19T23:23:02.703 DEBUG:boto:auth_path=/aktlji-9/ 2024-04-19T23:23:02.703 DEBUG:boto:Method: PUT 2024-04-19T23:23:02.703 DEBUG:boto:Path: /aktlji-9/ 2024-04-19T23:23:02.703 DEBUG:boto:Data: 2024-04-19T23:23:02.704 DEBUG:boto:Headers: {} 2024-04-19T23:23:02.704 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:02.704 DEBUG:boto:Port: 8001 2024-04-19T23:23:02.704 DEBUG:boto:Params: {} 2024-04-19T23:23:02.704 DEBUG:boto:Token: None 2024-04-19T23:23:02.704 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:02 GMT /aktlji-9/ 2024-04-19T23:23:02.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SCwkJ/gbzNQ/KUKUeBbe33J4eaw= 2024-04-19T23:23:02.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SCwkJ/gbzNQ/KUKUeBbe33J4eaw=', 'Content-Length': '0'} 2024-04-19T23:23:02.733 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a59a7bbdebfcb5fa-006622fcd6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:02.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:02.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:02.734 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:02.793 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:02.794 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:02.838 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:23:02.852 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:23:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:23:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:23:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:23:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568982.724459_68.1", 2024-04-19T23:23:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:02.724459Z" 2024-04-19T23:23:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:02.854 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:23:02.855 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568982.724459_68.1'} 2024-04-19T23:23:02.855 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:23:02.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:02.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:02.856 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:02.915 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:02.915 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:02.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:02.955+0000 7f4d05ce3780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:02.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:02.956+0000 7f4d05ce3780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:02.966 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:23:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:23:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:23:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:23:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:23:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:23:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:23:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:23:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:23:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:23:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:23:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:23:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:23:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:23:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:23:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:23:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:23:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568982.720319_66.1", 2024-04-19T23:23:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:02.720319Z", 2024-04-19T23:23:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:23:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:23:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:23:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:23:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:23:02Z" 2024-04-19T23:23:02.972 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:23:02.973 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568982.720319_66.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:02.720319Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:23:02Z" } 2024-04-19T23:23:02.974 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568982.720319_66.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:02.720319Z', 'realm_epoch': 2}}] 2024-04-19T23:23:02.974 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568982.724459_68.1'} 2024-04-19T23:23:02.974 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568982.720319_66.1'} 2024-04-19T23:23:02.974 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713568982.724459_68.1 target=1_1713568982.720319_66.1 2024-04-19T23:23:07.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:07.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:07.976 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:08.035 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:08.035 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:08.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:08.074+0000 7f10c94a0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:08.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:08.075+0000 7f10c94a0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:08.085 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:23:08.085 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:23:08.085 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:23:08.085 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:23:08.085 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:23:08.085 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:23:08.086 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:08.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:08.086 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:23:08.086 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:08.086 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:23:08.086 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:08.086 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:08.086 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:08.086 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:08.086 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:23:08.087 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:08.087 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:23:08.087 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:08.087 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:08.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:08.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:08.087 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:23:08.087 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:08.087 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:08.087 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:23:08.087 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:08.088 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:08.088 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:08.088 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:23:08.088 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:08.088 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:08.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:08.088 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:08.088 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:23:08.088 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:08.088 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:08.088 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:23:08.088 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:08.089 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:08.089 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:08.089 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:23:08.089 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:08.089 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:08.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:08.089 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:08.089 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:23:08.089 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:08.089 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:08.089 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568982.724459_68.1", 2024-04-19T23:23:08.090 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:08.090 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:08.090 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:08.090 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:02.724459Z", 2024-04-19T23:23:08.090 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:08.090 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:08.090 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:08.090 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:23:08.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:08.090 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:23:08.091 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:23:08.091 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:23:08.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:08.091 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:23:08Z" 2024-04-19T23:23:08.091 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:23:08.092 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568982.724459_68.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:02.724459Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:23:08Z" } 2024-04-19T23:23:08.092 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568982.724459_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:02.724459Z', 'realm_epoch': 2}}] 2024-04-19T23:23:08.092 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568982.724459_68.1'} 2024-04-19T23:23:08.092 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568982.724459_68.1'} 2024-04-19T23:23:08.092 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:23:08.093 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:23:08.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:08.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:08.093 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:08.151 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:08.152 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:08.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:08.190+0000 7f05dd0cc780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:08.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:08.191+0000 7f05dd0cc780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:08.201 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:23:08.201 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:23:08.201 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:23:08.201 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:23:08.201 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:23:08.201 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:23:08.201 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:08.201 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:08.202 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:23:08.202 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:08.202 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:23:08.202 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:08.202 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:08.202 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:08.202 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:08.202 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:23:08.202 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:08.202 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:23:08.203 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:08.203 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:08.203 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:08.203 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:08.203 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:23:08.203 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:08.203 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:08.203 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:08.203 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:08.203 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:08.203 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:08.203 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:23:08.204 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:08.204 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:08.204 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:08.204 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:08.204 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:23:08.204 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:08.204 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:08.204 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:08.204 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:08.205 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:08.205 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:08.205 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:23:08.205 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:08.205 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:08.205 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:08.205 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:08.205 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:23:08.205 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:08.205 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:08.205 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568982.724459_68.1", 2024-04-19T23:23:08.206 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:08.206 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:08.206 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:08.206 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:02.724459Z", 2024-04-19T23:23:08.206 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:08.206 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:08.206 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:08.206 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:23:08.206 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:08.206 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:23:08.206 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:23:08.207 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:23:08.207 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:08.207 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:23:08Z" 2024-04-19T23:23:08.207 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:23:08.208 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568982.724459_68.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:02.724459Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:23:08Z" } 2024-04-19T23:23:08.208 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568982.724459_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:02.724459Z', 'realm_epoch': 2}}] 2024-04-19T23:23:08.208 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568982.724459_68.1'} 2024-04-19T23:23:08.208 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568982.724459_68.1'} 2024-04-19T23:23:08.208 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:23:08.209 DEBUG:boto:path=/aktlji-7/ 2024-04-19T23:23:08.209 DEBUG:boto:auth_path=/aktlji-7/ 2024-04-19T23:23:08.209 DEBUG:boto:Method: HEAD 2024-04-19T23:23:08.209 DEBUG:boto:Path: /aktlji-7/ 2024-04-19T23:23:08.209 DEBUG:boto:Data: 2024-04-19T23:23:08.209 DEBUG:boto:Headers: {} 2024-04-19T23:23:08.209 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:08.209 DEBUG:boto:Port: 8000 2024-04-19T23:23:08.209 DEBUG:boto:Params: {} 2024-04-19T23:23:08.210 DEBUG:boto:Token: None 2024-04-19T23:23:08.210 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:08 GMT /aktlji-7/ 2024-04-19T23:23:08.210 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ltK4s/yG3Whh8HI2mbIPUKYn/9E= 2024-04-19T23:23:08.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ltK4s/yG3Whh8HI2mbIPUKYn/9E=', 'Content-Length': '0'} 2024-04-19T23:23:08.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007adfbc8c0229eabf-006622fcdc-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:08.212 DEBUG:boto:path=/aktlji-8/ 2024-04-19T23:23:08.213 DEBUG:boto:auth_path=/aktlji-8/ 2024-04-19T23:23:08.213 DEBUG:boto:Method: HEAD 2024-04-19T23:23:08.213 DEBUG:boto:Path: /aktlji-8/ 2024-04-19T23:23:08.213 DEBUG:boto:Data: 2024-04-19T23:23:08.213 DEBUG:boto:Headers: {} 2024-04-19T23:23:08.213 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:08.213 DEBUG:boto:Port: 8000 2024-04-19T23:23:08.213 DEBUG:boto:Params: {} 2024-04-19T23:23:08.213 DEBUG:boto:Token: None 2024-04-19T23:23:08.214 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:08 GMT /aktlji-8/ 2024-04-19T23:23:08.214 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EVYxzgcDzeh9r1k3McCzpo+GCWo= 2024-04-19T23:23:08.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EVYxzgcDzeh9r1k3McCzpo+GCWo=', 'Content-Length': '0'} 2024-04-19T23:23:08.216 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041312eec833bdf54-006622fcdc-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:08.216 DEBUG:boto:path=/aktlji-9/ 2024-04-19T23:23:08.216 DEBUG:boto:auth_path=/aktlji-9/ 2024-04-19T23:23:08.216 DEBUG:boto:Method: HEAD 2024-04-19T23:23:08.216 DEBUG:boto:Path: /aktlji-9/ 2024-04-19T23:23:08.216 DEBUG:boto:Data: 2024-04-19T23:23:08.217 DEBUG:boto:Headers: {} 2024-04-19T23:23:08.217 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:08.217 DEBUG:boto:Port: 8000 2024-04-19T23:23:08.217 DEBUG:boto:Params: {} 2024-04-19T23:23:08.217 DEBUG:boto:Token: None 2024-04-19T23:23:08.217 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:08 GMT /aktlji-9/ 2024-04-19T23:23:08.217 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MycqUNKyXkrPrwf9mCWA7ao3R+8= 2024-04-19T23:23:08.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MycqUNKyXkrPrwf9mCWA7ao3R+8=', 'Content-Length': '0'} 2024-04-19T23:23:08.220 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b24930997d095cef-006622fcdc-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:08.220 DEBUG:boto:path=/aktlji-7/ 2024-04-19T23:23:08.220 DEBUG:boto:auth_path=/aktlji-7/ 2024-04-19T23:23:08.220 DEBUG:boto:Method: HEAD 2024-04-19T23:23:08.220 DEBUG:boto:Path: /aktlji-7/ 2024-04-19T23:23:08.220 DEBUG:boto:Data: 2024-04-19T23:23:08.220 DEBUG:boto:Headers: {} 2024-04-19T23:23:08.220 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:08.220 DEBUG:boto:Port: 8000 2024-04-19T23:23:08.220 DEBUG:boto:Params: {} 2024-04-19T23:23:08.221 DEBUG:boto:Token: None 2024-04-19T23:23:08.221 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:08 GMT /aktlji-7/ 2024-04-19T23:23:08.221 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ltK4s/yG3Whh8HI2mbIPUKYn/9E= 2024-04-19T23:23:08.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ltK4s/yG3Whh8HI2mbIPUKYn/9E=', 'Content-Length': '0'} 2024-04-19T23:23:08.223 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026d0fd7b2a1fe1aa-006622fcdc-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:08.224 DEBUG:boto:path=/aktlji-8/ 2024-04-19T23:23:08.224 DEBUG:boto:auth_path=/aktlji-8/ 2024-04-19T23:23:08.224 DEBUG:boto:Method: HEAD 2024-04-19T23:23:08.224 DEBUG:boto:Path: /aktlji-8/ 2024-04-19T23:23:08.224 DEBUG:boto:Data: 2024-04-19T23:23:08.224 DEBUG:boto:Headers: {} 2024-04-19T23:23:08.224 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:08.224 DEBUG:boto:Port: 8000 2024-04-19T23:23:08.224 DEBUG:boto:Params: {} 2024-04-19T23:23:08.225 DEBUG:boto:Token: None 2024-04-19T23:23:08.225 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:08 GMT /aktlji-8/ 2024-04-19T23:23:08.225 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EVYxzgcDzeh9r1k3McCzpo+GCWo= 2024-04-19T23:23:08.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EVYxzgcDzeh9r1k3McCzpo+GCWo=', 'Content-Length': '0'} 2024-04-19T23:23:08.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000094cb673718a2313-006622fcdc-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:08.227 DEBUG:boto:path=/aktlji-9/ 2024-04-19T23:23:08.227 DEBUG:boto:auth_path=/aktlji-9/ 2024-04-19T23:23:08.227 DEBUG:boto:Method: HEAD 2024-04-19T23:23:08.228 DEBUG:boto:Path: /aktlji-9/ 2024-04-19T23:23:08.228 DEBUG:boto:Data: 2024-04-19T23:23:08.228 DEBUG:boto:Headers: {} 2024-04-19T23:23:08.228 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:08.228 DEBUG:boto:Port: 8000 2024-04-19T23:23:08.228 DEBUG:boto:Params: {} 2024-04-19T23:23:08.228 DEBUG:boto:Token: None 2024-04-19T23:23:08.229 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:08 GMT /aktlji-9/ 2024-04-19T23:23:08.229 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MycqUNKyXkrPrwf9mCWA7ao3R+8= 2024-04-19T23:23:08.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MycqUNKyXkrPrwf9mCWA7ao3R+8=', 'Content-Length': '0'} 2024-04-19T23:23:08.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000381f8667dae4ceed-006622fcdc-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:08.231 DEBUG:boto:path=/aktlji-7/ 2024-04-19T23:23:08.231 DEBUG:boto:auth_path=/aktlji-7/ 2024-04-19T23:23:08.231 DEBUG:boto:Method: HEAD 2024-04-19T23:23:08.231 DEBUG:boto:Path: /aktlji-7/ 2024-04-19T23:23:08.232 DEBUG:boto:Data: 2024-04-19T23:23:08.232 DEBUG:boto:Headers: {} 2024-04-19T23:23:08.232 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:08.232 DEBUG:boto:Port: 8001 2024-04-19T23:23:08.232 DEBUG:boto:Params: {} 2024-04-19T23:23:08.232 DEBUG:boto:Token: None 2024-04-19T23:23:08.232 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:08 GMT /aktlji-7/ 2024-04-19T23:23:08.232 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ltK4s/yG3Whh8HI2mbIPUKYn/9E= 2024-04-19T23:23:08.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ltK4s/yG3Whh8HI2mbIPUKYn/9E=', 'Content-Length': '0'} 2024-04-19T23:23:08.235 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020bcac17829a6a2b-006622fcdc-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:08.235 DEBUG:boto:path=/aktlji-8/ 2024-04-19T23:23:08.235 DEBUG:boto:auth_path=/aktlji-8/ 2024-04-19T23:23:08.235 DEBUG:boto:Method: HEAD 2024-04-19T23:23:08.235 DEBUG:boto:Path: /aktlji-8/ 2024-04-19T23:23:08.235 DEBUG:boto:Data: 2024-04-19T23:23:08.235 DEBUG:boto:Headers: {} 2024-04-19T23:23:08.235 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:08.235 DEBUG:boto:Port: 8001 2024-04-19T23:23:08.236 DEBUG:boto:Params: {} 2024-04-19T23:23:08.236 DEBUG:boto:Token: None 2024-04-19T23:23:08.236 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:08 GMT /aktlji-8/ 2024-04-19T23:23:08.236 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EVYxzgcDzeh9r1k3McCzpo+GCWo= 2024-04-19T23:23:08.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EVYxzgcDzeh9r1k3McCzpo+GCWo=', 'Content-Length': '0'} 2024-04-19T23:23:08.238 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d28926c30aa459c-006622fcdc-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:08.238 DEBUG:boto:path=/aktlji-9/ 2024-04-19T23:23:08.239 DEBUG:boto:auth_path=/aktlji-9/ 2024-04-19T23:23:08.239 DEBUG:boto:Method: HEAD 2024-04-19T23:23:08.239 DEBUG:boto:Path: /aktlji-9/ 2024-04-19T23:23:08.239 DEBUG:boto:Data: 2024-04-19T23:23:08.239 DEBUG:boto:Headers: {} 2024-04-19T23:23:08.239 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:08.239 DEBUG:boto:Port: 8001 2024-04-19T23:23:08.239 DEBUG:boto:Params: {} 2024-04-19T23:23:08.239 DEBUG:boto:Token: None 2024-04-19T23:23:08.239 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:08 GMT /aktlji-9/ 2024-04-19T23:23:08.240 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MycqUNKyXkrPrwf9mCWA7ao3R+8= 2024-04-19T23:23:08.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MycqUNKyXkrPrwf9mCWA7ao3R+8=', 'Content-Length': '0'} 2024-04-19T23:23:08.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7e8cd498a5b61b3-006622fcdc-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:08.242 DEBUG:boto:path=/aktlji-7/ 2024-04-19T23:23:08.242 DEBUG:boto:auth_path=/aktlji-7/ 2024-04-19T23:23:08.242 DEBUG:boto:Method: DELETE 2024-04-19T23:23:08.243 DEBUG:boto:Path: /aktlji-7/ 2024-04-19T23:23:08.243 DEBUG:boto:Data: 2024-04-19T23:23:08.243 DEBUG:boto:Headers: {} 2024-04-19T23:23:08.243 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:08.243 DEBUG:boto:Port: 8000 2024-04-19T23:23:08.243 DEBUG:boto:Params: {} 2024-04-19T23:23:08.243 DEBUG:boto:Token: None 2024-04-19T23:23:08.243 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:23:08 GMT /aktlji-7/ 2024-04-19T23:23:08.243 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dZ5t+3AO3fxVf8+ww12vnO1V46c= 2024-04-19T23:23:08.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dZ5t+3AO3fxVf8+ww12vnO1V46c=', 'Content-Length': '0'} 2024-04-19T23:23:08.275 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000044549c5e0c7a99db-006622fcdc-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:23:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:08.276 DEBUG:boto:path=/aktlji-8/ 2024-04-19T23:23:08.276 DEBUG:boto:auth_path=/aktlji-8/ 2024-04-19T23:23:08.276 DEBUG:boto:Method: DELETE 2024-04-19T23:23:08.276 DEBUG:boto:Path: /aktlji-8/ 2024-04-19T23:23:08.276 DEBUG:boto:Data: 2024-04-19T23:23:08.276 DEBUG:boto:Headers: {} 2024-04-19T23:23:08.277 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:08.277 DEBUG:boto:Port: 8000 2024-04-19T23:23:08.277 DEBUG:boto:Params: {} 2024-04-19T23:23:08.278 DEBUG:boto:Token: None 2024-04-19T23:23:08.278 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:23:08 GMT /aktlji-8/ 2024-04-19T23:23:08.278 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:prr7Hhp3Ic1ESQBziB04UHXCc68= 2024-04-19T23:23:08.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:prr7Hhp3Ic1ESQBziB04UHXCc68=', 'Content-Length': '0'} 2024-04-19T23:23:08.351 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009d1f6161f9e9b909-006622fcdc-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:23:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:08.351 DEBUG:boto:path=/aktlji-9/ 2024-04-19T23:23:08.351 DEBUG:boto:auth_path=/aktlji-9/ 2024-04-19T23:23:08.352 DEBUG:boto:Method: DELETE 2024-04-19T23:23:08.352 DEBUG:boto:Path: /aktlji-9/ 2024-04-19T23:23:08.352 DEBUG:boto:Data: 2024-04-19T23:23:08.352 DEBUG:boto:Headers: {} 2024-04-19T23:23:08.352 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:08.352 DEBUG:boto:Port: 8001 2024-04-19T23:23:08.352 DEBUG:boto:Params: {} 2024-04-19T23:23:08.352 DEBUG:boto:Token: None 2024-04-19T23:23:08.352 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:23:08 GMT /aktlji-9/ 2024-04-19T23:23:08.353 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6XAKur+30x/wMfK8QdouvL8WcI= 2024-04-19T23:23:08.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6XAKur+30x/wMfK8QdouvL8WcI=', 'Content-Length': '0'} 2024-04-19T23:23:08.424 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000026e93044931cf4c-006622fcdc-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:23:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:08.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:08.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:08.425 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:08.480 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:08.480 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:08.513 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:23:08.527 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:23:08.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:08.527 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:08.527 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:23:08.527 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:08.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:08.527 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:08.528 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:23:08.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:08.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:08.528 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:08.528 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:23:08.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:08.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:08.528 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:23:08.528 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:23:08.528 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:08.528 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:23:08.529 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:08.529 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:23:08.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:08.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:08.530 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:08.591 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:08.592 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:08.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:08.632+0000 7fcf64725780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:08.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:08.633+0000 7fcf64725780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:08.650 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:23:08.651 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:23:08.651 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:23:08.651 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:23:08.651 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:23:08.651 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:23:08.651 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:08.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:08.651 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:23:08.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:08.652 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:23:08.652 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:08.652 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:08.652 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:08.652 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:08.652 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:23:08.652 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:08.652 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:23:08.652 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:08.653 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:08.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:08.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:08.653 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:23:08.653 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:08.653 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:08.653 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:23:08.653 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:08.653 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:08.653 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:08.654 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:23:08.654 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:08.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:08.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:08.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:08.654 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:23:08.654 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:08.654 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:08.654 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:23:08.654 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:08.654 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:08.655 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:08.655 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:23:08.655 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:08.655 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:08.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:08.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:08.655 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:23:08.655 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:08.655 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:08.655 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.261293_69.1", 2024-04-19T23:23:08.655 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:08.656 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:08.656 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:08.656 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.261293Z", 2024-04-19T23:23:08.656 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:08.656 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:08.656 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:08.656 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:23:08.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:08.657 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:23:08.657 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:23:08.657 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:23:08.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:08.657 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:23:08Z" 2024-04-19T23:23:08.657 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:23:08.657 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.261293_69.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.261293Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:23:08Z" } 2024-04-19T23:23:08.658 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.261293_69.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.261293Z', 'realm_epoch': 2}}] 2024-04-19T23:23:08.658 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:08.658 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568988.261293_69.1'} 2024-04-19T23:23:08.658 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1713568988.397150_80.1 target=1_1713568988.261293_69.1 2024-04-19T23:23:13.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:13.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:13.660 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:13.718 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:13.719 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:13.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:13.758+0000 7f0531120780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:13.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:13.759+0000 7f0531120780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:13.769 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:23:13.770 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:23:13.770 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:23:13.770 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:23:13.770 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:23:13.770 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:23:13.770 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:13.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:13.770 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:23:13.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:13.771 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:23:13.771 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:13.771 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:13.771 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:13.771 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:13.771 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:23:13.771 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:13.771 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:23:13.771 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:13.772 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:13.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:13.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:13.772 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:23:13.772 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:13.772 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:13.772 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:23:13.772 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:13.772 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:13.772 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:13.772 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:23:13.773 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:13.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:13.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:13.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:13.773 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:23:13.773 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:13.773 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:13.773 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:23:13.773 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:13.773 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:13.773 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:13.774 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:23:13.774 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:13.774 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:13.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:13.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:13.774 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:23:13.774 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:13.774 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:13.774 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:23:13.774 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:13.774 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:13.775 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:13.775 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:23:13.775 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:13.775 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:13.775 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:13.775 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:23:13.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:13.775 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:23:13.775 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:23:13.775 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:23:13.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:13.776 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:23:13Z" 2024-04-19T23:23:13.776 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:23:13.777 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:23:13Z" } 2024-04-19T23:23:13.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:23:13.777 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:13.777 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:13.777 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:23:13.778 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:23:13.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:13.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:13.778 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:13.838 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:13.839 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:13.873 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:13.876+0000 7f147b3ca780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:13.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:13.878+0000 7f147b3ca780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:13.887 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:23:13.888 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:23:13.888 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:23:13.888 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:23:13.889 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:23:13.889 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:23:13.889 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:13.889 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:13.889 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:23:13.890 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:13.890 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:23:13.890 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:13.890 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:13.890 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:13.891 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:13.891 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:23:13.891 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:13.891 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:23:13.892 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:13.892 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:13.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:13.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:13.892 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:23:13.893 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:13.893 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:13.893 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:13.893 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:13.893 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:13.893 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:13.894 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:23:13.894 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:13.894 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:13.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:13.895 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:13.895 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:23:13.895 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:13.895 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:13.896 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:13.896 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:13.896 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:13.896 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:13.896 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:23:13.897 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:13.897 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:13.897 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:13.897 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:13.898 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:23:13.898 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:13.898 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:13.898 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:23:13.898 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:13.899 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:13.899 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:13.899 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:23:13.899 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:13.899 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:13.900 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:13.900 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:23:13.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:13.900 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:23:13.900 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:23:13.901 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:23:13.901 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:13.901 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:23:13Z" 2024-04-19T23:23:13.901 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:23:13.903 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:23:13Z" } 2024-04-19T23:23:13.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:23:13.904 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:13.904 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:13.904 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:23:13.904 DEBUG:boto:path=/aktlji-7/ 2024-04-19T23:23:13.904 DEBUG:boto:auth_path=/aktlji-7/ 2024-04-19T23:23:13.905 DEBUG:boto:Method: HEAD 2024-04-19T23:23:13.905 DEBUG:boto:Path: /aktlji-7/ 2024-04-19T23:23:13.905 DEBUG:boto:Data: 2024-04-19T23:23:13.905 DEBUG:boto:Headers: {} 2024-04-19T23:23:13.905 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:13.905 DEBUG:boto:Port: 8000 2024-04-19T23:23:13.905 DEBUG:boto:Params: {} 2024-04-19T23:23:13.905 DEBUG:boto:Token: None 2024-04-19T23:23:13.906 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:13 GMT /aktlji-7/ 2024-04-19T23:23:13.906 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZTPP1Qhxz9pLbWqWiukH5wCRN0A= 2024-04-19T23:23:13.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZTPP1Qhxz9pLbWqWiukH5wCRN0A=', 'Content-Length': '0'} 2024-04-19T23:23:13.907 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx0000035e8127ad68b0325-006622fce1-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:13.907 DEBUG:boto:path=/aktlji-8/ 2024-04-19T23:23:13.907 DEBUG:boto:auth_path=/aktlji-8/ 2024-04-19T23:23:13.907 DEBUG:boto:Method: HEAD 2024-04-19T23:23:13.907 DEBUG:boto:Path: /aktlji-8/ 2024-04-19T23:23:13.908 DEBUG:boto:Data: 2024-04-19T23:23:13.908 DEBUG:boto:Headers: {} 2024-04-19T23:23:13.908 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:13.908 DEBUG:boto:Port: 8000 2024-04-19T23:23:13.908 DEBUG:boto:Params: {} 2024-04-19T23:23:13.908 DEBUG:boto:Token: None 2024-04-19T23:23:13.908 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:13 GMT /aktlji-8/ 2024-04-19T23:23:13.908 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4tb+woQVPQ2qfiW+GJhMKDKXZbY= 2024-04-19T23:23:13.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4tb+woQVPQ2qfiW+GJhMKDKXZbY=', 'Content-Length': '0'} 2024-04-19T23:23:13.909 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000db07acadbd986574-006622fce1-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:13.909 DEBUG:boto:path=/aktlji-9/ 2024-04-19T23:23:13.909 DEBUG:boto:auth_path=/aktlji-9/ 2024-04-19T23:23:13.910 DEBUG:boto:Method: HEAD 2024-04-19T23:23:13.910 DEBUG:boto:Path: /aktlji-9/ 2024-04-19T23:23:13.910 DEBUG:boto:Data: 2024-04-19T23:23:13.910 DEBUG:boto:Headers: {} 2024-04-19T23:23:13.910 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:13.910 DEBUG:boto:Port: 8000 2024-04-19T23:23:13.910 DEBUG:boto:Params: {} 2024-04-19T23:23:13.910 DEBUG:boto:Token: None 2024-04-19T23:23:13.910 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:13 GMT /aktlji-9/ 2024-04-19T23:23:13.911 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YirhJ/O/o1fYIfzUJw0OegeXttM= 2024-04-19T23:23:13.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YirhJ/O/o1fYIfzUJw0OegeXttM=', 'Content-Length': '0'} 2024-04-19T23:23:13.911 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000cfd8f85a9bd17f58-006622fce1-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:13.912 DEBUG:boto:path=/aktlji-7/ 2024-04-19T23:23:13.912 DEBUG:boto:auth_path=/aktlji-7/ 2024-04-19T23:23:13.912 DEBUG:boto:Method: HEAD 2024-04-19T23:23:13.912 DEBUG:boto:Path: /aktlji-7/ 2024-04-19T23:23:13.912 DEBUG:boto:Data: 2024-04-19T23:23:13.912 DEBUG:boto:Headers: {} 2024-04-19T23:23:13.912 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:13.912 DEBUG:boto:Port: 8000 2024-04-19T23:23:13.912 DEBUG:boto:Params: {} 2024-04-19T23:23:13.913 DEBUG:boto:Token: None 2024-04-19T23:23:13.913 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:13 GMT /aktlji-7/ 2024-04-19T23:23:13.913 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZTPP1Qhxz9pLbWqWiukH5wCRN0A= 2024-04-19T23:23:13.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZTPP1Qhxz9pLbWqWiukH5wCRN0A=', 'Content-Length': '0'} 2024-04-19T23:23:13.914 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx0000055c3a6c5e8e6e5cd-006622fce1-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:13.914 DEBUG:boto:path=/aktlji-8/ 2024-04-19T23:23:13.914 DEBUG:boto:auth_path=/aktlji-8/ 2024-04-19T23:23:13.915 DEBUG:boto:Method: HEAD 2024-04-19T23:23:13.915 DEBUG:boto:Path: /aktlji-8/ 2024-04-19T23:23:13.915 DEBUG:boto:Data: 2024-04-19T23:23:13.915 DEBUG:boto:Headers: {} 2024-04-19T23:23:13.915 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:13.915 DEBUG:boto:Port: 8000 2024-04-19T23:23:13.915 DEBUG:boto:Params: {} 2024-04-19T23:23:13.915 DEBUG:boto:Token: None 2024-04-19T23:23:13.915 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:13 GMT /aktlji-8/ 2024-04-19T23:23:13.916 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4tb+woQVPQ2qfiW+GJhMKDKXZbY= 2024-04-19T23:23:13.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4tb+woQVPQ2qfiW+GJhMKDKXZbY=', 'Content-Length': '0'} 2024-04-19T23:23:13.916 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000416282dd4f72348b-006622fce1-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:13.917 DEBUG:boto:path=/aktlji-9/ 2024-04-19T23:23:13.917 DEBUG:boto:auth_path=/aktlji-9/ 2024-04-19T23:23:13.917 DEBUG:boto:Method: HEAD 2024-04-19T23:23:13.917 DEBUG:boto:Path: /aktlji-9/ 2024-04-19T23:23:13.918 DEBUG:boto:Data: 2024-04-19T23:23:13.918 DEBUG:boto:Headers: {} 2024-04-19T23:23:13.918 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:13.918 DEBUG:boto:Port: 8000 2024-04-19T23:23:13.918 DEBUG:boto:Params: {} 2024-04-19T23:23:13.918 DEBUG:boto:Token: None 2024-04-19T23:23:13.918 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:13 GMT /aktlji-9/ 2024-04-19T23:23:13.918 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YirhJ/O/o1fYIfzUJw0OegeXttM= 2024-04-19T23:23:13.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YirhJ/O/o1fYIfzUJw0OegeXttM=', 'Content-Length': '0'} 2024-04-19T23:23:13.920 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000eb3ae16f2064ba81-006622fce1-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:13.920 DEBUG:boto:path=/aktlji-7/ 2024-04-19T23:23:13.920 DEBUG:boto:auth_path=/aktlji-7/ 2024-04-19T23:23:13.920 DEBUG:boto:Method: HEAD 2024-04-19T23:23:13.920 DEBUG:boto:Path: /aktlji-7/ 2024-04-19T23:23:13.920 DEBUG:boto:Data: 2024-04-19T23:23:13.921 DEBUG:boto:Headers: {} 2024-04-19T23:23:13.921 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:13.921 DEBUG:boto:Port: 8001 2024-04-19T23:23:13.921 DEBUG:boto:Params: {} 2024-04-19T23:23:13.922 DEBUG:boto:Token: None 2024-04-19T23:23:13.922 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:13 GMT /aktlji-7/ 2024-04-19T23:23:13.922 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZTPP1Qhxz9pLbWqWiukH5wCRN0A= 2024-04-19T23:23:13.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZTPP1Qhxz9pLbWqWiukH5wCRN0A=', 'Content-Length': '0'} 2024-04-19T23:23:13.923 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000aba1076c835b6819-006622fce1-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:13.923 DEBUG:boto:path=/aktlji-8/ 2024-04-19T23:23:13.923 DEBUG:boto:auth_path=/aktlji-8/ 2024-04-19T23:23:13.923 DEBUG:boto:Method: HEAD 2024-04-19T23:23:13.924 DEBUG:boto:Path: /aktlji-8/ 2024-04-19T23:23:13.924 DEBUG:boto:Data: 2024-04-19T23:23:13.924 DEBUG:boto:Headers: {} 2024-04-19T23:23:13.924 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:13.924 DEBUG:boto:Port: 8001 2024-04-19T23:23:13.924 DEBUG:boto:Params: {} 2024-04-19T23:23:13.924 DEBUG:boto:Token: None 2024-04-19T23:23:13.924 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:13 GMT /aktlji-8/ 2024-04-19T23:23:13.924 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4tb+woQVPQ2qfiW+GJhMKDKXZbY= 2024-04-19T23:23:13.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4tb+woQVPQ2qfiW+GJhMKDKXZbY=', 'Content-Length': '0'} 2024-04-19T23:23:13.925 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000d9947a4f5f857926-006622fce1-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:13.925 DEBUG:boto:path=/aktlji-9/ 2024-04-19T23:23:13.926 DEBUG:boto:auth_path=/aktlji-9/ 2024-04-19T23:23:13.926 DEBUG:boto:Method: HEAD 2024-04-19T23:23:13.926 DEBUG:boto:Path: /aktlji-9/ 2024-04-19T23:23:13.926 DEBUG:boto:Data: 2024-04-19T23:23:13.926 DEBUG:boto:Headers: {} 2024-04-19T23:23:13.926 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:13.926 DEBUG:boto:Port: 8001 2024-04-19T23:23:13.926 DEBUG:boto:Params: {} 2024-04-19T23:23:13.926 DEBUG:boto:Token: None 2024-04-19T23:23:13.927 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:13 GMT /aktlji-9/ 2024-04-19T23:23:13.927 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YirhJ/O/o1fYIfzUJw0OegeXttM= 2024-04-19T23:23:13.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YirhJ/O/o1fYIfzUJw0OegeXttM=', 'Content-Length': '0'} 2024-04-19T23:23:13.928 DEBUG:boto:Response headers: [('Content-Length', '250'), ('x-amz-request-id', 'tx00000e0cf800aaa841b44-006622fce1-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:13.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2024-04-19T23:23:13.929 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-10 2024-04-19T23:23:13.929 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:13.929 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:13.929 DEBUG:boto:Method: PUT 2024-04-19T23:23:13.929 DEBUG:boto:Path: /aktlji-10/ 2024-04-19T23:23:13.929 DEBUG:boto:Data: 2024-04-19T23:23:13.929 DEBUG:boto:Headers: {} 2024-04-19T23:23:13.930 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:13.930 DEBUG:boto:Port: 8000 2024-04-19T23:23:13.930 DEBUG:boto:Params: {} 2024-04-19T23:23:13.930 DEBUG:boto:Token: None 2024-04-19T23:23:13.930 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:13 GMT /aktlji-10/ 2024-04-19T23:23:13.930 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mmqU+Wn9dnI09w8BrqPrEPq8iZQ= 2024-04-19T23:23:13.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mmqU+Wn9dnI09w8BrqPrEPq8iZQ=', 'Content-Length': '0'} 2024-04-19T23:23:13.946 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b3da9cb8db66be26-006622fce1-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:13.946 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-11 2024-04-19T23:23:13.946 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:23:13.946 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:23:13.947 DEBUG:boto:Method: PUT 2024-04-19T23:23:13.947 DEBUG:boto:Path: /aktlji-11/ 2024-04-19T23:23:13.947 DEBUG:boto:Data: 2024-04-19T23:23:13.947 DEBUG:boto:Headers: {} 2024-04-19T23:23:13.947 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:13.947 DEBUG:boto:Port: 8000 2024-04-19T23:23:13.947 DEBUG:boto:Params: {} 2024-04-19T23:23:13.947 DEBUG:boto:Token: None 2024-04-19T23:23:13.947 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:13 GMT /aktlji-11/ 2024-04-19T23:23:13.947 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UN6WaP+99bcyy5ZTbJEexTVilrI= 2024-04-19T23:23:13.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UN6WaP+99bcyy5ZTbJEexTVilrI=', 'Content-Length': '0'} 2024-04-19T23:23:13.976 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001a0cd685997cf11b-006622fce1-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:13.976 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-12 2024-04-19T23:23:13.976 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:23:13.976 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:23:13.976 DEBUG:boto:Method: PUT 2024-04-19T23:23:13.977 DEBUG:boto:Path: /aktlji-12/ 2024-04-19T23:23:13.977 DEBUG:boto:Data: 2024-04-19T23:23:13.977 DEBUG:boto:Headers: {} 2024-04-19T23:23:13.977 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:13.977 DEBUG:boto:Port: 8001 2024-04-19T23:23:13.977 DEBUG:boto:Params: {} 2024-04-19T23:23:13.977 DEBUG:boto:Token: None 2024-04-19T23:23:13.977 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:23:13 GMT /aktlji-12/ 2024-04-19T23:23:13.977 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T0GcCad7pC5vfiarNHplbngWwAw= 2024-04-19T23:23:13.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T0GcCad7pC5vfiarNHplbngWwAw=', 'Content-Length': '0'} 2024-04-19T23:23:14.012 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000888db5b97cf6cbbb-006622fce1-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.012 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:14.012 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:14.013 DEBUG:boto:Method: HEAD 2024-04-19T23:23:14.013 DEBUG:boto:Path: /aktlji-10/ 2024-04-19T23:23:14.013 DEBUG:boto:Data: 2024-04-19T23:23:14.013 DEBUG:boto:Headers: {} 2024-04-19T23:23:14.013 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:14.013 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.013 DEBUG:boto:Params: {} 2024-04-19T23:23:14.013 DEBUG:boto:Token: None 2024-04-19T23:23:14.013 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:14 GMT /aktlji-10/ 2024-04-19T23:23:14.014 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xqtqN5u0cabWbAo12gbs1KC/wWk= 2024-04-19T23:23:14.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xqtqN5u0cabWbAo12gbs1KC/wWk=', 'Content-Length': '0'} 2024-04-19T23:23:14.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000591936cca856203f-006622fce2-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.017 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:23:14.017 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:23:14.017 DEBUG:boto:Method: PUT 2024-04-19T23:23:14.017 DEBUG:boto:Path: /aktlji-10/myobj 2024-04-19T23:23:14.017 DEBUG:boto:Data: 2024-04-19T23:23:14.017 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:23:14.017 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:14.017 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.017 DEBUG:boto:Params: {} 2024-04-19T23:23:14.018 DEBUG:boto:Token: None 2024-04-19T23:23:14.018 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:23:14 GMT /aktlji-10/myobj 2024-04-19T23:23:14.018 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IQiFgceXiz7H80mWusF48vDFD/k= 2024-04-19T23:23:14.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IQiFgceXiz7H80mWusF48vDFD/k='} 2024-04-19T23:23:14.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e15890277561598d-006622fce2-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.032 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:14.032 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:14.032 DEBUG:boto:Method: HEAD 2024-04-19T23:23:14.032 DEBUG:boto:Path: /aktlji-10/ 2024-04-19T23:23:14.032 DEBUG:boto:Data: 2024-04-19T23:23:14.032 DEBUG:boto:Headers: {} 2024-04-19T23:23:14.033 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:14.033 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.033 DEBUG:boto:Params: {} 2024-04-19T23:23:14.033 DEBUG:boto:Token: None 2024-04-19T23:23:14.033 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:14 GMT /aktlji-10/ 2024-04-19T23:23:14.033 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xqtqN5u0cabWbAo12gbs1KC/wWk= 2024-04-19T23:23:14.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xqtqN5u0cabWbAo12gbs1KC/wWk=', 'Content-Length': '0'} 2024-04-19T23:23:14.035 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001371baba516dbf01-006622fce2-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.036 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:23:14.036 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:23:14.036 DEBUG:boto:Method: PUT 2024-04-19T23:23:14.036 DEBUG:boto:Path: /aktlji-10/_myobj 2024-04-19T23:23:14.036 DEBUG:boto:Data: 2024-04-19T23:23:14.036 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:23:14.036 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:14.036 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.037 DEBUG:boto:Params: {} 2024-04-19T23:23:14.037 DEBUG:boto:Token: None 2024-04-19T23:23:14.037 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:23:14 GMT /aktlji-10/_myobj 2024-04-19T23:23:14.037 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sPFDx3JPMddoRegDq9O50mx+Sk0= 2024-04-19T23:23:14.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sPFDx3JPMddoRegDq9O50mx+Sk0='} 2024-04-19T23:23:14.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071adc00945801c18-006622fce2-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.083 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:14.083 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:14.083 DEBUG:boto:Method: HEAD 2024-04-19T23:23:14.083 DEBUG:boto:Path: /aktlji-10/ 2024-04-19T23:23:14.083 DEBUG:boto:Data: 2024-04-19T23:23:14.083 DEBUG:boto:Headers: {} 2024-04-19T23:23:14.083 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:14.084 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.084 DEBUG:boto:Params: {} 2024-04-19T23:23:14.084 DEBUG:boto:Token: None 2024-04-19T23:23:14.084 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:14 GMT /aktlji-10/ 2024-04-19T23:23:14.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xqtqN5u0cabWbAo12gbs1KC/wWk= 2024-04-19T23:23:14.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xqtqN5u0cabWbAo12gbs1KC/wWk=', 'Content-Length': '0'} 2024-04-19T23:23:14.087 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049509ef172a4d81b-006622fce2-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.087 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:23:14.087 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:23:14.087 DEBUG:boto:Method: PUT 2024-04-19T23:23:14.087 DEBUG:boto:Path: /aktlji-10/%3A 2024-04-19T23:23:14.087 DEBUG:boto:Data: 2024-04-19T23:23:14.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:23:14.088 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:14.088 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.088 DEBUG:boto:Params: {} 2024-04-19T23:23:14.088 DEBUG:boto:Token: None 2024-04-19T23:23:14.088 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:23:14 GMT /aktlji-10/%3A 2024-04-19T23:23:14.088 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kjWQRNHCiH8rWND1NxeUevLU3y0= 2024-04-19T23:23:14.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kjWQRNHCiH8rWND1NxeUevLU3y0='} 2024-04-19T23:23:14.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000342136c490a11b11-006622fce2-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.098 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:14.098 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:14.098 DEBUG:boto:Method: HEAD 2024-04-19T23:23:14.099 DEBUG:boto:Path: /aktlji-10/ 2024-04-19T23:23:14.099 DEBUG:boto:Data: 2024-04-19T23:23:14.099 DEBUG:boto:Headers: {} 2024-04-19T23:23:14.099 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:14.099 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.099 DEBUG:boto:Params: {} 2024-04-19T23:23:14.099 DEBUG:boto:Token: None 2024-04-19T23:23:14.099 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:14 GMT /aktlji-10/ 2024-04-19T23:23:14.099 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xqtqN5u0cabWbAo12gbs1KC/wWk= 2024-04-19T23:23:14.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xqtqN5u0cabWbAo12gbs1KC/wWk=', 'Content-Length': '0'} 2024-04-19T23:23:14.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007af2137469f00387-006622fce2-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.102 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:23:14.102 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:23:14.102 DEBUG:boto:Method: PUT 2024-04-19T23:23:14.102 DEBUG:boto:Path: /aktlji-10/%26 2024-04-19T23:23:14.102 DEBUG:boto:Data: 2024-04-19T23:23:14.102 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:23:14.102 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:14.102 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.103 DEBUG:boto:Params: {} 2024-04-19T23:23:14.103 DEBUG:boto:Token: None 2024-04-19T23:23:14.103 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:23:14 GMT /aktlji-10/%26 2024-04-19T23:23:14.103 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u8cdi+B9PpkoekklLsJwxdLPi6U= 2024-04-19T23:23:14.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u8cdi+B9PpkoekklLsJwxdLPi6U='} 2024-04-19T23:23:14.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000758c416f4008b98c-006622fce2-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.147 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:23:14.147 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:23:14.147 DEBUG:boto:Method: HEAD 2024-04-19T23:23:14.148 DEBUG:boto:Path: /aktlji-11/ 2024-04-19T23:23:14.148 DEBUG:boto:Data: 2024-04-19T23:23:14.148 DEBUG:boto:Headers: {} 2024-04-19T23:23:14.148 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:14.148 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.148 DEBUG:boto:Params: {} 2024-04-19T23:23:14.148 DEBUG:boto:Token: None 2024-04-19T23:23:14.148 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:14 GMT /aktlji-11/ 2024-04-19T23:23:14.148 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2SWyKONIKmXaM7CcqMi8DzPM06c= 2024-04-19T23:23:14.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2SWyKONIKmXaM7CcqMi8DzPM06c=', 'Content-Length': '0'} 2024-04-19T23:23:14.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f22df842075753a0-006622fce2-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.151 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:23:14.152 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:23:14.152 DEBUG:boto:Method: PUT 2024-04-19T23:23:14.152 DEBUG:boto:Path: /aktlji-11/myobj 2024-04-19T23:23:14.152 DEBUG:boto:Data: 2024-04-19T23:23:14.152 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:23:14.152 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:14.152 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.152 DEBUG:boto:Params: {} 2024-04-19T23:23:14.152 DEBUG:boto:Token: None 2024-04-19T23:23:14.153 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:23:14 GMT /aktlji-11/myobj 2024-04-19T23:23:14.153 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iKiK+6chkMlt0/0PCvXyfHxsK00= 2024-04-19T23:23:14.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iKiK+6chkMlt0/0PCvXyfHxsK00='} 2024-04-19T23:23:14.165 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ceb0db5bbf629f72-006622fce2-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.166 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:23:14.166 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:23:14.166 DEBUG:boto:Method: HEAD 2024-04-19T23:23:14.166 DEBUG:boto:Path: /aktlji-11/ 2024-04-19T23:23:14.166 DEBUG:boto:Data: 2024-04-19T23:23:14.166 DEBUG:boto:Headers: {} 2024-04-19T23:23:14.166 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:14.166 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.167 DEBUG:boto:Params: {} 2024-04-19T23:23:14.167 DEBUG:boto:Token: None 2024-04-19T23:23:14.167 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:14 GMT /aktlji-11/ 2024-04-19T23:23:14.167 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2SWyKONIKmXaM7CcqMi8DzPM06c= 2024-04-19T23:23:14.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2SWyKONIKmXaM7CcqMi8DzPM06c=', 'Content-Length': '0'} 2024-04-19T23:23:14.170 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c322473af27ff3b-006622fce2-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.170 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:23:14.170 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:23:14.170 DEBUG:boto:Method: PUT 2024-04-19T23:23:14.170 DEBUG:boto:Path: /aktlji-11/_myobj 2024-04-19T23:23:14.170 DEBUG:boto:Data: 2024-04-19T23:23:14.171 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:23:14.171 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:14.171 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.171 DEBUG:boto:Params: {} 2024-04-19T23:23:14.171 DEBUG:boto:Token: None 2024-04-19T23:23:14.171 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:23:14 GMT /aktlji-11/_myobj 2024-04-19T23:23:14.171 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LXn/42cAuwt/4XBFUh0UODOtzVQ= 2024-04-19T23:23:14.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LXn/42cAuwt/4XBFUh0UODOtzVQ='} 2024-04-19T23:23:14.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004dd5644b3202b51c-006622fce2-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.182 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:23:14.182 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:23:14.182 DEBUG:boto:Method: HEAD 2024-04-19T23:23:14.182 DEBUG:boto:Path: /aktlji-11/ 2024-04-19T23:23:14.182 DEBUG:boto:Data: 2024-04-19T23:23:14.182 DEBUG:boto:Headers: {} 2024-04-19T23:23:14.183 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:14.183 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.183 DEBUG:boto:Params: {} 2024-04-19T23:23:14.183 DEBUG:boto:Token: None 2024-04-19T23:23:14.183 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:14 GMT /aktlji-11/ 2024-04-19T23:23:14.183 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2SWyKONIKmXaM7CcqMi8DzPM06c= 2024-04-19T23:23:14.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2SWyKONIKmXaM7CcqMi8DzPM06c=', 'Content-Length': '0'} 2024-04-19T23:23:14.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079eadaa06bc51931-006622fce2-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.186 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:23:14.186 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:23:14.186 DEBUG:boto:Method: PUT 2024-04-19T23:23:14.187 DEBUG:boto:Path: /aktlji-11/%3A 2024-04-19T23:23:14.187 DEBUG:boto:Data: 2024-04-19T23:23:14.187 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:23:14.187 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:14.187 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.187 DEBUG:boto:Params: {} 2024-04-19T23:23:14.187 DEBUG:boto:Token: None 2024-04-19T23:23:14.187 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:23:14 GMT /aktlji-11/%3A 2024-04-19T23:23:14.188 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6fKxz7XuWFK5OMT4xd7Ye+AcPm0= 2024-04-19T23:23:14.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6fKxz7XuWFK5OMT4xd7Ye+AcPm0='} 2024-04-19T23:23:14.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002910c1e87212b420-006622fce2-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.231 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:23:14.231 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:23:14.231 DEBUG:boto:Method: HEAD 2024-04-19T23:23:14.231 DEBUG:boto:Path: /aktlji-11/ 2024-04-19T23:23:14.231 DEBUG:boto:Data: 2024-04-19T23:23:14.231 DEBUG:boto:Headers: {} 2024-04-19T23:23:14.231 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:14.231 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.232 DEBUG:boto:Params: {} 2024-04-19T23:23:14.232 DEBUG:boto:Token: None 2024-04-19T23:23:14.232 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:14 GMT /aktlji-11/ 2024-04-19T23:23:14.232 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2SWyKONIKmXaM7CcqMi8DzPM06c= 2024-04-19T23:23:14.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2SWyKONIKmXaM7CcqMi8DzPM06c=', 'Content-Length': '0'} 2024-04-19T23:23:14.234 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe5ad99903a00843-006622fce2-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.235 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:23:14.235 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:23:14.235 DEBUG:boto:Method: PUT 2024-04-19T23:23:14.235 DEBUG:boto:Path: /aktlji-11/%26 2024-04-19T23:23:14.235 DEBUG:boto:Data: 2024-04-19T23:23:14.235 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:23:14.236 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:14.236 DEBUG:boto:Port: 8000 2024-04-19T23:23:14.236 DEBUG:boto:Params: {} 2024-04-19T23:23:14.236 DEBUG:boto:Token: None 2024-04-19T23:23:14.236 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:23:14 GMT /aktlji-11/%26 2024-04-19T23:23:14.236 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2ssmj9xVlrVMsxILHK6jFDAWpkY= 2024-04-19T23:23:14.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2ssmj9xVlrVMsxILHK6jFDAWpkY='} 2024-04-19T23:23:14.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008725217810de6ada-006622fce2-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.243 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:23:14.244 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:23:14.244 DEBUG:boto:Method: HEAD 2024-04-19T23:23:14.244 DEBUG:boto:Path: /aktlji-12/ 2024-04-19T23:23:14.244 DEBUG:boto:Data: 2024-04-19T23:23:14.244 DEBUG:boto:Headers: {} 2024-04-19T23:23:14.244 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:14.244 DEBUG:boto:Port: 8001 2024-04-19T23:23:14.244 DEBUG:boto:Params: {} 2024-04-19T23:23:14.244 DEBUG:boto:Token: None 2024-04-19T23:23:14.245 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:14 GMT /aktlji-12/ 2024-04-19T23:23:14.245 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iORQCBgjSdM4gVaOV7bFO4ylWJM= 2024-04-19T23:23:14.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iORQCBgjSdM4gVaOV7bFO4ylWJM=', 'Content-Length': '0'} 2024-04-19T23:23:14.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000444a3ac7b98a60cf-006622fce2-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.247 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:23:14.248 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:23:14.248 DEBUG:boto:Method: PUT 2024-04-19T23:23:14.248 DEBUG:boto:Path: /aktlji-12/myobj 2024-04-19T23:23:14.248 DEBUG:boto:Data: 2024-04-19T23:23:14.248 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:23:14.248 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:14.248 DEBUG:boto:Port: 8001 2024-04-19T23:23:14.248 DEBUG:boto:Params: {} 2024-04-19T23:23:14.248 DEBUG:boto:Token: None 2024-04-19T23:23:14.248 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:23:14 GMT /aktlji-12/myobj 2024-04-19T23:23:14.249 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9z7ACuDHvb7kxWT6kffcudbnam8= 2024-04-19T23:23:14.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9z7ACuDHvb7kxWT6kffcudbnam8='} 2024-04-19T23:23:14.265 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e94f857ba24e89a6-006622fce2-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.266 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:23:14.266 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:23:14.266 DEBUG:boto:Method: HEAD 2024-04-19T23:23:14.266 DEBUG:boto:Path: /aktlji-12/ 2024-04-19T23:23:14.266 DEBUG:boto:Data: 2024-04-19T23:23:14.266 DEBUG:boto:Headers: {} 2024-04-19T23:23:14.266 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:14.267 DEBUG:boto:Port: 8001 2024-04-19T23:23:14.267 DEBUG:boto:Params: {} 2024-04-19T23:23:14.267 DEBUG:boto:Token: None 2024-04-19T23:23:14.267 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:14 GMT /aktlji-12/ 2024-04-19T23:23:14.267 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iORQCBgjSdM4gVaOV7bFO4ylWJM= 2024-04-19T23:23:14.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iORQCBgjSdM4gVaOV7bFO4ylWJM=', 'Content-Length': '0'} 2024-04-19T23:23:14.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023d53e27fa73179d-006622fce2-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.270 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:23:14.270 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:23:14.270 DEBUG:boto:Method: PUT 2024-04-19T23:23:14.270 DEBUG:boto:Path: /aktlji-12/_myobj 2024-04-19T23:23:14.270 DEBUG:boto:Data: 2024-04-19T23:23:14.271 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:23:14.271 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:14.271 DEBUG:boto:Port: 8001 2024-04-19T23:23:14.271 DEBUG:boto:Params: {} 2024-04-19T23:23:14.271 DEBUG:boto:Token: None 2024-04-19T23:23:14.271 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:23:14 GMT /aktlji-12/_myobj 2024-04-19T23:23:14.271 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lwxI/u42m7Vw1LB/HAsp5tEwaus= 2024-04-19T23:23:14.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lwxI/u42m7Vw1LB/HAsp5tEwaus='} 2024-04-19T23:23:14.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000856040bfcef70a34-006622fce2-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.315 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:23:14.315 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:23:14.315 DEBUG:boto:Method: HEAD 2024-04-19T23:23:14.315 DEBUG:boto:Path: /aktlji-12/ 2024-04-19T23:23:14.315 DEBUG:boto:Data: 2024-04-19T23:23:14.315 DEBUG:boto:Headers: {} 2024-04-19T23:23:14.315 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:14.315 DEBUG:boto:Port: 8001 2024-04-19T23:23:14.315 DEBUG:boto:Params: {} 2024-04-19T23:23:14.316 DEBUG:boto:Token: None 2024-04-19T23:23:14.316 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:14 GMT /aktlji-12/ 2024-04-19T23:23:14.316 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iORQCBgjSdM4gVaOV7bFO4ylWJM= 2024-04-19T23:23:14.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iORQCBgjSdM4gVaOV7bFO4ylWJM=', 'Content-Length': '0'} 2024-04-19T23:23:14.319 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035e2271ccd63dff3-006622fce2-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.319 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:23:14.319 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:23:14.319 DEBUG:boto:Method: PUT 2024-04-19T23:23:14.319 DEBUG:boto:Path: /aktlji-12/%3A 2024-04-19T23:23:14.320 DEBUG:boto:Data: 2024-04-19T23:23:14.320 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:23:14.320 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:14.320 DEBUG:boto:Port: 8001 2024-04-19T23:23:14.320 DEBUG:boto:Params: {} 2024-04-19T23:23:14.320 DEBUG:boto:Token: None 2024-04-19T23:23:14.320 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:23:14 GMT /aktlji-12/%3A 2024-04-19T23:23:14.320 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Q6FLibQeYZNQ4oS7tXidLMXPfsg= 2024-04-19T23:23:14.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q6FLibQeYZNQ4oS7tXidLMXPfsg='} 2024-04-19T23:23:14.331 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbfd9f503fd95eda-006622fce2-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.332 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:23:14.332 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:23:14.332 DEBUG:boto:Method: HEAD 2024-04-19T23:23:14.332 DEBUG:boto:Path: /aktlji-12/ 2024-04-19T23:23:14.332 DEBUG:boto:Data: 2024-04-19T23:23:14.332 DEBUG:boto:Headers: {} 2024-04-19T23:23:14.332 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:14.332 DEBUG:boto:Port: 8001 2024-04-19T23:23:14.332 DEBUG:boto:Params: {} 2024-04-19T23:23:14.332 DEBUG:boto:Token: None 2024-04-19T23:23:14.333 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:14 GMT /aktlji-12/ 2024-04-19T23:23:14.333 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iORQCBgjSdM4gVaOV7bFO4ylWJM= 2024-04-19T23:23:14.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iORQCBgjSdM4gVaOV7bFO4ylWJM=', 'Content-Length': '0'} 2024-04-19T23:23:14.335 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b5af127d1b34666-006622fce2-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.335 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:23:14.335 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:23:14.335 DEBUG:boto:Method: PUT 2024-04-19T23:23:14.335 DEBUG:boto:Path: /aktlji-12/%26 2024-04-19T23:23:14.335 DEBUG:boto:Data: 2024-04-19T23:23:14.336 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:23:14.336 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:23:14.336 DEBUG:boto:Port: 8001 2024-04-19T23:23:14.336 DEBUG:boto:Params: {} 2024-04-19T23:23:14.336 DEBUG:boto:Token: None 2024-04-19T23:23:14.336 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:23:14 GMT /aktlji-12/%26 2024-04-19T23:23:14.336 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rHoPHzZwVEeMiapiSlXgRRO+moQ= 2024-04-19T23:23:14.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:23:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rHoPHzZwVEeMiapiSlXgRRO+moQ='} 2024-04-19T23:23:14.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0a899a5f75f7071-006622fce2-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:14.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:14.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:14.383 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:14.441 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:14.441 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:14.477 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:23:14.491 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:23:14.492 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:14.492 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:14.492 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:23:14.492 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:14.492 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:14.492 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:14.492 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:23:14.492 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:14.492 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:14.492 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568994.002748_38.1", 2024-04-19T23:23:14.492 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:14.002748Z" 2024-04-19T23:23:14.493 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:14.493 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:14.493 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:23:14.493 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:23:14.493 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:14.493 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:23:14.494 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713568994.002748_38.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:14.494 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:23:14.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:14.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:14.495 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:14.556 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:14.556 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:14.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:14.596+0000 7f019d0ef780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:14.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:14.597+0000 7f019d0ef780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:14.607 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:23:14.607 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:23:14.607 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:23:14.607 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:23:14.607 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:23:14.607 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:23:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:23:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:23:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:23:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:14.608 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:23:14.609 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:14.609 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:14.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:14.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:14.609 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:23:14.609 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:14.609 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:14.609 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:23:14.609 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:14.609 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:14.609 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:14.610 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:23:14.610 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:14.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:14.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:14.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:14.610 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:23:14.610 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:14.610 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:14.610 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568993.941668_27.1", 2024-04-19T23:23:14.610 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:14.610 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:14.610 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:14.611 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:13.941668Z", 2024-04-19T23:23:14.611 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:14.611 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:14.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:14.611 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:14.611 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:23:14.611 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:14.611 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:14.611 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:23:14.611 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:14.611 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:14.611 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:14.612 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:23:14.612 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:14.612 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:14.612 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:14.612 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:23:14.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:14.612 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:23:14.612 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:23:14.612 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:23:14.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:14.612 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:23:14Z" 2024-04-19T23:23:14.612 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:23:14.613 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713568993.941668_27.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:13.941668Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:23:14Z" } 2024-04-19T23:23:14.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713568993.941668_27.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:13.941668Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:23:14.614 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713568994.002748_38.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:14.614 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713568993.941668_27.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:14.614 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713568994.002748_38.1 target=1_1713568993.941668_27.1 2024-04-19T23:23:19.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:19.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:19.616 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:19.674 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:19.675 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:19.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:19.714+0000 7fad4a91f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:19.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:19.715+0000 7fad4a91f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:19.724 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:23:19.725 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:23:19.725 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:23:19.725 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:23:19.725 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:23:19.725 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:23:19.725 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:19.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:19.725 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:23:19.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:19.726 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:23:19.726 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:19.726 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:19.726 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:19.726 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:19.726 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:23:19.726 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:19.726 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:23:19.726 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:19.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:19.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:19.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:19.727 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:23:19.727 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:19.727 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:19.727 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:23:19.727 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:19.727 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:19.727 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:19.728 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:23:19.728 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:19.728 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:19.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:19.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:19.728 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:23:19.728 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:19.728 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:19.728 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568994.002748_38.1", 2024-04-19T23:23:19.728 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:19.728 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:19.729 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:19.729 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:14.002748Z", 2024-04-19T23:23:19.729 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:19.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:19.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:19.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:23:19.729 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:23:19.729 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:23:19.729 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:23:19.729 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:23:19.730 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:23:19.730 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:23:19.730 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:23:19.730 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:23:19.730 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:23:19.730 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:19.730 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:23:19.730 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:23:19.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:19.731 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:23:19.731 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:23:19.731 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:23:19.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:23:19.731 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:23:19Z" 2024-04-19T23:23:19.731 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:23:19.732 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713568994.002748_38.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:14.002748Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:23:19Z" } 2024-04-19T23:23:19.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713568994.002748_38.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:14.002748Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:23:19.733 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713568994.002748_38.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:19.733 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713568994.002748_38.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:19.733 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:23:19.733 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:23:19.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:19.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:23:19.734 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:23:19.793 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:19.793 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:19.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:19.834+0000 7f1b4b46e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:19.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:19.835+0000 7f1b4b46e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:23:19.845 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:23:19.845 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:23:19.845 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:23:19.846 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:23:19.846 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:23:19.846 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:23:19.846 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:19.846 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:19.846 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:23:19.846 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:19.846 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:23:19.846 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:19.847 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:19.847 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:23:19.847 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:19.847 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:23:19.847 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:19.847 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:23:19.847 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:19.847 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:19.847 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:19.847 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:19.847 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:23:19.848 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:19.848 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:19.848 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:23:19.848 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:19.848 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:19.848 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:19.848 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:23:19.848 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:19.848 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:19.848 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:19.849 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:19.849 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:23:19.849 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:19.849 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:19.849 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568994.002748_38.1", 2024-04-19T23:23:19.849 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:19.849 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:19.849 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:19.849 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:14.002748Z", 2024-04-19T23:23:19.849 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:19.850 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:19.850 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:19.850 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:23:19.850 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:23:19.850 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:23:19.850 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:23:19.850 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:23:19.850 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:23:19.850 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:23:19.850 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:23:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:23:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:23:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:23:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:23:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:23:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:23:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:23:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:23:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:23:19Z" 2024-04-19T23:23:19.852 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:23:19.853 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713568994.002748_38.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:14.002748Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:23:19Z" } 2024-04-19T23:23:19.853 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713568994.002748_38.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:14.002748Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:23:19.853 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713568994.002748_38.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:19.853 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713568994.002748_38.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:23:19.853 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:23:19.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:23:19.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:23:19.854 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:23:19.920 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:19.920 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:19.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:19.965+0000 7f0f7b6f3780 1 waiting to reach incremental sync.. 2024-04-19T23:23:24.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:24.966+0000 7f0f7b6f3780 1 waiting to reach incremental sync.. 2024-04-19T23:23:29.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:29.966+0000 7f0f7b6f3780 1 waiting to reach incremental sync.. 2024-04-19T23:23:31.744 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-19T23:23:31.748+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-19T23:23:34.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:34.968+0000 7f0f7b6f3780 1 bucket sync caught up with source: 2024-04-19T23:23:34.964 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000001.4.6, , 00000000001.6.6, , , , 00000000001.7.6, 00000000001.8.6, , ] 2024-04-19T23:23:34.964 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000001.4.6, , 00000000001.6.6, , , , 00000000001.7.6, 00000000001.8.6, , ] 2024-04-19T23:23:34.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:23:34.968+0000 7f0f7b6f3780 0 bucket checkpoint complete 2024-04-19T23:23:34.981 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-10 zones={test-zone2, test-zone1} 2024-04-19T23:23:34.981 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:34.981 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:34.982 DEBUG:boto:Method: HEAD 2024-04-19T23:23:34.982 DEBUG:boto:Path: /aktlji-10/ 2024-04-19T23:23:34.982 DEBUG:boto:Data: 2024-04-19T23:23:34.982 DEBUG:boto:Headers: {} 2024-04-19T23:23:34.982 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:34.982 DEBUG:boto:Port: 8000 2024-04-19T23:23:34.982 DEBUG:boto:Params: {} 2024-04-19T23:23:34.982 DEBUG:boto:Token: None 2024-04-19T23:23:34.984 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:34 GMT /aktlji-10/ 2024-04-19T23:23:34.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY1fYO8T4blrJoLgFrQBvpNb/68= 2024-04-19T23:23:34.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY1fYO8T4blrJoLgFrQBvpNb/68=', 'Content-Length': '0'} 2024-04-19T23:23:34.987 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a760d27b6db4c313-006622fcf6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:34.987 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:34.987 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:34.987 DEBUG:boto:Method: HEAD 2024-04-19T23:23:34.987 DEBUG:boto:Path: /aktlji-10/ 2024-04-19T23:23:34.987 DEBUG:boto:Data: 2024-04-19T23:23:34.987 DEBUG:boto:Headers: {} 2024-04-19T23:23:34.987 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:34.988 DEBUG:boto:Port: 8000 2024-04-19T23:23:34.988 DEBUG:boto:Params: {} 2024-04-19T23:23:34.988 DEBUG:boto:Token: None 2024-04-19T23:23:34.988 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:34 GMT /aktlji-10/ 2024-04-19T23:23:34.988 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY1fYO8T4blrJoLgFrQBvpNb/68= 2024-04-19T23:23:34.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY1fYO8T4blrJoLgFrQBvpNb/68=', 'Content-Length': '0'} 2024-04-19T23:23:34.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f798d9f549bb7192-006622fcf6-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:23:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:34.991 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:23:34.991 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:34.991 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:34.991 DEBUG:boto:path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:23:34.991 DEBUG:boto:auth_path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:23:34.991 DEBUG:boto:Method: GET 2024-04-19T23:23:34.991 DEBUG:boto:Path: /aktlji-10/?versions&max-keys=999 2024-04-19T23:23:34.992 DEBUG:boto:Data: 2024-04-19T23:23:34.992 DEBUG:boto:Headers: {} 2024-04-19T23:23:34.992 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:34.992 DEBUG:boto:Port: 8000 2024-04-19T23:23:34.992 DEBUG:boto:Params: {} 2024-04-19T23:23:34.992 DEBUG:boto:Token: None 2024-04-19T23:23:34.992 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:34 GMT /aktlji-10/?versions 2024-04-19T23:23:34.992 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YuDvONFHZeQMJpaKw2dSqrcsevU= 2024-04-19T23:23:34.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YuDvONFHZeQMJpaKw2dSqrcsevU=', 'Content-Length': '0'} 2024-04-19T23:23:34.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002de1fd926815f635-006622fcf6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:34.995 DEBUG:boto:b'aktlji-10999false&nulltrue2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:34.997 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-19T23:23:34.997 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-19T23:23:34.998 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-19T23:23:34.998 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:23:34.998 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:23:34.998 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:34.998 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:34.998 DEBUG:boto:path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:23:34.998 DEBUG:boto:auth_path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:23:34.998 DEBUG:boto:Method: GET 2024-04-19T23:23:34.998 DEBUG:boto:Path: /aktlji-10/?versions&max-keys=999 2024-04-19T23:23:34.998 DEBUG:boto:Data: 2024-04-19T23:23:34.999 DEBUG:boto:Headers: {} 2024-04-19T23:23:34.999 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:34.999 DEBUG:boto:Port: 8000 2024-04-19T23:23:34.999 DEBUG:boto:Params: {} 2024-04-19T23:23:34.999 DEBUG:boto:Token: None 2024-04-19T23:23:34.999 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:34 GMT /aktlji-10/?versions 2024-04-19T23:23:34.999 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YuDvONFHZeQMJpaKw2dSqrcsevU= 2024-04-19T23:23:34.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YuDvONFHZeQMJpaKw2dSqrcsevU=', 'Content-Length': '0'} 2024-04-19T23:23:35.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3f9ca015895eebb-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.002 DEBUG:boto:b'aktlji-10999false&nulltrue2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.002 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-19T23:23:35.002 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-19T23:23:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-19T23:23:35.003 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:23:35.003 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.003 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.003 DEBUG:boto:path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:23:35.003 DEBUG:boto:auth_path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:23:35.003 DEBUG:boto:Method: GET 2024-04-19T23:23:35.003 DEBUG:boto:Path: /aktlji-10/?versions&max-keys=999 2024-04-19T23:23:35.003 DEBUG:boto:Data: 2024-04-19T23:23:35.003 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.004 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.004 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.004 DEBUG:boto:Params: {} 2024-04-19T23:23:35.004 DEBUG:boto:Token: None 2024-04-19T23:23:35.004 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/?versions 2024-04-19T23:23:35.004 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XNTXCJHrfJcskBVH5ismonwYKwo= 2024-04-19T23:23:35.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XNTXCJHrfJcskBVH5ismonwYKwo=', 'Content-Length': '0'} 2024-04-19T23:23:35.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b206a112da76419-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.007 DEBUG:boto:b'aktlji-10999false&nulltrue2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.007 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.007 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.008 DEBUG:boto:path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:23:35.008 DEBUG:boto:auth_path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:23:35.008 DEBUG:boto:Method: GET 2024-04-19T23:23:35.008 DEBUG:boto:Path: /aktlji-10/?versions&max-keys=999 2024-04-19T23:23:35.008 DEBUG:boto:Data: 2024-04-19T23:23:35.008 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.008 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.008 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.008 DEBUG:boto:Params: {} 2024-04-19T23:23:35.008 DEBUG:boto:Token: None 2024-04-19T23:23:35.008 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/?versions 2024-04-19T23:23:35.009 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XNTXCJHrfJcskBVH5ismonwYKwo= 2024-04-19T23:23:35.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XNTXCJHrfJcskBVH5ismonwYKwo=', 'Content-Length': '0'} 2024-04-19T23:23:35.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d63f7744e8643532-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.011 DEBUG:boto:b'aktlji-10999false&nulltrue2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:23:35.012 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:23:35.012 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:23:35.012 DEBUG:boto:path=/aktlji-10/%26?versionId=null 2024-04-19T23:23:35.012 DEBUG:boto:auth_path=/aktlji-10/%26?versionId=null 2024-04-19T23:23:35.012 DEBUG:boto:Method: GET 2024-04-19T23:23:35.012 DEBUG:boto:Path: /aktlji-10/%26?versionId=null 2024-04-19T23:23:35.013 DEBUG:boto:Data: 2024-04-19T23:23:35.013 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.013 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.013 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.013 DEBUG:boto:Params: {} 2024-04-19T23:23:35.013 DEBUG:boto:Token: None 2024-04-19T23:23:35.013 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%26?versionId=null 2024-04-19T23:23:35.013 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TGLPk9zIAsZKB29ZVdUrPT91K+Q= 2024-04-19T23:23:35.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TGLPk9zIAsZKB29ZVdUrPT91K+Q=', 'Content-Length': '0'} 2024-04-19T23:23:35.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bd0995a48c7086f-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.062 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:23:35.062 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:23:35.063 DEBUG:boto:path=/aktlji-10/%26?versionId=null 2024-04-19T23:23:35.063 DEBUG:boto:auth_path=/aktlji-10/%26?versionId=null 2024-04-19T23:23:35.063 DEBUG:boto:Method: GET 2024-04-19T23:23:35.063 DEBUG:boto:Path: /aktlji-10/%26?versionId=null 2024-04-19T23:23:35.063 DEBUG:boto:Data: 2024-04-19T23:23:35.063 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.063 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.063 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.063 DEBUG:boto:Params: {} 2024-04-19T23:23:35.064 DEBUG:boto:Token: None 2024-04-19T23:23:35.065 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%26?versionId=null 2024-04-19T23:23:35.066 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TGLPk9zIAsZKB29ZVdUrPT91K+Q= 2024-04-19T23:23:35.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TGLPk9zIAsZKB29ZVdUrPT91K+Q=', 'Content-Length': '0'} 2024-04-19T23:23:35.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b134b6f532df04c6-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.068 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:23:35.069 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:23:35.069 DEBUG:boto:path=/aktlji-10/%26?versionId=null 2024-04-19T23:23:35.069 DEBUG:boto:auth_path=/aktlji-10/%26?versionId=null 2024-04-19T23:23:35.069 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.069 DEBUG:boto:Path: /aktlji-10/%26?versionId=null 2024-04-19T23:23:35.069 DEBUG:boto:Data: 2024-04-19T23:23:35.069 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.069 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.069 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.070 DEBUG:boto:Params: {} 2024-04-19T23:23:35.070 DEBUG:boto:Token: None 2024-04-19T23:23:35.070 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%26?versionId=null 2024-04-19T23:23:35.070 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pG2fiEHFkj4qs75t+Io6VfUomro= 2024-04-19T23:23:35.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pG2fiEHFkj4qs75t+Io6VfUomro=', 'Content-Length': '0'} 2024-04-19T23:23:35.072 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056c5b862c2835538-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.072 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:23:35.073 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:23:35.073 DEBUG:boto:path=/aktlji-10/%26?versionId=null 2024-04-19T23:23:35.073 DEBUG:boto:auth_path=/aktlji-10/%26?versionId=null 2024-04-19T23:23:35.073 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.073 DEBUG:boto:Path: /aktlji-10/%26?versionId=null 2024-04-19T23:23:35.073 DEBUG:boto:Data: 2024-04-19T23:23:35.073 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.073 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.073 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.073 DEBUG:boto:Params: {} 2024-04-19T23:23:35.074 DEBUG:boto:Token: None 2024-04-19T23:23:35.074 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%26?versionId=null 2024-04-19T23:23:35.074 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pG2fiEHFkj4qs75t+Io6VfUomro= 2024-04-19T23:23:35.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pG2fiEHFkj4qs75t+Io6VfUomro=', 'Content-Length': '0'} 2024-04-19T23:23:35.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f732cb9172de1c41-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:23:35.076 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:23:35.076 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:23:35.077 DEBUG:boto:path=/aktlji-10/%26?versionId=null 2024-04-19T23:23:35.077 DEBUG:boto:auth_path=/aktlji-10/%26?versionId=null 2024-04-19T23:23:35.077 DEBUG:boto:Method: GET 2024-04-19T23:23:35.077 DEBUG:boto:Path: /aktlji-10/%26?versionId=null 2024-04-19T23:23:35.077 DEBUG:boto:Data: 2024-04-19T23:23:35.077 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.077 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.077 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.077 DEBUG:boto:Params: {} 2024-04-19T23:23:35.077 DEBUG:boto:Token: None 2024-04-19T23:23:35.078 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%26?versionId=null 2024-04-19T23:23:35.078 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TGLPk9zIAsZKB29ZVdUrPT91K+Q= 2024-04-19T23:23:35.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TGLPk9zIAsZKB29ZVdUrPT91K+Q=', 'Content-Length': '0'} 2024-04-19T23:23:35.080 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006425cd0a4adb11db-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.080 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:23:35.080 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:23:35.080 DEBUG:boto:path=/aktlji-10/%26?versionId=null 2024-04-19T23:23:35.080 DEBUG:boto:auth_path=/aktlji-10/%26?versionId=null 2024-04-19T23:23:35.080 DEBUG:boto:Method: GET 2024-04-19T23:23:35.080 DEBUG:boto:Path: /aktlji-10/%26?versionId=null 2024-04-19T23:23:35.080 DEBUG:boto:Data: 2024-04-19T23:23:35.081 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.081 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.081 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.081 DEBUG:boto:Params: {} 2024-04-19T23:23:35.081 DEBUG:boto:Token: None 2024-04-19T23:23:35.081 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%26?versionId=null 2024-04-19T23:23:35.081 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TGLPk9zIAsZKB29ZVdUrPT91K+Q= 2024-04-19T23:23:35.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TGLPk9zIAsZKB29ZVdUrPT91K+Q=', 'Content-Length': '0'} 2024-04-19T23:23:35.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e57dc68b32f842b6-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.083 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.084 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.084 DEBUG:boto:path=/aktlji-10/?prefix=%26 2024-04-19T23:23:35.084 DEBUG:boto:auth_path=/aktlji-10/?prefix=%26 2024-04-19T23:23:35.084 DEBUG:boto:Method: GET 2024-04-19T23:23:35.084 DEBUG:boto:Path: /aktlji-10/?prefix=%26 2024-04-19T23:23:35.084 DEBUG:boto:Data: 2024-04-19T23:23:35.084 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.084 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.084 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.084 DEBUG:boto:Params: {} 2024-04-19T23:23:35.085 DEBUG:boto:Token: None 2024-04-19T23:23:35.085 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.085 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000602a54e575af6413-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.088 DEBUG:boto:b'aktlji-10&1000false&2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.088 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.088 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.088 DEBUG:boto:path=/aktlji-10/?prefix=%26 2024-04-19T23:23:35.088 DEBUG:boto:auth_path=/aktlji-10/?prefix=%26 2024-04-19T23:23:35.088 DEBUG:boto:Method: GET 2024-04-19T23:23:35.088 DEBUG:boto:Path: /aktlji-10/?prefix=%26 2024-04-19T23:23:35.089 DEBUG:boto:Data: 2024-04-19T23:23:35.089 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.089 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.089 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.089 DEBUG:boto:Params: {} 2024-04-19T23:23:35.089 DEBUG:boto:Token: None 2024-04-19T23:23:35.089 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.089 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dd822c8ba484ff5-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.092 DEBUG:boto:b'aktlji-10&1000false&2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.092 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:23:35.092 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:23:35.093 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.093 DEBUG:boto:Path: /aktlji-10/%26 2024-04-19T23:23:35.093 DEBUG:boto:Data: 2024-04-19T23:23:35.093 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.093 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.093 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.093 DEBUG:boto:Params: {} 2024-04-19T23:23:35.093 DEBUG:boto:Token: None 2024-04-19T23:23:35.093 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%26 2024-04-19T23:23:35.093 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KQZGqIyQMV5BnHJwn968K1I5+ck= 2024-04-19T23:23:35.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KQZGqIyQMV5BnHJwn968K1I5+ck=', 'Content-Length': '0'} 2024-04-19T23:23:35.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ecca27b7e4d819f-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.096 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:23:35.096 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:23:35.096 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.096 DEBUG:boto:Path: /aktlji-10/%26 2024-04-19T23:23:35.096 DEBUG:boto:Data: 2024-04-19T23:23:35.096 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.096 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.097 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.097 DEBUG:boto:Params: {} 2024-04-19T23:23:35.097 DEBUG:boto:Token: None 2024-04-19T23:23:35.097 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%26 2024-04-19T23:23:35.097 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KQZGqIyQMV5BnHJwn968K1I5+ck= 2024-04-19T23:23:35.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KQZGqIyQMV5BnHJwn968K1I5+ck=', 'Content-Length': '0'} 2024-04-19T23:23:35.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a19c1c6b99c3b8a-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:23:35.100 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:23:35.100 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:23:35.100 DEBUG:boto:Method: GET 2024-04-19T23:23:35.100 DEBUG:boto:Path: /aktlji-10/%26 2024-04-19T23:23:35.100 DEBUG:boto:Data: 2024-04-19T23:23:35.101 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.101 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.101 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.101 DEBUG:boto:Params: {} 2024-04-19T23:23:35.101 DEBUG:boto:Token: None 2024-04-19T23:23:35.101 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%26 2024-04-19T23:23:35.101 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yJvwQe7ABK2gzkRILS7KzI9Wf2o= 2024-04-19T23:23:35.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yJvwQe7ABK2gzkRILS7KzI9Wf2o=', 'Content-Length': '0'} 2024-04-19T23:23:35.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba6307de0da9ac8b-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.146 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:23:35.146 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:23:35.147 DEBUG:boto:Method: GET 2024-04-19T23:23:35.147 DEBUG:boto:Path: /aktlji-10/%26 2024-04-19T23:23:35.147 DEBUG:boto:Data: 2024-04-19T23:23:35.147 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.147 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.147 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.147 DEBUG:boto:Params: {} 2024-04-19T23:23:35.147 DEBUG:boto:Token: None 2024-04-19T23:23:35.147 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%26 2024-04-19T23:23:35.148 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yJvwQe7ABK2gzkRILS7KzI9Wf2o= 2024-04-19T23:23:35.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yJvwQe7ABK2gzkRILS7KzI9Wf2o=', 'Content-Length': '0'} 2024-04-19T23:23:35.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbb0b18a2befff3f-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.150 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.150 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.150 DEBUG:boto:path=/aktlji-10/?prefix=%26 2024-04-19T23:23:35.150 DEBUG:boto:auth_path=/aktlji-10/?prefix=%26 2024-04-19T23:23:35.150 DEBUG:boto:Method: GET 2024-04-19T23:23:35.150 DEBUG:boto:Path: /aktlji-10/?prefix=%26 2024-04-19T23:23:35.151 DEBUG:boto:Data: 2024-04-19T23:23:35.151 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.151 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.151 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.151 DEBUG:boto:Params: {} 2024-04-19T23:23:35.151 DEBUG:boto:Token: None 2024-04-19T23:23:35.151 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.151 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ea24140624c45cf-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.154 DEBUG:boto:b'aktlji-10&1000false&2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.154 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.154 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.154 DEBUG:boto:path=/aktlji-10/?prefix=%26 2024-04-19T23:23:35.155 DEBUG:boto:auth_path=/aktlji-10/?prefix=%26 2024-04-19T23:23:35.155 DEBUG:boto:Method: GET 2024-04-19T23:23:35.155 DEBUG:boto:Path: /aktlji-10/?prefix=%26 2024-04-19T23:23:35.155 DEBUG:boto:Data: 2024-04-19T23:23:35.155 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.155 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.155 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.155 DEBUG:boto:Params: {} 2024-04-19T23:23:35.155 DEBUG:boto:Token: None 2024-04-19T23:23:35.155 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.156 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d7dc2d6e110c43c-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.158 DEBUG:boto:b'aktlji-10&1000false&2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:23:35.159 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:23:35.159 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:23:35.159 DEBUG:boto:path=/aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.159 DEBUG:boto:auth_path=/aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.159 DEBUG:boto:Method: GET 2024-04-19T23:23:35.159 DEBUG:boto:Path: /aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.159 DEBUG:boto:Data: 2024-04-19T23:23:35.159 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.159 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.159 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.160 DEBUG:boto:Params: {} 2024-04-19T23:23:35.160 DEBUG:boto:Token: None 2024-04-19T23:23:35.160 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uct4FDwTXpwPWi7ZLPnFCaUdGC0= 2024-04-19T23:23:35.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uct4FDwTXpwPWi7ZLPnFCaUdGC0=', 'Content-Length': '0'} 2024-04-19T23:23:35.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1ba1fccfe16d1ee-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.202 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:23:35.202 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:23:35.203 DEBUG:boto:path=/aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.203 DEBUG:boto:auth_path=/aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.203 DEBUG:boto:Method: GET 2024-04-19T23:23:35.203 DEBUG:boto:Path: /aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.203 DEBUG:boto:Data: 2024-04-19T23:23:35.203 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.203 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.203 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.203 DEBUG:boto:Params: {} 2024-04-19T23:23:35.203 DEBUG:boto:Token: None 2024-04-19T23:23:35.204 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.204 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uct4FDwTXpwPWi7ZLPnFCaUdGC0= 2024-04-19T23:23:35.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uct4FDwTXpwPWi7ZLPnFCaUdGC0=', 'Content-Length': '0'} 2024-04-19T23:23:35.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e5e3b2bff86061c-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.206 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:23:35.206 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:23:35.206 DEBUG:boto:path=/aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.206 DEBUG:boto:auth_path=/aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.206 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.206 DEBUG:boto:Path: /aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.206 DEBUG:boto:Data: 2024-04-19T23:23:35.207 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.207 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.207 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.207 DEBUG:boto:Params: {} 2024-04-19T23:23:35.207 DEBUG:boto:Token: None 2024-04-19T23:23:35.207 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.207 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:D5ouyUvvAlOT+2NZPeWWWYQ4l0g= 2024-04-19T23:23:35.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:D5ouyUvvAlOT+2NZPeWWWYQ4l0g=', 'Content-Length': '0'} 2024-04-19T23:23:35.209 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbfef6644fcb7493-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.209 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:23:35.209 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:23:35.209 DEBUG:boto:path=/aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.209 DEBUG:boto:auth_path=/aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.210 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.210 DEBUG:boto:Path: /aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.210 DEBUG:boto:Data: 2024-04-19T23:23:35.210 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.210 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.210 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.210 DEBUG:boto:Params: {} 2024-04-19T23:23:35.210 DEBUG:boto:Token: None 2024-04-19T23:23:35.210 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.211 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:D5ouyUvvAlOT+2NZPeWWWYQ4l0g= 2024-04-19T23:23:35.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:D5ouyUvvAlOT+2NZPeWWWYQ4l0g=', 'Content-Length': '0'} 2024-04-19T23:23:35.212 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddad905a650d7d88-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:23:35.213 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:23:35.213 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:23:35.213 DEBUG:boto:path=/aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.213 DEBUG:boto:auth_path=/aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.213 DEBUG:boto:Method: GET 2024-04-19T23:23:35.213 DEBUG:boto:Path: /aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.213 DEBUG:boto:Data: 2024-04-19T23:23:35.213 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.214 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.214 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.214 DEBUG:boto:Params: {} 2024-04-19T23:23:35.214 DEBUG:boto:Token: None 2024-04-19T23:23:35.214 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.214 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uct4FDwTXpwPWi7ZLPnFCaUdGC0= 2024-04-19T23:23:35.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uct4FDwTXpwPWi7ZLPnFCaUdGC0=', 'Content-Length': '0'} 2024-04-19T23:23:35.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000feb364b86c695e39-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.216 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:23:35.217 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:23:35.217 DEBUG:boto:path=/aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.217 DEBUG:boto:auth_path=/aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.217 DEBUG:boto:Method: GET 2024-04-19T23:23:35.217 DEBUG:boto:Path: /aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.217 DEBUG:boto:Data: 2024-04-19T23:23:35.217 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.217 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.217 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.217 DEBUG:boto:Params: {} 2024-04-19T23:23:35.217 DEBUG:boto:Token: None 2024-04-19T23:23:35.218 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%3A?versionId=null 2024-04-19T23:23:35.218 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uct4FDwTXpwPWi7ZLPnFCaUdGC0= 2024-04-19T23:23:35.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uct4FDwTXpwPWi7ZLPnFCaUdGC0=', 'Content-Length': '0'} 2024-04-19T23:23:35.220 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009656eac6ef461846-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.220 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.220 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.220 DEBUG:boto:path=/aktlji-10/?prefix=%3A 2024-04-19T23:23:35.220 DEBUG:boto:auth_path=/aktlji-10/?prefix=%3A 2024-04-19T23:23:35.221 DEBUG:boto:Method: GET 2024-04-19T23:23:35.221 DEBUG:boto:Path: /aktlji-10/?prefix=%3A 2024-04-19T23:23:35.221 DEBUG:boto:Data: 2024-04-19T23:23:35.221 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.221 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.221 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.221 DEBUG:boto:Params: {} 2024-04-19T23:23:35.221 DEBUG:boto:Token: None 2024-04-19T23:23:35.221 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.221 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae1463ecc109a79c-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.224 DEBUG:boto:b'aktlji-10:1000false:2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.224 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.225 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.225 DEBUG:boto:path=/aktlji-10/?prefix=%3A 2024-04-19T23:23:35.225 DEBUG:boto:auth_path=/aktlji-10/?prefix=%3A 2024-04-19T23:23:35.225 DEBUG:boto:Method: GET 2024-04-19T23:23:35.225 DEBUG:boto:Path: /aktlji-10/?prefix=%3A 2024-04-19T23:23:35.225 DEBUG:boto:Data: 2024-04-19T23:23:35.225 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.225 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.225 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.225 DEBUG:boto:Params: {} 2024-04-19T23:23:35.225 DEBUG:boto:Token: None 2024-04-19T23:23:35.226 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.226 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063cbe5a72f7685e1-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.229 DEBUG:boto:b'aktlji-10:1000false:2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.229 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:23:35.229 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:23:35.229 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.229 DEBUG:boto:Path: /aktlji-10/%3A 2024-04-19T23:23:35.229 DEBUG:boto:Data: 2024-04-19T23:23:35.229 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.230 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.230 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.230 DEBUG:boto:Params: {} 2024-04-19T23:23:35.230 DEBUG:boto:Token: None 2024-04-19T23:23:35.231 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%3A 2024-04-19T23:23:35.231 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTh2HuMCj4UclmpCf0jhJfnJi40= 2024-04-19T23:23:35.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTh2HuMCj4UclmpCf0jhJfnJi40=', 'Content-Length': '0'} 2024-04-19T23:23:35.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b791841e86c2d80-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.233 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:23:35.233 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:23:35.233 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.233 DEBUG:boto:Path: /aktlji-10/%3A 2024-04-19T23:23:35.233 DEBUG:boto:Data: 2024-04-19T23:23:35.233 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.233 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.233 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.234 DEBUG:boto:Params: {} 2024-04-19T23:23:35.234 DEBUG:boto:Token: None 2024-04-19T23:23:35.234 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%3A 2024-04-19T23:23:35.234 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTh2HuMCj4UclmpCf0jhJfnJi40= 2024-04-19T23:23:35.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTh2HuMCj4UclmpCf0jhJfnJi40=', 'Content-Length': '0'} 2024-04-19T23:23:35.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3cc566be94cb2c6-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:23:35.236 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:23:35.236 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:23:35.236 DEBUG:boto:Method: GET 2024-04-19T23:23:35.236 DEBUG:boto:Path: /aktlji-10/%3A 2024-04-19T23:23:35.236 DEBUG:boto:Data: 2024-04-19T23:23:35.237 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.237 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.237 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.237 DEBUG:boto:Params: {} 2024-04-19T23:23:35.237 DEBUG:boto:Token: None 2024-04-19T23:23:35.237 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%3A 2024-04-19T23:23:35.237 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DgP+xzIq9T2twAqBLlMExin/f6I= 2024-04-19T23:23:35.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DgP+xzIq9T2twAqBLlMExin/f6I=', 'Content-Length': '0'} 2024-04-19T23:23:35.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d319c80bb6d5607e-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.282 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:23:35.282 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:23:35.283 DEBUG:boto:Method: GET 2024-04-19T23:23:35.283 DEBUG:boto:Path: /aktlji-10/%3A 2024-04-19T23:23:35.283 DEBUG:boto:Data: 2024-04-19T23:23:35.283 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.283 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.283 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.283 DEBUG:boto:Params: {} 2024-04-19T23:23:35.283 DEBUG:boto:Token: None 2024-04-19T23:23:35.283 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/%3A 2024-04-19T23:23:35.284 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DgP+xzIq9T2twAqBLlMExin/f6I= 2024-04-19T23:23:35.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DgP+xzIq9T2twAqBLlMExin/f6I=', 'Content-Length': '0'} 2024-04-19T23:23:35.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000278048e49062ab5d-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.286 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.286 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.286 DEBUG:boto:path=/aktlji-10/?prefix=%3A 2024-04-19T23:23:35.286 DEBUG:boto:auth_path=/aktlji-10/?prefix=%3A 2024-04-19T23:23:35.286 DEBUG:boto:Method: GET 2024-04-19T23:23:35.286 DEBUG:boto:Path: /aktlji-10/?prefix=%3A 2024-04-19T23:23:35.286 DEBUG:boto:Data: 2024-04-19T23:23:35.286 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.287 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.287 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.287 DEBUG:boto:Params: {} 2024-04-19T23:23:35.287 DEBUG:boto:Token: None 2024-04-19T23:23:35.287 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.287 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6c4dab0f0d38902-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.290 DEBUG:boto:b'aktlji-10:1000false:2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.290 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.290 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.290 DEBUG:boto:path=/aktlji-10/?prefix=%3A 2024-04-19T23:23:35.290 DEBUG:boto:auth_path=/aktlji-10/?prefix=%3A 2024-04-19T23:23:35.291 DEBUG:boto:Method: GET 2024-04-19T23:23:35.291 DEBUG:boto:Path: /aktlji-10/?prefix=%3A 2024-04-19T23:23:35.291 DEBUG:boto:Data: 2024-04-19T23:23:35.291 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.291 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.291 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.291 DEBUG:boto:Params: {} 2024-04-19T23:23:35.291 DEBUG:boto:Token: None 2024-04-19T23:23:35.291 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.291 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000250fd162ceb89748-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.294 DEBUG:boto:b'aktlji-10:1000false:2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:23:35.295 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:23:35.295 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:23:35.295 DEBUG:boto:path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.295 DEBUG:boto:auth_path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.295 DEBUG:boto:Method: GET 2024-04-19T23:23:35.295 DEBUG:boto:Path: /aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.295 DEBUG:boto:Data: 2024-04-19T23:23:35.295 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.295 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.295 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.295 DEBUG:boto:Params: {} 2024-04-19T23:23:35.296 DEBUG:boto:Token: None 2024-04-19T23:23:35.296 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.296 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:f9k4JbPNVzQsigQMGq+N3gwF2VY= 2024-04-19T23:23:35.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:f9k4JbPNVzQsigQMGq+N3gwF2VY=', 'Content-Length': '0'} 2024-04-19T23:23:35.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036e25be6b276815e-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.298 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:23:35.299 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:23:35.299 DEBUG:boto:path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.299 DEBUG:boto:auth_path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.299 DEBUG:boto:Method: GET 2024-04-19T23:23:35.299 DEBUG:boto:Path: /aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.299 DEBUG:boto:Data: 2024-04-19T23:23:35.299 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.299 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.299 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.299 DEBUG:boto:Params: {} 2024-04-19T23:23:35.299 DEBUG:boto:Token: None 2024-04-19T23:23:35.300 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.300 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:f9k4JbPNVzQsigQMGq+N3gwF2VY= 2024-04-19T23:23:35.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:f9k4JbPNVzQsigQMGq+N3gwF2VY=', 'Content-Length': '0'} 2024-04-19T23:23:35.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d90c12d7eb47c67-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.302 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:23:35.302 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:23:35.302 DEBUG:boto:path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.302 DEBUG:boto:auth_path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.302 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.302 DEBUG:boto:Path: /aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.302 DEBUG:boto:Data: 2024-04-19T23:23:35.302 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.303 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.303 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.303 DEBUG:boto:Params: {} 2024-04-19T23:23:35.303 DEBUG:boto:Token: None 2024-04-19T23:23:35.303 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.303 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:V2so4Mf9rUbLQnmFzrC/7L4oGuA= 2024-04-19T23:23:35.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:V2so4Mf9rUbLQnmFzrC/7L4oGuA=', 'Content-Length': '0'} 2024-04-19T23:23:35.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006286af42669f1c61-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.305 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:23:35.305 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:23:35.305 DEBUG:boto:path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.306 DEBUG:boto:auth_path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.306 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.306 DEBUG:boto:Path: /aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.306 DEBUG:boto:Data: 2024-04-19T23:23:35.306 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.306 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.306 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.306 DEBUG:boto:Params: {} 2024-04-19T23:23:35.306 DEBUG:boto:Token: None 2024-04-19T23:23:35.306 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.307 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:V2so4Mf9rUbLQnmFzrC/7L4oGuA= 2024-04-19T23:23:35.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:V2so4Mf9rUbLQnmFzrC/7L4oGuA=', 'Content-Length': '0'} 2024-04-19T23:23:35.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bac6ba042ea4df93-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:23:35.309 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:23:35.309 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:23:35.309 DEBUG:boto:path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.309 DEBUG:boto:auth_path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.309 DEBUG:boto:Method: GET 2024-04-19T23:23:35.309 DEBUG:boto:Path: /aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.309 DEBUG:boto:Data: 2024-04-19T23:23:35.309 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.310 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.310 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.310 DEBUG:boto:Params: {} 2024-04-19T23:23:35.310 DEBUG:boto:Token: None 2024-04-19T23:23:35.310 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.310 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:f9k4JbPNVzQsigQMGq+N3gwF2VY= 2024-04-19T23:23:35.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:f9k4JbPNVzQsigQMGq+N3gwF2VY=', 'Content-Length': '0'} 2024-04-19T23:23:35.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b33dbe40683eb84-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.312 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:23:35.313 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:23:35.313 DEBUG:boto:path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.313 DEBUG:boto:auth_path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.313 DEBUG:boto:Method: GET 2024-04-19T23:23:35.313 DEBUG:boto:Path: /aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.313 DEBUG:boto:Data: 2024-04-19T23:23:35.313 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.313 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.313 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.313 DEBUG:boto:Params: {} 2024-04-19T23:23:35.314 DEBUG:boto:Token: None 2024-04-19T23:23:35.314 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/_myobj?versionId=null 2024-04-19T23:23:35.314 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:f9k4JbPNVzQsigQMGq+N3gwF2VY= 2024-04-19T23:23:35.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:f9k4JbPNVzQsigQMGq+N3gwF2VY=', 'Content-Length': '0'} 2024-04-19T23:23:35.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081a4565a91b3962e-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.316 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.316 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.317 DEBUG:boto:path=/aktlji-10/?prefix=_myobj 2024-04-19T23:23:35.317 DEBUG:boto:auth_path=/aktlji-10/?prefix=_myobj 2024-04-19T23:23:35.317 DEBUG:boto:Method: GET 2024-04-19T23:23:35.317 DEBUG:boto:Path: /aktlji-10/?prefix=_myobj 2024-04-19T23:23:35.317 DEBUG:boto:Data: 2024-04-19T23:23:35.318 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.318 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.318 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.318 DEBUG:boto:Params: {} 2024-04-19T23:23:35.318 DEBUG:boto:Token: None 2024-04-19T23:23:35.318 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.318 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032e5e9abdd094739-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.321 DEBUG:boto:b'aktlji-10_myobj1000false_myobj2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.321 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.321 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.321 DEBUG:boto:path=/aktlji-10/?prefix=_myobj 2024-04-19T23:23:35.322 DEBUG:boto:auth_path=/aktlji-10/?prefix=_myobj 2024-04-19T23:23:35.322 DEBUG:boto:Method: GET 2024-04-19T23:23:35.322 DEBUG:boto:Path: /aktlji-10/?prefix=_myobj 2024-04-19T23:23:35.322 DEBUG:boto:Data: 2024-04-19T23:23:35.322 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.322 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.322 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.322 DEBUG:boto:Params: {} 2024-04-19T23:23:35.322 DEBUG:boto:Token: None 2024-04-19T23:23:35.322 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.323 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fa2cf9255495021-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.325 DEBUG:boto:b'aktlji-10_myobj1000false_myobj2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.326 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:23:35.326 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:23:35.326 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.326 DEBUG:boto:Path: /aktlji-10/_myobj 2024-04-19T23:23:35.326 DEBUG:boto:Data: 2024-04-19T23:23:35.326 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.326 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.326 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.327 DEBUG:boto:Params: {} 2024-04-19T23:23:35.327 DEBUG:boto:Token: None 2024-04-19T23:23:35.327 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/_myobj 2024-04-19T23:23:35.327 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:udGmoP8BR4K4SpfdCgCxYpgC0TQ= 2024-04-19T23:23:35.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:udGmoP8BR4K4SpfdCgCxYpgC0TQ=', 'Content-Length': '0'} 2024-04-19T23:23:35.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d473bc5e22dbf794-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.329 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:23:35.329 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:23:35.329 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.329 DEBUG:boto:Path: /aktlji-10/_myobj 2024-04-19T23:23:35.329 DEBUG:boto:Data: 2024-04-19T23:23:35.329 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.330 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.330 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.330 DEBUG:boto:Params: {} 2024-04-19T23:23:35.330 DEBUG:boto:Token: None 2024-04-19T23:23:35.330 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/_myobj 2024-04-19T23:23:35.330 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:udGmoP8BR4K4SpfdCgCxYpgC0TQ= 2024-04-19T23:23:35.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:udGmoP8BR4K4SpfdCgCxYpgC0TQ=', 'Content-Length': '0'} 2024-04-19T23:23:35.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9abb9956b9de0f1-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:23:35.332 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:23:35.333 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:23:35.333 DEBUG:boto:Method: GET 2024-04-19T23:23:35.333 DEBUG:boto:Path: /aktlji-10/_myobj 2024-04-19T23:23:35.333 DEBUG:boto:Data: 2024-04-19T23:23:35.333 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.333 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.333 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.333 DEBUG:boto:Params: {} 2024-04-19T23:23:35.333 DEBUG:boto:Token: None 2024-04-19T23:23:35.333 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/_myobj 2024-04-19T23:23:35.334 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ue81QfbfIWTAosTLUXwFu40ogAk= 2024-04-19T23:23:35.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ue81QfbfIWTAosTLUXwFu40ogAk=', 'Content-Length': '0'} 2024-04-19T23:23:35.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8379f2d368c1c65-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.336 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:23:35.336 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:23:35.336 DEBUG:boto:Method: GET 2024-04-19T23:23:35.336 DEBUG:boto:Path: /aktlji-10/_myobj 2024-04-19T23:23:35.336 DEBUG:boto:Data: 2024-04-19T23:23:35.336 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.337 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.337 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.337 DEBUG:boto:Params: {} 2024-04-19T23:23:35.337 DEBUG:boto:Token: None 2024-04-19T23:23:35.337 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/_myobj 2024-04-19T23:23:35.337 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ue81QfbfIWTAosTLUXwFu40ogAk= 2024-04-19T23:23:35.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ue81QfbfIWTAosTLUXwFu40ogAk=', 'Content-Length': '0'} 2024-04-19T23:23:35.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e72c081c5aeeeee-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.340 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.340 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.340 DEBUG:boto:path=/aktlji-10/?prefix=_myobj 2024-04-19T23:23:35.340 DEBUG:boto:auth_path=/aktlji-10/?prefix=_myobj 2024-04-19T23:23:35.340 DEBUG:boto:Method: GET 2024-04-19T23:23:35.340 DEBUG:boto:Path: /aktlji-10/?prefix=_myobj 2024-04-19T23:23:35.340 DEBUG:boto:Data: 2024-04-19T23:23:35.341 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.341 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.341 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.341 DEBUG:boto:Params: {} 2024-04-19T23:23:35.341 DEBUG:boto:Token: None 2024-04-19T23:23:35.341 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.341 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d99ec36ea5e9bacc-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.344 DEBUG:boto:b'aktlji-10_myobj1000false_myobj2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.344 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.344 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.345 DEBUG:boto:path=/aktlji-10/?prefix=_myobj 2024-04-19T23:23:35.345 DEBUG:boto:auth_path=/aktlji-10/?prefix=_myobj 2024-04-19T23:23:35.345 DEBUG:boto:Method: GET 2024-04-19T23:23:35.345 DEBUG:boto:Path: /aktlji-10/?prefix=_myobj 2024-04-19T23:23:35.345 DEBUG:boto:Data: 2024-04-19T23:23:35.345 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.345 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.345 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.345 DEBUG:boto:Params: {} 2024-04-19T23:23:35.345 DEBUG:boto:Token: None 2024-04-19T23:23:35.345 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.346 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee9fa52aabae44af-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.348 DEBUG:boto:b'aktlji-10_myobj1000false_myobj2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:23:35.349 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:23:35.349 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:23:35.349 DEBUG:boto:path=/aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.349 DEBUG:boto:auth_path=/aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.349 DEBUG:boto:Method: GET 2024-04-19T23:23:35.349 DEBUG:boto:Path: /aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.350 DEBUG:boto:Data: 2024-04-19T23:23:35.350 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.350 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.350 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.350 DEBUG:boto:Params: {} 2024-04-19T23:23:35.350 DEBUG:boto:Token: None 2024-04-19T23:23:35.350 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.350 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QaOATg0GPnu45tbk00FrwpNGxKI= 2024-04-19T23:23:35.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QaOATg0GPnu45tbk00FrwpNGxKI=', 'Content-Length': '0'} 2024-04-19T23:23:35.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079c4eac8f7127e3f-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.394 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:23:35.394 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:23:35.395 DEBUG:boto:path=/aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.395 DEBUG:boto:auth_path=/aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.395 DEBUG:boto:Method: GET 2024-04-19T23:23:35.395 DEBUG:boto:Path: /aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.395 DEBUG:boto:Data: 2024-04-19T23:23:35.395 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.395 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.395 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.395 DEBUG:boto:Params: {} 2024-04-19T23:23:35.395 DEBUG:boto:Token: None 2024-04-19T23:23:35.396 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.396 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QaOATg0GPnu45tbk00FrwpNGxKI= 2024-04-19T23:23:35.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QaOATg0GPnu45tbk00FrwpNGxKI=', 'Content-Length': '0'} 2024-04-19T23:23:35.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c05d5f16e2a955ce-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.398 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:23:35.398 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:23:35.398 DEBUG:boto:path=/aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.398 DEBUG:boto:auth_path=/aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.398 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.398 DEBUG:boto:Path: /aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.399 DEBUG:boto:Data: 2024-04-19T23:23:35.399 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.399 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.399 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.399 DEBUG:boto:Params: {} 2024-04-19T23:23:35.399 DEBUG:boto:Token: None 2024-04-19T23:23:35.399 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.399 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JqjTt4AaRIUFSkw8TkDkKrdnV/0= 2024-04-19T23:23:35.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JqjTt4AaRIUFSkw8TkDkKrdnV/0=', 'Content-Length': '0'} 2024-04-19T23:23:35.401 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a0a9b78b4a09acf-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.401 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:23:35.401 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:23:35.402 DEBUG:boto:path=/aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.402 DEBUG:boto:auth_path=/aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.402 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.402 DEBUG:boto:Path: /aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.402 DEBUG:boto:Data: 2024-04-19T23:23:35.402 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.402 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.402 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.402 DEBUG:boto:Params: {} 2024-04-19T23:23:35.402 DEBUG:boto:Token: None 2024-04-19T23:23:35.403 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.403 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JqjTt4AaRIUFSkw8TkDkKrdnV/0= 2024-04-19T23:23:35.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JqjTt4AaRIUFSkw8TkDkKrdnV/0=', 'Content-Length': '0'} 2024-04-19T23:23:35.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5831a280e4bd587-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:23:35.405 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:23:35.405 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:23:35.405 DEBUG:boto:path=/aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.405 DEBUG:boto:auth_path=/aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.405 DEBUG:boto:Method: GET 2024-04-19T23:23:35.405 DEBUG:boto:Path: /aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.405 DEBUG:boto:Data: 2024-04-19T23:23:35.406 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.406 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.406 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.406 DEBUG:boto:Params: {} 2024-04-19T23:23:35.406 DEBUG:boto:Token: None 2024-04-19T23:23:35.406 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.406 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QaOATg0GPnu45tbk00FrwpNGxKI= 2024-04-19T23:23:35.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QaOATg0GPnu45tbk00FrwpNGxKI=', 'Content-Length': '0'} 2024-04-19T23:23:35.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fc3d39dc3e5577e-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.408 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:23:35.408 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:23:35.409 DEBUG:boto:path=/aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.409 DEBUG:boto:auth_path=/aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.409 DEBUG:boto:Method: GET 2024-04-19T23:23:35.409 DEBUG:boto:Path: /aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.409 DEBUG:boto:Data: 2024-04-19T23:23:35.409 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.409 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.409 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.410 DEBUG:boto:Params: {} 2024-04-19T23:23:35.410 DEBUG:boto:Token: None 2024-04-19T23:23:35.410 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/myobj?versionId=null 2024-04-19T23:23:35.410 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QaOATg0GPnu45tbk00FrwpNGxKI= 2024-04-19T23:23:35.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QaOATg0GPnu45tbk00FrwpNGxKI=', 'Content-Length': '0'} 2024-04-19T23:23:35.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d58770c207ee366-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.412 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.412 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.413 DEBUG:boto:path=/aktlji-10/?prefix=myobj 2024-04-19T23:23:35.413 DEBUG:boto:auth_path=/aktlji-10/?prefix=myobj 2024-04-19T23:23:35.413 DEBUG:boto:Method: GET 2024-04-19T23:23:35.413 DEBUG:boto:Path: /aktlji-10/?prefix=myobj 2024-04-19T23:23:35.413 DEBUG:boto:Data: 2024-04-19T23:23:35.413 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.413 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.413 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.413 DEBUG:boto:Params: {} 2024-04-19T23:23:35.413 DEBUG:boto:Token: None 2024-04-19T23:23:35.414 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.414 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fba7e427284bfe30-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.416 DEBUG:boto:b'aktlji-10myobj1000falsemyobj2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.417 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.417 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.417 DEBUG:boto:path=/aktlji-10/?prefix=myobj 2024-04-19T23:23:35.417 DEBUG:boto:auth_path=/aktlji-10/?prefix=myobj 2024-04-19T23:23:35.418 DEBUG:boto:Method: GET 2024-04-19T23:23:35.418 DEBUG:boto:Path: /aktlji-10/?prefix=myobj 2024-04-19T23:23:35.418 DEBUG:boto:Data: 2024-04-19T23:23:35.418 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.418 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.418 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.418 DEBUG:boto:Params: {} 2024-04-19T23:23:35.418 DEBUG:boto:Token: None 2024-04-19T23:23:35.418 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.419 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd5a1232d35c2b28-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.421 DEBUG:boto:b'aktlji-10myobj1000falsemyobj2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.421 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:23:35.422 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:23:35.422 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.422 DEBUG:boto:Path: /aktlji-10/myobj 2024-04-19T23:23:35.422 DEBUG:boto:Data: 2024-04-19T23:23:35.422 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.422 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.422 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.422 DEBUG:boto:Params: {} 2024-04-19T23:23:35.423 DEBUG:boto:Token: None 2024-04-19T23:23:35.423 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/myobj 2024-04-19T23:23:35.423 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xgcNFmfExlhopf19Jbxja3osB9Q= 2024-04-19T23:23:35.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xgcNFmfExlhopf19Jbxja3osB9Q=', 'Content-Length': '0'} 2024-04-19T23:23:35.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a1064d14c567092-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.425 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:23:35.425 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:23:35.425 DEBUG:boto:Method: HEAD 2024-04-19T23:23:35.425 DEBUG:boto:Path: /aktlji-10/myobj 2024-04-19T23:23:35.425 DEBUG:boto:Data: 2024-04-19T23:23:35.425 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.425 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.426 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.426 DEBUG:boto:Params: {} 2024-04-19T23:23:35.426 DEBUG:boto:Token: None 2024-04-19T23:23:35.426 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/myobj 2024-04-19T23:23:35.426 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xgcNFmfExlhopf19Jbxja3osB9Q= 2024-04-19T23:23:35.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xgcNFmfExlhopf19Jbxja3osB9Q=', 'Content-Length': '0'} 2024-04-19T23:23:35.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ba5e9c004938191-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:23:35.428 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:23:35.429 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:23:35.429 DEBUG:boto:Method: GET 2024-04-19T23:23:35.429 DEBUG:boto:Path: /aktlji-10/myobj 2024-04-19T23:23:35.429 DEBUG:boto:Data: 2024-04-19T23:23:35.429 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.429 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.429 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.429 DEBUG:boto:Params: {} 2024-04-19T23:23:35.429 DEBUG:boto:Token: None 2024-04-19T23:23:35.430 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/myobj 2024-04-19T23:23:35.430 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zxpmxkba9ObpqBE5Ffj1WNEFTNk= 2024-04-19T23:23:35.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zxpmxkba9ObpqBE5Ffj1WNEFTNk=', 'Content-Length': '0'} 2024-04-19T23:23:35.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000098a3f0e803af120-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.432 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:23:35.432 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:23:35.432 DEBUG:boto:Method: GET 2024-04-19T23:23:35.432 DEBUG:boto:Path: /aktlji-10/myobj 2024-04-19T23:23:35.432 DEBUG:boto:Data: 2024-04-19T23:23:35.432 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.432 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.433 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.433 DEBUG:boto:Params: {} 2024-04-19T23:23:35.433 DEBUG:boto:Token: None 2024-04-19T23:23:35.433 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/myobj 2024-04-19T23:23:35.433 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zxpmxkba9ObpqBE5Ffj1WNEFTNk= 2024-04-19T23:23:35.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zxpmxkba9ObpqBE5Ffj1WNEFTNk=', 'Content-Length': '0'} 2024-04-19T23:23:35.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdd796c6b27d4109-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.435 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.435 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.436 DEBUG:boto:path=/aktlji-10/?prefix=myobj 2024-04-19T23:23:35.436 DEBUG:boto:auth_path=/aktlji-10/?prefix=myobj 2024-04-19T23:23:35.436 DEBUG:boto:Method: GET 2024-04-19T23:23:35.436 DEBUG:boto:Path: /aktlji-10/?prefix=myobj 2024-04-19T23:23:35.436 DEBUG:boto:Data: 2024-04-19T23:23:35.436 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.436 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:23:35.436 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.436 DEBUG:boto:Params: {} 2024-04-19T23:23:35.436 DEBUG:boto:Token: None 2024-04-19T23:23:35.437 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.437 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004e2422db144b04f-006622fcf7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.439 DEBUG:boto:b'aktlji-10myobj1000falsemyobj2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.439 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:23:35.440 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:23:35.440 DEBUG:boto:path=/aktlji-10/?prefix=myobj 2024-04-19T23:23:35.440 DEBUG:boto:auth_path=/aktlji-10/?prefix=myobj 2024-04-19T23:23:35.440 DEBUG:boto:Method: GET 2024-04-19T23:23:35.440 DEBUG:boto:Path: /aktlji-10/?prefix=myobj 2024-04-19T23:23:35.440 DEBUG:boto:Data: 2024-04-19T23:23:35.440 DEBUG:boto:Headers: {} 2024-04-19T23:23:35.440 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:23:35.440 DEBUG:boto:Port: 8000 2024-04-19T23:23:35.440 DEBUG:boto:Params: {} 2024-04-19T23:23:35.441 DEBUG:boto:Token: None 2024-04-19T23:23:35.441 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:23:35 GMT /aktlji-10/ 2024-04-19T23:23:35.441 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk= 2024-04-19T23:23:35.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:23:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QHQqbURIo9vkLk9jyd088xLOlpk=', 'Content-Length': '0'} 2024-04-19T23:23:35.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b867d702712ed08-006622fcf7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:23:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:23:35.444 DEBUG:boto:b'aktlji-10myobj1000falsemyobj2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:23:35.444 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-10 zones={test-zone2, test-zone1} 2024-04-19T23:23:35.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:23:35.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:23:35.445 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:23:35.513 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:23:35.513 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:23:35.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:35.557+0000 7f525f36d780 1 waiting to reach incremental sync.. 2024-04-19T23:23:40.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:40.558+0000 7f525f36d780 1 waiting to reach incremental sync.. 2024-04-19T23:23:45.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:45.559+0000 7f525f36d780 1 waiting to reach incremental sync.. 2024-04-19T23:23:50.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:50.560+0000 7f525f36d780 1 waiting to reach incremental sync.. 2024-04-19T23:23:55.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:23:55.561+0000 7f525f36d780 1 waiting to reach incremental sync.. 2024-04-19T23:24:00.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:00.563+0000 7f525f36d780 1 bucket sync caught up with source: 2024-04-19T23:24:00.558 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.4.6, , 00000000001.6.6, , , , 00000000001.7.6, 00000000001.8.6, , ] 2024-04-19T23:24:00.559 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000001.4.6, , 00000000001.6.6, , , , 00000000001.7.6, 00000000001.8.6, , ] 2024-04-19T23:24:00.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:00.563+0000 7f525f36d780 0 bucket checkpoint complete 2024-04-19T23:24:00.579 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-10 zones={test-zone3, test-zone1} 2024-04-19T23:24:00.579 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.579 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.580 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.580 DEBUG:boto:Path: /aktlji-10/ 2024-04-19T23:24:00.580 DEBUG:boto:Data: 2024-04-19T23:24:00.581 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.581 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.581 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.581 DEBUG:boto:Params: {} 2024-04-19T23:24:00.582 DEBUG:boto:Token: None 2024-04-19T23:24:00.582 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.582 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KaX24moh8SFQhbBUsGQqHVkAiFY= 2024-04-19T23:24:00.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KaX24moh8SFQhbBUsGQqHVkAiFY=', 'Content-Length': '0'} 2024-04-19T23:24:00.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003eebfb0871acadcb-006622fd10-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.586 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.586 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.586 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.586 DEBUG:boto:Path: /aktlji-10/ 2024-04-19T23:24:00.586 DEBUG:boto:Data: 2024-04-19T23:24:00.586 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.586 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.586 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.587 DEBUG:boto:Params: {} 2024-04-19T23:24:00.587 DEBUG:boto:Token: None 2024-04-19T23:24:00.587 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.587 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KaX24moh8SFQhbBUsGQqHVkAiFY= 2024-04-19T23:24:00.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KaX24moh8SFQhbBUsGQqHVkAiFY=', 'Content-Length': '0'} 2024-04-19T23:24:00.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb2d8c65ca064934-006622fd10-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.590 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:24:00.590 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.590 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.590 DEBUG:boto:path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:24:00.590 DEBUG:boto:auth_path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:24:00.590 DEBUG:boto:Method: GET 2024-04-19T23:24:00.590 DEBUG:boto:Path: /aktlji-10/?versions&max-keys=999 2024-04-19T23:24:00.590 DEBUG:boto:Data: 2024-04-19T23:24:00.590 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.590 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.591 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.591 DEBUG:boto:Params: {} 2024-04-19T23:24:00.591 DEBUG:boto:Token: None 2024-04-19T23:24:00.591 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/?versions 2024-04-19T23:24:00.591 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EUbOilWMtnv3ppLn7KoCwtGL3RU= 2024-04-19T23:24:00.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EUbOilWMtnv3ppLn7KoCwtGL3RU=', 'Content-Length': '0'} 2024-04-19T23:24:00.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e9752fa47369a20-006622fd10-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.594 DEBUG:boto:b'aktlji-10999false&nulltrue2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.594 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-19T23:24:00.594 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-19T23:24:00.594 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-19T23:24:00.594 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:00.595 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:24:00.595 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.595 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.595 DEBUG:boto:path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:24:00.595 DEBUG:boto:auth_path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:24:00.595 DEBUG:boto:Method: GET 2024-04-19T23:24:00.595 DEBUG:boto:Path: /aktlji-10/?versions&max-keys=999 2024-04-19T23:24:00.595 DEBUG:boto:Data: 2024-04-19T23:24:00.595 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.595 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.596 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.596 DEBUG:boto:Params: {} 2024-04-19T23:24:00.596 DEBUG:boto:Token: None 2024-04-19T23:24:00.596 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/?versions 2024-04-19T23:24:00.596 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EUbOilWMtnv3ppLn7KoCwtGL3RU= 2024-04-19T23:24:00.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EUbOilWMtnv3ppLn7KoCwtGL3RU=', 'Content-Length': '0'} 2024-04-19T23:24:00.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aa6d1bfd18ea0f5-006622fd10-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.599 DEBUG:boto:b'aktlji-10999false&nulltrue2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.600 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-19T23:24:00.600 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-19T23:24:00.600 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-19T23:24:00.600 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:00.600 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.601 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.601 DEBUG:boto:path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:24:00.601 DEBUG:boto:auth_path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:24:00.601 DEBUG:boto:Method: GET 2024-04-19T23:24:00.602 DEBUG:boto:Path: /aktlji-10/?versions&max-keys=999 2024-04-19T23:24:00.602 DEBUG:boto:Data: 2024-04-19T23:24:00.602 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.602 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.602 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.602 DEBUG:boto:Params: {} 2024-04-19T23:24:00.603 DEBUG:boto:Token: None 2024-04-19T23:24:00.603 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/?versions 2024-04-19T23:24:00.603 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EUbOilWMtnv3ppLn7KoCwtGL3RU= 2024-04-19T23:24:00.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EUbOilWMtnv3ppLn7KoCwtGL3RU=', 'Content-Length': '0'} 2024-04-19T23:24:00.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d7e4dffe887b82-006622fd10-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.606 DEBUG:boto:b'aktlji-10999false&nulltrue2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.607 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.608 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.608 DEBUG:boto:path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:24:00.608 DEBUG:boto:auth_path=/aktlji-10/?versions&max-keys=999 2024-04-19T23:24:00.608 DEBUG:boto:Method: GET 2024-04-19T23:24:00.608 DEBUG:boto:Path: /aktlji-10/?versions&max-keys=999 2024-04-19T23:24:00.608 DEBUG:boto:Data: 2024-04-19T23:24:00.608 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.609 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.609 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.609 DEBUG:boto:Params: {} 2024-04-19T23:24:00.609 DEBUG:boto:Token: None 2024-04-19T23:24:00.609 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/?versions 2024-04-19T23:24:00.610 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EUbOilWMtnv3ppLn7KoCwtGL3RU= 2024-04-19T23:24:00.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EUbOilWMtnv3ppLn7KoCwtGL3RU=', 'Content-Length': '0'} 2024-04-19T23:24:00.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023926f3d3741066a-006622fd10-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.613 DEBUG:boto:b'aktlji-10999false&nulltrue2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:00.614 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:24:00.614 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:24:00.614 DEBUG:boto:path=/aktlji-10/%26?versionId=null 2024-04-19T23:24:00.615 DEBUG:boto:auth_path=/aktlji-10/%26?versionId=null 2024-04-19T23:24:00.615 DEBUG:boto:Method: GET 2024-04-19T23:24:00.615 DEBUG:boto:Path: /aktlji-10/%26?versionId=null 2024-04-19T23:24:00.615 DEBUG:boto:Data: 2024-04-19T23:24:00.615 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.616 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.616 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.616 DEBUG:boto:Params: {} 2024-04-19T23:24:00.616 DEBUG:boto:Token: None 2024-04-19T23:24:00.616 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%26?versionId=null 2024-04-19T23:24:00.617 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BnCmaQ2oOg9Ed5IJCzwNlYBGsPg= 2024-04-19T23:24:00.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BnCmaQ2oOg9Ed5IJCzwNlYBGsPg=', 'Content-Length': '0'} 2024-04-19T23:24:00.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002f65a86eb894310-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.620 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:24:00.620 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:24:00.620 DEBUG:boto:path=/aktlji-10/%26?versionId=null 2024-04-19T23:24:00.620 DEBUG:boto:auth_path=/aktlji-10/%26?versionId=null 2024-04-19T23:24:00.620 DEBUG:boto:Method: GET 2024-04-19T23:24:00.621 DEBUG:boto:Path: /aktlji-10/%26?versionId=null 2024-04-19T23:24:00.621 DEBUG:boto:Data: 2024-04-19T23:24:00.621 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.621 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.621 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.622 DEBUG:boto:Params: {} 2024-04-19T23:24:00.622 DEBUG:boto:Token: None 2024-04-19T23:24:00.622 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%26?versionId=null 2024-04-19T23:24:00.622 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BnCmaQ2oOg9Ed5IJCzwNlYBGsPg= 2024-04-19T23:24:00.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BnCmaQ2oOg9Ed5IJCzwNlYBGsPg=', 'Content-Length': '0'} 2024-04-19T23:24:00.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bacad31c48ab6887-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.666 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:24:00.667 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:24:00.667 DEBUG:boto:path=/aktlji-10/%26?versionId=null 2024-04-19T23:24:00.667 DEBUG:boto:auth_path=/aktlji-10/%26?versionId=null 2024-04-19T23:24:00.667 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.667 DEBUG:boto:Path: /aktlji-10/%26?versionId=null 2024-04-19T23:24:00.667 DEBUG:boto:Data: 2024-04-19T23:24:00.667 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.667 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.667 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.667 DEBUG:boto:Params: {} 2024-04-19T23:24:00.668 DEBUG:boto:Token: None 2024-04-19T23:24:00.668 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%26?versionId=null 2024-04-19T23:24:00.668 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wEWZ3yfcL6YDf2nUT0LatnE8to4= 2024-04-19T23:24:00.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wEWZ3yfcL6YDf2nUT0LatnE8to4=', 'Content-Length': '0'} 2024-04-19T23:24:00.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9183557644dfcd7-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.670 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:24:00.670 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:24:00.670 DEBUG:boto:path=/aktlji-10/%26?versionId=null 2024-04-19T23:24:00.670 DEBUG:boto:auth_path=/aktlji-10/%26?versionId=null 2024-04-19T23:24:00.670 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.671 DEBUG:boto:Path: /aktlji-10/%26?versionId=null 2024-04-19T23:24:00.671 DEBUG:boto:Data: 2024-04-19T23:24:00.671 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.671 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.671 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.671 DEBUG:boto:Params: {} 2024-04-19T23:24:00.671 DEBUG:boto:Token: None 2024-04-19T23:24:00.671 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%26?versionId=null 2024-04-19T23:24:00.671 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wEWZ3yfcL6YDf2nUT0LatnE8to4= 2024-04-19T23:24:00.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wEWZ3yfcL6YDf2nUT0LatnE8to4=', 'Content-Length': '0'} 2024-04-19T23:24:00.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c6a97fb2296b488-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:00.674 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:24:00.674 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:24:00.674 DEBUG:boto:path=/aktlji-10/%26?versionId=null 2024-04-19T23:24:00.674 DEBUG:boto:auth_path=/aktlji-10/%26?versionId=null 2024-04-19T23:24:00.674 DEBUG:boto:Method: GET 2024-04-19T23:24:00.674 DEBUG:boto:Path: /aktlji-10/%26?versionId=null 2024-04-19T23:24:00.674 DEBUG:boto:Data: 2024-04-19T23:24:00.674 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.674 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.675 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.675 DEBUG:boto:Params: {} 2024-04-19T23:24:00.675 DEBUG:boto:Token: None 2024-04-19T23:24:00.675 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%26?versionId=null 2024-04-19T23:24:00.675 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BnCmaQ2oOg9Ed5IJCzwNlYBGsPg= 2024-04-19T23:24:00.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BnCmaQ2oOg9Ed5IJCzwNlYBGsPg=', 'Content-Length': '0'} 2024-04-19T23:24:00.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e57e813a12b5c64d-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.677 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:24:00.677 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:24:00.677 DEBUG:boto:path=/aktlji-10/%26?versionId=null 2024-04-19T23:24:00.678 DEBUG:boto:auth_path=/aktlji-10/%26?versionId=null 2024-04-19T23:24:00.678 DEBUG:boto:Method: GET 2024-04-19T23:24:00.678 DEBUG:boto:Path: /aktlji-10/%26?versionId=null 2024-04-19T23:24:00.678 DEBUG:boto:Data: 2024-04-19T23:24:00.678 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.678 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.678 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.678 DEBUG:boto:Params: {} 2024-04-19T23:24:00.679 DEBUG:boto:Token: None 2024-04-19T23:24:00.679 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%26?versionId=null 2024-04-19T23:24:00.679 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BnCmaQ2oOg9Ed5IJCzwNlYBGsPg= 2024-04-19T23:24:00.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BnCmaQ2oOg9Ed5IJCzwNlYBGsPg=', 'Content-Length': '0'} 2024-04-19T23:24:00.680 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003e68afc19c3667e-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.681 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.681 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.681 DEBUG:boto:path=/aktlji-10/?prefix=%26 2024-04-19T23:24:00.681 DEBUG:boto:auth_path=/aktlji-10/?prefix=%26 2024-04-19T23:24:00.681 DEBUG:boto:Method: GET 2024-04-19T23:24:00.681 DEBUG:boto:Path: /aktlji-10/?prefix=%26 2024-04-19T23:24:00.681 DEBUG:boto:Data: 2024-04-19T23:24:00.681 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.682 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.682 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.682 DEBUG:boto:Params: {} 2024-04-19T23:24:00.682 DEBUG:boto:Token: None 2024-04-19T23:24:00.682 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.682 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w= 2024-04-19T23:24:00.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w=', 'Content-Length': '0'} 2024-04-19T23:24:00.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026fd47c9e0f38808-006622fd10-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.685 DEBUG:boto:b'aktlji-10&1000false&2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.685 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.686 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.686 DEBUG:boto:path=/aktlji-10/?prefix=%26 2024-04-19T23:24:00.686 DEBUG:boto:auth_path=/aktlji-10/?prefix=%26 2024-04-19T23:24:00.686 DEBUG:boto:Method: GET 2024-04-19T23:24:00.686 DEBUG:boto:Path: /aktlji-10/?prefix=%26 2024-04-19T23:24:00.686 DEBUG:boto:Data: 2024-04-19T23:24:00.686 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.686 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.686 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.686 DEBUG:boto:Params: {} 2024-04-19T23:24:00.686 DEBUG:boto:Token: None 2024-04-19T23:24:00.687 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.687 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w= 2024-04-19T23:24:00.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w=', 'Content-Length': '0'} 2024-04-19T23:24:00.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3299d59158fd248-006622fd10-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.690 DEBUG:boto:b'aktlji-10&1000false&2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.690 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:24:00.690 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:24:00.690 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.690 DEBUG:boto:Path: /aktlji-10/%26 2024-04-19T23:24:00.690 DEBUG:boto:Data: 2024-04-19T23:24:00.691 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.691 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.691 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.691 DEBUG:boto:Params: {} 2024-04-19T23:24:00.691 DEBUG:boto:Token: None 2024-04-19T23:24:00.691 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%26 2024-04-19T23:24:00.691 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ABsj4MW3M8KpXrBi5qWGcCH2d4A= 2024-04-19T23:24:00.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ABsj4MW3M8KpXrBi5qWGcCH2d4A=', 'Content-Length': '0'} 2024-04-19T23:24:00.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e684c48012f20b96-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.693 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:24:00.693 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:24:00.693 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.693 DEBUG:boto:Path: /aktlji-10/%26 2024-04-19T23:24:00.694 DEBUG:boto:Data: 2024-04-19T23:24:00.694 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.694 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.694 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.694 DEBUG:boto:Params: {} 2024-04-19T23:24:00.694 DEBUG:boto:Token: None 2024-04-19T23:24:00.694 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%26 2024-04-19T23:24:00.694 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ABsj4MW3M8KpXrBi5qWGcCH2d4A= 2024-04-19T23:24:00.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ABsj4MW3M8KpXrBi5qWGcCH2d4A=', 'Content-Length': '0'} 2024-04-19T23:24:00.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca2e43369dc7b120-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:00.698 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:24:00.698 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:24:00.698 DEBUG:boto:Method: GET 2024-04-19T23:24:00.698 DEBUG:boto:Path: /aktlji-10/%26 2024-04-19T23:24:00.698 DEBUG:boto:Data: 2024-04-19T23:24:00.698 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.698 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.699 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.699 DEBUG:boto:Params: {} 2024-04-19T23:24:00.699 DEBUG:boto:Token: None 2024-04-19T23:24:00.699 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%26 2024-04-19T23:24:00.699 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:B5JqF2aQL3L/7FeTcOk+WyX0m/0= 2024-04-19T23:24:00.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B5JqF2aQL3L/7FeTcOk+WyX0m/0=', 'Content-Length': '0'} 2024-04-19T23:24:00.701 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b28fcbbd32d55ffa-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.701 DEBUG:boto:path=/aktlji-10/%26 2024-04-19T23:24:00.701 DEBUG:boto:auth_path=/aktlji-10/%26 2024-04-19T23:24:00.701 DEBUG:boto:Method: GET 2024-04-19T23:24:00.701 DEBUG:boto:Path: /aktlji-10/%26 2024-04-19T23:24:00.702 DEBUG:boto:Data: 2024-04-19T23:24:00.702 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.702 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.702 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.702 DEBUG:boto:Params: {} 2024-04-19T23:24:00.702 DEBUG:boto:Token: None 2024-04-19T23:24:00.702 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%26 2024-04-19T23:24:00.702 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:B5JqF2aQL3L/7FeTcOk+WyX0m/0= 2024-04-19T23:24:00.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B5JqF2aQL3L/7FeTcOk+WyX0m/0=', 'Content-Length': '0'} 2024-04-19T23:24:00.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000020ce4e830dd26d0-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.704 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.704 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.705 DEBUG:boto:path=/aktlji-10/?prefix=%26 2024-04-19T23:24:00.705 DEBUG:boto:auth_path=/aktlji-10/?prefix=%26 2024-04-19T23:24:00.705 DEBUG:boto:Method: GET 2024-04-19T23:24:00.705 DEBUG:boto:Path: /aktlji-10/?prefix=%26 2024-04-19T23:24:00.705 DEBUG:boto:Data: 2024-04-19T23:24:00.705 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.705 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.705 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.705 DEBUG:boto:Params: {} 2024-04-19T23:24:00.705 DEBUG:boto:Token: None 2024-04-19T23:24:00.705 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.706 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w= 2024-04-19T23:24:00.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w=', 'Content-Length': '0'} 2024-04-19T23:24:00.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4abe0f6cdd084f9-006622fd10-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.708 DEBUG:boto:b'aktlji-10&1000false&2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.709 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.709 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.709 DEBUG:boto:path=/aktlji-10/?prefix=%26 2024-04-19T23:24:00.709 DEBUG:boto:auth_path=/aktlji-10/?prefix=%26 2024-04-19T23:24:00.709 DEBUG:boto:Method: GET 2024-04-19T23:24:00.709 DEBUG:boto:Path: /aktlji-10/?prefix=%26 2024-04-19T23:24:00.709 DEBUG:boto:Data: 2024-04-19T23:24:00.709 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.709 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.709 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.709 DEBUG:boto:Params: {} 2024-04-19T23:24:00.710 DEBUG:boto:Token: None 2024-04-19T23:24:00.710 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.710 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w= 2024-04-19T23:24:00.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w=', 'Content-Length': '0'} 2024-04-19T23:24:00.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003badfd39c8e0a147-006622fd10-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.713 DEBUG:boto:b'aktlji-10&1000false&2024-04-19T23:23:14.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:00.713 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:24:00.714 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:24:00.714 DEBUG:boto:path=/aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.714 DEBUG:boto:auth_path=/aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.714 DEBUG:boto:Method: GET 2024-04-19T23:24:00.714 DEBUG:boto:Path: /aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.714 DEBUG:boto:Data: 2024-04-19T23:24:00.714 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.714 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.714 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.714 DEBUG:boto:Params: {} 2024-04-19T23:24:00.715 DEBUG:boto:Token: None 2024-04-19T23:24:00.715 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.715 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ySakBZTeKFh7BSGMkbvVXzvxqbw= 2024-04-19T23:24:00.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ySakBZTeKFh7BSGMkbvVXzvxqbw=', 'Content-Length': '0'} 2024-04-19T23:24:00.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6715da0bfb24276-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.717 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:24:00.717 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:24:00.717 DEBUG:boto:path=/aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.717 DEBUG:boto:auth_path=/aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.717 DEBUG:boto:Method: GET 2024-04-19T23:24:00.717 DEBUG:boto:Path: /aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.718 DEBUG:boto:Data: 2024-04-19T23:24:00.718 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.718 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.718 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.718 DEBUG:boto:Params: {} 2024-04-19T23:24:00.718 DEBUG:boto:Token: None 2024-04-19T23:24:00.718 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.718 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ySakBZTeKFh7BSGMkbvVXzvxqbw= 2024-04-19T23:24:00.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ySakBZTeKFh7BSGMkbvVXzvxqbw=', 'Content-Length': '0'} 2024-04-19T23:24:00.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000236b16aa898e7469-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.720 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:24:00.721 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:24:00.721 DEBUG:boto:path=/aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.721 DEBUG:boto:auth_path=/aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.721 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.721 DEBUG:boto:Path: /aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.721 DEBUG:boto:Data: 2024-04-19T23:24:00.721 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.721 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.721 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.721 DEBUG:boto:Params: {} 2024-04-19T23:24:00.721 DEBUG:boto:Token: None 2024-04-19T23:24:00.722 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.722 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:U6zqU+1nkNV5iPhAA4HbELw7kyI= 2024-04-19T23:24:00.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:U6zqU+1nkNV5iPhAA4HbELw7kyI=', 'Content-Length': '0'} 2024-04-19T23:24:00.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086d4358790be2582-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.724 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:24:00.724 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:24:00.724 DEBUG:boto:path=/aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.724 DEBUG:boto:auth_path=/aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.724 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.724 DEBUG:boto:Path: /aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.725 DEBUG:boto:Data: 2024-04-19T23:24:00.725 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.725 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.725 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.725 DEBUG:boto:Params: {} 2024-04-19T23:24:00.725 DEBUG:boto:Token: None 2024-04-19T23:24:00.725 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.725 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:U6zqU+1nkNV5iPhAA4HbELw7kyI= 2024-04-19T23:24:00.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:U6zqU+1nkNV5iPhAA4HbELw7kyI=', 'Content-Length': '0'} 2024-04-19T23:24:00.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040550b6d9d6f9ebc-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:00.727 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:24:00.728 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:24:00.728 DEBUG:boto:path=/aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.728 DEBUG:boto:auth_path=/aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.728 DEBUG:boto:Method: GET 2024-04-19T23:24:00.728 DEBUG:boto:Path: /aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.728 DEBUG:boto:Data: 2024-04-19T23:24:00.728 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.728 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.728 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.728 DEBUG:boto:Params: {} 2024-04-19T23:24:00.728 DEBUG:boto:Token: None 2024-04-19T23:24:00.729 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.729 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ySakBZTeKFh7BSGMkbvVXzvxqbw= 2024-04-19T23:24:00.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ySakBZTeKFh7BSGMkbvVXzvxqbw=', 'Content-Length': '0'} 2024-04-19T23:24:00.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058dac394a15329cf-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.731 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:24:00.731 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:24:00.731 DEBUG:boto:path=/aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.731 DEBUG:boto:auth_path=/aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.731 DEBUG:boto:Method: GET 2024-04-19T23:24:00.731 DEBUG:boto:Path: /aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.731 DEBUG:boto:Data: 2024-04-19T23:24:00.731 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.732 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.732 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.732 DEBUG:boto:Params: {} 2024-04-19T23:24:00.732 DEBUG:boto:Token: None 2024-04-19T23:24:00.732 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%3A?versionId=null 2024-04-19T23:24:00.732 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ySakBZTeKFh7BSGMkbvVXzvxqbw= 2024-04-19T23:24:00.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ySakBZTeKFh7BSGMkbvVXzvxqbw=', 'Content-Length': '0'} 2024-04-19T23:24:00.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011bd0c425debe691-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.734 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.734 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.735 DEBUG:boto:path=/aktlji-10/?prefix=%3A 2024-04-19T23:24:00.735 DEBUG:boto:auth_path=/aktlji-10/?prefix=%3A 2024-04-19T23:24:00.735 DEBUG:boto:Method: GET 2024-04-19T23:24:00.735 DEBUG:boto:Path: /aktlji-10/?prefix=%3A 2024-04-19T23:24:00.735 DEBUG:boto:Data: 2024-04-19T23:24:00.735 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.735 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.735 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.735 DEBUG:boto:Params: {} 2024-04-19T23:24:00.735 DEBUG:boto:Token: None 2024-04-19T23:24:00.736 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.736 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w= 2024-04-19T23:24:00.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w=', 'Content-Length': '0'} 2024-04-19T23:24:00.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004db816e82af0ad60-006622fd10-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.738 DEBUG:boto:b'aktlji-10:1000false:2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.739 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.739 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.739 DEBUG:boto:path=/aktlji-10/?prefix=%3A 2024-04-19T23:24:00.739 DEBUG:boto:auth_path=/aktlji-10/?prefix=%3A 2024-04-19T23:24:00.739 DEBUG:boto:Method: GET 2024-04-19T23:24:00.739 DEBUG:boto:Path: /aktlji-10/?prefix=%3A 2024-04-19T23:24:00.739 DEBUG:boto:Data: 2024-04-19T23:24:00.739 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.739 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.740 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.740 DEBUG:boto:Params: {} 2024-04-19T23:24:00.740 DEBUG:boto:Token: None 2024-04-19T23:24:00.740 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.740 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w= 2024-04-19T23:24:00.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w=', 'Content-Length': '0'} 2024-04-19T23:24:00.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046e5c42991cd3fd0-006622fd10-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.743 DEBUG:boto:b'aktlji-10:1000false:2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.743 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:24:00.743 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:24:00.743 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.743 DEBUG:boto:Path: /aktlji-10/%3A 2024-04-19T23:24:00.744 DEBUG:boto:Data: 2024-04-19T23:24:00.744 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.744 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.744 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.744 DEBUG:boto:Params: {} 2024-04-19T23:24:00.744 DEBUG:boto:Token: None 2024-04-19T23:24:00.744 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%3A 2024-04-19T23:24:00.744 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UvfOBKpDrqiI3GSVgLfkGcqkvdg= 2024-04-19T23:24:00.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UvfOBKpDrqiI3GSVgLfkGcqkvdg=', 'Content-Length': '0'} 2024-04-19T23:24:00.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008944aeed53602fd9-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.746 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:24:00.746 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:24:00.746 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.747 DEBUG:boto:Path: /aktlji-10/%3A 2024-04-19T23:24:00.747 DEBUG:boto:Data: 2024-04-19T23:24:00.747 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.747 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.747 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.747 DEBUG:boto:Params: {} 2024-04-19T23:24:00.747 DEBUG:boto:Token: None 2024-04-19T23:24:00.747 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%3A 2024-04-19T23:24:00.747 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UvfOBKpDrqiI3GSVgLfkGcqkvdg= 2024-04-19T23:24:00.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UvfOBKpDrqiI3GSVgLfkGcqkvdg=', 'Content-Length': '0'} 2024-04-19T23:24:00.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d04221e88a7b2cd0-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:00.749 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:24:00.749 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:24:00.750 DEBUG:boto:Method: GET 2024-04-19T23:24:00.750 DEBUG:boto:Path: /aktlji-10/%3A 2024-04-19T23:24:00.750 DEBUG:boto:Data: 2024-04-19T23:24:00.750 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.750 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.750 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.750 DEBUG:boto:Params: {} 2024-04-19T23:24:00.750 DEBUG:boto:Token: None 2024-04-19T23:24:00.750 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%3A 2024-04-19T23:24:00.750 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NfLePqcDV7O6ZahXwOXhf7BQpjc= 2024-04-19T23:24:00.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NfLePqcDV7O6ZahXwOXhf7BQpjc=', 'Content-Length': '0'} 2024-04-19T23:24:00.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000f63e98c9366437-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.753 DEBUG:boto:path=/aktlji-10/%3A 2024-04-19T23:24:00.753 DEBUG:boto:auth_path=/aktlji-10/%3A 2024-04-19T23:24:00.753 DEBUG:boto:Method: GET 2024-04-19T23:24:00.753 DEBUG:boto:Path: /aktlji-10/%3A 2024-04-19T23:24:00.753 DEBUG:boto:Data: 2024-04-19T23:24:00.753 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.753 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.753 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.753 DEBUG:boto:Params: {} 2024-04-19T23:24:00.754 DEBUG:boto:Token: None 2024-04-19T23:24:00.754 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/%3A 2024-04-19T23:24:00.754 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NfLePqcDV7O6ZahXwOXhf7BQpjc= 2024-04-19T23:24:00.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NfLePqcDV7O6ZahXwOXhf7BQpjc=', 'Content-Length': '0'} 2024-04-19T23:24:00.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070d19b13def3cbbf-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.802 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.802 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.803 DEBUG:boto:path=/aktlji-10/?prefix=%3A 2024-04-19T23:24:00.803 DEBUG:boto:auth_path=/aktlji-10/?prefix=%3A 2024-04-19T23:24:00.803 DEBUG:boto:Method: GET 2024-04-19T23:24:00.803 DEBUG:boto:Path: /aktlji-10/?prefix=%3A 2024-04-19T23:24:00.803 DEBUG:boto:Data: 2024-04-19T23:24:00.803 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.803 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.803 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.803 DEBUG:boto:Params: {} 2024-04-19T23:24:00.803 DEBUG:boto:Token: None 2024-04-19T23:24:00.804 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w= 2024-04-19T23:24:00.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w=', 'Content-Length': '0'} 2024-04-19T23:24:00.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bdca59296838aba-006622fd10-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.807 DEBUG:boto:b'aktlji-10:1000false:2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.807 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.807 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.807 DEBUG:boto:path=/aktlji-10/?prefix=%3A 2024-04-19T23:24:00.807 DEBUG:boto:auth_path=/aktlji-10/?prefix=%3A 2024-04-19T23:24:00.807 DEBUG:boto:Method: GET 2024-04-19T23:24:00.808 DEBUG:boto:Path: /aktlji-10/?prefix=%3A 2024-04-19T23:24:00.808 DEBUG:boto:Data: 2024-04-19T23:24:00.808 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.808 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.808 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.808 DEBUG:boto:Params: {} 2024-04-19T23:24:00.808 DEBUG:boto:Token: None 2024-04-19T23:24:00.808 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.808 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w= 2024-04-19T23:24:00.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w=', 'Content-Length': '0'} 2024-04-19T23:24:00.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000180005e973088c43-006622fd10-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.812 DEBUG:boto:b'aktlji-10:1000false:2024-04-19T23:23:14.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:00.812 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:24:00.812 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:24:00.812 DEBUG:boto:path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.812 DEBUG:boto:auth_path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.812 DEBUG:boto:Method: GET 2024-04-19T23:24:00.812 DEBUG:boto:Path: /aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.813 DEBUG:boto:Data: 2024-04-19T23:24:00.813 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.813 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.813 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.813 DEBUG:boto:Params: {} 2024-04-19T23:24:00.813 DEBUG:boto:Token: None 2024-04-19T23:24:00.813 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.813 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:638gd7J7df6vRY+jcnC0yuwwXJ8= 2024-04-19T23:24:00.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:638gd7J7df6vRY+jcnC0yuwwXJ8=', 'Content-Length': '0'} 2024-04-19T23:24:00.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b075e90458556afe-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.858 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:24:00.858 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:24:00.859 DEBUG:boto:path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.859 DEBUG:boto:auth_path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.859 DEBUG:boto:Method: GET 2024-04-19T23:24:00.859 DEBUG:boto:Path: /aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.859 DEBUG:boto:Data: 2024-04-19T23:24:00.859 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.859 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.859 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.859 DEBUG:boto:Params: {} 2024-04-19T23:24:00.859 DEBUG:boto:Token: None 2024-04-19T23:24:00.860 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.860 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:638gd7J7df6vRY+jcnC0yuwwXJ8= 2024-04-19T23:24:00.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:638gd7J7df6vRY+jcnC0yuwwXJ8=', 'Content-Length': '0'} 2024-04-19T23:24:00.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8f7be6b5e116c34-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.862 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:24:00.862 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:24:00.862 DEBUG:boto:path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.862 DEBUG:boto:auth_path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.862 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.862 DEBUG:boto:Path: /aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.862 DEBUG:boto:Data: 2024-04-19T23:24:00.863 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.863 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.863 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.863 DEBUG:boto:Params: {} 2024-04-19T23:24:00.863 DEBUG:boto:Token: None 2024-04-19T23:24:00.863 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.863 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ivJa0XwUcT7WwX5Np9EboRQS/as= 2024-04-19T23:24:00.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ivJa0XwUcT7WwX5Np9EboRQS/as=', 'Content-Length': '0'} 2024-04-19T23:24:00.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f91737ece262436-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.865 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:24:00.865 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:24:00.866 DEBUG:boto:path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.866 DEBUG:boto:auth_path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.866 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.866 DEBUG:boto:Path: /aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.866 DEBUG:boto:Data: 2024-04-19T23:24:00.866 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.866 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.866 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.866 DEBUG:boto:Params: {} 2024-04-19T23:24:00.866 DEBUG:boto:Token: None 2024-04-19T23:24:00.867 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.867 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ivJa0XwUcT7WwX5Np9EboRQS/as= 2024-04-19T23:24:00.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ivJa0XwUcT7WwX5Np9EboRQS/as=', 'Content-Length': '0'} 2024-04-19T23:24:00.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000077d5234cb4aea01-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:00.869 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:24:00.869 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:24:00.869 DEBUG:boto:path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.869 DEBUG:boto:auth_path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.869 DEBUG:boto:Method: GET 2024-04-19T23:24:00.869 DEBUG:boto:Path: /aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.870 DEBUG:boto:Data: 2024-04-19T23:24:00.870 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.870 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.870 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.870 DEBUG:boto:Params: {} 2024-04-19T23:24:00.870 DEBUG:boto:Token: None 2024-04-19T23:24:00.870 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.870 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:638gd7J7df6vRY+jcnC0yuwwXJ8= 2024-04-19T23:24:00.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:638gd7J7df6vRY+jcnC0yuwwXJ8=', 'Content-Length': '0'} 2024-04-19T23:24:00.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fe7969f9b068129-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.914 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:24:00.915 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:24:00.915 DEBUG:boto:path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.915 DEBUG:boto:auth_path=/aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.915 DEBUG:boto:Method: GET 2024-04-19T23:24:00.915 DEBUG:boto:Path: /aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.915 DEBUG:boto:Data: 2024-04-19T23:24:00.915 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.915 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.915 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.915 DEBUG:boto:Params: {} 2024-04-19T23:24:00.915 DEBUG:boto:Token: None 2024-04-19T23:24:00.916 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/_myobj?versionId=null 2024-04-19T23:24:00.916 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:638gd7J7df6vRY+jcnC0yuwwXJ8= 2024-04-19T23:24:00.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:638gd7J7df6vRY+jcnC0yuwwXJ8=', 'Content-Length': '0'} 2024-04-19T23:24:00.918 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b54d9a969e8c7a29-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.918 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.918 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.918 DEBUG:boto:path=/aktlji-10/?prefix=_myobj 2024-04-19T23:24:00.919 DEBUG:boto:auth_path=/aktlji-10/?prefix=_myobj 2024-04-19T23:24:00.919 DEBUG:boto:Method: GET 2024-04-19T23:24:00.919 DEBUG:boto:Path: /aktlji-10/?prefix=_myobj 2024-04-19T23:24:00.919 DEBUG:boto:Data: 2024-04-19T23:24:00.919 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.919 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.919 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.919 DEBUG:boto:Params: {} 2024-04-19T23:24:00.919 DEBUG:boto:Token: None 2024-04-19T23:24:00.919 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.920 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w= 2024-04-19T23:24:00.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w=', 'Content-Length': '0'} 2024-04-19T23:24:00.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006018d9bf7e9d5905-006622fd10-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.922 DEBUG:boto:b'aktlji-10_myobj1000false_myobj2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.923 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.923 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.923 DEBUG:boto:path=/aktlji-10/?prefix=_myobj 2024-04-19T23:24:00.923 DEBUG:boto:auth_path=/aktlji-10/?prefix=_myobj 2024-04-19T23:24:00.923 DEBUG:boto:Method: GET 2024-04-19T23:24:00.923 DEBUG:boto:Path: /aktlji-10/?prefix=_myobj 2024-04-19T23:24:00.923 DEBUG:boto:Data: 2024-04-19T23:24:00.924 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.924 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.924 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.924 DEBUG:boto:Params: {} 2024-04-19T23:24:00.924 DEBUG:boto:Token: None 2024-04-19T23:24:00.924 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.924 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w= 2024-04-19T23:24:00.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w=', 'Content-Length': '0'} 2024-04-19T23:24:00.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053dee2c2cc78b1fe-006622fd10-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.927 DEBUG:boto:b'aktlji-10_myobj1000false_myobj2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.927 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:24:00.927 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:24:00.928 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.928 DEBUG:boto:Path: /aktlji-10/_myobj 2024-04-19T23:24:00.928 DEBUG:boto:Data: 2024-04-19T23:24:00.928 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.928 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.928 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.928 DEBUG:boto:Params: {} 2024-04-19T23:24:00.928 DEBUG:boto:Token: None 2024-04-19T23:24:00.928 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/_myobj 2024-04-19T23:24:00.928 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:J9iwFVWvXW8rU9bFIQtqsEMxgZY= 2024-04-19T23:24:00.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:J9iwFVWvXW8rU9bFIQtqsEMxgZY=', 'Content-Length': '0'} 2024-04-19T23:24:00.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d793e27afda7d62b-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.931 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:24:00.931 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:24:00.931 DEBUG:boto:Method: HEAD 2024-04-19T23:24:00.931 DEBUG:boto:Path: /aktlji-10/_myobj 2024-04-19T23:24:00.931 DEBUG:boto:Data: 2024-04-19T23:24:00.931 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.931 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.931 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.932 DEBUG:boto:Params: {} 2024-04-19T23:24:00.932 DEBUG:boto:Token: None 2024-04-19T23:24:00.932 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/_myobj 2024-04-19T23:24:00.932 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:J9iwFVWvXW8rU9bFIQtqsEMxgZY= 2024-04-19T23:24:00.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:J9iwFVWvXW8rU9bFIQtqsEMxgZY=', 'Content-Length': '0'} 2024-04-19T23:24:00.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc863ddcd0795b29-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:00.934 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:24:00.934 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:24:00.934 DEBUG:boto:Method: GET 2024-04-19T23:24:00.934 DEBUG:boto:Path: /aktlji-10/_myobj 2024-04-19T23:24:00.934 DEBUG:boto:Data: 2024-04-19T23:24:00.935 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.935 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.935 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.935 DEBUG:boto:Params: {} 2024-04-19T23:24:00.935 DEBUG:boto:Token: None 2024-04-19T23:24:00.935 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/_myobj 2024-04-19T23:24:00.935 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OIcN4nSZO1TEdaxzD8cylF4tINs= 2024-04-19T23:24:00.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OIcN4nSZO1TEdaxzD8cylF4tINs=', 'Content-Length': '0'} 2024-04-19T23:24:00.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ab851728af6863f-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.978 DEBUG:boto:path=/aktlji-10/_myobj 2024-04-19T23:24:00.979 DEBUG:boto:auth_path=/aktlji-10/_myobj 2024-04-19T23:24:00.979 DEBUG:boto:Method: GET 2024-04-19T23:24:00.979 DEBUG:boto:Path: /aktlji-10/_myobj 2024-04-19T23:24:00.979 DEBUG:boto:Data: 2024-04-19T23:24:00.979 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.979 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.979 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.979 DEBUG:boto:Params: {} 2024-04-19T23:24:00.979 DEBUG:boto:Token: None 2024-04-19T23:24:00.980 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/_myobj 2024-04-19T23:24:00.980 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OIcN4nSZO1TEdaxzD8cylF4tINs= 2024-04-19T23:24:00.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OIcN4nSZO1TEdaxzD8cylF4tINs=', 'Content-Length': '0'} 2024-04-19T23:24:00.982 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003340d9f100460a88-006622fd10-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.982 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.982 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.983 DEBUG:boto:path=/aktlji-10/?prefix=_myobj 2024-04-19T23:24:00.983 DEBUG:boto:auth_path=/aktlji-10/?prefix=_myobj 2024-04-19T23:24:00.983 DEBUG:boto:Method: GET 2024-04-19T23:24:00.983 DEBUG:boto:Path: /aktlji-10/?prefix=_myobj 2024-04-19T23:24:00.983 DEBUG:boto:Data: 2024-04-19T23:24:00.983 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.983 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.983 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.983 DEBUG:boto:Params: {} 2024-04-19T23:24:00.983 DEBUG:boto:Token: None 2024-04-19T23:24:00.984 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w= 2024-04-19T23:24:00.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w=', 'Content-Length': '0'} 2024-04-19T23:24:00.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006df8d07b1c9ced99-006622fd10-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.986 DEBUG:boto:b'aktlji-10_myobj1000false_myobj2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.987 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:00.987 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:00.987 DEBUG:boto:path=/aktlji-10/?prefix=_myobj 2024-04-19T23:24:00.987 DEBUG:boto:auth_path=/aktlji-10/?prefix=_myobj 2024-04-19T23:24:00.987 DEBUG:boto:Method: GET 2024-04-19T23:24:00.987 DEBUG:boto:Path: /aktlji-10/?prefix=_myobj 2024-04-19T23:24:00.987 DEBUG:boto:Data: 2024-04-19T23:24:00.988 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.988 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.988 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.988 DEBUG:boto:Params: {} 2024-04-19T23:24:00.988 DEBUG:boto:Token: None 2024-04-19T23:24:00.988 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/ 2024-04-19T23:24:00.988 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w= 2024-04-19T23:24:00.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iSKFH5dLs9QkKYyppg7aHMpxl9w=', 'Content-Length': '0'} 2024-04-19T23:24:00.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015d499a01a11a04a-006622fd10-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.991 DEBUG:boto:b'aktlji-10_myobj1000false_myobj2024-04-19T23:23:14.044Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:00.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:00.992 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:24:00.992 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:24:00.992 DEBUG:boto:path=/aktlji-10/myobj?versionId=null 2024-04-19T23:24:00.992 DEBUG:boto:auth_path=/aktlji-10/myobj?versionId=null 2024-04-19T23:24:00.992 DEBUG:boto:Method: GET 2024-04-19T23:24:00.992 DEBUG:boto:Path: /aktlji-10/myobj?versionId=null 2024-04-19T23:24:00.992 DEBUG:boto:Data: 2024-04-19T23:24:00.992 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.992 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:00.992 DEBUG:boto:Port: 8001 2024-04-19T23:24:00.993 DEBUG:boto:Params: {} 2024-04-19T23:24:00.993 DEBUG:boto:Token: None 2024-04-19T23:24:00.993 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/myobj?versionId=null 2024-04-19T23:24:00.993 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EcfWqk9ekrHsLkBpNqH1tz/xbEM= 2024-04-19T23:24:00.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EcfWqk9ekrHsLkBpNqH1tz/xbEM=', 'Content-Length': '0'} 2024-04-19T23:24:00.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d4c5fb6cd1018aa-006622fd10-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:00.995 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:24:00.995 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:24:00.995 DEBUG:boto:path=/aktlji-10/myobj?versionId=null 2024-04-19T23:24:00.995 DEBUG:boto:auth_path=/aktlji-10/myobj?versionId=null 2024-04-19T23:24:00.996 DEBUG:boto:Method: GET 2024-04-19T23:24:00.996 DEBUG:boto:Path: /aktlji-10/myobj?versionId=null 2024-04-19T23:24:00.996 DEBUG:boto:Data: 2024-04-19T23:24:00.996 DEBUG:boto:Headers: {} 2024-04-19T23:24:00.996 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:00.996 DEBUG:boto:Port: 8000 2024-04-19T23:24:00.996 DEBUG:boto:Params: {} 2024-04-19T23:24:00.996 DEBUG:boto:Token: None 2024-04-19T23:24:00.996 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:00 GMT /aktlji-10/myobj?versionId=null 2024-04-19T23:24:00.996 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EcfWqk9ekrHsLkBpNqH1tz/xbEM= 2024-04-19T23:24:00.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EcfWqk9ekrHsLkBpNqH1tz/xbEM=', 'Content-Length': '0'} 2024-04-19T23:24:00.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9faa8ba06cfc650-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.046 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:24:01.047 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:24:01.047 DEBUG:boto:path=/aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.047 DEBUG:boto:auth_path=/aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.047 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.047 DEBUG:boto:Path: /aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.047 DEBUG:boto:Data: 2024-04-19T23:24:01.047 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.047 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.047 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.047 DEBUG:boto:Params: {} 2024-04-19T23:24:01.048 DEBUG:boto:Token: None 2024-04-19T23:24:01.048 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.048 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:v1UpOBjXHNgJrz69JPYQNHT/B+U= 2024-04-19T23:24:01.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:v1UpOBjXHNgJrz69JPYQNHT/B+U=', 'Content-Length': '0'} 2024-04-19T23:24:01.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000301ea47917062aa1-006622fd11-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.050 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:24:01.050 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:24:01.051 DEBUG:boto:path=/aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.051 DEBUG:boto:auth_path=/aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.051 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.051 DEBUG:boto:Path: /aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.051 DEBUG:boto:Data: 2024-04-19T23:24:01.051 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.051 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.051 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.051 DEBUG:boto:Params: {} 2024-04-19T23:24:01.051 DEBUG:boto:Token: None 2024-04-19T23:24:01.052 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.052 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:v1UpOBjXHNgJrz69JPYQNHT/B+U= 2024-04-19T23:24:01.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:v1UpOBjXHNgJrz69JPYQNHT/B+U=', 'Content-Length': '0'} 2024-04-19T23:24:01.053 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df428920156aba64-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:01.054 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:24:01.054 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:24:01.054 DEBUG:boto:path=/aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.054 DEBUG:boto:auth_path=/aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.054 DEBUG:boto:Method: GET 2024-04-19T23:24:01.054 DEBUG:boto:Path: /aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.054 DEBUG:boto:Data: 2024-04-19T23:24:01.054 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.054 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.055 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.055 DEBUG:boto:Params: {} 2024-04-19T23:24:01.055 DEBUG:boto:Token: None 2024-04-19T23:24:01.055 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.055 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FBmtjyziBPRa71osD3j2ZX4GlqA= 2024-04-19T23:24:01.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FBmtjyziBPRa71osD3j2ZX4GlqA=', 'Content-Length': '0'} 2024-04-19T23:24:01.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004e402909738f5ef-006622fd11-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.057 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:24:01.057 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:24:01.057 DEBUG:boto:path=/aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.057 DEBUG:boto:auth_path=/aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.058 DEBUG:boto:Method: GET 2024-04-19T23:24:01.058 DEBUG:boto:Path: /aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.058 DEBUG:boto:Data: 2024-04-19T23:24:01.058 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.058 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.058 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.058 DEBUG:boto:Params: {} 2024-04-19T23:24:01.058 DEBUG:boto:Token: None 2024-04-19T23:24:01.059 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-10/myobj?versionId=null 2024-04-19T23:24:01.059 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FBmtjyziBPRa71osD3j2ZX4GlqA= 2024-04-19T23:24:01.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FBmtjyziBPRa71osD3j2ZX4GlqA=', 'Content-Length': '0'} 2024-04-19T23:24:01.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002bc82a3195743703-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.061 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:01.061 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:01.061 DEBUG:boto:path=/aktlji-10/?prefix=myobj 2024-04-19T23:24:01.061 DEBUG:boto:auth_path=/aktlji-10/?prefix=myobj 2024-04-19T23:24:01.061 DEBUG:boto:Method: GET 2024-04-19T23:24:01.061 DEBUG:boto:Path: /aktlji-10/?prefix=myobj 2024-04-19T23:24:01.062 DEBUG:boto:Data: 2024-04-19T23:24:01.062 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.062 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.062 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.062 DEBUG:boto:Params: {} 2024-04-19T23:24:01.062 DEBUG:boto:Token: None 2024-04-19T23:24:01.062 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-10/ 2024-04-19T23:24:01.062 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HQGFVuMqIhy/rQwKaUXt2I1d/EM= 2024-04-19T23:24:01.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HQGFVuMqIhy/rQwKaUXt2I1d/EM=', 'Content-Length': '0'} 2024-04-19T23:24:01.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8cac9940fc7bb1b-006622fd11-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.065 DEBUG:boto:b'aktlji-10myobj1000falsemyobj2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.066 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:01.066 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:01.066 DEBUG:boto:path=/aktlji-10/?prefix=myobj 2024-04-19T23:24:01.066 DEBUG:boto:auth_path=/aktlji-10/?prefix=myobj 2024-04-19T23:24:01.066 DEBUG:boto:Method: GET 2024-04-19T23:24:01.066 DEBUG:boto:Path: /aktlji-10/?prefix=myobj 2024-04-19T23:24:01.066 DEBUG:boto:Data: 2024-04-19T23:24:01.066 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.066 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.067 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.067 DEBUG:boto:Params: {} 2024-04-19T23:24:01.067 DEBUG:boto:Token: None 2024-04-19T23:24:01.067 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-10/ 2024-04-19T23:24:01.067 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HQGFVuMqIhy/rQwKaUXt2I1d/EM= 2024-04-19T23:24:01.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HQGFVuMqIhy/rQwKaUXt2I1d/EM=', 'Content-Length': '0'} 2024-04-19T23:24:01.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e32468f54a42431-006622fd11-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.070 DEBUG:boto:b'aktlji-10myobj1000falsemyobj2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.070 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:24:01.070 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:24:01.070 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.070 DEBUG:boto:Path: /aktlji-10/myobj 2024-04-19T23:24:01.070 DEBUG:boto:Data: 2024-04-19T23:24:01.071 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.071 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.071 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.071 DEBUG:boto:Params: {} 2024-04-19T23:24:01.071 DEBUG:boto:Token: None 2024-04-19T23:24:01.072 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-10/myobj 2024-04-19T23:24:01.072 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:grmqL3zXbk+euPIRttfPg+gJCCs= 2024-04-19T23:24:01.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:grmqL3zXbk+euPIRttfPg+gJCCs=', 'Content-Length': '0'} 2024-04-19T23:24:01.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac293c65ddd66544-006622fd11-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.074 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:24:01.074 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:24:01.075 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.075 DEBUG:boto:Path: /aktlji-10/myobj 2024-04-19T23:24:01.075 DEBUG:boto:Data: 2024-04-19T23:24:01.075 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.075 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.075 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.075 DEBUG:boto:Params: {} 2024-04-19T23:24:01.075 DEBUG:boto:Token: None 2024-04-19T23:24:01.075 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-10/myobj 2024-04-19T23:24:01.075 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:grmqL3zXbk+euPIRttfPg+gJCCs= 2024-04-19T23:24:01.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:grmqL3zXbk+euPIRttfPg+gJCCs=', 'Content-Length': '0'} 2024-04-19T23:24:01.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5355a190e898ce0-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:01.078 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:24:01.078 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:24:01.078 DEBUG:boto:Method: GET 2024-04-19T23:24:01.078 DEBUG:boto:Path: /aktlji-10/myobj 2024-04-19T23:24:01.078 DEBUG:boto:Data: 2024-04-19T23:24:01.078 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.078 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.078 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.078 DEBUG:boto:Params: {} 2024-04-19T23:24:01.079 DEBUG:boto:Token: None 2024-04-19T23:24:01.079 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-10/myobj 2024-04-19T23:24:01.079 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1HC2dJmDMwVrxO90Pjxk+vCjT4A= 2024-04-19T23:24:01.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1HC2dJmDMwVrxO90Pjxk+vCjT4A=', 'Content-Length': '0'} 2024-04-19T23:24:01.080 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c2b1f48957b7fc6-006622fd11-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.081 DEBUG:boto:path=/aktlji-10/myobj 2024-04-19T23:24:01.081 DEBUG:boto:auth_path=/aktlji-10/myobj 2024-04-19T23:24:01.081 DEBUG:boto:Method: GET 2024-04-19T23:24:01.081 DEBUG:boto:Path: /aktlji-10/myobj 2024-04-19T23:24:01.081 DEBUG:boto:Data: 2024-04-19T23:24:01.081 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.081 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.082 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.082 DEBUG:boto:Params: {} 2024-04-19T23:24:01.082 DEBUG:boto:Token: None 2024-04-19T23:24:01.082 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-10/myobj 2024-04-19T23:24:01.082 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1HC2dJmDMwVrxO90Pjxk+vCjT4A= 2024-04-19T23:24:01.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1HC2dJmDMwVrxO90Pjxk+vCjT4A=', 'Content-Length': '0'} 2024-04-19T23:24:01.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082a2e4d61e31f966-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.085 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:01.085 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:01.085 DEBUG:boto:path=/aktlji-10/?prefix=myobj 2024-04-19T23:24:01.086 DEBUG:boto:auth_path=/aktlji-10/?prefix=myobj 2024-04-19T23:24:01.086 DEBUG:boto:Method: GET 2024-04-19T23:24:01.086 DEBUG:boto:Path: /aktlji-10/?prefix=myobj 2024-04-19T23:24:01.086 DEBUG:boto:Data: 2024-04-19T23:24:01.086 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.086 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.087 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.087 DEBUG:boto:Params: {} 2024-04-19T23:24:01.087 DEBUG:boto:Token: None 2024-04-19T23:24:01.087 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-10/ 2024-04-19T23:24:01.088 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HQGFVuMqIhy/rQwKaUXt2I1d/EM= 2024-04-19T23:24:01.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HQGFVuMqIhy/rQwKaUXt2I1d/EM=', 'Content-Length': '0'} 2024-04-19T23:24:01.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a8d794591f50211-006622fd11-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.091 DEBUG:boto:b'aktlji-10myobj1000falsemyobj2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.091 DEBUG:boto:path=/aktlji-10/ 2024-04-19T23:24:01.092 DEBUG:boto:auth_path=/aktlji-10/ 2024-04-19T23:24:01.092 DEBUG:boto:path=/aktlji-10/?prefix=myobj 2024-04-19T23:24:01.092 DEBUG:boto:auth_path=/aktlji-10/?prefix=myobj 2024-04-19T23:24:01.092 DEBUG:boto:Method: GET 2024-04-19T23:24:01.092 DEBUG:boto:Path: /aktlji-10/?prefix=myobj 2024-04-19T23:24:01.092 DEBUG:boto:Data: 2024-04-19T23:24:01.092 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.092 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.092 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.092 DEBUG:boto:Params: {} 2024-04-19T23:24:01.093 DEBUG:boto:Token: None 2024-04-19T23:24:01.093 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-10/ 2024-04-19T23:24:01.093 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HQGFVuMqIhy/rQwKaUXt2I1d/EM= 2024-04-19T23:24:01.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HQGFVuMqIhy/rQwKaUXt2I1d/EM=', 'Content-Length': '0'} 2024-04-19T23:24:01.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0d405423bfda15b-006622fd11-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.095 DEBUG:boto:b'aktlji-10myobj1000falsemyobj2024-04-19T23:23:14.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.096 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-10 zones={test-zone3, test-zone1} 2024-04-19T23:24:01.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:01.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:01.096 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:24:01.162 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:01.162 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:01.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:01.209+0000 7f9c1164f780 1 bucket sync caught up with source: 2024-04-19T23:24:01.205 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , 00000000001.6.6, , , , 00000000001.8.6, 00000000001.17.6, , ] 2024-04-19T23:24:01.205 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000001.10.6, , 00000000001.6.6, , , , 00000000001.8.6, 00000000001.17.6, , ] 2024-04-19T23:24:01.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:01.209+0000 7f9c1164f780 0 bucket checkpoint complete 2024-04-19T23:24:01.219 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-11 zones={test-zone1, test-zone2} 2024-04-19T23:24:01.220 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.220 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.220 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.220 DEBUG:boto:Path: /aktlji-11/ 2024-04-19T23:24:01.220 DEBUG:boto:Data: 2024-04-19T23:24:01.220 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.220 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.220 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.220 DEBUG:boto:Params: {} 2024-04-19T23:24:01.220 DEBUG:boto:Token: None 2024-04-19T23:24:01.221 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.221 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2zlZpXUx9mPdqukB4mYX/iSg/3s= 2024-04-19T23:24:01.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2zlZpXUx9mPdqukB4mYX/iSg/3s=', 'Content-Length': '0'} 2024-04-19T23:24:01.223 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e985fca5112be41-006622fd11-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.223 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.223 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.223 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.224 DEBUG:boto:Path: /aktlji-11/ 2024-04-19T23:24:01.224 DEBUG:boto:Data: 2024-04-19T23:24:01.224 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.224 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.224 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.224 DEBUG:boto:Params: {} 2024-04-19T23:24:01.224 DEBUG:boto:Token: None 2024-04-19T23:24:01.224 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.224 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2zlZpXUx9mPdqukB4mYX/iSg/3s= 2024-04-19T23:24:01.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2zlZpXUx9mPdqukB4mYX/iSg/3s=', 'Content-Length': '0'} 2024-04-19T23:24:01.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009521427420a0f60a-006622fd11-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.227 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:24:01.227 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.227 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.227 DEBUG:boto:path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.227 DEBUG:boto:auth_path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.227 DEBUG:boto:Method: GET 2024-04-19T23:24:01.227 DEBUG:boto:Path: /aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.228 DEBUG:boto:Data: 2024-04-19T23:24:01.228 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.228 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.228 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.228 DEBUG:boto:Params: {} 2024-04-19T23:24:01.228 DEBUG:boto:Token: None 2024-04-19T23:24:01.228 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/?versions 2024-04-19T23:24:01.228 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY= 2024-04-19T23:24:01.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY=', 'Content-Length': '0'} 2024-04-19T23:24:01.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6db665fa0d24573-006622fd11-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.231 DEBUG:boto:b'aktlji-11999false&nulltrue2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.231 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-19T23:24:01.231 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-19T23:24:01.231 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-19T23:24:01.232 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:01.232 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:24:01.232 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.232 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.232 DEBUG:boto:path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.232 DEBUG:boto:auth_path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.232 DEBUG:boto:Method: GET 2024-04-19T23:24:01.232 DEBUG:boto:Path: /aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.232 DEBUG:boto:Data: 2024-04-19T23:24:01.232 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.233 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.233 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.233 DEBUG:boto:Params: {} 2024-04-19T23:24:01.233 DEBUG:boto:Token: None 2024-04-19T23:24:01.233 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/?versions 2024-04-19T23:24:01.233 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY= 2024-04-19T23:24:01.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY=', 'Content-Length': '0'} 2024-04-19T23:24:01.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098f0af1fdf3267ab-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.236 DEBUG:boto:b'aktlji-11999false&nulltrue2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.236 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-19T23:24:01.237 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-19T23:24:01.237 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-19T23:24:01.237 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:01.237 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.237 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.237 DEBUG:boto:path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.237 DEBUG:boto:auth_path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.237 DEBUG:boto:Method: GET 2024-04-19T23:24:01.237 DEBUG:boto:Path: /aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.237 DEBUG:boto:Data: 2024-04-19T23:24:01.238 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.238 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.238 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.238 DEBUG:boto:Params: {} 2024-04-19T23:24:01.238 DEBUG:boto:Token: None 2024-04-19T23:24:01.238 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/?versions 2024-04-19T23:24:01.238 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY= 2024-04-19T23:24:01.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY=', 'Content-Length': '0'} 2024-04-19T23:24:01.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008608c98abdae768f-006622fd11-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.241 DEBUG:boto:b'aktlji-11999false&nulltrue2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.241 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.241 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.241 DEBUG:boto:path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.241 DEBUG:boto:auth_path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.242 DEBUG:boto:Method: GET 2024-04-19T23:24:01.242 DEBUG:boto:Path: /aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.242 DEBUG:boto:Data: 2024-04-19T23:24:01.242 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.242 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.242 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.242 DEBUG:boto:Params: {} 2024-04-19T23:24:01.242 DEBUG:boto:Token: None 2024-04-19T23:24:01.242 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/?versions 2024-04-19T23:24:01.243 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY= 2024-04-19T23:24:01.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY=', 'Content-Length': '0'} 2024-04-19T23:24:01.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000952bec151883619f-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.245 DEBUG:boto:b'aktlji-11999false&nulltrue2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:01.246 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.246 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.246 DEBUG:boto:path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.246 DEBUG:boto:auth_path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.246 DEBUG:boto:Method: GET 2024-04-19T23:24:01.246 DEBUG:boto:Path: /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.246 DEBUG:boto:Data: 2024-04-19T23:24:01.246 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.247 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.247 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.247 DEBUG:boto:Params: {} 2024-04-19T23:24:01.247 DEBUG:boto:Token: None 2024-04-19T23:24:01.247 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.247 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E= 2024-04-19T23:24:01.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E=', 'Content-Length': '0'} 2024-04-19T23:24:01.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025675988b2b1649e-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.290 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.291 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.291 DEBUG:boto:path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.291 DEBUG:boto:auth_path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.291 DEBUG:boto:Method: GET 2024-04-19T23:24:01.291 DEBUG:boto:Path: /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.291 DEBUG:boto:Data: 2024-04-19T23:24:01.291 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.291 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.291 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.291 DEBUG:boto:Params: {} 2024-04-19T23:24:01.292 DEBUG:boto:Token: None 2024-04-19T23:24:01.292 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.292 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E= 2024-04-19T23:24:01.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E=', 'Content-Length': '0'} 2024-04-19T23:24:01.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edb12ecde02eb77e-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.294 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.294 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.294 DEBUG:boto:path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.294 DEBUG:boto:auth_path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.294 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.295 DEBUG:boto:Path: /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.295 DEBUG:boto:Data: 2024-04-19T23:24:01.295 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.295 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.295 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.295 DEBUG:boto:Params: {} 2024-04-19T23:24:01.295 DEBUG:boto:Token: None 2024-04-19T23:24:01.295 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.295 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:v+Xq3T/ja5kHuNO+1egdGDjP85s= 2024-04-19T23:24:01.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:v+Xq3T/ja5kHuNO+1egdGDjP85s=', 'Content-Length': '0'} 2024-04-19T23:24:01.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f29a52f91931a8e-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.297 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.297 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.298 DEBUG:boto:path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.298 DEBUG:boto:auth_path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.298 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.298 DEBUG:boto:Path: /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.298 DEBUG:boto:Data: 2024-04-19T23:24:01.298 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.298 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.298 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.298 DEBUG:boto:Params: {} 2024-04-19T23:24:01.298 DEBUG:boto:Token: None 2024-04-19T23:24:01.298 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.299 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:v+Xq3T/ja5kHuNO+1egdGDjP85s= 2024-04-19T23:24:01.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:v+Xq3T/ja5kHuNO+1egdGDjP85s=', 'Content-Length': '0'} 2024-04-19T23:24:01.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000626e9f6b0f8f8323-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:01.301 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.301 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.301 DEBUG:boto:path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.301 DEBUG:boto:auth_path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.301 DEBUG:boto:Method: GET 2024-04-19T23:24:01.301 DEBUG:boto:Path: /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.301 DEBUG:boto:Data: 2024-04-19T23:24:01.301 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.301 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.301 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.302 DEBUG:boto:Params: {} 2024-04-19T23:24:01.302 DEBUG:boto:Token: None 2024-04-19T23:24:01.302 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.302 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E= 2024-04-19T23:24:01.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E=', 'Content-Length': '0'} 2024-04-19T23:24:01.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa4290a9474707ae-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.304 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.304 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.304 DEBUG:boto:path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.304 DEBUG:boto:auth_path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.304 DEBUG:boto:Method: GET 2024-04-19T23:24:01.305 DEBUG:boto:Path: /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.305 DEBUG:boto:Data: 2024-04-19T23:24:01.305 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.305 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.305 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.305 DEBUG:boto:Params: {} 2024-04-19T23:24:01.305 DEBUG:boto:Token: None 2024-04-19T23:24:01.305 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.305 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E= 2024-04-19T23:24:01.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E=', 'Content-Length': '0'} 2024-04-19T23:24:01.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e75fd63eec734dd-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.307 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.308 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.308 DEBUG:boto:path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.308 DEBUG:boto:auth_path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.308 DEBUG:boto:Method: GET 2024-04-19T23:24:01.308 DEBUG:boto:Path: /aktlji-11/?prefix=%26 2024-04-19T23:24:01.308 DEBUG:boto:Data: 2024-04-19T23:24:01.308 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.308 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.308 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.308 DEBUG:boto:Params: {} 2024-04-19T23:24:01.308 DEBUG:boto:Token: None 2024-04-19T23:24:01.309 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.309 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000744d957851eb9ddf-006622fd11-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.311 DEBUG:boto:b'aktlji-11&1000false&2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.314 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.314 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.314 DEBUG:boto:path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.314 DEBUG:boto:auth_path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.314 DEBUG:boto:Method: GET 2024-04-19T23:24:01.314 DEBUG:boto:Path: /aktlji-11/?prefix=%26 2024-04-19T23:24:01.315 DEBUG:boto:Data: 2024-04-19T23:24:01.315 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.315 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.315 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.315 DEBUG:boto:Params: {} 2024-04-19T23:24:01.315 DEBUG:boto:Token: None 2024-04-19T23:24:01.315 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.315 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007669cd225e8fb52d-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.318 DEBUG:boto:b'aktlji-11&1000false&2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.318 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.318 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.318 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.319 DEBUG:boto:Path: /aktlji-11/%26 2024-04-19T23:24:01.319 DEBUG:boto:Data: 2024-04-19T23:24:01.319 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.319 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.319 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.319 DEBUG:boto:Params: {} 2024-04-19T23:24:01.319 DEBUG:boto:Token: None 2024-04-19T23:24:01.319 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26 2024-04-19T23:24:01.319 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:36sVR5ZwBByGaiHyMU3wr0fUXnA= 2024-04-19T23:24:01.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:36sVR5ZwBByGaiHyMU3wr0fUXnA=', 'Content-Length': '0'} 2024-04-19T23:24:01.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000067fc05d405ec27-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.321 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.322 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.322 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.322 DEBUG:boto:Path: /aktlji-11/%26 2024-04-19T23:24:01.322 DEBUG:boto:Data: 2024-04-19T23:24:01.322 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.322 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.322 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.322 DEBUG:boto:Params: {} 2024-04-19T23:24:01.322 DEBUG:boto:Token: None 2024-04-19T23:24:01.322 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26 2024-04-19T23:24:01.323 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:36sVR5ZwBByGaiHyMU3wr0fUXnA= 2024-04-19T23:24:01.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:36sVR5ZwBByGaiHyMU3wr0fUXnA=', 'Content-Length': '0'} 2024-04-19T23:24:01.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afde07f57443af0c-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:01.325 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.325 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.325 DEBUG:boto:Method: GET 2024-04-19T23:24:01.325 DEBUG:boto:Path: /aktlji-11/%26 2024-04-19T23:24:01.325 DEBUG:boto:Data: 2024-04-19T23:24:01.325 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.326 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.326 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.326 DEBUG:boto:Params: {} 2024-04-19T23:24:01.326 DEBUG:boto:Token: None 2024-04-19T23:24:01.326 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26 2024-04-19T23:24:01.326 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:W/K26wTnsy613JAaKfHBx2WQrlg= 2024-04-19T23:24:01.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W/K26wTnsy613JAaKfHBx2WQrlg=', 'Content-Length': '0'} 2024-04-19T23:24:01.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000068caa9edff8b5ae-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.328 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.328 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.329 DEBUG:boto:Method: GET 2024-04-19T23:24:01.329 DEBUG:boto:Path: /aktlji-11/%26 2024-04-19T23:24:01.329 DEBUG:boto:Data: 2024-04-19T23:24:01.329 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.329 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.329 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.329 DEBUG:boto:Params: {} 2024-04-19T23:24:01.329 DEBUG:boto:Token: None 2024-04-19T23:24:01.329 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26 2024-04-19T23:24:01.329 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:W/K26wTnsy613JAaKfHBx2WQrlg= 2024-04-19T23:24:01.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W/K26wTnsy613JAaKfHBx2WQrlg=', 'Content-Length': '0'} 2024-04-19T23:24:01.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085d30704ab99a99c-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.332 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.332 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.332 DEBUG:boto:path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.332 DEBUG:boto:auth_path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.332 DEBUG:boto:Method: GET 2024-04-19T23:24:01.332 DEBUG:boto:Path: /aktlji-11/?prefix=%26 2024-04-19T23:24:01.332 DEBUG:boto:Data: 2024-04-19T23:24:01.332 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.332 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.332 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.332 DEBUG:boto:Params: {} 2024-04-19T23:24:01.333 DEBUG:boto:Token: None 2024-04-19T23:24:01.333 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.333 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0e2798125a778a0-006622fd11-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.335 DEBUG:boto:b'aktlji-11&1000false&2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.336 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.336 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.336 DEBUG:boto:path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.336 DEBUG:boto:auth_path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.336 DEBUG:boto:Method: GET 2024-04-19T23:24:01.336 DEBUG:boto:Path: /aktlji-11/?prefix=%26 2024-04-19T23:24:01.336 DEBUG:boto:Data: 2024-04-19T23:24:01.336 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.337 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.337 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.337 DEBUG:boto:Params: {} 2024-04-19T23:24:01.337 DEBUG:boto:Token: None 2024-04-19T23:24:01.337 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.337 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000897572b6d5a8e4ca-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.340 DEBUG:boto:b'aktlji-11&1000false&2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:01.340 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:01.340 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:01.341 DEBUG:boto:path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.341 DEBUG:boto:auth_path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.341 DEBUG:boto:Method: GET 2024-04-19T23:24:01.341 DEBUG:boto:Path: /aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.341 DEBUG:boto:Data: 2024-04-19T23:24:01.341 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.341 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.341 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.341 DEBUG:boto:Params: {} 2024-04-19T23:24:01.341 DEBUG:boto:Token: None 2024-04-19T23:24:01.342 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.342 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EB0DyBLZsSwCBKsnPyNM3U9HkQA= 2024-04-19T23:24:01.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EB0DyBLZsSwCBKsnPyNM3U9HkQA=', 'Content-Length': '0'} 2024-04-19T23:24:01.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002bdbe1f2322ed63c-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.387 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:01.387 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:01.387 DEBUG:boto:path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.387 DEBUG:boto:auth_path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.387 DEBUG:boto:Method: GET 2024-04-19T23:24:01.387 DEBUG:boto:Path: /aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.387 DEBUG:boto:Data: 2024-04-19T23:24:01.387 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.387 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.388 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.388 DEBUG:boto:Params: {} 2024-04-19T23:24:01.388 DEBUG:boto:Token: None 2024-04-19T23:24:01.388 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.388 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EB0DyBLZsSwCBKsnPyNM3U9HkQA= 2024-04-19T23:24:01.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EB0DyBLZsSwCBKsnPyNM3U9HkQA=', 'Content-Length': '0'} 2024-04-19T23:24:01.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002724583869a71547-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.390 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:01.390 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:01.391 DEBUG:boto:path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.391 DEBUG:boto:auth_path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.391 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.391 DEBUG:boto:Path: /aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.391 DEBUG:boto:Data: 2024-04-19T23:24:01.391 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.391 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.391 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.391 DEBUG:boto:Params: {} 2024-04-19T23:24:01.391 DEBUG:boto:Token: None 2024-04-19T23:24:01.392 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.392 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tu3ltH7mLF01BNGZh2NezNperTI= 2024-04-19T23:24:01.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tu3ltH7mLF01BNGZh2NezNperTI=', 'Content-Length': '0'} 2024-04-19T23:24:01.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fd907f583cff3e6-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.394 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:01.394 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:01.394 DEBUG:boto:path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.394 DEBUG:boto:auth_path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.395 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.395 DEBUG:boto:Path: /aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.395 DEBUG:boto:Data: 2024-04-19T23:24:01.395 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.395 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.395 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.395 DEBUG:boto:Params: {} 2024-04-19T23:24:01.395 DEBUG:boto:Token: None 2024-04-19T23:24:01.395 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.395 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tu3ltH7mLF01BNGZh2NezNperTI= 2024-04-19T23:24:01.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tu3ltH7mLF01BNGZh2NezNperTI=', 'Content-Length': '0'} 2024-04-19T23:24:01.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005304248c9de7d1dc-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:01.398 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:01.398 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:01.398 DEBUG:boto:path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.398 DEBUG:boto:auth_path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.398 DEBUG:boto:Method: GET 2024-04-19T23:24:01.398 DEBUG:boto:Path: /aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.398 DEBUG:boto:Data: 2024-04-19T23:24:01.399 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.399 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.399 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.399 DEBUG:boto:Params: {} 2024-04-19T23:24:01.399 DEBUG:boto:Token: None 2024-04-19T23:24:01.399 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.399 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EB0DyBLZsSwCBKsnPyNM3U9HkQA= 2024-04-19T23:24:01.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EB0DyBLZsSwCBKsnPyNM3U9HkQA=', 'Content-Length': '0'} 2024-04-19T23:24:01.401 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6248cb03a927913-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.442 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:01.442 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:01.443 DEBUG:boto:path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.443 DEBUG:boto:auth_path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.443 DEBUG:boto:Method: GET 2024-04-19T23:24:01.443 DEBUG:boto:Path: /aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.443 DEBUG:boto:Data: 2024-04-19T23:24:01.443 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.443 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.443 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.443 DEBUG:boto:Params: {} 2024-04-19T23:24:01.443 DEBUG:boto:Token: None 2024-04-19T23:24:01.444 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%3A?versionId=null 2024-04-19T23:24:01.444 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EB0DyBLZsSwCBKsnPyNM3U9HkQA= 2024-04-19T23:24:01.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EB0DyBLZsSwCBKsnPyNM3U9HkQA=', 'Content-Length': '0'} 2024-04-19T23:24:01.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec66e871feb54b93-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.446 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.446 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.446 DEBUG:boto:path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:01.446 DEBUG:boto:auth_path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:01.446 DEBUG:boto:Method: GET 2024-04-19T23:24:01.447 DEBUG:boto:Path: /aktlji-11/?prefix=%3A 2024-04-19T23:24:01.447 DEBUG:boto:Data: 2024-04-19T23:24:01.447 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.447 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.447 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.447 DEBUG:boto:Params: {} 2024-04-19T23:24:01.447 DEBUG:boto:Token: None 2024-04-19T23:24:01.447 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.447 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddef5e9a249ddc4c-006622fd11-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.450 DEBUG:boto:b'aktlji-11:1000false:2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.450 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.451 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.451 DEBUG:boto:path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:01.451 DEBUG:boto:auth_path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:01.451 DEBUG:boto:Method: GET 2024-04-19T23:24:01.451 DEBUG:boto:Path: /aktlji-11/?prefix=%3A 2024-04-19T23:24:01.451 DEBUG:boto:Data: 2024-04-19T23:24:01.451 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.451 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.451 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.451 DEBUG:boto:Params: {} 2024-04-19T23:24:01.452 DEBUG:boto:Token: None 2024-04-19T23:24:01.452 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.452 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a75149c2f657e88-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.454 DEBUG:boto:b'aktlji-11:1000false:2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.455 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:01.455 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:01.455 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.455 DEBUG:boto:Path: /aktlji-11/%3A 2024-04-19T23:24:01.455 DEBUG:boto:Data: 2024-04-19T23:24:01.455 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.455 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.455 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.456 DEBUG:boto:Params: {} 2024-04-19T23:24:01.456 DEBUG:boto:Token: None 2024-04-19T23:24:01.456 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%3A 2024-04-19T23:24:01.456 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:B8h60EpVeRkHW5ar08I0+rntiJM= 2024-04-19T23:24:01.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B8h60EpVeRkHW5ar08I0+rntiJM=', 'Content-Length': '0'} 2024-04-19T23:24:01.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065f984b0a9085d5c-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.458 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:01.458 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:01.458 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.458 DEBUG:boto:Path: /aktlji-11/%3A 2024-04-19T23:24:01.458 DEBUG:boto:Data: 2024-04-19T23:24:01.459 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.459 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.459 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.459 DEBUG:boto:Params: {} 2024-04-19T23:24:01.459 DEBUG:boto:Token: None 2024-04-19T23:24:01.459 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%3A 2024-04-19T23:24:01.459 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:B8h60EpVeRkHW5ar08I0+rntiJM= 2024-04-19T23:24:01.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B8h60EpVeRkHW5ar08I0+rntiJM=', 'Content-Length': '0'} 2024-04-19T23:24:01.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a99f87b7aa96322-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:01.461 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:01.461 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:01.461 DEBUG:boto:Method: GET 2024-04-19T23:24:01.462 DEBUG:boto:Path: /aktlji-11/%3A 2024-04-19T23:24:01.462 DEBUG:boto:Data: 2024-04-19T23:24:01.462 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.462 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.462 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.462 DEBUG:boto:Params: {} 2024-04-19T23:24:01.462 DEBUG:boto:Token: None 2024-04-19T23:24:01.462 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%3A 2024-04-19T23:24:01.462 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ah+WgioVAyQozaVtPaaRhIFHw7Q= 2024-04-19T23:24:01.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ah+WgioVAyQozaVtPaaRhIFHw7Q=', 'Content-Length': '0'} 2024-04-19T23:24:01.464 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad7fb869024bc881-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.510 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:01.511 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:01.511 DEBUG:boto:Method: GET 2024-04-19T23:24:01.511 DEBUG:boto:Path: /aktlji-11/%3A 2024-04-19T23:24:01.511 DEBUG:boto:Data: 2024-04-19T23:24:01.511 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.511 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.511 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.511 DEBUG:boto:Params: {} 2024-04-19T23:24:01.511 DEBUG:boto:Token: None 2024-04-19T23:24:01.511 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%3A 2024-04-19T23:24:01.512 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ah+WgioVAyQozaVtPaaRhIFHw7Q= 2024-04-19T23:24:01.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ah+WgioVAyQozaVtPaaRhIFHw7Q=', 'Content-Length': '0'} 2024-04-19T23:24:01.513 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed53cf48fd500bac-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.514 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.514 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.514 DEBUG:boto:path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:01.514 DEBUG:boto:auth_path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:01.514 DEBUG:boto:Method: GET 2024-04-19T23:24:01.514 DEBUG:boto:Path: /aktlji-11/?prefix=%3A 2024-04-19T23:24:01.514 DEBUG:boto:Data: 2024-04-19T23:24:01.514 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.514 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.514 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.515 DEBUG:boto:Params: {} 2024-04-19T23:24:01.515 DEBUG:boto:Token: None 2024-04-19T23:24:01.515 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.515 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c09dc7286a89e98-006622fd11-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.558 DEBUG:boto:b'aktlji-11:1000false:2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.559 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.559 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.559 DEBUG:boto:path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:01.559 DEBUG:boto:auth_path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:01.559 DEBUG:boto:Method: GET 2024-04-19T23:24:01.559 DEBUG:boto:Path: /aktlji-11/?prefix=%3A 2024-04-19T23:24:01.559 DEBUG:boto:Data: 2024-04-19T23:24:01.559 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.560 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.560 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.560 DEBUG:boto:Params: {} 2024-04-19T23:24:01.560 DEBUG:boto:Token: None 2024-04-19T23:24:01.560 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e49091ca0c7e0b1-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.563 DEBUG:boto:b'aktlji-11:1000false:2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:01.563 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:01.564 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:01.564 DEBUG:boto:path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.564 DEBUG:boto:auth_path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.564 DEBUG:boto:Method: GET 2024-04-19T23:24:01.564 DEBUG:boto:Path: /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.564 DEBUG:boto:Data: 2024-04-19T23:24:01.564 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.564 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.564 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.564 DEBUG:boto:Params: {} 2024-04-19T23:24:01.564 DEBUG:boto:Token: None 2024-04-19T23:24:01.565 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.565 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:30c5vWOzkWJvamP0xWWQbZnLlqY= 2024-04-19T23:24:01.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:30c5vWOzkWJvamP0xWWQbZnLlqY=', 'Content-Length': '0'} 2024-04-19T23:24:01.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000371a26e8cf79362b-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.567 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:01.567 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:01.567 DEBUG:boto:path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.567 DEBUG:boto:auth_path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.567 DEBUG:boto:Method: GET 2024-04-19T23:24:01.567 DEBUG:boto:Path: /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.567 DEBUG:boto:Data: 2024-04-19T23:24:01.567 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.568 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.568 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.568 DEBUG:boto:Params: {} 2024-04-19T23:24:01.568 DEBUG:boto:Token: None 2024-04-19T23:24:01.568 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.568 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:30c5vWOzkWJvamP0xWWQbZnLlqY= 2024-04-19T23:24:01.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:30c5vWOzkWJvamP0xWWQbZnLlqY=', 'Content-Length': '0'} 2024-04-19T23:24:01.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2a977625d16ae6c-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.570 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:01.570 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:01.571 DEBUG:boto:path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.571 DEBUG:boto:auth_path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.571 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.571 DEBUG:boto:Path: /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.571 DEBUG:boto:Data: 2024-04-19T23:24:01.571 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.571 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.571 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.571 DEBUG:boto:Params: {} 2024-04-19T23:24:01.571 DEBUG:boto:Token: None 2024-04-19T23:24:01.571 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.572 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HOgx/GgO3M+uT4n/cNdmQmrWwrQ= 2024-04-19T23:24:01.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HOgx/GgO3M+uT4n/cNdmQmrWwrQ=', 'Content-Length': '0'} 2024-04-19T23:24:01.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f79881db1c8c6a68-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.574 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:01.574 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:01.574 DEBUG:boto:path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.574 DEBUG:boto:auth_path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.574 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.574 DEBUG:boto:Path: /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.574 DEBUG:boto:Data: 2024-04-19T23:24:01.575 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.575 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.575 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.575 DEBUG:boto:Params: {} 2024-04-19T23:24:01.575 DEBUG:boto:Token: None 2024-04-19T23:24:01.575 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.575 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HOgx/GgO3M+uT4n/cNdmQmrWwrQ= 2024-04-19T23:24:01.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HOgx/GgO3M+uT4n/cNdmQmrWwrQ=', 'Content-Length': '0'} 2024-04-19T23:24:01.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008e863a3fe2a0431-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:01.578 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:01.578 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:01.578 DEBUG:boto:path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.578 DEBUG:boto:auth_path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.578 DEBUG:boto:Method: GET 2024-04-19T23:24:01.578 DEBUG:boto:Path: /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.578 DEBUG:boto:Data: 2024-04-19T23:24:01.578 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.578 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.579 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.579 DEBUG:boto:Params: {} 2024-04-19T23:24:01.579 DEBUG:boto:Token: None 2024-04-19T23:24:01.579 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.579 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:30c5vWOzkWJvamP0xWWQbZnLlqY= 2024-04-19T23:24:01.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:30c5vWOzkWJvamP0xWWQbZnLlqY=', 'Content-Length': '0'} 2024-04-19T23:24:01.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0b5b65ed0dcbb08-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.581 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:01.581 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:01.581 DEBUG:boto:path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.581 DEBUG:boto:auth_path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.581 DEBUG:boto:Method: GET 2024-04-19T23:24:01.581 DEBUG:boto:Path: /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.582 DEBUG:boto:Data: 2024-04-19T23:24:01.582 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.582 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.582 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.582 DEBUG:boto:Params: {} 2024-04-19T23:24:01.582 DEBUG:boto:Token: None 2024-04-19T23:24:01.582 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:01.582 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:30c5vWOzkWJvamP0xWWQbZnLlqY= 2024-04-19T23:24:01.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:30c5vWOzkWJvamP0xWWQbZnLlqY=', 'Content-Length': '0'} 2024-04-19T23:24:01.584 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061808f217868eeca-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.584 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.585 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.585 DEBUG:boto:path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:01.585 DEBUG:boto:auth_path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:01.585 DEBUG:boto:Method: GET 2024-04-19T23:24:01.586 DEBUG:boto:Path: /aktlji-11/?prefix=_myobj 2024-04-19T23:24:01.586 DEBUG:boto:Data: 2024-04-19T23:24:01.586 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.586 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.586 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.586 DEBUG:boto:Params: {} 2024-04-19T23:24:01.586 DEBUG:boto:Token: None 2024-04-19T23:24:01.586 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.587 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eb0490334535da5-006622fd11-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.589 DEBUG:boto:b'aktlji-11_myobj1000false_myobj2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.589 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.590 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.590 DEBUG:boto:path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:01.590 DEBUG:boto:auth_path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:01.590 DEBUG:boto:Method: GET 2024-04-19T23:24:01.590 DEBUG:boto:Path: /aktlji-11/?prefix=_myobj 2024-04-19T23:24:01.590 DEBUG:boto:Data: 2024-04-19T23:24:01.590 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.590 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.590 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.590 DEBUG:boto:Params: {} 2024-04-19T23:24:01.590 DEBUG:boto:Token: None 2024-04-19T23:24:01.591 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.591 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e09d1ab75219166d-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.594 DEBUG:boto:b'aktlji-11_myobj1000false_myobj2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.594 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:01.594 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:01.594 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.594 DEBUG:boto:Path: /aktlji-11/_myobj 2024-04-19T23:24:01.594 DEBUG:boto:Data: 2024-04-19T23:24:01.595 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.595 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.595 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.595 DEBUG:boto:Params: {} 2024-04-19T23:24:01.595 DEBUG:boto:Token: None 2024-04-19T23:24:01.595 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/_myobj 2024-04-19T23:24:01.595 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0X8B/8dcZ2XJbHEe4uBnLoC6uUA= 2024-04-19T23:24:01.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0X8B/8dcZ2XJbHEe4uBnLoC6uUA=', 'Content-Length': '0'} 2024-04-19T23:24:01.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037c112976a53a2c7-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.597 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:01.597 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:01.597 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.597 DEBUG:boto:Path: /aktlji-11/_myobj 2024-04-19T23:24:01.598 DEBUG:boto:Data: 2024-04-19T23:24:01.598 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.598 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.598 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.598 DEBUG:boto:Params: {} 2024-04-19T23:24:01.598 DEBUG:boto:Token: None 2024-04-19T23:24:01.598 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/_myobj 2024-04-19T23:24:01.598 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0X8B/8dcZ2XJbHEe4uBnLoC6uUA= 2024-04-19T23:24:01.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0X8B/8dcZ2XJbHEe4uBnLoC6uUA=', 'Content-Length': '0'} 2024-04-19T23:24:01.600 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a533117ae2bd597-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:01.601 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:01.601 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:01.601 DEBUG:boto:Method: GET 2024-04-19T23:24:01.601 DEBUG:boto:Path: /aktlji-11/_myobj 2024-04-19T23:24:01.601 DEBUG:boto:Data: 2024-04-19T23:24:01.601 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.601 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.601 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.601 DEBUG:boto:Params: {} 2024-04-19T23:24:01.601 DEBUG:boto:Token: None 2024-04-19T23:24:01.602 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/_myobj 2024-04-19T23:24:01.602 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gD0h4qNlTaHdQLtHmXHU4IwnVAI= 2024-04-19T23:24:01.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gD0h4qNlTaHdQLtHmXHU4IwnVAI=', 'Content-Length': '0'} 2024-04-19T23:24:01.603 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006253e24e590f01a5-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.604 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:01.604 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:01.604 DEBUG:boto:Method: GET 2024-04-19T23:24:01.604 DEBUG:boto:Path: /aktlji-11/_myobj 2024-04-19T23:24:01.604 DEBUG:boto:Data: 2024-04-19T23:24:01.604 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.604 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.604 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.604 DEBUG:boto:Params: {} 2024-04-19T23:24:01.605 DEBUG:boto:Token: None 2024-04-19T23:24:01.605 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/_myobj 2024-04-19T23:24:01.605 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gD0h4qNlTaHdQLtHmXHU4IwnVAI= 2024-04-19T23:24:01.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gD0h4qNlTaHdQLtHmXHU4IwnVAI=', 'Content-Length': '0'} 2024-04-19T23:24:01.607 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3036d66fffd202d-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.607 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.607 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.607 DEBUG:boto:path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:01.607 DEBUG:boto:auth_path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:01.607 DEBUG:boto:Method: GET 2024-04-19T23:24:01.607 DEBUG:boto:Path: /aktlji-11/?prefix=_myobj 2024-04-19T23:24:01.607 DEBUG:boto:Data: 2024-04-19T23:24:01.608 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.608 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.608 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.608 DEBUG:boto:Params: {} 2024-04-19T23:24:01.608 DEBUG:boto:Token: None 2024-04-19T23:24:01.608 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.608 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f69d2b00b673c063-006622fd11-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.611 DEBUG:boto:b'aktlji-11_myobj1000false_myobj2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.611 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.611 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.611 DEBUG:boto:path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:01.611 DEBUG:boto:auth_path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:01.611 DEBUG:boto:Method: GET 2024-04-19T23:24:01.611 DEBUG:boto:Path: /aktlji-11/?prefix=_myobj 2024-04-19T23:24:01.612 DEBUG:boto:Data: 2024-04-19T23:24:01.612 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.612 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.612 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.612 DEBUG:boto:Params: {} 2024-04-19T23:24:01.612 DEBUG:boto:Token: None 2024-04-19T23:24:01.612 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.612 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac6fccd98e27416f-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.615 DEBUG:boto:b'aktlji-11_myobj1000false_myobj2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:01.615 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:01.616 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:01.616 DEBUG:boto:path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.616 DEBUG:boto:auth_path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.616 DEBUG:boto:Method: GET 2024-04-19T23:24:01.616 DEBUG:boto:Path: /aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.616 DEBUG:boto:Data: 2024-04-19T23:24:01.616 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.616 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.616 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.616 DEBUG:boto:Params: {} 2024-04-19T23:24:01.616 DEBUG:boto:Token: None 2024-04-19T23:24:01.617 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.617 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ImwZDHAA+cdCG8FwSSu9Nqx3X7k= 2024-04-19T23:24:01.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ImwZDHAA+cdCG8FwSSu9Nqx3X7k=', 'Content-Length': '0'} 2024-04-19T23:24:01.618 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2fcb821ef49128d-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.662 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:01.663 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:01.663 DEBUG:boto:path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.663 DEBUG:boto:auth_path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.663 DEBUG:boto:Method: GET 2024-04-19T23:24:01.663 DEBUG:boto:Path: /aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.663 DEBUG:boto:Data: 2024-04-19T23:24:01.663 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.663 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.663 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.663 DEBUG:boto:Params: {} 2024-04-19T23:24:01.663 DEBUG:boto:Token: None 2024-04-19T23:24:01.664 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.664 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ImwZDHAA+cdCG8FwSSu9Nqx3X7k= 2024-04-19T23:24:01.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ImwZDHAA+cdCG8FwSSu9Nqx3X7k=', 'Content-Length': '0'} 2024-04-19T23:24:01.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd4404b50415d13d-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.666 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:01.666 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:01.666 DEBUG:boto:path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.666 DEBUG:boto:auth_path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.667 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.667 DEBUG:boto:Path: /aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.667 DEBUG:boto:Data: 2024-04-19T23:24:01.667 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.667 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.667 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.667 DEBUG:boto:Params: {} 2024-04-19T23:24:01.667 DEBUG:boto:Token: None 2024-04-19T23:24:01.667 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.667 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UU94nkVYCjXYvdk9kSqjAEe8084= 2024-04-19T23:24:01.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UU94nkVYCjXYvdk9kSqjAEe8084=', 'Content-Length': '0'} 2024-04-19T23:24:01.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a881b47f94de2563-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.669 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:01.670 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:01.670 DEBUG:boto:path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.670 DEBUG:boto:auth_path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.670 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.670 DEBUG:boto:Path: /aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.670 DEBUG:boto:Data: 2024-04-19T23:24:01.670 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.670 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.670 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.670 DEBUG:boto:Params: {} 2024-04-19T23:24:01.670 DEBUG:boto:Token: None 2024-04-19T23:24:01.671 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.671 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UU94nkVYCjXYvdk9kSqjAEe8084= 2024-04-19T23:24:01.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UU94nkVYCjXYvdk9kSqjAEe8084=', 'Content-Length': '0'} 2024-04-19T23:24:01.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebb90cc0d50836b7-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:01.673 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:01.673 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:01.673 DEBUG:boto:path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.673 DEBUG:boto:auth_path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.674 DEBUG:boto:Method: GET 2024-04-19T23:24:01.674 DEBUG:boto:Path: /aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.674 DEBUG:boto:Data: 2024-04-19T23:24:01.674 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.674 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.674 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.674 DEBUG:boto:Params: {} 2024-04-19T23:24:01.674 DEBUG:boto:Token: None 2024-04-19T23:24:01.674 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.675 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ImwZDHAA+cdCG8FwSSu9Nqx3X7k= 2024-04-19T23:24:01.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ImwZDHAA+cdCG8FwSSu9Nqx3X7k=', 'Content-Length': '0'} 2024-04-19T23:24:01.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbc120f299b2ad05-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.677 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:01.677 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:01.677 DEBUG:boto:path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.677 DEBUG:boto:auth_path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.677 DEBUG:boto:Method: GET 2024-04-19T23:24:01.677 DEBUG:boto:Path: /aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.677 DEBUG:boto:Data: 2024-04-19T23:24:01.677 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.677 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.677 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.678 DEBUG:boto:Params: {} 2024-04-19T23:24:01.678 DEBUG:boto:Token: None 2024-04-19T23:24:01.678 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/myobj?versionId=null 2024-04-19T23:24:01.678 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ImwZDHAA+cdCG8FwSSu9Nqx3X7k= 2024-04-19T23:24:01.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ImwZDHAA+cdCG8FwSSu9Nqx3X7k=', 'Content-Length': '0'} 2024-04-19T23:24:01.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a12d4582c1b077a-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.680 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.680 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.680 DEBUG:boto:path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:01.680 DEBUG:boto:auth_path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:01.680 DEBUG:boto:Method: GET 2024-04-19T23:24:01.680 DEBUG:boto:Path: /aktlji-11/?prefix=myobj 2024-04-19T23:24:01.680 DEBUG:boto:Data: 2024-04-19T23:24:01.681 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.681 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.681 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.681 DEBUG:boto:Params: {} 2024-04-19T23:24:01.681 DEBUG:boto:Token: None 2024-04-19T23:24:01.681 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.681 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad04c64d74ddb990-006622fd11-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.684 DEBUG:boto:b'aktlji-11myobj1000falsemyobj2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.684 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.684 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.684 DEBUG:boto:path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:01.684 DEBUG:boto:auth_path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:01.684 DEBUG:boto:Method: GET 2024-04-19T23:24:01.685 DEBUG:boto:Path: /aktlji-11/?prefix=myobj 2024-04-19T23:24:01.685 DEBUG:boto:Data: 2024-04-19T23:24:01.685 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.685 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.685 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.685 DEBUG:boto:Params: {} 2024-04-19T23:24:01.685 DEBUG:boto:Token: None 2024-04-19T23:24:01.685 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.685 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005616c1748793e813-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.688 DEBUG:boto:b'aktlji-11myobj1000falsemyobj2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.688 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:01.688 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:01.689 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.689 DEBUG:boto:Path: /aktlji-11/myobj 2024-04-19T23:24:01.689 DEBUG:boto:Data: 2024-04-19T23:24:01.689 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.689 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.689 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.689 DEBUG:boto:Params: {} 2024-04-19T23:24:01.689 DEBUG:boto:Token: None 2024-04-19T23:24:01.689 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/myobj 2024-04-19T23:24:01.689 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8ZVg3txi9K6ULZ1o8xW2dOhNQVo= 2024-04-19T23:24:01.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8ZVg3txi9K6ULZ1o8xW2dOhNQVo=', 'Content-Length': '0'} 2024-04-19T23:24:01.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcfe6dcc53be31a9-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.691 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:01.691 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:01.692 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.692 DEBUG:boto:Path: /aktlji-11/myobj 2024-04-19T23:24:01.692 DEBUG:boto:Data: 2024-04-19T23:24:01.692 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.692 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.692 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.692 DEBUG:boto:Params: {} 2024-04-19T23:24:01.692 DEBUG:boto:Token: None 2024-04-19T23:24:01.692 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/myobj 2024-04-19T23:24:01.693 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8ZVg3txi9K6ULZ1o8xW2dOhNQVo= 2024-04-19T23:24:01.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8ZVg3txi9K6ULZ1o8xW2dOhNQVo=', 'Content-Length': '0'} 2024-04-19T23:24:01.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cb05124647f7de7-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:01.695 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:01.695 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:01.695 DEBUG:boto:Method: GET 2024-04-19T23:24:01.695 DEBUG:boto:Path: /aktlji-11/myobj 2024-04-19T23:24:01.695 DEBUG:boto:Data: 2024-04-19T23:24:01.695 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.695 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.695 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.695 DEBUG:boto:Params: {} 2024-04-19T23:24:01.695 DEBUG:boto:Token: None 2024-04-19T23:24:01.696 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/myobj 2024-04-19T23:24:01.696 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GpRTpSIg9aVWqkbsWc+Z6obRYwQ= 2024-04-19T23:24:01.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GpRTpSIg9aVWqkbsWc+Z6obRYwQ=', 'Content-Length': '0'} 2024-04-19T23:24:01.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013852f64cbc0b7de-006622fd11-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.738 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:01.738 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:01.739 DEBUG:boto:Method: GET 2024-04-19T23:24:01.739 DEBUG:boto:Path: /aktlji-11/myobj 2024-04-19T23:24:01.739 DEBUG:boto:Data: 2024-04-19T23:24:01.739 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.739 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.739 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.739 DEBUG:boto:Params: {} 2024-04-19T23:24:01.739 DEBUG:boto:Token: None 2024-04-19T23:24:01.739 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/myobj 2024-04-19T23:24:01.740 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GpRTpSIg9aVWqkbsWc+Z6obRYwQ= 2024-04-19T23:24:01.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GpRTpSIg9aVWqkbsWc+Z6obRYwQ=', 'Content-Length': '0'} 2024-04-19T23:24:01.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c538dd671bd28cd-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.742 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.742 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.742 DEBUG:boto:path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:01.742 DEBUG:boto:auth_path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:01.742 DEBUG:boto:Method: GET 2024-04-19T23:24:01.742 DEBUG:boto:Path: /aktlji-11/?prefix=myobj 2024-04-19T23:24:01.742 DEBUG:boto:Data: 2024-04-19T23:24:01.742 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.743 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:01.743 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.743 DEBUG:boto:Params: {} 2024-04-19T23:24:01.743 DEBUG:boto:Token: None 2024-04-19T23:24:01.743 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.743 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cc8297c1f18cdca-006622fd11-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.746 DEBUG:boto:b'aktlji-11myobj1000falsemyobj2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.746 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.746 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.746 DEBUG:boto:path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:01.746 DEBUG:boto:auth_path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:01.747 DEBUG:boto:Method: GET 2024-04-19T23:24:01.747 DEBUG:boto:Path: /aktlji-11/?prefix=myobj 2024-04-19T23:24:01.747 DEBUG:boto:Data: 2024-04-19T23:24:01.747 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.747 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.747 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.747 DEBUG:boto:Params: {} 2024-04-19T23:24:01.747 DEBUG:boto:Token: None 2024-04-19T23:24:01.747 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.747 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035c3a41e2789851d-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.750 DEBUG:boto:b'aktlji-11myobj1000falsemyobj2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.750 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-11 zones={test-zone1, test-zone2} 2024-04-19T23:24:01.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:01.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:01.751 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:24:01.818 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:01.818 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:01.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:01.866+0000 7fcda99a3780 1 bucket sync caught up with source: 2024-04-19T23:24:01.862 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , 00000000001.6.6, , , , 00000000001.8.6, 00000000001.17.6, , ] 2024-04-19T23:24:01.862 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000001.10.6, , 00000000001.6.6, , , , 00000000001.8.6, 00000000001.17.6, , ] 2024-04-19T23:24:01.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:01.866+0000 7fcda99a3780 0 bucket checkpoint complete 2024-04-19T23:24:01.876 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-11 zones={test-zone3, test-zone2} 2024-04-19T23:24:01.876 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.876 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.876 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.876 DEBUG:boto:Path: /aktlji-11/ 2024-04-19T23:24:01.876 DEBUG:boto:Data: 2024-04-19T23:24:01.877 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.877 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.877 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.877 DEBUG:boto:Params: {} 2024-04-19T23:24:01.877 DEBUG:boto:Token: None 2024-04-19T23:24:01.877 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.877 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2zlZpXUx9mPdqukB4mYX/iSg/3s= 2024-04-19T23:24:01.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2zlZpXUx9mPdqukB4mYX/iSg/3s=', 'Content-Length': '0'} 2024-04-19T23:24:01.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000268f523a7c825f42-006622fd11-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.880 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.880 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.880 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.880 DEBUG:boto:Path: /aktlji-11/ 2024-04-19T23:24:01.881 DEBUG:boto:Data: 2024-04-19T23:24:01.881 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.881 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.881 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.881 DEBUG:boto:Params: {} 2024-04-19T23:24:01.881 DEBUG:boto:Token: None 2024-04-19T23:24:01.881 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.881 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2zlZpXUx9mPdqukB4mYX/iSg/3s= 2024-04-19T23:24:01.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2zlZpXUx9mPdqukB4mYX/iSg/3s=', 'Content-Length': '0'} 2024-04-19T23:24:01.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092c618db4bfde40b-006622fd11-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.884 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:24:01.884 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.884 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.884 DEBUG:boto:path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.884 DEBUG:boto:auth_path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.884 DEBUG:boto:Method: GET 2024-04-19T23:24:01.885 DEBUG:boto:Path: /aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.885 DEBUG:boto:Data: 2024-04-19T23:24:01.885 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.885 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.885 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.885 DEBUG:boto:Params: {} 2024-04-19T23:24:01.885 DEBUG:boto:Token: None 2024-04-19T23:24:01.885 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/?versions 2024-04-19T23:24:01.885 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY= 2024-04-19T23:24:01.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY=', 'Content-Length': '0'} 2024-04-19T23:24:01.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bf56c624b69314e-006622fd11-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.888 DEBUG:boto:b'aktlji-11999false&nulltrue2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.888 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-19T23:24:01.888 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-19T23:24:01.889 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-19T23:24:01.889 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:01.889 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:24:01.889 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.889 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.889 DEBUG:boto:path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.889 DEBUG:boto:auth_path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.889 DEBUG:boto:Method: GET 2024-04-19T23:24:01.889 DEBUG:boto:Path: /aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.889 DEBUG:boto:Data: 2024-04-19T23:24:01.889 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.890 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.890 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.890 DEBUG:boto:Params: {} 2024-04-19T23:24:01.890 DEBUG:boto:Token: None 2024-04-19T23:24:01.890 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/?versions 2024-04-19T23:24:01.890 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY= 2024-04-19T23:24:01.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY=', 'Content-Length': '0'} 2024-04-19T23:24:01.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086567c1a9683c47f-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.893 DEBUG:boto:b'aktlji-11999false&nulltrue2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.893 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-19T23:24:01.893 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-19T23:24:01.893 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-19T23:24:01.894 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:01.894 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.894 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.894 DEBUG:boto:path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.894 DEBUG:boto:auth_path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.894 DEBUG:boto:Method: GET 2024-04-19T23:24:01.895 DEBUG:boto:Path: /aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.895 DEBUG:boto:Data: 2024-04-19T23:24:01.895 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.895 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.895 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.895 DEBUG:boto:Params: {} 2024-04-19T23:24:01.895 DEBUG:boto:Token: None 2024-04-19T23:24:01.895 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/?versions 2024-04-19T23:24:01.896 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY= 2024-04-19T23:24:01.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY=', 'Content-Length': '0'} 2024-04-19T23:24:01.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdb79c65cc1e3746-006622fd11-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.898 DEBUG:boto:b'aktlji-11999false&nulltrue2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.899 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.901 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.901 DEBUG:boto:path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.901 DEBUG:boto:auth_path=/aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.901 DEBUG:boto:Method: GET 2024-04-19T23:24:01.901 DEBUG:boto:Path: /aktlji-11/?versions&max-keys=999 2024-04-19T23:24:01.901 DEBUG:boto:Data: 2024-04-19T23:24:01.901 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.901 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.901 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.902 DEBUG:boto:Params: {} 2024-04-19T23:24:01.902 DEBUG:boto:Token: None 2024-04-19T23:24:01.902 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/?versions 2024-04-19T23:24:01.902 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY= 2024-04-19T23:24:01.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1dzIspw9cuA3zXqoBs8yh+Y/btY=', 'Content-Length': '0'} 2024-04-19T23:24:01.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b02256bffc54155b-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.905 DEBUG:boto:b'aktlji-11999false&nulltrue2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:01.905 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.906 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.906 DEBUG:boto:path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.906 DEBUG:boto:auth_path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.906 DEBUG:boto:Method: GET 2024-04-19T23:24:01.906 DEBUG:boto:Path: /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.906 DEBUG:boto:Data: 2024-04-19T23:24:01.906 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.906 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.907 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.907 DEBUG:boto:Params: {} 2024-04-19T23:24:01.907 DEBUG:boto:Token: None 2024-04-19T23:24:01.907 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.907 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E= 2024-04-19T23:24:01.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E=', 'Content-Length': '0'} 2024-04-19T23:24:01.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002aaba5872b529d16-006622fd11-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.954 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.955 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.955 DEBUG:boto:path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.955 DEBUG:boto:auth_path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.955 DEBUG:boto:Method: GET 2024-04-19T23:24:01.955 DEBUG:boto:Path: /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.955 DEBUG:boto:Data: 2024-04-19T23:24:01.955 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.955 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.955 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.955 DEBUG:boto:Params: {} 2024-04-19T23:24:01.956 DEBUG:boto:Token: None 2024-04-19T23:24:01.956 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.956 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E= 2024-04-19T23:24:01.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E=', 'Content-Length': '0'} 2024-04-19T23:24:01.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8317f259262c20e-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.958 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.959 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.959 DEBUG:boto:path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.959 DEBUG:boto:auth_path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.959 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.959 DEBUG:boto:Path: /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.959 DEBUG:boto:Data: 2024-04-19T23:24:01.959 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.959 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.959 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.959 DEBUG:boto:Params: {} 2024-04-19T23:24:01.959 DEBUG:boto:Token: None 2024-04-19T23:24:01.960 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.960 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:v+Xq3T/ja5kHuNO+1egdGDjP85s= 2024-04-19T23:24:01.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:v+Xq3T/ja5kHuNO+1egdGDjP85s=', 'Content-Length': '0'} 2024-04-19T23:24:01.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1f1384dff8d87f8-006622fd11-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.962 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.962 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.962 DEBUG:boto:path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.962 DEBUG:boto:auth_path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.963 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.963 DEBUG:boto:Path: /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.963 DEBUG:boto:Data: 2024-04-19T23:24:01.963 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.963 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.963 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.963 DEBUG:boto:Params: {} 2024-04-19T23:24:01.963 DEBUG:boto:Token: None 2024-04-19T23:24:01.963 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.963 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:v+Xq3T/ja5kHuNO+1egdGDjP85s= 2024-04-19T23:24:01.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:v+Xq3T/ja5kHuNO+1egdGDjP85s=', 'Content-Length': '0'} 2024-04-19T23:24:01.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee6f363154e4c584-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:01.966 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.966 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.966 DEBUG:boto:path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.966 DEBUG:boto:auth_path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.966 DEBUG:boto:Method: GET 2024-04-19T23:24:01.966 DEBUG:boto:Path: /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.966 DEBUG:boto:Data: 2024-04-19T23:24:01.967 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.967 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.967 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.967 DEBUG:boto:Params: {} 2024-04-19T23:24:01.967 DEBUG:boto:Token: None 2024-04-19T23:24:01.967 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.967 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E= 2024-04-19T23:24:01.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E=', 'Content-Length': '0'} 2024-04-19T23:24:01.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f0c38e79aa73a80-006622fd11-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.969 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.969 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.970 DEBUG:boto:path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.970 DEBUG:boto:auth_path=/aktlji-11/%26?versionId=null 2024-04-19T23:24:01.970 DEBUG:boto:Method: GET 2024-04-19T23:24:01.970 DEBUG:boto:Path: /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.970 DEBUG:boto:Data: 2024-04-19T23:24:01.970 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.970 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.970 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.970 DEBUG:boto:Params: {} 2024-04-19T23:24:01.970 DEBUG:boto:Token: None 2024-04-19T23:24:01.971 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26?versionId=null 2024-04-19T23:24:01.971 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E= 2024-04-19T23:24:01.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:45Kfn+AhLktxOc8VOmDGlnhz95E=', 'Content-Length': '0'} 2024-04-19T23:24:01.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000379ce8d9de83995b-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.973 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.973 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.973 DEBUG:boto:path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.973 DEBUG:boto:auth_path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.973 DEBUG:boto:Method: GET 2024-04-19T23:24:01.973 DEBUG:boto:Path: /aktlji-11/?prefix=%26 2024-04-19T23:24:01.973 DEBUG:boto:Data: 2024-04-19T23:24:01.974 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.974 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.974 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.974 DEBUG:boto:Params: {} 2024-04-19T23:24:01.974 DEBUG:boto:Token: None 2024-04-19T23:24:01.974 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.974 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acdc2de0a321e2e4-006622fd11-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.977 DEBUG:boto:b'aktlji-11&1000false&2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.977 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.977 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.977 DEBUG:boto:path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.977 DEBUG:boto:auth_path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.977 DEBUG:boto:Method: GET 2024-04-19T23:24:01.977 DEBUG:boto:Path: /aktlji-11/?prefix=%26 2024-04-19T23:24:01.978 DEBUG:boto:Data: 2024-04-19T23:24:01.978 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.978 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.978 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.978 DEBUG:boto:Params: {} 2024-04-19T23:24:01.978 DEBUG:boto:Token: None 2024-04-19T23:24:01.978 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.978 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6b0502a8800baf5-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.981 DEBUG:boto:b'aktlji-11&1000false&2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.981 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.981 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.981 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.981 DEBUG:boto:Path: /aktlji-11/%26 2024-04-19T23:24:01.982 DEBUG:boto:Data: 2024-04-19T23:24:01.982 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.982 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.982 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.982 DEBUG:boto:Params: {} 2024-04-19T23:24:01.982 DEBUG:boto:Token: None 2024-04-19T23:24:01.982 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26 2024-04-19T23:24:01.982 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:36sVR5ZwBByGaiHyMU3wr0fUXnA= 2024-04-19T23:24:01.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:36sVR5ZwBByGaiHyMU3wr0fUXnA=', 'Content-Length': '0'} 2024-04-19T23:24:01.984 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007174839f6b0d7b0-006622fd11-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.984 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.984 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.984 DEBUG:boto:Method: HEAD 2024-04-19T23:24:01.985 DEBUG:boto:Path: /aktlji-11/%26 2024-04-19T23:24:01.985 DEBUG:boto:Data: 2024-04-19T23:24:01.985 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.985 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.985 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.985 DEBUG:boto:Params: {} 2024-04-19T23:24:01.985 DEBUG:boto:Token: None 2024-04-19T23:24:01.985 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26 2024-04-19T23:24:01.985 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:36sVR5ZwBByGaiHyMU3wr0fUXnA= 2024-04-19T23:24:01.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:36sVR5ZwBByGaiHyMU3wr0fUXnA=', 'Content-Length': '0'} 2024-04-19T23:24:01.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f40f6c2070ea67f-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:01.987 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.987 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.988 DEBUG:boto:Method: GET 2024-04-19T23:24:01.988 DEBUG:boto:Path: /aktlji-11/%26 2024-04-19T23:24:01.988 DEBUG:boto:Data: 2024-04-19T23:24:01.988 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.988 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.988 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.988 DEBUG:boto:Params: {} 2024-04-19T23:24:01.988 DEBUG:boto:Token: None 2024-04-19T23:24:01.989 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26 2024-04-19T23:24:01.989 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:W/K26wTnsy613JAaKfHBx2WQrlg= 2024-04-19T23:24:01.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W/K26wTnsy613JAaKfHBx2WQrlg=', 'Content-Length': '0'} 2024-04-19T23:24:01.990 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f699c333f75f3ab-006622fd11-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.991 DEBUG:boto:path=/aktlji-11/%26 2024-04-19T23:24:01.991 DEBUG:boto:auth_path=/aktlji-11/%26 2024-04-19T23:24:01.991 DEBUG:boto:Method: GET 2024-04-19T23:24:01.991 DEBUG:boto:Path: /aktlji-11/%26 2024-04-19T23:24:01.991 DEBUG:boto:Data: 2024-04-19T23:24:01.991 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.991 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:01.991 DEBUG:boto:Port: 8000 2024-04-19T23:24:01.991 DEBUG:boto:Params: {} 2024-04-19T23:24:01.992 DEBUG:boto:Token: None 2024-04-19T23:24:01.992 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/%26 2024-04-19T23:24:01.992 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:W/K26wTnsy613JAaKfHBx2WQrlg= 2024-04-19T23:24:01.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W/K26wTnsy613JAaKfHBx2WQrlg=', 'Content-Length': '0'} 2024-04-19T23:24:01.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a41a357458f0850b-006622fd11-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.994 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.994 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.994 DEBUG:boto:path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.994 DEBUG:boto:auth_path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.995 DEBUG:boto:Method: GET 2024-04-19T23:24:01.995 DEBUG:boto:Path: /aktlji-11/?prefix=%26 2024-04-19T23:24:01.995 DEBUG:boto:Data: 2024-04-19T23:24:01.995 DEBUG:boto:Headers: {} 2024-04-19T23:24:01.995 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:01.995 DEBUG:boto:Port: 8001 2024-04-19T23:24:01.995 DEBUG:boto:Params: {} 2024-04-19T23:24:01.995 DEBUG:boto:Token: None 2024-04-19T23:24:01.995 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:01 GMT /aktlji-11/ 2024-04-19T23:24:01.995 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU= 2024-04-19T23:24:01.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k5q0PFmvE+JhQ6jLNOtT+9gTTEU=', 'Content-Length': '0'} 2024-04-19T23:24:01.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eafbdc38ce8254ff-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:01.998 DEBUG:boto:b'aktlji-11&1000false&2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:01.999 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:01.999 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:01.999 DEBUG:boto:path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.999 DEBUG:boto:auth_path=/aktlji-11/?prefix=%26 2024-04-19T23:24:01.999 DEBUG:boto:Method: GET 2024-04-19T23:24:01.999 DEBUG:boto:Path: /aktlji-11/?prefix=%26 2024-04-19T23:24:01.999 DEBUG:boto:Data: 2024-04-19T23:24:01.999 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.000 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.000 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.000 DEBUG:boto:Params: {} 2024-04-19T23:24:02.000 DEBUG:boto:Token: None 2024-04-19T23:24:02.000 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.000 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6989d8560bca82b-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.003 DEBUG:boto:b'aktlji-11&1000false&2024-04-19T23:23:14.243Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:02.003 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:02.003 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:02.003 DEBUG:boto:path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.004 DEBUG:boto:auth_path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.004 DEBUG:boto:Method: GET 2024-04-19T23:24:02.004 DEBUG:boto:Path: /aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.004 DEBUG:boto:Data: 2024-04-19T23:24:02.004 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.004 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.004 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.004 DEBUG:boto:Params: {} 2024-04-19T23:24:02.004 DEBUG:boto:Token: None 2024-04-19T23:24:02.004 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.005 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g1PWa1xMlCb2eNCs9Pj/sMGryxA= 2024-04-19T23:24:02.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g1PWa1xMlCb2eNCs9Pj/sMGryxA=', 'Content-Length': '0'} 2024-04-19T23:24:02.007 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059d518c60347ddf9-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.050 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:02.051 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:02.051 DEBUG:boto:path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.051 DEBUG:boto:auth_path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.051 DEBUG:boto:Method: GET 2024-04-19T23:24:02.051 DEBUG:boto:Path: /aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.051 DEBUG:boto:Data: 2024-04-19T23:24:02.051 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.051 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.051 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.051 DEBUG:boto:Params: {} 2024-04-19T23:24:02.052 DEBUG:boto:Token: None 2024-04-19T23:24:02.052 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.052 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g1PWa1xMlCb2eNCs9Pj/sMGryxA= 2024-04-19T23:24:02.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g1PWa1xMlCb2eNCs9Pj/sMGryxA=', 'Content-Length': '0'} 2024-04-19T23:24:02.054 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000639b891b62202137-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.054 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:02.054 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:02.054 DEBUG:boto:path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.054 DEBUG:boto:auth_path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.055 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.055 DEBUG:boto:Path: /aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.055 DEBUG:boto:Data: 2024-04-19T23:24:02.055 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.055 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.055 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.055 DEBUG:boto:Params: {} 2024-04-19T23:24:02.055 DEBUG:boto:Token: None 2024-04-19T23:24:02.055 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.055 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jfloz3P3HI3TSRBGC7d8NB2bv2A= 2024-04-19T23:24:02.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jfloz3P3HI3TSRBGC7d8NB2bv2A=', 'Content-Length': '0'} 2024-04-19T23:24:02.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000250685470a1684cc-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.057 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:02.057 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:02.058 DEBUG:boto:path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.058 DEBUG:boto:auth_path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.058 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.058 DEBUG:boto:Path: /aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.058 DEBUG:boto:Data: 2024-04-19T23:24:02.058 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.058 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.058 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.058 DEBUG:boto:Params: {} 2024-04-19T23:24:02.058 DEBUG:boto:Token: None 2024-04-19T23:24:02.059 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.059 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jfloz3P3HI3TSRBGC7d8NB2bv2A= 2024-04-19T23:24:02.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jfloz3P3HI3TSRBGC7d8NB2bv2A=', 'Content-Length': '0'} 2024-04-19T23:24:02.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007af9dc03e7d16b8a-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:02.061 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:02.061 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:02.061 DEBUG:boto:path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.061 DEBUG:boto:auth_path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.062 DEBUG:boto:Method: GET 2024-04-19T23:24:02.062 DEBUG:boto:Path: /aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.062 DEBUG:boto:Data: 2024-04-19T23:24:02.062 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.062 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.062 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.062 DEBUG:boto:Params: {} 2024-04-19T23:24:02.062 DEBUG:boto:Token: None 2024-04-19T23:24:02.062 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.062 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g1PWa1xMlCb2eNCs9Pj/sMGryxA= 2024-04-19T23:24:02.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g1PWa1xMlCb2eNCs9Pj/sMGryxA=', 'Content-Length': '0'} 2024-04-19T23:24:02.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008b99f7882430cee-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.064 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:02.065 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:02.065 DEBUG:boto:path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.065 DEBUG:boto:auth_path=/aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.065 DEBUG:boto:Method: GET 2024-04-19T23:24:02.065 DEBUG:boto:Path: /aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.065 DEBUG:boto:Data: 2024-04-19T23:24:02.065 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.065 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.065 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.066 DEBUG:boto:Params: {} 2024-04-19T23:24:02.066 DEBUG:boto:Token: None 2024-04-19T23:24:02.066 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/%3A?versionId=null 2024-04-19T23:24:02.066 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g1PWa1xMlCb2eNCs9Pj/sMGryxA= 2024-04-19T23:24:02.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g1PWa1xMlCb2eNCs9Pj/sMGryxA=', 'Content-Length': '0'} 2024-04-19T23:24:02.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001504f9379ef7c7c-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.068 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:02.069 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:02.069 DEBUG:boto:path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:02.069 DEBUG:boto:auth_path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:02.069 DEBUG:boto:Method: GET 2024-04-19T23:24:02.069 DEBUG:boto:Path: /aktlji-11/?prefix=%3A 2024-04-19T23:24:02.069 DEBUG:boto:Data: 2024-04-19T23:24:02.069 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.069 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.069 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.069 DEBUG:boto:Params: {} 2024-04-19T23:24:02.070 DEBUG:boto:Token: None 2024-04-19T23:24:02.070 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.070 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbd9f5185570370d-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.073 DEBUG:boto:b'aktlji-11:1000false:2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.073 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:02.074 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:02.074 DEBUG:boto:path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:02.074 DEBUG:boto:auth_path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:02.074 DEBUG:boto:Method: GET 2024-04-19T23:24:02.074 DEBUG:boto:Path: /aktlji-11/?prefix=%3A 2024-04-19T23:24:02.074 DEBUG:boto:Data: 2024-04-19T23:24:02.074 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.074 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.075 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.075 DEBUG:boto:Params: {} 2024-04-19T23:24:02.075 DEBUG:boto:Token: None 2024-04-19T23:24:02.075 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.075 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064a78d0cd59dda99-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.078 DEBUG:boto:b'aktlji-11:1000false:2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.078 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:02.078 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:02.078 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.078 DEBUG:boto:Path: /aktlji-11/%3A 2024-04-19T23:24:02.078 DEBUG:boto:Data: 2024-04-19T23:24:02.078 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.078 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.079 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.079 DEBUG:boto:Params: {} 2024-04-19T23:24:02.079 DEBUG:boto:Token: None 2024-04-19T23:24:02.079 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/%3A 2024-04-19T23:24:02.079 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OPC6u0qqNO2e7l2fePFxdA3TeGo= 2024-04-19T23:24:02.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OPC6u0qqNO2e7l2fePFxdA3TeGo=', 'Content-Length': '0'} 2024-04-19T23:24:02.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1250dfc0ae0b153-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.081 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:02.081 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:02.081 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.081 DEBUG:boto:Path: /aktlji-11/%3A 2024-04-19T23:24:02.081 DEBUG:boto:Data: 2024-04-19T23:24:02.081 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.081 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.082 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.082 DEBUG:boto:Params: {} 2024-04-19T23:24:02.082 DEBUG:boto:Token: None 2024-04-19T23:24:02.082 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/%3A 2024-04-19T23:24:02.082 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OPC6u0qqNO2e7l2fePFxdA3TeGo= 2024-04-19T23:24:02.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OPC6u0qqNO2e7l2fePFxdA3TeGo=', 'Content-Length': '0'} 2024-04-19T23:24:02.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009009323ed761227f-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:02.084 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:02.084 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:02.084 DEBUG:boto:Method: GET 2024-04-19T23:24:02.084 DEBUG:boto:Path: /aktlji-11/%3A 2024-04-19T23:24:02.085 DEBUG:boto:Data: 2024-04-19T23:24:02.085 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.085 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.085 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.085 DEBUG:boto:Params: {} 2024-04-19T23:24:02.085 DEBUG:boto:Token: None 2024-04-19T23:24:02.085 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/%3A 2024-04-19T23:24:02.085 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XGbNfmUgumF5xpXfHetP9EENP+4= 2024-04-19T23:24:02.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XGbNfmUgumF5xpXfHetP9EENP+4=', 'Content-Length': '0'} 2024-04-19T23:24:02.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000248f9071886a7e1b-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.087 DEBUG:boto:path=/aktlji-11/%3A 2024-04-19T23:24:02.088 DEBUG:boto:auth_path=/aktlji-11/%3A 2024-04-19T23:24:02.088 DEBUG:boto:Method: GET 2024-04-19T23:24:02.088 DEBUG:boto:Path: /aktlji-11/%3A 2024-04-19T23:24:02.088 DEBUG:boto:Data: 2024-04-19T23:24:02.088 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.088 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.088 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.088 DEBUG:boto:Params: {} 2024-04-19T23:24:02.088 DEBUG:boto:Token: None 2024-04-19T23:24:02.088 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/%3A 2024-04-19T23:24:02.089 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XGbNfmUgumF5xpXfHetP9EENP+4= 2024-04-19T23:24:02.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XGbNfmUgumF5xpXfHetP9EENP+4=', 'Content-Length': '0'} 2024-04-19T23:24:02.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1c3ebc8a44d8b94-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.091 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:02.091 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:02.091 DEBUG:boto:path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:02.091 DEBUG:boto:auth_path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:02.091 DEBUG:boto:Method: GET 2024-04-19T23:24:02.091 DEBUG:boto:Path: /aktlji-11/?prefix=%3A 2024-04-19T23:24:02.091 DEBUG:boto:Data: 2024-04-19T23:24:02.091 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.091 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.091 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.091 DEBUG:boto:Params: {} 2024-04-19T23:24:02.092 DEBUG:boto:Token: None 2024-04-19T23:24:02.092 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.092 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3f2997c28ad979c-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.094 DEBUG:boto:b'aktlji-11:1000false:2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.095 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:02.095 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:02.095 DEBUG:boto:path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:02.095 DEBUG:boto:auth_path=/aktlji-11/?prefix=%3A 2024-04-19T23:24:02.095 DEBUG:boto:Method: GET 2024-04-19T23:24:02.095 DEBUG:boto:Path: /aktlji-11/?prefix=%3A 2024-04-19T23:24:02.095 DEBUG:boto:Data: 2024-04-19T23:24:02.095 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.095 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.096 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.096 DEBUG:boto:Params: {} 2024-04-19T23:24:02.096 DEBUG:boto:Token: None 2024-04-19T23:24:02.096 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.096 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e37296556213812-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.143 DEBUG:boto:b'aktlji-11:1000false:2024-04-19T23:23:14.195Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:02.143 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:02.143 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:02.143 DEBUG:boto:path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.144 DEBUG:boto:auth_path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.144 DEBUG:boto:Method: GET 2024-04-19T23:24:02.144 DEBUG:boto:Path: /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.144 DEBUG:boto:Data: 2024-04-19T23:24:02.144 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.144 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.144 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.144 DEBUG:boto:Params: {} 2024-04-19T23:24:02.145 DEBUG:boto:Token: None 2024-04-19T23:24:02.145 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.145 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PUVnZcwvPTj4rX5lu2RvdBQtJk8= 2024-04-19T23:24:02.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PUVnZcwvPTj4rX5lu2RvdBQtJk8=', 'Content-Length': '0'} 2024-04-19T23:24:02.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000646e97bf9fa08b1d-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.147 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:02.147 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:02.147 DEBUG:boto:path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.147 DEBUG:boto:auth_path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.148 DEBUG:boto:Method: GET 2024-04-19T23:24:02.148 DEBUG:boto:Path: /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.148 DEBUG:boto:Data: 2024-04-19T23:24:02.148 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.148 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.148 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.148 DEBUG:boto:Params: {} 2024-04-19T23:24:02.148 DEBUG:boto:Token: None 2024-04-19T23:24:02.148 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.148 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PUVnZcwvPTj4rX5lu2RvdBQtJk8= 2024-04-19T23:24:02.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PUVnZcwvPTj4rX5lu2RvdBQtJk8=', 'Content-Length': '0'} 2024-04-19T23:24:02.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e08f795fd498ac94-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.151 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:02.151 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:02.151 DEBUG:boto:path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.151 DEBUG:boto:auth_path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.151 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.151 DEBUG:boto:Path: /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.151 DEBUG:boto:Data: 2024-04-19T23:24:02.151 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.151 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.151 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.152 DEBUG:boto:Params: {} 2024-04-19T23:24:02.152 DEBUG:boto:Token: None 2024-04-19T23:24:02.152 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.152 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IK18JFKntBS2amBUBkpMiIBfpYg= 2024-04-19T23:24:02.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IK18JFKntBS2amBUBkpMiIBfpYg=', 'Content-Length': '0'} 2024-04-19T23:24:02.154 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf7af9e567701226-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.154 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:02.154 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:02.154 DEBUG:boto:path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.154 DEBUG:boto:auth_path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.154 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.154 DEBUG:boto:Path: /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.154 DEBUG:boto:Data: 2024-04-19T23:24:02.155 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.155 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.155 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.155 DEBUG:boto:Params: {} 2024-04-19T23:24:02.155 DEBUG:boto:Token: None 2024-04-19T23:24:02.155 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.155 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IK18JFKntBS2amBUBkpMiIBfpYg= 2024-04-19T23:24:02.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IK18JFKntBS2amBUBkpMiIBfpYg=', 'Content-Length': '0'} 2024-04-19T23:24:02.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002036f9a671a4c052-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:02.157 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:02.157 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:02.157 DEBUG:boto:path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.158 DEBUG:boto:auth_path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.158 DEBUG:boto:Method: GET 2024-04-19T23:24:02.158 DEBUG:boto:Path: /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.158 DEBUG:boto:Data: 2024-04-19T23:24:02.158 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.158 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.158 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.158 DEBUG:boto:Params: {} 2024-04-19T23:24:02.158 DEBUG:boto:Token: None 2024-04-19T23:24:02.158 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.159 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PUVnZcwvPTj4rX5lu2RvdBQtJk8= 2024-04-19T23:24:02.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PUVnZcwvPTj4rX5lu2RvdBQtJk8=', 'Content-Length': '0'} 2024-04-19T23:24:02.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000152242b8240bc883-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.202 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:02.203 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:02.203 DEBUG:boto:path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.203 DEBUG:boto:auth_path=/aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.203 DEBUG:boto:Method: GET 2024-04-19T23:24:02.203 DEBUG:boto:Path: /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.203 DEBUG:boto:Data: 2024-04-19T23:24:02.203 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.203 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.204 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.204 DEBUG:boto:Params: {} 2024-04-19T23:24:02.204 DEBUG:boto:Token: None 2024-04-19T23:24:02.204 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/_myobj?versionId=null 2024-04-19T23:24:02.204 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PUVnZcwvPTj4rX5lu2RvdBQtJk8= 2024-04-19T23:24:02.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PUVnZcwvPTj4rX5lu2RvdBQtJk8=', 'Content-Length': '0'} 2024-04-19T23:24:02.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a96206dd1fdbcdd0-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.206 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:02.206 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:02.207 DEBUG:boto:path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:02.207 DEBUG:boto:auth_path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:02.207 DEBUG:boto:Method: GET 2024-04-19T23:24:02.207 DEBUG:boto:Path: /aktlji-11/?prefix=_myobj 2024-04-19T23:24:02.207 DEBUG:boto:Data: 2024-04-19T23:24:02.207 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.207 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.207 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.207 DEBUG:boto:Params: {} 2024-04-19T23:24:02.207 DEBUG:boto:Token: None 2024-04-19T23:24:02.208 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.208 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3036545dbd19194-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.210 DEBUG:boto:b'aktlji-11_myobj1000false_myobj2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.211 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:02.211 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:02.211 DEBUG:boto:path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:02.211 DEBUG:boto:auth_path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:02.211 DEBUG:boto:Method: GET 2024-04-19T23:24:02.211 DEBUG:boto:Path: /aktlji-11/?prefix=_myobj 2024-04-19T23:24:02.211 DEBUG:boto:Data: 2024-04-19T23:24:02.211 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.212 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.212 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.212 DEBUG:boto:Params: {} 2024-04-19T23:24:02.212 DEBUG:boto:Token: None 2024-04-19T23:24:02.212 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.212 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc97406469aed92f-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.215 DEBUG:boto:b'aktlji-11_myobj1000false_myobj2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.215 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:02.215 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:02.215 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.215 DEBUG:boto:Path: /aktlji-11/_myobj 2024-04-19T23:24:02.215 DEBUG:boto:Data: 2024-04-19T23:24:02.216 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.216 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.216 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.216 DEBUG:boto:Params: {} 2024-04-19T23:24:02.216 DEBUG:boto:Token: None 2024-04-19T23:24:02.216 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/_myobj 2024-04-19T23:24:02.216 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R8nLhvb9eZjf1P2m19bRTH/wizw= 2024-04-19T23:24:02.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R8nLhvb9eZjf1P2m19bRTH/wizw=', 'Content-Length': '0'} 2024-04-19T23:24:02.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f03328bf1ca74cba-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.218 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:02.219 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:02.219 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.219 DEBUG:boto:Path: /aktlji-11/_myobj 2024-04-19T23:24:02.219 DEBUG:boto:Data: 2024-04-19T23:24:02.219 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.219 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.219 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.219 DEBUG:boto:Params: {} 2024-04-19T23:24:02.220 DEBUG:boto:Token: None 2024-04-19T23:24:02.220 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/_myobj 2024-04-19T23:24:02.220 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R8nLhvb9eZjf1P2m19bRTH/wizw= 2024-04-19T23:24:02.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R8nLhvb9eZjf1P2m19bRTH/wizw=', 'Content-Length': '0'} 2024-04-19T23:24:02.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a993ddb1af4977a-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:02.222 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:02.222 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:02.222 DEBUG:boto:Method: GET 2024-04-19T23:24:02.222 DEBUG:boto:Path: /aktlji-11/_myobj 2024-04-19T23:24:02.222 DEBUG:boto:Data: 2024-04-19T23:24:02.222 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.222 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.223 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.223 DEBUG:boto:Params: {} 2024-04-19T23:24:02.223 DEBUG:boto:Token: None 2024-04-19T23:24:02.223 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/_myobj 2024-04-19T23:24:02.223 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mGy1OQLCdOHxkhNYPebPZqSVsa4= 2024-04-19T23:24:02.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mGy1OQLCdOHxkhNYPebPZqSVsa4=', 'Content-Length': '0'} 2024-04-19T23:24:02.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bea23c2f1fb484f6-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.225 DEBUG:boto:path=/aktlji-11/_myobj 2024-04-19T23:24:02.225 DEBUG:boto:auth_path=/aktlji-11/_myobj 2024-04-19T23:24:02.225 DEBUG:boto:Method: GET 2024-04-19T23:24:02.225 DEBUG:boto:Path: /aktlji-11/_myobj 2024-04-19T23:24:02.226 DEBUG:boto:Data: 2024-04-19T23:24:02.226 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.226 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.226 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.226 DEBUG:boto:Params: {} 2024-04-19T23:24:02.226 DEBUG:boto:Token: None 2024-04-19T23:24:02.226 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/_myobj 2024-04-19T23:24:02.226 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mGy1OQLCdOHxkhNYPebPZqSVsa4= 2024-04-19T23:24:02.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mGy1OQLCdOHxkhNYPebPZqSVsa4=', 'Content-Length': '0'} 2024-04-19T23:24:02.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004146866cf52daa69-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.274 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:02.275 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:02.275 DEBUG:boto:path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:02.275 DEBUG:boto:auth_path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:02.275 DEBUG:boto:Method: GET 2024-04-19T23:24:02.275 DEBUG:boto:Path: /aktlji-11/?prefix=_myobj 2024-04-19T23:24:02.275 DEBUG:boto:Data: 2024-04-19T23:24:02.275 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.275 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.276 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.276 DEBUG:boto:Params: {} 2024-04-19T23:24:02.276 DEBUG:boto:Token: None 2024-04-19T23:24:02.276 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.276 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000578b8358328ba17a-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.279 DEBUG:boto:b'aktlji-11_myobj1000false_myobj2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.279 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:02.279 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:02.280 DEBUG:boto:path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:02.280 DEBUG:boto:auth_path=/aktlji-11/?prefix=_myobj 2024-04-19T23:24:02.280 DEBUG:boto:Method: GET 2024-04-19T23:24:02.280 DEBUG:boto:Path: /aktlji-11/?prefix=_myobj 2024-04-19T23:24:02.280 DEBUG:boto:Data: 2024-04-19T23:24:02.280 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.280 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.280 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.280 DEBUG:boto:Params: {} 2024-04-19T23:24:02.280 DEBUG:boto:Token: None 2024-04-19T23:24:02.281 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.281 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000944df726c404c9f0-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.284 DEBUG:boto:b'aktlji-11_myobj1000false_myobj2024-04-19T23:23:14.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:02.284 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:02.285 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:02.285 DEBUG:boto:path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.285 DEBUG:boto:auth_path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.285 DEBUG:boto:Method: GET 2024-04-19T23:24:02.285 DEBUG:boto:Path: /aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.285 DEBUG:boto:Data: 2024-04-19T23:24:02.285 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.285 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.285 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.285 DEBUG:boto:Params: {} 2024-04-19T23:24:02.285 DEBUG:boto:Token: None 2024-04-19T23:24:02.286 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.286 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YnweQNJUaZCYtcINuLMEtqLz9WU= 2024-04-19T23:24:02.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YnweQNJUaZCYtcINuLMEtqLz9WU=', 'Content-Length': '0'} 2024-04-19T23:24:02.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002141d2f64bfc05d-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.330 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:02.331 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:02.331 DEBUG:boto:path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.331 DEBUG:boto:auth_path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.331 DEBUG:boto:Method: GET 2024-04-19T23:24:02.331 DEBUG:boto:Path: /aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.331 DEBUG:boto:Data: 2024-04-19T23:24:02.331 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.331 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.331 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.331 DEBUG:boto:Params: {} 2024-04-19T23:24:02.332 DEBUG:boto:Token: None 2024-04-19T23:24:02.332 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YnweQNJUaZCYtcINuLMEtqLz9WU= 2024-04-19T23:24:02.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YnweQNJUaZCYtcINuLMEtqLz9WU=', 'Content-Length': '0'} 2024-04-19T23:24:02.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e31c32df9432ce32-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.334 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:02.334 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:02.334 DEBUG:boto:path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.335 DEBUG:boto:auth_path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.335 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.335 DEBUG:boto:Path: /aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.335 DEBUG:boto:Data: 2024-04-19T23:24:02.335 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.335 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.335 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.335 DEBUG:boto:Params: {} 2024-04-19T23:24:02.336 DEBUG:boto:Token: None 2024-04-19T23:24:02.336 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.336 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+npQiENmNmuYMd1FKgoPzTJF5Pc= 2024-04-19T23:24:02.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+npQiENmNmuYMd1FKgoPzTJF5Pc=', 'Content-Length': '0'} 2024-04-19T23:24:02.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7db8b72fdd4f205-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.338 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:02.338 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:02.338 DEBUG:boto:path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.338 DEBUG:boto:auth_path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.338 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.339 DEBUG:boto:Path: /aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.339 DEBUG:boto:Data: 2024-04-19T23:24:02.339 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.339 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.339 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.339 DEBUG:boto:Params: {} 2024-04-19T23:24:02.339 DEBUG:boto:Token: None 2024-04-19T23:24:02.339 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.339 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+npQiENmNmuYMd1FKgoPzTJF5Pc= 2024-04-19T23:24:02.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+npQiENmNmuYMd1FKgoPzTJF5Pc=', 'Content-Length': '0'} 2024-04-19T23:24:02.341 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008921930351f16011-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:02.341 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:02.341 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:02.342 DEBUG:boto:path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.342 DEBUG:boto:auth_path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.342 DEBUG:boto:Method: GET 2024-04-19T23:24:02.342 DEBUG:boto:Path: /aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.342 DEBUG:boto:Data: 2024-04-19T23:24:02.342 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.342 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.342 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.342 DEBUG:boto:Params: {} 2024-04-19T23:24:02.342 DEBUG:boto:Token: None 2024-04-19T23:24:02.343 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.343 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YnweQNJUaZCYtcINuLMEtqLz9WU= 2024-04-19T23:24:02.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YnweQNJUaZCYtcINuLMEtqLz9WU=', 'Content-Length': '0'} 2024-04-19T23:24:02.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079e97c5f38249766-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.345 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:02.345 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:02.345 DEBUG:boto:path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.345 DEBUG:boto:auth_path=/aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.345 DEBUG:boto:Method: GET 2024-04-19T23:24:02.345 DEBUG:boto:Path: /aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.345 DEBUG:boto:Data: 2024-04-19T23:24:02.345 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.346 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.346 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.346 DEBUG:boto:Params: {} 2024-04-19T23:24:02.346 DEBUG:boto:Token: None 2024-04-19T23:24:02.346 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/myobj?versionId=null 2024-04-19T23:24:02.346 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YnweQNJUaZCYtcINuLMEtqLz9WU= 2024-04-19T23:24:02.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YnweQNJUaZCYtcINuLMEtqLz9WU=', 'Content-Length': '0'} 2024-04-19T23:24:02.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b43797ce8ac3533c-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.348 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:02.348 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:02.348 DEBUG:boto:path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:02.349 DEBUG:boto:auth_path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:02.349 DEBUG:boto:Method: GET 2024-04-19T23:24:02.349 DEBUG:boto:Path: /aktlji-11/?prefix=myobj 2024-04-19T23:24:02.349 DEBUG:boto:Data: 2024-04-19T23:24:02.349 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.349 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.349 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.349 DEBUG:boto:Params: {} 2024-04-19T23:24:02.349 DEBUG:boto:Token: None 2024-04-19T23:24:02.349 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.349 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b87c0f4ccfb31a6-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.352 DEBUG:boto:b'aktlji-11myobj1000falsemyobj2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.352 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:02.353 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:02.353 DEBUG:boto:path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:02.353 DEBUG:boto:auth_path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:02.353 DEBUG:boto:Method: GET 2024-04-19T23:24:02.353 DEBUG:boto:Path: /aktlji-11/?prefix=myobj 2024-04-19T23:24:02.353 DEBUG:boto:Data: 2024-04-19T23:24:02.353 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.353 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.353 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.353 DEBUG:boto:Params: {} 2024-04-19T23:24:02.353 DEBUG:boto:Token: None 2024-04-19T23:24:02.354 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.354 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb9bb448884697cb-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.356 DEBUG:boto:b'aktlji-11myobj1000falsemyobj2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.357 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:02.357 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:02.357 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.357 DEBUG:boto:Path: /aktlji-11/myobj 2024-04-19T23:24:02.357 DEBUG:boto:Data: 2024-04-19T23:24:02.357 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.357 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.357 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.357 DEBUG:boto:Params: {} 2024-04-19T23:24:02.358 DEBUG:boto:Token: None 2024-04-19T23:24:02.358 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/myobj 2024-04-19T23:24:02.358 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VBVmac7NxnjXkoIq2EHyOOJFIUg= 2024-04-19T23:24:02.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VBVmac7NxnjXkoIq2EHyOOJFIUg=', 'Content-Length': '0'} 2024-04-19T23:24:02.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000435b233f1b62b8e5-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.360 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:02.360 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:02.360 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.360 DEBUG:boto:Path: /aktlji-11/myobj 2024-04-19T23:24:02.360 DEBUG:boto:Data: 2024-04-19T23:24:02.360 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.360 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.360 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.361 DEBUG:boto:Params: {} 2024-04-19T23:24:02.361 DEBUG:boto:Token: None 2024-04-19T23:24:02.361 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/myobj 2024-04-19T23:24:02.361 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VBVmac7NxnjXkoIq2EHyOOJFIUg= 2024-04-19T23:24:02.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VBVmac7NxnjXkoIq2EHyOOJFIUg=', 'Content-Length': '0'} 2024-04-19T23:24:02.363 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097939cf559ff667c-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:02.363 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:02.363 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:02.363 DEBUG:boto:Method: GET 2024-04-19T23:24:02.363 DEBUG:boto:Path: /aktlji-11/myobj 2024-04-19T23:24:02.363 DEBUG:boto:Data: 2024-04-19T23:24:02.363 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.363 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.364 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.364 DEBUG:boto:Params: {} 2024-04-19T23:24:02.364 DEBUG:boto:Token: None 2024-04-19T23:24:02.364 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/myobj 2024-04-19T23:24:02.364 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fBnAj2Q1mB3QaGo3DbV0hhZA37A= 2024-04-19T23:24:02.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fBnAj2Q1mB3QaGo3DbV0hhZA37A=', 'Content-Length': '0'} 2024-04-19T23:24:02.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c0c49750524cb4c-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.366 DEBUG:boto:path=/aktlji-11/myobj 2024-04-19T23:24:02.366 DEBUG:boto:auth_path=/aktlji-11/myobj 2024-04-19T23:24:02.366 DEBUG:boto:Method: GET 2024-04-19T23:24:02.366 DEBUG:boto:Path: /aktlji-11/myobj 2024-04-19T23:24:02.367 DEBUG:boto:Data: 2024-04-19T23:24:02.367 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.367 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.367 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.367 DEBUG:boto:Params: {} 2024-04-19T23:24:02.367 DEBUG:boto:Token: None 2024-04-19T23:24:02.367 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/myobj 2024-04-19T23:24:02.367 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fBnAj2Q1mB3QaGo3DbV0hhZA37A= 2024-04-19T23:24:02.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fBnAj2Q1mB3QaGo3DbV0hhZA37A=', 'Content-Length': '0'} 2024-04-19T23:24:02.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b673eae80eb96766-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.369 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:02.369 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:02.369 DEBUG:boto:path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:02.370 DEBUG:boto:auth_path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:02.370 DEBUG:boto:Method: GET 2024-04-19T23:24:02.370 DEBUG:boto:Path: /aktlji-11/?prefix=myobj 2024-04-19T23:24:02.370 DEBUG:boto:Data: 2024-04-19T23:24:02.370 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.370 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.370 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.370 DEBUG:boto:Params: {} 2024-04-19T23:24:02.370 DEBUG:boto:Token: None 2024-04-19T23:24:02.370 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.371 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab12d35d159c6c8c-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.373 DEBUG:boto:b'aktlji-11myobj1000falsemyobj2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.374 DEBUG:boto:path=/aktlji-11/ 2024-04-19T23:24:02.374 DEBUG:boto:auth_path=/aktlji-11/ 2024-04-19T23:24:02.374 DEBUG:boto:path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:02.374 DEBUG:boto:auth_path=/aktlji-11/?prefix=myobj 2024-04-19T23:24:02.374 DEBUG:boto:Method: GET 2024-04-19T23:24:02.374 DEBUG:boto:Path: /aktlji-11/?prefix=myobj 2024-04-19T23:24:02.374 DEBUG:boto:Data: 2024-04-19T23:24:02.374 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.374 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:02.374 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.375 DEBUG:boto:Params: {} 2024-04-19T23:24:02.375 DEBUG:boto:Token: None 2024-04-19T23:24:02.375 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-11/ 2024-04-19T23:24:02.375 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU= 2024-04-19T23:24:02.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0vG2QqCUAxyOS2W4yC1saWCiDvU=', 'Content-Length': '0'} 2024-04-19T23:24:02.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fb241c0212d47c7-006622fd12-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.378 DEBUG:boto:b'aktlji-11myobj1000falsemyobj2024-04-19T23:23:14.160Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.378 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-11 zones={test-zone3, test-zone2} 2024-04-19T23:24:02.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:02.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:02.378 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:24:02.444 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:02.445 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:02.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:02.492+0000 7f160ec76780 1 bucket sync caught up with source: 2024-04-19T23:24:02.500 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.6.6, , 00000000001.15.6, , , , 00000000001.10.6, 00000000001.6.6, , ] 2024-04-19T23:24:02.500 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000001.6.6, , 00000000001.15.6, , , , 00000000001.10.6, 00000000001.6.6, , ] 2024-04-19T23:24:02.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:02.492+0000 7f160ec76780 0 bucket checkpoint complete 2024-04-19T23:24:02.502 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-12 zones={test-zone1, test-zone3} 2024-04-19T23:24:02.502 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.502 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.502 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.502 DEBUG:boto:Path: /aktlji-12/ 2024-04-19T23:24:02.502 DEBUG:boto:Data: 2024-04-19T23:24:02.502 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.503 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.503 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.503 DEBUG:boto:Params: {} 2024-04-19T23:24:02.503 DEBUG:boto:Token: None 2024-04-19T23:24:02.503 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.503 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dsDKwBJu7l3ukO+J9iwjpITkt1c= 2024-04-19T23:24:02.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dsDKwBJu7l3ukO+J9iwjpITkt1c=', 'Content-Length': '0'} 2024-04-19T23:24:02.505 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011ed6c0e2cecba15-006622fd12-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.506 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.506 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.506 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.506 DEBUG:boto:Path: /aktlji-12/ 2024-04-19T23:24:02.506 DEBUG:boto:Data: 2024-04-19T23:24:02.506 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.506 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.506 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.506 DEBUG:boto:Params: {} 2024-04-19T23:24:02.507 DEBUG:boto:Token: None 2024-04-19T23:24:02.507 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.507 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dsDKwBJu7l3ukO+J9iwjpITkt1c= 2024-04-19T23:24:02.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dsDKwBJu7l3ukO+J9iwjpITkt1c=', 'Content-Length': '0'} 2024-04-19T23:24:02.509 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080f4abbdfa9ded17-006622fd12-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.509 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:24:02.510 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.510 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.510 DEBUG:boto:path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:02.510 DEBUG:boto:auth_path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:02.510 DEBUG:boto:Method: GET 2024-04-19T23:24:02.510 DEBUG:boto:Path: /aktlji-12/?versions&max-keys=999 2024-04-19T23:24:02.510 DEBUG:boto:Data: 2024-04-19T23:24:02.510 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.510 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.510 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.510 DEBUG:boto:Params: {} 2024-04-19T23:24:02.511 DEBUG:boto:Token: None 2024-04-19T23:24:02.511 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/?versions 2024-04-19T23:24:02.511 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qkRdTKK24jOGilE3O4JyDPOe9a4= 2024-04-19T23:24:02.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qkRdTKK24jOGilE3O4JyDPOe9a4=', 'Content-Length': '0'} 2024-04-19T23:24:02.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b261f44db0570b10-006622fd12-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.513 DEBUG:boto:b'aktlji-12999false&nulltrue2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.514 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-19T23:24:02.514 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-19T23:24:02.514 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-19T23:24:02.514 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:02.514 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:24:02.514 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.514 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.514 DEBUG:boto:path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:02.514 DEBUG:boto:auth_path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:02.514 DEBUG:boto:Method: GET 2024-04-19T23:24:02.515 DEBUG:boto:Path: /aktlji-12/?versions&max-keys=999 2024-04-19T23:24:02.515 DEBUG:boto:Data: 2024-04-19T23:24:02.515 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.515 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.515 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.515 DEBUG:boto:Params: {} 2024-04-19T23:24:02.515 DEBUG:boto:Token: None 2024-04-19T23:24:02.515 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/?versions 2024-04-19T23:24:02.515 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qkRdTKK24jOGilE3O4JyDPOe9a4= 2024-04-19T23:24:02.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qkRdTKK24jOGilE3O4JyDPOe9a4=', 'Content-Length': '0'} 2024-04-19T23:24:02.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4ae1d34534a1bca-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.518 DEBUG:boto:b'aktlji-12999false&nulltrue2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.518 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-19T23:24:02.519 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-19T23:24:02.519 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-19T23:24:02.519 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:02.519 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.519 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.519 DEBUG:boto:path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:02.519 DEBUG:boto:auth_path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:02.519 DEBUG:boto:Method: GET 2024-04-19T23:24:02.519 DEBUG:boto:Path: /aktlji-12/?versions&max-keys=999 2024-04-19T23:24:02.519 DEBUG:boto:Data: 2024-04-19T23:24:02.519 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.520 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.520 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.520 DEBUG:boto:Params: {} 2024-04-19T23:24:02.520 DEBUG:boto:Token: None 2024-04-19T23:24:02.520 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/?versions 2024-04-19T23:24:02.520 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qkRdTKK24jOGilE3O4JyDPOe9a4= 2024-04-19T23:24:02.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qkRdTKK24jOGilE3O4JyDPOe9a4=', 'Content-Length': '0'} 2024-04-19T23:24:02.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a236cab6ed9eb01-006622fd12-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.522 DEBUG:boto:b'aktlji-12999false&nulltrue2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.523 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.523 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.523 DEBUG:boto:path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:02.523 DEBUG:boto:auth_path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:02.523 DEBUG:boto:Method: GET 2024-04-19T23:24:02.523 DEBUG:boto:Path: /aktlji-12/?versions&max-keys=999 2024-04-19T23:24:02.523 DEBUG:boto:Data: 2024-04-19T23:24:02.523 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.523 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.524 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.524 DEBUG:boto:Params: {} 2024-04-19T23:24:02.524 DEBUG:boto:Token: None 2024-04-19T23:24:02.524 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/?versions 2024-04-19T23:24:02.524 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qkRdTKK24jOGilE3O4JyDPOe9a4= 2024-04-19T23:24:02.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qkRdTKK24jOGilE3O4JyDPOe9a4=', 'Content-Length': '0'} 2024-04-19T23:24:02.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bcb3cdb28e55f1d-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.526 DEBUG:boto:b'aktlji-12999false&nulltrue2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:02.527 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:02.527 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:02.527 DEBUG:boto:path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:02.527 DEBUG:boto:auth_path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:02.527 DEBUG:boto:Method: GET 2024-04-19T23:24:02.527 DEBUG:boto:Path: /aktlji-12/%26?versionId=null 2024-04-19T23:24:02.527 DEBUG:boto:Data: 2024-04-19T23:24:02.527 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.528 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.528 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.528 DEBUG:boto:Params: {} 2024-04-19T23:24:02.528 DEBUG:boto:Token: None 2024-04-19T23:24:02.528 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%26?versionId=null 2024-04-19T23:24:02.528 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PNm4CAIZmqaKf9ltd8IIlfiUSKo= 2024-04-19T23:24:02.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PNm4CAIZmqaKf9ltd8IIlfiUSKo=', 'Content-Length': '0'} 2024-04-19T23:24:02.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000559c6655c794f545-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.530 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:02.530 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:02.531 DEBUG:boto:path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:02.531 DEBUG:boto:auth_path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:02.531 DEBUG:boto:Method: GET 2024-04-19T23:24:02.531 DEBUG:boto:Path: /aktlji-12/%26?versionId=null 2024-04-19T23:24:02.531 DEBUG:boto:Data: 2024-04-19T23:24:02.531 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.531 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.531 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.531 DEBUG:boto:Params: {} 2024-04-19T23:24:02.531 DEBUG:boto:Token: None 2024-04-19T23:24:02.531 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%26?versionId=null 2024-04-19T23:24:02.532 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PNm4CAIZmqaKf9ltd8IIlfiUSKo= 2024-04-19T23:24:02.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PNm4CAIZmqaKf9ltd8IIlfiUSKo=', 'Content-Length': '0'} 2024-04-19T23:24:02.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007abadec41a04b86f-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.574 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:02.574 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:02.575 DEBUG:boto:path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:02.575 DEBUG:boto:auth_path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:02.575 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.575 DEBUG:boto:Path: /aktlji-12/%26?versionId=null 2024-04-19T23:24:02.575 DEBUG:boto:Data: 2024-04-19T23:24:02.575 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.575 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.575 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.575 DEBUG:boto:Params: {} 2024-04-19T23:24:02.576 DEBUG:boto:Token: None 2024-04-19T23:24:02.576 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%26?versionId=null 2024-04-19T23:24:02.576 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pqnXakNI/ez4Mtv+FaMDSbUx+tw= 2024-04-19T23:24:02.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pqnXakNI/ez4Mtv+FaMDSbUx+tw=', 'Content-Length': '0'} 2024-04-19T23:24:02.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a462a692ffc4154c-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.578 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:02.579 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:02.579 DEBUG:boto:path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:02.579 DEBUG:boto:auth_path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:02.579 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.579 DEBUG:boto:Path: /aktlji-12/%26?versionId=null 2024-04-19T23:24:02.579 DEBUG:boto:Data: 2024-04-19T23:24:02.579 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.579 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.579 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.579 DEBUG:boto:Params: {} 2024-04-19T23:24:02.579 DEBUG:boto:Token: None 2024-04-19T23:24:02.580 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%26?versionId=null 2024-04-19T23:24:02.580 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pqnXakNI/ez4Mtv+FaMDSbUx+tw= 2024-04-19T23:24:02.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pqnXakNI/ez4Mtv+FaMDSbUx+tw=', 'Content-Length': '0'} 2024-04-19T23:24:02.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050cbf6f8ba57e6cd-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:02.582 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:02.582 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:02.582 DEBUG:boto:path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:02.582 DEBUG:boto:auth_path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:02.582 DEBUG:boto:Method: GET 2024-04-19T23:24:02.582 DEBUG:boto:Path: /aktlji-12/%26?versionId=null 2024-04-19T23:24:02.582 DEBUG:boto:Data: 2024-04-19T23:24:02.582 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.582 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.582 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.583 DEBUG:boto:Params: {} 2024-04-19T23:24:02.583 DEBUG:boto:Token: None 2024-04-19T23:24:02.583 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%26?versionId=null 2024-04-19T23:24:02.583 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PNm4CAIZmqaKf9ltd8IIlfiUSKo= 2024-04-19T23:24:02.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PNm4CAIZmqaKf9ltd8IIlfiUSKo=', 'Content-Length': '0'} 2024-04-19T23:24:02.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ec33277c7405fe4-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.585 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:02.585 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:02.585 DEBUG:boto:path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:02.585 DEBUG:boto:auth_path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:02.586 DEBUG:boto:Method: GET 2024-04-19T23:24:02.586 DEBUG:boto:Path: /aktlji-12/%26?versionId=null 2024-04-19T23:24:02.586 DEBUG:boto:Data: 2024-04-19T23:24:02.586 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.586 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.586 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.586 DEBUG:boto:Params: {} 2024-04-19T23:24:02.586 DEBUG:boto:Token: None 2024-04-19T23:24:02.586 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%26?versionId=null 2024-04-19T23:24:02.586 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PNm4CAIZmqaKf9ltd8IIlfiUSKo= 2024-04-19T23:24:02.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PNm4CAIZmqaKf9ltd8IIlfiUSKo=', 'Content-Length': '0'} 2024-04-19T23:24:02.588 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b8985eef54ab40a-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.589 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.589 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.589 DEBUG:boto:path=/aktlji-12/?prefix=%26 2024-04-19T23:24:02.589 DEBUG:boto:auth_path=/aktlji-12/?prefix=%26 2024-04-19T23:24:02.589 DEBUG:boto:Method: GET 2024-04-19T23:24:02.590 DEBUG:boto:Path: /aktlji-12/?prefix=%26 2024-04-19T23:24:02.590 DEBUG:boto:Data: 2024-04-19T23:24:02.590 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.590 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.590 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.590 DEBUG:boto:Params: {} 2024-04-19T23:24:02.590 DEBUG:boto:Token: None 2024-04-19T23:24:02.593 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.594 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA= 2024-04-19T23:24:02.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA=', 'Content-Length': '0'} 2024-04-19T23:24:02.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095fac80faee06a02-006622fd12-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.597 DEBUG:boto:b'aktlji-12&1000false&2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.597 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.597 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.597 DEBUG:boto:path=/aktlji-12/?prefix=%26 2024-04-19T23:24:02.597 DEBUG:boto:auth_path=/aktlji-12/?prefix=%26 2024-04-19T23:24:02.598 DEBUG:boto:Method: GET 2024-04-19T23:24:02.598 DEBUG:boto:Path: /aktlji-12/?prefix=%26 2024-04-19T23:24:02.598 DEBUG:boto:Data: 2024-04-19T23:24:02.598 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.598 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.598 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.598 DEBUG:boto:Params: {} 2024-04-19T23:24:02.598 DEBUG:boto:Token: None 2024-04-19T23:24:02.598 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.599 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA= 2024-04-19T23:24:02.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA=', 'Content-Length': '0'} 2024-04-19T23:24:02.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a54f8778fb52cabb-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.601 DEBUG:boto:b'aktlji-12&1000false&2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.601 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:02.602 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:02.602 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.602 DEBUG:boto:Path: /aktlji-12/%26 2024-04-19T23:24:02.602 DEBUG:boto:Data: 2024-04-19T23:24:02.602 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.602 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.602 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.602 DEBUG:boto:Params: {} 2024-04-19T23:24:02.602 DEBUG:boto:Token: None 2024-04-19T23:24:02.603 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%26 2024-04-19T23:24:02.603 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KcFa3A2PLp66uzT1TJ/wH8rcJi4= 2024-04-19T23:24:02.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KcFa3A2PLp66uzT1TJ/wH8rcJi4=', 'Content-Length': '0'} 2024-04-19T23:24:02.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ca0e8d0dda700e6-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.605 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:02.605 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:02.605 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.605 DEBUG:boto:Path: /aktlji-12/%26 2024-04-19T23:24:02.605 DEBUG:boto:Data: 2024-04-19T23:24:02.606 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.606 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.606 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.606 DEBUG:boto:Params: {} 2024-04-19T23:24:02.606 DEBUG:boto:Token: None 2024-04-19T23:24:02.606 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%26 2024-04-19T23:24:02.606 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KcFa3A2PLp66uzT1TJ/wH8rcJi4= 2024-04-19T23:24:02.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KcFa3A2PLp66uzT1TJ/wH8rcJi4=', 'Content-Length': '0'} 2024-04-19T23:24:02.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1f64cbcfa7aaff5-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:02.609 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:02.609 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:02.609 DEBUG:boto:Method: GET 2024-04-19T23:24:02.609 DEBUG:boto:Path: /aktlji-12/%26 2024-04-19T23:24:02.609 DEBUG:boto:Data: 2024-04-19T23:24:02.609 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.609 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.609 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.609 DEBUG:boto:Params: {} 2024-04-19T23:24:02.609 DEBUG:boto:Token: None 2024-04-19T23:24:02.610 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%26 2024-04-19T23:24:02.610 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Sxs8qfskK15Ghx+ZEmqzkEG/WZE= 2024-04-19T23:24:02.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Sxs8qfskK15Ghx+ZEmqzkEG/WZE=', 'Content-Length': '0'} 2024-04-19T23:24:02.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017c0123926d90b22-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.612 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:02.612 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:02.612 DEBUG:boto:Method: GET 2024-04-19T23:24:02.612 DEBUG:boto:Path: /aktlji-12/%26 2024-04-19T23:24:02.612 DEBUG:boto:Data: 2024-04-19T23:24:02.612 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.612 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.612 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.612 DEBUG:boto:Params: {} 2024-04-19T23:24:02.613 DEBUG:boto:Token: None 2024-04-19T23:24:02.613 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%26 2024-04-19T23:24:02.613 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Sxs8qfskK15Ghx+ZEmqzkEG/WZE= 2024-04-19T23:24:02.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Sxs8qfskK15Ghx+ZEmqzkEG/WZE=', 'Content-Length': '0'} 2024-04-19T23:24:02.614 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d28c5edd3121eb56-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.615 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.615 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.615 DEBUG:boto:path=/aktlji-12/?prefix=%26 2024-04-19T23:24:02.615 DEBUG:boto:auth_path=/aktlji-12/?prefix=%26 2024-04-19T23:24:02.615 DEBUG:boto:Method: GET 2024-04-19T23:24:02.615 DEBUG:boto:Path: /aktlji-12/?prefix=%26 2024-04-19T23:24:02.615 DEBUG:boto:Data: 2024-04-19T23:24:02.615 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.615 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.616 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.616 DEBUG:boto:Params: {} 2024-04-19T23:24:02.616 DEBUG:boto:Token: None 2024-04-19T23:24:02.616 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.616 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA= 2024-04-19T23:24:02.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA=', 'Content-Length': '0'} 2024-04-19T23:24:02.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e197bbf6db0f3249-006622fd12-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.618 DEBUG:boto:b'aktlji-12&1000false&2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.619 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.619 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.619 DEBUG:boto:path=/aktlji-12/?prefix=%26 2024-04-19T23:24:02.619 DEBUG:boto:auth_path=/aktlji-12/?prefix=%26 2024-04-19T23:24:02.619 DEBUG:boto:Method: GET 2024-04-19T23:24:02.619 DEBUG:boto:Path: /aktlji-12/?prefix=%26 2024-04-19T23:24:02.619 DEBUG:boto:Data: 2024-04-19T23:24:02.619 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.620 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.620 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.620 DEBUG:boto:Params: {} 2024-04-19T23:24:02.620 DEBUG:boto:Token: None 2024-04-19T23:24:02.620 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.620 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA= 2024-04-19T23:24:02.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA=', 'Content-Length': '0'} 2024-04-19T23:24:02.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033ef234ee2325a67-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.623 DEBUG:boto:b'aktlji-12&1000false&2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:02.623 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:02.623 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:02.623 DEBUG:boto:path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.624 DEBUG:boto:auth_path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.624 DEBUG:boto:Method: GET 2024-04-19T23:24:02.624 DEBUG:boto:Path: /aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.624 DEBUG:boto:Data: 2024-04-19T23:24:02.624 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.624 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.624 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.624 DEBUG:boto:Params: {} 2024-04-19T23:24:02.624 DEBUG:boto:Token: None 2024-04-19T23:24:02.624 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.625 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9xl1HSREotXTA0EnlHl2AaOW6Z8= 2024-04-19T23:24:02.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9xl1HSREotXTA0EnlHl2AaOW6Z8=', 'Content-Length': '0'} 2024-04-19T23:24:02.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab711e817151d2fa-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.674 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:02.675 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:02.675 DEBUG:boto:path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.675 DEBUG:boto:auth_path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.675 DEBUG:boto:Method: GET 2024-04-19T23:24:02.675 DEBUG:boto:Path: /aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.675 DEBUG:boto:Data: 2024-04-19T23:24:02.675 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.675 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.675 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.675 DEBUG:boto:Params: {} 2024-04-19T23:24:02.676 DEBUG:boto:Token: None 2024-04-19T23:24:02.676 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9xl1HSREotXTA0EnlHl2AaOW6Z8= 2024-04-19T23:24:02.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9xl1HSREotXTA0EnlHl2AaOW6Z8=', 'Content-Length': '0'} 2024-04-19T23:24:02.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e83f42ed81a0a21-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.678 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:02.679 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:02.679 DEBUG:boto:path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.679 DEBUG:boto:auth_path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.679 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.679 DEBUG:boto:Path: /aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.679 DEBUG:boto:Data: 2024-04-19T23:24:02.679 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.679 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.679 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.679 DEBUG:boto:Params: {} 2024-04-19T23:24:02.680 DEBUG:boto:Token: None 2024-04-19T23:24:02.680 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.680 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bx6NZS7WzlRw16GhZTqgzmu+dmQ= 2024-04-19T23:24:02.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bx6NZS7WzlRw16GhZTqgzmu+dmQ=', 'Content-Length': '0'} 2024-04-19T23:24:02.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000575cacfcad50d857-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.682 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:02.682 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:02.682 DEBUG:boto:path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.682 DEBUG:boto:auth_path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.682 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.682 DEBUG:boto:Path: /aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.682 DEBUG:boto:Data: 2024-04-19T23:24:02.683 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.683 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.683 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.683 DEBUG:boto:Params: {} 2024-04-19T23:24:02.683 DEBUG:boto:Token: None 2024-04-19T23:24:02.683 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.683 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bx6NZS7WzlRw16GhZTqgzmu+dmQ= 2024-04-19T23:24:02.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bx6NZS7WzlRw16GhZTqgzmu+dmQ=', 'Content-Length': '0'} 2024-04-19T23:24:02.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5f06315da34524b-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:02.686 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:02.686 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:02.686 DEBUG:boto:path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.686 DEBUG:boto:auth_path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.686 DEBUG:boto:Method: GET 2024-04-19T23:24:02.686 DEBUG:boto:Path: /aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.686 DEBUG:boto:Data: 2024-04-19T23:24:02.686 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.687 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.687 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.687 DEBUG:boto:Params: {} 2024-04-19T23:24:02.687 DEBUG:boto:Token: None 2024-04-19T23:24:02.687 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.687 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9xl1HSREotXTA0EnlHl2AaOW6Z8= 2024-04-19T23:24:02.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9xl1HSREotXTA0EnlHl2AaOW6Z8=', 'Content-Length': '0'} 2024-04-19T23:24:02.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a719a8f8eee4fb91-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.689 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:02.689 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:02.689 DEBUG:boto:path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.690 DEBUG:boto:auth_path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.690 DEBUG:boto:Method: GET 2024-04-19T23:24:02.690 DEBUG:boto:Path: /aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.690 DEBUG:boto:Data: 2024-04-19T23:24:02.690 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.690 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.690 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.690 DEBUG:boto:Params: {} 2024-04-19T23:24:02.690 DEBUG:boto:Token: None 2024-04-19T23:24:02.691 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%3A?versionId=null 2024-04-19T23:24:02.691 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9xl1HSREotXTA0EnlHl2AaOW6Z8= 2024-04-19T23:24:02.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9xl1HSREotXTA0EnlHl2AaOW6Z8=', 'Content-Length': '0'} 2024-04-19T23:24:02.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e26d02e9bc040e62-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.738 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.738 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.739 DEBUG:boto:path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:02.739 DEBUG:boto:auth_path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:02.739 DEBUG:boto:Method: GET 2024-04-19T23:24:02.739 DEBUG:boto:Path: /aktlji-12/?prefix=%3A 2024-04-19T23:24:02.739 DEBUG:boto:Data: 2024-04-19T23:24:02.739 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.739 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.740 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.740 DEBUG:boto:Params: {} 2024-04-19T23:24:02.740 DEBUG:boto:Token: None 2024-04-19T23:24:02.740 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.740 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA= 2024-04-19T23:24:02.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA=', 'Content-Length': '0'} 2024-04-19T23:24:02.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eb8e4a7fe7471b9-006622fd12-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.744 DEBUG:boto:b'aktlji-12:1000false:2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.744 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.744 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.744 DEBUG:boto:path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:02.744 DEBUG:boto:auth_path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:02.744 DEBUG:boto:Method: GET 2024-04-19T23:24:02.745 DEBUG:boto:Path: /aktlji-12/?prefix=%3A 2024-04-19T23:24:02.745 DEBUG:boto:Data: 2024-04-19T23:24:02.745 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.745 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.745 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.745 DEBUG:boto:Params: {} 2024-04-19T23:24:02.745 DEBUG:boto:Token: None 2024-04-19T23:24:02.745 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.745 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA= 2024-04-19T23:24:02.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA=', 'Content-Length': '0'} 2024-04-19T23:24:02.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e269bf0d8f58dcc7-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.750 DEBUG:boto:b'aktlji-12:1000false:2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.751 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:02.751 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:02.751 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.751 DEBUG:boto:Path: /aktlji-12/%3A 2024-04-19T23:24:02.751 DEBUG:boto:Data: 2024-04-19T23:24:02.751 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.751 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.751 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.751 DEBUG:boto:Params: {} 2024-04-19T23:24:02.751 DEBUG:boto:Token: None 2024-04-19T23:24:02.752 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%3A 2024-04-19T23:24:02.752 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lXfH27rAJVka7C3TNbiwadiGkDU= 2024-04-19T23:24:02.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lXfH27rAJVka7C3TNbiwadiGkDU=', 'Content-Length': '0'} 2024-04-19T23:24:02.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007d9222ce14131b5-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.754 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:02.754 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:02.754 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.754 DEBUG:boto:Path: /aktlji-12/%3A 2024-04-19T23:24:02.754 DEBUG:boto:Data: 2024-04-19T23:24:02.754 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.754 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.755 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.755 DEBUG:boto:Params: {} 2024-04-19T23:24:02.755 DEBUG:boto:Token: None 2024-04-19T23:24:02.755 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%3A 2024-04-19T23:24:02.755 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lXfH27rAJVka7C3TNbiwadiGkDU= 2024-04-19T23:24:02.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lXfH27rAJVka7C3TNbiwadiGkDU=', 'Content-Length': '0'} 2024-04-19T23:24:02.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038e999447cf518f9-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:02.757 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:02.757 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:02.757 DEBUG:boto:Method: GET 2024-04-19T23:24:02.757 DEBUG:boto:Path: /aktlji-12/%3A 2024-04-19T23:24:02.757 DEBUG:boto:Data: 2024-04-19T23:24:02.758 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.758 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.758 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.758 DEBUG:boto:Params: {} 2024-04-19T23:24:02.758 DEBUG:boto:Token: None 2024-04-19T23:24:02.758 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%3A 2024-04-19T23:24:02.758 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LnR8aoB/BfOP026iCOClpQsRLKk= 2024-04-19T23:24:02.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LnR8aoB/BfOP026iCOClpQsRLKk=', 'Content-Length': '0'} 2024-04-19T23:24:02.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ceda385375f56e2-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.760 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:02.760 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:02.761 DEBUG:boto:Method: GET 2024-04-19T23:24:02.761 DEBUG:boto:Path: /aktlji-12/%3A 2024-04-19T23:24:02.761 DEBUG:boto:Data: 2024-04-19T23:24:02.761 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.761 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.761 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.761 DEBUG:boto:Params: {} 2024-04-19T23:24:02.761 DEBUG:boto:Token: None 2024-04-19T23:24:02.761 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/%3A 2024-04-19T23:24:02.761 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LnR8aoB/BfOP026iCOClpQsRLKk= 2024-04-19T23:24:02.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LnR8aoB/BfOP026iCOClpQsRLKk=', 'Content-Length': '0'} 2024-04-19T23:24:02.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002471682a111642d1-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.764 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.764 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.764 DEBUG:boto:path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:02.764 DEBUG:boto:auth_path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:02.764 DEBUG:boto:Method: GET 2024-04-19T23:24:02.764 DEBUG:boto:Path: /aktlji-12/?prefix=%3A 2024-04-19T23:24:02.764 DEBUG:boto:Data: 2024-04-19T23:24:02.764 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.765 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.765 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.765 DEBUG:boto:Params: {} 2024-04-19T23:24:02.765 DEBUG:boto:Token: None 2024-04-19T23:24:02.765 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.765 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA= 2024-04-19T23:24:02.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA=', 'Content-Length': '0'} 2024-04-19T23:24:02.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b0757b7b23e6f6-006622fd12-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.767 DEBUG:boto:b'aktlji-12:1000false:2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.768 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.768 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.768 DEBUG:boto:path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:02.768 DEBUG:boto:auth_path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:02.768 DEBUG:boto:Method: GET 2024-04-19T23:24:02.768 DEBUG:boto:Path: /aktlji-12/?prefix=%3A 2024-04-19T23:24:02.768 DEBUG:boto:Data: 2024-04-19T23:24:02.768 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.769 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.769 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.769 DEBUG:boto:Params: {} 2024-04-19T23:24:02.769 DEBUG:boto:Token: None 2024-04-19T23:24:02.769 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.769 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA= 2024-04-19T23:24:02.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA=', 'Content-Length': '0'} 2024-04-19T23:24:02.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f74acd7bc2c9a9a3-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.814 DEBUG:boto:b'aktlji-12:1000false:2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:02.815 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:02.815 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:02.815 DEBUG:boto:path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.815 DEBUG:boto:auth_path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.815 DEBUG:boto:Method: GET 2024-04-19T23:24:02.815 DEBUG:boto:Path: /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.816 DEBUG:boto:Data: 2024-04-19T23:24:02.816 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.816 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.816 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.816 DEBUG:boto:Params: {} 2024-04-19T23:24:02.816 DEBUG:boto:Token: None 2024-04-19T23:24:02.816 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.816 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qwfXhpZxo2tC3D1NE85QpsMaN6M= 2024-04-19T23:24:02.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qwfXhpZxo2tC3D1NE85QpsMaN6M=', 'Content-Length': '0'} 2024-04-19T23:24:02.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b506a160b8a5736-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.819 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:02.819 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:02.819 DEBUG:boto:path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.819 DEBUG:boto:auth_path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.819 DEBUG:boto:Method: GET 2024-04-19T23:24:02.819 DEBUG:boto:Path: /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.819 DEBUG:boto:Data: 2024-04-19T23:24:02.820 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.820 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.820 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.820 DEBUG:boto:Params: {} 2024-04-19T23:24:02.820 DEBUG:boto:Token: None 2024-04-19T23:24:02.820 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.820 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qwfXhpZxo2tC3D1NE85QpsMaN6M= 2024-04-19T23:24:02.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qwfXhpZxo2tC3D1NE85QpsMaN6M=', 'Content-Length': '0'} 2024-04-19T23:24:02.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000aaa5c6642873412-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.822 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:02.822 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:02.822 DEBUG:boto:path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.823 DEBUG:boto:auth_path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.823 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.823 DEBUG:boto:Path: /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.823 DEBUG:boto:Data: 2024-04-19T23:24:02.823 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.823 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.823 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.823 DEBUG:boto:Params: {} 2024-04-19T23:24:02.823 DEBUG:boto:Token: None 2024-04-19T23:24:02.823 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/YE/36AIXE9oxNN4zRiM1LuAXk8= 2024-04-19T23:24:02.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/YE/36AIXE9oxNN4zRiM1LuAXk8=', 'Content-Length': '0'} 2024-04-19T23:24:02.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efe164ed2dd03bb2-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.826 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:02.826 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:02.826 DEBUG:boto:path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.826 DEBUG:boto:auth_path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.826 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.826 DEBUG:boto:Path: /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.826 DEBUG:boto:Data: 2024-04-19T23:24:02.826 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.826 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.827 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.827 DEBUG:boto:Params: {} 2024-04-19T23:24:02.827 DEBUG:boto:Token: None 2024-04-19T23:24:02.827 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.827 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/YE/36AIXE9oxNN4zRiM1LuAXk8= 2024-04-19T23:24:02.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/YE/36AIXE9oxNN4zRiM1LuAXk8=', 'Content-Length': '0'} 2024-04-19T23:24:02.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000adc59766579cc8b-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:02.829 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:02.829 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:02.829 DEBUG:boto:path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.829 DEBUG:boto:auth_path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.830 DEBUG:boto:Method: GET 2024-04-19T23:24:02.830 DEBUG:boto:Path: /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.830 DEBUG:boto:Data: 2024-04-19T23:24:02.830 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.830 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.830 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.830 DEBUG:boto:Params: {} 2024-04-19T23:24:02.830 DEBUG:boto:Token: None 2024-04-19T23:24:02.830 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.831 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qwfXhpZxo2tC3D1NE85QpsMaN6M= 2024-04-19T23:24:02.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qwfXhpZxo2tC3D1NE85QpsMaN6M=', 'Content-Length': '0'} 2024-04-19T23:24:02.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064640a5336352435-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.874 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:02.875 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:02.875 DEBUG:boto:path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.875 DEBUG:boto:auth_path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.875 DEBUG:boto:Method: GET 2024-04-19T23:24:02.875 DEBUG:boto:Path: /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.875 DEBUG:boto:Data: 2024-04-19T23:24:02.875 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.875 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.875 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.875 DEBUG:boto:Params: {} 2024-04-19T23:24:02.876 DEBUG:boto:Token: None 2024-04-19T23:24:02.876 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:02.876 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qwfXhpZxo2tC3D1NE85QpsMaN6M= 2024-04-19T23:24:02.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qwfXhpZxo2tC3D1NE85QpsMaN6M=', 'Content-Length': '0'} 2024-04-19T23:24:02.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000512b26808b2e3c7f-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.878 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.879 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.879 DEBUG:boto:path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:02.879 DEBUG:boto:auth_path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:02.879 DEBUG:boto:Method: GET 2024-04-19T23:24:02.879 DEBUG:boto:Path: /aktlji-12/?prefix=_myobj 2024-04-19T23:24:02.879 DEBUG:boto:Data: 2024-04-19T23:24:02.879 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.879 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.879 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.880 DEBUG:boto:Params: {} 2024-04-19T23:24:02.880 DEBUG:boto:Token: None 2024-04-19T23:24:02.880 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA= 2024-04-19T23:24:02.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA=', 'Content-Length': '0'} 2024-04-19T23:24:02.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cab020da37bc6d68-006622fd12-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.883 DEBUG:boto:b'aktlji-12_myobj1000false_myobj2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.883 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.883 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.883 DEBUG:boto:path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:02.884 DEBUG:boto:auth_path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:02.884 DEBUG:boto:Method: GET 2024-04-19T23:24:02.884 DEBUG:boto:Path: /aktlji-12/?prefix=_myobj 2024-04-19T23:24:02.884 DEBUG:boto:Data: 2024-04-19T23:24:02.884 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.884 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.884 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.884 DEBUG:boto:Params: {} 2024-04-19T23:24:02.884 DEBUG:boto:Token: None 2024-04-19T23:24:02.884 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.885 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA= 2024-04-19T23:24:02.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA=', 'Content-Length': '0'} 2024-04-19T23:24:02.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8ed2f18d3ad39b2-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.887 DEBUG:boto:b'aktlji-12_myobj1000false_myobj2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.888 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:02.888 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:02.888 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.888 DEBUG:boto:Path: /aktlji-12/_myobj 2024-04-19T23:24:02.888 DEBUG:boto:Data: 2024-04-19T23:24:02.888 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.888 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.888 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.888 DEBUG:boto:Params: {} 2024-04-19T23:24:02.889 DEBUG:boto:Token: None 2024-04-19T23:24:02.889 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/_myobj 2024-04-19T23:24:02.889 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fA/pz/ildYPCm59FDuFeYRjsZdU= 2024-04-19T23:24:02.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fA/pz/ildYPCm59FDuFeYRjsZdU=', 'Content-Length': '0'} 2024-04-19T23:24:02.890 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f67ee8ac31d7da3-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.891 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:02.891 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:02.891 DEBUG:boto:Method: HEAD 2024-04-19T23:24:02.891 DEBUG:boto:Path: /aktlji-12/_myobj 2024-04-19T23:24:02.891 DEBUG:boto:Data: 2024-04-19T23:24:02.891 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.891 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.891 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.892 DEBUG:boto:Params: {} 2024-04-19T23:24:02.892 DEBUG:boto:Token: None 2024-04-19T23:24:02.892 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/_myobj 2024-04-19T23:24:02.892 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fA/pz/ildYPCm59FDuFeYRjsZdU= 2024-04-19T23:24:02.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fA/pz/ildYPCm59FDuFeYRjsZdU=', 'Content-Length': '0'} 2024-04-19T23:24:02.894 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a1f37886673402d-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:02.894 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:02.894 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:02.894 DEBUG:boto:Method: GET 2024-04-19T23:24:02.894 DEBUG:boto:Path: /aktlji-12/_myobj 2024-04-19T23:24:02.895 DEBUG:boto:Data: 2024-04-19T23:24:02.895 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.895 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.895 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.895 DEBUG:boto:Params: {} 2024-04-19T23:24:02.895 DEBUG:boto:Token: None 2024-04-19T23:24:02.895 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/_myobj 2024-04-19T23:24:02.895 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vDyJ1OytPK+ipyEE/tgDNDI7bFg= 2024-04-19T23:24:02.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vDyJ1OytPK+ipyEE/tgDNDI7bFg=', 'Content-Length': '0'} 2024-04-19T23:24:02.897 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047efa1a3eafcf5ca-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.942 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:02.943 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:02.943 DEBUG:boto:Method: GET 2024-04-19T23:24:02.943 DEBUG:boto:Path: /aktlji-12/_myobj 2024-04-19T23:24:02.943 DEBUG:boto:Data: 2024-04-19T23:24:02.943 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.943 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.943 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.943 DEBUG:boto:Params: {} 2024-04-19T23:24:02.944 DEBUG:boto:Token: None 2024-04-19T23:24:02.944 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/_myobj 2024-04-19T23:24:02.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vDyJ1OytPK+ipyEE/tgDNDI7bFg= 2024-04-19T23:24:02.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vDyJ1OytPK+ipyEE/tgDNDI7bFg=', 'Content-Length': '0'} 2024-04-19T23:24:02.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c553a72fd8befd1f-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.946 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.946 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.947 DEBUG:boto:path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:02.947 DEBUG:boto:auth_path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:02.947 DEBUG:boto:Method: GET 2024-04-19T23:24:02.947 DEBUG:boto:Path: /aktlji-12/?prefix=_myobj 2024-04-19T23:24:02.947 DEBUG:boto:Data: 2024-04-19T23:24:02.947 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.947 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.947 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.947 DEBUG:boto:Params: {} 2024-04-19T23:24:02.948 DEBUG:boto:Token: None 2024-04-19T23:24:02.948 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.948 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA= 2024-04-19T23:24:02.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA=', 'Content-Length': '0'} 2024-04-19T23:24:02.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f22118d26d48aed8-006622fd12-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.951 DEBUG:boto:b'aktlji-12_myobj1000false_myobj2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.951 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:02.951 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:02.951 DEBUG:boto:path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:02.951 DEBUG:boto:auth_path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:02.951 DEBUG:boto:Method: GET 2024-04-19T23:24:02.952 DEBUG:boto:Path: /aktlji-12/?prefix=_myobj 2024-04-19T23:24:02.952 DEBUG:boto:Data: 2024-04-19T23:24:02.952 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.952 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.952 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.952 DEBUG:boto:Params: {} 2024-04-19T23:24:02.952 DEBUG:boto:Token: None 2024-04-19T23:24:02.952 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/ 2024-04-19T23:24:02.953 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA= 2024-04-19T23:24:02.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5/eVna9DTA8QoBmCgp/jFTbs0HA=', 'Content-Length': '0'} 2024-04-19T23:24:02.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026855f4f6253a4a4-006622fd12-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.958 DEBUG:boto:b'aktlji-12_myobj1000false_myobj2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:02.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:02.958 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:02.959 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:02.959 DEBUG:boto:path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:02.959 DEBUG:boto:auth_path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:02.959 DEBUG:boto:Method: GET 2024-04-19T23:24:02.959 DEBUG:boto:Path: /aktlji-12/myobj?versionId=null 2024-04-19T23:24:02.959 DEBUG:boto:Data: 2024-04-19T23:24:02.959 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.959 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:02.959 DEBUG:boto:Port: 8000 2024-04-19T23:24:02.959 DEBUG:boto:Params: {} 2024-04-19T23:24:02.960 DEBUG:boto:Token: None 2024-04-19T23:24:02.960 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/myobj?versionId=null 2024-04-19T23:24:02.960 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eES4l/LNuKkDxL/6hx5HGrP1YmQ= 2024-04-19T23:24:02.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eES4l/LNuKkDxL/6hx5HGrP1YmQ=', 'Content-Length': '0'} 2024-04-19T23:24:02.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a125eeaffe1dc15c-006622fd12-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:02.962 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:02.962 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:02.962 DEBUG:boto:path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:02.962 DEBUG:boto:auth_path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:02.962 DEBUG:boto:Method: GET 2024-04-19T23:24:02.962 DEBUG:boto:Path: /aktlji-12/myobj?versionId=null 2024-04-19T23:24:02.962 DEBUG:boto:Data: 2024-04-19T23:24:02.963 DEBUG:boto:Headers: {} 2024-04-19T23:24:02.963 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:02.963 DEBUG:boto:Port: 8001 2024-04-19T23:24:02.963 DEBUG:boto:Params: {} 2024-04-19T23:24:02.963 DEBUG:boto:Token: None 2024-04-19T23:24:02.963 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:02 GMT /aktlji-12/myobj?versionId=null 2024-04-19T23:24:02.964 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eES4l/LNuKkDxL/6hx5HGrP1YmQ= 2024-04-19T23:24:02.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eES4l/LNuKkDxL/6hx5HGrP1YmQ=', 'Content-Length': '0'} 2024-04-19T23:24:02.965 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc0272a5d9638825-006622fd12-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.006 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.006 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.007 DEBUG:boto:path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.007 DEBUG:boto:auth_path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.007 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.007 DEBUG:boto:Path: /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.007 DEBUG:boto:Data: 2024-04-19T23:24:03.007 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.008 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:03.008 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.008 DEBUG:boto:Params: {} 2024-04-19T23:24:03.008 DEBUG:boto:Token: None 2024-04-19T23:24:03.008 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.008 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaA2oWB4ONWOqhnt7U/rqByMdhk= 2024-04-19T23:24:03.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaA2oWB4ONWOqhnt7U/rqByMdhk=', 'Content-Length': '0'} 2024-04-19T23:24:03.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af0564c0da20a381-006622fd13-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.011 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.011 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.011 DEBUG:boto:path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.011 DEBUG:boto:auth_path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.011 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.011 DEBUG:boto:Path: /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.012 DEBUG:boto:Data: 2024-04-19T23:24:03.012 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.012 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.012 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.012 DEBUG:boto:Params: {} 2024-04-19T23:24:03.012 DEBUG:boto:Token: None 2024-04-19T23:24:03.012 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.012 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaA2oWB4ONWOqhnt7U/rqByMdhk= 2024-04-19T23:24:03.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaA2oWB4ONWOqhnt7U/rqByMdhk=', 'Content-Length': '0'} 2024-04-19T23:24:03.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a4e2fad3cfffdb2-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:03.015 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.015 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.015 DEBUG:boto:path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.015 DEBUG:boto:auth_path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.015 DEBUG:boto:Method: GET 2024-04-19T23:24:03.015 DEBUG:boto:Path: /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.015 DEBUG:boto:Data: 2024-04-19T23:24:03.016 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.016 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:03.016 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.016 DEBUG:boto:Params: {} 2024-04-19T23:24:03.016 DEBUG:boto:Token: None 2024-04-19T23:24:03.016 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.016 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zaMbPw3eCJIGI8HEmEW+ETdyD3M= 2024-04-19T23:24:03.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zaMbPw3eCJIGI8HEmEW+ETdyD3M=', 'Content-Length': '0'} 2024-04-19T23:24:03.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba4e003144a82f19-006622fd13-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.063 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.063 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.063 DEBUG:boto:path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.063 DEBUG:boto:auth_path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.063 DEBUG:boto:Method: GET 2024-04-19T23:24:03.064 DEBUG:boto:Path: /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.064 DEBUG:boto:Data: 2024-04-19T23:24:03.064 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.064 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.064 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.065 DEBUG:boto:Params: {} 2024-04-19T23:24:03.065 DEBUG:boto:Token: None 2024-04-19T23:24:03.065 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.065 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zaMbPw3eCJIGI8HEmEW+ETdyD3M= 2024-04-19T23:24:03.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zaMbPw3eCJIGI8HEmEW+ETdyD3M=', 'Content-Length': '0'} 2024-04-19T23:24:03.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de1f74d64de4fdd5-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.068 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.069 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.069 DEBUG:boto:path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.069 DEBUG:boto:auth_path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.069 DEBUG:boto:Method: GET 2024-04-19T23:24:03.069 DEBUG:boto:Path: /aktlji-12/?prefix=myobj 2024-04-19T23:24:03.069 DEBUG:boto:Data: 2024-04-19T23:24:03.069 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.069 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:03.070 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.070 DEBUG:boto:Params: {} 2024-04-19T23:24:03.070 DEBUG:boto:Token: None 2024-04-19T23:24:03.070 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.070 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003c7e408e178897b-006622fd13-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.073 DEBUG:boto:b'aktlji-12myobj1000falsemyobj2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.074 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.074 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.074 DEBUG:boto:path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.074 DEBUG:boto:auth_path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.075 DEBUG:boto:Method: GET 2024-04-19T23:24:03.075 DEBUG:boto:Path: /aktlji-12/?prefix=myobj 2024-04-19T23:24:03.075 DEBUG:boto:Data: 2024-04-19T23:24:03.075 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.075 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.075 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.075 DEBUG:boto:Params: {} 2024-04-19T23:24:03.075 DEBUG:boto:Token: None 2024-04-19T23:24:03.076 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.076 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065c809f048ea6004-006622fd13-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.080 DEBUG:boto:b'aktlji-12myobj1000falsemyobj2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.081 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.081 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.081 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.081 DEBUG:boto:Path: /aktlji-12/myobj 2024-04-19T23:24:03.081 DEBUG:boto:Data: 2024-04-19T23:24:03.081 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.082 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:03.082 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.082 DEBUG:boto:Params: {} 2024-04-19T23:24:03.082 DEBUG:boto:Token: None 2024-04-19T23:24:03.082 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj 2024-04-19T23:24:03.082 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OeV644+IRrwcZkL6Ph6AwZEcv1U= 2024-04-19T23:24:03.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OeV644+IRrwcZkL6Ph6AwZEcv1U=', 'Content-Length': '0'} 2024-04-19T23:24:03.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de4e24c13f6918f3-006622fd13-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.085 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.085 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.085 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.086 DEBUG:boto:Path: /aktlji-12/myobj 2024-04-19T23:24:03.086 DEBUG:boto:Data: 2024-04-19T23:24:03.086 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.086 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.086 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.086 DEBUG:boto:Params: {} 2024-04-19T23:24:03.086 DEBUG:boto:Token: None 2024-04-19T23:24:03.086 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj 2024-04-19T23:24:03.086 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OeV644+IRrwcZkL6Ph6AwZEcv1U= 2024-04-19T23:24:03.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OeV644+IRrwcZkL6Ph6AwZEcv1U=', 'Content-Length': '0'} 2024-04-19T23:24:03.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e03d70b9c2753a44-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:03.089 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.089 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.089 DEBUG:boto:Method: GET 2024-04-19T23:24:03.089 DEBUG:boto:Path: /aktlji-12/myobj 2024-04-19T23:24:03.089 DEBUG:boto:Data: 2024-04-19T23:24:03.089 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.090 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:03.090 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.090 DEBUG:boto:Params: {} 2024-04-19T23:24:03.090 DEBUG:boto:Token: None 2024-04-19T23:24:03.090 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj 2024-04-19T23:24:03.090 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PevKsrnZqMQSsWXWgzvQ5OyejDQ= 2024-04-19T23:24:03.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PevKsrnZqMQSsWXWgzvQ5OyejDQ=', 'Content-Length': '0'} 2024-04-19T23:24:03.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005766df84e81f0bb1-006622fd13-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.092 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.093 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.093 DEBUG:boto:Method: GET 2024-04-19T23:24:03.093 DEBUG:boto:Path: /aktlji-12/myobj 2024-04-19T23:24:03.093 DEBUG:boto:Data: 2024-04-19T23:24:03.093 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.094 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.094 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.094 DEBUG:boto:Params: {} 2024-04-19T23:24:03.094 DEBUG:boto:Token: None 2024-04-19T23:24:03.094 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj 2024-04-19T23:24:03.094 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PevKsrnZqMQSsWXWgzvQ5OyejDQ= 2024-04-19T23:24:03.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PevKsrnZqMQSsWXWgzvQ5OyejDQ=', 'Content-Length': '0'} 2024-04-19T23:24:03.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4006e37a089fe8c-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.096 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.097 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.097 DEBUG:boto:path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.097 DEBUG:boto:auth_path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.097 DEBUG:boto:Method: GET 2024-04-19T23:24:03.097 DEBUG:boto:Path: /aktlji-12/?prefix=myobj 2024-04-19T23:24:03.097 DEBUG:boto:Data: 2024-04-19T23:24:03.097 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.097 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:03.097 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.098 DEBUG:boto:Params: {} 2024-04-19T23:24:03.098 DEBUG:boto:Token: None 2024-04-19T23:24:03.098 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.098 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea83bdd7b43de97a-006622fd13-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.100 DEBUG:boto:b'aktlji-12myobj1000falsemyobj2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.101 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.101 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.101 DEBUG:boto:path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.101 DEBUG:boto:auth_path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.102 DEBUG:boto:Method: GET 2024-04-19T23:24:03.102 DEBUG:boto:Path: /aktlji-12/?prefix=myobj 2024-04-19T23:24:03.102 DEBUG:boto:Data: 2024-04-19T23:24:03.102 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.102 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.102 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.103 DEBUG:boto:Params: {} 2024-04-19T23:24:03.103 DEBUG:boto:Token: None 2024-04-19T23:24:03.103 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.103 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aed609eb10f4df74-006622fd13-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.107 DEBUG:boto:b'aktlji-12myobj1000falsemyobj2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.107 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-12 zones={test-zone1, test-zone3} 2024-04-19T23:24:03.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:03.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:03.108 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:24:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:03.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:03.224+0000 7f33ad444780 1 bucket sync caught up with source: 2024-04-19T23:24:03.221 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000001.6.6, , 00000000001.15.6, , , , 00000000001.10.6, 00000000001.6.6, , ] 2024-04-19T23:24:03.221 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000001.6.6, , 00000000001.15.6, , , , 00000000001.10.6, 00000000001.6.6, , ] 2024-04-19T23:24:03.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:03.224+0000 7f33ad444780 0 bucket checkpoint complete 2024-04-19T23:24:03.236 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-12 zones={test-zone2, test-zone3} 2024-04-19T23:24:03.236 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.236 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.236 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.236 DEBUG:boto:Path: /aktlji-12/ 2024-04-19T23:24:03.236 DEBUG:boto:Data: 2024-04-19T23:24:03.236 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.236 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.236 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.237 DEBUG:boto:Params: {} 2024-04-19T23:24:03.237 DEBUG:boto:Token: None 2024-04-19T23:24:03.237 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.237 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4inZJecZ6a7a5AdRU+AePiQsaKo= 2024-04-19T23:24:03.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4inZJecZ6a7a5AdRU+AePiQsaKo=', 'Content-Length': '0'} 2024-04-19T23:24:03.240 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bee83081fff9842-006622fd13-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.240 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.240 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.241 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.241 DEBUG:boto:Path: /aktlji-12/ 2024-04-19T23:24:03.241 DEBUG:boto:Data: 2024-04-19T23:24:03.241 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.241 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.241 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.241 DEBUG:boto:Params: {} 2024-04-19T23:24:03.241 DEBUG:boto:Token: None 2024-04-19T23:24:03.242 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.242 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4inZJecZ6a7a5AdRU+AePiQsaKo= 2024-04-19T23:24:03.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4inZJecZ6a7a5AdRU+AePiQsaKo=', 'Content-Length': '0'} 2024-04-19T23:24:03.244 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f374f42b38c4a2ef-006622fd13-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.244 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:24:03.245 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.245 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.245 DEBUG:boto:path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:03.245 DEBUG:boto:auth_path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:03.245 DEBUG:boto:Method: GET 2024-04-19T23:24:03.245 DEBUG:boto:Path: /aktlji-12/?versions&max-keys=999 2024-04-19T23:24:03.245 DEBUG:boto:Data: 2024-04-19T23:24:03.245 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.245 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.245 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.246 DEBUG:boto:Params: {} 2024-04-19T23:24:03.246 DEBUG:boto:Token: None 2024-04-19T23:24:03.246 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/?versions 2024-04-19T23:24:03.246 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IkGNFFZCU5CDuK23P9Caeq3hoMs= 2024-04-19T23:24:03.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IkGNFFZCU5CDuK23P9Caeq3hoMs=', 'Content-Length': '0'} 2024-04-19T23:24:03.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa8d1d56a2d1d349-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.249 DEBUG:boto:b'aktlji-12999false&nulltrue2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.249 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-19T23:24:03.250 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-19T23:24:03.250 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-19T23:24:03.250 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:03.250 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:24:03.250 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.250 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.250 DEBUG:boto:path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:03.250 DEBUG:boto:auth_path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:03.251 DEBUG:boto:Method: GET 2024-04-19T23:24:03.251 DEBUG:boto:Path: /aktlji-12/?versions&max-keys=999 2024-04-19T23:24:03.251 DEBUG:boto:Data: 2024-04-19T23:24:03.251 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.251 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.251 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.251 DEBUG:boto:Params: {} 2024-04-19T23:24:03.251 DEBUG:boto:Token: None 2024-04-19T23:24:03.251 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/?versions 2024-04-19T23:24:03.252 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IkGNFFZCU5CDuK23P9Caeq3hoMs= 2024-04-19T23:24:03.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IkGNFFZCU5CDuK23P9Caeq3hoMs=', 'Content-Length': '0'} 2024-04-19T23:24:03.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064d40465238aa3dc-006622fd13-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.254 DEBUG:boto:b'aktlji-12999false&nulltrue2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.254 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-19T23:24:03.255 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-19T23:24:03.255 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-19T23:24:03.255 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:03.255 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.255 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.255 DEBUG:boto:path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:03.255 DEBUG:boto:auth_path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:03.255 DEBUG:boto:Method: GET 2024-04-19T23:24:03.256 DEBUG:boto:Path: /aktlji-12/?versions&max-keys=999 2024-04-19T23:24:03.256 DEBUG:boto:Data: 2024-04-19T23:24:03.256 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.256 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.256 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.256 DEBUG:boto:Params: {} 2024-04-19T23:24:03.256 DEBUG:boto:Token: None 2024-04-19T23:24:03.256 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/?versions 2024-04-19T23:24:03.257 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IkGNFFZCU5CDuK23P9Caeq3hoMs= 2024-04-19T23:24:03.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IkGNFFZCU5CDuK23P9Caeq3hoMs=', 'Content-Length': '0'} 2024-04-19T23:24:03.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a60f56d3da0640c-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.259 DEBUG:boto:b'aktlji-12999false&nulltrue2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.260 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.260 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.260 DEBUG:boto:path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:03.260 DEBUG:boto:auth_path=/aktlji-12/?versions&max-keys=999 2024-04-19T23:24:03.260 DEBUG:boto:Method: GET 2024-04-19T23:24:03.260 DEBUG:boto:Path: /aktlji-12/?versions&max-keys=999 2024-04-19T23:24:03.260 DEBUG:boto:Data: 2024-04-19T23:24:03.260 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.261 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.261 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.261 DEBUG:boto:Params: {} 2024-04-19T23:24:03.261 DEBUG:boto:Token: None 2024-04-19T23:24:03.261 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/?versions 2024-04-19T23:24:03.261 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IkGNFFZCU5CDuK23P9Caeq3hoMs= 2024-04-19T23:24:03.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IkGNFFZCU5CDuK23P9Caeq3hoMs=', 'Content-Length': '0'} 2024-04-19T23:24:03.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007561629bc39e15bd-006622fd13-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.264 DEBUG:boto:b'aktlji-12999false&nulltrue2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:03.264 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:03.265 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:03.265 DEBUG:boto:path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:03.265 DEBUG:boto:auth_path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:03.265 DEBUG:boto:Method: GET 2024-04-19T23:24:03.265 DEBUG:boto:Path: /aktlji-12/%26?versionId=null 2024-04-19T23:24:03.265 DEBUG:boto:Data: 2024-04-19T23:24:03.265 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.265 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.265 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.265 DEBUG:boto:Params: {} 2024-04-19T23:24:03.266 DEBUG:boto:Token: None 2024-04-19T23:24:03.266 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%26?versionId=null 2024-04-19T23:24:03.266 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7tvnj5P0RJe+zlwoDOdw/K3xyX8= 2024-04-19T23:24:03.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7tvnj5P0RJe+zlwoDOdw/K3xyX8=', 'Content-Length': '0'} 2024-04-19T23:24:03.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009821cb77c4e5a047-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.314 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:03.314 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:03.315 DEBUG:boto:path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:03.315 DEBUG:boto:auth_path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:03.315 DEBUG:boto:Method: GET 2024-04-19T23:24:03.315 DEBUG:boto:Path: /aktlji-12/%26?versionId=null 2024-04-19T23:24:03.315 DEBUG:boto:Data: 2024-04-19T23:24:03.315 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.315 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.315 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.315 DEBUG:boto:Params: {} 2024-04-19T23:24:03.316 DEBUG:boto:Token: None 2024-04-19T23:24:03.316 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%26?versionId=null 2024-04-19T23:24:03.316 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7tvnj5P0RJe+zlwoDOdw/K3xyX8= 2024-04-19T23:24:03.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7tvnj5P0RJe+zlwoDOdw/K3xyX8=', 'Content-Length': '0'} 2024-04-19T23:24:03.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060d0e58261d58920-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.319 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:03.319 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:03.319 DEBUG:boto:path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:03.319 DEBUG:boto:auth_path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:03.319 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.319 DEBUG:boto:Path: /aktlji-12/%26?versionId=null 2024-04-19T23:24:03.319 DEBUG:boto:Data: 2024-04-19T23:24:03.319 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.319 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.320 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.320 DEBUG:boto:Params: {} 2024-04-19T23:24:03.320 DEBUG:boto:Token: None 2024-04-19T23:24:03.320 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%26?versionId=null 2024-04-19T23:24:03.320 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4fUNUzP+B8zyEYQ/1EYtyDgOuQQ= 2024-04-19T23:24:03.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4fUNUzP+B8zyEYQ/1EYtyDgOuQQ=', 'Content-Length': '0'} 2024-04-19T23:24:03.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f7a7fe1e8da5153-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.323 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:03.323 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:03.323 DEBUG:boto:path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:03.323 DEBUG:boto:auth_path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:03.323 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.324 DEBUG:boto:Path: /aktlji-12/%26?versionId=null 2024-04-19T23:24:03.324 DEBUG:boto:Data: 2024-04-19T23:24:03.324 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.324 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.324 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.324 DEBUG:boto:Params: {} 2024-04-19T23:24:03.324 DEBUG:boto:Token: None 2024-04-19T23:24:03.324 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%26?versionId=null 2024-04-19T23:24:03.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4fUNUzP+B8zyEYQ/1EYtyDgOuQQ= 2024-04-19T23:24:03.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4fUNUzP+B8zyEYQ/1EYtyDgOuQQ=', 'Content-Length': '0'} 2024-04-19T23:24:03.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0e42bbb64585955-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:03.327 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:03.327 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:03.327 DEBUG:boto:path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:03.327 DEBUG:boto:auth_path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:03.327 DEBUG:boto:Method: GET 2024-04-19T23:24:03.328 DEBUG:boto:Path: /aktlji-12/%26?versionId=null 2024-04-19T23:24:03.328 DEBUG:boto:Data: 2024-04-19T23:24:03.328 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.328 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.328 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.328 DEBUG:boto:Params: {} 2024-04-19T23:24:03.328 DEBUG:boto:Token: None 2024-04-19T23:24:03.328 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%26?versionId=null 2024-04-19T23:24:03.328 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7tvnj5P0RJe+zlwoDOdw/K3xyX8= 2024-04-19T23:24:03.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7tvnj5P0RJe+zlwoDOdw/K3xyX8=', 'Content-Length': '0'} 2024-04-19T23:24:03.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000728540dfcab91e76-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.331 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:03.331 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:03.331 DEBUG:boto:path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:03.331 DEBUG:boto:auth_path=/aktlji-12/%26?versionId=null 2024-04-19T23:24:03.331 DEBUG:boto:Method: GET 2024-04-19T23:24:03.331 DEBUG:boto:Path: /aktlji-12/%26?versionId=null 2024-04-19T23:24:03.331 DEBUG:boto:Data: 2024-04-19T23:24:03.331 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.332 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.332 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.332 DEBUG:boto:Params: {} 2024-04-19T23:24:03.332 DEBUG:boto:Token: None 2024-04-19T23:24:03.332 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%26?versionId=null 2024-04-19T23:24:03.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7tvnj5P0RJe+zlwoDOdw/K3xyX8= 2024-04-19T23:24:03.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7tvnj5P0RJe+zlwoDOdw/K3xyX8=', 'Content-Length': '0'} 2024-04-19T23:24:03.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e07776b284495f7a-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.378 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.379 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.379 DEBUG:boto:path=/aktlji-12/?prefix=%26 2024-04-19T23:24:03.379 DEBUG:boto:auth_path=/aktlji-12/?prefix=%26 2024-04-19T23:24:03.379 DEBUG:boto:Method: GET 2024-04-19T23:24:03.379 DEBUG:boto:Path: /aktlji-12/?prefix=%26 2024-04-19T23:24:03.379 DEBUG:boto:Data: 2024-04-19T23:24:03.379 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.379 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.379 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.379 DEBUG:boto:Params: {} 2024-04-19T23:24:03.380 DEBUG:boto:Token: None 2024-04-19T23:24:03.380 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.380 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbb33c74f9c2268a-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.383 DEBUG:boto:b'aktlji-12&1000false&2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.383 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.384 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.384 DEBUG:boto:path=/aktlji-12/?prefix=%26 2024-04-19T23:24:03.384 DEBUG:boto:auth_path=/aktlji-12/?prefix=%26 2024-04-19T23:24:03.384 DEBUG:boto:Method: GET 2024-04-19T23:24:03.384 DEBUG:boto:Path: /aktlji-12/?prefix=%26 2024-04-19T23:24:03.384 DEBUG:boto:Data: 2024-04-19T23:24:03.384 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.384 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.384 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.385 DEBUG:boto:Params: {} 2024-04-19T23:24:03.385 DEBUG:boto:Token: None 2024-04-19T23:24:03.385 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.385 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f36ac6d500a04463-006622fd13-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.388 DEBUG:boto:b'aktlji-12&1000false&2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.389 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:03.389 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:03.389 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.389 DEBUG:boto:Path: /aktlji-12/%26 2024-04-19T23:24:03.389 DEBUG:boto:Data: 2024-04-19T23:24:03.389 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.390 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.390 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.390 DEBUG:boto:Params: {} 2024-04-19T23:24:03.390 DEBUG:boto:Token: None 2024-04-19T23:24:03.390 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%26 2024-04-19T23:24:03.390 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:r4a3727pux9ee4qz+Fb75C78aKc= 2024-04-19T23:24:03.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r4a3727pux9ee4qz+Fb75C78aKc=', 'Content-Length': '0'} 2024-04-19T23:24:03.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e2f68363c08c656-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.393 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:03.393 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:03.393 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.393 DEBUG:boto:Path: /aktlji-12/%26 2024-04-19T23:24:03.393 DEBUG:boto:Data: 2024-04-19T23:24:03.393 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.394 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.394 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.394 DEBUG:boto:Params: {} 2024-04-19T23:24:03.394 DEBUG:boto:Token: None 2024-04-19T23:24:03.394 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%26 2024-04-19T23:24:03.394 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:r4a3727pux9ee4qz+Fb75C78aKc= 2024-04-19T23:24:03.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r4a3727pux9ee4qz+Fb75C78aKc=', 'Content-Length': '0'} 2024-04-19T23:24:03.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c857e26ee2499f13-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-19T23:24:03.397 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:03.397 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:03.397 DEBUG:boto:Method: GET 2024-04-19T23:24:03.397 DEBUG:boto:Path: /aktlji-12/%26 2024-04-19T23:24:03.397 DEBUG:boto:Data: 2024-04-19T23:24:03.397 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.397 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.397 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.397 DEBUG:boto:Params: {} 2024-04-19T23:24:03.398 DEBUG:boto:Token: None 2024-04-19T23:24:03.398 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%26 2024-04-19T23:24:03.398 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:n7DnSLnNCLfKHj2a84QxxOicqsU= 2024-04-19T23:24:03.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:n7DnSLnNCLfKHj2a84QxxOicqsU=', 'Content-Length': '0'} 2024-04-19T23:24:03.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b00635bf8f4f4999-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.442 DEBUG:boto:path=/aktlji-12/%26 2024-04-19T23:24:03.443 DEBUG:boto:auth_path=/aktlji-12/%26 2024-04-19T23:24:03.443 DEBUG:boto:Method: GET 2024-04-19T23:24:03.443 DEBUG:boto:Path: /aktlji-12/%26 2024-04-19T23:24:03.443 DEBUG:boto:Data: 2024-04-19T23:24:03.443 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.443 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.443 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.444 DEBUG:boto:Params: {} 2024-04-19T23:24:03.444 DEBUG:boto:Token: None 2024-04-19T23:24:03.444 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%26 2024-04-19T23:24:03.444 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:n7DnSLnNCLfKHj2a84QxxOicqsU= 2024-04-19T23:24:03.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:n7DnSLnNCLfKHj2a84QxxOicqsU=', 'Content-Length': '0'} 2024-04-19T23:24:03.447 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053e9a587513f8c56-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.447 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.447 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.447 DEBUG:boto:path=/aktlji-12/?prefix=%26 2024-04-19T23:24:03.447 DEBUG:boto:auth_path=/aktlji-12/?prefix=%26 2024-04-19T23:24:03.447 DEBUG:boto:Method: GET 2024-04-19T23:24:03.448 DEBUG:boto:Path: /aktlji-12/?prefix=%26 2024-04-19T23:24:03.448 DEBUG:boto:Data: 2024-04-19T23:24:03.448 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.448 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.448 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.448 DEBUG:boto:Params: {} 2024-04-19T23:24:03.448 DEBUG:boto:Token: None 2024-04-19T23:24:03.448 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.449 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005adfd3e4ce22d301-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.452 DEBUG:boto:b'aktlji-12&1000false&2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.452 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.452 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.452 DEBUG:boto:path=/aktlji-12/?prefix=%26 2024-04-19T23:24:03.452 DEBUG:boto:auth_path=/aktlji-12/?prefix=%26 2024-04-19T23:24:03.453 DEBUG:boto:Method: GET 2024-04-19T23:24:03.453 DEBUG:boto:Path: /aktlji-12/?prefix=%26 2024-04-19T23:24:03.453 DEBUG:boto:Data: 2024-04-19T23:24:03.453 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.453 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.453 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.453 DEBUG:boto:Params: {} 2024-04-19T23:24:03.453 DEBUG:boto:Token: None 2024-04-19T23:24:03.454 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.454 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016ed63d81ab6bd9a-006622fd13-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.498 DEBUG:boto:b'aktlji-12&1000false&2024-04-19T23:23:14.344Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:03.499 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:03.499 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:03.500 DEBUG:boto:path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.500 DEBUG:boto:auth_path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.500 DEBUG:boto:Method: GET 2024-04-19T23:24:03.500 DEBUG:boto:Path: /aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.500 DEBUG:boto:Data: 2024-04-19T23:24:03.501 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.501 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.501 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.501 DEBUG:boto:Params: {} 2024-04-19T23:24:03.501 DEBUG:boto:Token: None 2024-04-19T23:24:03.501 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.501 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wLEssOS0A/PPRHuPrpRYH/Xhrp8= 2024-04-19T23:24:03.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wLEssOS0A/PPRHuPrpRYH/Xhrp8=', 'Content-Length': '0'} 2024-04-19T23:24:03.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a22180200266038-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.504 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:03.504 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:03.504 DEBUG:boto:path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.504 DEBUG:boto:auth_path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.504 DEBUG:boto:Method: GET 2024-04-19T23:24:03.504 DEBUG:boto:Path: /aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.504 DEBUG:boto:Data: 2024-04-19T23:24:03.504 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.505 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.505 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.505 DEBUG:boto:Params: {} 2024-04-19T23:24:03.505 DEBUG:boto:Token: None 2024-04-19T23:24:03.505 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.505 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wLEssOS0A/PPRHuPrpRYH/Xhrp8= 2024-04-19T23:24:03.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wLEssOS0A/PPRHuPrpRYH/Xhrp8=', 'Content-Length': '0'} 2024-04-19T23:24:03.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089139778262e88d5-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.508 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:03.508 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:03.508 DEBUG:boto:path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.508 DEBUG:boto:auth_path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.508 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.508 DEBUG:boto:Path: /aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.508 DEBUG:boto:Data: 2024-04-19T23:24:03.508 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.508 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.508 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.508 DEBUG:boto:Params: {} 2024-04-19T23:24:03.509 DEBUG:boto:Token: None 2024-04-19T23:24:03.509 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.509 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rPfaPA3moSBayW8lS++Cd1WsbJw= 2024-04-19T23:24:03.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rPfaPA3moSBayW8lS++Cd1WsbJw=', 'Content-Length': '0'} 2024-04-19T23:24:03.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ce05e7a49346e97-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.511 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:03.511 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:03.511 DEBUG:boto:path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.511 DEBUG:boto:auth_path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.511 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.511 DEBUG:boto:Path: /aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.512 DEBUG:boto:Data: 2024-04-19T23:24:03.512 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.512 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.512 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.512 DEBUG:boto:Params: {} 2024-04-19T23:24:03.512 DEBUG:boto:Token: None 2024-04-19T23:24:03.512 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.512 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rPfaPA3moSBayW8lS++Cd1WsbJw= 2024-04-19T23:24:03.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rPfaPA3moSBayW8lS++Cd1WsbJw=', 'Content-Length': '0'} 2024-04-19T23:24:03.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000251c18f29ae3f6f5-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:03.515 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:03.515 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:03.515 DEBUG:boto:path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.515 DEBUG:boto:auth_path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.515 DEBUG:boto:Method: GET 2024-04-19T23:24:03.515 DEBUG:boto:Path: /aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.515 DEBUG:boto:Data: 2024-04-19T23:24:03.515 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.515 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.515 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.515 DEBUG:boto:Params: {} 2024-04-19T23:24:03.516 DEBUG:boto:Token: None 2024-04-19T23:24:03.516 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.516 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wLEssOS0A/PPRHuPrpRYH/Xhrp8= 2024-04-19T23:24:03.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wLEssOS0A/PPRHuPrpRYH/Xhrp8=', 'Content-Length': '0'} 2024-04-19T23:24:03.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015d294bc10fb8a60-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.518 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:03.519 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:03.519 DEBUG:boto:path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.519 DEBUG:boto:auth_path=/aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.519 DEBUG:boto:Method: GET 2024-04-19T23:24:03.519 DEBUG:boto:Path: /aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.519 DEBUG:boto:Data: 2024-04-19T23:24:03.519 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.519 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.519 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.519 DEBUG:boto:Params: {} 2024-04-19T23:24:03.520 DEBUG:boto:Token: None 2024-04-19T23:24:03.520 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%3A?versionId=null 2024-04-19T23:24:03.520 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wLEssOS0A/PPRHuPrpRYH/Xhrp8= 2024-04-19T23:24:03.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wLEssOS0A/PPRHuPrpRYH/Xhrp8=', 'Content-Length': '0'} 2024-04-19T23:24:03.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b32cf599ccbbe09-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.562 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.563 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.563 DEBUG:boto:path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:03.563 DEBUG:boto:auth_path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:03.563 DEBUG:boto:Method: GET 2024-04-19T23:24:03.563 DEBUG:boto:Path: /aktlji-12/?prefix=%3A 2024-04-19T23:24:03.563 DEBUG:boto:Data: 2024-04-19T23:24:03.563 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.563 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.563 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.563 DEBUG:boto:Params: {} 2024-04-19T23:24:03.564 DEBUG:boto:Token: None 2024-04-19T23:24:03.564 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.564 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c03a1e674e82bb3-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.567 DEBUG:boto:b'aktlji-12:1000false:2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.568 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.568 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.568 DEBUG:boto:path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:03.568 DEBUG:boto:auth_path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:03.568 DEBUG:boto:Method: GET 2024-04-19T23:24:03.568 DEBUG:boto:Path: /aktlji-12/?prefix=%3A 2024-04-19T23:24:03.568 DEBUG:boto:Data: 2024-04-19T23:24:03.568 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.568 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.568 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.568 DEBUG:boto:Params: {} 2024-04-19T23:24:03.569 DEBUG:boto:Token: None 2024-04-19T23:24:03.569 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.569 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e9db9b32fa00072-006622fd13-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.574 DEBUG:boto:b'aktlji-12:1000false:2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.574 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:03.574 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:03.575 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.575 DEBUG:boto:Path: /aktlji-12/%3A 2024-04-19T23:24:03.575 DEBUG:boto:Data: 2024-04-19T23:24:03.575 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.575 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.575 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.575 DEBUG:boto:Params: {} 2024-04-19T23:24:03.575 DEBUG:boto:Token: None 2024-04-19T23:24:03.575 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%3A 2024-04-19T23:24:03.575 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:byq7+omUjTOPh8lAEwQrYXxLA5w= 2024-04-19T23:24:03.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:byq7+omUjTOPh8lAEwQrYXxLA5w=', 'Content-Length': '0'} 2024-04-19T23:24:03.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca2876a1a46a4f08-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.578 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:03.578 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:03.578 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.578 DEBUG:boto:Path: /aktlji-12/%3A 2024-04-19T23:24:03.578 DEBUG:boto:Data: 2024-04-19T23:24:03.578 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.578 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.578 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.578 DEBUG:boto:Params: {} 2024-04-19T23:24:03.579 DEBUG:boto:Token: None 2024-04-19T23:24:03.579 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%3A 2024-04-19T23:24:03.579 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:byq7+omUjTOPh8lAEwQrYXxLA5w= 2024-04-19T23:24:03.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:byq7+omUjTOPh8lAEwQrYXxLA5w=', 'Content-Length': '0'} 2024-04-19T23:24:03.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9287e801e73d24b-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-19T23:24:03.581 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:03.581 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:03.581 DEBUG:boto:Method: GET 2024-04-19T23:24:03.581 DEBUG:boto:Path: /aktlji-12/%3A 2024-04-19T23:24:03.582 DEBUG:boto:Data: 2024-04-19T23:24:03.582 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.582 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.582 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.582 DEBUG:boto:Params: {} 2024-04-19T23:24:03.582 DEBUG:boto:Token: None 2024-04-19T23:24:03.582 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%3A 2024-04-19T23:24:03.582 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WfNoTj9yXnVZrDFvnEfdOiL4qoY= 2024-04-19T23:24:03.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WfNoTj9yXnVZrDFvnEfdOiL4qoY=', 'Content-Length': '0'} 2024-04-19T23:24:03.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d901b77e1b67f380-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.630 DEBUG:boto:path=/aktlji-12/%3A 2024-04-19T23:24:03.631 DEBUG:boto:auth_path=/aktlji-12/%3A 2024-04-19T23:24:03.631 DEBUG:boto:Method: GET 2024-04-19T23:24:03.631 DEBUG:boto:Path: /aktlji-12/%3A 2024-04-19T23:24:03.631 DEBUG:boto:Data: 2024-04-19T23:24:03.631 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.631 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.631 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.631 DEBUG:boto:Params: {} 2024-04-19T23:24:03.631 DEBUG:boto:Token: None 2024-04-19T23:24:03.631 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/%3A 2024-04-19T23:24:03.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WfNoTj9yXnVZrDFvnEfdOiL4qoY= 2024-04-19T23:24:03.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WfNoTj9yXnVZrDFvnEfdOiL4qoY=', 'Content-Length': '0'} 2024-04-19T23:24:03.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001489a8b761210242-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.634 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.634 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.634 DEBUG:boto:path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:03.634 DEBUG:boto:auth_path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:03.634 DEBUG:boto:Method: GET 2024-04-19T23:24:03.634 DEBUG:boto:Path: /aktlji-12/?prefix=%3A 2024-04-19T23:24:03.634 DEBUG:boto:Data: 2024-04-19T23:24:03.634 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.635 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.635 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.635 DEBUG:boto:Params: {} 2024-04-19T23:24:03.635 DEBUG:boto:Token: None 2024-04-19T23:24:03.635 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.635 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa2a1f7d5e4e2584-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.638 DEBUG:boto:b'aktlji-12:1000false:2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.638 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.638 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.638 DEBUG:boto:path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:03.638 DEBUG:boto:auth_path=/aktlji-12/?prefix=%3A 2024-04-19T23:24:03.639 DEBUG:boto:Method: GET 2024-04-19T23:24:03.639 DEBUG:boto:Path: /aktlji-12/?prefix=%3A 2024-04-19T23:24:03.639 DEBUG:boto:Data: 2024-04-19T23:24:03.639 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.639 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.639 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.639 DEBUG:boto:Params: {} 2024-04-19T23:24:03.639 DEBUG:boto:Token: None 2024-04-19T23:24:03.639 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.640 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0f31242a89ea520-006622fd13-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.642 DEBUG:boto:b'aktlji-12:1000false:2024-04-19T23:23:14.328Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:03.643 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:03.643 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:03.643 DEBUG:boto:path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.643 DEBUG:boto:auth_path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.643 DEBUG:boto:Method: GET 2024-04-19T23:24:03.643 DEBUG:boto:Path: /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.643 DEBUG:boto:Data: 2024-04-19T23:24:03.643 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.643 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.643 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.643 DEBUG:boto:Params: {} 2024-04-19T23:24:03.644 DEBUG:boto:Token: None 2024-04-19T23:24:03.644 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.644 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u1YnAMQYtOtmXe0uXjjVTb7NWiU= 2024-04-19T23:24:03.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u1YnAMQYtOtmXe0uXjjVTb7NWiU=', 'Content-Length': '0'} 2024-04-19T23:24:03.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f0701b4ede66e62-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.646 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:03.646 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:03.646 DEBUG:boto:path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.646 DEBUG:boto:auth_path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.646 DEBUG:boto:Method: GET 2024-04-19T23:24:03.647 DEBUG:boto:Path: /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.647 DEBUG:boto:Data: 2024-04-19T23:24:03.647 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.647 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.647 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.647 DEBUG:boto:Params: {} 2024-04-19T23:24:03.647 DEBUG:boto:Token: None 2024-04-19T23:24:03.647 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.647 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u1YnAMQYtOtmXe0uXjjVTb7NWiU= 2024-04-19T23:24:03.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u1YnAMQYtOtmXe0uXjjVTb7NWiU=', 'Content-Length': '0'} 2024-04-19T23:24:03.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006c1f9bc36ddacec-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.650 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:03.650 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:03.650 DEBUG:boto:path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.650 DEBUG:boto:auth_path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.650 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.650 DEBUG:boto:Path: /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.650 DEBUG:boto:Data: 2024-04-19T23:24:03.650 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.650 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.651 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.651 DEBUG:boto:Params: {} 2024-04-19T23:24:03.651 DEBUG:boto:Token: None 2024-04-19T23:24:03.651 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.651 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rPZ+Ybfy1Cmr2XQS58eWek0j8cE= 2024-04-19T23:24:03.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rPZ+Ybfy1Cmr2XQS58eWek0j8cE=', 'Content-Length': '0'} 2024-04-19T23:24:03.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000579e6b2764a4e5d-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.653 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:03.653 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:03.653 DEBUG:boto:path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.653 DEBUG:boto:auth_path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.653 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.654 DEBUG:boto:Path: /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.654 DEBUG:boto:Data: 2024-04-19T23:24:03.654 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.654 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.654 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.654 DEBUG:boto:Params: {} 2024-04-19T23:24:03.654 DEBUG:boto:Token: None 2024-04-19T23:24:03.654 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.654 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rPZ+Ybfy1Cmr2XQS58eWek0j8cE= 2024-04-19T23:24:03.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rPZ+Ybfy1Cmr2XQS58eWek0j8cE=', 'Content-Length': '0'} 2024-04-19T23:24:03.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d6bf553bfc04cb4-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:03.657 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:03.657 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:03.657 DEBUG:boto:path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.657 DEBUG:boto:auth_path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.657 DEBUG:boto:Method: GET 2024-04-19T23:24:03.657 DEBUG:boto:Path: /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.657 DEBUG:boto:Data: 2024-04-19T23:24:03.657 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.657 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.657 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.657 DEBUG:boto:Params: {} 2024-04-19T23:24:03.658 DEBUG:boto:Token: None 2024-04-19T23:24:03.658 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.658 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u1YnAMQYtOtmXe0uXjjVTb7NWiU= 2024-04-19T23:24:03.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u1YnAMQYtOtmXe0uXjjVTb7NWiU=', 'Content-Length': '0'} 2024-04-19T23:24:03.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e587252a81193b5b-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.660 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:03.660 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:03.660 DEBUG:boto:path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.660 DEBUG:boto:auth_path=/aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.660 DEBUG:boto:Method: GET 2024-04-19T23:24:03.660 DEBUG:boto:Path: /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.661 DEBUG:boto:Data: 2024-04-19T23:24:03.661 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.661 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.661 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.661 DEBUG:boto:Params: {} 2024-04-19T23:24:03.661 DEBUG:boto:Token: None 2024-04-19T23:24:03.661 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/_myobj?versionId=null 2024-04-19T23:24:03.661 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u1YnAMQYtOtmXe0uXjjVTb7NWiU= 2024-04-19T23:24:03.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u1YnAMQYtOtmXe0uXjjVTb7NWiU=', 'Content-Length': '0'} 2024-04-19T23:24:03.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c175e20c3ec2d265-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.663 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.664 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.664 DEBUG:boto:path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:03.664 DEBUG:boto:auth_path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:03.664 DEBUG:boto:Method: GET 2024-04-19T23:24:03.664 DEBUG:boto:Path: /aktlji-12/?prefix=_myobj 2024-04-19T23:24:03.664 DEBUG:boto:Data: 2024-04-19T23:24:03.664 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.664 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.664 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.664 DEBUG:boto:Params: {} 2024-04-19T23:24:03.664 DEBUG:boto:Token: None 2024-04-19T23:24:03.665 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.665 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c5471378659d7e7-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.667 DEBUG:boto:b'aktlji-12_myobj1000false_myobj2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.668 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.668 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.668 DEBUG:boto:path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:03.668 DEBUG:boto:auth_path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:03.668 DEBUG:boto:Method: GET 2024-04-19T23:24:03.668 DEBUG:boto:Path: /aktlji-12/?prefix=_myobj 2024-04-19T23:24:03.668 DEBUG:boto:Data: 2024-04-19T23:24:03.668 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.668 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.668 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.669 DEBUG:boto:Params: {} 2024-04-19T23:24:03.669 DEBUG:boto:Token: None 2024-04-19T23:24:03.669 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.669 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f978be6ee65dab1d-006622fd13-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.718 DEBUG:boto:b'aktlji-12_myobj1000false_myobj2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.719 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:03.719 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:03.719 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.719 DEBUG:boto:Path: /aktlji-12/_myobj 2024-04-19T23:24:03.719 DEBUG:boto:Data: 2024-04-19T23:24:03.719 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.719 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.719 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.719 DEBUG:boto:Params: {} 2024-04-19T23:24:03.719 DEBUG:boto:Token: None 2024-04-19T23:24:03.720 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/_myobj 2024-04-19T23:24:03.720 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TTwlGon3hkBzdsF1HSjb9uO2Fr4= 2024-04-19T23:24:03.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TTwlGon3hkBzdsF1HSjb9uO2Fr4=', 'Content-Length': '0'} 2024-04-19T23:24:03.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cce7e939676539ca-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.722 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:03.722 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:03.722 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.722 DEBUG:boto:Path: /aktlji-12/_myobj 2024-04-19T23:24:03.722 DEBUG:boto:Data: 2024-04-19T23:24:03.723 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.723 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.723 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.723 DEBUG:boto:Params: {} 2024-04-19T23:24:03.723 DEBUG:boto:Token: None 2024-04-19T23:24:03.723 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/_myobj 2024-04-19T23:24:03.723 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TTwlGon3hkBzdsF1HSjb9uO2Fr4= 2024-04-19T23:24:03.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TTwlGon3hkBzdsF1HSjb9uO2Fr4=', 'Content-Length': '0'} 2024-04-19T23:24:03.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017da4be719f17fe3-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-19T23:24:03.725 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:03.726 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:03.726 DEBUG:boto:Method: GET 2024-04-19T23:24:03.726 DEBUG:boto:Path: /aktlji-12/_myobj 2024-04-19T23:24:03.726 DEBUG:boto:Data: 2024-04-19T23:24:03.726 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.726 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.726 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.726 DEBUG:boto:Params: {} 2024-04-19T23:24:03.726 DEBUG:boto:Token: None 2024-04-19T23:24:03.726 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/_myobj 2024-04-19T23:24:03.727 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+rbuJgXtbm1FfL+1MbAHg8zpsbM= 2024-04-19T23:24:03.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+rbuJgXtbm1FfL+1MbAHg8zpsbM=', 'Content-Length': '0'} 2024-04-19T23:24:03.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f246a40a2e761fa-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.729 DEBUG:boto:path=/aktlji-12/_myobj 2024-04-19T23:24:03.729 DEBUG:boto:auth_path=/aktlji-12/_myobj 2024-04-19T23:24:03.729 DEBUG:boto:Method: GET 2024-04-19T23:24:03.729 DEBUG:boto:Path: /aktlji-12/_myobj 2024-04-19T23:24:03.729 DEBUG:boto:Data: 2024-04-19T23:24:03.730 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.730 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.730 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.730 DEBUG:boto:Params: {} 2024-04-19T23:24:03.730 DEBUG:boto:Token: None 2024-04-19T23:24:03.730 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/_myobj 2024-04-19T23:24:03.730 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+rbuJgXtbm1FfL+1MbAHg8zpsbM= 2024-04-19T23:24:03.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+rbuJgXtbm1FfL+1MbAHg8zpsbM=', 'Content-Length': '0'} 2024-04-19T23:24:03.732 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000235762f0eee45290-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.732 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.732 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.732 DEBUG:boto:path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:03.733 DEBUG:boto:auth_path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:03.733 DEBUG:boto:Method: GET 2024-04-19T23:24:03.733 DEBUG:boto:Path: /aktlji-12/?prefix=_myobj 2024-04-19T23:24:03.733 DEBUG:boto:Data: 2024-04-19T23:24:03.733 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.733 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.733 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.733 DEBUG:boto:Params: {} 2024-04-19T23:24:03.733 DEBUG:boto:Token: None 2024-04-19T23:24:03.733 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.734 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d485966fca9ec75-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.736 DEBUG:boto:b'aktlji-12_myobj1000false_myobj2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.736 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.737 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.737 DEBUG:boto:path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:03.737 DEBUG:boto:auth_path=/aktlji-12/?prefix=_myobj 2024-04-19T23:24:03.737 DEBUG:boto:Method: GET 2024-04-19T23:24:03.737 DEBUG:boto:Path: /aktlji-12/?prefix=_myobj 2024-04-19T23:24:03.737 DEBUG:boto:Data: 2024-04-19T23:24:03.737 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.737 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.737 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.737 DEBUG:boto:Params: {} 2024-04-19T23:24:03.737 DEBUG:boto:Token: None 2024-04-19T23:24:03.738 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.738 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6e6d0b17b85f09b-006622fd13-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.740 DEBUG:boto:b'aktlji-12_myobj1000false_myobj2024-04-19T23:23:14.278Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:03.741 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.741 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.741 DEBUG:boto:path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.741 DEBUG:boto:auth_path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.741 DEBUG:boto:Method: GET 2024-04-19T23:24:03.741 DEBUG:boto:Path: /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.741 DEBUG:boto:Data: 2024-04-19T23:24:03.741 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.741 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.742 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.742 DEBUG:boto:Params: {} 2024-04-19T23:24:03.742 DEBUG:boto:Token: None 2024-04-19T23:24:03.742 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.742 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zaMbPw3eCJIGI8HEmEW+ETdyD3M= 2024-04-19T23:24:03.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zaMbPw3eCJIGI8HEmEW+ETdyD3M=', 'Content-Length': '0'} 2024-04-19T23:24:03.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000947219514bf269de-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.744 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.745 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.745 DEBUG:boto:path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.745 DEBUG:boto:auth_path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.745 DEBUG:boto:Method: GET 2024-04-19T23:24:03.745 DEBUG:boto:Path: /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.745 DEBUG:boto:Data: 2024-04-19T23:24:03.745 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.745 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.745 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.745 DEBUG:boto:Params: {} 2024-04-19T23:24:03.746 DEBUG:boto:Token: None 2024-04-19T23:24:03.746 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.746 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zaMbPw3eCJIGI8HEmEW+ETdyD3M= 2024-04-19T23:24:03.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zaMbPw3eCJIGI8HEmEW+ETdyD3M=', 'Content-Length': '0'} 2024-04-19T23:24:03.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e546e31f0cba4146-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.748 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.748 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.748 DEBUG:boto:path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.748 DEBUG:boto:auth_path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.748 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.748 DEBUG:boto:Path: /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.748 DEBUG:boto:Data: 2024-04-19T23:24:03.749 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.749 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.749 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.749 DEBUG:boto:Params: {} 2024-04-19T23:24:03.749 DEBUG:boto:Token: None 2024-04-19T23:24:03.749 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.749 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaA2oWB4ONWOqhnt7U/rqByMdhk= 2024-04-19T23:24:03.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaA2oWB4ONWOqhnt7U/rqByMdhk=', 'Content-Length': '0'} 2024-04-19T23:24:03.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed45f18760d3ed2f-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.751 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.751 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.751 DEBUG:boto:path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.751 DEBUG:boto:auth_path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.751 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.752 DEBUG:boto:Path: /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.752 DEBUG:boto:Data: 2024-04-19T23:24:03.752 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.752 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.752 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.752 DEBUG:boto:Params: {} 2024-04-19T23:24:03.752 DEBUG:boto:Token: None 2024-04-19T23:24:03.752 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.752 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaA2oWB4ONWOqhnt7U/rqByMdhk= 2024-04-19T23:24:03.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaA2oWB4ONWOqhnt7U/rqByMdhk=', 'Content-Length': '0'} 2024-04-19T23:24:03.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004714c835ae768f78-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:03.755 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.755 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.755 DEBUG:boto:path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.755 DEBUG:boto:auth_path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.755 DEBUG:boto:Method: GET 2024-04-19T23:24:03.755 DEBUG:boto:Path: /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.755 DEBUG:boto:Data: 2024-04-19T23:24:03.755 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.755 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.756 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.756 DEBUG:boto:Params: {} 2024-04-19T23:24:03.756 DEBUG:boto:Token: None 2024-04-19T23:24:03.756 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.756 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zaMbPw3eCJIGI8HEmEW+ETdyD3M= 2024-04-19T23:24:03.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zaMbPw3eCJIGI8HEmEW+ETdyD3M=', 'Content-Length': '0'} 2024-04-19T23:24:03.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a38609a8dbcb15b-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.758 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.758 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.759 DEBUG:boto:path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.759 DEBUG:boto:auth_path=/aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.759 DEBUG:boto:Method: GET 2024-04-19T23:24:03.759 DEBUG:boto:Path: /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.759 DEBUG:boto:Data: 2024-04-19T23:24:03.759 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.759 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.759 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.759 DEBUG:boto:Params: {} 2024-04-19T23:24:03.759 DEBUG:boto:Token: None 2024-04-19T23:24:03.759 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj?versionId=null 2024-04-19T23:24:03.760 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zaMbPw3eCJIGI8HEmEW+ETdyD3M= 2024-04-19T23:24:03.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zaMbPw3eCJIGI8HEmEW+ETdyD3M=', 'Content-Length': '0'} 2024-04-19T23:24:03.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c588d10c1c88ec93-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.762 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.762 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.762 DEBUG:boto:path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.762 DEBUG:boto:auth_path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.762 DEBUG:boto:Method: GET 2024-04-19T23:24:03.762 DEBUG:boto:Path: /aktlji-12/?prefix=myobj 2024-04-19T23:24:03.763 DEBUG:boto:Data: 2024-04-19T23:24:03.763 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.763 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.763 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.763 DEBUG:boto:Params: {} 2024-04-19T23:24:03.763 DEBUG:boto:Token: None 2024-04-19T23:24:03.763 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.763 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000377d894bc78c502f-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.766 DEBUG:boto:b'aktlji-12myobj1000falsemyobj2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.766 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.766 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.766 DEBUG:boto:path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.766 DEBUG:boto:auth_path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.767 DEBUG:boto:Method: GET 2024-04-19T23:24:03.767 DEBUG:boto:Path: /aktlji-12/?prefix=myobj 2024-04-19T23:24:03.767 DEBUG:boto:Data: 2024-04-19T23:24:03.767 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.767 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.767 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.767 DEBUG:boto:Params: {} 2024-04-19T23:24:03.767 DEBUG:boto:Token: None 2024-04-19T23:24:03.767 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.767 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006501e69bab8feb4c-006622fd13-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.770 DEBUG:boto:b'aktlji-12myobj1000falsemyobj2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.771 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.771 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.771 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.771 DEBUG:boto:Path: /aktlji-12/myobj 2024-04-19T23:24:03.771 DEBUG:boto:Data: 2024-04-19T23:24:03.771 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.771 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.771 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.771 DEBUG:boto:Params: {} 2024-04-19T23:24:03.772 DEBUG:boto:Token: None 2024-04-19T23:24:03.772 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj 2024-04-19T23:24:03.772 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OeV644+IRrwcZkL6Ph6AwZEcv1U= 2024-04-19T23:24:03.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OeV644+IRrwcZkL6Ph6AwZEcv1U=', 'Content-Length': '0'} 2024-04-19T23:24:03.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f36e8d09f04ae45-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.774 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.774 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.775 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.775 DEBUG:boto:Path: /aktlji-12/myobj 2024-04-19T23:24:03.775 DEBUG:boto:Data: 2024-04-19T23:24:03.775 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.775 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.775 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.775 DEBUG:boto:Params: {} 2024-04-19T23:24:03.777 DEBUG:boto:Token: None 2024-04-19T23:24:03.777 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj 2024-04-19T23:24:03.777 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OeV644+IRrwcZkL6Ph6AwZEcv1U= 2024-04-19T23:24:03.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OeV644+IRrwcZkL6Ph6AwZEcv1U=', 'Content-Length': '0'} 2024-04-19T23:24:03.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db2464c1781795b4-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:03.780 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.780 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.780 DEBUG:boto:Method: GET 2024-04-19T23:24:03.780 DEBUG:boto:Path: /aktlji-12/myobj 2024-04-19T23:24:03.780 DEBUG:boto:Data: 2024-04-19T23:24:03.780 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.780 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.780 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.780 DEBUG:boto:Params: {} 2024-04-19T23:24:03.781 DEBUG:boto:Token: None 2024-04-19T23:24:03.781 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj 2024-04-19T23:24:03.781 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PevKsrnZqMQSsWXWgzvQ5OyejDQ= 2024-04-19T23:24:03.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PevKsrnZqMQSsWXWgzvQ5OyejDQ=', 'Content-Length': '0'} 2024-04-19T23:24:03.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094896c72c23a9625-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.783 DEBUG:boto:path=/aktlji-12/myobj 2024-04-19T23:24:03.783 DEBUG:boto:auth_path=/aktlji-12/myobj 2024-04-19T23:24:03.783 DEBUG:boto:Method: GET 2024-04-19T23:24:03.783 DEBUG:boto:Path: /aktlji-12/myobj 2024-04-19T23:24:03.783 DEBUG:boto:Data: 2024-04-19T23:24:03.783 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.784 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.784 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.784 DEBUG:boto:Params: {} 2024-04-19T23:24:03.784 DEBUG:boto:Token: None 2024-04-19T23:24:03.784 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/myobj 2024-04-19T23:24:03.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PevKsrnZqMQSsWXWgzvQ5OyejDQ= 2024-04-19T23:24:03.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PevKsrnZqMQSsWXWgzvQ5OyejDQ=', 'Content-Length': '0'} 2024-04-19T23:24:03.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:23:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000591f8db4e8a3f755-006622fd13-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.830 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.830 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.830 DEBUG:boto:path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.831 DEBUG:boto:auth_path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.831 DEBUG:boto:Method: GET 2024-04-19T23:24:03.831 DEBUG:boto:Path: /aktlji-12/?prefix=myobj 2024-04-19T23:24:03.831 DEBUG:boto:Data: 2024-04-19T23:24:03.831 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.831 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.831 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.831 DEBUG:boto:Params: {} 2024-04-19T23:24:03.831 DEBUG:boto:Token: None 2024-04-19T23:24:03.831 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.832 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d08a809727f2754-006622fd13-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.834 DEBUG:boto:b'aktlji-12myobj1000falsemyobj2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.835 DEBUG:boto:path=/aktlji-12/ 2024-04-19T23:24:03.835 DEBUG:boto:auth_path=/aktlji-12/ 2024-04-19T23:24:03.835 DEBUG:boto:path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.835 DEBUG:boto:auth_path=/aktlji-12/?prefix=myobj 2024-04-19T23:24:03.835 DEBUG:boto:Method: GET 2024-04-19T23:24:03.835 DEBUG:boto:Path: /aktlji-12/?prefix=myobj 2024-04-19T23:24:03.835 DEBUG:boto:Data: 2024-04-19T23:24:03.835 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.836 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.836 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.836 DEBUG:boto:Params: {} 2024-04-19T23:24:03.836 DEBUG:boto:Token: None 2024-04-19T23:24:03.836 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:03 GMT /aktlji-12/ 2024-04-19T23:24:03.836 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4= 2024-04-19T23:24:03.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Erx9kzjxrIV0WusO2aZ5BtONld4=', 'Content-Length': '0'} 2024-04-19T23:24:03.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9236df325853493-006622fd13-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.838 DEBUG:boto:b'aktlji-12myobj1000falsemyobj2024-04-19T23:23:14.256Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:03.839 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-12 zones={test-zone2, test-zone3} 2024-04-19T23:24:03.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2024-04-19T23:24:03.840 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-13 2024-04-19T23:24:03.841 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:03.841 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:03.841 DEBUG:boto:Method: PUT 2024-04-19T23:24:03.841 DEBUG:boto:Path: /aktlji-13/ 2024-04-19T23:24:03.841 DEBUG:boto:Data: 2024-04-19T23:24:03.841 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.841 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:03.841 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.841 DEBUG:boto:Params: {} 2024-04-19T23:24:03.841 DEBUG:boto:Token: None 2024-04-19T23:24:03.841 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:24:03 GMT /aktlji-13/ 2024-04-19T23:24:03.842 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OaoevxVLlZTKu80wxYHI9PiOdE0= 2024-04-19T23:24:03.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OaoevxVLlZTKu80wxYHI9PiOdE0=', 'Content-Length': '0'} 2024-04-19T23:24:03.858 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f16288dfb289b8d6-006622fd13-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.858 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-14 2024-04-19T23:24:03.858 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:03.858 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:03.859 DEBUG:boto:Method: PUT 2024-04-19T23:24:03.859 DEBUG:boto:Path: /aktlji-14/ 2024-04-19T23:24:03.859 DEBUG:boto:Data: 2024-04-19T23:24:03.859 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.859 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.859 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.859 DEBUG:boto:Params: {} 2024-04-19T23:24:03.859 DEBUG:boto:Token: None 2024-04-19T23:24:03.859 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:24:03 GMT /aktlji-14/ 2024-04-19T23:24:03.860 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:23GOfzyt+pxPqHrpELjPEd3nr4A= 2024-04-19T23:24:03.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:23GOfzyt+pxPqHrpELjPEd3nr4A=', 'Content-Length': '0'} 2024-04-19T23:24:03.887 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007b190bd2664be8db-006622fd13-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.887 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-15 2024-04-19T23:24:03.887 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:03.887 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:03.887 DEBUG:boto:Method: PUT 2024-04-19T23:24:03.887 DEBUG:boto:Path: /aktlji-15/ 2024-04-19T23:24:03.887 DEBUG:boto:Data: 2024-04-19T23:24:03.888 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.888 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.888 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.888 DEBUG:boto:Params: {} 2024-04-19T23:24:03.888 DEBUG:boto:Token: None 2024-04-19T23:24:03.888 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:24:03 GMT /aktlji-15/ 2024-04-19T23:24:03.888 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z7stTxBTUq8kCRpPz41cSJMrhmE= 2024-04-19T23:24:03.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z7stTxBTUq8kCRpPz41cSJMrhmE=', 'Content-Length': '0'} 2024-04-19T23:24:03.916 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000047824e6ed9ce8762-006622fd13-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.916 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:03.916 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:03.916 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.916 DEBUG:boto:Path: /aktlji-13/ 2024-04-19T23:24:03.916 DEBUG:boto:Data: 2024-04-19T23:24:03.916 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.917 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:03.917 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.917 DEBUG:boto:Params: {} 2024-04-19T23:24:03.917 DEBUG:boto:Token: None 2024-04-19T23:24:03.917 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-13/ 2024-04-19T23:24:03.917 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fUngcO7VS9wguqawMB6ox0eVBbg= 2024-04-19T23:24:03.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fUngcO7VS9wguqawMB6ox0eVBbg=', 'Content-Length': '0'} 2024-04-19T23:24:03.920 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2ccbea674c21ad0-006622fd13-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.921 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:03.921 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:03.921 DEBUG:boto:Method: PUT 2024-04-19T23:24:03.921 DEBUG:boto:Path: /aktlji-13/myobj 2024-04-19T23:24:03.921 DEBUG:boto:Data: 2024-04-19T23:24:03.921 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:24:03.921 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:03.921 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.921 DEBUG:boto:Params: {} 2024-04-19T23:24:03.921 DEBUG:boto:Token: None 2024-04-19T23:24:03.922 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:24:03 GMT /aktlji-13/myobj 2024-04-19T23:24:03.922 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:J91f0eG/Ut1WRtrCP4y6acy6QH8= 2024-04-19T23:24:03.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:J91f0eG/Ut1WRtrCP4y6acy6QH8='} 2024-04-19T23:24:03.929 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000674a29bd4ecd79d2-006622fd13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.929 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:03.929 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:03.929 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.929 DEBUG:boto:Path: /aktlji-14/ 2024-04-19T23:24:03.929 DEBUG:boto:Data: 2024-04-19T23:24:03.929 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.929 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.929 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.929 DEBUG:boto:Params: {} 2024-04-19T23:24:03.930 DEBUG:boto:Token: None 2024-04-19T23:24:03.930 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-14/ 2024-04-19T23:24:03.930 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mzFf3ZNfZx6ZhimQay++nKgrCt8= 2024-04-19T23:24:03.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mzFf3ZNfZx6ZhimQay++nKgrCt8=', 'Content-Length': '0'} 2024-04-19T23:24:03.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098768d3a3c8c2a57-006622fd13-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.933 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:03.933 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:03.933 DEBUG:boto:Method: PUT 2024-04-19T23:24:03.933 DEBUG:boto:Path: /aktlji-14/myobj 2024-04-19T23:24:03.933 DEBUG:boto:Data: 2024-04-19T23:24:03.933 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:24:03.933 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:03.933 DEBUG:boto:Port: 8000 2024-04-19T23:24:03.933 DEBUG:boto:Params: {} 2024-04-19T23:24:03.933 DEBUG:boto:Token: None 2024-04-19T23:24:03.934 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:24:03 GMT /aktlji-14/myobj 2024-04-19T23:24:03.934 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KFeZBsLn1pDfpYyZj78q5LRzO3o= 2024-04-19T23:24:03.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KFeZBsLn1pDfpYyZj78q5LRzO3o='} 2024-04-19T23:24:03.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e001c763a9b4b5d-006622fd13-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.986 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:03.986 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:03.986 DEBUG:boto:Method: HEAD 2024-04-19T23:24:03.986 DEBUG:boto:Path: /aktlji-15/ 2024-04-19T23:24:03.987 DEBUG:boto:Data: 2024-04-19T23:24:03.987 DEBUG:boto:Headers: {} 2024-04-19T23:24:03.987 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.987 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.987 DEBUG:boto:Params: {} 2024-04-19T23:24:03.987 DEBUG:boto:Token: None 2024-04-19T23:24:03.987 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:03 GMT /aktlji-15/ 2024-04-19T23:24:03.987 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:e515yx7qCQ3o5IsQE9CA9URbeOw= 2024-04-19T23:24:03.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:e515yx7qCQ3o5IsQE9CA9URbeOw=', 'Content-Length': '0'} 2024-04-19T23:24:03.990 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbc831fed977523a-006622fd13-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.990 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:03.990 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:03.991 DEBUG:boto:Method: PUT 2024-04-19T23:24:03.991 DEBUG:boto:Path: /aktlji-15/myobj 2024-04-19T23:24:03.991 DEBUG:boto:Data: 2024-04-19T23:24:03.991 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:24:03.991 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:03.991 DEBUG:boto:Port: 8001 2024-04-19T23:24:03.991 DEBUG:boto:Params: {} 2024-04-19T23:24:03.991 DEBUG:boto:Token: None 2024-04-19T23:24:03.991 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:24:03 GMT /aktlji-15/myobj 2024-04-19T23:24:03.992 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XC3+4TSUaO6GPP3uLYPTBBRAeu8= 2024-04-19T23:24:03.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:24:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XC3+4TSUaO6GPP3uLYPTBBRAeu8='} 2024-04-19T23:24:03.999 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005825faecea633085-006622fd13-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:24:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:03.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:24:03.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:24:03.999 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:24:04.065 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:04.065 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:04.105 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:24:04.119 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:24:04.119 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:24:04.119 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:24:04.119 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:24:04.119 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:24:04.119 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:24:04.119 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:24:04.120 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:24:04.120 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:24:04.120 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:24:04.120 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713569043.909356_52.1", 2024-04-19T23:24:04.120 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:24:03.909356Z" 2024-04-19T23:24:04.120 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:24:04.120 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:24:04.120 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:24:04.120 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:24:04.120 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:24:04.120 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:24:04.121 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569043.909356_52.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:24:04.122 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:24:04.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:24:04.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:24:04.122 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:24:04.180 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:04.181 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:04.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:04.221+0000 7f9a62e24780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:24:04.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:04.222+0000 7f9a62e24780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:24:04.232 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:24:04.233 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:24:04.233 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:24:04.233 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:24:04.233 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:24:04.233 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:24:04.233 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:24:04.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:24:04.233 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:24:04.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:24:04.234 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:24:04.234 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:24:04.234 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:24:04.234 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:24:04.234 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:24:04.234 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:24:04.234 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:24:04.234 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:24:04.234 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:24:04.234 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:24:04.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:24:04.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:24:04.235 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:24:04.235 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:24:04.235 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:24:04.235 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:24:04.235 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:24:04.235 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:24:04.235 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:24:04.235 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:24:04.235 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:24:04.236 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:24:04.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:24:04.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:24:04.236 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:24:04.236 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:24:04.236 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:24:04.236 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713569043.872890_45.1", 2024-04-19T23:24:04.236 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:24:04.236 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:24:04.236 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:24:04.237 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:24:03.872890Z", 2024-04-19T23:24:04.237 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:24:04.237 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:24:04.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:24:04.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:24:04.237 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:24:04.237 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:24:04.237 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:24:04.237 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:24:04.237 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:24:04.237 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:24:04.238 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:24:04.238 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:24:04.238 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:24:04.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:24:04.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:24:04.238 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:24:04.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:24:04.238 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:24:04.238 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:24:04.239 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:24:04.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:24:04.239 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:24:04Z" 2024-04-19T23:24:04.239 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:24:04.240 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569043.872890_45.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:24:03.872890Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:24:04Z" } 2024-04-19T23:24:04.240 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569043.872890_45.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:24:03.872890Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:24:04.240 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569043.909356_52.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:24:04.240 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569043.872890_45.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:24:04.240 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713569043.909356_52.1 target=1_1713569043.872890_45.1 2024-04-19T23:24:09.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:24:09.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:24:09.242 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:24:09.302 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:09.302 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:09.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:09.341+0000 7f0a28e87780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:24:09.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:09.342+0000 7f0a28e87780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:24:09.352 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:24:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:24:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:24:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:24:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:24:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:24:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:24:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:24:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:24:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:24:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:24:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:24:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:24:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:24:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:24:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:24:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:24:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:24:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:24:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:24:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:24:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:24:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:24:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:24:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:24:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:24:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:24:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:24:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:24:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:24:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:24:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:24:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:24:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:24:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:24:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:24:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:24:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713569043.909356_52.1", 2024-04-19T23:24:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:24:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:24:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:24:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:24:03.909356Z", 2024-04-19T23:24:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:24:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:24:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:24:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:24:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:24:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:24:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:24:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:24:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:24:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:24:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:24:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:24:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:24:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:24:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:24:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:24:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:24:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:24:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:24:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:24:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:24:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:24:09Z" 2024-04-19T23:24:09.359 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:24:09.360 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569043.909356_52.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:24:03.909356Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:24:09Z" } 2024-04-19T23:24:09.360 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569043.909356_52.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:24:03.909356Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:24:09.360 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569043.909356_52.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:24:09.361 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569043.909356_52.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:24:09.361 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:24:09.361 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:24:09.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:24:09.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:24:09.361 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:24:09.421 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:09.421 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:09.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:09.462+0000 7fd01a36c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:24:09.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:09.464+0000 7fd01a36c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:24:09.472 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:24:09.472 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:24:09.472 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:24:09.473 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:24:09.473 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:24:09.473 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:24:09.473 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:24:09.473 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:24:09.473 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:24:09.473 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:24:09.473 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:24:09.473 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:24:09.473 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:24:09.473 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:24:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:24:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:24:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:24:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:24:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:24:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:24:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:24:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:24:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:24:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:24:09.474 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:24:09.475 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:24:09.475 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:24:09.475 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:24:09.475 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:24:09.475 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:24:09.475 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:24:09.475 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:24:09.475 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:24:09.475 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:24:09.475 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:24:09.475 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:24:09.476 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:24:09.476 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713569043.909356_52.1", 2024-04-19T23:24:09.476 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:24:09.476 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:24:09.476 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:24:09.476 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:24:03.909356Z", 2024-04-19T23:24:09.476 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:24:09.476 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:24:09.476 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:24:09.476 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:24:09.477 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:24:09.477 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:24:09.477 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:24:09.477 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:24:09.477 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:24:09.477 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:24:09.477 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:24:09.477 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:24:09.477 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:24:09.477 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:24:09.477 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:24:09.478 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:24:09.478 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:24:09.478 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:24:09.478 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:24:09.478 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:24:09.478 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:24:09.478 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:24:09Z" 2024-04-19T23:24:09.478 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:24:09.479 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569043.909356_52.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:24:03.909356Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:24:09Z" } 2024-04-19T23:24:09.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569043.909356_52.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:24:03.909356Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:24:09.480 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569043.909356_52.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:24:09.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569043.909356_52.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:24:09.480 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:24:09.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:09.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:09.480 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:24:09.547 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:09.548 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:09.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:09.592+0000 7f9811008780 1 waiting to reach incremental sync.. 2024-04-19T23:24:14.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:14.593+0000 7f9811008780 1 bucket sync caught up with source: 2024-04-19T23:24:14.590 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:24:14.590 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:24:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:14.593+0000 7f9811008780 0 bucket checkpoint complete 2024-04-19T23:24:14.607 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-13 zones={test-zone2, test-zone1} 2024-04-19T23:24:14.607 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:14.607 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:14.607 DEBUG:boto:Method: HEAD 2024-04-19T23:24:14.607 DEBUG:boto:Path: /aktlji-13/ 2024-04-19T23:24:14.607 DEBUG:boto:Data: 2024-04-19T23:24:14.607 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.607 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:14.608 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.608 DEBUG:boto:Params: {} 2024-04-19T23:24:14.608 DEBUG:boto:Token: None 2024-04-19T23:24:14.608 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/ 2024-04-19T23:24:14.608 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bHNwT4jNs22NgWP9XPKKm9FkDB8= 2024-04-19T23:24:14.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bHNwT4jNs22NgWP9XPKKm9FkDB8=', 'Content-Length': '0'} 2024-04-19T23:24:14.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000743e6fc68d40a25b-006622fd1e-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.611 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:14.611 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:14.611 DEBUG:boto:Method: HEAD 2024-04-19T23:24:14.611 DEBUG:boto:Path: /aktlji-13/ 2024-04-19T23:24:14.612 DEBUG:boto:Data: 2024-04-19T23:24:14.612 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.612 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:14.612 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.612 DEBUG:boto:Params: {} 2024-04-19T23:24:14.612 DEBUG:boto:Token: None 2024-04-19T23:24:14.612 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/ 2024-04-19T23:24:14.612 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bHNwT4jNs22NgWP9XPKKm9FkDB8= 2024-04-19T23:24:14.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bHNwT4jNs22NgWP9XPKKm9FkDB8=', 'Content-Length': '0'} 2024-04-19T23:24:14.615 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005146b87caeda6373-006622fd1e-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.615 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:24:14.615 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:14.615 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:14.615 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:14.615 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:14.615 DEBUG:boto:Method: GET 2024-04-19T23:24:14.616 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:24:14.616 DEBUG:boto:Data: 2024-04-19T23:24:14.616 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.616 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:14.616 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.616 DEBUG:boto:Params: {} 2024-04-19T23:24:14.616 DEBUG:boto:Token: None 2024-04-19T23:24:14.616 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/?versions 2024-04-19T23:24:14.616 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ONNXihGzMoyqjqg92+LuAq+LHPE= 2024-04-19T23:24:14.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ONNXihGzMoyqjqg92+LuAq+LHPE=', 'Content-Length': '0'} 2024-04-19T23:24:14.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c72238ea130d32a-006622fd1e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.619 DEBUG:boto:b'aktlji-13999falsemyobjnulltrue2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:14.620 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:14.620 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:24:14.621 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:14.621 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:14.621 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:14.621 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:14.621 DEBUG:boto:Method: GET 2024-04-19T23:24:14.621 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:24:14.621 DEBUG:boto:Data: 2024-04-19T23:24:14.621 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.621 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:14.621 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.621 DEBUG:boto:Params: {} 2024-04-19T23:24:14.621 DEBUG:boto:Token: None 2024-04-19T23:24:14.622 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/?versions 2024-04-19T23:24:14.622 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ONNXihGzMoyqjqg92+LuAq+LHPE= 2024-04-19T23:24:14.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ONNXihGzMoyqjqg92+LuAq+LHPE=', 'Content-Length': '0'} 2024-04-19T23:24:14.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1d0f4013346c50d-006622fd1e-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.670 DEBUG:boto:b'aktlji-13999falsemyobjnulltrue2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:14.671 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:14.671 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:14.671 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:14.671 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:14.672 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:14.672 DEBUG:boto:Method: GET 2024-04-19T23:24:14.672 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:24:14.672 DEBUG:boto:Data: 2024-04-19T23:24:14.672 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.672 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:14.672 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.672 DEBUG:boto:Params: {} 2024-04-19T23:24:14.673 DEBUG:boto:Token: None 2024-04-19T23:24:14.673 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/?versions 2024-04-19T23:24:14.673 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ONNXihGzMoyqjqg92+LuAq+LHPE= 2024-04-19T23:24:14.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ONNXihGzMoyqjqg92+LuAq+LHPE=', 'Content-Length': '0'} 2024-04-19T23:24:14.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025e4a67af549bab8-006622fd1e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.676 DEBUG:boto:b'aktlji-13999falsemyobjnulltrue2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:14.676 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:14.676 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:14.676 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:14.676 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:14.677 DEBUG:boto:Method: GET 2024-04-19T23:24:14.677 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:24:14.677 DEBUG:boto:Data: 2024-04-19T23:24:14.677 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.677 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:14.677 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.677 DEBUG:boto:Params: {} 2024-04-19T23:24:14.677 DEBUG:boto:Token: None 2024-04-19T23:24:14.677 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/?versions 2024-04-19T23:24:14.677 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ONNXihGzMoyqjqg92+LuAq+LHPE= 2024-04-19T23:24:14.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ONNXihGzMoyqjqg92+LuAq+LHPE=', 'Content-Length': '0'} 2024-04-19T23:24:14.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1c0e3259601a373-006622fd1e-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.680 DEBUG:boto:b'aktlji-13999falsemyobjnulltrue2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:14.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:14.680 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:14.680 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:14.681 DEBUG:boto:path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.681 DEBUG:boto:auth_path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.681 DEBUG:boto:Method: GET 2024-04-19T23:24:14.681 DEBUG:boto:Path: /aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.681 DEBUG:boto:Data: 2024-04-19T23:24:14.681 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.681 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:14.682 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.682 DEBUG:boto:Params: {} 2024-04-19T23:24:14.682 DEBUG:boto:Token: None 2024-04-19T23:24:14.682 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.682 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BD1zntmwiPPvo9R9pPaXfDna+Ak= 2024-04-19T23:24:14.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BD1zntmwiPPvo9R9pPaXfDna+Ak=', 'Content-Length': '0'} 2024-04-19T23:24:14.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ff7994551033dce-006622fd1e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.685 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:14.685 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:14.685 DEBUG:boto:path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.685 DEBUG:boto:auth_path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.685 DEBUG:boto:Method: GET 2024-04-19T23:24:14.685 DEBUG:boto:Path: /aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.685 DEBUG:boto:Data: 2024-04-19T23:24:14.685 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.685 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:14.685 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.685 DEBUG:boto:Params: {} 2024-04-19T23:24:14.686 DEBUG:boto:Token: None 2024-04-19T23:24:14.686 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.686 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BD1zntmwiPPvo9R9pPaXfDna+Ak= 2024-04-19T23:24:14.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BD1zntmwiPPvo9R9pPaXfDna+Ak=', 'Content-Length': '0'} 2024-04-19T23:24:14.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000690ea8e35eaaf044-006622fd1e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.688 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:14.688 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:14.688 DEBUG:boto:path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.689 DEBUG:boto:auth_path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.689 DEBUG:boto:Method: HEAD 2024-04-19T23:24:14.689 DEBUG:boto:Path: /aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.689 DEBUG:boto:Data: 2024-04-19T23:24:14.689 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.689 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:14.689 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.689 DEBUG:boto:Params: {} 2024-04-19T23:24:14.689 DEBUG:boto:Token: None 2024-04-19T23:24:14.689 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.690 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VA5WknTWm+/xzWZ52bT+WvJMGYM= 2024-04-19T23:24:14.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VA5WknTWm+/xzWZ52bT+WvJMGYM=', 'Content-Length': '0'} 2024-04-19T23:24:14.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c8521eb5ed7af7a-006622fd1e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.692 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:14.692 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:14.692 DEBUG:boto:path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.692 DEBUG:boto:auth_path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.692 DEBUG:boto:Method: HEAD 2024-04-19T23:24:14.693 DEBUG:boto:Path: /aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.693 DEBUG:boto:Data: 2024-04-19T23:24:14.693 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.693 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:14.693 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.693 DEBUG:boto:Params: {} 2024-04-19T23:24:14.693 DEBUG:boto:Token: None 2024-04-19T23:24:14.693 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.693 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VA5WknTWm+/xzWZ52bT+WvJMGYM= 2024-04-19T23:24:14.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VA5WknTWm+/xzWZ52bT+WvJMGYM=', 'Content-Length': '0'} 2024-04-19T23:24:14.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050431c1b8ac98b5f-006622fd1e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:14.696 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:14.696 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:14.696 DEBUG:boto:path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.696 DEBUG:boto:auth_path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.696 DEBUG:boto:Method: GET 2024-04-19T23:24:14.696 DEBUG:boto:Path: /aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.696 DEBUG:boto:Data: 2024-04-19T23:24:14.696 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.696 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:14.696 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.696 DEBUG:boto:Params: {} 2024-04-19T23:24:14.697 DEBUG:boto:Token: None 2024-04-19T23:24:14.697 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.697 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BD1zntmwiPPvo9R9pPaXfDna+Ak= 2024-04-19T23:24:14.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BD1zntmwiPPvo9R9pPaXfDna+Ak=', 'Content-Length': '0'} 2024-04-19T23:24:14.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000980fbb5b016e20d2-006622fd1e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.699 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:14.699 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:14.699 DEBUG:boto:path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.699 DEBUG:boto:auth_path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.699 DEBUG:boto:Method: GET 2024-04-19T23:24:14.699 DEBUG:boto:Path: /aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.700 DEBUG:boto:Data: 2024-04-19T23:24:14.700 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.700 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:14.700 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.700 DEBUG:boto:Params: {} 2024-04-19T23:24:14.700 DEBUG:boto:Token: None 2024-04-19T23:24:14.700 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/myobj?versionId=null 2024-04-19T23:24:14.700 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BD1zntmwiPPvo9R9pPaXfDna+Ak= 2024-04-19T23:24:14.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BD1zntmwiPPvo9R9pPaXfDna+Ak=', 'Content-Length': '0'} 2024-04-19T23:24:14.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044790c28bc378a09-006622fd1e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.746 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:14.746 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:14.747 DEBUG:boto:path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:14.747 DEBUG:boto:auth_path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:14.747 DEBUG:boto:Method: GET 2024-04-19T23:24:14.747 DEBUG:boto:Path: /aktlji-13/?prefix=myobj 2024-04-19T23:24:14.747 DEBUG:boto:Data: 2024-04-19T23:24:14.747 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.747 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:14.747 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.747 DEBUG:boto:Params: {} 2024-04-19T23:24:14.747 DEBUG:boto:Token: None 2024-04-19T23:24:14.748 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/ 2024-04-19T23:24:14.748 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yk1S7lm+lV4nr7TaDsDu9Gq/q0c= 2024-04-19T23:24:14.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yk1S7lm+lV4nr7TaDsDu9Gq/q0c=', 'Content-Length': '0'} 2024-04-19T23:24:14.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3cc7a75fbc87d50-006622fd1e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.751 DEBUG:boto:b'aktlji-13myobj1000falsemyobj2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:14.751 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:14.751 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:14.751 DEBUG:boto:path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:14.751 DEBUG:boto:auth_path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:14.752 DEBUG:boto:Method: GET 2024-04-19T23:24:14.752 DEBUG:boto:Path: /aktlji-13/?prefix=myobj 2024-04-19T23:24:14.752 DEBUG:boto:Data: 2024-04-19T23:24:14.752 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.752 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:14.752 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.752 DEBUG:boto:Params: {} 2024-04-19T23:24:14.752 DEBUG:boto:Token: None 2024-04-19T23:24:14.752 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/ 2024-04-19T23:24:14.753 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yk1S7lm+lV4nr7TaDsDu9Gq/q0c= 2024-04-19T23:24:14.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yk1S7lm+lV4nr7TaDsDu9Gq/q0c=', 'Content-Length': '0'} 2024-04-19T23:24:14.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000786d4f74bf439a97-006622fd1e-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.756 DEBUG:boto:b'aktlji-13myobj1000falsemyobj2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:14.756 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:14.756 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:14.756 DEBUG:boto:Method: HEAD 2024-04-19T23:24:14.756 DEBUG:boto:Path: /aktlji-13/myobj 2024-04-19T23:24:14.756 DEBUG:boto:Data: 2024-04-19T23:24:14.756 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.757 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:14.757 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.757 DEBUG:boto:Params: {} 2024-04-19T23:24:14.757 DEBUG:boto:Token: None 2024-04-19T23:24:14.757 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/myobj 2024-04-19T23:24:14.757 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l7HuIQCj2IpyQ/2nPZtgh09k5SI= 2024-04-19T23:24:14.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l7HuIQCj2IpyQ/2nPZtgh09k5SI=', 'Content-Length': '0'} 2024-04-19T23:24:14.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000182128e4718319f3-006622fd1e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.759 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:14.759 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:14.759 DEBUG:boto:Method: HEAD 2024-04-19T23:24:14.759 DEBUG:boto:Path: /aktlji-13/myobj 2024-04-19T23:24:14.760 DEBUG:boto:Data: 2024-04-19T23:24:14.760 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.760 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:14.760 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.760 DEBUG:boto:Params: {} 2024-04-19T23:24:14.760 DEBUG:boto:Token: None 2024-04-19T23:24:14.760 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/myobj 2024-04-19T23:24:14.760 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l7HuIQCj2IpyQ/2nPZtgh09k5SI= 2024-04-19T23:24:14.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l7HuIQCj2IpyQ/2nPZtgh09k5SI=', 'Content-Length': '0'} 2024-04-19T23:24:14.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000837efcd87e1a3281-006622fd1e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:14.762 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:14.763 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:14.763 DEBUG:boto:Method: GET 2024-04-19T23:24:14.763 DEBUG:boto:Path: /aktlji-13/myobj 2024-04-19T23:24:14.763 DEBUG:boto:Data: 2024-04-19T23:24:14.763 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.763 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:14.763 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.763 DEBUG:boto:Params: {} 2024-04-19T23:24:14.763 DEBUG:boto:Token: None 2024-04-19T23:24:14.764 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/myobj 2024-04-19T23:24:14.764 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fkWM94sG0LqcjYanjtHNySmi8C0= 2024-04-19T23:24:14.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fkWM94sG0LqcjYanjtHNySmi8C0=', 'Content-Length': '0'} 2024-04-19T23:24:14.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003752383b108b2e18-006622fd1e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.806 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:14.806 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:14.807 DEBUG:boto:Method: GET 2024-04-19T23:24:14.807 DEBUG:boto:Path: /aktlji-13/myobj 2024-04-19T23:24:14.807 DEBUG:boto:Data: 2024-04-19T23:24:14.807 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.807 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:14.807 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.807 DEBUG:boto:Params: {} 2024-04-19T23:24:14.807 DEBUG:boto:Token: None 2024-04-19T23:24:14.807 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/myobj 2024-04-19T23:24:14.807 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fkWM94sG0LqcjYanjtHNySmi8C0= 2024-04-19T23:24:14.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fkWM94sG0LqcjYanjtHNySmi8C0=', 'Content-Length': '0'} 2024-04-19T23:24:14.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b7f495a55564a94-006622fd1e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.809 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:14.810 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:14.810 DEBUG:boto:path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:14.810 DEBUG:boto:auth_path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:14.810 DEBUG:boto:Method: GET 2024-04-19T23:24:14.811 DEBUG:boto:Path: /aktlji-13/?prefix=myobj 2024-04-19T23:24:14.811 DEBUG:boto:Data: 2024-04-19T23:24:14.811 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.811 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:14.811 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.811 DEBUG:boto:Params: {} 2024-04-19T23:24:14.811 DEBUG:boto:Token: None 2024-04-19T23:24:14.812 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/ 2024-04-19T23:24:14.812 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yk1S7lm+lV4nr7TaDsDu9Gq/q0c= 2024-04-19T23:24:14.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yk1S7lm+lV4nr7TaDsDu9Gq/q0c=', 'Content-Length': '0'} 2024-04-19T23:24:14.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000744b24d99195b2e6-006622fd1e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.815 DEBUG:boto:b'aktlji-13myobj1000falsemyobj2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:14.815 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:14.815 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:14.816 DEBUG:boto:path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:14.816 DEBUG:boto:auth_path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:14.816 DEBUG:boto:Method: GET 2024-04-19T23:24:14.816 DEBUG:boto:Path: /aktlji-13/?prefix=myobj 2024-04-19T23:24:14.816 DEBUG:boto:Data: 2024-04-19T23:24:14.816 DEBUG:boto:Headers: {} 2024-04-19T23:24:14.816 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:14.816 DEBUG:boto:Port: 8000 2024-04-19T23:24:14.816 DEBUG:boto:Params: {} 2024-04-19T23:24:14.816 DEBUG:boto:Token: None 2024-04-19T23:24:14.817 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:14 GMT /aktlji-13/ 2024-04-19T23:24:14.817 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yk1S7lm+lV4nr7TaDsDu9Gq/q0c= 2024-04-19T23:24:14.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yk1S7lm+lV4nr7TaDsDu9Gq/q0c=', 'Content-Length': '0'} 2024-04-19T23:24:14.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000143425bf86643e3a-006622fd1e-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:14.819 DEBUG:boto:b'aktlji-13myobj1000falsemyobj2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:14.820 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-13 zones={test-zone2, test-zone1} 2024-04-19T23:24:14.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:14.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:14.820 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:24:14.887 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:14.887 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:14.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:14.933+0000 7f07574c5780 1 waiting to reach incremental sync.. 2024-04-19T23:24:19.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:19.933+0000 7f07574c5780 1 bucket sync caught up with source: 2024-04-19T23:24:19.929 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:24:19.929 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:24:19.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:19.933+0000 7f07574c5780 0 bucket checkpoint complete 2024-04-19T23:24:19.946 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-13 zones={test-zone3, test-zone1} 2024-04-19T23:24:19.946 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:19.946 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:19.947 DEBUG:boto:Method: HEAD 2024-04-19T23:24:19.947 DEBUG:boto:Path: /aktlji-13/ 2024-04-19T23:24:19.947 DEBUG:boto:Data: 2024-04-19T23:24:19.947 DEBUG:boto:Headers: {} 2024-04-19T23:24:19.947 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:19.947 DEBUG:boto:Port: 8001 2024-04-19T23:24:19.947 DEBUG:boto:Params: {} 2024-04-19T23:24:19.947 DEBUG:boto:Token: None 2024-04-19T23:24:19.947 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:19 GMT /aktlji-13/ 2024-04-19T23:24:19.948 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N/qyaGFz9kHgQj2verdNbH+FbQQ= 2024-04-19T23:24:19.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N/qyaGFz9kHgQj2verdNbH+FbQQ=', 'Content-Length': '0'} 2024-04-19T23:24:19.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a891166278679a11-006622fd23-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:19.950 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:19.951 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:19.951 DEBUG:boto:Method: HEAD 2024-04-19T23:24:19.951 DEBUG:boto:Path: /aktlji-13/ 2024-04-19T23:24:19.951 DEBUG:boto:Data: 2024-04-19T23:24:19.951 DEBUG:boto:Headers: {} 2024-04-19T23:24:19.951 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:19.951 DEBUG:boto:Port: 8000 2024-04-19T23:24:19.951 DEBUG:boto:Params: {} 2024-04-19T23:24:19.951 DEBUG:boto:Token: None 2024-04-19T23:24:19.951 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:19 GMT /aktlji-13/ 2024-04-19T23:24:19.952 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N/qyaGFz9kHgQj2verdNbH+FbQQ= 2024-04-19T23:24:19.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N/qyaGFz9kHgQj2verdNbH+FbQQ=', 'Content-Length': '0'} 2024-04-19T23:24:19.954 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b36621e49ee07ec1-006622fd23-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:19.954 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:24:19.954 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:19.954 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:19.954 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:19.954 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:19.955 DEBUG:boto:Method: GET 2024-04-19T23:24:19.955 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:24:19.955 DEBUG:boto:Data: 2024-04-19T23:24:19.955 DEBUG:boto:Headers: {} 2024-04-19T23:24:19.955 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:19.955 DEBUG:boto:Port: 8001 2024-04-19T23:24:19.955 DEBUG:boto:Params: {} 2024-04-19T23:24:19.955 DEBUG:boto:Token: None 2024-04-19T23:24:19.955 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:19 GMT /aktlji-13/?versions 2024-04-19T23:24:19.955 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:04ruazRckkjgV2DrqyzxEq/nCzI= 2024-04-19T23:24:19.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:04ruazRckkjgV2DrqyzxEq/nCzI=', 'Content-Length': '0'} 2024-04-19T23:24:19.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046b5b17fea5b1e20-006622fd23-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:19.958 DEBUG:boto:b'aktlji-13999falsemyobjnulltrue2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:19.958 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:19.959 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:24:19.959 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:19.959 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:19.959 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:19.959 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:19.959 DEBUG:boto:Method: GET 2024-04-19T23:24:19.959 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:24:19.960 DEBUG:boto:Data: 2024-04-19T23:24:19.960 DEBUG:boto:Headers: {} 2024-04-19T23:24:19.960 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:19.960 DEBUG:boto:Port: 8000 2024-04-19T23:24:19.960 DEBUG:boto:Params: {} 2024-04-19T23:24:19.960 DEBUG:boto:Token: None 2024-04-19T23:24:19.960 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:19 GMT /aktlji-13/?versions 2024-04-19T23:24:19.960 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:04ruazRckkjgV2DrqyzxEq/nCzI= 2024-04-19T23:24:19.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:04ruazRckkjgV2DrqyzxEq/nCzI=', 'Content-Length': '0'} 2024-04-19T23:24:19.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d50ddbebafa7bd-006622fd23-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:19.963 DEBUG:boto:b'aktlji-13999falsemyobjnulltrue2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:19.963 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:19.963 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:19.964 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:19.964 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:19.964 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:19.964 DEBUG:boto:Method: GET 2024-04-19T23:24:19.964 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:24:19.964 DEBUG:boto:Data: 2024-04-19T23:24:19.964 DEBUG:boto:Headers: {} 2024-04-19T23:24:19.964 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:19.964 DEBUG:boto:Port: 8001 2024-04-19T23:24:19.964 DEBUG:boto:Params: {} 2024-04-19T23:24:19.964 DEBUG:boto:Token: None 2024-04-19T23:24:19.965 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:19 GMT /aktlji-13/?versions 2024-04-19T23:24:19.965 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:04ruazRckkjgV2DrqyzxEq/nCzI= 2024-04-19T23:24:19.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:04ruazRckkjgV2DrqyzxEq/nCzI=', 'Content-Length': '0'} 2024-04-19T23:24:19.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056e183bd839b6cfd-006622fd23-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:19.967 DEBUG:boto:b'aktlji-13999falsemyobjnulltrue2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:19.968 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:19.968 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:19.968 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:19.968 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:19.968 DEBUG:boto:Method: GET 2024-04-19T23:24:19.968 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:24:19.968 DEBUG:boto:Data: 2024-04-19T23:24:19.969 DEBUG:boto:Headers: {} 2024-04-19T23:24:19.969 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:19.969 DEBUG:boto:Port: 8000 2024-04-19T23:24:19.969 DEBUG:boto:Params: {} 2024-04-19T23:24:19.969 DEBUG:boto:Token: None 2024-04-19T23:24:19.969 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:19 GMT /aktlji-13/?versions 2024-04-19T23:24:19.969 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:04ruazRckkjgV2DrqyzxEq/nCzI= 2024-04-19T23:24:19.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:04ruazRckkjgV2DrqyzxEq/nCzI=', 'Content-Length': '0'} 2024-04-19T23:24:19.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e172a9bed807532-006622fd23-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.018 DEBUG:boto:b'aktlji-13999falsemyobjnulltrue2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:20.019 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:20.019 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:20.019 DEBUG:boto:path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.019 DEBUG:boto:auth_path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.019 DEBUG:boto:Method: GET 2024-04-19T23:24:20.019 DEBUG:boto:Path: /aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.020 DEBUG:boto:Data: 2024-04-19T23:24:20.020 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.020 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.020 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.020 DEBUG:boto:Params: {} 2024-04-19T23:24:20.020 DEBUG:boto:Token: None 2024-04-19T23:24:20.020 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.020 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kx7DjZXkIoW/bcZB0xgh7J56w8E= 2024-04-19T23:24:20.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kx7DjZXkIoW/bcZB0xgh7J56w8E=', 'Content-Length': '0'} 2024-04-19T23:24:20.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6a4c3517b95e22e-006622fd24-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.023 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:20.023 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:20.023 DEBUG:boto:path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.023 DEBUG:boto:auth_path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.023 DEBUG:boto:Method: GET 2024-04-19T23:24:20.024 DEBUG:boto:Path: /aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.024 DEBUG:boto:Data: 2024-04-19T23:24:20.024 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.024 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.024 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.024 DEBUG:boto:Params: {} 2024-04-19T23:24:20.024 DEBUG:boto:Token: None 2024-04-19T23:24:20.024 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.025 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kx7DjZXkIoW/bcZB0xgh7J56w8E= 2024-04-19T23:24:20.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kx7DjZXkIoW/bcZB0xgh7J56w8E=', 'Content-Length': '0'} 2024-04-19T23:24:20.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfedc027e41179be-006622fd24-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.027 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:20.027 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:20.027 DEBUG:boto:path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.027 DEBUG:boto:auth_path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.027 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.027 DEBUG:boto:Path: /aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.027 DEBUG:boto:Data: 2024-04-19T23:24:20.027 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.027 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.027 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.028 DEBUG:boto:Params: {} 2024-04-19T23:24:20.028 DEBUG:boto:Token: None 2024-04-19T23:24:20.028 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.028 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qUzm/VZP4pUF9H9kOReK5pd86zU= 2024-04-19T23:24:20.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qUzm/VZP4pUF9H9kOReK5pd86zU=', 'Content-Length': '0'} 2024-04-19T23:24:20.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3aa23cbff83890a-006622fd24-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.030 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:20.030 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:20.031 DEBUG:boto:path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.031 DEBUG:boto:auth_path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.031 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.031 DEBUG:boto:Path: /aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.031 DEBUG:boto:Data: 2024-04-19T23:24:20.031 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.031 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.031 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.031 DEBUG:boto:Params: {} 2024-04-19T23:24:20.031 DEBUG:boto:Token: None 2024-04-19T23:24:20.032 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.032 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qUzm/VZP4pUF9H9kOReK5pd86zU= 2024-04-19T23:24:20.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qUzm/VZP4pUF9H9kOReK5pd86zU=', 'Content-Length': '0'} 2024-04-19T23:24:20.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b31227430d59673c-006622fd24-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:20.034 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:20.034 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:20.034 DEBUG:boto:path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.034 DEBUG:boto:auth_path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.034 DEBUG:boto:Method: GET 2024-04-19T23:24:20.034 DEBUG:boto:Path: /aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.034 DEBUG:boto:Data: 2024-04-19T23:24:20.034 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.034 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.034 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.035 DEBUG:boto:Params: {} 2024-04-19T23:24:20.035 DEBUG:boto:Token: None 2024-04-19T23:24:20.035 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.035 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kx7DjZXkIoW/bcZB0xgh7J56w8E= 2024-04-19T23:24:20.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kx7DjZXkIoW/bcZB0xgh7J56w8E=', 'Content-Length': '0'} 2024-04-19T23:24:20.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f134bff3b77ce06f-006622fd24-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.082 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:20.083 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:20.083 DEBUG:boto:path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.083 DEBUG:boto:auth_path=/aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.083 DEBUG:boto:Method: GET 2024-04-19T23:24:20.083 DEBUG:boto:Path: /aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.083 DEBUG:boto:Data: 2024-04-19T23:24:20.083 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.083 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.083 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.084 DEBUG:boto:Params: {} 2024-04-19T23:24:20.084 DEBUG:boto:Token: None 2024-04-19T23:24:20.084 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/myobj?versionId=null 2024-04-19T23:24:20.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kx7DjZXkIoW/bcZB0xgh7J56w8E= 2024-04-19T23:24:20.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kx7DjZXkIoW/bcZB0xgh7J56w8E=', 'Content-Length': '0'} 2024-04-19T23:24:20.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000101f188112876a79-006622fd24-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.086 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:20.086 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:20.086 DEBUG:boto:path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:20.086 DEBUG:boto:auth_path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:20.087 DEBUG:boto:Method: GET 2024-04-19T23:24:20.087 DEBUG:boto:Path: /aktlji-13/?prefix=myobj 2024-04-19T23:24:20.087 DEBUG:boto:Data: 2024-04-19T23:24:20.087 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.087 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.087 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.087 DEBUG:boto:Params: {} 2024-04-19T23:24:20.087 DEBUG:boto:Token: None 2024-04-19T23:24:20.087 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/ 2024-04-19T23:24:20.087 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ijCwRYCNnN1W+0w6NYjYtmxz3x4= 2024-04-19T23:24:20.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ijCwRYCNnN1W+0w6NYjYtmxz3x4=', 'Content-Length': '0'} 2024-04-19T23:24:20.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af6e4cdf273b1698-006622fd24-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.090 DEBUG:boto:b'aktlji-13myobj1000falsemyobj2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.090 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:20.091 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:20.091 DEBUG:boto:path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:20.091 DEBUG:boto:auth_path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:20.091 DEBUG:boto:Method: GET 2024-04-19T23:24:20.091 DEBUG:boto:Path: /aktlji-13/?prefix=myobj 2024-04-19T23:24:20.091 DEBUG:boto:Data: 2024-04-19T23:24:20.091 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.091 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.091 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.091 DEBUG:boto:Params: {} 2024-04-19T23:24:20.092 DEBUG:boto:Token: None 2024-04-19T23:24:20.092 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/ 2024-04-19T23:24:20.092 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ijCwRYCNnN1W+0w6NYjYtmxz3x4= 2024-04-19T23:24:20.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ijCwRYCNnN1W+0w6NYjYtmxz3x4=', 'Content-Length': '0'} 2024-04-19T23:24:20.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000154688afbe55ea59-006622fd24-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.094 DEBUG:boto:b'aktlji-13myobj1000falsemyobj2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.094 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:20.095 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:20.095 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.095 DEBUG:boto:Path: /aktlji-13/myobj 2024-04-19T23:24:20.095 DEBUG:boto:Data: 2024-04-19T23:24:20.095 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.095 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.095 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.095 DEBUG:boto:Params: {} 2024-04-19T23:24:20.095 DEBUG:boto:Token: None 2024-04-19T23:24:20.095 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/myobj 2024-04-19T23:24:20.096 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1sIhrwf5U7S/ggr7XGwW6G2/KKs= 2024-04-19T23:24:20.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1sIhrwf5U7S/ggr7XGwW6G2/KKs=', 'Content-Length': '0'} 2024-04-19T23:24:20.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9a5df3a8b31489e-006622fd24-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.098 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:20.098 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:20.098 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.098 DEBUG:boto:Path: /aktlji-13/myobj 2024-04-19T23:24:20.098 DEBUG:boto:Data: 2024-04-19T23:24:20.098 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.098 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.098 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.098 DEBUG:boto:Params: {} 2024-04-19T23:24:20.099 DEBUG:boto:Token: None 2024-04-19T23:24:20.099 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/myobj 2024-04-19T23:24:20.099 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1sIhrwf5U7S/ggr7XGwW6G2/KKs= 2024-04-19T23:24:20.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1sIhrwf5U7S/ggr7XGwW6G2/KKs=', 'Content-Length': '0'} 2024-04-19T23:24:20.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000393b123a4567f269-006622fd24-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:20.101 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:20.101 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:20.101 DEBUG:boto:Method: GET 2024-04-19T23:24:20.101 DEBUG:boto:Path: /aktlji-13/myobj 2024-04-19T23:24:20.101 DEBUG:boto:Data: 2024-04-19T23:24:20.102 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.102 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.102 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.102 DEBUG:boto:Params: {} 2024-04-19T23:24:20.102 DEBUG:boto:Token: None 2024-04-19T23:24:20.102 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/myobj 2024-04-19T23:24:20.102 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vXT5G3a4qESR9wnhTonucoAT8N0= 2024-04-19T23:24:20.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vXT5G3a4qESR9wnhTonucoAT8N0=', 'Content-Length': '0'} 2024-04-19T23:24:20.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e566cf5e6bd90a42-006622fd24-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.104 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:20.104 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:20.104 DEBUG:boto:Method: GET 2024-04-19T23:24:20.105 DEBUG:boto:Path: /aktlji-13/myobj 2024-04-19T23:24:20.105 DEBUG:boto:Data: 2024-04-19T23:24:20.105 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.105 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.105 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.105 DEBUG:boto:Params: {} 2024-04-19T23:24:20.105 DEBUG:boto:Token: None 2024-04-19T23:24:20.105 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/myobj 2024-04-19T23:24:20.105 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vXT5G3a4qESR9wnhTonucoAT8N0= 2024-04-19T23:24:20.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vXT5G3a4qESR9wnhTonucoAT8N0=', 'Content-Length': '0'} 2024-04-19T23:24:20.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085a6387430f20229-006622fd24-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.107 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:20.107 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:20.108 DEBUG:boto:path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:20.108 DEBUG:boto:auth_path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:20.108 DEBUG:boto:Method: GET 2024-04-19T23:24:20.108 DEBUG:boto:Path: /aktlji-13/?prefix=myobj 2024-04-19T23:24:20.108 DEBUG:boto:Data: 2024-04-19T23:24:20.108 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.108 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.108 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.108 DEBUG:boto:Params: {} 2024-04-19T23:24:20.108 DEBUG:boto:Token: None 2024-04-19T23:24:20.109 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/ 2024-04-19T23:24:20.109 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ijCwRYCNnN1W+0w6NYjYtmxz3x4= 2024-04-19T23:24:20.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ijCwRYCNnN1W+0w6NYjYtmxz3x4=', 'Content-Length': '0'} 2024-04-19T23:24:20.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001011284c6c951dbe-006622fd24-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.111 DEBUG:boto:b'aktlji-13myobj1000falsemyobj2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.112 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:20.112 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:20.112 DEBUG:boto:path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:20.112 DEBUG:boto:auth_path=/aktlji-13/?prefix=myobj 2024-04-19T23:24:20.112 DEBUG:boto:Method: GET 2024-04-19T23:24:20.112 DEBUG:boto:Path: /aktlji-13/?prefix=myobj 2024-04-19T23:24:20.112 DEBUG:boto:Data: 2024-04-19T23:24:20.112 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.112 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.112 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.113 DEBUG:boto:Params: {} 2024-04-19T23:24:20.113 DEBUG:boto:Token: None 2024-04-19T23:24:20.113 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-13/ 2024-04-19T23:24:20.113 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ijCwRYCNnN1W+0w6NYjYtmxz3x4= 2024-04-19T23:24:20.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ijCwRYCNnN1W+0w6NYjYtmxz3x4=', 'Content-Length': '0'} 2024-04-19T23:24:20.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a301c65907071f0-006622fd24-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.158 DEBUG:boto:b'aktlji-13myobj1000falsemyobj2024-04-19T23:24:03.929Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.159 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-13 zones={test-zone3, test-zone1} 2024-04-19T23:24:20.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:20.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:20.160 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:24:20.228 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:20.228 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:20.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:20.275+0000 7f3201a10780 1 bucket sync caught up with source: 2024-04-19T23:24:20.271 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2024-04-19T23:24:20.271 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000001.4.6, , , , , , , , , ] 2024-04-19T23:24:20.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:20.275+0000 7f3201a10780 0 bucket checkpoint complete 2024-04-19T23:24:20.285 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-14 zones={test-zone1, test-zone2} 2024-04-19T23:24:20.285 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.285 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.285 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.285 DEBUG:boto:Path: /aktlji-14/ 2024-04-19T23:24:20.285 DEBUG:boto:Data: 2024-04-19T23:24:20.286 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.286 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.286 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.286 DEBUG:boto:Params: {} 2024-04-19T23:24:20.286 DEBUG:boto:Token: None 2024-04-19T23:24:20.286 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/ 2024-04-19T23:24:20.286 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2tziK7ZYE5Uv/g+hVVu0CDLhNGI= 2024-04-19T23:24:20.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2tziK7ZYE5Uv/g+hVVu0CDLhNGI=', 'Content-Length': '0'} 2024-04-19T23:24:20.289 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051545af6acefc8c5-006622fd24-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.289 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.289 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.289 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.289 DEBUG:boto:Path: /aktlji-14/ 2024-04-19T23:24:20.289 DEBUG:boto:Data: 2024-04-19T23:24:20.289 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.290 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.290 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.290 DEBUG:boto:Params: {} 2024-04-19T23:24:20.290 DEBUG:boto:Token: None 2024-04-19T23:24:20.290 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/ 2024-04-19T23:24:20.290 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2tziK7ZYE5Uv/g+hVVu0CDLhNGI= 2024-04-19T23:24:20.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2tziK7ZYE5Uv/g+hVVu0CDLhNGI=', 'Content-Length': '0'} 2024-04-19T23:24:20.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f89bb42e0a397070-006622fd24-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.293 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:24:20.293 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.293 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.293 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.293 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.293 DEBUG:boto:Method: GET 2024-04-19T23:24:20.293 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.293 DEBUG:boto:Data: 2024-04-19T23:24:20.293 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.294 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.294 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.294 DEBUG:boto:Params: {} 2024-04-19T23:24:20.294 DEBUG:boto:Token: None 2024-04-19T23:24:20.294 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/?versions 2024-04-19T23:24:20.294 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ= 2024-04-19T23:24:20.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7ab50fba220523-006622fd24-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.297 DEBUG:boto:b'aktlji-14999falsemyobjnulltrue2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.297 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:20.297 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:24:20.298 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.298 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.298 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.298 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.298 DEBUG:boto:Method: GET 2024-04-19T23:24:20.298 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.298 DEBUG:boto:Data: 2024-04-19T23:24:20.298 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.298 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.298 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.298 DEBUG:boto:Params: {} 2024-04-19T23:24:20.299 DEBUG:boto:Token: None 2024-04-19T23:24:20.299 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/?versions 2024-04-19T23:24:20.299 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ= 2024-04-19T23:24:20.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ff644ec0275b6c5-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.302 DEBUG:boto:b'aktlji-14999falsemyobjnulltrue2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.303 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:20.303 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.303 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.303 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.303 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.303 DEBUG:boto:Method: GET 2024-04-19T23:24:20.304 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.304 DEBUG:boto:Data: 2024-04-19T23:24:20.304 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.304 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.304 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.304 DEBUG:boto:Params: {} 2024-04-19T23:24:20.304 DEBUG:boto:Token: None 2024-04-19T23:24:20.305 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/?versions 2024-04-19T23:24:20.305 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ= 2024-04-19T23:24:20.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066eff5ba8c1d293d-006622fd24-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.307 DEBUG:boto:b'aktlji-14999falsemyobjnulltrue2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.308 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.308 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.308 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.308 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.308 DEBUG:boto:Method: GET 2024-04-19T23:24:20.309 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.309 DEBUG:boto:Data: 2024-04-19T23:24:20.309 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.309 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.309 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.309 DEBUG:boto:Params: {} 2024-04-19T23:24:20.309 DEBUG:boto:Token: None 2024-04-19T23:24:20.309 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/?versions 2024-04-19T23:24:20.309 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ= 2024-04-19T23:24:20.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027bbe6b0235d6aed-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.312 DEBUG:boto:b'aktlji-14999falsemyobjnulltrue2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:20.312 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.312 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.312 DEBUG:boto:path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.313 DEBUG:boto:auth_path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.313 DEBUG:boto:Method: GET 2024-04-19T23:24:20.313 DEBUG:boto:Path: /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.313 DEBUG:boto:Data: 2024-04-19T23:24:20.313 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.313 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.313 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.313 DEBUG:boto:Params: {} 2024-04-19T23:24:20.313 DEBUG:boto:Token: None 2024-04-19T23:24:20.314 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.314 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg= 2024-04-19T23:24:20.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg=', 'Content-Length': '0'} 2024-04-19T23:24:20.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c691c017fb948e9-006622fd24-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.316 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.316 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.316 DEBUG:boto:path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.316 DEBUG:boto:auth_path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.316 DEBUG:boto:Method: GET 2024-04-19T23:24:20.316 DEBUG:boto:Path: /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.317 DEBUG:boto:Data: 2024-04-19T23:24:20.317 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.317 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.317 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.317 DEBUG:boto:Params: {} 2024-04-19T23:24:20.317 DEBUG:boto:Token: None 2024-04-19T23:24:20.317 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.317 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg= 2024-04-19T23:24:20.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg=', 'Content-Length': '0'} 2024-04-19T23:24:20.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000816affeb29e58b38-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.319 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.320 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.320 DEBUG:boto:path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.320 DEBUG:boto:auth_path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.320 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.320 DEBUG:boto:Path: /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.320 DEBUG:boto:Data: 2024-04-19T23:24:20.320 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.320 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.320 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.320 DEBUG:boto:Params: {} 2024-04-19T23:24:20.320 DEBUG:boto:Token: None 2024-04-19T23:24:20.321 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.321 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/I6qWTrH4hhssPMvuAYgZ5p7UQ0= 2024-04-19T23:24:20.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/I6qWTrH4hhssPMvuAYgZ5p7UQ0=', 'Content-Length': '0'} 2024-04-19T23:24:20.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d959782e1c5ff860-006622fd24-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.323 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.323 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.323 DEBUG:boto:path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.323 DEBUG:boto:auth_path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.323 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.323 DEBUG:boto:Path: /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.323 DEBUG:boto:Data: 2024-04-19T23:24:20.323 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.323 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.323 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.323 DEBUG:boto:Params: {} 2024-04-19T23:24:20.324 DEBUG:boto:Token: None 2024-04-19T23:24:20.324 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/I6qWTrH4hhssPMvuAYgZ5p7UQ0= 2024-04-19T23:24:20.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/I6qWTrH4hhssPMvuAYgZ5p7UQ0=', 'Content-Length': '0'} 2024-04-19T23:24:20.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea24c78a00c10e32-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:20.326 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.326 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.326 DEBUG:boto:path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.326 DEBUG:boto:auth_path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.326 DEBUG:boto:Method: GET 2024-04-19T23:24:20.326 DEBUG:boto:Path: /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.326 DEBUG:boto:Data: 2024-04-19T23:24:20.327 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.327 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.327 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.327 DEBUG:boto:Params: {} 2024-04-19T23:24:20.327 DEBUG:boto:Token: None 2024-04-19T23:24:20.327 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.327 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg= 2024-04-19T23:24:20.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg=', 'Content-Length': '0'} 2024-04-19T23:24:20.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5f8103b8a05eb8a-006622fd24-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.330 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.330 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.330 DEBUG:boto:path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.330 DEBUG:boto:auth_path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.330 DEBUG:boto:Method: GET 2024-04-19T23:24:20.330 DEBUG:boto:Path: /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.330 DEBUG:boto:Data: 2024-04-19T23:24:20.330 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.331 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.331 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.331 DEBUG:boto:Params: {} 2024-04-19T23:24:20.331 DEBUG:boto:Token: None 2024-04-19T23:24:20.331 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.331 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg= 2024-04-19T23:24:20.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg=', 'Content-Length': '0'} 2024-04-19T23:24:20.333 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e5f7681c13e3f63-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.374 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.375 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.375 DEBUG:boto:path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.375 DEBUG:boto:auth_path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.375 DEBUG:boto:Method: GET 2024-04-19T23:24:20.375 DEBUG:boto:Path: /aktlji-14/?prefix=myobj 2024-04-19T23:24:20.375 DEBUG:boto:Data: 2024-04-19T23:24:20.375 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.375 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.375 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.375 DEBUG:boto:Params: {} 2024-04-19T23:24:20.375 DEBUG:boto:Token: None 2024-04-19T23:24:20.376 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/ 2024-04-19T23:24:20.376 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ= 2024-04-19T23:24:20.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000187f9cd1e223257b-006622fd24-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.378 DEBUG:boto:b'aktlji-14myobj1000falsemyobj2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.379 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.379 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.379 DEBUG:boto:path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.379 DEBUG:boto:auth_path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.379 DEBUG:boto:Method: GET 2024-04-19T23:24:20.379 DEBUG:boto:Path: /aktlji-14/?prefix=myobj 2024-04-19T23:24:20.379 DEBUG:boto:Data: 2024-04-19T23:24:20.379 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.379 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.380 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.380 DEBUG:boto:Params: {} 2024-04-19T23:24:20.380 DEBUG:boto:Token: None 2024-04-19T23:24:20.380 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/ 2024-04-19T23:24:20.380 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ= 2024-04-19T23:24:20.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1418f6ca95aad75-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.383 DEBUG:boto:b'aktlji-14myobj1000falsemyobj2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.383 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.383 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.383 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.383 DEBUG:boto:Path: /aktlji-14/myobj 2024-04-19T23:24:20.383 DEBUG:boto:Data: 2024-04-19T23:24:20.384 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.384 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.384 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.384 DEBUG:boto:Params: {} 2024-04-19T23:24:20.384 DEBUG:boto:Token: None 2024-04-19T23:24:20.384 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj 2024-04-19T23:24:20.384 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GTqSReZ7EvmH4ofu6fwNDl5Ynsg= 2024-04-19T23:24:20.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GTqSReZ7EvmH4ofu6fwNDl5Ynsg=', 'Content-Length': '0'} 2024-04-19T23:24:20.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f66c24d0923efe4-006622fd24-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.386 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.387 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.387 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.387 DEBUG:boto:Path: /aktlji-14/myobj 2024-04-19T23:24:20.387 DEBUG:boto:Data: 2024-04-19T23:24:20.387 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.387 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.387 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.387 DEBUG:boto:Params: {} 2024-04-19T23:24:20.387 DEBUG:boto:Token: None 2024-04-19T23:24:20.387 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj 2024-04-19T23:24:20.388 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GTqSReZ7EvmH4ofu6fwNDl5Ynsg= 2024-04-19T23:24:20.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GTqSReZ7EvmH4ofu6fwNDl5Ynsg=', 'Content-Length': '0'} 2024-04-19T23:24:20.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040a5fb391f8a1262-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:20.390 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.390 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.390 DEBUG:boto:Method: GET 2024-04-19T23:24:20.390 DEBUG:boto:Path: /aktlji-14/myobj 2024-04-19T23:24:20.390 DEBUG:boto:Data: 2024-04-19T23:24:20.390 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.390 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.390 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.390 DEBUG:boto:Params: {} 2024-04-19T23:24:20.391 DEBUG:boto:Token: None 2024-04-19T23:24:20.391 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj 2024-04-19T23:24:20.391 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4JqED1y8ToSsPXMCcc9D6fEYRPA= 2024-04-19T23:24:20.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4JqED1y8ToSsPXMCcc9D6fEYRPA=', 'Content-Length': '0'} 2024-04-19T23:24:20.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fbe517890b5e0c8-006622fd24-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.438 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.439 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.439 DEBUG:boto:Method: GET 2024-04-19T23:24:20.439 DEBUG:boto:Path: /aktlji-14/myobj 2024-04-19T23:24:20.439 DEBUG:boto:Data: 2024-04-19T23:24:20.439 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.439 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.439 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.439 DEBUG:boto:Params: {} 2024-04-19T23:24:20.439 DEBUG:boto:Token: None 2024-04-19T23:24:20.440 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj 2024-04-19T23:24:20.440 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4JqED1y8ToSsPXMCcc9D6fEYRPA= 2024-04-19T23:24:20.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4JqED1y8ToSsPXMCcc9D6fEYRPA=', 'Content-Length': '0'} 2024-04-19T23:24:20.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000978be3e8e4128357-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.442 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.442 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.442 DEBUG:boto:path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.442 DEBUG:boto:auth_path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.443 DEBUG:boto:Method: GET 2024-04-19T23:24:20.443 DEBUG:boto:Path: /aktlji-14/?prefix=myobj 2024-04-19T23:24:20.443 DEBUG:boto:Data: 2024-04-19T23:24:20.443 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.443 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.443 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.443 DEBUG:boto:Params: {} 2024-04-19T23:24:20.443 DEBUG:boto:Token: None 2024-04-19T23:24:20.443 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/ 2024-04-19T23:24:20.443 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ= 2024-04-19T23:24:20.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce66e3a6503f65d5-006622fd24-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.446 DEBUG:boto:b'aktlji-14myobj1000falsemyobj2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.446 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.446 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.447 DEBUG:boto:path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.447 DEBUG:boto:auth_path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.447 DEBUG:boto:Method: GET 2024-04-19T23:24:20.447 DEBUG:boto:Path: /aktlji-14/?prefix=myobj 2024-04-19T23:24:20.447 DEBUG:boto:Data: 2024-04-19T23:24:20.447 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.447 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.447 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.447 DEBUG:boto:Params: {} 2024-04-19T23:24:20.447 DEBUG:boto:Token: None 2024-04-19T23:24:20.448 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/ 2024-04-19T23:24:20.448 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ= 2024-04-19T23:24:20.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3f889fef478caa0-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.450 DEBUG:boto:b'aktlji-14myobj1000falsemyobj2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.451 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-14 zones={test-zone1, test-zone2} 2024-04-19T23:24:20.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:20.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:20.451 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:24:20.518 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:20.519 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:20.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:20.564+0000 7f8f78a3d780 1 bucket sync caught up with source: 2024-04-19T23:24:20.560 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2024-04-19T23:24:20.560 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000001.4.6, , , , , , , , , ] 2024-04-19T23:24:20.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:20.564+0000 7f8f78a3d780 0 bucket checkpoint complete 2024-04-19T23:24:20.574 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-14 zones={test-zone3, test-zone2} 2024-04-19T23:24:20.574 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.574 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.574 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.574 DEBUG:boto:Path: /aktlji-14/ 2024-04-19T23:24:20.575 DEBUG:boto:Data: 2024-04-19T23:24:20.575 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.575 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.575 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.575 DEBUG:boto:Params: {} 2024-04-19T23:24:20.575 DEBUG:boto:Token: None 2024-04-19T23:24:20.575 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/ 2024-04-19T23:24:20.575 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2tziK7ZYE5Uv/g+hVVu0CDLhNGI= 2024-04-19T23:24:20.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2tziK7ZYE5Uv/g+hVVu0CDLhNGI=', 'Content-Length': '0'} 2024-04-19T23:24:20.578 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020959533f697201a-006622fd24-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.578 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.578 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.579 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.579 DEBUG:boto:Path: /aktlji-14/ 2024-04-19T23:24:20.579 DEBUG:boto:Data: 2024-04-19T23:24:20.579 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.579 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.580 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.580 DEBUG:boto:Params: {} 2024-04-19T23:24:20.580 DEBUG:boto:Token: None 2024-04-19T23:24:20.580 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/ 2024-04-19T23:24:20.580 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2tziK7ZYE5Uv/g+hVVu0CDLhNGI= 2024-04-19T23:24:20.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2tziK7ZYE5Uv/g+hVVu0CDLhNGI=', 'Content-Length': '0'} 2024-04-19T23:24:20.583 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000223a9e6b64033255-006622fd24-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.583 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:24:20.583 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.583 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.583 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.583 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.584 DEBUG:boto:Method: GET 2024-04-19T23:24:20.584 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.584 DEBUG:boto:Data: 2024-04-19T23:24:20.584 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.584 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.584 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.584 DEBUG:boto:Params: {} 2024-04-19T23:24:20.584 DEBUG:boto:Token: None 2024-04-19T23:24:20.584 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/?versions 2024-04-19T23:24:20.584 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ= 2024-04-19T23:24:20.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b4a342f7c291a86-006622fd24-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.587 DEBUG:boto:b'aktlji-14999falsemyobjnulltrue2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.587 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:20.587 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:24:20.588 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.588 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.588 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.588 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.588 DEBUG:boto:Method: GET 2024-04-19T23:24:20.588 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.588 DEBUG:boto:Data: 2024-04-19T23:24:20.588 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.588 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.588 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.589 DEBUG:boto:Params: {} 2024-04-19T23:24:20.589 DEBUG:boto:Token: None 2024-04-19T23:24:20.589 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/?versions 2024-04-19T23:24:20.589 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ= 2024-04-19T23:24:20.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e1f9046bbf41803-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.638 DEBUG:boto:b'aktlji-14999falsemyobjnulltrue2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.639 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:20.639 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.639 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.639 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.639 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.640 DEBUG:boto:Method: GET 2024-04-19T23:24:20.640 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.640 DEBUG:boto:Data: 2024-04-19T23:24:20.640 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.640 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.640 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.640 DEBUG:boto:Params: {} 2024-04-19T23:24:20.640 DEBUG:boto:Token: None 2024-04-19T23:24:20.640 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/?versions 2024-04-19T23:24:20.641 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ= 2024-04-19T23:24:20.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a821eca7f5eb1a36-006622fd24-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.644 DEBUG:boto:b'aktlji-14999falsemyobjnulltrue2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.645 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.645 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.645 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.645 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.645 DEBUG:boto:Method: GET 2024-04-19T23:24:20.646 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:24:20.646 DEBUG:boto:Data: 2024-04-19T23:24:20.646 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.646 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.646 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.646 DEBUG:boto:Params: {} 2024-04-19T23:24:20.647 DEBUG:boto:Token: None 2024-04-19T23:24:20.647 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/?versions 2024-04-19T23:24:20.647 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ= 2024-04-19T23:24:20.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3/xHmQuavyatw8QUi0i4kFhXneQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eda93322c8173562-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.651 DEBUG:boto:b'aktlji-14999falsemyobjnulltrue2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:20.651 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.652 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.652 DEBUG:boto:path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.652 DEBUG:boto:auth_path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.652 DEBUG:boto:Method: GET 2024-04-19T23:24:20.653 DEBUG:boto:Path: /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.653 DEBUG:boto:Data: 2024-04-19T23:24:20.653 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.653 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.654 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.654 DEBUG:boto:Params: {} 2024-04-19T23:24:20.654 DEBUG:boto:Token: None 2024-04-19T23:24:20.654 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.655 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg= 2024-04-19T23:24:20.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg=', 'Content-Length': '0'} 2024-04-19T23:24:20.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb59b8e8f19271e5-006622fd24-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.659 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.659 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.659 DEBUG:boto:path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.659 DEBUG:boto:auth_path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.659 DEBUG:boto:Method: GET 2024-04-19T23:24:20.660 DEBUG:boto:Path: /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.660 DEBUG:boto:Data: 2024-04-19T23:24:20.660 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.660 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.660 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.660 DEBUG:boto:Params: {} 2024-04-19T23:24:20.660 DEBUG:boto:Token: None 2024-04-19T23:24:20.660 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.661 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg= 2024-04-19T23:24:20.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg=', 'Content-Length': '0'} 2024-04-19T23:24:20.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d55d61b8760a83f-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.663 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.663 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.664 DEBUG:boto:path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.664 DEBUG:boto:auth_path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.664 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.664 DEBUG:boto:Path: /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.664 DEBUG:boto:Data: 2024-04-19T23:24:20.664 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.664 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.664 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.664 DEBUG:boto:Params: {} 2024-04-19T23:24:20.665 DEBUG:boto:Token: None 2024-04-19T23:24:20.665 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.665 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/I6qWTrH4hhssPMvuAYgZ5p7UQ0= 2024-04-19T23:24:20.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/I6qWTrH4hhssPMvuAYgZ5p7UQ0=', 'Content-Length': '0'} 2024-04-19T23:24:20.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000879209ed6b4dbdc1-006622fd24-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.667 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.668 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.668 DEBUG:boto:path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.668 DEBUG:boto:auth_path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.668 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.668 DEBUG:boto:Path: /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.668 DEBUG:boto:Data: 2024-04-19T23:24:20.668 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.668 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.668 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.668 DEBUG:boto:Params: {} 2024-04-19T23:24:20.668 DEBUG:boto:Token: None 2024-04-19T23:24:20.669 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.669 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/I6qWTrH4hhssPMvuAYgZ5p7UQ0= 2024-04-19T23:24:20.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/I6qWTrH4hhssPMvuAYgZ5p7UQ0=', 'Content-Length': '0'} 2024-04-19T23:24:20.671 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bff454cf04098491-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:20.672 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.672 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.672 DEBUG:boto:path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.672 DEBUG:boto:auth_path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.672 DEBUG:boto:Method: GET 2024-04-19T23:24:20.673 DEBUG:boto:Path: /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.673 DEBUG:boto:Data: 2024-04-19T23:24:20.673 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.673 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.673 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.673 DEBUG:boto:Params: {} 2024-04-19T23:24:20.673 DEBUG:boto:Token: None 2024-04-19T23:24:20.673 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.674 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg= 2024-04-19T23:24:20.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg=', 'Content-Length': '0'} 2024-04-19T23:24:20.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042efe09a194705c5-006622fd24-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.719 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.719 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.719 DEBUG:boto:path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.719 DEBUG:boto:auth_path=/aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.720 DEBUG:boto:Method: GET 2024-04-19T23:24:20.720 DEBUG:boto:Path: /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.720 DEBUG:boto:Data: 2024-04-19T23:24:20.720 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.720 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.720 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.720 DEBUG:boto:Params: {} 2024-04-19T23:24:20.720 DEBUG:boto:Token: None 2024-04-19T23:24:20.720 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj?versionId=null 2024-04-19T23:24:20.721 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg= 2024-04-19T23:24:20.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dWNmvf2LLDNPxW1OwPBhqO333Qg=', 'Content-Length': '0'} 2024-04-19T23:24:20.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3a6a1ea330892c0-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.723 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.723 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.724 DEBUG:boto:path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.724 DEBUG:boto:auth_path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.724 DEBUG:boto:Method: GET 2024-04-19T23:24:20.724 DEBUG:boto:Path: /aktlji-14/?prefix=myobj 2024-04-19T23:24:20.724 DEBUG:boto:Data: 2024-04-19T23:24:20.724 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.724 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.724 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.724 DEBUG:boto:Params: {} 2024-04-19T23:24:20.725 DEBUG:boto:Token: None 2024-04-19T23:24:20.725 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/ 2024-04-19T23:24:20.725 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ= 2024-04-19T23:24:20.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000379a056f634e5d0f-006622fd24-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.728 DEBUG:boto:b'aktlji-14myobj1000falsemyobj2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.728 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.728 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.728 DEBUG:boto:path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.728 DEBUG:boto:auth_path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.729 DEBUG:boto:Method: GET 2024-04-19T23:24:20.729 DEBUG:boto:Path: /aktlji-14/?prefix=myobj 2024-04-19T23:24:20.729 DEBUG:boto:Data: 2024-04-19T23:24:20.729 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.729 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.729 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.729 DEBUG:boto:Params: {} 2024-04-19T23:24:20.729 DEBUG:boto:Token: None 2024-04-19T23:24:20.729 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/ 2024-04-19T23:24:20.729 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ= 2024-04-19T23:24:20.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076748b2260e4a138-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.732 DEBUG:boto:b'aktlji-14myobj1000falsemyobj2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.732 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.732 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.733 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.733 DEBUG:boto:Path: /aktlji-14/myobj 2024-04-19T23:24:20.733 DEBUG:boto:Data: 2024-04-19T23:24:20.733 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.733 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.733 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.733 DEBUG:boto:Params: {} 2024-04-19T23:24:20.733 DEBUG:boto:Token: None 2024-04-19T23:24:20.733 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj 2024-04-19T23:24:20.733 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GTqSReZ7EvmH4ofu6fwNDl5Ynsg= 2024-04-19T23:24:20.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GTqSReZ7EvmH4ofu6fwNDl5Ynsg=', 'Content-Length': '0'} 2024-04-19T23:24:20.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083e166f476d932fe-006622fd24-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.738 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.738 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.738 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.739 DEBUG:boto:Path: /aktlji-14/myobj 2024-04-19T23:24:20.739 DEBUG:boto:Data: 2024-04-19T23:24:20.739 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.739 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.739 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.739 DEBUG:boto:Params: {} 2024-04-19T23:24:20.739 DEBUG:boto:Token: None 2024-04-19T23:24:20.740 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj 2024-04-19T23:24:20.740 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GTqSReZ7EvmH4ofu6fwNDl5Ynsg= 2024-04-19T23:24:20.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GTqSReZ7EvmH4ofu6fwNDl5Ynsg=', 'Content-Length': '0'} 2024-04-19T23:24:20.742 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c10a3570864f5a84-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:20.743 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.743 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.743 DEBUG:boto:Method: GET 2024-04-19T23:24:20.743 DEBUG:boto:Path: /aktlji-14/myobj 2024-04-19T23:24:20.743 DEBUG:boto:Data: 2024-04-19T23:24:20.743 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.743 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.743 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.743 DEBUG:boto:Params: {} 2024-04-19T23:24:20.744 DEBUG:boto:Token: None 2024-04-19T23:24:20.744 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj 2024-04-19T23:24:20.744 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4JqED1y8ToSsPXMCcc9D6fEYRPA= 2024-04-19T23:24:20.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4JqED1y8ToSsPXMCcc9D6fEYRPA=', 'Content-Length': '0'} 2024-04-19T23:24:20.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fbd599b919823e5-006622fd24-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.790 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:24:20.791 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:24:20.791 DEBUG:boto:Method: GET 2024-04-19T23:24:20.791 DEBUG:boto:Path: /aktlji-14/myobj 2024-04-19T23:24:20.791 DEBUG:boto:Data: 2024-04-19T23:24:20.791 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.791 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.791 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.791 DEBUG:boto:Params: {} 2024-04-19T23:24:20.792 DEBUG:boto:Token: None 2024-04-19T23:24:20.792 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/myobj 2024-04-19T23:24:20.792 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4JqED1y8ToSsPXMCcc9D6fEYRPA= 2024-04-19T23:24:20.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4JqED1y8ToSsPXMCcc9D6fEYRPA=', 'Content-Length': '0'} 2024-04-19T23:24:20.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3b151fb2e37e90f-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.794 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.794 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.794 DEBUG:boto:path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.794 DEBUG:boto:auth_path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.795 DEBUG:boto:Method: GET 2024-04-19T23:24:20.795 DEBUG:boto:Path: /aktlji-14/?prefix=myobj 2024-04-19T23:24:20.795 DEBUG:boto:Data: 2024-04-19T23:24:20.795 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.795 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.795 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.795 DEBUG:boto:Params: {} 2024-04-19T23:24:20.795 DEBUG:boto:Token: None 2024-04-19T23:24:20.795 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/ 2024-04-19T23:24:20.795 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ= 2024-04-19T23:24:20.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c8ddfa33a735ca4-006622fd24-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.798 DEBUG:boto:b'aktlji-14myobj1000falsemyobj2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.798 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:24:20.798 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:24:20.798 DEBUG:boto:path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.799 DEBUG:boto:auth_path=/aktlji-14/?prefix=myobj 2024-04-19T23:24:20.799 DEBUG:boto:Method: GET 2024-04-19T23:24:20.799 DEBUG:boto:Path: /aktlji-14/?prefix=myobj 2024-04-19T23:24:20.799 DEBUG:boto:Data: 2024-04-19T23:24:20.799 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.799 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:20.799 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.799 DEBUG:boto:Params: {} 2024-04-19T23:24:20.799 DEBUG:boto:Token: None 2024-04-19T23:24:20.799 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-14/ 2024-04-19T23:24:20.799 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ= 2024-04-19T23:24:20.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0h58y9TKdVVaPFkXGTQGy8evuhQ=', 'Content-Length': '0'} 2024-04-19T23:24:20.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fe3d3aa4d2cd7bd-006622fd24-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.803 DEBUG:boto:b'aktlji-14myobj1000falsemyobj2024-04-19T23:24:03.941Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.803 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-14 zones={test-zone3, test-zone2} 2024-04-19T23:24:20.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:20.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:20.804 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:24:20.870 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:20.871 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:20.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:20.916+0000 7fea7b0ab780 1 bucket sync caught up with source: 2024-04-19T23:24:20.912 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2024-04-19T23:24:20.912 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000001.18.6, , , , , , , , , ] 2024-04-19T23:24:20.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:20.916+0000 7fea7b0ab780 0 bucket checkpoint complete 2024-04-19T23:24:20.926 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-15 zones={test-zone1, test-zone3} 2024-04-19T23:24:20.926 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:20.926 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:20.926 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.926 DEBUG:boto:Path: /aktlji-15/ 2024-04-19T23:24:20.927 DEBUG:boto:Data: 2024-04-19T23:24:20.927 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.927 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.927 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.927 DEBUG:boto:Params: {} 2024-04-19T23:24:20.927 DEBUG:boto:Token: None 2024-04-19T23:24:20.927 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-15/ 2024-04-19T23:24:20.928 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aLkS8zRn/JPy5zB495tk+pIfv+Q= 2024-04-19T23:24:20.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aLkS8zRn/JPy5zB495tk+pIfv+Q=', 'Content-Length': '0'} 2024-04-19T23:24:20.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000674b7d73fd0651ba-006622fd24-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.930 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:20.930 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:20.930 DEBUG:boto:Method: HEAD 2024-04-19T23:24:20.931 DEBUG:boto:Path: /aktlji-15/ 2024-04-19T23:24:20.931 DEBUG:boto:Data: 2024-04-19T23:24:20.931 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.931 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.931 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.931 DEBUG:boto:Params: {} 2024-04-19T23:24:20.931 DEBUG:boto:Token: None 2024-04-19T23:24:20.931 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:20 GMT /aktlji-15/ 2024-04-19T23:24:20.931 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aLkS8zRn/JPy5zB495tk+pIfv+Q= 2024-04-19T23:24:20.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aLkS8zRn/JPy5zB495tk+pIfv+Q=', 'Content-Length': '0'} 2024-04-19T23:24:20.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f23c425cdb64907d-006622fd24-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.934 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:24:20.934 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:20.934 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:20.934 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:20.934 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:20.934 DEBUG:boto:Method: GET 2024-04-19T23:24:20.935 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:24:20.935 DEBUG:boto:Data: 2024-04-19T23:24:20.935 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.935 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.935 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.935 DEBUG:boto:Params: {} 2024-04-19T23:24:20.935 DEBUG:boto:Token: None 2024-04-19T23:24:20.936 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-15/?versions 2024-04-19T23:24:20.936 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XV49Doe2RpU+PC/wsUMZAsrpr2Q= 2024-04-19T23:24:20.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XV49Doe2RpU+PC/wsUMZAsrpr2Q=', 'Content-Length': '0'} 2024-04-19T23:24:20.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f757d5c8f1cd76f2-006622fd24-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.938 DEBUG:boto:b'aktlji-15999falsemyobjnulltrue2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.939 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:20.939 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:24:20.939 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:20.939 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:20.940 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:20.940 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:20.940 DEBUG:boto:Method: GET 2024-04-19T23:24:20.940 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:24:20.940 DEBUG:boto:Data: 2024-04-19T23:24:20.940 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.940 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.940 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.940 DEBUG:boto:Params: {} 2024-04-19T23:24:20.941 DEBUG:boto:Token: None 2024-04-19T23:24:20.941 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-15/?versions 2024-04-19T23:24:20.941 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XV49Doe2RpU+PC/wsUMZAsrpr2Q= 2024-04-19T23:24:20.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XV49Doe2RpU+PC/wsUMZAsrpr2Q=', 'Content-Length': '0'} 2024-04-19T23:24:20.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c680f16e5cd46fe5-006622fd24-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.943 DEBUG:boto:b'aktlji-15999falsemyobjnulltrue2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.944 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:20.944 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:20.944 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:20.944 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:20.944 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:20.944 DEBUG:boto:Method: GET 2024-04-19T23:24:20.945 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:24:20.945 DEBUG:boto:Data: 2024-04-19T23:24:20.945 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.945 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.945 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.945 DEBUG:boto:Params: {} 2024-04-19T23:24:20.945 DEBUG:boto:Token: None 2024-04-19T23:24:20.945 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-15/?versions 2024-04-19T23:24:20.945 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XV49Doe2RpU+PC/wsUMZAsrpr2Q= 2024-04-19T23:24:20.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XV49Doe2RpU+PC/wsUMZAsrpr2Q=', 'Content-Length': '0'} 2024-04-19T23:24:20.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e7a6238efa4f6fb-006622fd24-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.948 DEBUG:boto:b'aktlji-15999falsemyobjnulltrue2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.949 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:20.949 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:20.949 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:20.949 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:20.949 DEBUG:boto:Method: GET 2024-04-19T23:24:20.949 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:24:20.949 DEBUG:boto:Data: 2024-04-19T23:24:20.949 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.949 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.949 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.950 DEBUG:boto:Params: {} 2024-04-19T23:24:20.950 DEBUG:boto:Token: None 2024-04-19T23:24:20.950 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-15/?versions 2024-04-19T23:24:20.950 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XV49Doe2RpU+PC/wsUMZAsrpr2Q= 2024-04-19T23:24:20.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XV49Doe2RpU+PC/wsUMZAsrpr2Q=', 'Content-Length': '0'} 2024-04-19T23:24:20.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e3e1686a37a091b-006622fd24-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.953 DEBUG:boto:b'aktlji-15999falsemyobjnulltrue2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:20.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:20.953 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:20.954 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:20.954 DEBUG:boto:path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:20.954 DEBUG:boto:auth_path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:20.954 DEBUG:boto:Method: GET 2024-04-19T23:24:20.954 DEBUG:boto:Path: /aktlji-15/myobj?versionId=null 2024-04-19T23:24:20.954 DEBUG:boto:Data: 2024-04-19T23:24:20.954 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.954 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:20.955 DEBUG:boto:Port: 8000 2024-04-19T23:24:20.955 DEBUG:boto:Params: {} 2024-04-19T23:24:20.955 DEBUG:boto:Token: None 2024-04-19T23:24:20.955 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:20 GMT /aktlji-15/myobj?versionId=null 2024-04-19T23:24:20.955 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GLU596FL632mv3ijKqAT3sY6oiE= 2024-04-19T23:24:20.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GLU596FL632mv3ijKqAT3sY6oiE=', 'Content-Length': '0'} 2024-04-19T23:24:20.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e19f14819f84965-006622fd24-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:20.998 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:20.999 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:20.999 DEBUG:boto:path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:20.999 DEBUG:boto:auth_path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:20.999 DEBUG:boto:Method: GET 2024-04-19T23:24:20.999 DEBUG:boto:Path: /aktlji-15/myobj?versionId=null 2024-04-19T23:24:20.999 DEBUG:boto:Data: 2024-04-19T23:24:20.999 DEBUG:boto:Headers: {} 2024-04-19T23:24:20.999 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:20.999 DEBUG:boto:Port: 8001 2024-04-19T23:24:20.999 DEBUG:boto:Params: {} 2024-04-19T23:24:21.000 DEBUG:boto:Token: None 2024-04-19T23:24:21.000 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.000 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8= 2024-04-19T23:24:21.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8=', 'Content-Length': '0'} 2024-04-19T23:24:21.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebf707cf1e6af302-006622fd25-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.002 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.002 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.002 DEBUG:boto:path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.003 DEBUG:boto:auth_path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.003 DEBUG:boto:Method: HEAD 2024-04-19T23:24:21.003 DEBUG:boto:Path: /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.003 DEBUG:boto:Data: 2024-04-19T23:24:21.003 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.003 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:21.003 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.003 DEBUG:boto:Params: {} 2024-04-19T23:24:21.003 DEBUG:boto:Token: None 2024-04-19T23:24:21.003 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.003 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SadYw6iYCUb32r5Oi5NqwxAu2hg= 2024-04-19T23:24:21.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SadYw6iYCUb32r5Oi5NqwxAu2hg=', 'Content-Length': '0'} 2024-04-19T23:24:21.005 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cbaad0b74839900-006622fd25-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.006 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.006 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.006 DEBUG:boto:path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.006 DEBUG:boto:auth_path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.006 DEBUG:boto:Method: HEAD 2024-04-19T23:24:21.006 DEBUG:boto:Path: /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.006 DEBUG:boto:Data: 2024-04-19T23:24:21.006 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.007 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.007 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.007 DEBUG:boto:Params: {} 2024-04-19T23:24:21.007 DEBUG:boto:Token: None 2024-04-19T23:24:21.007 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.007 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SadYw6iYCUb32r5Oi5NqwxAu2hg= 2024-04-19T23:24:21.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SadYw6iYCUb32r5Oi5NqwxAu2hg=', 'Content-Length': '0'} 2024-04-19T23:24:21.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001131ae00e4672293-006622fd25-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:21.009 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.009 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.010 DEBUG:boto:path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.010 DEBUG:boto:auth_path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.010 DEBUG:boto:Method: GET 2024-04-19T23:24:21.010 DEBUG:boto:Path: /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.010 DEBUG:boto:Data: 2024-04-19T23:24:21.010 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.010 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:21.010 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.010 DEBUG:boto:Params: {} 2024-04-19T23:24:21.010 DEBUG:boto:Token: None 2024-04-19T23:24:21.011 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.011 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8= 2024-04-19T23:24:21.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8=', 'Content-Length': '0'} 2024-04-19T23:24:21.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd98ff6b98f1ef7b-006622fd25-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.014 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.014 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.014 DEBUG:boto:path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.014 DEBUG:boto:auth_path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.014 DEBUG:boto:Method: GET 2024-04-19T23:24:21.014 DEBUG:boto:Path: /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.014 DEBUG:boto:Data: 2024-04-19T23:24:21.014 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.014 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.014 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.014 DEBUG:boto:Params: {} 2024-04-19T23:24:21.015 DEBUG:boto:Token: None 2024-04-19T23:24:21.015 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.015 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8= 2024-04-19T23:24:21.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8=', 'Content-Length': '0'} 2024-04-19T23:24:21.017 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059fa0317f1db8fd3-006622fd25-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.017 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.017 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.017 DEBUG:boto:path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.017 DEBUG:boto:auth_path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.017 DEBUG:boto:Method: GET 2024-04-19T23:24:21.017 DEBUG:boto:Path: /aktlji-15/?prefix=myobj 2024-04-19T23:24:21.018 DEBUG:boto:Data: 2024-04-19T23:24:21.018 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.018 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:21.018 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.018 DEBUG:boto:Params: {} 2024-04-19T23:24:21.018 DEBUG:boto:Token: None 2024-04-19T23:24:21.018 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/ 2024-04-19T23:24:21.018 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU= 2024-04-19T23:24:21.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU=', 'Content-Length': '0'} 2024-04-19T23:24:21.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000db0c95bec249849-006622fd25-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.021 DEBUG:boto:b'aktlji-15myobj1000falsemyobj2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:21.021 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.022 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.022 DEBUG:boto:path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.022 DEBUG:boto:auth_path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.022 DEBUG:boto:Method: GET 2024-04-19T23:24:21.022 DEBUG:boto:Path: /aktlji-15/?prefix=myobj 2024-04-19T23:24:21.022 DEBUG:boto:Data: 2024-04-19T23:24:21.022 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.022 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.022 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.022 DEBUG:boto:Params: {} 2024-04-19T23:24:21.023 DEBUG:boto:Token: None 2024-04-19T23:24:21.023 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/ 2024-04-19T23:24:21.023 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU= 2024-04-19T23:24:21.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU=', 'Content-Length': '0'} 2024-04-19T23:24:21.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d89db060ceda0db8-006622fd25-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.025 DEBUG:boto:b'aktlji-15myobj1000falsemyobj2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:21.026 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.026 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.026 DEBUG:boto:Method: HEAD 2024-04-19T23:24:21.026 DEBUG:boto:Path: /aktlji-15/myobj 2024-04-19T23:24:21.026 DEBUG:boto:Data: 2024-04-19T23:24:21.026 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.026 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:21.026 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.026 DEBUG:boto:Params: {} 2024-04-19T23:24:21.027 DEBUG:boto:Token: None 2024-04-19T23:24:21.027 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj 2024-04-19T23:24:21.027 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oWuCyYkzCLwf4A+wEsM7FFhujsI= 2024-04-19T23:24:21.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oWuCyYkzCLwf4A+wEsM7FFhujsI=', 'Content-Length': '0'} 2024-04-19T23:24:21.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1b89336d2b27662-006622fd25-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.029 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.029 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.029 DEBUG:boto:Method: HEAD 2024-04-19T23:24:21.029 DEBUG:boto:Path: /aktlji-15/myobj 2024-04-19T23:24:21.029 DEBUG:boto:Data: 2024-04-19T23:24:21.029 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.029 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.029 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.030 DEBUG:boto:Params: {} 2024-04-19T23:24:21.030 DEBUG:boto:Token: None 2024-04-19T23:24:21.030 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj 2024-04-19T23:24:21.030 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oWuCyYkzCLwf4A+wEsM7FFhujsI= 2024-04-19T23:24:21.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oWuCyYkzCLwf4A+wEsM7FFhujsI=', 'Content-Length': '0'} 2024-04-19T23:24:21.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033f34dc785224ac0-006622fd25-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:21.032 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.032 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.032 DEBUG:boto:Method: GET 2024-04-19T23:24:21.032 DEBUG:boto:Path: /aktlji-15/myobj 2024-04-19T23:24:21.032 DEBUG:boto:Data: 2024-04-19T23:24:21.032 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.033 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:21.033 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.033 DEBUG:boto:Params: {} 2024-04-19T23:24:21.033 DEBUG:boto:Token: None 2024-04-19T23:24:21.033 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj 2024-04-19T23:24:21.033 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GoZCmNZkvvn1aEJzGjXOzBOJoog= 2024-04-19T23:24:21.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GoZCmNZkvvn1aEJzGjXOzBOJoog=', 'Content-Length': '0'} 2024-04-19T23:24:21.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bff76ffc1bd76696-006622fd25-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.035 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.035 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.035 DEBUG:boto:Method: GET 2024-04-19T23:24:21.036 DEBUG:boto:Path: /aktlji-15/myobj 2024-04-19T23:24:21.036 DEBUG:boto:Data: 2024-04-19T23:24:21.036 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.036 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.036 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.036 DEBUG:boto:Params: {} 2024-04-19T23:24:21.036 DEBUG:boto:Token: None 2024-04-19T23:24:21.036 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj 2024-04-19T23:24:21.036 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GoZCmNZkvvn1aEJzGjXOzBOJoog= 2024-04-19T23:24:21.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GoZCmNZkvvn1aEJzGjXOzBOJoog=', 'Content-Length': '0'} 2024-04-19T23:24:21.038 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb8888f81392957a-006622fd25-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.038 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.039 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.039 DEBUG:boto:path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.039 DEBUG:boto:auth_path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.039 DEBUG:boto:Method: GET 2024-04-19T23:24:21.039 DEBUG:boto:Path: /aktlji-15/?prefix=myobj 2024-04-19T23:24:21.039 DEBUG:boto:Data: 2024-04-19T23:24:21.039 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.039 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:21.039 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.039 DEBUG:boto:Params: {} 2024-04-19T23:24:21.039 DEBUG:boto:Token: None 2024-04-19T23:24:21.040 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/ 2024-04-19T23:24:21.040 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU= 2024-04-19T23:24:21.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU=', 'Content-Length': '0'} 2024-04-19T23:24:21.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c9c7305941719cc-006622fd25-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.042 DEBUG:boto:b'aktlji-15myobj1000falsemyobj2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:21.043 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.043 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.043 DEBUG:boto:path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.043 DEBUG:boto:auth_path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.043 DEBUG:boto:Method: GET 2024-04-19T23:24:21.043 DEBUG:boto:Path: /aktlji-15/?prefix=myobj 2024-04-19T23:24:21.043 DEBUG:boto:Data: 2024-04-19T23:24:21.043 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.043 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.043 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.044 DEBUG:boto:Params: {} 2024-04-19T23:24:21.044 DEBUG:boto:Token: None 2024-04-19T23:24:21.044 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/ 2024-04-19T23:24:21.044 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU= 2024-04-19T23:24:21.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU=', 'Content-Length': '0'} 2024-04-19T23:24:21.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9a57eabd9ac73ed-006622fd25-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.090 DEBUG:boto:b'aktlji-15myobj1000falsemyobj2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:21.091 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-15 zones={test-zone1, test-zone3} 2024-04-19T23:24:21.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:21.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:21.092 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:24:21.158 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:21.159 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:21.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:21.207+0000 7f2890b65780 1 bucket sync caught up with source: 2024-04-19T23:24:21.203 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2024-04-19T23:24:21.203 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000001.18.6, , , , , , , , , ] 2024-04-19T23:24:21.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:21.207+0000 7f2890b65780 0 bucket checkpoint complete 2024-04-19T23:24:21.217 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-15 zones={test-zone2, test-zone3} 2024-04-19T23:24:21.218 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.218 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.218 DEBUG:boto:Method: HEAD 2024-04-19T23:24:21.218 DEBUG:boto:Path: /aktlji-15/ 2024-04-19T23:24:21.218 DEBUG:boto:Data: 2024-04-19T23:24:21.218 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.218 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:21.218 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.218 DEBUG:boto:Params: {} 2024-04-19T23:24:21.219 DEBUG:boto:Token: None 2024-04-19T23:24:21.219 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/ 2024-04-19T23:24:21.219 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:M10mfthr9vO2fH1CgPOhwm+EgeI= 2024-04-19T23:24:21.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:M10mfthr9vO2fH1CgPOhwm+EgeI=', 'Content-Length': '0'} 2024-04-19T23:24:21.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4793003ae9cb377-006622fd25-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.221 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.221 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.222 DEBUG:boto:Method: HEAD 2024-04-19T23:24:21.222 DEBUG:boto:Path: /aktlji-15/ 2024-04-19T23:24:21.222 DEBUG:boto:Data: 2024-04-19T23:24:21.222 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.222 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.222 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.222 DEBUG:boto:Params: {} 2024-04-19T23:24:21.223 DEBUG:boto:Token: None 2024-04-19T23:24:21.223 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/ 2024-04-19T23:24:21.223 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:M10mfthr9vO2fH1CgPOhwm+EgeI= 2024-04-19T23:24:21.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:M10mfthr9vO2fH1CgPOhwm+EgeI=', 'Content-Length': '0'} 2024-04-19T23:24:21.225 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe01a96f4632df69-006622fd25-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.225 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:24:21.225 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.225 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.225 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:21.225 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:21.226 DEBUG:boto:Method: GET 2024-04-19T23:24:21.226 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:24:21.226 DEBUG:boto:Data: 2024-04-19T23:24:21.226 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.226 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:21.226 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.226 DEBUG:boto:Params: {} 2024-04-19T23:24:21.226 DEBUG:boto:Token: None 2024-04-19T23:24:21.226 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/?versions 2024-04-19T23:24:21.227 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d9BXQ8vLakBCFFpni4ZDwTM52MM= 2024-04-19T23:24:21.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d9BXQ8vLakBCFFpni4ZDwTM52MM=', 'Content-Length': '0'} 2024-04-19T23:24:21.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa26eb2d999aff5f-006622fd25-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.229 DEBUG:boto:b'aktlji-15999falsemyobjnulltrue2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:21.229 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:21.229 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:24:21.230 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.230 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.230 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:21.230 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:21.230 DEBUG:boto:Method: GET 2024-04-19T23:24:21.230 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:24:21.230 DEBUG:boto:Data: 2024-04-19T23:24:21.230 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.230 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.230 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.231 DEBUG:boto:Params: {} 2024-04-19T23:24:21.231 DEBUG:boto:Token: None 2024-04-19T23:24:21.231 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/?versions 2024-04-19T23:24:21.231 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d9BXQ8vLakBCFFpni4ZDwTM52MM= 2024-04-19T23:24:21.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d9BXQ8vLakBCFFpni4ZDwTM52MM=', 'Content-Length': '0'} 2024-04-19T23:24:21.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c0f6137db341d65-006622fd25-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.234 DEBUG:boto:b'aktlji-15999falsemyobjnulltrue2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:21.234 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-19T23:24:21.234 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.234 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.234 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:21.234 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:21.235 DEBUG:boto:Method: GET 2024-04-19T23:24:21.235 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:24:21.235 DEBUG:boto:Data: 2024-04-19T23:24:21.235 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.235 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:21.235 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.235 DEBUG:boto:Params: {} 2024-04-19T23:24:21.235 DEBUG:boto:Token: None 2024-04-19T23:24:21.235 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/?versions 2024-04-19T23:24:21.235 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d9BXQ8vLakBCFFpni4ZDwTM52MM= 2024-04-19T23:24:21.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d9BXQ8vLakBCFFpni4ZDwTM52MM=', 'Content-Length': '0'} 2024-04-19T23:24:21.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee89d1c17a57eb1-006622fd25-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.238 DEBUG:boto:b'aktlji-15999falsemyobjnulltrue2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:21.238 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.238 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.238 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:21.239 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:24:21.239 DEBUG:boto:Method: GET 2024-04-19T23:24:21.239 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:24:21.239 DEBUG:boto:Data: 2024-04-19T23:24:21.239 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.239 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.239 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.239 DEBUG:boto:Params: {} 2024-04-19T23:24:21.239 DEBUG:boto:Token: None 2024-04-19T23:24:21.239 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/?versions 2024-04-19T23:24:21.240 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d9BXQ8vLakBCFFpni4ZDwTM52MM= 2024-04-19T23:24:21.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d9BXQ8vLakBCFFpni4ZDwTM52MM=', 'Content-Length': '0'} 2024-04-19T23:24:21.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000041c4806a9086b3e-006622fd25-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.242 DEBUG:boto:b'aktlji-15999falsemyobjnulltrue2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:21.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:21.243 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.243 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.243 DEBUG:boto:path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.243 DEBUG:boto:auth_path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.243 DEBUG:boto:Method: GET 2024-04-19T23:24:21.243 DEBUG:boto:Path: /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.243 DEBUG:boto:Data: 2024-04-19T23:24:21.243 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.243 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:21.243 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.244 DEBUG:boto:Params: {} 2024-04-19T23:24:21.244 DEBUG:boto:Token: None 2024-04-19T23:24:21.244 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.244 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8= 2024-04-19T23:24:21.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8=', 'Content-Length': '0'} 2024-04-19T23:24:21.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e724a910933cc81-006622fd25-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.246 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.246 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.246 DEBUG:boto:path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.246 DEBUG:boto:auth_path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.246 DEBUG:boto:Method: GET 2024-04-19T23:24:21.246 DEBUG:boto:Path: /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.247 DEBUG:boto:Data: 2024-04-19T23:24:21.247 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.247 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.247 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.247 DEBUG:boto:Params: {} 2024-04-19T23:24:21.247 DEBUG:boto:Token: None 2024-04-19T23:24:21.247 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.247 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8= 2024-04-19T23:24:21.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8=', 'Content-Length': '0'} 2024-04-19T23:24:21.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f19c4532ce98e3b5-006622fd25-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.250 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.250 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.250 DEBUG:boto:path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.250 DEBUG:boto:auth_path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.250 DEBUG:boto:Method: HEAD 2024-04-19T23:24:21.250 DEBUG:boto:Path: /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.250 DEBUG:boto:Data: 2024-04-19T23:24:21.250 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.250 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:21.250 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.250 DEBUG:boto:Params: {} 2024-04-19T23:24:21.251 DEBUG:boto:Token: None 2024-04-19T23:24:21.251 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.251 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SadYw6iYCUb32r5Oi5NqwxAu2hg= 2024-04-19T23:24:21.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SadYw6iYCUb32r5Oi5NqwxAu2hg=', 'Content-Length': '0'} 2024-04-19T23:24:21.252 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003530a5f4c4fd11ad-006622fd25-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.253 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.253 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.253 DEBUG:boto:path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.253 DEBUG:boto:auth_path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.253 DEBUG:boto:Method: HEAD 2024-04-19T23:24:21.253 DEBUG:boto:Path: /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.253 DEBUG:boto:Data: 2024-04-19T23:24:21.253 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.254 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.254 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.254 DEBUG:boto:Params: {} 2024-04-19T23:24:21.254 DEBUG:boto:Token: None 2024-04-19T23:24:21.254 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.254 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SadYw6iYCUb32r5Oi5NqwxAu2hg= 2024-04-19T23:24:21.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SadYw6iYCUb32r5Oi5NqwxAu2hg=', 'Content-Length': '0'} 2024-04-19T23:24:21.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aff24957d793fd1a-006622fd25-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:21.256 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.256 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.256 DEBUG:boto:path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.257 DEBUG:boto:auth_path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.257 DEBUG:boto:Method: GET 2024-04-19T23:24:21.257 DEBUG:boto:Path: /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.257 DEBUG:boto:Data: 2024-04-19T23:24:21.257 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.257 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:21.257 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.257 DEBUG:boto:Params: {} 2024-04-19T23:24:21.257 DEBUG:boto:Token: None 2024-04-19T23:24:21.257 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.258 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8= 2024-04-19T23:24:21.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8=', 'Content-Length': '0'} 2024-04-19T23:24:21.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4ae98d372b02457-006622fd25-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.259 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.260 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.260 DEBUG:boto:path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.260 DEBUG:boto:auth_path=/aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.260 DEBUG:boto:Method: GET 2024-04-19T23:24:21.260 DEBUG:boto:Path: /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.260 DEBUG:boto:Data: 2024-04-19T23:24:21.260 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.260 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.260 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.260 DEBUG:boto:Params: {} 2024-04-19T23:24:21.261 DEBUG:boto:Token: None 2024-04-19T23:24:21.261 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj?versionId=null 2024-04-19T23:24:21.261 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8= 2024-04-19T23:24:21.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jmYIGUsuA5e1Wtsu4eyNqC4SmL8=', 'Content-Length': '0'} 2024-04-19T23:24:21.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc3e46638808b5a9-006622fd25-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.263 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.263 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.263 DEBUG:boto:path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.263 DEBUG:boto:auth_path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.263 DEBUG:boto:Method: GET 2024-04-19T23:24:21.263 DEBUG:boto:Path: /aktlji-15/?prefix=myobj 2024-04-19T23:24:21.264 DEBUG:boto:Data: 2024-04-19T23:24:21.264 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.264 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:21.264 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.264 DEBUG:boto:Params: {} 2024-04-19T23:24:21.264 DEBUG:boto:Token: None 2024-04-19T23:24:21.264 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/ 2024-04-19T23:24:21.264 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU= 2024-04-19T23:24:21.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU=', 'Content-Length': '0'} 2024-04-19T23:24:21.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a24673e4f5b4071-006622fd25-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.314 DEBUG:boto:b'aktlji-15myobj1000falsemyobj2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:21.315 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.315 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.315 DEBUG:boto:path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.315 DEBUG:boto:auth_path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.315 DEBUG:boto:Method: GET 2024-04-19T23:24:21.315 DEBUG:boto:Path: /aktlji-15/?prefix=myobj 2024-04-19T23:24:21.315 DEBUG:boto:Data: 2024-04-19T23:24:21.315 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.315 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.316 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.316 DEBUG:boto:Params: {} 2024-04-19T23:24:21.316 DEBUG:boto:Token: None 2024-04-19T23:24:21.316 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/ 2024-04-19T23:24:21.316 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU= 2024-04-19T23:24:21.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU=', 'Content-Length': '0'} 2024-04-19T23:24:21.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fe5b11f420ac129-006622fd25-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.319 DEBUG:boto:b'aktlji-15myobj1000falsemyobj2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:21.319 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.319 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.319 DEBUG:boto:Method: HEAD 2024-04-19T23:24:21.319 DEBUG:boto:Path: /aktlji-15/myobj 2024-04-19T23:24:21.319 DEBUG:boto:Data: 2024-04-19T23:24:21.320 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.320 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:21.320 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.320 DEBUG:boto:Params: {} 2024-04-19T23:24:21.320 DEBUG:boto:Token: None 2024-04-19T23:24:21.320 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj 2024-04-19T23:24:21.320 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oWuCyYkzCLwf4A+wEsM7FFhujsI= 2024-04-19T23:24:21.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oWuCyYkzCLwf4A+wEsM7FFhujsI=', 'Content-Length': '0'} 2024-04-19T23:24:21.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000596d1a6b2e1007e6-006622fd25-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.322 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.322 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.322 DEBUG:boto:Method: HEAD 2024-04-19T23:24:21.323 DEBUG:boto:Path: /aktlji-15/myobj 2024-04-19T23:24:21.323 DEBUG:boto:Data: 2024-04-19T23:24:21.323 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.323 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.323 DEBUG:boto:Params: {} 2024-04-19T23:24:21.323 DEBUG:boto:Token: None 2024-04-19T23:24:21.323 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj 2024-04-19T23:24:21.323 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oWuCyYkzCLwf4A+wEsM7FFhujsI= 2024-04-19T23:24:21.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oWuCyYkzCLwf4A+wEsM7FFhujsI=', 'Content-Length': '0'} 2024-04-19T23:24:21.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000559748bca7f4da60-006622fd25-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-19T23:24:21.326 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.326 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.326 DEBUG:boto:Method: GET 2024-04-19T23:24:21.326 DEBUG:boto:Path: /aktlji-15/myobj 2024-04-19T23:24:21.326 DEBUG:boto:Data: 2024-04-19T23:24:21.326 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.326 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:21.326 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.326 DEBUG:boto:Params: {} 2024-04-19T23:24:21.327 DEBUG:boto:Token: None 2024-04-19T23:24:21.327 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj 2024-04-19T23:24:21.327 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GoZCmNZkvvn1aEJzGjXOzBOJoog= 2024-04-19T23:24:21.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GoZCmNZkvvn1aEJzGjXOzBOJoog=', 'Content-Length': '0'} 2024-04-19T23:24:21.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003761019ea5df9110-006622fd25-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.329 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:24:21.329 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:24:21.329 DEBUG:boto:Method: GET 2024-04-19T23:24:21.329 DEBUG:boto:Path: /aktlji-15/myobj 2024-04-19T23:24:21.329 DEBUG:boto:Data: 2024-04-19T23:24:21.329 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.329 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.330 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.330 DEBUG:boto:Params: {} 2024-04-19T23:24:21.330 DEBUG:boto:Token: None 2024-04-19T23:24:21.330 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/myobj 2024-04-19T23:24:21.330 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GoZCmNZkvvn1aEJzGjXOzBOJoog= 2024-04-19T23:24:21.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GoZCmNZkvvn1aEJzGjXOzBOJoog=', 'Content-Length': '0'} 2024-04-19T23:24:21.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ede357cf19aa14fb-006622fd25-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.332 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.332 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.332 DEBUG:boto:path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.332 DEBUG:boto:auth_path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.333 DEBUG:boto:Method: GET 2024-04-19T23:24:21.333 DEBUG:boto:Path: /aktlji-15/?prefix=myobj 2024-04-19T23:24:21.333 DEBUG:boto:Data: 2024-04-19T23:24:21.333 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.333 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:21.333 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.333 DEBUG:boto:Params: {} 2024-04-19T23:24:21.333 DEBUG:boto:Token: None 2024-04-19T23:24:21.333 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/ 2024-04-19T23:24:21.333 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU= 2024-04-19T23:24:21.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU=', 'Content-Length': '0'} 2024-04-19T23:24:21.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad5f926b058fd2f4-006622fd25-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.336 DEBUG:boto:b'aktlji-15myobj1000falsemyobj2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:21.336 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:24:21.337 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:24:21.337 DEBUG:boto:path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.337 DEBUG:boto:auth_path=/aktlji-15/?prefix=myobj 2024-04-19T23:24:21.337 DEBUG:boto:Method: GET 2024-04-19T23:24:21.337 DEBUG:boto:Path: /aktlji-15/?prefix=myobj 2024-04-19T23:24:21.337 DEBUG:boto:Data: 2024-04-19T23:24:21.337 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.337 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:24:21.337 DEBUG:boto:Port: 8001 2024-04-19T23:24:21.337 DEBUG:boto:Params: {} 2024-04-19T23:24:21.338 DEBUG:boto:Token: None 2024-04-19T23:24:21.338 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:21 GMT /aktlji-15/ 2024-04-19T23:24:21.338 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU= 2024-04-19T23:24:21.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LvwMArxL/g7HiUD1KDi87WeQJkU=', 'Content-Length': '0'} 2024-04-19T23:24:21.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f142f75e81f4a66c-006622fd25-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.340 DEBUG:boto:b'aktlji-15myobj1000falsemyobj2024-04-19T23:24:03.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:24:21.341 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-15 zones={test-zone2, test-zone3} 2024-04-19T23:24:21.341 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:21.341 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:21.341 DEBUG:boto:Method: HEAD 2024-04-19T23:24:21.341 DEBUG:boto:Path: /aktlji-13/ 2024-04-19T23:24:21.341 DEBUG:boto:Data: 2024-04-19T23:24:21.341 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.341 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:21.341 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.342 DEBUG:boto:Params: {} 2024-04-19T23:24:21.342 DEBUG:boto:Token: None 2024-04-19T23:24:21.342 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:21 GMT /aktlji-13/ 2024-04-19T23:24:21.342 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hPYOGVxY/t3+99G8OCHQlkyHv/A= 2024-04-19T23:24:21.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hPYOGVxY/t3+99G8OCHQlkyHv/A=', 'Content-Length': '0'} 2024-04-19T23:24:21.345 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000287a15f62c87e7ec-006622fd25-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.345 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:21.345 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:21.345 DEBUG:boto:Method: HEAD 2024-04-19T23:24:21.345 DEBUG:boto:Path: /aktlji-13/myobj 2024-04-19T23:24:21.345 DEBUG:boto:Data: 2024-04-19T23:24:21.345 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.345 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:21.345 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.345 DEBUG:boto:Params: {} 2024-04-19T23:24:21.346 DEBUG:boto:Token: None 2024-04-19T23:24:21.346 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:21 GMT /aktlji-13/myobj 2024-04-19T23:24:21.346 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qpU6c5NIT2d4ACcYPPGd6eWtFds= 2024-04-19T23:24:21.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qpU6c5NIT2d4ACcYPPGd6eWtFds=', 'Content-Length': '0'} 2024-04-19T23:24:21.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007788aef27f0a6c73-006622fd25-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.348 DEBUG:boto:path=/aktlji-13/myobj 2024-04-19T23:24:21.348 DEBUG:boto:auth_path=/aktlji-13/myobj 2024-04-19T23:24:21.348 DEBUG:boto:Method: DELETE 2024-04-19T23:24:21.348 DEBUG:boto:Path: /aktlji-13/myobj 2024-04-19T23:24:21.348 DEBUG:boto:Data: 2024-04-19T23:24:21.348 DEBUG:boto:Headers: {} 2024-04-19T23:24:21.348 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:21.348 DEBUG:boto:Port: 8000 2024-04-19T23:24:21.349 DEBUG:boto:Params: {} 2024-04-19T23:24:21.349 DEBUG:boto:Token: None 2024-04-19T23:24:21.349 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:24:21 GMT /aktlji-13/myobj 2024-04-19T23:24:21.349 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ko5UnIYFD1hEo3UDLAS7dfzEQqw= 2024-04-19T23:24:21.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ko5UnIYFD1hEo3UDLAS7dfzEQqw=', 'Content-Length': '0'} 2024-04-19T23:24:21.353 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d5aadd5d32ae412f-006622fd25-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:24:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:21.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:21.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:21.354 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:24:21.421 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:21.422 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:21.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:21.468+0000 7f9383b5e780 1 waiting for incremental sync to catch up: 2024-04-19T23:24:21.464 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:24:21.464 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:24:26.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:26.469+0000 7f9383b5e780 1 waiting for incremental sync to catch up: 2024-04-19T23:24:26.466 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:24:26.466 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:24:31.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:31.471+0000 7f9383b5e780 1 waiting for incremental sync to catch up: 2024-04-19T23:24:31.467 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:24:31.467 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:24:36.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:36.472+0000 7f9383b5e780 1 bucket sync caught up with source: 2024-04-19T23:24:36.468 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:24:36.468 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:24:36.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:24:36.472+0000 7f9383b5e780 0 bucket checkpoint complete 2024-04-19T23:24:36.485 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-13 zones={test-zone2, test-zone1} 2024-04-19T23:24:36.485 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:36.485 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:36.486 DEBUG:boto:Method: HEAD 2024-04-19T23:24:36.486 DEBUG:boto:Path: /aktlji-13/ 2024-04-19T23:24:36.486 DEBUG:boto:Data: 2024-04-19T23:24:36.486 DEBUG:boto:Headers: {} 2024-04-19T23:24:36.486 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:36.486 DEBUG:boto:Port: 8000 2024-04-19T23:24:36.486 DEBUG:boto:Params: {} 2024-04-19T23:24:36.486 DEBUG:boto:Token: None 2024-04-19T23:24:36.487 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:36 GMT /aktlji-13/ 2024-04-19T23:24:36.487 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UOY53gOnRxDQ4MG/i8PnID+9gZE= 2024-04-19T23:24:36.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UOY53gOnRxDQ4MG/i8PnID+9gZE=', 'Content-Length': '0'} 2024-04-19T23:24:36.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c4d5c1f648276f0-006622fd34-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:36.490 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:36.490 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:36.490 DEBUG:boto:Method: HEAD 2024-04-19T23:24:36.490 DEBUG:boto:Path: /aktlji-13/ 2024-04-19T23:24:36.490 DEBUG:boto:Data: 2024-04-19T23:24:36.490 DEBUG:boto:Headers: {} 2024-04-19T23:24:36.490 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:36.490 DEBUG:boto:Port: 8000 2024-04-19T23:24:36.491 DEBUG:boto:Params: {} 2024-04-19T23:24:36.491 DEBUG:boto:Token: None 2024-04-19T23:24:36.491 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:24:36 GMT /aktlji-13/ 2024-04-19T23:24:36.491 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UOY53gOnRxDQ4MG/i8PnID+9gZE= 2024-04-19T23:24:36.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UOY53gOnRxDQ4MG/i8PnID+9gZE=', 'Content-Length': '0'} 2024-04-19T23:24:36.494 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089ccc2281c8cf073-006622fd34-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:24:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:36.494 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:24:36.494 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:36.494 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:36.494 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:36.494 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:36.494 DEBUG:boto:Method: GET 2024-04-19T23:24:36.495 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:24:36.495 DEBUG:boto:Data: 2024-04-19T23:24:36.495 DEBUG:boto:Headers: {} 2024-04-19T23:24:36.495 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:36.495 DEBUG:boto:Port: 8000 2024-04-19T23:24:36.495 DEBUG:boto:Params: {} 2024-04-19T23:24:36.495 DEBUG:boto:Token: None 2024-04-19T23:24:36.495 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:36 GMT /aktlji-13/?versions 2024-04-19T23:24:36.495 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kvhIjmlVjXx5NhsnwvaSy8zBv+k= 2024-04-19T23:24:36.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kvhIjmlVjXx5NhsnwvaSy8zBv+k=', 'Content-Length': '0'} 2024-04-19T23:24:36.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057edc37d95c51b90-006622fd34-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:36.498 DEBUG:boto:b'aktlji-13999false' 2024-04-19T23:24:36.498 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:24:36.499 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:36.499 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:36.499 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:36.499 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:36.499 DEBUG:boto:Method: GET 2024-04-19T23:24:36.499 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:24:36.499 DEBUG:boto:Data: 2024-04-19T23:24:36.499 DEBUG:boto:Headers: {} 2024-04-19T23:24:36.499 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:36.500 DEBUG:boto:Port: 8000 2024-04-19T23:24:36.500 DEBUG:boto:Params: {} 2024-04-19T23:24:36.500 DEBUG:boto:Token: None 2024-04-19T23:24:36.500 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:36 GMT /aktlji-13/?versions 2024-04-19T23:24:36.500 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kvhIjmlVjXx5NhsnwvaSy8zBv+k= 2024-04-19T23:24:36.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kvhIjmlVjXx5NhsnwvaSy8zBv+k=', 'Content-Length': '0'} 2024-04-19T23:24:36.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009051ce19141ebdcd-006622fd34-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:36.503 DEBUG:boto:b'aktlji-13999false' 2024-04-19T23:24:36.503 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:36.503 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:36.504 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:36.504 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:36.504 DEBUG:boto:Method: GET 2024-04-19T23:24:36.504 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:24:36.504 DEBUG:boto:Data: 2024-04-19T23:24:36.504 DEBUG:boto:Headers: {} 2024-04-19T23:24:36.504 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:24:36.504 DEBUG:boto:Port: 8000 2024-04-19T23:24:36.504 DEBUG:boto:Params: {} 2024-04-19T23:24:36.504 DEBUG:boto:Token: None 2024-04-19T23:24:36.504 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:36 GMT /aktlji-13/?versions 2024-04-19T23:24:36.505 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kvhIjmlVjXx5NhsnwvaSy8zBv+k= 2024-04-19T23:24:36.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kvhIjmlVjXx5NhsnwvaSy8zBv+k=', 'Content-Length': '0'} 2024-04-19T23:24:36.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003885599a136671c9-006622fd34-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:36.508 DEBUG:boto:b'aktlji-13999false' 2024-04-19T23:24:36.508 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:24:36.508 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:24:36.508 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:36.508 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:24:36.509 DEBUG:boto:Method: GET 2024-04-19T23:24:36.509 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:24:36.509 DEBUG:boto:Data: 2024-04-19T23:24:36.509 DEBUG:boto:Headers: {} 2024-04-19T23:24:36.509 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:24:36.509 DEBUG:boto:Port: 8000 2024-04-19T23:24:36.509 DEBUG:boto:Params: {} 2024-04-19T23:24:36.509 DEBUG:boto:Token: None 2024-04-19T23:24:36.509 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:24:36 GMT /aktlji-13/?versions 2024-04-19T23:24:36.509 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kvhIjmlVjXx5NhsnwvaSy8zBv+k= 2024-04-19T23:24:36.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:24:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kvhIjmlVjXx5NhsnwvaSy8zBv+k=', 'Content-Length': '0'} 2024-04-19T23:24:36.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2c2a431a64b7ef2-006622fd34-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:24:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:24:36.554 DEBUG:boto:b'aktlji-13999false' 2024-04-19T23:24:36.555 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-13 zones={test-zone2, test-zone1} 2024-04-19T23:24:36.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:36.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:24:36.555 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:24:36.623 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:24:36.623 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:24:36.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:36.668+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:24:36.665 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:24:36.665 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:24:41.666 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:41.669+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:24:41.666 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:24:41.666 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:24:46.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:46.671+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:24:46.668 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:24:46.668 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:24:51.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:51.672+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:24:51.669 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:24:51.669 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:24:56.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:24:56.674+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:24:56.671 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:24:56.671 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:25:01.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:01.675+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:25:01.672 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:25:01.672 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:06.676+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:25:11.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:11.678+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:25:11.675 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:25:11.675 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:25:16.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:16.679+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:25:16.676 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:25:16.676 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:25:21.677 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:21.680+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:25:21.677 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:25:21.677 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:25:26.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:26.682+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:25:26.678 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:25:26.678 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:25:31.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:31.683+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:25:31.680 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:25:31.680 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:25:36.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:36.685+0000 7f31765d5780 1 waiting for incremental sync to catch up: 2024-04-19T23:25:36.681 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.10.6, , , , , , , , , ] 2024-04-19T23:25:36.681 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:25:41.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:41.687+0000 7f31765d5780 1 bucket sync caught up with source: 2024-04-19T23:25:41.683 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:25:41.683 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.12.6, , , , , , , , , ] 2024-04-19T23:25:41.683 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:41.687+0000 7f31765d5780 0 bucket checkpoint complete 2024-04-19T23:25:41.699 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-13 zones={test-zone3, test-zone1} 2024-04-19T23:25:41.699 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:25:41.700 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:25:41.700 DEBUG:boto:Method: HEAD 2024-04-19T23:25:41.700 DEBUG:boto:Path: /aktlji-13/ 2024-04-19T23:25:41.700 DEBUG:boto:Data: 2024-04-19T23:25:41.700 DEBUG:boto:Headers: {} 2024-04-19T23:25:41.700 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:25:41.700 DEBUG:boto:Port: 8001 2024-04-19T23:25:41.700 DEBUG:boto:Params: {} 2024-04-19T23:25:41.701 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-19T23:25:41.701 DEBUG:boto:Token: None 2024-04-19T23:25:41.701 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:25:41 GMT /aktlji-13/ 2024-04-19T23:25:41.701 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qvn731cZ/XlhoMiVCgDiWaSzsl8= 2024-04-19T23:25:41.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qvn731cZ/XlhoMiVCgDiWaSzsl8=', 'Content-Length': '0'} 2024-04-19T23:25:41.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d5d2b56aa1e791b-006622fd75-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:41.707 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:25:41.707 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:25:41.707 DEBUG:boto:Method: HEAD 2024-04-19T23:25:41.707 DEBUG:boto:Path: /aktlji-13/ 2024-04-19T23:25:41.707 DEBUG:boto:Data: 2024-04-19T23:25:41.707 DEBUG:boto:Headers: {} 2024-04-19T23:25:41.707 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:25:41.708 DEBUG:boto:Port: 8000 2024-04-19T23:25:41.708 DEBUG:boto:Params: {} 2024-04-19T23:25:41.708 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:25:41.708 DEBUG:boto:Token: None 2024-04-19T23:25:41.708 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:25:41 GMT /aktlji-13/ 2024-04-19T23:25:41.708 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qvn731cZ/XlhoMiVCgDiWaSzsl8= 2024-04-19T23:25:41.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qvn731cZ/XlhoMiVCgDiWaSzsl8=', 'Content-Length': '0'} 2024-04-19T23:25:41.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0d340b5373015e2-006622fd75-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:41.712 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:25:41.713 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:25:41.713 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:25:41.713 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:25:41.713 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:25:41.713 DEBUG:boto:Method: GET 2024-04-19T23:25:41.713 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:25:41.713 DEBUG:boto:Data: 2024-04-19T23:25:41.713 DEBUG:boto:Headers: {} 2024-04-19T23:25:41.713 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:25:41.714 DEBUG:boto:Port: 8001 2024-04-19T23:25:41.714 DEBUG:boto:Params: {} 2024-04-19T23:25:41.714 DEBUG:boto:Token: None 2024-04-19T23:25:41.714 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:25:41 GMT /aktlji-13/?versions 2024-04-19T23:25:41.714 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qu8Sw0hnPDfqvkXFFGi90C2OVBE= 2024-04-19T23:25:41.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qu8Sw0hnPDfqvkXFFGi90C2OVBE=', 'Content-Length': '0'} 2024-04-19T23:25:41.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccdabfd627695879-006622fd75-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:41.717 DEBUG:boto:b'aktlji-13999false' 2024-04-19T23:25:41.717 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:25:41.717 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:25:41.718 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:25:41.718 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:25:41.718 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:25:41.718 DEBUG:boto:Method: GET 2024-04-19T23:25:41.718 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:25:41.718 DEBUG:boto:Data: 2024-04-19T23:25:41.718 DEBUG:boto:Headers: {} 2024-04-19T23:25:41.718 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:25:41.718 DEBUG:boto:Port: 8000 2024-04-19T23:25:41.719 DEBUG:boto:Params: {} 2024-04-19T23:25:41.719 DEBUG:boto:Token: None 2024-04-19T23:25:41.719 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:25:41 GMT /aktlji-13/?versions 2024-04-19T23:25:41.719 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qu8Sw0hnPDfqvkXFFGi90C2OVBE= 2024-04-19T23:25:41.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qu8Sw0hnPDfqvkXFFGi90C2OVBE=', 'Content-Length': '0'} 2024-04-19T23:25:41.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047a053acbf2ac56d-006622fd75-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:41.722 DEBUG:boto:b'aktlji-13999false' 2024-04-19T23:25:41.722 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:25:41.722 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:25:41.722 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:25:41.722 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:25:41.723 DEBUG:boto:Method: GET 2024-04-19T23:25:41.723 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:25:41.723 DEBUG:boto:Data: 2024-04-19T23:25:41.723 DEBUG:boto:Headers: {} 2024-04-19T23:25:41.723 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:25:41.723 DEBUG:boto:Port: 8001 2024-04-19T23:25:41.723 DEBUG:boto:Params: {} 2024-04-19T23:25:41.723 DEBUG:boto:Token: None 2024-04-19T23:25:41.723 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:25:41 GMT /aktlji-13/?versions 2024-04-19T23:25:41.723 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qu8Sw0hnPDfqvkXFFGi90C2OVBE= 2024-04-19T23:25:41.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qu8Sw0hnPDfqvkXFFGi90C2OVBE=', 'Content-Length': '0'} 2024-04-19T23:25:41.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bdc914dac1eb1d7-006622fd75-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:41.726 DEBUG:boto:b'aktlji-13999false' 2024-04-19T23:25:41.727 DEBUG:boto:path=/aktlji-13/ 2024-04-19T23:25:41.727 DEBUG:boto:auth_path=/aktlji-13/ 2024-04-19T23:25:41.727 DEBUG:boto:path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:25:41.727 DEBUG:boto:auth_path=/aktlji-13/?versions&max-keys=999 2024-04-19T23:25:41.727 DEBUG:boto:Method: GET 2024-04-19T23:25:41.727 DEBUG:boto:Path: /aktlji-13/?versions&max-keys=999 2024-04-19T23:25:41.727 DEBUG:boto:Data: 2024-04-19T23:25:41.727 DEBUG:boto:Headers: {} 2024-04-19T23:25:41.727 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:25:41.727 DEBUG:boto:Port: 8000 2024-04-19T23:25:41.727 DEBUG:boto:Params: {} 2024-04-19T23:25:41.728 DEBUG:boto:Token: None 2024-04-19T23:25:41.728 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:25:41 GMT /aktlji-13/?versions 2024-04-19T23:25:41.728 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qu8Sw0hnPDfqvkXFFGi90C2OVBE= 2024-04-19T23:25:41.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qu8Sw0hnPDfqvkXFFGi90C2OVBE=', 'Content-Length': '0'} 2024-04-19T23:25:41.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bea72e12a8e9db3e-006622fd75-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:41.731 DEBUG:boto:b'aktlji-13999false' 2024-04-19T23:25:41.731 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-13 zones={test-zone3, test-zone1} 2024-04-19T23:25:41.731 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:41.731 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:41.731 DEBUG:boto:Method: HEAD 2024-04-19T23:25:41.731 DEBUG:boto:Path: /aktlji-14/ 2024-04-19T23:25:41.732 DEBUG:boto:Data: 2024-04-19T23:25:41.732 DEBUG:boto:Headers: {} 2024-04-19T23:25:41.732 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:25:41.732 DEBUG:boto:Port: 8000 2024-04-19T23:25:41.732 DEBUG:boto:Params: {} 2024-04-19T23:25:41.732 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:25:41.732 DEBUG:boto:Token: None 2024-04-19T23:25:41.733 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:25:41 GMT /aktlji-14/ 2024-04-19T23:25:41.733 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jf3siKnJu4SqPbRU0bU86JPPBDY= 2024-04-19T23:25:41.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jf3siKnJu4SqPbRU0bU86JPPBDY=', 'Content-Length': '0'} 2024-04-19T23:25:41.737 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab3115901a3ba869-006622fd75-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:41.737 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:25:41.737 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:25:41.737 DEBUG:boto:Method: HEAD 2024-04-19T23:25:41.737 DEBUG:boto:Path: /aktlji-14/myobj 2024-04-19T23:25:41.737 DEBUG:boto:Data: 2024-04-19T23:25:41.737 DEBUG:boto:Headers: {} 2024-04-19T23:25:41.738 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:25:41.738 DEBUG:boto:Port: 8000 2024-04-19T23:25:41.738 DEBUG:boto:Params: {} 2024-04-19T23:25:41.738 DEBUG:boto:Token: None 2024-04-19T23:25:41.738 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:25:41 GMT /aktlji-14/myobj 2024-04-19T23:25:41.738 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mARub39KRJ1lchcAnjQa4o949Z4= 2024-04-19T23:25:41.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mARub39KRJ1lchcAnjQa4o949Z4=', 'Content-Length': '0'} 2024-04-19T23:25:41.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054700560bb238e3f-006622fd75-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:41.740 DEBUG:boto:path=/aktlji-14/myobj 2024-04-19T23:25:41.741 DEBUG:boto:auth_path=/aktlji-14/myobj 2024-04-19T23:25:41.741 DEBUG:boto:Method: DELETE 2024-04-19T23:25:41.741 DEBUG:boto:Path: /aktlji-14/myobj 2024-04-19T23:25:41.741 DEBUG:boto:Data: 2024-04-19T23:25:41.741 DEBUG:boto:Headers: {} 2024-04-19T23:25:41.741 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:25:41.741 DEBUG:boto:Port: 8000 2024-04-19T23:25:41.741 DEBUG:boto:Params: {} 2024-04-19T23:25:41.741 DEBUG:boto:Token: None 2024-04-19T23:25:41.741 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:25:41 GMT /aktlji-14/myobj 2024-04-19T23:25:41.742 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tZtQtQO1GXMSCxnes2zuLPEEyuM= 2024-04-19T23:25:41.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tZtQtQO1GXMSCxnes2zuLPEEyuM=', 'Content-Length': '0'} 2024-04-19T23:25:41.748 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a64e44d49fdb5ab3-006622fd75-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:41.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:25:41.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:25:41.749 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:25:41.815 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:25:41.815 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:25:41.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:41.863+0000 7fddc948b780 1 waiting for incremental sync to catch up: 2024-04-19T23:25:41.858 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2024-04-19T23:25:41.858 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-19T23:25:46.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:46.864+0000 7fddc948b780 1 waiting for incremental sync to catch up: 2024-04-19T23:25:46.860 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2024-04-19T23:25:46.860 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-19T23:25:51.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:51.865+0000 7fddc948b780 1 bucket sync caught up with source: 2024-04-19T23:25:51.861 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000002.7.6, , , , , , , , , ] 2024-04-19T23:25:51.861 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-19T23:25:51.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:51.865+0000 7fddc948b780 0 bucket checkpoint complete 2024-04-19T23:25:51.880 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-14 zones={test-zone1, test-zone2} 2024-04-19T23:25:51.880 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:51.880 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:51.880 DEBUG:boto:Method: HEAD 2024-04-19T23:25:51.881 DEBUG:boto:Path: /aktlji-14/ 2024-04-19T23:25:51.881 DEBUG:boto:Data: 2024-04-19T23:25:51.881 DEBUG:boto:Headers: {} 2024-04-19T23:25:51.881 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:25:51.881 DEBUG:boto:Port: 8000 2024-04-19T23:25:51.881 DEBUG:boto:Params: {} 2024-04-19T23:25:51.882 DEBUG:boto:Token: None 2024-04-19T23:25:51.882 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:25:51 GMT /aktlji-14/ 2024-04-19T23:25:51.882 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g2X0W1Lt7VzsIExeiYrAzi/6Wu4= 2024-04-19T23:25:51.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:51 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g2X0W1Lt7VzsIExeiYrAzi/6Wu4=', 'Content-Length': '0'} 2024-04-19T23:25:51.885 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026f0af896b66c57c-006622fd7f-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:51.885 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:51.885 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:51.886 DEBUG:boto:Method: HEAD 2024-04-19T23:25:51.886 DEBUG:boto:Path: /aktlji-14/ 2024-04-19T23:25:51.886 DEBUG:boto:Data: 2024-04-19T23:25:51.887 DEBUG:boto:Headers: {} 2024-04-19T23:25:51.887 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:25:51.887 DEBUG:boto:Port: 8000 2024-04-19T23:25:51.887 DEBUG:boto:Params: {} 2024-04-19T23:25:51.887 DEBUG:boto:Token: None 2024-04-19T23:25:51.887 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:25:51 GMT /aktlji-14/ 2024-04-19T23:25:51.887 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g2X0W1Lt7VzsIExeiYrAzi/6Wu4= 2024-04-19T23:25:51.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:51 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g2X0W1Lt7VzsIExeiYrAzi/6Wu4=', 'Content-Length': '0'} 2024-04-19T23:25:51.890 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002de0f244df00094d-006622fd7f-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:51.890 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:25:51.890 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:51.890 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:51.891 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:51.891 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:51.891 DEBUG:boto:Method: GET 2024-04-19T23:25:51.891 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:25:51.891 DEBUG:boto:Data: 2024-04-19T23:25:51.891 DEBUG:boto:Headers: {} 2024-04-19T23:25:51.891 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:25:51.891 DEBUG:boto:Port: 8000 2024-04-19T23:25:51.891 DEBUG:boto:Params: {} 2024-04-19T23:25:51.891 DEBUG:boto:Token: None 2024-04-19T23:25:51.892 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:25:51 GMT /aktlji-14/?versions 2024-04-19T23:25:51.892 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:em6sC+sWBFZiWCwhp/r1u9HNSkw= 2024-04-19T23:25:51.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:51 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:em6sC+sWBFZiWCwhp/r1u9HNSkw=', 'Content-Length': '0'} 2024-04-19T23:25:51.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003d53aa2b8a4600d-006622fd7f-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:51.894 DEBUG:boto:b'aktlji-14999false' 2024-04-19T23:25:51.895 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:25:51.895 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:51.895 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:51.895 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:51.895 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:51.895 DEBUG:boto:Method: GET 2024-04-19T23:25:51.895 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:25:51.895 DEBUG:boto:Data: 2024-04-19T23:25:51.895 DEBUG:boto:Headers: {} 2024-04-19T23:25:51.896 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:25:51.896 DEBUG:boto:Port: 8000 2024-04-19T23:25:51.896 DEBUG:boto:Params: {} 2024-04-19T23:25:51.896 DEBUG:boto:Token: None 2024-04-19T23:25:51.896 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:25:51 GMT /aktlji-14/?versions 2024-04-19T23:25:51.896 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:em6sC+sWBFZiWCwhp/r1u9HNSkw= 2024-04-19T23:25:51.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:51 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:em6sC+sWBFZiWCwhp/r1u9HNSkw=', 'Content-Length': '0'} 2024-04-19T23:25:51.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3d26952e6bb54c1-006622fd7f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:51.899 DEBUG:boto:b'aktlji-14999false' 2024-04-19T23:25:51.899 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:51.900 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:51.900 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:51.900 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:51.900 DEBUG:boto:Method: GET 2024-04-19T23:25:51.900 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:25:51.900 DEBUG:boto:Data: 2024-04-19T23:25:51.900 DEBUG:boto:Headers: {} 2024-04-19T23:25:51.900 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:25:51.900 DEBUG:boto:Port: 8000 2024-04-19T23:25:51.900 DEBUG:boto:Params: {} 2024-04-19T23:25:51.901 DEBUG:boto:Token: None 2024-04-19T23:25:51.901 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:25:51 GMT /aktlji-14/?versions 2024-04-19T23:25:51.901 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:em6sC+sWBFZiWCwhp/r1u9HNSkw= 2024-04-19T23:25:51.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:51 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:em6sC+sWBFZiWCwhp/r1u9HNSkw=', 'Content-Length': '0'} 2024-04-19T23:25:51.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a733ffd5f10fd9f-006622fd7f-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:51.903 DEBUG:boto:b'aktlji-14999false' 2024-04-19T23:25:51.903 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:51.904 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:51.904 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:51.904 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:51.904 DEBUG:boto:Method: GET 2024-04-19T23:25:51.904 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:25:51.904 DEBUG:boto:Data: 2024-04-19T23:25:51.904 DEBUG:boto:Headers: {} 2024-04-19T23:25:51.904 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:25:51.904 DEBUG:boto:Port: 8000 2024-04-19T23:25:51.904 DEBUG:boto:Params: {} 2024-04-19T23:25:51.905 DEBUG:boto:Token: None 2024-04-19T23:25:51.905 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:25:51 GMT /aktlji-14/?versions 2024-04-19T23:25:51.905 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:em6sC+sWBFZiWCwhp/r1u9HNSkw= 2024-04-19T23:25:51.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:51 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:em6sC+sWBFZiWCwhp/r1u9HNSkw=', 'Content-Length': '0'} 2024-04-19T23:25:51.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000168c7e22e928b7e5-006622fd7f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:51.907 DEBUG:boto:b'aktlji-14999false' 2024-04-19T23:25:51.907 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-14 zones={test-zone1, test-zone2} 2024-04-19T23:25:51.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:25:51.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:25:51.908 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:25:51.974 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:25:51.975 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:25:52.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:52.021+0000 7fb6188fd780 1 waiting for incremental sync to catch up: 2024-04-19T23:25:52.017 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2024-04-19T23:25:52.017 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-19T23:25:57.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:57.023+0000 7fb6188fd780 1 bucket sync caught up with source: 2024-04-19T23:25:57.018 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000002.7.6, , , , , , , , , ] 2024-04-19T23:25:57.019 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-19T23:25:57.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:57.023+0000 7fb6188fd780 0 bucket checkpoint complete 2024-04-19T23:25:57.035 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-14 zones={test-zone3, test-zone2} 2024-04-19T23:25:57.035 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:57.035 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:57.036 DEBUG:boto:Method: HEAD 2024-04-19T23:25:57.036 DEBUG:boto:Path: /aktlji-14/ 2024-04-19T23:25:57.036 DEBUG:boto:Data: 2024-04-19T23:25:57.036 DEBUG:boto:Headers: {} 2024-04-19T23:25:57.036 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:25:57.036 DEBUG:boto:Port: 8001 2024-04-19T23:25:57.036 DEBUG:boto:Params: {} 2024-04-19T23:25:57.036 DEBUG:boto:Token: None 2024-04-19T23:25:57.037 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:25:57 GMT /aktlji-14/ 2024-04-19T23:25:57.037 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wHSioN2ghPLi41th/sxzFeXvzrk= 2024-04-19T23:25:57.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wHSioN2ghPLi41th/sxzFeXvzrk=', 'Content-Length': '0'} 2024-04-19T23:25:57.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009adc35ba732861cc-006622fd85-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:25:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:57.039 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:57.040 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:57.040 DEBUG:boto:Method: HEAD 2024-04-19T23:25:57.040 DEBUG:boto:Path: /aktlji-14/ 2024-04-19T23:25:57.040 DEBUG:boto:Data: 2024-04-19T23:25:57.040 DEBUG:boto:Headers: {} 2024-04-19T23:25:57.040 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:25:57.040 DEBUG:boto:Port: 8000 2024-04-19T23:25:57.040 DEBUG:boto:Params: {} 2024-04-19T23:25:57.040 DEBUG:boto:Token: None 2024-04-19T23:25:57.040 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:25:57 GMT /aktlji-14/ 2024-04-19T23:25:57.041 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wHSioN2ghPLi41th/sxzFeXvzrk= 2024-04-19T23:25:57.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wHSioN2ghPLi41th/sxzFeXvzrk=', 'Content-Length': '0'} 2024-04-19T23:25:57.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa5a5de07c2bc052-006622fd85-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:25:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:57.043 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:25:57.043 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:57.043 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:57.043 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:57.044 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:57.044 DEBUG:boto:Method: GET 2024-04-19T23:25:57.044 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:25:57.044 DEBUG:boto:Data: 2024-04-19T23:25:57.044 DEBUG:boto:Headers: {} 2024-04-19T23:25:57.044 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:25:57.044 DEBUG:boto:Port: 8001 2024-04-19T23:25:57.044 DEBUG:boto:Params: {} 2024-04-19T23:25:57.044 DEBUG:boto:Token: None 2024-04-19T23:25:57.044 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:25:57 GMT /aktlji-14/?versions 2024-04-19T23:25:57.044 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LkqDgYoKz8P9i+1rZ93jIMEY9TU= 2024-04-19T23:25:57.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LkqDgYoKz8P9i+1rZ93jIMEY9TU=', 'Content-Length': '0'} 2024-04-19T23:25:57.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7a9a4f5c67608a1-006622fd85-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:25:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:57.047 DEBUG:boto:b'aktlji-14999false' 2024-04-19T23:25:57.047 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:25:57.047 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:57.048 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:57.048 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:57.048 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:57.048 DEBUG:boto:Method: GET 2024-04-19T23:25:57.048 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:25:57.048 DEBUG:boto:Data: 2024-04-19T23:25:57.048 DEBUG:boto:Headers: {} 2024-04-19T23:25:57.048 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:25:57.048 DEBUG:boto:Port: 8000 2024-04-19T23:25:57.048 DEBUG:boto:Params: {} 2024-04-19T23:25:57.049 DEBUG:boto:Token: None 2024-04-19T23:25:57.049 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:25:57 GMT /aktlji-14/?versions 2024-04-19T23:25:57.049 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LkqDgYoKz8P9i+1rZ93jIMEY9TU= 2024-04-19T23:25:57.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LkqDgYoKz8P9i+1rZ93jIMEY9TU=', 'Content-Length': '0'} 2024-04-19T23:25:57.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2b80f92792fd011-006622fd85-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:25:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:57.051 DEBUG:boto:b'aktlji-14999false' 2024-04-19T23:25:57.052 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:57.052 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:57.052 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:57.052 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:57.052 DEBUG:boto:Method: GET 2024-04-19T23:25:57.052 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:25:57.052 DEBUG:boto:Data: 2024-04-19T23:25:57.052 DEBUG:boto:Headers: {} 2024-04-19T23:25:57.053 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:25:57.053 DEBUG:boto:Port: 8001 2024-04-19T23:25:57.053 DEBUG:boto:Params: {} 2024-04-19T23:25:57.053 DEBUG:boto:Token: None 2024-04-19T23:25:57.053 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:25:57 GMT /aktlji-14/?versions 2024-04-19T23:25:57.053 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LkqDgYoKz8P9i+1rZ93jIMEY9TU= 2024-04-19T23:25:57.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LkqDgYoKz8P9i+1rZ93jIMEY9TU=', 'Content-Length': '0'} 2024-04-19T23:25:57.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7d18acb5c9c11a8-006622fd85-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:25:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:57.056 DEBUG:boto:b'aktlji-14999false' 2024-04-19T23:25:57.056 DEBUG:boto:path=/aktlji-14/ 2024-04-19T23:25:57.056 DEBUG:boto:auth_path=/aktlji-14/ 2024-04-19T23:25:57.056 DEBUG:boto:path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:57.056 DEBUG:boto:auth_path=/aktlji-14/?versions&max-keys=999 2024-04-19T23:25:57.056 DEBUG:boto:Method: GET 2024-04-19T23:25:57.056 DEBUG:boto:Path: /aktlji-14/?versions&max-keys=999 2024-04-19T23:25:57.056 DEBUG:boto:Data: 2024-04-19T23:25:57.057 DEBUG:boto:Headers: {} 2024-04-19T23:25:57.057 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:25:57.057 DEBUG:boto:Port: 8000 2024-04-19T23:25:57.057 DEBUG:boto:Params: {} 2024-04-19T23:25:57.057 DEBUG:boto:Token: None 2024-04-19T23:25:57.057 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:25:57 GMT /aktlji-14/?versions 2024-04-19T23:25:57.057 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LkqDgYoKz8P9i+1rZ93jIMEY9TU= 2024-04-19T23:25:57.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LkqDgYoKz8P9i+1rZ93jIMEY9TU=', 'Content-Length': '0'} 2024-04-19T23:25:57.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d85798145c27e1d3-006622fd85-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:25:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:57.102 DEBUG:boto:b'aktlji-14999false' 2024-04-19T23:25:57.103 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-14 zones={test-zone3, test-zone2} 2024-04-19T23:25:57.103 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:25:57.103 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:25:57.103 DEBUG:boto:Method: HEAD 2024-04-19T23:25:57.103 DEBUG:boto:Path: /aktlji-15/ 2024-04-19T23:25:57.103 DEBUG:boto:Data: 2024-04-19T23:25:57.103 DEBUG:boto:Headers: {} 2024-04-19T23:25:57.103 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:25:57.103 DEBUG:boto:Port: 8001 2024-04-19T23:25:57.104 DEBUG:boto:Params: {} 2024-04-19T23:25:57.104 DEBUG:boto:Token: None 2024-04-19T23:25:57.104 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:25:57 GMT /aktlji-15/ 2024-04-19T23:25:57.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jSTN6HRdEcNE8MwRd8HNDL8vztI= 2024-04-19T23:25:57.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jSTN6HRdEcNE8MwRd8HNDL8vztI=', 'Content-Length': '0'} 2024-04-19T23:25:57.106 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000155f82730d5ce5d0-006622fd85-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:25:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:57.107 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:25:57.107 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:25:57.107 DEBUG:boto:Method: HEAD 2024-04-19T23:25:57.107 DEBUG:boto:Path: /aktlji-15/myobj 2024-04-19T23:25:57.107 DEBUG:boto:Data: 2024-04-19T23:25:57.107 DEBUG:boto:Headers: {} 2024-04-19T23:25:57.107 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:25:57.107 DEBUG:boto:Port: 8001 2024-04-19T23:25:57.108 DEBUG:boto:Params: {} 2024-04-19T23:25:57.108 DEBUG:boto:Token: None 2024-04-19T23:25:57.108 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:25:57 GMT /aktlji-15/myobj 2024-04-19T23:25:57.108 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VeWveb5tcm+RZaBRSk6v7GR8QFY= 2024-04-19T23:25:57.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VeWveb5tcm+RZaBRSk6v7GR8QFY=', 'Content-Length': '0'} 2024-04-19T23:25:57.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af94f1ecf8dcf6cc-006622fd85-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:25:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:57.110 DEBUG:boto:path=/aktlji-15/myobj 2024-04-19T23:25:57.111 DEBUG:boto:auth_path=/aktlji-15/myobj 2024-04-19T23:25:57.111 DEBUG:boto:Method: DELETE 2024-04-19T23:25:57.111 DEBUG:boto:Path: /aktlji-15/myobj 2024-04-19T23:25:57.111 DEBUG:boto:Data: 2024-04-19T23:25:57.111 DEBUG:boto:Headers: {} 2024-04-19T23:25:57.111 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:25:57.111 DEBUG:boto:Port: 8001 2024-04-19T23:25:57.111 DEBUG:boto:Params: {} 2024-04-19T23:25:57.111 DEBUG:boto:Token: None 2024-04-19T23:25:57.111 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:25:57 GMT /aktlji-15/myobj 2024-04-19T23:25:57.112 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:v0xDh+s/aV1YJ2X776FK1+VJDBE= 2024-04-19T23:25:57.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:25:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:v0xDh+s/aV1YJ2X776FK1+VJDBE=', 'Content-Length': '0'} 2024-04-19T23:25:57.118 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000072d9cc19a667c9eb-006622fd85-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:25:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:25:57.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:25:57.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:25:57.119 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:25:57.186 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:25:57.186 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:25:57.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:25:57.234+0000 7f29375ac780 1 waiting for incremental sync to catch up: 2024-04-19T23:25:57.229 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2024-04-19T23:25:57.229 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.24.6, , , , , , , , , ] 2024-04-19T23:26:02.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:26:02.235+0000 7f29375ac780 1 waiting for incremental sync to catch up: 2024-04-19T23:26:02.231 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2024-04-19T23:26:02.231 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.24.6, , , , , , , , , ] 2024-04-19T23:26:07.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:26:07.237+0000 7f29375ac780 1 waiting for incremental sync to catch up: 2024-04-19T23:26:07.232 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.18.6, , , , , , , , , ] 2024-04-19T23:26:07.232 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.24.6, , , , , , , , , ] 2024-04-19T23:26:12.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:26:12.238+0000 7f29375ac780 1 bucket sync caught up with source: 2024-04-19T23:26:12.233 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000002.24.6, , , , , , , , , ] 2024-04-19T23:26:12.233 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.24.6, , , , , , , , , ] 2024-04-19T23:26:12.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:26:12.238+0000 7f29375ac780 0 bucket checkpoint complete 2024-04-19T23:26:12.250 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-15 zones={test-zone1, test-zone3} 2024-04-19T23:26:12.251 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:26:12.251 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:26:12.251 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.251 DEBUG:boto:Path: /aktlji-15/ 2024-04-19T23:26:12.251 DEBUG:boto:Data: 2024-04-19T23:26:12.251 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.251 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.252 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.252 DEBUG:boto:Params: {} 2024-04-19T23:26:12.252 DEBUG:boto:Token: None 2024-04-19T23:26:12.252 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-15/ 2024-04-19T23:26:12.252 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:E3T04wlmkuKuU/YmPeA8dNlzor0= 2024-04-19T23:26:12.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E3T04wlmkuKuU/YmPeA8dNlzor0=', 'Content-Length': '0'} 2024-04-19T23:26:12.255 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003418f4d8d4828a76-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.255 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:26:12.255 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:26:12.255 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.255 DEBUG:boto:Path: /aktlji-15/ 2024-04-19T23:26:12.255 DEBUG:boto:Data: 2024-04-19T23:26:12.256 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.256 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:12.256 DEBUG:boto:Port: 8001 2024-04-19T23:26:12.256 DEBUG:boto:Params: {} 2024-04-19T23:26:12.256 DEBUG:boto:Token: None 2024-04-19T23:26:12.256 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-15/ 2024-04-19T23:26:12.256 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:E3T04wlmkuKuU/YmPeA8dNlzor0= 2024-04-19T23:26:12.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E3T04wlmkuKuU/YmPeA8dNlzor0=', 'Content-Length': '0'} 2024-04-19T23:26:12.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000294d9fa1a92076d8-006622fd94-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.259 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:26:12.259 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:26:12.259 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:26:12.259 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.259 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.259 DEBUG:boto:Method: GET 2024-04-19T23:26:12.259 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.260 DEBUG:boto:Data: 2024-04-19T23:26:12.260 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.260 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.260 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.260 DEBUG:boto:Params: {} 2024-04-19T23:26:12.260 DEBUG:boto:Token: None 2024-04-19T23:26:12.260 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:12 GMT /aktlji-15/?versions 2024-04-19T23:26:12.260 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw= 2024-04-19T23:26:12.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw=', 'Content-Length': '0'} 2024-04-19T23:26:12.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2fa97df182a5bf2-006622fd94-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.263 DEBUG:boto:b'aktlji-15999false' 2024-04-19T23:26:12.263 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:26:12.263 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:26:12.263 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:26:12.264 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.264 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.264 DEBUG:boto:Method: GET 2024-04-19T23:26:12.264 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.264 DEBUG:boto:Data: 2024-04-19T23:26:12.264 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.264 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:12.264 DEBUG:boto:Port: 8001 2024-04-19T23:26:12.264 DEBUG:boto:Params: {} 2024-04-19T23:26:12.264 DEBUG:boto:Token: None 2024-04-19T23:26:12.265 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:12 GMT /aktlji-15/?versions 2024-04-19T23:26:12.265 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw= 2024-04-19T23:26:12.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw=', 'Content-Length': '0'} 2024-04-19T23:26:12.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a51063dd99d9ed07-006622fd94-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.267 DEBUG:boto:b'aktlji-15999false' 2024-04-19T23:26:12.267 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:26:12.267 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:26:12.267 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.268 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.268 DEBUG:boto:Method: GET 2024-04-19T23:26:12.268 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.268 DEBUG:boto:Data: 2024-04-19T23:26:12.268 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.268 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.268 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.268 DEBUG:boto:Params: {} 2024-04-19T23:26:12.268 DEBUG:boto:Token: None 2024-04-19T23:26:12.268 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:12 GMT /aktlji-15/?versions 2024-04-19T23:26:12.269 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw= 2024-04-19T23:26:12.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw=', 'Content-Length': '0'} 2024-04-19T23:26:12.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c86b75f3ec6f5299-006622fd94-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.271 DEBUG:boto:b'aktlji-15999false' 2024-04-19T23:26:12.271 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:26:12.271 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:26:12.271 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.271 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.272 DEBUG:boto:Method: GET 2024-04-19T23:26:12.272 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.272 DEBUG:boto:Data: 2024-04-19T23:26:12.272 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.272 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:12.272 DEBUG:boto:Port: 8001 2024-04-19T23:26:12.272 DEBUG:boto:Params: {} 2024-04-19T23:26:12.272 DEBUG:boto:Token: None 2024-04-19T23:26:12.272 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:12 GMT /aktlji-15/?versions 2024-04-19T23:26:12.273 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw= 2024-04-19T23:26:12.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw=', 'Content-Length': '0'} 2024-04-19T23:26:12.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cce4998134a4c4f1-006622fd94-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.275 DEBUG:boto:b'aktlji-15999false' 2024-04-19T23:26:12.275 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-15 zones={test-zone1, test-zone3} 2024-04-19T23:26:12.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:26:12.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:26:12.276 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:26:12.342 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:26:12.342 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:26:12.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:26:12.388+0000 7fe706d32780 1 bucket sync caught up with source: 2024-04-19T23:26:12.384 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000002.24.6, , , , , , , , , ] 2024-04-19T23:26:12.384 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000002.24.6, , , , , , , , , ] 2024-04-19T23:26:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:26:12.388+0000 7fe706d32780 0 bucket checkpoint complete 2024-04-19T23:26:12.399 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-15 zones={test-zone2, test-zone3} 2024-04-19T23:26:12.399 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:26:12.399 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:26:12.399 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.400 DEBUG:boto:Path: /aktlji-15/ 2024-04-19T23:26:12.400 DEBUG:boto:Data: 2024-04-19T23:26:12.400 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.400 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:12.400 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.400 DEBUG:boto:Params: {} 2024-04-19T23:26:12.400 DEBUG:boto:Token: None 2024-04-19T23:26:12.400 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-15/ 2024-04-19T23:26:12.401 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:E3T04wlmkuKuU/YmPeA8dNlzor0= 2024-04-19T23:26:12.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E3T04wlmkuKuU/YmPeA8dNlzor0=', 'Content-Length': '0'} 2024-04-19T23:26:12.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfa25567b6a45df1-006622fd94-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.403 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:26:12.403 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:26:12.403 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.404 DEBUG:boto:Path: /aktlji-15/ 2024-04-19T23:26:12.404 DEBUG:boto:Data: 2024-04-19T23:26:12.404 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.404 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:12.404 DEBUG:boto:Port: 8001 2024-04-19T23:26:12.404 DEBUG:boto:Params: {} 2024-04-19T23:26:12.404 DEBUG:boto:Token: None 2024-04-19T23:26:12.404 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-15/ 2024-04-19T23:26:12.404 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:E3T04wlmkuKuU/YmPeA8dNlzor0= 2024-04-19T23:26:12.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E3T04wlmkuKuU/YmPeA8dNlzor0=', 'Content-Length': '0'} 2024-04-19T23:26:12.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9ae03b2b8347664-006622fd94-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.407 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:26:12.407 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:26:12.407 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:26:12.407 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.408 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.408 DEBUG:boto:Method: GET 2024-04-19T23:26:12.408 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.408 DEBUG:boto:Data: 2024-04-19T23:26:12.408 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.408 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:12.408 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.408 DEBUG:boto:Params: {} 2024-04-19T23:26:12.408 DEBUG:boto:Token: None 2024-04-19T23:26:12.409 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:12 GMT /aktlji-15/?versions 2024-04-19T23:26:12.409 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw= 2024-04-19T23:26:12.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw=', 'Content-Length': '0'} 2024-04-19T23:26:12.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8c55f8f68f6d8b2-006622fd94-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.411 DEBUG:boto:b'aktlji-15999false' 2024-04-19T23:26:12.412 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:26:12.412 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:26:12.412 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:26:12.412 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.412 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.412 DEBUG:boto:Method: GET 2024-04-19T23:26:12.412 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.412 DEBUG:boto:Data: 2024-04-19T23:26:12.412 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.412 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:12.413 DEBUG:boto:Port: 8001 2024-04-19T23:26:12.413 DEBUG:boto:Params: {} 2024-04-19T23:26:12.413 DEBUG:boto:Token: None 2024-04-19T23:26:12.413 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:12 GMT /aktlji-15/?versions 2024-04-19T23:26:12.413 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw= 2024-04-19T23:26:12.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw=', 'Content-Length': '0'} 2024-04-19T23:26:12.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e744582552e0290c-006622fd94-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.458 DEBUG:boto:b'aktlji-15999false' 2024-04-19T23:26:12.459 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:26:12.459 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:26:12.459 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.459 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.459 DEBUG:boto:Method: GET 2024-04-19T23:26:12.459 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.459 DEBUG:boto:Data: 2024-04-19T23:26:12.459 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.459 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:12.459 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.460 DEBUG:boto:Params: {} 2024-04-19T23:26:12.460 DEBUG:boto:Token: None 2024-04-19T23:26:12.460 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:12 GMT /aktlji-15/?versions 2024-04-19T23:26:12.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw= 2024-04-19T23:26:12.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw=', 'Content-Length': '0'} 2024-04-19T23:26:12.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000240349239760f6ff-006622fd94-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.463 DEBUG:boto:b'aktlji-15999false' 2024-04-19T23:26:12.463 DEBUG:boto:path=/aktlji-15/ 2024-04-19T23:26:12.463 DEBUG:boto:auth_path=/aktlji-15/ 2024-04-19T23:26:12.463 DEBUG:boto:path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.463 DEBUG:boto:auth_path=/aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.463 DEBUG:boto:Method: GET 2024-04-19T23:26:12.464 DEBUG:boto:Path: /aktlji-15/?versions&max-keys=999 2024-04-19T23:26:12.464 DEBUG:boto:Data: 2024-04-19T23:26:12.464 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.464 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:12.464 DEBUG:boto:Port: 8001 2024-04-19T23:26:12.464 DEBUG:boto:Params: {} 2024-04-19T23:26:12.464 DEBUG:boto:Token: None 2024-04-19T23:26:12.464 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:12 GMT /aktlji-15/?versions 2024-04-19T23:26:12.464 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw= 2024-04-19T23:26:12.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TfSu+PzfvAsDYgxnX/m267iY6hw=', 'Content-Length': '0'} 2024-04-19T23:26:12.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c4984fc8c4e679f-006622fd94-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.467 DEBUG:boto:b'aktlji-15999false' 2024-04-19T23:26:12.467 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-15 zones={test-zone2, test-zone3} 2024-04-19T23:26:12.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2024-04-19T23:26:12.469 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-16 2024-04-19T23:26:12.469 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.469 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.469 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.469 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.469 DEBUG:boto:Data: 2024-04-19T23:26:12.470 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.470 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.470 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.470 DEBUG:boto:Params: {} 2024-04-19T23:26:12.470 DEBUG:boto:Token: None 2024-04-19T23:26:12.470 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.470 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Fdy5qym9tKkRbJ1L+Q6B5u/LOsk= 2024-04-19T23:26:12.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Fdy5qym9tKkRbJ1L+Q6B5u/LOsk=', 'Content-Length': '0'} 2024-04-19T23:26:12.489 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006d87aa821579882b-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.489 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-17 2024-04-19T23:26:12.489 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:12.489 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:12.489 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.489 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:12.489 DEBUG:boto:Data: 2024-04-19T23:26:12.489 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.490 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:12.490 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.490 DEBUG:boto:Params: {} 2024-04-19T23:26:12.490 DEBUG:boto:Token: None 2024-04-19T23:26:12.490 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:26:12 GMT /aktlji-17/ 2024-04-19T23:26:12.490 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9gwSrDQDB+HVMdc52KIdsDB3uXw= 2024-04-19T23:26:12.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9gwSrDQDB+HVMdc52KIdsDB3uXw=', 'Content-Length': '0'} 2024-04-19T23:26:12.522 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009da37aeeb92c4353-006622fd94-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.523 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-18 2024-04-19T23:26:12.523 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:12.523 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:12.523 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.523 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:12.523 DEBUG:boto:Data: 2024-04-19T23:26:12.523 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.523 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:12.523 DEBUG:boto:Port: 8001 2024-04-19T23:26:12.523 DEBUG:boto:Params: {} 2024-04-19T23:26:12.524 DEBUG:boto:Token: None 2024-04-19T23:26:12.524 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:26:12 GMT /aktlji-18/ 2024-04-19T23:26:12.524 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cj3QUeeRUqB8FBww8AxMcALs68g= 2024-04-19T23:26:12.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cj3QUeeRUqB8FBww8AxMcALs68g=', 'Content-Length': '0'} 2024-04-19T23:26:12.553 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001a6cdd7e58d01ed2-006622fd94-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.553 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.553 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.553 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.553 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.554 DEBUG:boto:Data: 2024-04-19T23:26:12.554 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.554 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.554 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.554 DEBUG:boto:Params: {} 2024-04-19T23:26:12.554 DEBUG:boto:Token: None 2024-04-19T23:26:12.554 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.554 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.557 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f30de14e5c648e5-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.557 DEBUG:boto:path=/aktlji-16/obj1 2024-04-19T23:26:12.557 DEBUG:boto:auth_path=/aktlji-16/obj1 2024-04-19T23:26:12.558 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.558 DEBUG:boto:Path: /aktlji-16/obj1 2024-04-19T23:26:12.558 DEBUG:boto:Data: 2024-04-19T23:26:12.558 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.558 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.558 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.558 DEBUG:boto:Params: {} 2024-04-19T23:26:12.558 DEBUG:boto:Token: None 2024-04-19T23:26:12.558 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj1 2024-04-19T23:26:12.558 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NAwmiyIP1jRPTqlIdlYvGm5OUck= 2024-04-19T23:26:12.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NAwmiyIP1jRPTqlIdlYvGm5OUck='} 2024-04-19T23:26:12.565 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a858f89ab0f1a607-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.565 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.566 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.566 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.566 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.566 DEBUG:boto:Data: 2024-04-19T23:26:12.566 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.566 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.566 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.566 DEBUG:boto:Params: {} 2024-04-19T23:26:12.566 DEBUG:boto:Token: None 2024-04-19T23:26:12.566 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.567 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.569 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da2a575b3af48338-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.569 DEBUG:boto:path=/aktlji-16/obj2 2024-04-19T23:26:12.569 DEBUG:boto:auth_path=/aktlji-16/obj2 2024-04-19T23:26:12.569 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.569 DEBUG:boto:Path: /aktlji-16/obj2 2024-04-19T23:26:12.570 DEBUG:boto:Data: 2024-04-19T23:26:12.570 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.570 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.570 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.570 DEBUG:boto:Params: {} 2024-04-19T23:26:12.570 DEBUG:boto:Token: None 2024-04-19T23:26:12.570 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj2 2024-04-19T23:26:12.570 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5X0whiAt2Nj1qvB71b3I8NFjpaE= 2024-04-19T23:26:12.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5X0whiAt2Nj1qvB71b3I8NFjpaE='} 2024-04-19T23:26:12.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d86642fae1e9a58-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.619 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.619 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.619 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.619 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.619 DEBUG:boto:Data: 2024-04-19T23:26:12.619 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.619 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.619 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.619 DEBUG:boto:Params: {} 2024-04-19T23:26:12.619 DEBUG:boto:Token: None 2024-04-19T23:26:12.620 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.620 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021524f144042bd41-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.622 DEBUG:boto:path=/aktlji-16/obj3 2024-04-19T23:26:12.622 DEBUG:boto:auth_path=/aktlji-16/obj3 2024-04-19T23:26:12.623 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.623 DEBUG:boto:Path: /aktlji-16/obj3 2024-04-19T23:26:12.623 DEBUG:boto:Data: 2024-04-19T23:26:12.623 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.623 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.623 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.623 DEBUG:boto:Params: {} 2024-04-19T23:26:12.623 DEBUG:boto:Token: None 2024-04-19T23:26:12.623 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj3 2024-04-19T23:26:12.623 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PbmNrvau/Xv2tKmWI2aj1+msVQM= 2024-04-19T23:26:12.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PbmNrvau/Xv2tKmWI2aj1+msVQM='} 2024-04-19T23:26:12.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4bb612760e4b69d-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.630 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.630 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.630 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.630 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.630 DEBUG:boto:Data: 2024-04-19T23:26:12.630 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.630 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.630 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.630 DEBUG:boto:Params: {} 2024-04-19T23:26:12.631 DEBUG:boto:Token: None 2024-04-19T23:26:12.631 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.631 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a53e547da9435b76-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.633 DEBUG:boto:path=/aktlji-16/obj4 2024-04-19T23:26:12.633 DEBUG:boto:auth_path=/aktlji-16/obj4 2024-04-19T23:26:12.634 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.634 DEBUG:boto:Path: /aktlji-16/obj4 2024-04-19T23:26:12.634 DEBUG:boto:Data: 2024-04-19T23:26:12.634 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.634 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.634 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.634 DEBUG:boto:Params: {} 2024-04-19T23:26:12.634 DEBUG:boto:Token: None 2024-04-19T23:26:12.634 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj4 2024-04-19T23:26:12.635 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VueLeQX6Yogg4LNN9ZjogZzZbAk= 2024-04-19T23:26:12.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VueLeQX6Yogg4LNN9ZjogZzZbAk='} 2024-04-19T23:26:12.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d7d4cad04770dae-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.641 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.641 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.641 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.641 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.641 DEBUG:boto:Data: 2024-04-19T23:26:12.642 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.642 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.642 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.642 DEBUG:boto:Params: {} 2024-04-19T23:26:12.642 DEBUG:boto:Token: None 2024-04-19T23:26:12.642 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.642 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.644 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000176abfc364fcd2ae-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.644 DEBUG:boto:path=/aktlji-16/obj5 2024-04-19T23:26:12.645 DEBUG:boto:auth_path=/aktlji-16/obj5 2024-04-19T23:26:12.645 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.645 DEBUG:boto:Path: /aktlji-16/obj5 2024-04-19T23:26:12.645 DEBUG:boto:Data: 2024-04-19T23:26:12.645 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.645 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.645 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.645 DEBUG:boto:Params: {} 2024-04-19T23:26:12.645 DEBUG:boto:Token: None 2024-04-19T23:26:12.646 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj5 2024-04-19T23:26:12.646 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kdzaMR2GZaOywdOSYyKYbIeFV0c= 2024-04-19T23:26:12.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kdzaMR2GZaOywdOSYyKYbIeFV0c='} 2024-04-19T23:26:12.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006959d6d0ece6bfb8-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.691 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.691 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.691 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.691 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.691 DEBUG:boto:Data: 2024-04-19T23:26:12.691 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.691 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.691 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.691 DEBUG:boto:Params: {} 2024-04-19T23:26:12.691 DEBUG:boto:Token: None 2024-04-19T23:26:12.692 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.692 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.695 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e4e5a317ace199a-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.695 DEBUG:boto:path=/aktlji-16/obj6 2024-04-19T23:26:12.695 DEBUG:boto:auth_path=/aktlji-16/obj6 2024-04-19T23:26:12.695 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.695 DEBUG:boto:Path: /aktlji-16/obj6 2024-04-19T23:26:12.695 DEBUG:boto:Data: 2024-04-19T23:26:12.696 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.696 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.696 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.696 DEBUG:boto:Params: {} 2024-04-19T23:26:12.696 DEBUG:boto:Token: None 2024-04-19T23:26:12.696 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj6 2024-04-19T23:26:12.696 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:r5WpE3zObe4vLVY4E/Uu9e800wA= 2024-04-19T23:26:12.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r5WpE3zObe4vLVY4E/Uu9e800wA='} 2024-04-19T23:26:12.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000160a111885a8d2d6-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.708 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.708 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.708 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.708 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.708 DEBUG:boto:Data: 2024-04-19T23:26:12.709 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.709 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.709 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.709 DEBUG:boto:Params: {} 2024-04-19T23:26:12.709 DEBUG:boto:Token: None 2024-04-19T23:26:12.709 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.709 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9df2f260fa7728a-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.712 DEBUG:boto:path=/aktlji-16/obj7 2024-04-19T23:26:12.712 DEBUG:boto:auth_path=/aktlji-16/obj7 2024-04-19T23:26:12.712 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.713 DEBUG:boto:Path: /aktlji-16/obj7 2024-04-19T23:26:12.713 DEBUG:boto:Data: 2024-04-19T23:26:12.713 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.713 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.713 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.713 DEBUG:boto:Params: {} 2024-04-19T23:26:12.713 DEBUG:boto:Token: None 2024-04-19T23:26:12.713 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj7 2024-04-19T23:26:12.713 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OWwN2FtAwf9NTCQleVyYeeKe0C4= 2024-04-19T23:26:12.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OWwN2FtAwf9NTCQleVyYeeKe0C4='} 2024-04-19T23:26:12.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001bc0ef2bc02ad06-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.763 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.763 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.763 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.763 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.763 DEBUG:boto:Data: 2024-04-19T23:26:12.763 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.763 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.764 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.764 DEBUG:boto:Params: {} 2024-04-19T23:26:12.764 DEBUG:boto:Token: None 2024-04-19T23:26:12.764 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.764 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.767 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3e68fe6fc4c9a16-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.767 DEBUG:boto:path=/aktlji-16/obj8 2024-04-19T23:26:12.767 DEBUG:boto:auth_path=/aktlji-16/obj8 2024-04-19T23:26:12.767 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.767 DEBUG:boto:Path: /aktlji-16/obj8 2024-04-19T23:26:12.767 DEBUG:boto:Data: 2024-04-19T23:26:12.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.767 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.768 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.768 DEBUG:boto:Params: {} 2024-04-19T23:26:12.768 DEBUG:boto:Token: None 2024-04-19T23:26:12.768 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj8 2024-04-19T23:26:12.768 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NU+IJJz4lB0TaqDW2RcGOscElVc= 2024-04-19T23:26:12.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NU+IJJz4lB0TaqDW2RcGOscElVc='} 2024-04-19T23:26:12.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f97a7767c89d6df-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.773 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.773 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.773 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.774 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.774 DEBUG:boto:Data: 2024-04-19T23:26:12.774 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.774 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.774 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.774 DEBUG:boto:Params: {} 2024-04-19T23:26:12.774 DEBUG:boto:Token: None 2024-04-19T23:26:12.774 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.775 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aba5fd520e9eecf5-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.777 DEBUG:boto:path=/aktlji-16/obj9 2024-04-19T23:26:12.777 DEBUG:boto:auth_path=/aktlji-16/obj9 2024-04-19T23:26:12.777 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.777 DEBUG:boto:Path: /aktlji-16/obj9 2024-04-19T23:26:12.777 DEBUG:boto:Data: 2024-04-19T23:26:12.777 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.777 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.778 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.778 DEBUG:boto:Params: {} 2024-04-19T23:26:12.778 DEBUG:boto:Token: None 2024-04-19T23:26:12.778 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj9 2024-04-19T23:26:12.778 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cKE5TKoykuRyOp49vUq6kllG168= 2024-04-19T23:26:12.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cKE5TKoykuRyOp49vUq6kllG168='} 2024-04-19T23:26:12.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009cc4080120c58147-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.823 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.824 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.824 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.824 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.824 DEBUG:boto:Data: 2024-04-19T23:26:12.824 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.824 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.824 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.824 DEBUG:boto:Params: {} 2024-04-19T23:26:12.824 DEBUG:boto:Token: None 2024-04-19T23:26:12.825 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.825 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8f2edf84602c4d3-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.827 DEBUG:boto:path=/aktlji-16/obj10 2024-04-19T23:26:12.827 DEBUG:boto:auth_path=/aktlji-16/obj10 2024-04-19T23:26:12.828 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.828 DEBUG:boto:Path: /aktlji-16/obj10 2024-04-19T23:26:12.828 DEBUG:boto:Data: 2024-04-19T23:26:12.828 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.828 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.828 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.828 DEBUG:boto:Params: {} 2024-04-19T23:26:12.828 DEBUG:boto:Token: None 2024-04-19T23:26:12.828 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj10 2024-04-19T23:26:12.829 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sk1lYCzJ5Vx3tum7LNsGxRbHWXM= 2024-04-19T23:26:12.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sk1lYCzJ5Vx3tum7LNsGxRbHWXM='} 2024-04-19T23:26:12.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003fed8bd8893b3c99-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.836 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.836 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.837 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.837 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.837 DEBUG:boto:Data: 2024-04-19T23:26:12.837 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.837 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.837 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.837 DEBUG:boto:Params: {} 2024-04-19T23:26:12.837 DEBUG:boto:Token: None 2024-04-19T23:26:12.837 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.838 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d88e5de0f334b4a8-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.840 DEBUG:boto:path=/aktlji-16/obj11 2024-04-19T23:26:12.840 DEBUG:boto:auth_path=/aktlji-16/obj11 2024-04-19T23:26:12.841 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.841 DEBUG:boto:Path: /aktlji-16/obj11 2024-04-19T23:26:12.841 DEBUG:boto:Data: 2024-04-19T23:26:12.841 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.841 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.841 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.841 DEBUG:boto:Params: {} 2024-04-19T23:26:12.841 DEBUG:boto:Token: None 2024-04-19T23:26:12.841 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj11 2024-04-19T23:26:12.842 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0qZneRnU/sdUjrHty05q3q6Oz4Q= 2024-04-19T23:26:12.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0qZneRnU/sdUjrHty05q3q6Oz4Q='} 2024-04-19T23:26:12.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0724c873e36dbf8-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.887 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.887 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.887 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.887 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.887 DEBUG:boto:Data: 2024-04-19T23:26:12.887 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.887 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.887 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.887 DEBUG:boto:Params: {} 2024-04-19T23:26:12.887 DEBUG:boto:Token: None 2024-04-19T23:26:12.888 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.888 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.890 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5267516ab0bb6b7-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.890 DEBUG:boto:path=/aktlji-16/obj12 2024-04-19T23:26:12.890 DEBUG:boto:auth_path=/aktlji-16/obj12 2024-04-19T23:26:12.891 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.891 DEBUG:boto:Path: /aktlji-16/obj12 2024-04-19T23:26:12.891 DEBUG:boto:Data: 2024-04-19T23:26:12.891 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.891 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.891 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.891 DEBUG:boto:Params: {} 2024-04-19T23:26:12.891 DEBUG:boto:Token: None 2024-04-19T23:26:12.891 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj12 2024-04-19T23:26:12.892 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BcAItnDa+7ddczxARgfn0Jd6oUQ= 2024-04-19T23:26:12.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BcAItnDa+7ddczxARgfn0Jd6oUQ='} 2024-04-19T23:26:12.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000685fa46c9ccac02a-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.896 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.896 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.896 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.897 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.897 DEBUG:boto:Data: 2024-04-19T23:26:12.897 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.897 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.897 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.897 DEBUG:boto:Params: {} 2024-04-19T23:26:12.897 DEBUG:boto:Token: None 2024-04-19T23:26:12.897 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.897 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b19d6f26e778d3a-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.900 DEBUG:boto:path=/aktlji-16/obj13 2024-04-19T23:26:12.900 DEBUG:boto:auth_path=/aktlji-16/obj13 2024-04-19T23:26:12.900 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.900 DEBUG:boto:Path: /aktlji-16/obj13 2024-04-19T23:26:12.900 DEBUG:boto:Data: 2024-04-19T23:26:12.900 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.900 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.900 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.900 DEBUG:boto:Params: {} 2024-04-19T23:26:12.901 DEBUG:boto:Token: None 2024-04-19T23:26:12.901 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj13 2024-04-19T23:26:12.901 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Y7i9o6zNThPJosAMQDYirb2x2lk= 2024-04-19T23:26:12.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y7i9o6zNThPJosAMQDYirb2x2lk='} 2024-04-19T23:26:12.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d56325476abfc889-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.942 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.943 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.943 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.943 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.943 DEBUG:boto:Data: 2024-04-19T23:26:12.943 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.943 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.943 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.943 DEBUG:boto:Params: {} 2024-04-19T23:26:12.943 DEBUG:boto:Token: None 2024-04-19T23:26:12.943 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037195bd767548e9b-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.946 DEBUG:boto:path=/aktlji-16/obj14 2024-04-19T23:26:12.946 DEBUG:boto:auth_path=/aktlji-16/obj14 2024-04-19T23:26:12.946 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.946 DEBUG:boto:Path: /aktlji-16/obj14 2024-04-19T23:26:12.947 DEBUG:boto:Data: 2024-04-19T23:26:12.947 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.947 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.947 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.947 DEBUG:boto:Params: {} 2024-04-19T23:26:12.947 DEBUG:boto:Token: None 2024-04-19T23:26:12.947 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj14 2024-04-19T23:26:12.947 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hOkk8hfHTnsf9bS6cv/5o2E8uLw= 2024-04-19T23:26:12.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hOkk8hfHTnsf9bS6cv/5o2E8uLw='} 2024-04-19T23:26:12.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed5f7e29fb2dbbc2-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.952 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:12.952 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:12.953 DEBUG:boto:Method: HEAD 2024-04-19T23:26:12.953 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:12.953 DEBUG:boto:Data: 2024-04-19T23:26:12.953 DEBUG:boto:Headers: {} 2024-04-19T23:26:12.953 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.953 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.953 DEBUG:boto:Params: {} 2024-04-19T23:26:12.953 DEBUG:boto:Token: None 2024-04-19T23:26:12.953 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/ 2024-04-19T23:26:12.954 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-19T23:26:12.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-19T23:26:12.956 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d291b4156d4b495-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:12.956 DEBUG:boto:path=/aktlji-16/obj15 2024-04-19T23:26:12.956 DEBUG:boto:auth_path=/aktlji-16/obj15 2024-04-19T23:26:12.957 DEBUG:boto:Method: PUT 2024-04-19T23:26:12.957 DEBUG:boto:Path: /aktlji-16/obj15 2024-04-19T23:26:12.957 DEBUG:boto:Data: 2024-04-19T23:26:12.957 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:12.957 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:12.957 DEBUG:boto:Port: 8000 2024-04-19T23:26:12.957 DEBUG:boto:Params: {} 2024-04-19T23:26:12.957 DEBUG:boto:Token: None 2024-04-19T23:26:12.958 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:12 GMT /aktlji-16/obj15 2024-04-19T23:26:12.958 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:c4mM2EpNBY8eUf7WlYEjbBpmpZY= 2024-04-19T23:26:12.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c4mM2EpNBY8eUf7WlYEjbBpmpZY='} 2024-04-19T23:26:13.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000472b10cd9c0d537f-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.006 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.006 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.007 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.007 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.007 DEBUG:boto:Data: 2024-04-19T23:26:13.007 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.007 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.007 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.007 DEBUG:boto:Params: {} 2024-04-19T23:26:13.007 DEBUG:boto:Token: None 2024-04-19T23:26:13.007 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.008 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.010 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1a79c035f4c4b0d-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.010 DEBUG:boto:path=/aktlji-16/obj16 2024-04-19T23:26:13.011 DEBUG:boto:auth_path=/aktlji-16/obj16 2024-04-19T23:26:13.011 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.011 DEBUG:boto:Path: /aktlji-16/obj16 2024-04-19T23:26:13.011 DEBUG:boto:Data: 2024-04-19T23:26:13.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.011 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.011 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.011 DEBUG:boto:Params: {} 2024-04-19T23:26:13.011 DEBUG:boto:Token: None 2024-04-19T23:26:13.011 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj16 2024-04-19T23:26:13.012 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/8B32ntaHTU2iIo43FoIaxh9E9E= 2024-04-19T23:26:13.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/8B32ntaHTU2iIo43FoIaxh9E9E='} 2024-04-19T23:26:13.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f487a4fa98a5acde-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.018 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.018 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.018 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.019 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.019 DEBUG:boto:Data: 2024-04-19T23:26:13.019 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.019 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.019 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.019 DEBUG:boto:Params: {} 2024-04-19T23:26:13.019 DEBUG:boto:Token: None 2024-04-19T23:26:13.019 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.019 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a532d964c60a4a9-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.022 DEBUG:boto:path=/aktlji-16/obj17 2024-04-19T23:26:13.022 DEBUG:boto:auth_path=/aktlji-16/obj17 2024-04-19T23:26:13.022 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.023 DEBUG:boto:Path: /aktlji-16/obj17 2024-04-19T23:26:13.023 DEBUG:boto:Data: 2024-04-19T23:26:13.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.023 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.023 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.023 DEBUG:boto:Params: {} 2024-04-19T23:26:13.023 DEBUG:boto:Token: None 2024-04-19T23:26:13.023 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj17 2024-04-19T23:26:13.023 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EgsoaJrxor7zMQm7ternFnTS+Pk= 2024-04-19T23:26:13.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EgsoaJrxor7zMQm7ternFnTS+Pk='} 2024-04-19T23:26:13.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b29d9a5fc7908427-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.028 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.029 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.029 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.029 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.029 DEBUG:boto:Data: 2024-04-19T23:26:13.029 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.029 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.029 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.029 DEBUG:boto:Params: {} 2024-04-19T23:26:13.030 DEBUG:boto:Token: None 2024-04-19T23:26:13.030 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.030 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.032 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4c0c8952661a4c9-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.032 DEBUG:boto:path=/aktlji-16/obj18 2024-04-19T23:26:13.033 DEBUG:boto:auth_path=/aktlji-16/obj18 2024-04-19T23:26:13.033 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.033 DEBUG:boto:Path: /aktlji-16/obj18 2024-04-19T23:26:13.033 DEBUG:boto:Data: 2024-04-19T23:26:13.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.033 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.033 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.033 DEBUG:boto:Params: {} 2024-04-19T23:26:13.034 DEBUG:boto:Token: None 2024-04-19T23:26:13.034 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj18 2024-04-19T23:26:13.034 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:E5YFg1R00gImpRK8V9FNaH23Lxg= 2024-04-19T23:26:13.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E5YFg1R00gImpRK8V9FNaH23Lxg='} 2024-04-19T23:26:13.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067447fa4e72006dd-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.083 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.083 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.083 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.083 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.083 DEBUG:boto:Data: 2024-04-19T23:26:13.083 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.083 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.083 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.083 DEBUG:boto:Params: {} 2024-04-19T23:26:13.084 DEBUG:boto:Token: None 2024-04-19T23:26:13.084 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a13cb628acc613c8-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.086 DEBUG:boto:path=/aktlji-16/obj19 2024-04-19T23:26:13.087 DEBUG:boto:auth_path=/aktlji-16/obj19 2024-04-19T23:26:13.087 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.087 DEBUG:boto:Path: /aktlji-16/obj19 2024-04-19T23:26:13.087 DEBUG:boto:Data: 2024-04-19T23:26:13.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.087 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.087 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.087 DEBUG:boto:Params: {} 2024-04-19T23:26:13.087 DEBUG:boto:Token: None 2024-04-19T23:26:13.088 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj19 2024-04-19T23:26:13.088 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vZKsyGeWQ1xQK1ziv2+XTVKSM1s= 2024-04-19T23:26:13.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vZKsyGeWQ1xQK1ziv2+XTVKSM1s='} 2024-04-19T23:26:13.092 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f87bb5de21d73ed5-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.092 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.093 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.093 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.093 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.093 DEBUG:boto:Data: 2024-04-19T23:26:13.093 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.093 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.093 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.093 DEBUG:boto:Params: {} 2024-04-19T23:26:13.093 DEBUG:boto:Token: None 2024-04-19T23:26:13.093 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.094 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.096 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054c5b0ccaa9f1e29-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.096 DEBUG:boto:path=/aktlji-16/obj20 2024-04-19T23:26:13.097 DEBUG:boto:auth_path=/aktlji-16/obj20 2024-04-19T23:26:13.097 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.097 DEBUG:boto:Path: /aktlji-16/obj20 2024-04-19T23:26:13.097 DEBUG:boto:Data: 2024-04-19T23:26:13.097 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.097 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.097 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.097 DEBUG:boto:Params: {} 2024-04-19T23:26:13.097 DEBUG:boto:Token: None 2024-04-19T23:26:13.098 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj20 2024-04-19T23:26:13.098 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7E8VrFDMIsOAzrh16DI1KMjezEA= 2024-04-19T23:26:13.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7E8VrFDMIsOAzrh16DI1KMjezEA='} 2024-04-19T23:26:13.104 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8b240dbbaba704c-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.104 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.104 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.104 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.104 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.104 DEBUG:boto:Data: 2024-04-19T23:26:13.105 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.105 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.105 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.105 DEBUG:boto:Params: {} 2024-04-19T23:26:13.105 DEBUG:boto:Token: None 2024-04-19T23:26:13.105 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.105 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.107 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000003f0146cfddb3c7-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.108 DEBUG:boto:path=/aktlji-16/obj21 2024-04-19T23:26:13.108 DEBUG:boto:auth_path=/aktlji-16/obj21 2024-04-19T23:26:13.108 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.108 DEBUG:boto:Path: /aktlji-16/obj21 2024-04-19T23:26:13.108 DEBUG:boto:Data: 2024-04-19T23:26:13.108 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.108 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.108 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.109 DEBUG:boto:Params: {} 2024-04-19T23:26:13.109 DEBUG:boto:Token: None 2024-04-19T23:26:13.109 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj21 2024-04-19T23:26:13.109 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8Dj4SFSOyA+NqAsraGlyuSM9RGk= 2024-04-19T23:26:13.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8Dj4SFSOyA+NqAsraGlyuSM9RGk='} 2024-04-19T23:26:13.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055271d97a174723b-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.151 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.151 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.151 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.151 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.151 DEBUG:boto:Data: 2024-04-19T23:26:13.151 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.151 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.151 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.151 DEBUG:boto:Params: {} 2024-04-19T23:26:13.151 DEBUG:boto:Token: None 2024-04-19T23:26:13.152 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.152 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cd9459e8570a7be-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.155 DEBUG:boto:path=/aktlji-16/obj22 2024-04-19T23:26:13.155 DEBUG:boto:auth_path=/aktlji-16/obj22 2024-04-19T23:26:13.155 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.155 DEBUG:boto:Path: /aktlji-16/obj22 2024-04-19T23:26:13.155 DEBUG:boto:Data: 2024-04-19T23:26:13.155 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.155 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.155 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.155 DEBUG:boto:Params: {} 2024-04-19T23:26:13.155 DEBUG:boto:Token: None 2024-04-19T23:26:13.156 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj22 2024-04-19T23:26:13.156 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9JOTjg1pN06skhW2e+Ev1iiBIxI= 2024-04-19T23:26:13.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9JOTjg1pN06skhW2e+Ev1iiBIxI='} 2024-04-19T23:26:13.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ee85cd8d6b05044-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.161 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.161 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.161 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.161 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.161 DEBUG:boto:Data: 2024-04-19T23:26:13.161 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.161 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.161 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.161 DEBUG:boto:Params: {} 2024-04-19T23:26:13.161 DEBUG:boto:Token: None 2024-04-19T23:26:13.161 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.162 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.164 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a5831ec1f0b8512-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.164 DEBUG:boto:path=/aktlji-16/obj23 2024-04-19T23:26:13.164 DEBUG:boto:auth_path=/aktlji-16/obj23 2024-04-19T23:26:13.164 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.164 DEBUG:boto:Path: /aktlji-16/obj23 2024-04-19T23:26:13.165 DEBUG:boto:Data: 2024-04-19T23:26:13.165 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.165 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.165 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.165 DEBUG:boto:Params: {} 2024-04-19T23:26:13.165 DEBUG:boto:Token: None 2024-04-19T23:26:13.165 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj23 2024-04-19T23:26:13.165 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:B3wzCtoPYlB3dgaQ1AhMdXSq+/4= 2024-04-19T23:26:13.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B3wzCtoPYlB3dgaQ1AhMdXSq+/4='} 2024-04-19T23:26:13.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f33ae953747bbd7-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.215 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.215 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.215 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.216 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.216 DEBUG:boto:Data: 2024-04-19T23:26:13.216 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.216 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.216 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.216 DEBUG:boto:Params: {} 2024-04-19T23:26:13.216 DEBUG:boto:Token: None 2024-04-19T23:26:13.216 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.217 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.219 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2adf18b28e5be3d-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.220 DEBUG:boto:path=/aktlji-16/obj24 2024-04-19T23:26:13.220 DEBUG:boto:auth_path=/aktlji-16/obj24 2024-04-19T23:26:13.220 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.220 DEBUG:boto:Path: /aktlji-16/obj24 2024-04-19T23:26:13.220 DEBUG:boto:Data: 2024-04-19T23:26:13.220 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.220 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.221 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.221 DEBUG:boto:Params: {} 2024-04-19T23:26:13.221 DEBUG:boto:Token: None 2024-04-19T23:26:13.221 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj24 2024-04-19T23:26:13.221 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4gv7O0ENkNwPHyF2ITHq+Jz4k+A= 2024-04-19T23:26:13.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4gv7O0ENkNwPHyF2ITHq+Jz4k+A='} 2024-04-19T23:26:13.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2cd3bdb34f7648b-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.226 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.226 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.226 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.226 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.226 DEBUG:boto:Data: 2024-04-19T23:26:13.226 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.226 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.226 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.227 DEBUG:boto:Params: {} 2024-04-19T23:26:13.227 DEBUG:boto:Token: None 2024-04-19T23:26:13.227 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.227 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.229 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d73a439f877597dc-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.229 DEBUG:boto:path=/aktlji-16/obj25 2024-04-19T23:26:13.230 DEBUG:boto:auth_path=/aktlji-16/obj25 2024-04-19T23:26:13.230 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.230 DEBUG:boto:Path: /aktlji-16/obj25 2024-04-19T23:26:13.230 DEBUG:boto:Data: 2024-04-19T23:26:13.230 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.230 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.230 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.230 DEBUG:boto:Params: {} 2024-04-19T23:26:13.230 DEBUG:boto:Token: None 2024-04-19T23:26:13.230 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj25 2024-04-19T23:26:13.231 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pUzcHYaE78TqEdkrgZyVPUcf7DM= 2024-04-19T23:26:13.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pUzcHYaE78TqEdkrgZyVPUcf7DM='} 2024-04-19T23:26:13.275 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000543e2c252a910eb3-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.275 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.275 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.275 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.275 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.275 DEBUG:boto:Data: 2024-04-19T23:26:13.275 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.275 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.275 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.276 DEBUG:boto:Params: {} 2024-04-19T23:26:13.276 DEBUG:boto:Token: None 2024-04-19T23:26:13.276 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.276 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.278 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbf8f5afbbb3e978-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.279 DEBUG:boto:path=/aktlji-16/obj26 2024-04-19T23:26:13.279 DEBUG:boto:auth_path=/aktlji-16/obj26 2024-04-19T23:26:13.279 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.279 DEBUG:boto:Path: /aktlji-16/obj26 2024-04-19T23:26:13.279 DEBUG:boto:Data: 2024-04-19T23:26:13.279 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.279 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.279 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.279 DEBUG:boto:Params: {} 2024-04-19T23:26:13.280 DEBUG:boto:Token: None 2024-04-19T23:26:13.280 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj26 2024-04-19T23:26:13.280 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T4zvxCTU4b06pNJEeQMQXXm1v5w= 2024-04-19T23:26:13.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T4zvxCTU4b06pNJEeQMQXXm1v5w='} 2024-04-19T23:26:13.284 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ebc8c44de4a494f-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.285 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.285 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.285 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.285 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.285 DEBUG:boto:Data: 2024-04-19T23:26:13.285 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.285 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.286 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.286 DEBUG:boto:Params: {} 2024-04-19T23:26:13.286 DEBUG:boto:Token: None 2024-04-19T23:26:13.286 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.286 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cdd7e74afbccc14-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.289 DEBUG:boto:path=/aktlji-16/obj27 2024-04-19T23:26:13.289 DEBUG:boto:auth_path=/aktlji-16/obj27 2024-04-19T23:26:13.289 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.289 DEBUG:boto:Path: /aktlji-16/obj27 2024-04-19T23:26:13.289 DEBUG:boto:Data: 2024-04-19T23:26:13.289 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.289 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.289 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.289 DEBUG:boto:Params: {} 2024-04-19T23:26:13.289 DEBUG:boto:Token: None 2024-04-19T23:26:13.290 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj27 2024-04-19T23:26:13.290 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GFLKbOWrtSMf+9xoN8H90r8prE0= 2024-04-19T23:26:13.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GFLKbOWrtSMf+9xoN8H90r8prE0='} 2024-04-19T23:26:13.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbc7abba149b3347-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.339 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.339 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.339 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.339 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.339 DEBUG:boto:Data: 2024-04-19T23:26:13.339 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.339 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.339 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.339 DEBUG:boto:Params: {} 2024-04-19T23:26:13.339 DEBUG:boto:Token: None 2024-04-19T23:26:13.340 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.340 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e77690ae9476c44-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.342 DEBUG:boto:path=/aktlji-16/obj28 2024-04-19T23:26:13.342 DEBUG:boto:auth_path=/aktlji-16/obj28 2024-04-19T23:26:13.343 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.343 DEBUG:boto:Path: /aktlji-16/obj28 2024-04-19T23:26:13.343 DEBUG:boto:Data: 2024-04-19T23:26:13.343 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.343 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.343 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.343 DEBUG:boto:Params: {} 2024-04-19T23:26:13.343 DEBUG:boto:Token: None 2024-04-19T23:26:13.343 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj28 2024-04-19T23:26:13.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:E6j3X480HgIyW8Cwr/OC8i3euYQ= 2024-04-19T23:26:13.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E6j3X480HgIyW8Cwr/OC8i3euYQ='} 2024-04-19T23:26:13.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6cf322bde94d404-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.348 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.348 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.349 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.349 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.349 DEBUG:boto:Data: 2024-04-19T23:26:13.349 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.349 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.349 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.349 DEBUG:boto:Params: {} 2024-04-19T23:26:13.349 DEBUG:boto:Token: None 2024-04-19T23:26:13.349 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.349 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009989e3c5f16c9244-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.352 DEBUG:boto:path=/aktlji-16/obj29 2024-04-19T23:26:13.352 DEBUG:boto:auth_path=/aktlji-16/obj29 2024-04-19T23:26:13.353 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.353 DEBUG:boto:Path: /aktlji-16/obj29 2024-04-19T23:26:13.353 DEBUG:boto:Data: 2024-04-19T23:26:13.353 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.353 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.353 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.353 DEBUG:boto:Params: {} 2024-04-19T23:26:13.353 DEBUG:boto:Token: None 2024-04-19T23:26:13.353 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj29 2024-04-19T23:26:13.354 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:puXjnMzbFORwHJpYcf2epxskhF0= 2024-04-19T23:26:13.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:puXjnMzbFORwHJpYcf2epxskhF0='} 2024-04-19T23:26:13.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003a06a3d22bc3200-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.359 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.360 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.360 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.360 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.360 DEBUG:boto:Data: 2024-04-19T23:26:13.360 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.360 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.360 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.360 DEBUG:boto:Params: {} 2024-04-19T23:26:13.360 DEBUG:boto:Token: None 2024-04-19T23:26:13.360 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.361 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024cea54ce25296a9-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.363 DEBUG:boto:path=/aktlji-16/obj30 2024-04-19T23:26:13.363 DEBUG:boto:auth_path=/aktlji-16/obj30 2024-04-19T23:26:13.363 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.363 DEBUG:boto:Path: /aktlji-16/obj30 2024-04-19T23:26:13.363 DEBUG:boto:Data: 2024-04-19T23:26:13.363 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.364 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.364 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.364 DEBUG:boto:Params: {} 2024-04-19T23:26:13.364 DEBUG:boto:Token: None 2024-04-19T23:26:13.364 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj30 2024-04-19T23:26:13.364 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hG51rjXT+J49UXS6KXCGiOd/zsk= 2024-04-19T23:26:13.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hG51rjXT+J49UXS6KXCGiOd/zsk='} 2024-04-19T23:26:13.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bd38f356d707637-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.407 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.407 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.407 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.407 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.407 DEBUG:boto:Data: 2024-04-19T23:26:13.407 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.407 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.408 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.408 DEBUG:boto:Params: {} 2024-04-19T23:26:13.408 DEBUG:boto:Token: None 2024-04-19T23:26:13.408 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.408 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e2631c622851295-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.411 DEBUG:boto:path=/aktlji-16/obj31 2024-04-19T23:26:13.411 DEBUG:boto:auth_path=/aktlji-16/obj31 2024-04-19T23:26:13.411 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.412 DEBUG:boto:Path: /aktlji-16/obj31 2024-04-19T23:26:13.412 DEBUG:boto:Data: 2024-04-19T23:26:13.412 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.412 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.412 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.412 DEBUG:boto:Params: {} 2024-04-19T23:26:13.412 DEBUG:boto:Token: None 2024-04-19T23:26:13.412 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj31 2024-04-19T23:26:13.412 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g2mBpl78cWx5oAD4rA40UNA0CI0= 2024-04-19T23:26:13.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g2mBpl78cWx5oAD4rA40UNA0CI0='} 2024-04-19T23:26:13.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000741b74009c4d88fb-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.418 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.418 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.418 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.418 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.418 DEBUG:boto:Data: 2024-04-19T23:26:13.418 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.419 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.419 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.419 DEBUG:boto:Params: {} 2024-04-19T23:26:13.419 DEBUG:boto:Token: None 2024-04-19T23:26:13.419 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.419 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3d19efdbef7bba7-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.422 DEBUG:boto:path=/aktlji-16/obj32 2024-04-19T23:26:13.422 DEBUG:boto:auth_path=/aktlji-16/obj32 2024-04-19T23:26:13.422 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.422 DEBUG:boto:Path: /aktlji-16/obj32 2024-04-19T23:26:13.422 DEBUG:boto:Data: 2024-04-19T23:26:13.422 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.422 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.423 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.423 DEBUG:boto:Params: {} 2024-04-19T23:26:13.423 DEBUG:boto:Token: None 2024-04-19T23:26:13.423 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj32 2024-04-19T23:26:13.423 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aHg+nE0cV+0fi+uhGh8SDdUcI50= 2024-04-19T23:26:13.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aHg+nE0cV+0fi+uhGh8SDdUcI50='} 2024-04-19T23:26:13.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6391c3345f2f76e-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.471 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.471 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.471 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.471 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.471 DEBUG:boto:Data: 2024-04-19T23:26:13.471 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.471 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.471 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.471 DEBUG:boto:Params: {} 2024-04-19T23:26:13.472 DEBUG:boto:Token: None 2024-04-19T23:26:13.472 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.472 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020eeb57a2bde8857-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.474 DEBUG:boto:path=/aktlji-16/obj33 2024-04-19T23:26:13.474 DEBUG:boto:auth_path=/aktlji-16/obj33 2024-04-19T23:26:13.475 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.475 DEBUG:boto:Path: /aktlji-16/obj33 2024-04-19T23:26:13.475 DEBUG:boto:Data: 2024-04-19T23:26:13.475 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.475 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.475 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.475 DEBUG:boto:Params: {} 2024-04-19T23:26:13.475 DEBUG:boto:Token: None 2024-04-19T23:26:13.475 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj33 2024-04-19T23:26:13.476 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pK/+sYNFIO7ZG0TiRYSbStxsj5o= 2024-04-19T23:26:13.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pK/+sYNFIO7ZG0TiRYSbStxsj5o='} 2024-04-19T23:26:13.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bc5253809091a61-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.480 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.480 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.480 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.481 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.481 DEBUG:boto:Data: 2024-04-19T23:26:13.481 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.481 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.481 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.481 DEBUG:boto:Params: {} 2024-04-19T23:26:13.481 DEBUG:boto:Token: None 2024-04-19T23:26:13.481 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.481 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013c6cd2b7274e702-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.484 DEBUG:boto:path=/aktlji-16/obj34 2024-04-19T23:26:13.484 DEBUG:boto:auth_path=/aktlji-16/obj34 2024-04-19T23:26:13.484 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.484 DEBUG:boto:Path: /aktlji-16/obj34 2024-04-19T23:26:13.484 DEBUG:boto:Data: 2024-04-19T23:26:13.484 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.485 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.485 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.485 DEBUG:boto:Params: {} 2024-04-19T23:26:13.485 DEBUG:boto:Token: None 2024-04-19T23:26:13.485 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj34 2024-04-19T23:26:13.485 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1lYk4n7y419jjrDqkfRIOSTeQ14= 2024-04-19T23:26:13.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1lYk4n7y419jjrDqkfRIOSTeQ14='} 2024-04-19T23:26:13.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7fba3537354c123-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.535 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.535 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.535 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.535 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.535 DEBUG:boto:Data: 2024-04-19T23:26:13.535 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.535 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.535 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.535 DEBUG:boto:Params: {} 2024-04-19T23:26:13.536 DEBUG:boto:Token: None 2024-04-19T23:26:13.536 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.536 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000456814b552afe91b-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.539 DEBUG:boto:path=/aktlji-16/obj35 2024-04-19T23:26:13.539 DEBUG:boto:auth_path=/aktlji-16/obj35 2024-04-19T23:26:13.539 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.539 DEBUG:boto:Path: /aktlji-16/obj35 2024-04-19T23:26:13.539 DEBUG:boto:Data: 2024-04-19T23:26:13.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.539 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.539 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.540 DEBUG:boto:Params: {} 2024-04-19T23:26:13.540 DEBUG:boto:Token: None 2024-04-19T23:26:13.540 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj35 2024-04-19T23:26:13.540 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vCuI5njxZ/XVn6/BOdTA000aHAE= 2024-04-19T23:26:13.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vCuI5njxZ/XVn6/BOdTA000aHAE='} 2024-04-19T23:26:13.545 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f15b18d5e20e7990-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.545 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.545 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.545 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.545 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.545 DEBUG:boto:Data: 2024-04-19T23:26:13.545 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.545 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.545 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.546 DEBUG:boto:Params: {} 2024-04-19T23:26:13.546 DEBUG:boto:Token: None 2024-04-19T23:26:13.546 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.546 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5dfc7ff960a5619-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.549 DEBUG:boto:path=/aktlji-16/obj36 2024-04-19T23:26:13.549 DEBUG:boto:auth_path=/aktlji-16/obj36 2024-04-19T23:26:13.549 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.549 DEBUG:boto:Path: /aktlji-16/obj36 2024-04-19T23:26:13.549 DEBUG:boto:Data: 2024-04-19T23:26:13.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.549 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.549 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.549 DEBUG:boto:Params: {} 2024-04-19T23:26:13.549 DEBUG:boto:Token: None 2024-04-19T23:26:13.550 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj36 2024-04-19T23:26:13.550 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FjHT0EfwQGD8ZvM9RqdiRnoNhdY= 2024-04-19T23:26:13.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FjHT0EfwQGD8ZvM9RqdiRnoNhdY='} 2024-04-19T23:26:13.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a094e3669b83443-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.595 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.595 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.595 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.595 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.595 DEBUG:boto:Data: 2024-04-19T23:26:13.595 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.595 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.595 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.595 DEBUG:boto:Params: {} 2024-04-19T23:26:13.595 DEBUG:boto:Token: None 2024-04-19T23:26:13.596 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.596 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000604b8886fcb3f564-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.599 DEBUG:boto:path=/aktlji-16/obj37 2024-04-19T23:26:13.599 DEBUG:boto:auth_path=/aktlji-16/obj37 2024-04-19T23:26:13.599 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.599 DEBUG:boto:Path: /aktlji-16/obj37 2024-04-19T23:26:13.599 DEBUG:boto:Data: 2024-04-19T23:26:13.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.599 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.599 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.599 DEBUG:boto:Params: {} 2024-04-19T23:26:13.600 DEBUG:boto:Token: None 2024-04-19T23:26:13.600 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj37 2024-04-19T23:26:13.600 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PeSpm2Bm89aj3vLyT2UpcwbD6KI= 2024-04-19T23:26:13.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PeSpm2Bm89aj3vLyT2UpcwbD6KI='} 2024-04-19T23:26:13.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f1c70b4cbe5f75a-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.605 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.605 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.605 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.605 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.605 DEBUG:boto:Data: 2024-04-19T23:26:13.605 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.605 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.605 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.605 DEBUG:boto:Params: {} 2024-04-19T23:26:13.606 DEBUG:boto:Token: None 2024-04-19T23:26:13.606 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.606 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.608 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084f9756c9734cacd-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.608 DEBUG:boto:path=/aktlji-16/obj38 2024-04-19T23:26:13.608 DEBUG:boto:auth_path=/aktlji-16/obj38 2024-04-19T23:26:13.609 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.609 DEBUG:boto:Path: /aktlji-16/obj38 2024-04-19T23:26:13.609 DEBUG:boto:Data: 2024-04-19T23:26:13.609 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.609 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.609 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.609 DEBUG:boto:Params: {} 2024-04-19T23:26:13.609 DEBUG:boto:Token: None 2024-04-19T23:26:13.609 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj38 2024-04-19T23:26:13.610 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GruXFFJDPDJ+oeJBk9zxFCVgpfo= 2024-04-19T23:26:13.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GruXFFJDPDJ+oeJBk9zxFCVgpfo='} 2024-04-19T23:26:13.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007bd9eb69d07711ec-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.655 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.655 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.655 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.655 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.656 DEBUG:boto:Data: 2024-04-19T23:26:13.656 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.656 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.656 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.656 DEBUG:boto:Params: {} 2024-04-19T23:26:13.656 DEBUG:boto:Token: None 2024-04-19T23:26:13.656 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.656 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000014a8c7e85719b0a-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.659 DEBUG:boto:path=/aktlji-16/obj39 2024-04-19T23:26:13.659 DEBUG:boto:auth_path=/aktlji-16/obj39 2024-04-19T23:26:13.659 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.660 DEBUG:boto:Path: /aktlji-16/obj39 2024-04-19T23:26:13.660 DEBUG:boto:Data: 2024-04-19T23:26:13.660 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.660 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.660 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.660 DEBUG:boto:Params: {} 2024-04-19T23:26:13.660 DEBUG:boto:Token: None 2024-04-19T23:26:13.661 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj39 2024-04-19T23:26:13.661 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6ReF5i75TFtw6femqMFS5xyO/eA= 2024-04-19T23:26:13.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ReF5i75TFtw6femqMFS5xyO/eA='} 2024-04-19T23:26:13.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000393eb47622798bc5-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.665 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.665 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.665 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.665 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.665 DEBUG:boto:Data: 2024-04-19T23:26:13.666 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.666 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.666 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.666 DEBUG:boto:Params: {} 2024-04-19T23:26:13.666 DEBUG:boto:Token: None 2024-04-19T23:26:13.666 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.666 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.669 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d30aedb198cdf2c5-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.669 DEBUG:boto:path=/aktlji-16/obj40 2024-04-19T23:26:13.669 DEBUG:boto:auth_path=/aktlji-16/obj40 2024-04-19T23:26:13.669 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.669 DEBUG:boto:Path: /aktlji-16/obj40 2024-04-19T23:26:13.669 DEBUG:boto:Data: 2024-04-19T23:26:13.669 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.670 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.670 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.670 DEBUG:boto:Params: {} 2024-04-19T23:26:13.670 DEBUG:boto:Token: None 2024-04-19T23:26:13.670 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj40 2024-04-19T23:26:13.670 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EErmUrS+SsBhJLn0Ud3ML7QVN1I= 2024-04-19T23:26:13.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EErmUrS+SsBhJLn0Ud3ML7QVN1I='} 2024-04-19T23:26:13.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a794e489a221409-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.719 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.719 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.719 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.719 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.719 DEBUG:boto:Data: 2024-04-19T23:26:13.719 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.719 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.719 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.720 DEBUG:boto:Params: {} 2024-04-19T23:26:13.720 DEBUG:boto:Token: None 2024-04-19T23:26:13.720 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.720 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005be27f2376894f9b-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.722 DEBUG:boto:path=/aktlji-16/obj41 2024-04-19T23:26:13.723 DEBUG:boto:auth_path=/aktlji-16/obj41 2024-04-19T23:26:13.723 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.723 DEBUG:boto:Path: /aktlji-16/obj41 2024-04-19T23:26:13.723 DEBUG:boto:Data: 2024-04-19T23:26:13.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.723 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.723 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.723 DEBUG:boto:Params: {} 2024-04-19T23:26:13.723 DEBUG:boto:Token: None 2024-04-19T23:26:13.724 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj41 2024-04-19T23:26:13.724 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZYXEkoxpPvNuytXIxz4g8st1vVc= 2024-04-19T23:26:13.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZYXEkoxpPvNuytXIxz4g8st1vVc='} 2024-04-19T23:26:13.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e57436f54bbaff50-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.728 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.728 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.728 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.728 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.729 DEBUG:boto:Data: 2024-04-19T23:26:13.729 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.729 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.729 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.729 DEBUG:boto:Params: {} 2024-04-19T23:26:13.729 DEBUG:boto:Token: None 2024-04-19T23:26:13.729 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.729 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c589e43ca9b4cda-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.732 DEBUG:boto:path=/aktlji-16/obj42 2024-04-19T23:26:13.732 DEBUG:boto:auth_path=/aktlji-16/obj42 2024-04-19T23:26:13.732 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.732 DEBUG:boto:Path: /aktlji-16/obj42 2024-04-19T23:26:13.732 DEBUG:boto:Data: 2024-04-19T23:26:13.732 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.732 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.732 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.732 DEBUG:boto:Params: {} 2024-04-19T23:26:13.733 DEBUG:boto:Token: None 2024-04-19T23:26:13.733 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj42 2024-04-19T23:26:13.733 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZT7aZjfC2dQPh42jq8J581UQgQ= 2024-04-19T23:26:13.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZT7aZjfC2dQPh42jq8J581UQgQ='} 2024-04-19T23:26:13.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000918c4196581d59d7-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.779 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.779 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.779 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.779 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.779 DEBUG:boto:Data: 2024-04-19T23:26:13.780 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.780 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.780 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.780 DEBUG:boto:Params: {} 2024-04-19T23:26:13.780 DEBUG:boto:Token: None 2024-04-19T23:26:13.780 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.780 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.783 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a99ab4653130000-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.783 DEBUG:boto:path=/aktlji-16/obj43 2024-04-19T23:26:13.783 DEBUG:boto:auth_path=/aktlji-16/obj43 2024-04-19T23:26:13.783 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.784 DEBUG:boto:Path: /aktlji-16/obj43 2024-04-19T23:26:13.784 DEBUG:boto:Data: 2024-04-19T23:26:13.784 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.784 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.784 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.784 DEBUG:boto:Params: {} 2024-04-19T23:26:13.784 DEBUG:boto:Token: None 2024-04-19T23:26:13.784 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj43 2024-04-19T23:26:13.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X1M7/rRWV1Tld4d1TbM34t4k02A= 2024-04-19T23:26:13.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X1M7/rRWV1Tld4d1TbM34t4k02A='} 2024-04-19T23:26:13.789 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a57844bc600332c-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.789 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.790 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.790 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.790 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.790 DEBUG:boto:Data: 2024-04-19T23:26:13.790 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.790 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.790 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.790 DEBUG:boto:Params: {} 2024-04-19T23:26:13.790 DEBUG:boto:Token: None 2024-04-19T23:26:13.790 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.791 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d25b684d94d830d-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.793 DEBUG:boto:path=/aktlji-16/obj44 2024-04-19T23:26:13.793 DEBUG:boto:auth_path=/aktlji-16/obj44 2024-04-19T23:26:13.793 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.793 DEBUG:boto:Path: /aktlji-16/obj44 2024-04-19T23:26:13.793 DEBUG:boto:Data: 2024-04-19T23:26:13.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.794 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.794 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.794 DEBUG:boto:Params: {} 2024-04-19T23:26:13.794 DEBUG:boto:Token: None 2024-04-19T23:26:13.794 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj44 2024-04-19T23:26:13.794 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2r6QIkuFHIylAFubQre5IPPXMiM= 2024-04-19T23:26:13.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2r6QIkuFHIylAFubQre5IPPXMiM='} 2024-04-19T23:26:13.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cf2d3a0207cad3d-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.839 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.839 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.839 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.839 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.839 DEBUG:boto:Data: 2024-04-19T23:26:13.839 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.839 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.839 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.839 DEBUG:boto:Params: {} 2024-04-19T23:26:13.840 DEBUG:boto:Token: None 2024-04-19T23:26:13.840 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.840 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.843 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b21ab38925acbe7-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.843 DEBUG:boto:path=/aktlji-16/obj45 2024-04-19T23:26:13.843 DEBUG:boto:auth_path=/aktlji-16/obj45 2024-04-19T23:26:13.843 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.843 DEBUG:boto:Path: /aktlji-16/obj45 2024-04-19T23:26:13.843 DEBUG:boto:Data: 2024-04-19T23:26:13.844 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.844 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.844 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.844 DEBUG:boto:Params: {} 2024-04-19T23:26:13.844 DEBUG:boto:Token: None 2024-04-19T23:26:13.844 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj45 2024-04-19T23:26:13.844 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:S+zu7L6RpN4hEO/bXkFxlSvTBDg= 2024-04-19T23:26:13.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:S+zu7L6RpN4hEO/bXkFxlSvTBDg='} 2024-04-19T23:26:13.849 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000371f8584106fdffe-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.849 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.849 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.849 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.850 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.850 DEBUG:boto:Data: 2024-04-19T23:26:13.850 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.850 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.850 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.850 DEBUG:boto:Params: {} 2024-04-19T23:26:13.850 DEBUG:boto:Token: None 2024-04-19T23:26:13.850 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.850 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a53feba8fa14508b-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.853 DEBUG:boto:path=/aktlji-16/obj46 2024-04-19T23:26:13.853 DEBUG:boto:auth_path=/aktlji-16/obj46 2024-04-19T23:26:13.853 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.853 DEBUG:boto:Path: /aktlji-16/obj46 2024-04-19T23:26:13.853 DEBUG:boto:Data: 2024-04-19T23:26:13.853 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.853 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.854 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.854 DEBUG:boto:Params: {} 2024-04-19T23:26:13.854 DEBUG:boto:Token: None 2024-04-19T23:26:13.854 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj46 2024-04-19T23:26:13.854 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BNQJ/JOXwY/YzXvd31cB/SSIvGo= 2024-04-19T23:26:13.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BNQJ/JOXwY/YzXvd31cB/SSIvGo='} 2024-04-19T23:26:13.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cfda88f8e957d8c-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.899 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.899 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.899 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.899 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.899 DEBUG:boto:Data: 2024-04-19T23:26:13.899 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.899 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.899 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.899 DEBUG:boto:Params: {} 2024-04-19T23:26:13.899 DEBUG:boto:Token: None 2024-04-19T23:26:13.900 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.900 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.902 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042117a310a95af87-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.903 DEBUG:boto:path=/aktlji-16/obj47 2024-04-19T23:26:13.903 DEBUG:boto:auth_path=/aktlji-16/obj47 2024-04-19T23:26:13.903 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.903 DEBUG:boto:Path: /aktlji-16/obj47 2024-04-19T23:26:13.903 DEBUG:boto:Data: 2024-04-19T23:26:13.903 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.903 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.903 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.903 DEBUG:boto:Params: {} 2024-04-19T23:26:13.904 DEBUG:boto:Token: None 2024-04-19T23:26:13.904 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj47 2024-04-19T23:26:13.904 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MpCznzp2+XNZ/6nCgw251J5hoqE= 2024-04-19T23:26:13.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MpCznzp2+XNZ/6nCgw251J5hoqE='} 2024-04-19T23:26:13.909 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001594a264cc10f7fb-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.909 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.909 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.909 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.909 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.909 DEBUG:boto:Data: 2024-04-19T23:26:13.909 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.910 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.910 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.910 DEBUG:boto:Params: {} 2024-04-19T23:26:13.910 DEBUG:boto:Token: None 2024-04-19T23:26:13.910 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.910 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004225f081890b1461-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.912 DEBUG:boto:path=/aktlji-16/obj48 2024-04-19T23:26:13.913 DEBUG:boto:auth_path=/aktlji-16/obj48 2024-04-19T23:26:13.913 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.913 DEBUG:boto:Path: /aktlji-16/obj48 2024-04-19T23:26:13.913 DEBUG:boto:Data: 2024-04-19T23:26:13.913 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.913 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.913 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.913 DEBUG:boto:Params: {} 2024-04-19T23:26:13.913 DEBUG:boto:Token: None 2024-04-19T23:26:13.913 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj48 2024-04-19T23:26:13.914 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yo9KrHU+1F0rujx+TgpkQbiG3SE= 2024-04-19T23:26:13.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yo9KrHU+1F0rujx+TgpkQbiG3SE='} 2024-04-19T23:26:13.959 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b86a8702d1e45a38-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.959 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:13.959 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:13.959 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.959 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:13.959 DEBUG:boto:Data: 2024-04-19T23:26:13.959 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.960 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.960 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.960 DEBUG:boto:Params: {} 2024-04-19T23:26:13.960 DEBUG:boto:Token: None 2024-04-19T23:26:13.960 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/ 2024-04-19T23:26:13.960 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-19T23:26:13.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-19T23:26:13.963 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000671b8f0ee82db957-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.963 DEBUG:boto:path=/aktlji-16/obj49 2024-04-19T23:26:13.963 DEBUG:boto:auth_path=/aktlji-16/obj49 2024-04-19T23:26:13.963 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.963 DEBUG:boto:Path: /aktlji-16/obj49 2024-04-19T23:26:13.963 DEBUG:boto:Data: 2024-04-19T23:26:13.963 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.964 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:13.964 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.964 DEBUG:boto:Params: {} 2024-04-19T23:26:13.964 DEBUG:boto:Token: None 2024-04-19T23:26:13.964 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-16/obj49 2024-04-19T23:26:13.965 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:F8+QA60Y3MVYAIsGXO6RGqsO2DY= 2024-04-19T23:26:13.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:F8+QA60Y3MVYAIsGXO6RGqsO2DY='} 2024-04-19T23:26:13.969 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b370c078912b4d3-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.969 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:13.969 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:13.970 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.970 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:13.970 DEBUG:boto:Data: 2024-04-19T23:26:13.970 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.970 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:13.970 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.970 DEBUG:boto:Params: {} 2024-04-19T23:26:13.971 DEBUG:boto:Token: None 2024-04-19T23:26:13.971 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-17/ 2024-04-19T23:26:13.971 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3DE1rSFZ03+Qf5qd9bhyfT7oJGc= 2024-04-19T23:26:13.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3DE1rSFZ03+Qf5qd9bhyfT7oJGc=', 'Content-Length': '0'} 2024-04-19T23:26:13.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a2c7641715e61fd-006622fd95-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.974 DEBUG:boto:path=/aktlji-17/obj1 2024-04-19T23:26:13.974 DEBUG:boto:auth_path=/aktlji-17/obj1 2024-04-19T23:26:13.974 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.974 DEBUG:boto:Path: /aktlji-17/obj1 2024-04-19T23:26:13.975 DEBUG:boto:Data: 2024-04-19T23:26:13.975 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.975 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:13.975 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.975 DEBUG:boto:Params: {} 2024-04-19T23:26:13.975 DEBUG:boto:Token: None 2024-04-19T23:26:13.975 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-17/obj1 2024-04-19T23:26:13.976 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O5Q/PYjortp6nyeGsLV0fIfQsxQ= 2024-04-19T23:26:13.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O5Q/PYjortp6nyeGsLV0fIfQsxQ='} 2024-04-19T23:26:13.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5a7a6c64cd09066-006622fd95-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.986 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:13.986 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:13.986 DEBUG:boto:Method: HEAD 2024-04-19T23:26:13.986 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:13.986 DEBUG:boto:Data: 2024-04-19T23:26:13.986 DEBUG:boto:Headers: {} 2024-04-19T23:26:13.986 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:13.986 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.987 DEBUG:boto:Params: {} 2024-04-19T23:26:13.987 DEBUG:boto:Token: None 2024-04-19T23:26:13.987 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:13 GMT /aktlji-17/ 2024-04-19T23:26:13.987 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3DE1rSFZ03+Qf5qd9bhyfT7oJGc= 2024-04-19T23:26:13.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3DE1rSFZ03+Qf5qd9bhyfT7oJGc=', 'Content-Length': '0'} 2024-04-19T23:26:13.989 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032c463672ba32824-006622fd95-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:13.990 DEBUG:boto:path=/aktlji-17/obj2 2024-04-19T23:26:13.990 DEBUG:boto:auth_path=/aktlji-17/obj2 2024-04-19T23:26:13.990 DEBUG:boto:Method: PUT 2024-04-19T23:26:13.990 DEBUG:boto:Path: /aktlji-17/obj2 2024-04-19T23:26:13.990 DEBUG:boto:Data: 2024-04-19T23:26:13.990 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:13.990 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:13.991 DEBUG:boto:Port: 8000 2024-04-19T23:26:13.991 DEBUG:boto:Params: {} 2024-04-19T23:26:13.991 DEBUG:boto:Token: None 2024-04-19T23:26:13.991 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:13 GMT /aktlji-17/obj2 2024-04-19T23:26:13.991 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tOo1ew6zRSuquO6wf46RXSfpMcw= 2024-04-19T23:26:13.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tOo1ew6zRSuquO6wf46RXSfpMcw='} 2024-04-19T23:26:14.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b77e07a2886a545-006622fd95-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.035 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.035 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.036 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.036 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.036 DEBUG:boto:Data: 2024-04-19T23:26:14.036 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.036 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.036 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.036 DEBUG:boto:Params: {} 2024-04-19T23:26:14.036 DEBUG:boto:Token: None 2024-04-19T23:26:14.036 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.037 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a52d588373699721-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.039 DEBUG:boto:path=/aktlji-17/obj3 2024-04-19T23:26:14.040 DEBUG:boto:auth_path=/aktlji-17/obj3 2024-04-19T23:26:14.040 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.040 DEBUG:boto:Path: /aktlji-17/obj3 2024-04-19T23:26:14.040 DEBUG:boto:Data: 2024-04-19T23:26:14.040 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.040 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.040 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.040 DEBUG:boto:Params: {} 2024-04-19T23:26:14.040 DEBUG:boto:Token: None 2024-04-19T23:26:14.041 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj3 2024-04-19T23:26:14.041 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hsbr4AqCH6ISSBsUs4mdddIQ5vU= 2024-04-19T23:26:14.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hsbr4AqCH6ISSBsUs4mdddIQ5vU='} 2024-04-19T23:26:14.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093159f5d94cf4bf0-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.048 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.048 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.048 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.048 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.049 DEBUG:boto:Data: 2024-04-19T23:26:14.049 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.049 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.049 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.049 DEBUG:boto:Params: {} 2024-04-19T23:26:14.049 DEBUG:boto:Token: None 2024-04-19T23:26:14.049 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.049 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.052 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a48d3e9fd7e26e94-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.052 DEBUG:boto:path=/aktlji-17/obj4 2024-04-19T23:26:14.052 DEBUG:boto:auth_path=/aktlji-17/obj4 2024-04-19T23:26:14.052 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.052 DEBUG:boto:Path: /aktlji-17/obj4 2024-04-19T23:26:14.052 DEBUG:boto:Data: 2024-04-19T23:26:14.052 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.052 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.052 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.053 DEBUG:boto:Params: {} 2024-04-19T23:26:14.053 DEBUG:boto:Token: None 2024-04-19T23:26:14.053 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj4 2024-04-19T23:26:14.053 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:S9NMizasssifnE8teLsG//49xBM= 2024-04-19T23:26:14.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:S9NMizasssifnE8teLsG//49xBM='} 2024-04-19T23:26:14.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b43e643207784d1-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.095 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.095 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.095 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.095 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.095 DEBUG:boto:Data: 2024-04-19T23:26:14.095 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.095 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.095 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.096 DEBUG:boto:Params: {} 2024-04-19T23:26:14.096 DEBUG:boto:Token: None 2024-04-19T23:26:14.096 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.096 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.099 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000802a67d6bf671682-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.099 DEBUG:boto:path=/aktlji-17/obj5 2024-04-19T23:26:14.099 DEBUG:boto:auth_path=/aktlji-17/obj5 2024-04-19T23:26:14.099 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.099 DEBUG:boto:Path: /aktlji-17/obj5 2024-04-19T23:26:14.100 DEBUG:boto:Data: 2024-04-19T23:26:14.100 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.100 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.100 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.100 DEBUG:boto:Params: {} 2024-04-19T23:26:14.100 DEBUG:boto:Token: None 2024-04-19T23:26:14.100 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj5 2024-04-19T23:26:14.100 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:E/WVdpfRhlP+4+HgfRVUJzhkZmY= 2024-04-19T23:26:14.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E/WVdpfRhlP+4+HgfRVUJzhkZmY='} 2024-04-19T23:26:14.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d2917dc35177d45-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.106 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.107 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.107 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.107 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.107 DEBUG:boto:Data: 2024-04-19T23:26:14.107 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.107 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.107 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.107 DEBUG:boto:Params: {} 2024-04-19T23:26:14.107 DEBUG:boto:Token: None 2024-04-19T23:26:14.107 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.108 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000320723f01d78811-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.110 DEBUG:boto:path=/aktlji-17/obj6 2024-04-19T23:26:14.110 DEBUG:boto:auth_path=/aktlji-17/obj6 2024-04-19T23:26:14.111 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.111 DEBUG:boto:Path: /aktlji-17/obj6 2024-04-19T23:26:14.111 DEBUG:boto:Data: 2024-04-19T23:26:14.111 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.111 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.111 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.111 DEBUG:boto:Params: {} 2024-04-19T23:26:14.111 DEBUG:boto:Token: None 2024-04-19T23:26:14.111 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj6 2024-04-19T23:26:14.111 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UIhaC3B/6P2ls+qXb9VTeRY+Fy8= 2024-04-19T23:26:14.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UIhaC3B/6P2ls+qXb9VTeRY+Fy8='} 2024-04-19T23:26:14.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b16160e21ae3650-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.155 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.155 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.155 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.155 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.155 DEBUG:boto:Data: 2024-04-19T23:26:14.155 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.155 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.155 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.155 DEBUG:boto:Params: {} 2024-04-19T23:26:14.155 DEBUG:boto:Token: None 2024-04-19T23:26:14.156 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.156 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a693bef5416f49df-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.158 DEBUG:boto:path=/aktlji-17/obj7 2024-04-19T23:26:14.159 DEBUG:boto:auth_path=/aktlji-17/obj7 2024-04-19T23:26:14.159 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.159 DEBUG:boto:Path: /aktlji-17/obj7 2024-04-19T23:26:14.159 DEBUG:boto:Data: 2024-04-19T23:26:14.159 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.159 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.159 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.159 DEBUG:boto:Params: {} 2024-04-19T23:26:14.159 DEBUG:boto:Token: None 2024-04-19T23:26:14.160 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj7 2024-04-19T23:26:14.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lJg9LClQiT1yhdsh9QI8o64GcNM= 2024-04-19T23:26:14.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lJg9LClQiT1yhdsh9QI8o64GcNM='} 2024-04-19T23:26:14.165 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000566433853f55944d-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.165 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.165 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.165 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.165 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.165 DEBUG:boto:Data: 2024-04-19T23:26:14.165 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.166 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.166 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.166 DEBUG:boto:Params: {} 2024-04-19T23:26:14.166 DEBUG:boto:Token: None 2024-04-19T23:26:14.166 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.166 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfef5db118260df0-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.169 DEBUG:boto:path=/aktlji-17/obj8 2024-04-19T23:26:14.169 DEBUG:boto:auth_path=/aktlji-17/obj8 2024-04-19T23:26:14.169 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.169 DEBUG:boto:Path: /aktlji-17/obj8 2024-04-19T23:26:14.169 DEBUG:boto:Data: 2024-04-19T23:26:14.169 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.169 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.169 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.169 DEBUG:boto:Params: {} 2024-04-19T23:26:14.170 DEBUG:boto:Token: None 2024-04-19T23:26:14.170 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj8 2024-04-19T23:26:14.170 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IIRMRdxDGfRJh8mT4Vdbp+AU+g0= 2024-04-19T23:26:14.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IIRMRdxDGfRJh8mT4Vdbp+AU+g0='} 2024-04-19T23:26:14.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bab453503ec51378-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.175 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.175 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.175 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.175 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.175 DEBUG:boto:Data: 2024-04-19T23:26:14.175 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.175 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.175 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.175 DEBUG:boto:Params: {} 2024-04-19T23:26:14.176 DEBUG:boto:Token: None 2024-04-19T23:26:14.176 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.176 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000722f483e33a45949-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.178 DEBUG:boto:path=/aktlji-17/obj9 2024-04-19T23:26:14.179 DEBUG:boto:auth_path=/aktlji-17/obj9 2024-04-19T23:26:14.179 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.179 DEBUG:boto:Path: /aktlji-17/obj9 2024-04-19T23:26:14.179 DEBUG:boto:Data: 2024-04-19T23:26:14.179 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.179 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.179 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.179 DEBUG:boto:Params: {} 2024-04-19T23:26:14.179 DEBUG:boto:Token: None 2024-04-19T23:26:14.180 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj9 2024-04-19T23:26:14.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SOD6aCLakjilIMrSrzIyhgEvKm8= 2024-04-19T23:26:14.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SOD6aCLakjilIMrSrzIyhgEvKm8='} 2024-04-19T23:26:14.222 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044bf0c11a267beab-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.223 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.223 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.223 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.223 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.223 DEBUG:boto:Data: 2024-04-19T23:26:14.223 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.223 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.223 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.223 DEBUG:boto:Params: {} 2024-04-19T23:26:14.224 DEBUG:boto:Token: None 2024-04-19T23:26:14.224 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.224 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4ffce0cd8654fab-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.226 DEBUG:boto:path=/aktlji-17/obj10 2024-04-19T23:26:14.227 DEBUG:boto:auth_path=/aktlji-17/obj10 2024-04-19T23:26:14.227 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.227 DEBUG:boto:Path: /aktlji-17/obj10 2024-04-19T23:26:14.227 DEBUG:boto:Data: 2024-04-19T23:26:14.227 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.227 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.227 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.227 DEBUG:boto:Params: {} 2024-04-19T23:26:14.227 DEBUG:boto:Token: None 2024-04-19T23:26:14.227 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj10 2024-04-19T23:26:14.228 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BYcUSPoBtWx5xd1q8/JHMdw+P+E= 2024-04-19T23:26:14.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BYcUSPoBtWx5xd1q8/JHMdw+P+E='} 2024-04-19T23:26:14.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f70781643f6edf6-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.233 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.233 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.234 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.234 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.234 DEBUG:boto:Data: 2024-04-19T23:26:14.234 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.234 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.234 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.234 DEBUG:boto:Params: {} 2024-04-19T23:26:14.234 DEBUG:boto:Token: None 2024-04-19T23:26:14.234 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.234 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.237 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2a80d1643496283-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.237 DEBUG:boto:path=/aktlji-17/obj11 2024-04-19T23:26:14.237 DEBUG:boto:auth_path=/aktlji-17/obj11 2024-04-19T23:26:14.238 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.238 DEBUG:boto:Path: /aktlji-17/obj11 2024-04-19T23:26:14.238 DEBUG:boto:Data: 2024-04-19T23:26:14.238 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.238 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.238 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.238 DEBUG:boto:Params: {} 2024-04-19T23:26:14.238 DEBUG:boto:Token: None 2024-04-19T23:26:14.238 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj11 2024-04-19T23:26:14.239 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JHS7wCQ52H1G94E4Ste7i17b7L4= 2024-04-19T23:26:14.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JHS7wCQ52H1G94E4Ste7i17b7L4='} 2024-04-19T23:26:14.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000049bbcd086a38c25-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.283 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.283 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.283 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.283 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.283 DEBUG:boto:Data: 2024-04-19T23:26:14.283 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.283 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.284 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.284 DEBUG:boto:Params: {} 2024-04-19T23:26:14.284 DEBUG:boto:Token: None 2024-04-19T23:26:14.284 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.284 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000443f6fd93093ac53-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.287 DEBUG:boto:path=/aktlji-17/obj12 2024-04-19T23:26:14.287 DEBUG:boto:auth_path=/aktlji-17/obj12 2024-04-19T23:26:14.287 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.287 DEBUG:boto:Path: /aktlji-17/obj12 2024-04-19T23:26:14.287 DEBUG:boto:Data: 2024-04-19T23:26:14.287 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.287 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.287 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.287 DEBUG:boto:Params: {} 2024-04-19T23:26:14.288 DEBUG:boto:Token: None 2024-04-19T23:26:14.288 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj12 2024-04-19T23:26:14.288 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZBhgwfk1CnflJA3LZrsiLQ++V5w= 2024-04-19T23:26:14.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZBhgwfk1CnflJA3LZrsiLQ++V5w='} 2024-04-19T23:26:14.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022aa2cd49a2259c7-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.293 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.293 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.293 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.293 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.293 DEBUG:boto:Data: 2024-04-19T23:26:14.293 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.293 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.293 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.293 DEBUG:boto:Params: {} 2024-04-19T23:26:14.293 DEBUG:boto:Token: None 2024-04-19T23:26:14.294 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.294 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.296 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca9233845ea31213-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.296 DEBUG:boto:path=/aktlji-17/obj13 2024-04-19T23:26:14.296 DEBUG:boto:auth_path=/aktlji-17/obj13 2024-04-19T23:26:14.297 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.297 DEBUG:boto:Path: /aktlji-17/obj13 2024-04-19T23:26:14.297 DEBUG:boto:Data: 2024-04-19T23:26:14.297 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.297 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.297 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.297 DEBUG:boto:Params: {} 2024-04-19T23:26:14.297 DEBUG:boto:Token: None 2024-04-19T23:26:14.297 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj13 2024-04-19T23:26:14.298 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zr1LNGipUDnLdRrhdcsWU6YvBHY= 2024-04-19T23:26:14.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zr1LNGipUDnLdRrhdcsWU6YvBHY='} 2024-04-19T23:26:14.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a797da1da684de1f-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.343 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.343 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.343 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.343 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.343 DEBUG:boto:Data: 2024-04-19T23:26:14.343 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.343 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.343 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.343 DEBUG:boto:Params: {} 2024-04-19T23:26:14.344 DEBUG:boto:Token: None 2024-04-19T23:26:14.344 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.346 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d9d9d703720740c-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.347 DEBUG:boto:path=/aktlji-17/obj14 2024-04-19T23:26:14.347 DEBUG:boto:auth_path=/aktlji-17/obj14 2024-04-19T23:26:14.347 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.347 DEBUG:boto:Path: /aktlji-17/obj14 2024-04-19T23:26:14.347 DEBUG:boto:Data: 2024-04-19T23:26:14.347 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.347 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.347 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.348 DEBUG:boto:Params: {} 2024-04-19T23:26:14.348 DEBUG:boto:Token: None 2024-04-19T23:26:14.348 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj14 2024-04-19T23:26:14.348 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2NZPmfyA0aCTTL+G+LP7S/lG5bE= 2024-04-19T23:26:14.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2NZPmfyA0aCTTL+G+LP7S/lG5bE='} 2024-04-19T23:26:14.353 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd16188396f1c35d-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.353 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.353 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.353 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.353 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.353 DEBUG:boto:Data: 2024-04-19T23:26:14.353 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.354 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.354 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.354 DEBUG:boto:Params: {} 2024-04-19T23:26:14.354 DEBUG:boto:Token: None 2024-04-19T23:26:14.354 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.354 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a182f76eb47f9d0-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.357 DEBUG:boto:path=/aktlji-17/obj15 2024-04-19T23:26:14.357 DEBUG:boto:auth_path=/aktlji-17/obj15 2024-04-19T23:26:14.357 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.357 DEBUG:boto:Path: /aktlji-17/obj15 2024-04-19T23:26:14.357 DEBUG:boto:Data: 2024-04-19T23:26:14.357 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.357 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.357 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.357 DEBUG:boto:Params: {} 2024-04-19T23:26:14.358 DEBUG:boto:Token: None 2024-04-19T23:26:14.358 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj15 2024-04-19T23:26:14.358 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RX1k3mpY57FOhLwM9R82OF8cBa0= 2024-04-19T23:26:14.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RX1k3mpY57FOhLwM9R82OF8cBa0='} 2024-04-19T23:26:14.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d8173f285c26493-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.403 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.403 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.403 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.403 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.403 DEBUG:boto:Data: 2024-04-19T23:26:14.403 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.403 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.403 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.403 DEBUG:boto:Params: {} 2024-04-19T23:26:14.403 DEBUG:boto:Token: None 2024-04-19T23:26:14.404 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.404 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c68a92fccd490ea0-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.406 DEBUG:boto:path=/aktlji-17/obj16 2024-04-19T23:26:14.407 DEBUG:boto:auth_path=/aktlji-17/obj16 2024-04-19T23:26:14.407 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.407 DEBUG:boto:Path: /aktlji-17/obj16 2024-04-19T23:26:14.407 DEBUG:boto:Data: 2024-04-19T23:26:14.407 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.407 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.407 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.407 DEBUG:boto:Params: {} 2024-04-19T23:26:14.407 DEBUG:boto:Token: None 2024-04-19T23:26:14.407 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj16 2024-04-19T23:26:14.408 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jA5MEK6WfPRYstoXiH2EcbE2gu0= 2024-04-19T23:26:14.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jA5MEK6WfPRYstoXiH2EcbE2gu0='} 2024-04-19T23:26:14.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea70a1d467624789-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.414 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.414 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.414 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.414 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.414 DEBUG:boto:Data: 2024-04-19T23:26:14.414 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.414 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.414 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.414 DEBUG:boto:Params: {} 2024-04-19T23:26:14.415 DEBUG:boto:Token: None 2024-04-19T23:26:14.415 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.415 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.417 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1b56e08f5c3d132-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.418 DEBUG:boto:path=/aktlji-17/obj17 2024-04-19T23:26:14.418 DEBUG:boto:auth_path=/aktlji-17/obj17 2024-04-19T23:26:14.418 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.418 DEBUG:boto:Path: /aktlji-17/obj17 2024-04-19T23:26:14.418 DEBUG:boto:Data: 2024-04-19T23:26:14.418 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.418 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.418 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.418 DEBUG:boto:Params: {} 2024-04-19T23:26:14.419 DEBUG:boto:Token: None 2024-04-19T23:26:14.419 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj17 2024-04-19T23:26:14.419 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m56esVs2YIioovLUDaWwmSFBJiM= 2024-04-19T23:26:14.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m56esVs2YIioovLUDaWwmSFBJiM='} 2024-04-19T23:26:14.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000777f77ce45291002-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.463 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.463 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.463 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.463 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.463 DEBUG:boto:Data: 2024-04-19T23:26:14.463 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.464 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.464 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.464 DEBUG:boto:Params: {} 2024-04-19T23:26:14.464 DEBUG:boto:Token: None 2024-04-19T23:26:14.464 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.464 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.467 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001aa57846606134dc-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.467 DEBUG:boto:path=/aktlji-17/obj18 2024-04-19T23:26:14.467 DEBUG:boto:auth_path=/aktlji-17/obj18 2024-04-19T23:26:14.467 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.467 DEBUG:boto:Path: /aktlji-17/obj18 2024-04-19T23:26:14.467 DEBUG:boto:Data: 2024-04-19T23:26:14.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.467 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.468 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.468 DEBUG:boto:Params: {} 2024-04-19T23:26:14.468 DEBUG:boto:Token: None 2024-04-19T23:26:14.468 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj18 2024-04-19T23:26:14.468 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:K7Mi6rl/M4NhWhkdhiiDWZT9A2w= 2024-04-19T23:26:14.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K7Mi6rl/M4NhWhkdhiiDWZT9A2w='} 2024-04-19T23:26:14.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043a2f93172a1a8c8-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.473 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.473 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.473 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.473 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.473 DEBUG:boto:Data: 2024-04-19T23:26:14.473 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.473 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.473 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.473 DEBUG:boto:Params: {} 2024-04-19T23:26:14.474 DEBUG:boto:Token: None 2024-04-19T23:26:14.474 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.474 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.476 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bc8e784d68c5998-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.476 DEBUG:boto:path=/aktlji-17/obj19 2024-04-19T23:26:14.476 DEBUG:boto:auth_path=/aktlji-17/obj19 2024-04-19T23:26:14.476 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.476 DEBUG:boto:Path: /aktlji-17/obj19 2024-04-19T23:26:14.477 DEBUG:boto:Data: 2024-04-19T23:26:14.477 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.477 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.477 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.477 DEBUG:boto:Params: {} 2024-04-19T23:26:14.477 DEBUG:boto:Token: None 2024-04-19T23:26:14.477 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj19 2024-04-19T23:26:14.477 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AwqqdMqK9sEO9kCsNgj1KLsaug8= 2024-04-19T23:26:14.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AwqqdMqK9sEO9kCsNgj1KLsaug8='} 2024-04-19T23:26:14.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0581b798f8ae649-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.523 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.523 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.523 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.523 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.523 DEBUG:boto:Data: 2024-04-19T23:26:14.523 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.523 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.523 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.523 DEBUG:boto:Params: {} 2024-04-19T23:26:14.523 DEBUG:boto:Token: None 2024-04-19T23:26:14.524 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.524 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015490f2889e072b0-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.526 DEBUG:boto:path=/aktlji-17/obj20 2024-04-19T23:26:14.527 DEBUG:boto:auth_path=/aktlji-17/obj20 2024-04-19T23:26:14.527 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.527 DEBUG:boto:Path: /aktlji-17/obj20 2024-04-19T23:26:14.527 DEBUG:boto:Data: 2024-04-19T23:26:14.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.527 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.527 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.527 DEBUG:boto:Params: {} 2024-04-19T23:26:14.528 DEBUG:boto:Token: None 2024-04-19T23:26:14.528 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj20 2024-04-19T23:26:14.528 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MxXmhkLI9WD0ngZoiqdqXoVpIJo= 2024-04-19T23:26:14.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MxXmhkLI9WD0ngZoiqdqXoVpIJo='} 2024-04-19T23:26:14.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ef7a0eb2aed67fd-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.535 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.535 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.535 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.535 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.535 DEBUG:boto:Data: 2024-04-19T23:26:14.536 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.536 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.536 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.536 DEBUG:boto:Params: {} 2024-04-19T23:26:14.536 DEBUG:boto:Token: None 2024-04-19T23:26:14.536 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.536 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.539 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000898537f0ec24d744-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.539 DEBUG:boto:path=/aktlji-17/obj21 2024-04-19T23:26:14.539 DEBUG:boto:auth_path=/aktlji-17/obj21 2024-04-19T23:26:14.539 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.539 DEBUG:boto:Path: /aktlji-17/obj21 2024-04-19T23:26:14.540 DEBUG:boto:Data: 2024-04-19T23:26:14.540 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.540 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.540 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.540 DEBUG:boto:Params: {} 2024-04-19T23:26:14.540 DEBUG:boto:Token: None 2024-04-19T23:26:14.540 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj21 2024-04-19T23:26:14.541 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:G27s6TOna8BbxiEDWeas2GXF7gc= 2024-04-19T23:26:14.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:G27s6TOna8BbxiEDWeas2GXF7gc='} 2024-04-19T23:26:14.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cc3c4fa5e64753f-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.583 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.583 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.583 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.583 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.583 DEBUG:boto:Data: 2024-04-19T23:26:14.583 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.583 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.583 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.583 DEBUG:boto:Params: {} 2024-04-19T23:26:14.584 DEBUG:boto:Token: None 2024-04-19T23:26:14.584 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.584 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa23b30c72f3e2d2-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.587 DEBUG:boto:path=/aktlji-17/obj22 2024-04-19T23:26:14.587 DEBUG:boto:auth_path=/aktlji-17/obj22 2024-04-19T23:26:14.587 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.587 DEBUG:boto:Path: /aktlji-17/obj22 2024-04-19T23:26:14.587 DEBUG:boto:Data: 2024-04-19T23:26:14.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.587 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.587 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.587 DEBUG:boto:Params: {} 2024-04-19T23:26:14.588 DEBUG:boto:Token: None 2024-04-19T23:26:14.588 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj22 2024-04-19T23:26:14.588 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uXM8ijlmJHaM2e9ECdRCiMzTYQo= 2024-04-19T23:26:14.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uXM8ijlmJHaM2e9ECdRCiMzTYQo='} 2024-04-19T23:26:14.593 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa3e92a6db25ac18-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.593 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.593 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.594 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.594 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.594 DEBUG:boto:Data: 2024-04-19T23:26:14.594 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.594 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.594 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.594 DEBUG:boto:Params: {} 2024-04-19T23:26:14.594 DEBUG:boto:Token: None 2024-04-19T23:26:14.594 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.594 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f9f1cbe0fb5333f-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.597 DEBUG:boto:path=/aktlji-17/obj23 2024-04-19T23:26:14.597 DEBUG:boto:auth_path=/aktlji-17/obj23 2024-04-19T23:26:14.597 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.598 DEBUG:boto:Path: /aktlji-17/obj23 2024-04-19T23:26:14.598 DEBUG:boto:Data: 2024-04-19T23:26:14.598 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.598 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.598 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.598 DEBUG:boto:Params: {} 2024-04-19T23:26:14.598 DEBUG:boto:Token: None 2024-04-19T23:26:14.598 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj23 2024-04-19T23:26:14.599 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/9JBsknAhtptiIHRR87M7//0uc4= 2024-04-19T23:26:14.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/9JBsknAhtptiIHRR87M7//0uc4='} 2024-04-19T23:26:14.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d96137e395d7931-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.643 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.643 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.643 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.644 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.644 DEBUG:boto:Data: 2024-04-19T23:26:14.644 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.644 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.644 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.644 DEBUG:boto:Params: {} 2024-04-19T23:26:14.644 DEBUG:boto:Token: None 2024-04-19T23:26:14.644 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.644 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004be0ea7206ff7c9-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.647 DEBUG:boto:path=/aktlji-17/obj24 2024-04-19T23:26:14.647 DEBUG:boto:auth_path=/aktlji-17/obj24 2024-04-19T23:26:14.647 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.647 DEBUG:boto:Path: /aktlji-17/obj24 2024-04-19T23:26:14.647 DEBUG:boto:Data: 2024-04-19T23:26:14.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.647 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.648 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.648 DEBUG:boto:Params: {} 2024-04-19T23:26:14.648 DEBUG:boto:Token: None 2024-04-19T23:26:14.648 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj24 2024-04-19T23:26:14.648 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qF+cXbeQfRwxp2rDFtmu6tnKIOk= 2024-04-19T23:26:14.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qF+cXbeQfRwxp2rDFtmu6tnKIOk='} 2024-04-19T23:26:14.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000363c59e60f736969-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.653 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.653 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.654 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.654 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.654 DEBUG:boto:Data: 2024-04-19T23:26:14.654 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.654 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.654 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.654 DEBUG:boto:Params: {} 2024-04-19T23:26:14.654 DEBUG:boto:Token: None 2024-04-19T23:26:14.654 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.655 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.657 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000494262e4a5ccd54a-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.657 DEBUG:boto:path=/aktlji-17/obj25 2024-04-19T23:26:14.657 DEBUG:boto:auth_path=/aktlji-17/obj25 2024-04-19T23:26:14.657 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.658 DEBUG:boto:Path: /aktlji-17/obj25 2024-04-19T23:26:14.658 DEBUG:boto:Data: 2024-04-19T23:26:14.658 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.658 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.658 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.658 DEBUG:boto:Params: {} 2024-04-19T23:26:14.658 DEBUG:boto:Token: None 2024-04-19T23:26:14.658 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj25 2024-04-19T23:26:14.658 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UPZxTzE0qveDKb29vcE0m57i6BM= 2024-04-19T23:26:14.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UPZxTzE0qveDKb29vcE0m57i6BM='} 2024-04-19T23:26:14.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000396d51068a107d94-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.703 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.703 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.703 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.703 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.703 DEBUG:boto:Data: 2024-04-19T23:26:14.703 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.703 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.704 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.704 DEBUG:boto:Params: {} 2024-04-19T23:26:14.704 DEBUG:boto:Token: None 2024-04-19T23:26:14.704 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001580667f6b91370f-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.707 DEBUG:boto:path=/aktlji-17/obj26 2024-04-19T23:26:14.707 DEBUG:boto:auth_path=/aktlji-17/obj26 2024-04-19T23:26:14.707 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.707 DEBUG:boto:Path: /aktlji-17/obj26 2024-04-19T23:26:14.707 DEBUG:boto:Data: 2024-04-19T23:26:14.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.707 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.707 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.707 DEBUG:boto:Params: {} 2024-04-19T23:26:14.708 DEBUG:boto:Token: None 2024-04-19T23:26:14.708 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj26 2024-04-19T23:26:14.708 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Oo223t4cBZw6tFw4tLYUbKzXiF8= 2024-04-19T23:26:14.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Oo223t4cBZw6tFw4tLYUbKzXiF8='} 2024-04-19T23:26:14.713 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000503b2a1d77a89deb-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.713 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.713 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.713 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.713 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.713 DEBUG:boto:Data: 2024-04-19T23:26:14.713 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.713 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.713 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.713 DEBUG:boto:Params: {} 2024-04-19T23:26:14.714 DEBUG:boto:Token: None 2024-04-19T23:26:14.714 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.714 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061512adaaa65deb9-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.717 DEBUG:boto:path=/aktlji-17/obj27 2024-04-19T23:26:14.717 DEBUG:boto:auth_path=/aktlji-17/obj27 2024-04-19T23:26:14.717 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.717 DEBUG:boto:Path: /aktlji-17/obj27 2024-04-19T23:26:14.717 DEBUG:boto:Data: 2024-04-19T23:26:14.717 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.717 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.717 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.717 DEBUG:boto:Params: {} 2024-04-19T23:26:14.717 DEBUG:boto:Token: None 2024-04-19T23:26:14.718 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj27 2024-04-19T23:26:14.718 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6wJR3sfJ2qH5gtrk5TiOXCluQfE= 2024-04-19T23:26:14.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6wJR3sfJ2qH5gtrk5TiOXCluQfE='} 2024-04-19T23:26:14.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b8e099b25fbc48d-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.763 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.763 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.763 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.763 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.763 DEBUG:boto:Data: 2024-04-19T23:26:14.763 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.763 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.763 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.763 DEBUG:boto:Params: {} 2024-04-19T23:26:14.764 DEBUG:boto:Token: None 2024-04-19T23:26:14.764 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.764 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.767 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a60767afc8493da-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.767 DEBUG:boto:path=/aktlji-17/obj28 2024-04-19T23:26:14.767 DEBUG:boto:auth_path=/aktlji-17/obj28 2024-04-19T23:26:14.768 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.768 DEBUG:boto:Path: /aktlji-17/obj28 2024-04-19T23:26:14.768 DEBUG:boto:Data: 2024-04-19T23:26:14.768 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.768 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.768 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.768 DEBUG:boto:Params: {} 2024-04-19T23:26:14.768 DEBUG:boto:Token: None 2024-04-19T23:26:14.768 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj28 2024-04-19T23:26:14.769 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:s09QlKKeU8SGZvjEr5PNGJpmhJk= 2024-04-19T23:26:14.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:s09QlKKeU8SGZvjEr5PNGJpmhJk='} 2024-04-19T23:26:14.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001032eb7b3d3854d1-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.774 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.774 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.774 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.774 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.774 DEBUG:boto:Data: 2024-04-19T23:26:14.774 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.774 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.774 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.774 DEBUG:boto:Params: {} 2024-04-19T23:26:14.775 DEBUG:boto:Token: None 2024-04-19T23:26:14.775 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.775 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002760a040d3efffd4-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.777 DEBUG:boto:path=/aktlji-17/obj29 2024-04-19T23:26:14.778 DEBUG:boto:auth_path=/aktlji-17/obj29 2024-04-19T23:26:14.778 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.778 DEBUG:boto:Path: /aktlji-17/obj29 2024-04-19T23:26:14.778 DEBUG:boto:Data: 2024-04-19T23:26:14.778 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.778 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.778 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.778 DEBUG:boto:Params: {} 2024-04-19T23:26:14.778 DEBUG:boto:Token: None 2024-04-19T23:26:14.779 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj29 2024-04-19T23:26:14.779 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ImXdDxY/yoNId4EplSCSvIw+eqM= 2024-04-19T23:26:14.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ImXdDxY/yoNId4EplSCSvIw+eqM='} 2024-04-19T23:26:14.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001220b7457b3dded5-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.823 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.823 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.823 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.823 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.823 DEBUG:boto:Data: 2024-04-19T23:26:14.823 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.823 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.824 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.824 DEBUG:boto:Params: {} 2024-04-19T23:26:14.824 DEBUG:boto:Token: None 2024-04-19T23:26:14.824 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060b554843eaba598-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.827 DEBUG:boto:path=/aktlji-17/obj30 2024-04-19T23:26:14.827 DEBUG:boto:auth_path=/aktlji-17/obj30 2024-04-19T23:26:14.827 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.827 DEBUG:boto:Path: /aktlji-17/obj30 2024-04-19T23:26:14.827 DEBUG:boto:Data: 2024-04-19T23:26:14.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.827 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.828 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.828 DEBUG:boto:Params: {} 2024-04-19T23:26:14.828 DEBUG:boto:Token: None 2024-04-19T23:26:14.828 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj30 2024-04-19T23:26:14.828 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MTWGFuGuxuV442YHUcpCsK+g0Cc= 2024-04-19T23:26:14.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MTWGFuGuxuV442YHUcpCsK+g0Cc='} 2024-04-19T23:26:14.833 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000dee92bd1c0d7192-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.833 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.833 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.833 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.833 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.834 DEBUG:boto:Data: 2024-04-19T23:26:14.834 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.834 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.834 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.834 DEBUG:boto:Params: {} 2024-04-19T23:26:14.834 DEBUG:boto:Token: None 2024-04-19T23:26:14.834 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.834 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df497eb8271a805b-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.837 DEBUG:boto:path=/aktlji-17/obj31 2024-04-19T23:26:14.838 DEBUG:boto:auth_path=/aktlji-17/obj31 2024-04-19T23:26:14.838 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.838 DEBUG:boto:Path: /aktlji-17/obj31 2024-04-19T23:26:14.838 DEBUG:boto:Data: 2024-04-19T23:26:14.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.838 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.838 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.838 DEBUG:boto:Params: {} 2024-04-19T23:26:14.838 DEBUG:boto:Token: None 2024-04-19T23:26:14.839 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj31 2024-04-19T23:26:14.839 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BMQl0x8o0iwb6WUa7AA6rgB1BXE= 2024-04-19T23:26:14.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BMQl0x8o0iwb6WUa7AA6rgB1BXE='} 2024-04-19T23:26:14.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1d0437998be04e9-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.887 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.887 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.887 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.887 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.887 DEBUG:boto:Data: 2024-04-19T23:26:14.887 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.887 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.887 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.888 DEBUG:boto:Params: {} 2024-04-19T23:26:14.888 DEBUG:boto:Token: None 2024-04-19T23:26:14.888 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.888 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c168e97897e12ef-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.891 DEBUG:boto:path=/aktlji-17/obj32 2024-04-19T23:26:14.891 DEBUG:boto:auth_path=/aktlji-17/obj32 2024-04-19T23:26:14.891 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.891 DEBUG:boto:Path: /aktlji-17/obj32 2024-04-19T23:26:14.891 DEBUG:boto:Data: 2024-04-19T23:26:14.892 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.892 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.892 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.892 DEBUG:boto:Params: {} 2024-04-19T23:26:14.892 DEBUG:boto:Token: None 2024-04-19T23:26:14.892 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj32 2024-04-19T23:26:14.892 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O4TCVpu+aHGELesGdbva2Mk8QL4= 2024-04-19T23:26:14.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O4TCVpu+aHGELesGdbva2Mk8QL4='} 2024-04-19T23:26:14.897 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5a2eba7dfb2e4a8-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.897 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.897 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.897 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.897 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.897 DEBUG:boto:Data: 2024-04-19T23:26:14.898 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.898 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.898 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.898 DEBUG:boto:Params: {} 2024-04-19T23:26:14.898 DEBUG:boto:Token: None 2024-04-19T23:26:14.898 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.898 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1680c11a643c86e-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.901 DEBUG:boto:path=/aktlji-17/obj33 2024-04-19T23:26:14.901 DEBUG:boto:auth_path=/aktlji-17/obj33 2024-04-19T23:26:14.901 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.901 DEBUG:boto:Path: /aktlji-17/obj33 2024-04-19T23:26:14.901 DEBUG:boto:Data: 2024-04-19T23:26:14.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.901 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.901 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.902 DEBUG:boto:Params: {} 2024-04-19T23:26:14.902 DEBUG:boto:Token: None 2024-04-19T23:26:14.902 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj33 2024-04-19T23:26:14.902 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0R0OTKUWWU8+isymrXCIGulCOt0= 2024-04-19T23:26:14.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0R0OTKUWWU8+isymrXCIGulCOt0='} 2024-04-19T23:26:14.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009dc832288d8be83d-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.947 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.947 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.947 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.947 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.947 DEBUG:boto:Data: 2024-04-19T23:26:14.947 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.947 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.947 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.948 DEBUG:boto:Params: {} 2024-04-19T23:26:14.948 DEBUG:boto:Token: None 2024-04-19T23:26:14.948 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.948 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cb214f72c357799-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.951 DEBUG:boto:path=/aktlji-17/obj34 2024-04-19T23:26:14.951 DEBUG:boto:auth_path=/aktlji-17/obj34 2024-04-19T23:26:14.951 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.951 DEBUG:boto:Path: /aktlji-17/obj34 2024-04-19T23:26:14.951 DEBUG:boto:Data: 2024-04-19T23:26:14.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.951 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.951 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.952 DEBUG:boto:Params: {} 2024-04-19T23:26:14.952 DEBUG:boto:Token: None 2024-04-19T23:26:14.952 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj34 2024-04-19T23:26:14.952 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dtgm+JQF68bxvGKgBif0kwmpUF0= 2024-04-19T23:26:14.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dtgm+JQF68bxvGKgBif0kwmpUF0='} 2024-04-19T23:26:14.957 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2cbd0953aaa346d-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.957 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:14.957 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:14.957 DEBUG:boto:Method: HEAD 2024-04-19T23:26:14.957 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:14.957 DEBUG:boto:Data: 2024-04-19T23:26:14.957 DEBUG:boto:Headers: {} 2024-04-19T23:26:14.958 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.958 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.958 DEBUG:boto:Params: {} 2024-04-19T23:26:14.958 DEBUG:boto:Token: None 2024-04-19T23:26:14.958 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/ 2024-04-19T23:26:14.958 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-19T23:26:14.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-19T23:26:14.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6726fa94c3bd26c-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:14.961 DEBUG:boto:path=/aktlji-17/obj35 2024-04-19T23:26:14.961 DEBUG:boto:auth_path=/aktlji-17/obj35 2024-04-19T23:26:14.961 DEBUG:boto:Method: PUT 2024-04-19T23:26:14.961 DEBUG:boto:Path: /aktlji-17/obj35 2024-04-19T23:26:14.961 DEBUG:boto:Data: 2024-04-19T23:26:14.961 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:14.961 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:14.961 DEBUG:boto:Port: 8000 2024-04-19T23:26:14.962 DEBUG:boto:Params: {} 2024-04-19T23:26:14.962 DEBUG:boto:Token: None 2024-04-19T23:26:14.962 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:14 GMT /aktlji-17/obj35 2024-04-19T23:26:14.962 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u58gI82upxJpHbNVcivcFirnrtc= 2024-04-19T23:26:14.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u58gI82upxJpHbNVcivcFirnrtc='} 2024-04-19T23:26:15.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c866024420c4006-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.011 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.011 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.011 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.011 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.011 DEBUG:boto:Data: 2024-04-19T23:26:15.011 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.011 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.011 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.012 DEBUG:boto:Params: {} 2024-04-19T23:26:15.012 DEBUG:boto:Token: None 2024-04-19T23:26:15.012 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.012 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c181f621709ef72-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.015 DEBUG:boto:path=/aktlji-17/obj36 2024-04-19T23:26:15.015 DEBUG:boto:auth_path=/aktlji-17/obj36 2024-04-19T23:26:15.015 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.015 DEBUG:boto:Path: /aktlji-17/obj36 2024-04-19T23:26:15.015 DEBUG:boto:Data: 2024-04-19T23:26:15.016 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.016 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.016 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.016 DEBUG:boto:Params: {} 2024-04-19T23:26:15.016 DEBUG:boto:Token: None 2024-04-19T23:26:15.017 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj36 2024-04-19T23:26:15.017 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L10kwF74t0F1UneocU39NZDc9SU= 2024-04-19T23:26:15.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L10kwF74t0F1UneocU39NZDc9SU='} 2024-04-19T23:26:15.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb08ecaa41b2c7e3-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.022 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.022 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.022 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.022 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.022 DEBUG:boto:Data: 2024-04-19T23:26:15.022 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.022 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.022 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.022 DEBUG:boto:Params: {} 2024-04-19T23:26:15.023 DEBUG:boto:Token: None 2024-04-19T23:26:15.023 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.023 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.025 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098629b3781f0ac64-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.025 DEBUG:boto:path=/aktlji-17/obj37 2024-04-19T23:26:15.026 DEBUG:boto:auth_path=/aktlji-17/obj37 2024-04-19T23:26:15.026 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.026 DEBUG:boto:Path: /aktlji-17/obj37 2024-04-19T23:26:15.026 DEBUG:boto:Data: 2024-04-19T23:26:15.026 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.026 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.026 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.026 DEBUG:boto:Params: {} 2024-04-19T23:26:15.026 DEBUG:boto:Token: None 2024-04-19T23:26:15.027 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj37 2024-04-19T23:26:15.027 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+EOMmiFwukcCzlPxwGke+1M2bRY= 2024-04-19T23:26:15.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+EOMmiFwukcCzlPxwGke+1M2bRY='} 2024-04-19T23:26:15.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f169253e9679968-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.071 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.071 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.071 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.071 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.071 DEBUG:boto:Data: 2024-04-19T23:26:15.071 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.071 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.071 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.071 DEBUG:boto:Params: {} 2024-04-19T23:26:15.072 DEBUG:boto:Token: None 2024-04-19T23:26:15.072 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.072 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.074 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7da0798b6ed4d98-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.075 DEBUG:boto:path=/aktlji-17/obj38 2024-04-19T23:26:15.075 DEBUG:boto:auth_path=/aktlji-17/obj38 2024-04-19T23:26:15.075 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.075 DEBUG:boto:Path: /aktlji-17/obj38 2024-04-19T23:26:15.075 DEBUG:boto:Data: 2024-04-19T23:26:15.075 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.075 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.075 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.075 DEBUG:boto:Params: {} 2024-04-19T23:26:15.076 DEBUG:boto:Token: None 2024-04-19T23:26:15.076 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj38 2024-04-19T23:26:15.076 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JeBN1NTFEcYom6agK3gwWNIQ9M4= 2024-04-19T23:26:15.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JeBN1NTFEcYom6agK3gwWNIQ9M4='} 2024-04-19T23:26:15.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009165b32c1217af66-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.081 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.081 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.081 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.081 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.081 DEBUG:boto:Data: 2024-04-19T23:26:15.081 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.081 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.081 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.081 DEBUG:boto:Params: {} 2024-04-19T23:26:15.081 DEBUG:boto:Token: None 2024-04-19T23:26:15.082 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.082 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097048091f51a46a6-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.084 DEBUG:boto:path=/aktlji-17/obj39 2024-04-19T23:26:15.084 DEBUG:boto:auth_path=/aktlji-17/obj39 2024-04-19T23:26:15.085 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.085 DEBUG:boto:Path: /aktlji-17/obj39 2024-04-19T23:26:15.085 DEBUG:boto:Data: 2024-04-19T23:26:15.085 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.085 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.085 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.085 DEBUG:boto:Params: {} 2024-04-19T23:26:15.085 DEBUG:boto:Token: None 2024-04-19T23:26:15.085 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj39 2024-04-19T23:26:15.086 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HyMYVoqDJbfi25ca5zqcpgumrCk= 2024-04-19T23:26:15.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HyMYVoqDJbfi25ca5zqcpgumrCk='} 2024-04-19T23:26:15.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfd7eb2f31c6c053-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.135 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.135 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.135 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.135 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.135 DEBUG:boto:Data: 2024-04-19T23:26:15.135 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.135 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.135 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.136 DEBUG:boto:Params: {} 2024-04-19T23:26:15.136 DEBUG:boto:Token: None 2024-04-19T23:26:15.136 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.136 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.139 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001573993fdad66427-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.139 DEBUG:boto:path=/aktlji-17/obj40 2024-04-19T23:26:15.139 DEBUG:boto:auth_path=/aktlji-17/obj40 2024-04-19T23:26:15.139 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.139 DEBUG:boto:Path: /aktlji-17/obj40 2024-04-19T23:26:15.140 DEBUG:boto:Data: 2024-04-19T23:26:15.140 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.140 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.140 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.140 DEBUG:boto:Params: {} 2024-04-19T23:26:15.140 DEBUG:boto:Token: None 2024-04-19T23:26:15.140 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj40 2024-04-19T23:26:15.140 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uoT0qpq4/JrCeX1QYSnp9O/WjXo= 2024-04-19T23:26:15.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uoT0qpq4/JrCeX1QYSnp9O/WjXo='} 2024-04-19T23:26:15.145 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbfdafa56839e87b-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.145 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.146 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.146 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.146 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.146 DEBUG:boto:Data: 2024-04-19T23:26:15.146 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.146 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.146 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.146 DEBUG:boto:Params: {} 2024-04-19T23:26:15.146 DEBUG:boto:Token: None 2024-04-19T23:26:15.147 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.147 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008ca024969909069-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.150 DEBUG:boto:path=/aktlji-17/obj41 2024-04-19T23:26:15.150 DEBUG:boto:auth_path=/aktlji-17/obj41 2024-04-19T23:26:15.150 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.150 DEBUG:boto:Path: /aktlji-17/obj41 2024-04-19T23:26:15.150 DEBUG:boto:Data: 2024-04-19T23:26:15.150 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.150 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.151 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.151 DEBUG:boto:Params: {} 2024-04-19T23:26:15.151 DEBUG:boto:Token: None 2024-04-19T23:26:15.151 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj41 2024-04-19T23:26:15.151 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shEuJ1l9pdfkqDWo/7R7JGkr8Fg= 2024-04-19T23:26:15.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shEuJ1l9pdfkqDWo/7R7JGkr8Fg='} 2024-04-19T23:26:15.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1e0f704e9726ea0-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.199 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.199 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.199 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.199 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.199 DEBUG:boto:Data: 2024-04-19T23:26:15.199 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.200 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.200 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.200 DEBUG:boto:Params: {} 2024-04-19T23:26:15.200 DEBUG:boto:Token: None 2024-04-19T23:26:15.200 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.200 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddd84036c96228f1-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.203 DEBUG:boto:path=/aktlji-17/obj42 2024-04-19T23:26:15.203 DEBUG:boto:auth_path=/aktlji-17/obj42 2024-04-19T23:26:15.203 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.204 DEBUG:boto:Path: /aktlji-17/obj42 2024-04-19T23:26:15.204 DEBUG:boto:Data: 2024-04-19T23:26:15.204 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.204 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.204 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.204 DEBUG:boto:Params: {} 2024-04-19T23:26:15.204 DEBUG:boto:Token: None 2024-04-19T23:26:15.204 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj42 2024-04-19T23:26:15.204 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jM1ZO3iNjFwhZUV6lEWpHZTiyaM= 2024-04-19T23:26:15.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jM1ZO3iNjFwhZUV6lEWpHZTiyaM='} 2024-04-19T23:26:15.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6a8866c0983ee5c-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.209 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.209 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.209 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.210 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.210 DEBUG:boto:Data: 2024-04-19T23:26:15.210 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.210 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.210 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.211 DEBUG:boto:Params: {} 2024-04-19T23:26:15.211 DEBUG:boto:Token: None 2024-04-19T23:26:15.211 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.212 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c2563b6a4d68af4-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.215 DEBUG:boto:path=/aktlji-17/obj43 2024-04-19T23:26:15.215 DEBUG:boto:auth_path=/aktlji-17/obj43 2024-04-19T23:26:15.215 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.215 DEBUG:boto:Path: /aktlji-17/obj43 2024-04-19T23:26:15.215 DEBUG:boto:Data: 2024-04-19T23:26:15.216 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.216 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.216 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.216 DEBUG:boto:Params: {} 2024-04-19T23:26:15.216 DEBUG:boto:Token: None 2024-04-19T23:26:15.217 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj43 2024-04-19T23:26:15.217 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hi3EG6MKOAJNkVsRtha3J58976I= 2024-04-19T23:26:15.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hi3EG6MKOAJNkVsRtha3J58976I='} 2024-04-19T23:26:15.267 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040fc93da88acd5ea-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.267 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.268 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.268 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.268 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.268 DEBUG:boto:Data: 2024-04-19T23:26:15.268 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.268 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.269 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.269 DEBUG:boto:Params: {} 2024-04-19T23:26:15.270 DEBUG:boto:Token: None 2024-04-19T23:26:15.270 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.270 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000483c030b96ea5cc-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.273 DEBUG:boto:path=/aktlji-17/obj44 2024-04-19T23:26:15.274 DEBUG:boto:auth_path=/aktlji-17/obj44 2024-04-19T23:26:15.274 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.274 DEBUG:boto:Path: /aktlji-17/obj44 2024-04-19T23:26:15.274 DEBUG:boto:Data: 2024-04-19T23:26:15.274 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.274 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.274 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.274 DEBUG:boto:Params: {} 2024-04-19T23:26:15.275 DEBUG:boto:Token: None 2024-04-19T23:26:15.275 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj44 2024-04-19T23:26:15.275 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/hE6Jc9JM5ic02eFaulJ5T3965A= 2024-04-19T23:26:15.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/hE6Jc9JM5ic02eFaulJ5T3965A='} 2024-04-19T23:26:15.281 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044aa7c2516d01eb9-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.281 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.281 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.281 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.281 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.281 DEBUG:boto:Data: 2024-04-19T23:26:15.282 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.282 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.282 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.282 DEBUG:boto:Params: {} 2024-04-19T23:26:15.282 DEBUG:boto:Token: None 2024-04-19T23:26:15.282 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.282 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000431bff69b9cc1c6d-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.285 DEBUG:boto:path=/aktlji-17/obj45 2024-04-19T23:26:15.285 DEBUG:boto:auth_path=/aktlji-17/obj45 2024-04-19T23:26:15.285 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.285 DEBUG:boto:Path: /aktlji-17/obj45 2024-04-19T23:26:15.285 DEBUG:boto:Data: 2024-04-19T23:26:15.285 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.285 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.286 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.286 DEBUG:boto:Params: {} 2024-04-19T23:26:15.286 DEBUG:boto:Token: None 2024-04-19T23:26:15.286 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj45 2024-04-19T23:26:15.286 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YCviTTsyhptBvtExr2bhUYeM7Ww= 2024-04-19T23:26:15.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YCviTTsyhptBvtExr2bhUYeM7Ww='} 2024-04-19T23:26:15.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4c7944fe2999c57-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.291 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.291 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.291 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.291 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.292 DEBUG:boto:Data: 2024-04-19T23:26:15.292 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.292 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.292 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.292 DEBUG:boto:Params: {} 2024-04-19T23:26:15.292 DEBUG:boto:Token: None 2024-04-19T23:26:15.292 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.292 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b308badc38279e12-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.295 DEBUG:boto:path=/aktlji-17/obj46 2024-04-19T23:26:15.295 DEBUG:boto:auth_path=/aktlji-17/obj46 2024-04-19T23:26:15.295 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.295 DEBUG:boto:Path: /aktlji-17/obj46 2024-04-19T23:26:15.295 DEBUG:boto:Data: 2024-04-19T23:26:15.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.295 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.296 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.296 DEBUG:boto:Params: {} 2024-04-19T23:26:15.296 DEBUG:boto:Token: None 2024-04-19T23:26:15.296 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj46 2024-04-19T23:26:15.296 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:No711zyjEkOkleO/h+12gLFN3LM= 2024-04-19T23:26:15.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:No711zyjEkOkleO/h+12gLFN3LM='} 2024-04-19T23:26:15.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011dae0c8aa4be76b-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.339 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.339 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.339 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.339 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.339 DEBUG:boto:Data: 2024-04-19T23:26:15.339 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.339 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.339 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.339 DEBUG:boto:Params: {} 2024-04-19T23:26:15.340 DEBUG:boto:Token: None 2024-04-19T23:26:15.340 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.340 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ae0fa1085007024-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.343 DEBUG:boto:path=/aktlji-17/obj47 2024-04-19T23:26:15.343 DEBUG:boto:auth_path=/aktlji-17/obj47 2024-04-19T23:26:15.343 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.343 DEBUG:boto:Path: /aktlji-17/obj47 2024-04-19T23:26:15.343 DEBUG:boto:Data: 2024-04-19T23:26:15.343 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.343 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.344 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.344 DEBUG:boto:Params: {} 2024-04-19T23:26:15.344 DEBUG:boto:Token: None 2024-04-19T23:26:15.344 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj47 2024-04-19T23:26:15.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zpmKlmIl2Aj8AwT3uLjn74Kmbo4= 2024-04-19T23:26:15.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zpmKlmIl2Aj8AwT3uLjn74Kmbo4='} 2024-04-19T23:26:15.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000736e407748dd87b0-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.349 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.349 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.349 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.349 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.349 DEBUG:boto:Data: 2024-04-19T23:26:15.349 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.349 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.349 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.349 DEBUG:boto:Params: {} 2024-04-19T23:26:15.349 DEBUG:boto:Token: None 2024-04-19T23:26:15.349 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.350 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0538a4317e9f89a-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.352 DEBUG:boto:path=/aktlji-17/obj48 2024-04-19T23:26:15.353 DEBUG:boto:auth_path=/aktlji-17/obj48 2024-04-19T23:26:15.353 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.353 DEBUG:boto:Path: /aktlji-17/obj48 2024-04-19T23:26:15.353 DEBUG:boto:Data: 2024-04-19T23:26:15.353 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.353 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.353 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.353 DEBUG:boto:Params: {} 2024-04-19T23:26:15.354 DEBUG:boto:Token: None 2024-04-19T23:26:15.354 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj48 2024-04-19T23:26:15.354 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lhRoUD1YXh17gVHzNA0XQfjgfa0= 2024-04-19T23:26:15.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lhRoUD1YXh17gVHzNA0XQfjgfa0='} 2024-04-19T23:26:15.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddec167df5c46e8b-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.403 DEBUG:boto:path=/aktlji-17/ 2024-04-19T23:26:15.403 DEBUG:boto:auth_path=/aktlji-17/ 2024-04-19T23:26:15.403 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.403 DEBUG:boto:Path: /aktlji-17/ 2024-04-19T23:26:15.403 DEBUG:boto:Data: 2024-04-19T23:26:15.404 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.404 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.404 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.404 DEBUG:boto:Params: {} 2024-04-19T23:26:15.404 DEBUG:boto:Token: None 2024-04-19T23:26:15.404 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/ 2024-04-19T23:26:15.405 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-19T23:26:15.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-19T23:26:15.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000682a5b11c8d9faa5-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.407 DEBUG:boto:path=/aktlji-17/obj49 2024-04-19T23:26:15.407 DEBUG:boto:auth_path=/aktlji-17/obj49 2024-04-19T23:26:15.408 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.408 DEBUG:boto:Path: /aktlji-17/obj49 2024-04-19T23:26:15.408 DEBUG:boto:Data: 2024-04-19T23:26:15.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.408 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:15.408 DEBUG:boto:Port: 8000 2024-04-19T23:26:15.408 DEBUG:boto:Params: {} 2024-04-19T23:26:15.408 DEBUG:boto:Token: None 2024-04-19T23:26:15.408 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-17/obj49 2024-04-19T23:26:15.409 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Xye4pUdRDWE96O9I5Xpj7UHTXYk= 2024-04-19T23:26:15.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Xye4pUdRDWE96O9I5Xpj7UHTXYk='} 2024-04-19T23:26:15.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c8c6f701da13424-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.413 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.414 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.414 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.414 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.414 DEBUG:boto:Data: 2024-04-19T23:26:15.414 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.414 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.414 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.414 DEBUG:boto:Params: {} 2024-04-19T23:26:15.414 DEBUG:boto:Token: None 2024-04-19T23:26:15.414 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.415 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.417 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6a03bb368e1789d-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.417 DEBUG:boto:path=/aktlji-18/obj1 2024-04-19T23:26:15.417 DEBUG:boto:auth_path=/aktlji-18/obj1 2024-04-19T23:26:15.418 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.418 DEBUG:boto:Path: /aktlji-18/obj1 2024-04-19T23:26:15.418 DEBUG:boto:Data: 2024-04-19T23:26:15.418 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.418 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.418 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.418 DEBUG:boto:Params: {} 2024-04-19T23:26:15.418 DEBUG:boto:Token: None 2024-04-19T23:26:15.418 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj1 2024-04-19T23:26:15.418 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EzzlANHvoz2SC+yNgbOmXAByzI8= 2024-04-19T23:26:15.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EzzlANHvoz2SC+yNgbOmXAByzI8='} 2024-04-19T23:26:15.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024c946e42175d2ef-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.426 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.426 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.426 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.426 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.426 DEBUG:boto:Data: 2024-04-19T23:26:15.426 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.427 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.427 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.427 DEBUG:boto:Params: {} 2024-04-19T23:26:15.427 DEBUG:boto:Token: None 2024-04-19T23:26:15.427 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.427 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.429 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000916d1771dc8a7daa-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.430 DEBUG:boto:path=/aktlji-18/obj2 2024-04-19T23:26:15.430 DEBUG:boto:auth_path=/aktlji-18/obj2 2024-04-19T23:26:15.430 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.430 DEBUG:boto:Path: /aktlji-18/obj2 2024-04-19T23:26:15.430 DEBUG:boto:Data: 2024-04-19T23:26:15.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.430 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.430 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.430 DEBUG:boto:Params: {} 2024-04-19T23:26:15.431 DEBUG:boto:Token: None 2024-04-19T23:26:15.431 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj2 2024-04-19T23:26:15.431 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1tJXKrqMtGA51lp8lBUkDd4CVg8= 2024-04-19T23:26:15.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1tJXKrqMtGA51lp8lBUkDd4CVg8='} 2024-04-19T23:26:15.441 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009fbb54fb08a5b296-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.441 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.441 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.441 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.441 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.441 DEBUG:boto:Data: 2024-04-19T23:26:15.441 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.441 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.442 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.442 DEBUG:boto:Params: {} 2024-04-19T23:26:15.442 DEBUG:boto:Token: None 2024-04-19T23:26:15.442 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.442 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001db6983606ac01bc-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.445 DEBUG:boto:path=/aktlji-18/obj3 2024-04-19T23:26:15.445 DEBUG:boto:auth_path=/aktlji-18/obj3 2024-04-19T23:26:15.445 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.445 DEBUG:boto:Path: /aktlji-18/obj3 2024-04-19T23:26:15.445 DEBUG:boto:Data: 2024-04-19T23:26:15.445 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.445 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.445 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.445 DEBUG:boto:Params: {} 2024-04-19T23:26:15.446 DEBUG:boto:Token: None 2024-04-19T23:26:15.446 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj3 2024-04-19T23:26:15.446 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Tp77ofTsLFsUf1IO9qbNyple3os= 2024-04-19T23:26:15.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Tp77ofTsLFsUf1IO9qbNyple3os='} 2024-04-19T23:26:15.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000244f00d46c4361d1-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.491 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.491 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.491 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.491 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.491 DEBUG:boto:Data: 2024-04-19T23:26:15.491 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.491 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.491 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.492 DEBUG:boto:Params: {} 2024-04-19T23:26:15.492 DEBUG:boto:Token: None 2024-04-19T23:26:15.492 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.492 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.494 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012e8a9c906234425-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.495 DEBUG:boto:path=/aktlji-18/obj4 2024-04-19T23:26:15.495 DEBUG:boto:auth_path=/aktlji-18/obj4 2024-04-19T23:26:15.495 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.495 DEBUG:boto:Path: /aktlji-18/obj4 2024-04-19T23:26:15.495 DEBUG:boto:Data: 2024-04-19T23:26:15.495 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.495 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.495 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.495 DEBUG:boto:Params: {} 2024-04-19T23:26:15.496 DEBUG:boto:Token: None 2024-04-19T23:26:15.496 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj4 2024-04-19T23:26:15.496 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jUNcbEu4rkpKAc/jLPhUZ3gNP1A= 2024-04-19T23:26:15.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jUNcbEu4rkpKAc/jLPhUZ3gNP1A='} 2024-04-19T23:26:15.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a165b4b832e317ed-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.502 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.502 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.502 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.503 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.503 DEBUG:boto:Data: 2024-04-19T23:26:15.503 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.503 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.503 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.503 DEBUG:boto:Params: {} 2024-04-19T23:26:15.503 DEBUG:boto:Token: None 2024-04-19T23:26:15.503 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.503 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085d9b48c1854294e-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.506 DEBUG:boto:path=/aktlji-18/obj5 2024-04-19T23:26:15.506 DEBUG:boto:auth_path=/aktlji-18/obj5 2024-04-19T23:26:15.506 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.506 DEBUG:boto:Path: /aktlji-18/obj5 2024-04-19T23:26:15.506 DEBUG:boto:Data: 2024-04-19T23:26:15.506 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.507 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.507 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.507 DEBUG:boto:Params: {} 2024-04-19T23:26:15.507 DEBUG:boto:Token: None 2024-04-19T23:26:15.507 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj5 2024-04-19T23:26:15.507 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:A4fpJURRtZFKPPoTmdsAt3LFNko= 2024-04-19T23:26:15.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:A4fpJURRtZFKPPoTmdsAt3LFNko='} 2024-04-19T23:26:15.555 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009dbbeccec282ce2c-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.555 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.555 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.555 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.556 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.556 DEBUG:boto:Data: 2024-04-19T23:26:15.556 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.556 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.556 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.556 DEBUG:boto:Params: {} 2024-04-19T23:26:15.557 DEBUG:boto:Token: None 2024-04-19T23:26:15.557 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.557 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073dc70944b564783-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.560 DEBUG:boto:path=/aktlji-18/obj6 2024-04-19T23:26:15.561 DEBUG:boto:auth_path=/aktlji-18/obj6 2024-04-19T23:26:15.561 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.561 DEBUG:boto:Path: /aktlji-18/obj6 2024-04-19T23:26:15.561 DEBUG:boto:Data: 2024-04-19T23:26:15.561 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.562 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.562 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.562 DEBUG:boto:Params: {} 2024-04-19T23:26:15.562 DEBUG:boto:Token: None 2024-04-19T23:26:15.562 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj6 2024-04-19T23:26:15.563 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FThfJx43ljM1ZvpP5H8u0cK2GVM= 2024-04-19T23:26:15.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FThfJx43ljM1ZvpP5H8u0cK2GVM='} 2024-04-19T23:26:15.568 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6d2ffac51d6fcad-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.568 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.568 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.568 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.569 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.569 DEBUG:boto:Data: 2024-04-19T23:26:15.569 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.569 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.569 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.569 DEBUG:boto:Params: {} 2024-04-19T23:26:15.569 DEBUG:boto:Token: None 2024-04-19T23:26:15.570 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.570 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.572 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bc7d628ea5b930e-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.572 DEBUG:boto:path=/aktlji-18/obj7 2024-04-19T23:26:15.573 DEBUG:boto:auth_path=/aktlji-18/obj7 2024-04-19T23:26:15.573 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.573 DEBUG:boto:Path: /aktlji-18/obj7 2024-04-19T23:26:15.573 DEBUG:boto:Data: 2024-04-19T23:26:15.573 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.573 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.573 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.573 DEBUG:boto:Params: {} 2024-04-19T23:26:15.573 DEBUG:boto:Token: None 2024-04-19T23:26:15.574 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj7 2024-04-19T23:26:15.574 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JMqLBDZikKsrD/hYZZwkWz4KkYE= 2024-04-19T23:26:15.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JMqLBDZikKsrD/hYZZwkWz4KkYE='} 2024-04-19T23:26:15.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000350c54daa4fd105e-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.619 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.619 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.619 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.619 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.619 DEBUG:boto:Data: 2024-04-19T23:26:15.619 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.619 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.619 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.619 DEBUG:boto:Params: {} 2024-04-19T23:26:15.620 DEBUG:boto:Token: None 2024-04-19T23:26:15.620 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.620 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed148de40c907539-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.623 DEBUG:boto:path=/aktlji-18/obj8 2024-04-19T23:26:15.623 DEBUG:boto:auth_path=/aktlji-18/obj8 2024-04-19T23:26:15.623 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.623 DEBUG:boto:Path: /aktlji-18/obj8 2024-04-19T23:26:15.623 DEBUG:boto:Data: 2024-04-19T23:26:15.623 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.623 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.623 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.623 DEBUG:boto:Params: {} 2024-04-19T23:26:15.623 DEBUG:boto:Token: None 2024-04-19T23:26:15.624 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj8 2024-04-19T23:26:15.624 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oADasiTYZJ9X7W3dMhgMsYEuQ4g= 2024-04-19T23:26:15.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oADasiTYZJ9X7W3dMhgMsYEuQ4g='} 2024-04-19T23:26:15.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c63620f1fe84b7c1-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.629 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.629 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.629 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.629 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.629 DEBUG:boto:Data: 2024-04-19T23:26:15.629 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.629 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.629 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.629 DEBUG:boto:Params: {} 2024-04-19T23:26:15.630 DEBUG:boto:Token: None 2024-04-19T23:26:15.630 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.630 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.632 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d90b3049770b504-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.632 DEBUG:boto:path=/aktlji-18/obj9 2024-04-19T23:26:15.633 DEBUG:boto:auth_path=/aktlji-18/obj9 2024-04-19T23:26:15.633 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.633 DEBUG:boto:Path: /aktlji-18/obj9 2024-04-19T23:26:15.633 DEBUG:boto:Data: 2024-04-19T23:26:15.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.633 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.633 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.633 DEBUG:boto:Params: {} 2024-04-19T23:26:15.633 DEBUG:boto:Token: None 2024-04-19T23:26:15.633 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj9 2024-04-19T23:26:15.634 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6awl+O73y5Yr98TPwCca8WbYcdI= 2024-04-19T23:26:15.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6awl+O73y5Yr98TPwCca8WbYcdI='} 2024-04-19T23:26:15.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000403e421dedd57d84-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.679 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.679 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.679 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.679 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.679 DEBUG:boto:Data: 2024-04-19T23:26:15.679 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.679 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.680 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.680 DEBUG:boto:Params: {} 2024-04-19T23:26:15.680 DEBUG:boto:Token: None 2024-04-19T23:26:15.680 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.680 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008488b1e6c28eb39d-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.683 DEBUG:boto:path=/aktlji-18/obj10 2024-04-19T23:26:15.683 DEBUG:boto:auth_path=/aktlji-18/obj10 2024-04-19T23:26:15.683 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.684 DEBUG:boto:Path: /aktlji-18/obj10 2024-04-19T23:26:15.684 DEBUG:boto:Data: 2024-04-19T23:26:15.684 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.684 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.684 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.684 DEBUG:boto:Params: {} 2024-04-19T23:26:15.684 DEBUG:boto:Token: None 2024-04-19T23:26:15.684 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj10 2024-04-19T23:26:15.684 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:euXsdLrcYoY3ARZi7cgcJwSt/Og= 2024-04-19T23:26:15.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:euXsdLrcYoY3ARZi7cgcJwSt/Og='} 2024-04-19T23:26:15.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5e05cfb8272eea8-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.689 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.689 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.690 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.690 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.690 DEBUG:boto:Data: 2024-04-19T23:26:15.690 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.690 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.690 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.690 DEBUG:boto:Params: {} 2024-04-19T23:26:15.690 DEBUG:boto:Token: None 2024-04-19T23:26:15.690 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.690 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.693 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002de9aa21579b0493-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.693 DEBUG:boto:path=/aktlji-18/obj11 2024-04-19T23:26:15.693 DEBUG:boto:auth_path=/aktlji-18/obj11 2024-04-19T23:26:15.693 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.693 DEBUG:boto:Path: /aktlji-18/obj11 2024-04-19T23:26:15.693 DEBUG:boto:Data: 2024-04-19T23:26:15.693 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.693 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.694 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.694 DEBUG:boto:Params: {} 2024-04-19T23:26:15.694 DEBUG:boto:Token: None 2024-04-19T23:26:15.694 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj11 2024-04-19T23:26:15.694 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZBqvZcswP2k5E3ahdcLAAnKTD00= 2024-04-19T23:26:15.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZBqvZcswP2k5E3ahdcLAAnKTD00='} 2024-04-19T23:26:15.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bec5ca025b95699-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.739 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.739 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.739 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.739 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.739 DEBUG:boto:Data: 2024-04-19T23:26:15.739 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.739 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.739 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.740 DEBUG:boto:Params: {} 2024-04-19T23:26:15.740 DEBUG:boto:Token: None 2024-04-19T23:26:15.740 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.740 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.742 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000308adb801676a816-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.743 DEBUG:boto:path=/aktlji-18/obj12 2024-04-19T23:26:15.743 DEBUG:boto:auth_path=/aktlji-18/obj12 2024-04-19T23:26:15.743 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.743 DEBUG:boto:Path: /aktlji-18/obj12 2024-04-19T23:26:15.743 DEBUG:boto:Data: 2024-04-19T23:26:15.743 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.743 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.743 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.743 DEBUG:boto:Params: {} 2024-04-19T23:26:15.743 DEBUG:boto:Token: None 2024-04-19T23:26:15.744 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj12 2024-04-19T23:26:15.744 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:w7r3j2PfjkkVeIOx+7VXTh0ZIn8= 2024-04-19T23:26:15.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:w7r3j2PfjkkVeIOx+7VXTh0ZIn8='} 2024-04-19T23:26:15.749 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068deea84e2738d42-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.749 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.749 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.749 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.749 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.750 DEBUG:boto:Data: 2024-04-19T23:26:15.750 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.750 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.750 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.750 DEBUG:boto:Params: {} 2024-04-19T23:26:15.750 DEBUG:boto:Token: None 2024-04-19T23:26:15.750 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.750 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.753 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000128c001b80866f1f-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.753 DEBUG:boto:path=/aktlji-18/obj13 2024-04-19T23:26:15.753 DEBUG:boto:auth_path=/aktlji-18/obj13 2024-04-19T23:26:15.753 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.753 DEBUG:boto:Path: /aktlji-18/obj13 2024-04-19T23:26:15.753 DEBUG:boto:Data: 2024-04-19T23:26:15.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.754 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.754 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.754 DEBUG:boto:Params: {} 2024-04-19T23:26:15.754 DEBUG:boto:Token: None 2024-04-19T23:26:15.754 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj13 2024-04-19T23:26:15.754 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qVITO1YIoMtQCoNPAoyiMqgYlqE= 2024-04-19T23:26:15.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qVITO1YIoMtQCoNPAoyiMqgYlqE='} 2024-04-19T23:26:15.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7918ef00c3bf173-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.800 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.800 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.800 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.800 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.800 DEBUG:boto:Data: 2024-04-19T23:26:15.800 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.800 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.801 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.801 DEBUG:boto:Params: {} 2024-04-19T23:26:15.801 DEBUG:boto:Token: None 2024-04-19T23:26:15.801 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.801 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.803 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005a0459f23f4c461-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.804 DEBUG:boto:path=/aktlji-18/obj14 2024-04-19T23:26:15.804 DEBUG:boto:auth_path=/aktlji-18/obj14 2024-04-19T23:26:15.804 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.804 DEBUG:boto:Path: /aktlji-18/obj14 2024-04-19T23:26:15.804 DEBUG:boto:Data: 2024-04-19T23:26:15.804 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.804 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.804 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.805 DEBUG:boto:Params: {} 2024-04-19T23:26:15.805 DEBUG:boto:Token: None 2024-04-19T23:26:15.805 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj14 2024-04-19T23:26:15.805 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HpcItr8ChjJ1dJGrPRJInacS9uk= 2024-04-19T23:26:15.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HpcItr8ChjJ1dJGrPRJInacS9uk='} 2024-04-19T23:26:15.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001370d57edbde094d-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.810 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.810 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.810 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.810 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.811 DEBUG:boto:Data: 2024-04-19T23:26:15.811 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.811 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.811 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.811 DEBUG:boto:Params: {} 2024-04-19T23:26:15.811 DEBUG:boto:Token: None 2024-04-19T23:26:15.811 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.811 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.814 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e3739fc3ea3bb9a-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.814 DEBUG:boto:path=/aktlji-18/obj15 2024-04-19T23:26:15.814 DEBUG:boto:auth_path=/aktlji-18/obj15 2024-04-19T23:26:15.814 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.814 DEBUG:boto:Path: /aktlji-18/obj15 2024-04-19T23:26:15.814 DEBUG:boto:Data: 2024-04-19T23:26:15.814 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.815 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.815 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.815 DEBUG:boto:Params: {} 2024-04-19T23:26:15.815 DEBUG:boto:Token: None 2024-04-19T23:26:15.815 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj15 2024-04-19T23:26:15.815 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8PvfRoHPMIrGgl86hdRWetJvelo= 2024-04-19T23:26:15.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8PvfRoHPMIrGgl86hdRWetJvelo='} 2024-04-19T23:26:15.858 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061dd27e5755963d1-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.859 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.859 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.859 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.859 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.859 DEBUG:boto:Data: 2024-04-19T23:26:15.859 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.859 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.859 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.859 DEBUG:boto:Params: {} 2024-04-19T23:26:15.860 DEBUG:boto:Token: None 2024-04-19T23:26:15.860 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.860 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.862 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cc628b6981e7d99-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.862 DEBUG:boto:path=/aktlji-18/obj16 2024-04-19T23:26:15.863 DEBUG:boto:auth_path=/aktlji-18/obj16 2024-04-19T23:26:15.863 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.863 DEBUG:boto:Path: /aktlji-18/obj16 2024-04-19T23:26:15.863 DEBUG:boto:Data: 2024-04-19T23:26:15.863 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.863 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.863 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.863 DEBUG:boto:Params: {} 2024-04-19T23:26:15.863 DEBUG:boto:Token: None 2024-04-19T23:26:15.863 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj16 2024-04-19T23:26:15.864 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wAEKlbFndGLTvOpYubKvMzs+JI0= 2024-04-19T23:26:15.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wAEKlbFndGLTvOpYubKvMzs+JI0='} 2024-04-19T23:26:15.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033c238a859235298-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.872 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.873 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.873 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.873 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.873 DEBUG:boto:Data: 2024-04-19T23:26:15.873 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.873 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.873 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.873 DEBUG:boto:Params: {} 2024-04-19T23:26:15.873 DEBUG:boto:Token: None 2024-04-19T23:26:15.874 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.874 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000203bdbbb72bed969-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.876 DEBUG:boto:path=/aktlji-18/obj17 2024-04-19T23:26:15.876 DEBUG:boto:auth_path=/aktlji-18/obj17 2024-04-19T23:26:15.877 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.877 DEBUG:boto:Path: /aktlji-18/obj17 2024-04-19T23:26:15.877 DEBUG:boto:Data: 2024-04-19T23:26:15.877 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.877 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.877 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.877 DEBUG:boto:Params: {} 2024-04-19T23:26:15.877 DEBUG:boto:Token: None 2024-04-19T23:26:15.877 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj17 2024-04-19T23:26:15.877 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LWlyW0s2zQNEXEKKxUG+3ahKiqg= 2024-04-19T23:26:15.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LWlyW0s2zQNEXEKKxUG+3ahKiqg='} 2024-04-19T23:26:15.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a5b35344a99a88c-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.927 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.927 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.927 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.927 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.927 DEBUG:boto:Data: 2024-04-19T23:26:15.927 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.927 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.927 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.928 DEBUG:boto:Params: {} 2024-04-19T23:26:15.928 DEBUG:boto:Token: None 2024-04-19T23:26:15.928 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.928 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000104c81ec5c99e63b-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.931 DEBUG:boto:path=/aktlji-18/obj18 2024-04-19T23:26:15.931 DEBUG:boto:auth_path=/aktlji-18/obj18 2024-04-19T23:26:15.931 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.931 DEBUG:boto:Path: /aktlji-18/obj18 2024-04-19T23:26:15.931 DEBUG:boto:Data: 2024-04-19T23:26:15.931 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.931 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.931 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.931 DEBUG:boto:Params: {} 2024-04-19T23:26:15.932 DEBUG:boto:Token: None 2024-04-19T23:26:15.932 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj18 2024-04-19T23:26:15.932 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g5peiNdug1v2IRCtznBQ8M9DJN8= 2024-04-19T23:26:15.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g5peiNdug1v2IRCtznBQ8M9DJN8='} 2024-04-19T23:26:15.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d65b07650b1c3e8e-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.937 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.937 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.937 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.937 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.937 DEBUG:boto:Data: 2024-04-19T23:26:15.937 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.937 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.937 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.937 DEBUG:boto:Params: {} 2024-04-19T23:26:15.937 DEBUG:boto:Token: None 2024-04-19T23:26:15.938 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.938 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000816323bca1f41a3f-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.940 DEBUG:boto:path=/aktlji-18/obj19 2024-04-19T23:26:15.940 DEBUG:boto:auth_path=/aktlji-18/obj19 2024-04-19T23:26:15.940 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.940 DEBUG:boto:Path: /aktlji-18/obj19 2024-04-19T23:26:15.940 DEBUG:boto:Data: 2024-04-19T23:26:15.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.941 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.941 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.941 DEBUG:boto:Params: {} 2024-04-19T23:26:15.942 DEBUG:boto:Token: None 2024-04-19T23:26:15.942 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj19 2024-04-19T23:26:15.942 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1M28GfQPOWc3lOqSLnrSvE90KDs= 2024-04-19T23:26:15.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1M28GfQPOWc3lOqSLnrSvE90KDs='} 2024-04-19T23:26:15.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebfe121d3ab9d795-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.991 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:15.991 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:15.991 DEBUG:boto:Method: HEAD 2024-04-19T23:26:15.991 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:15.991 DEBUG:boto:Data: 2024-04-19T23:26:15.992 DEBUG:boto:Headers: {} 2024-04-19T23:26:15.992 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.992 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.992 DEBUG:boto:Params: {} 2024-04-19T23:26:15.992 DEBUG:boto:Token: None 2024-04-19T23:26:15.992 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/ 2024-04-19T23:26:15.993 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-19T23:26:15.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-19T23:26:15.995 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f8173cd99aea7fc-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:15.995 DEBUG:boto:path=/aktlji-18/obj20 2024-04-19T23:26:15.996 DEBUG:boto:auth_path=/aktlji-18/obj20 2024-04-19T23:26:15.996 DEBUG:boto:Method: PUT 2024-04-19T23:26:15.996 DEBUG:boto:Path: /aktlji-18/obj20 2024-04-19T23:26:15.996 DEBUG:boto:Data: 2024-04-19T23:26:15.996 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:15.996 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:15.996 DEBUG:boto:Port: 8001 2024-04-19T23:26:15.997 DEBUG:boto:Params: {} 2024-04-19T23:26:15.997 DEBUG:boto:Token: None 2024-04-19T23:26:15.997 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:15 GMT /aktlji-18/obj20 2024-04-19T23:26:15.997 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JwWq06MUogFrjiRbz9yQXxa3vac= 2024-04-19T23:26:15.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JwWq06MUogFrjiRbz9yQXxa3vac='} 2024-04-19T23:26:16.003 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6f24a551e166889-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.004 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.004 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.004 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.004 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.004 DEBUG:boto:Data: 2024-04-19T23:26:16.004 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.004 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.004 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.004 DEBUG:boto:Params: {} 2024-04-19T23:26:16.004 DEBUG:boto:Token: None 2024-04-19T23:26:16.005 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.005 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003190502b2cbff0c-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.007 DEBUG:boto:path=/aktlji-18/obj21 2024-04-19T23:26:16.007 DEBUG:boto:auth_path=/aktlji-18/obj21 2024-04-19T23:26:16.008 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.008 DEBUG:boto:Path: /aktlji-18/obj21 2024-04-19T23:26:16.008 DEBUG:boto:Data: 2024-04-19T23:26:16.008 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.008 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.008 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.008 DEBUG:boto:Params: {} 2024-04-19T23:26:16.008 DEBUG:boto:Token: None 2024-04-19T23:26:16.008 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj21 2024-04-19T23:26:16.009 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OyKDZ9JglhWGjzflPEp19RCh6aM= 2024-04-19T23:26:16.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OyKDZ9JglhWGjzflPEp19RCh6aM='} 2024-04-19T23:26:16.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000138cd8fcb9c085d6-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.051 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.051 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.051 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.051 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.052 DEBUG:boto:Data: 2024-04-19T23:26:16.052 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.053 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.053 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.053 DEBUG:boto:Params: {} 2024-04-19T23:26:16.053 DEBUG:boto:Token: None 2024-04-19T23:26:16.053 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.054 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.056 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a929d48b4e03c32-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.056 DEBUG:boto:path=/aktlji-18/obj22 2024-04-19T23:26:16.057 DEBUG:boto:auth_path=/aktlji-18/obj22 2024-04-19T23:26:16.057 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.057 DEBUG:boto:Path: /aktlji-18/obj22 2024-04-19T23:26:16.057 DEBUG:boto:Data: 2024-04-19T23:26:16.057 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.057 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.058 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.058 DEBUG:boto:Params: {} 2024-04-19T23:26:16.058 DEBUG:boto:Token: None 2024-04-19T23:26:16.058 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj22 2024-04-19T23:26:16.058 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1xBUus32uTqE0kg55fflNSxJ7MA= 2024-04-19T23:26:16.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1xBUus32uTqE0kg55fflNSxJ7MA='} 2024-04-19T23:26:16.064 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4ed40e414cd2adc-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.064 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.065 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.065 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.065 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.065 DEBUG:boto:Data: 2024-04-19T23:26:16.065 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.065 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.066 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.066 DEBUG:boto:Params: {} 2024-04-19T23:26:16.066 DEBUG:boto:Token: None 2024-04-19T23:26:16.066 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.066 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.069 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096f0550a9cff5ae6-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.069 DEBUG:boto:path=/aktlji-18/obj23 2024-04-19T23:26:16.070 DEBUG:boto:auth_path=/aktlji-18/obj23 2024-04-19T23:26:16.070 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.070 DEBUG:boto:Path: /aktlji-18/obj23 2024-04-19T23:26:16.070 DEBUG:boto:Data: 2024-04-19T23:26:16.070 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.071 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.073 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.073 DEBUG:boto:Params: {} 2024-04-19T23:26:16.074 DEBUG:boto:Token: None 2024-04-19T23:26:16.074 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj23 2024-04-19T23:26:16.074 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:H2444GM3GGY6WZOym+AbzNaU0Dk= 2024-04-19T23:26:16.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:H2444GM3GGY6WZOym+AbzNaU0Dk='} 2024-04-19T23:26:16.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007449a4514c60f395-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.079 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.079 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.079 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.079 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.079 DEBUG:boto:Data: 2024-04-19T23:26:16.079 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.079 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.079 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.079 DEBUG:boto:Params: {} 2024-04-19T23:26:16.080 DEBUG:boto:Token: None 2024-04-19T23:26:16.080 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.080 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.082 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4780b381874b315-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.083 DEBUG:boto:path=/aktlji-18/obj24 2024-04-19T23:26:16.083 DEBUG:boto:auth_path=/aktlji-18/obj24 2024-04-19T23:26:16.083 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.083 DEBUG:boto:Path: /aktlji-18/obj24 2024-04-19T23:26:16.083 DEBUG:boto:Data: 2024-04-19T23:26:16.083 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.083 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.083 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.083 DEBUG:boto:Params: {} 2024-04-19T23:26:16.083 DEBUG:boto:Token: None 2024-04-19T23:26:16.084 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj24 2024-04-19T23:26:16.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Y7NIh3JcVSuibJBe90xfRjK+dqY= 2024-04-19T23:26:16.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y7NIh3JcVSuibJBe90xfRjK+dqY='} 2024-04-19T23:26:16.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000345a73f36b7bd7e2-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.127 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.127 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.127 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.127 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.127 DEBUG:boto:Data: 2024-04-19T23:26:16.127 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.127 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.127 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.127 DEBUG:boto:Params: {} 2024-04-19T23:26:16.128 DEBUG:boto:Token: None 2024-04-19T23:26:16.128 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.128 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085701fae72a1c85b-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.131 DEBUG:boto:path=/aktlji-18/obj25 2024-04-19T23:26:16.131 DEBUG:boto:auth_path=/aktlji-18/obj25 2024-04-19T23:26:16.131 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.131 DEBUG:boto:Path: /aktlji-18/obj25 2024-04-19T23:26:16.131 DEBUG:boto:Data: 2024-04-19T23:26:16.131 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.131 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.131 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.131 DEBUG:boto:Params: {} 2024-04-19T23:26:16.131 DEBUG:boto:Token: None 2024-04-19T23:26:16.132 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj25 2024-04-19T23:26:16.132 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+45YH47SHGoDwBuejIkiuFQoiy8= 2024-04-19T23:26:16.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+45YH47SHGoDwBuejIkiuFQoiy8='} 2024-04-19T23:26:16.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f098d6f3c250d58-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.137 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.137 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.137 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.137 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.137 DEBUG:boto:Data: 2024-04-19T23:26:16.137 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.137 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.137 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.138 DEBUG:boto:Params: {} 2024-04-19T23:26:16.138 DEBUG:boto:Token: None 2024-04-19T23:26:16.138 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.138 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088ffc4367b58f352-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.140 DEBUG:boto:path=/aktlji-18/obj26 2024-04-19T23:26:16.140 DEBUG:boto:auth_path=/aktlji-18/obj26 2024-04-19T23:26:16.141 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.141 DEBUG:boto:Path: /aktlji-18/obj26 2024-04-19T23:26:16.141 DEBUG:boto:Data: 2024-04-19T23:26:16.141 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.141 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.141 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.141 DEBUG:boto:Params: {} 2024-04-19T23:26:16.141 DEBUG:boto:Token: None 2024-04-19T23:26:16.142 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj26 2024-04-19T23:26:16.142 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EKuElpgQkhNRMqPrP22UMOncDdk= 2024-04-19T23:26:16.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EKuElpgQkhNRMqPrP22UMOncDdk='} 2024-04-19T23:26:16.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000281e1b0700891178-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.147 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.147 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.147 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.147 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.147 DEBUG:boto:Data: 2024-04-19T23:26:16.147 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.147 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.147 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.147 DEBUG:boto:Params: {} 2024-04-19T23:26:16.148 DEBUG:boto:Token: None 2024-04-19T23:26:16.148 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.148 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fed36d57da16c33-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.150 DEBUG:boto:path=/aktlji-18/obj27 2024-04-19T23:26:16.150 DEBUG:boto:auth_path=/aktlji-18/obj27 2024-04-19T23:26:16.151 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.151 DEBUG:boto:Path: /aktlji-18/obj27 2024-04-19T23:26:16.151 DEBUG:boto:Data: 2024-04-19T23:26:16.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.151 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.151 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.151 DEBUG:boto:Params: {} 2024-04-19T23:26:16.151 DEBUG:boto:Token: None 2024-04-19T23:26:16.151 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj27 2024-04-19T23:26:16.152 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Li0/Aj494HJcekIeuPYoGu9//7g= 2024-04-19T23:26:16.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Li0/Aj494HJcekIeuPYoGu9//7g='} 2024-04-19T23:26:16.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b97e92a963a5ecf4-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.195 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.195 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.195 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.195 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.195 DEBUG:boto:Data: 2024-04-19T23:26:16.195 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.195 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.195 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.195 DEBUG:boto:Params: {} 2024-04-19T23:26:16.196 DEBUG:boto:Token: None 2024-04-19T23:26:16.196 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.196 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000032a9ca5b87a3d3c-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.199 DEBUG:boto:path=/aktlji-18/obj28 2024-04-19T23:26:16.199 DEBUG:boto:auth_path=/aktlji-18/obj28 2024-04-19T23:26:16.199 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.199 DEBUG:boto:Path: /aktlji-18/obj28 2024-04-19T23:26:16.199 DEBUG:boto:Data: 2024-04-19T23:26:16.199 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.199 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.199 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.199 DEBUG:boto:Params: {} 2024-04-19T23:26:16.200 DEBUG:boto:Token: None 2024-04-19T23:26:16.200 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj28 2024-04-19T23:26:16.200 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nTFt8K/1tRyydQEQVDe+p0Ssd48= 2024-04-19T23:26:16.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nTFt8K/1tRyydQEQVDe+p0Ssd48='} 2024-04-19T23:26:16.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002271ffc24ad76b3-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.204 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.205 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.205 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.205 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.205 DEBUG:boto:Data: 2024-04-19T23:26:16.205 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.205 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.205 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.205 DEBUG:boto:Params: {} 2024-04-19T23:26:16.205 DEBUG:boto:Token: None 2024-04-19T23:26:16.205 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.206 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.208 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088ac754b41e1a069-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.208 DEBUG:boto:path=/aktlji-18/obj29 2024-04-19T23:26:16.208 DEBUG:boto:auth_path=/aktlji-18/obj29 2024-04-19T23:26:16.208 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.208 DEBUG:boto:Path: /aktlji-18/obj29 2024-04-19T23:26:16.208 DEBUG:boto:Data: 2024-04-19T23:26:16.209 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.209 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.209 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.209 DEBUG:boto:Params: {} 2024-04-19T23:26:16.209 DEBUG:boto:Token: None 2024-04-19T23:26:16.209 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj29 2024-04-19T23:26:16.209 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DR4Cmpk2ptSNd2NcpEN5BVbUvN8= 2024-04-19T23:26:16.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DR4Cmpk2ptSNd2NcpEN5BVbUvN8='} 2024-04-19T23:26:16.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009af3b3f3ff8a4b03-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.255 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.255 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.255 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.255 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.255 DEBUG:boto:Data: 2024-04-19T23:26:16.255 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.255 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.255 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.255 DEBUG:boto:Params: {} 2024-04-19T23:26:16.255 DEBUG:boto:Token: None 2024-04-19T23:26:16.256 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.256 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000904fbc638475bed4-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.258 DEBUG:boto:path=/aktlji-18/obj30 2024-04-19T23:26:16.258 DEBUG:boto:auth_path=/aktlji-18/obj30 2024-04-19T23:26:16.259 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.259 DEBUG:boto:Path: /aktlji-18/obj30 2024-04-19T23:26:16.259 DEBUG:boto:Data: 2024-04-19T23:26:16.259 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.259 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.259 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.259 DEBUG:boto:Params: {} 2024-04-19T23:26:16.259 DEBUG:boto:Token: None 2024-04-19T23:26:16.259 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj30 2024-04-19T23:26:16.259 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z9toIhsYZKvkjMRJ5rHClXbxVgo= 2024-04-19T23:26:16.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z9toIhsYZKvkjMRJ5rHClXbxVgo='} 2024-04-19T23:26:16.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000354e6eca8b5e6002-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.264 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.265 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.265 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.265 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.265 DEBUG:boto:Data: 2024-04-19T23:26:16.265 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.265 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.265 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.265 DEBUG:boto:Params: {} 2024-04-19T23:26:16.265 DEBUG:boto:Token: None 2024-04-19T23:26:16.266 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.266 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000456f15e36a510c14-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.268 DEBUG:boto:path=/aktlji-18/obj31 2024-04-19T23:26:16.268 DEBUG:boto:auth_path=/aktlji-18/obj31 2024-04-19T23:26:16.268 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.268 DEBUG:boto:Path: /aktlji-18/obj31 2024-04-19T23:26:16.268 DEBUG:boto:Data: 2024-04-19T23:26:16.269 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.269 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.269 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.269 DEBUG:boto:Params: {} 2024-04-19T23:26:16.269 DEBUG:boto:Token: None 2024-04-19T23:26:16.269 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj31 2024-04-19T23:26:16.269 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X0G6b6m9gyCFzeHZJEGNatlBZaM= 2024-04-19T23:26:16.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X0G6b6m9gyCFzeHZJEGNatlBZaM='} 2024-04-19T23:26:16.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f63c42bd839611fc-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.274 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.274 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.274 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.274 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.275 DEBUG:boto:Data: 2024-04-19T23:26:16.275 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.275 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.275 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.275 DEBUG:boto:Params: {} 2024-04-19T23:26:16.275 DEBUG:boto:Token: None 2024-04-19T23:26:16.275 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.275 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000269e9e0bd4ba3cbb-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.279 DEBUG:boto:path=/aktlji-18/obj32 2024-04-19T23:26:16.279 DEBUG:boto:auth_path=/aktlji-18/obj32 2024-04-19T23:26:16.280 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.280 DEBUG:boto:Path: /aktlji-18/obj32 2024-04-19T23:26:16.280 DEBUG:boto:Data: 2024-04-19T23:26:16.280 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.280 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.280 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.280 DEBUG:boto:Params: {} 2024-04-19T23:26:16.280 DEBUG:boto:Token: None 2024-04-19T23:26:16.280 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj32 2024-04-19T23:26:16.281 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4IBA+u+/NjegAOcGdq8SIjvlxXM= 2024-04-19T23:26:16.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4IBA+u+/NjegAOcGdq8SIjvlxXM='} 2024-04-19T23:26:16.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d84cb34e45f06e8c-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.323 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.323 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.323 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.323 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.323 DEBUG:boto:Data: 2024-04-19T23:26:16.323 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.323 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.323 DEBUG:boto:Params: {} 2024-04-19T23:26:16.323 DEBUG:boto:Token: None 2024-04-19T23:26:16.324 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f548d2ab2afec58-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.326 DEBUG:boto:path=/aktlji-18/obj33 2024-04-19T23:26:16.326 DEBUG:boto:auth_path=/aktlji-18/obj33 2024-04-19T23:26:16.327 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.327 DEBUG:boto:Path: /aktlji-18/obj33 2024-04-19T23:26:16.327 DEBUG:boto:Data: 2024-04-19T23:26:16.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.327 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.327 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.327 DEBUG:boto:Params: {} 2024-04-19T23:26:16.327 DEBUG:boto:Token: None 2024-04-19T23:26:16.327 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj33 2024-04-19T23:26:16.327 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XEMUm7QZO2uBXr4djCyJ5h7skpE= 2024-04-19T23:26:16.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XEMUm7QZO2uBXr4djCyJ5h7skpE='} 2024-04-19T23:26:16.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b56c6730ea3fcff-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.333 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.333 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.333 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.333 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.333 DEBUG:boto:Data: 2024-04-19T23:26:16.333 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.333 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.333 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.333 DEBUG:boto:Params: {} 2024-04-19T23:26:16.334 DEBUG:boto:Token: None 2024-04-19T23:26:16.334 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.334 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d93ad28164bc8d3e-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.337 DEBUG:boto:path=/aktlji-18/obj34 2024-04-19T23:26:16.337 DEBUG:boto:auth_path=/aktlji-18/obj34 2024-04-19T23:26:16.337 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.337 DEBUG:boto:Path: /aktlji-18/obj34 2024-04-19T23:26:16.337 DEBUG:boto:Data: 2024-04-19T23:26:16.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.337 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.337 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.337 DEBUG:boto:Params: {} 2024-04-19T23:26:16.338 DEBUG:boto:Token: None 2024-04-19T23:26:16.338 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj34 2024-04-19T23:26:16.338 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pX1Glq8+bo1dsQn5rQcJZSKziLU= 2024-04-19T23:26:16.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pX1Glq8+bo1dsQn5rQcJZSKziLU='} 2024-04-19T23:26:16.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6228e22d4bccdb9-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.342 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.342 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.343 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.343 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.343 DEBUG:boto:Data: 2024-04-19T23:26:16.343 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.343 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.343 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.343 DEBUG:boto:Params: {} 2024-04-19T23:26:16.343 DEBUG:boto:Token: None 2024-04-19T23:26:16.344 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.347 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005166b65817011787-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.347 DEBUG:boto:path=/aktlji-18/obj35 2024-04-19T23:26:16.347 DEBUG:boto:auth_path=/aktlji-18/obj35 2024-04-19T23:26:16.347 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.347 DEBUG:boto:Path: /aktlji-18/obj35 2024-04-19T23:26:16.347 DEBUG:boto:Data: 2024-04-19T23:26:16.347 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.348 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.348 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.348 DEBUG:boto:Params: {} 2024-04-19T23:26:16.348 DEBUG:boto:Token: None 2024-04-19T23:26:16.348 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj35 2024-04-19T23:26:16.348 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BY4/TSaFrQ6HJ7gqZOqSB2IgxZY= 2024-04-19T23:26:16.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BY4/TSaFrQ6HJ7gqZOqSB2IgxZY='} 2024-04-19T23:26:16.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa362e74ed72089a-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.390 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.391 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.391 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.391 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.391 DEBUG:boto:Data: 2024-04-19T23:26:16.391 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.391 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.391 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.391 DEBUG:boto:Params: {} 2024-04-19T23:26:16.391 DEBUG:boto:Token: None 2024-04-19T23:26:16.392 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.392 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f1ee0b0abe89c45-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.394 DEBUG:boto:path=/aktlji-18/obj36 2024-04-19T23:26:16.394 DEBUG:boto:auth_path=/aktlji-18/obj36 2024-04-19T23:26:16.394 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.395 DEBUG:boto:Path: /aktlji-18/obj36 2024-04-19T23:26:16.395 DEBUG:boto:Data: 2024-04-19T23:26:16.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.395 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.395 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.395 DEBUG:boto:Params: {} 2024-04-19T23:26:16.395 DEBUG:boto:Token: None 2024-04-19T23:26:16.395 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj36 2024-04-19T23:26:16.396 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AAHQb8bShg3zaFM6Gc0ENJ3/xfQ= 2024-04-19T23:26:16.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AAHQb8bShg3zaFM6Gc0ENJ3/xfQ='} 2024-04-19T23:26:16.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a3b12320f63977a-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.401 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.401 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.401 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.401 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.401 DEBUG:boto:Data: 2024-04-19T23:26:16.401 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.401 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.401 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.402 DEBUG:boto:Params: {} 2024-04-19T23:26:16.402 DEBUG:boto:Token: None 2024-04-19T23:26:16.402 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.402 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000135c1675381cf3f9-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.404 DEBUG:boto:path=/aktlji-18/obj37 2024-04-19T23:26:16.404 DEBUG:boto:auth_path=/aktlji-18/obj37 2024-04-19T23:26:16.405 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.405 DEBUG:boto:Path: /aktlji-18/obj37 2024-04-19T23:26:16.405 DEBUG:boto:Data: 2024-04-19T23:26:16.405 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.405 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.405 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.405 DEBUG:boto:Params: {} 2024-04-19T23:26:16.405 DEBUG:boto:Token: None 2024-04-19T23:26:16.406 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj37 2024-04-19T23:26:16.406 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UcEWM5pZG7u/RgtB11nnkV5KAqw= 2024-04-19T23:26:16.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UcEWM5pZG7u/RgtB11nnkV5KAqw='} 2024-04-19T23:26:16.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc1af73bac530013-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.411 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.411 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.411 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.411 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.411 DEBUG:boto:Data: 2024-04-19T23:26:16.411 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.411 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.411 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.412 DEBUG:boto:Params: {} 2024-04-19T23:26:16.412 DEBUG:boto:Token: None 2024-04-19T23:26:16.412 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.412 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.414 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b993e51052fa73d0-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.415 DEBUG:boto:path=/aktlji-18/obj38 2024-04-19T23:26:16.415 DEBUG:boto:auth_path=/aktlji-18/obj38 2024-04-19T23:26:16.415 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.415 DEBUG:boto:Path: /aktlji-18/obj38 2024-04-19T23:26:16.415 DEBUG:boto:Data: 2024-04-19T23:26:16.415 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.415 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.415 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.416 DEBUG:boto:Params: {} 2024-04-19T23:26:16.416 DEBUG:boto:Token: None 2024-04-19T23:26:16.416 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj38 2024-04-19T23:26:16.416 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DqDMJG3WA6GkEYMkjrqMdKpT3p0= 2024-04-19T23:26:16.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DqDMJG3WA6GkEYMkjrqMdKpT3p0='} 2024-04-19T23:26:16.459 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec0a43c07ba6db65-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.459 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.459 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.459 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.459 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.459 DEBUG:boto:Data: 2024-04-19T23:26:16.459 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.459 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.459 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.460 DEBUG:boto:Params: {} 2024-04-19T23:26:16.460 DEBUG:boto:Token: None 2024-04-19T23:26:16.460 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1a65fab7d103414-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.463 DEBUG:boto:path=/aktlji-18/obj39 2024-04-19T23:26:16.463 DEBUG:boto:auth_path=/aktlji-18/obj39 2024-04-19T23:26:16.463 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.463 DEBUG:boto:Path: /aktlji-18/obj39 2024-04-19T23:26:16.463 DEBUG:boto:Data: 2024-04-19T23:26:16.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.463 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.464 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.464 DEBUG:boto:Params: {} 2024-04-19T23:26:16.464 DEBUG:boto:Token: None 2024-04-19T23:26:16.464 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj39 2024-04-19T23:26:16.464 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zis0z0lIF6OPsMtQ/bCytnv2lAo= 2024-04-19T23:26:16.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zis0z0lIF6OPsMtQ/bCytnv2lAo='} 2024-04-19T23:26:16.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008175277f5f15b80a-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.469 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.469 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.469 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.470 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.470 DEBUG:boto:Data: 2024-04-19T23:26:16.470 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.470 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.470 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.470 DEBUG:boto:Params: {} 2024-04-19T23:26:16.470 DEBUG:boto:Token: None 2024-04-19T23:26:16.470 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.470 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7945cbf7c5c9df1-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.473 DEBUG:boto:path=/aktlji-18/obj40 2024-04-19T23:26:16.473 DEBUG:boto:auth_path=/aktlji-18/obj40 2024-04-19T23:26:16.473 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.473 DEBUG:boto:Path: /aktlji-18/obj40 2024-04-19T23:26:16.473 DEBUG:boto:Data: 2024-04-19T23:26:16.473 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.473 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.473 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.474 DEBUG:boto:Params: {} 2024-04-19T23:26:16.474 DEBUG:boto:Token: None 2024-04-19T23:26:16.474 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj40 2024-04-19T23:26:16.474 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EVfg1M6XqYXoN8I+BpPOFqknAYE= 2024-04-19T23:26:16.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EVfg1M6XqYXoN8I+BpPOFqknAYE='} 2024-04-19T23:26:16.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092c1a6540419725b-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.523 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.523 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.523 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.523 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.523 DEBUG:boto:Data: 2024-04-19T23:26:16.523 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.523 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.523 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.523 DEBUG:boto:Params: {} 2024-04-19T23:26:16.524 DEBUG:boto:Token: None 2024-04-19T23:26:16.524 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.524 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfcff0b01a495a95-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.527 DEBUG:boto:path=/aktlji-18/obj41 2024-04-19T23:26:16.527 DEBUG:boto:auth_path=/aktlji-18/obj41 2024-04-19T23:26:16.527 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.527 DEBUG:boto:Path: /aktlji-18/obj41 2024-04-19T23:26:16.527 DEBUG:boto:Data: 2024-04-19T23:26:16.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.527 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.527 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.527 DEBUG:boto:Params: {} 2024-04-19T23:26:16.528 DEBUG:boto:Token: None 2024-04-19T23:26:16.528 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj41 2024-04-19T23:26:16.528 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eKP0bC+NQWrGEtlJClIS3S4plAU= 2024-04-19T23:26:16.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eKP0bC+NQWrGEtlJClIS3S4plAU='} 2024-04-19T23:26:16.533 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ac685037373ecaf-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.533 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.533 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.533 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.533 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.533 DEBUG:boto:Data: 2024-04-19T23:26:16.533 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.533 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.533 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.533 DEBUG:boto:Params: {} 2024-04-19T23:26:16.534 DEBUG:boto:Token: None 2024-04-19T23:26:16.534 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.534 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ce6f526ea362128-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.536 DEBUG:boto:path=/aktlji-18/obj42 2024-04-19T23:26:16.537 DEBUG:boto:auth_path=/aktlji-18/obj42 2024-04-19T23:26:16.537 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.537 DEBUG:boto:Path: /aktlji-18/obj42 2024-04-19T23:26:16.537 DEBUG:boto:Data: 2024-04-19T23:26:16.537 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.537 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.537 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.537 DEBUG:boto:Params: {} 2024-04-19T23:26:16.537 DEBUG:boto:Token: None 2024-04-19T23:26:16.537 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj42 2024-04-19T23:26:16.538 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HvY49/quLj+pVeQCGrkf915AEvA= 2024-04-19T23:26:16.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HvY49/quLj+pVeQCGrkf915AEvA='} 2024-04-19T23:26:16.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a69aac434ee470b-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.583 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.583 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.583 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.583 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.583 DEBUG:boto:Data: 2024-04-19T23:26:16.583 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.583 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.584 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.584 DEBUG:boto:Params: {} 2024-04-19T23:26:16.584 DEBUG:boto:Token: None 2024-04-19T23:26:16.584 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.584 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd788d467526b96a-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.587 DEBUG:boto:path=/aktlji-18/obj43 2024-04-19T23:26:16.587 DEBUG:boto:auth_path=/aktlji-18/obj43 2024-04-19T23:26:16.587 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.587 DEBUG:boto:Path: /aktlji-18/obj43 2024-04-19T23:26:16.587 DEBUG:boto:Data: 2024-04-19T23:26:16.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.587 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.587 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.587 DEBUG:boto:Params: {} 2024-04-19T23:26:16.587 DEBUG:boto:Token: None 2024-04-19T23:26:16.588 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj43 2024-04-19T23:26:16.588 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7nS8rDOP5SbZHpaAH8NolVg5VTo= 2024-04-19T23:26:16.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7nS8rDOP5SbZHpaAH8NolVg5VTo='} 2024-04-19T23:26:16.593 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff3a49931a839712-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.593 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.593 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.593 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.593 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.593 DEBUG:boto:Data: 2024-04-19T23:26:16.593 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.593 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.594 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.594 DEBUG:boto:Params: {} 2024-04-19T23:26:16.594 DEBUG:boto:Token: None 2024-04-19T23:26:16.594 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.594 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e96050e0bf3e03e9-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.596 DEBUG:boto:path=/aktlji-18/obj44 2024-04-19T23:26:16.597 DEBUG:boto:auth_path=/aktlji-18/obj44 2024-04-19T23:26:16.597 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.597 DEBUG:boto:Path: /aktlji-18/obj44 2024-04-19T23:26:16.597 DEBUG:boto:Data: 2024-04-19T23:26:16.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.597 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.597 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.597 DEBUG:boto:Params: {} 2024-04-19T23:26:16.597 DEBUG:boto:Token: None 2024-04-19T23:26:16.598 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj44 2024-04-19T23:26:16.598 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:THZJlQMekW1D6l4w4+Od2APXGG4= 2024-04-19T23:26:16.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:THZJlQMekW1D6l4w4+Od2APXGG4='} 2024-04-19T23:26:16.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b63ab855b95012f8-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.643 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.643 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.643 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.643 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.643 DEBUG:boto:Data: 2024-04-19T23:26:16.643 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.643 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.643 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.644 DEBUG:boto:Params: {} 2024-04-19T23:26:16.644 DEBUG:boto:Token: None 2024-04-19T23:26:16.644 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.644 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecd97848bf6da389-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.647 DEBUG:boto:path=/aktlji-18/obj45 2024-04-19T23:26:16.647 DEBUG:boto:auth_path=/aktlji-18/obj45 2024-04-19T23:26:16.647 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.647 DEBUG:boto:Path: /aktlji-18/obj45 2024-04-19T23:26:16.647 DEBUG:boto:Data: 2024-04-19T23:26:16.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.647 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.647 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.647 DEBUG:boto:Params: {} 2024-04-19T23:26:16.647 DEBUG:boto:Token: None 2024-04-19T23:26:16.648 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj45 2024-04-19T23:26:16.648 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X8gSmTp9mB1FPYsJZOoV8SxIgnQ= 2024-04-19T23:26:16.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X8gSmTp9mB1FPYsJZOoV8SxIgnQ='} 2024-04-19T23:26:16.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000abec812fb87b63c4-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.652 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.652 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.653 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.653 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.653 DEBUG:boto:Data: 2024-04-19T23:26:16.653 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.653 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.653 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.653 DEBUG:boto:Params: {} 2024-04-19T23:26:16.653 DEBUG:boto:Token: None 2024-04-19T23:26:16.653 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.653 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035d4f18f072abad9-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.656 DEBUG:boto:path=/aktlji-18/obj46 2024-04-19T23:26:16.656 DEBUG:boto:auth_path=/aktlji-18/obj46 2024-04-19T23:26:16.656 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.656 DEBUG:boto:Path: /aktlji-18/obj46 2024-04-19T23:26:16.656 DEBUG:boto:Data: 2024-04-19T23:26:16.656 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.657 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.657 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.657 DEBUG:boto:Params: {} 2024-04-19T23:26:16.657 DEBUG:boto:Token: None 2024-04-19T23:26:16.657 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj46 2024-04-19T23:26:16.657 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4vyEPJ2KtqEtxDcS15BZP5QudDw= 2024-04-19T23:26:16.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4vyEPJ2KtqEtxDcS15BZP5QudDw='} 2024-04-19T23:26:16.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083e63804ef38d1a2-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.703 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.703 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.703 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.703 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.703 DEBUG:boto:Data: 2024-04-19T23:26:16.703 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.703 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.703 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.704 DEBUG:boto:Params: {} 2024-04-19T23:26:16.704 DEBUG:boto:Token: None 2024-04-19T23:26:16.704 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c4bba95ae607919-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.707 DEBUG:boto:path=/aktlji-18/obj47 2024-04-19T23:26:16.707 DEBUG:boto:auth_path=/aktlji-18/obj47 2024-04-19T23:26:16.707 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.707 DEBUG:boto:Path: /aktlji-18/obj47 2024-04-19T23:26:16.707 DEBUG:boto:Data: 2024-04-19T23:26:16.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.707 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.707 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.708 DEBUG:boto:Params: {} 2024-04-19T23:26:16.708 DEBUG:boto:Token: None 2024-04-19T23:26:16.708 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj47 2024-04-19T23:26:16.708 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LI9gLUYIS0i2YJNdJkEj40e5w0s= 2024-04-19T23:26:16.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LI9gLUYIS0i2YJNdJkEj40e5w0s='} 2024-04-19T23:26:16.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c79cb796e5552a9-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.713 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.713 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.713 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.713 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.713 DEBUG:boto:Data: 2024-04-19T23:26:16.713 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.713 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.713 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.713 DEBUG:boto:Params: {} 2024-04-19T23:26:16.714 DEBUG:boto:Token: None 2024-04-19T23:26:16.714 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.714 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9ba6975b3dcfbdf-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.716 DEBUG:boto:path=/aktlji-18/obj48 2024-04-19T23:26:16.716 DEBUG:boto:auth_path=/aktlji-18/obj48 2024-04-19T23:26:16.716 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.717 DEBUG:boto:Path: /aktlji-18/obj48 2024-04-19T23:26:16.717 DEBUG:boto:Data: 2024-04-19T23:26:16.717 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.717 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.717 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.717 DEBUG:boto:Params: {} 2024-04-19T23:26:16.717 DEBUG:boto:Token: None 2024-04-19T23:26:16.717 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj48 2024-04-19T23:26:16.717 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ach+lJTqbzOUNRNg14qUIhZXdtY= 2024-04-19T23:26:16.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ach+lJTqbzOUNRNg14qUIhZXdtY='} 2024-04-19T23:26:16.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a77f7622a44e8678-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.763 DEBUG:boto:path=/aktlji-18/ 2024-04-19T23:26:16.763 DEBUG:boto:auth_path=/aktlji-18/ 2024-04-19T23:26:16.763 DEBUG:boto:Method: HEAD 2024-04-19T23:26:16.763 DEBUG:boto:Path: /aktlji-18/ 2024-04-19T23:26:16.763 DEBUG:boto:Data: 2024-04-19T23:26:16.763 DEBUG:boto:Headers: {} 2024-04-19T23:26:16.763 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.764 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.764 DEBUG:boto:Params: {} 2024-04-19T23:26:16.764 DEBUG:boto:Token: None 2024-04-19T23:26:16.764 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/ 2024-04-19T23:26:16.764 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-19T23:26:16.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-19T23:26:16.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f70439d710d58ca8-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.767 DEBUG:boto:path=/aktlji-18/obj49 2024-04-19T23:26:16.767 DEBUG:boto:auth_path=/aktlji-18/obj49 2024-04-19T23:26:16.767 DEBUG:boto:Method: PUT 2024-04-19T23:26:16.767 DEBUG:boto:Path: /aktlji-18/obj49 2024-04-19T23:26:16.767 DEBUG:boto:Data: 2024-04-19T23:26:16.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-19T23:26:16.767 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:26:16.767 DEBUG:boto:Port: 8001 2024-04-19T23:26:16.768 DEBUG:boto:Params: {} 2024-04-19T23:26:16.768 DEBUG:boto:Token: None 2024-04-19T23:26:16.768 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 19 Apr 2024 23:26:16 GMT /aktlji-18/obj49 2024-04-19T23:26:16.768 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qpGVEyRPFneq10BlyykjPuMzpSk= 2024-04-19T23:26:16.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qpGVEyRPFneq10BlyykjPuMzpSk='} 2024-04-19T23:26:16.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8832f2cf7137b0d-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:16.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:26:16.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:26:16.773 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:26:16.832 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:26:16.832 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:26:16.872 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:26:16.894 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:26:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:26:16.894 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:26:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:26:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:26:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:26:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:26:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:26:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:26:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:26:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713569172.547279_68.1", 2024-04-19T23:26:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:26:12.547279Z" 2024-04-19T23:26:16.895 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:26:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:26:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:26:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:26:16.896 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:26:16.896 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:26:16.897 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569172.547279_68.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:26:16.898 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:26:16.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:26:16.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:26:16.898 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:26:16.958 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:26:16.958 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:26:16.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:26:16.998+0000 7f94eea35780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:26:16.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:26:17.002+0000 7f94eea35780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:26:17.012 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:26:17.012 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:26:17.012 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:26:17.013 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:26:17.013 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:26:17.013 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:26:17.013 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:26:17.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:26:17.013 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:26:17.013 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:26:17.013 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:26:17.013 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:26:17.013 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:26:17.013 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:26:17.014 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:26:17.014 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:26:17.014 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:26:17.014 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:26:17.014 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:26:17.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:26:17.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:26:17.014 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:26:17.014 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:26:17.014 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:26:17.015 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:26:17.015 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:26:17.015 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:26:17.015 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:26:17.015 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:26:17.015 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:26:17.015 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:26:17.015 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:26:17.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:26:17.015 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:26:17.016 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:26:17.016 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:26:17.016 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:26:17.016 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713569172.547279_68.1", 2024-04-19T23:26:17.016 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:26:17.016 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:26:17.016 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:26:17.016 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:12.547279Z", 2024-04-19T23:26:17.016 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:26:17.016 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:26:17.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:26:17.017 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:26:17.017 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:26:17.017 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:26:17.017 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:26:17.017 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:26:17.017 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:26:17.017 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:26:17.017 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:26:17.017 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:26:17.017 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:26:17.018 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:26:17.018 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:26:17.018 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:26:17.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:26:17.018 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:26:17.018 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:26:17.018 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:26:17.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:26:17.018 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:26:17Z" 2024-04-19T23:26:17.018 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:26:17.019 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569172.547279_68.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:12.547279Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:26:17Z" } 2024-04-19T23:26:17.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569172.547279_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:12.547279Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:26:17.020 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569172.547279_68.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:26:17.020 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569172.547279_68.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:26:17.020 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:26:17.020 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:26:17.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:26:17.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:26:17.020 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:26:17.121 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:26:17.121 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:26:17.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:26:17.164+0000 7f8ca0460780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:26:17.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:26:17.165+0000 7f8ca0460780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:26:17.174 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:26:17.174 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:26:17.174 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:26:17.174 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:26:17.174 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:26:17.174 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:26:17.175 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:26:17.175 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:26:17.175 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:26:17.175 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:26:17.175 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:26:17.175 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:26:17.175 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:26:17.175 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:26:17.175 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:26:17.175 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:26:17.175 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:26:17.175 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:26:17.176 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:26:17.176 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:26:17.176 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:26:17.176 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:26:17.176 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:26:17.176 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:26:17.176 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:26:17.176 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:26:17.176 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:26:17.176 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:26:17.176 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:26:17.177 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:26:17.177 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:26:17.177 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:26:17.177 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:26:17.177 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:26:17.177 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:26:17.177 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:26:17.177 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:26:17.177 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713569172.547279_68.1", 2024-04-19T23:26:17.177 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:26:17.177 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:26:17.178 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:26:17.178 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:26:12.547279Z", 2024-04-19T23:26:17.178 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:26:17.178 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:26:17.178 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:26:17.178 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:26:17.178 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:26:17.178 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:26:17.178 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:26:17.178 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:26:17.178 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:26:17.179 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:26:17.179 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:26:17.179 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:26:17.179 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:26:17.179 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:26:17.179 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:26:17.179 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:26:17.179 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:26:17.179 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:26:17.179 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:26:17.179 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:26:17.180 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:26:17.180 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:26:17Z" 2024-04-19T23:26:17.180 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:26:17.181 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569172.547279_68.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:12.547279Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:26:17Z" } 2024-04-19T23:26:17.181 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569172.547279_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:12.547279Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:26:17.181 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569172.547279_68.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:26:17.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569172.547279_68.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:26:17.181 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:26:17.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:26:17.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:26:17.182 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:26:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:26:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:26:17.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:26:17.297+0000 7f0bf22d2780 1 waiting for incremental sync to catch up: 2024-04-19T23:26:17.293 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [00000000007.39.6, , 00000000002.11.6, , 00000000006.16.6, 00000000001.5.6, , 00000000004.17.6, 00000000002.18.6, , 00000000001.17.6] 2024-04-19T23:26:17.293 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [00000000007.39.6, 00000000004.41.6, 00000000004.15.6, 00000000004.26.6, 00000000006.16.6, 00000000005.13.6, 00000000004.21.6, 00000000005.19.6, 00000000002.18.6, 00000000004.22.6, 00000000004.35.6] 2024-04-19T23:26:22.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:26:22.298+0000 7f0bf22d2780 1 waiting for incremental sync to catch up: 2024-04-19T23:26:22.294 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [00000000007.39.6, , 00000000002.11.6, , 00000000006.16.6, 00000000001.5.6, , 00000000004.17.6, 00000000002.18.6, , 00000000001.17.6] 2024-04-19T23:26:22.294 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [00000000007.39.6, 00000000004.41.6, 00000000004.15.6, 00000000004.26.6, 00000000006.16.6, 00000000005.13.6, 00000000004.21.6, 00000000005.19.6, 00000000002.18.6, 00000000004.22.6, 00000000004.35.6] 2024-04-19T23:26:27.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:26:27.299+0000 7f0bf22d2780 1 waiting for incremental sync to catch up: 2024-04-19T23:26:27.295 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [00000000007.39.6, , 00000000002.11.6, , 00000000006.16.6, 00000000001.5.6, , 00000000004.17.6, 00000000002.18.6, , 00000000001.17.6] 2024-04-19T23:26:27.295 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [00000000007.39.6, 00000000004.41.6, 00000000004.15.6, 00000000004.26.6, 00000000006.16.6, 00000000005.13.6, 00000000004.21.6, 00000000005.19.6, 00000000002.18.6, 00000000004.22.6, 00000000004.35.6] 2024-04-19T23:26:32.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:26:32.300+0000 7f0bf22d2780 1 waiting for incremental sync to catch up: 2024-04-19T23:26:32.296 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [00000000007.39.6, , 00000000004.15.6, , 00000000006.16.6, 00000000001.5.6, 00000000004.21.6, 00000000004.17.6, 00000000002.18.6, , 00000000003.25.6] 2024-04-19T23:26:32.297 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [00000000007.39.6, 00000000004.41.6, 00000000004.15.6, 00000000004.26.6, 00000000006.16.6, 00000000005.13.6, 00000000004.21.6, 00000000005.19.6, 00000000002.18.6, 00000000004.22.6, 00000000004.35.6] 2024-04-19T23:26:37.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:26:37.302+0000 7f0bf22d2780 1 bucket sync caught up with source: 2024-04-19T23:26:37.298 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [00000000007.39.6, 00000000004.41.6, 00000000004.15.6, 00000000004.26.6, 00000000006.16.6, 00000000005.13.6, 00000000004.21.6, 00000000005.19.6, 00000000002.18.6, 00000000004.22.6, 00000000004.35.6] 2024-04-19T23:26:37.298 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [00000000007.39.6, 00000000004.41.6, 00000000004.15.6, 00000000004.26.6, 00000000006.16.6, 00000000005.13.6, 00000000004.21.6, 00000000005.19.6, 00000000002.18.6, 00000000004.22.6, 00000000004.35.6] 2024-04-19T23:26:37.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:26:37.302+0000 7f0bf22d2780 0 bucket checkpoint complete 2024-04-19T23:26:37.315 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-16 zones={test-zone2, test-zone1} 2024-04-19T23:26:37.315 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.315 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.315 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.315 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:37.315 DEBUG:boto:Data: 2024-04-19T23:26:37.316 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.316 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.316 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.316 DEBUG:boto:Params: {} 2024-04-19T23:26:37.316 DEBUG:boto:Token: None 2024-04-19T23:26:37.316 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.316 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u2pNxeEWJmssPF+lFy9B9vFghh4= 2024-04-19T23:26:37.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u2pNxeEWJmssPF+lFy9B9vFghh4=', 'Content-Length': '0'} 2024-04-19T23:26:37.319 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '294'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e19773c4b608bdd5-006622fdad-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.319 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.319 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.320 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.320 DEBUG:boto:Path: /aktlji-16/ 2024-04-19T23:26:37.320 DEBUG:boto:Data: 2024-04-19T23:26:37.320 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.320 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.320 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.320 DEBUG:boto:Params: {} 2024-04-19T23:26:37.320 DEBUG:boto:Token: None 2024-04-19T23:26:37.320 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.321 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u2pNxeEWJmssPF+lFy9B9vFghh4= 2024-04-19T23:26:37.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u2pNxeEWJmssPF+lFy9B9vFghh4=', 'Content-Length': '0'} 2024-04-19T23:26:37.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '294'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1644e597b1a2bb3-006622fdad-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.323 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:26:37.323 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.323 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.323 DEBUG:boto:path=/aktlji-16/?versions&max-keys=999 2024-04-19T23:26:37.324 DEBUG:boto:auth_path=/aktlji-16/?versions&max-keys=999 2024-04-19T23:26:37.324 DEBUG:boto:Method: GET 2024-04-19T23:26:37.324 DEBUG:boto:Path: /aktlji-16/?versions&max-keys=999 2024-04-19T23:26:37.324 DEBUG:boto:Data: 2024-04-19T23:26:37.324 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.324 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.324 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.324 DEBUG:boto:Params: {} 2024-04-19T23:26:37.324 DEBUG:boto:Token: None 2024-04-19T23:26:37.325 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/?versions 2024-04-19T23:26:37.325 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ= 2024-04-19T23:26:37.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ=', 'Content-Length': '0'} 2024-04-19T23:26:37.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c45ea7f43351c271-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.328 DEBUG:boto:b'aktlji-16999falseobj1nulltrue2024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-19T23:26:37.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj10 2024-04-19T23:26:37.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-19T23:26:37.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj12 2024-04-19T23:26:37.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj13 2024-04-19T23:26:37.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj14 2024-04-19T23:26:37.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj15 2024-04-19T23:26:37.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj16 2024-04-19T23:26:37.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-19T23:26:37.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj18 2024-04-19T23:26:37.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-19T23:26:37.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-19T23:26:37.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj20 2024-04-19T23:26:37.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj21 2024-04-19T23:26:37.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-19T23:26:37.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-19T23:26:37.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj24 2024-04-19T23:26:37.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj25 2024-04-19T23:26:37.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj26 2024-04-19T23:26:37.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj27 2024-04-19T23:26:37.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj28 2024-04-19T23:26:37.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj30 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj32 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj35 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj36 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj39 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-19T23:26:37.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj40 2024-04-19T23:26:37.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj41 2024-04-19T23:26:37.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj42 2024-04-19T23:26:37.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj43 2024-04-19T23:26:37.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-19T23:26:37.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj45 2024-04-19T23:26:37.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-19T23:26:37.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj47 2024-04-19T23:26:37.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj48 2024-04-19T23:26:37.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj49 2024-04-19T23:26:37.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-19T23:26:37.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-19T23:26:37.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-19T23:26:37.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-19T23:26:37.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj9 2024-04-19T23:26:37.335 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:26:37.335 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.335 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.335 DEBUG:boto:path=/aktlji-16/?versions&max-keys=999 2024-04-19T23:26:37.335 DEBUG:boto:auth_path=/aktlji-16/?versions&max-keys=999 2024-04-19T23:26:37.335 DEBUG:boto:Method: GET 2024-04-19T23:26:37.335 DEBUG:boto:Path: /aktlji-16/?versions&max-keys=999 2024-04-19T23:26:37.335 DEBUG:boto:Data: 2024-04-19T23:26:37.335 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.336 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.336 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.336 DEBUG:boto:Params: {} 2024-04-19T23:26:37.336 DEBUG:boto:Token: None 2024-04-19T23:26:37.336 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/?versions 2024-04-19T23:26:37.336 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ= 2024-04-19T23:26:37.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ=', 'Content-Length': '0'} 2024-04-19T23:26:37.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c463b7d7cbeb904-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.340 DEBUG:boto:b'aktlji-16999falseobj1nulltrue2024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-19T23:26:37.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj10 2024-04-19T23:26:37.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-19T23:26:37.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj12 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj13 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj14 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj15 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj16 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj18 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj20 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj21 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-19T23:26:37.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj24 2024-04-19T23:26:37.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj25 2024-04-19T23:26:37.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj26 2024-04-19T23:26:37.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj27 2024-04-19T23:26:37.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj28 2024-04-19T23:26:37.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-19T23:26:37.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-19T23:26:37.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj30 2024-04-19T23:26:37.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-19T23:26:37.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj32 2024-04-19T23:26:37.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-19T23:26:37.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-19T23:26:37.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj35 2024-04-19T23:26:37.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj36 2024-04-19T23:26:37.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-19T23:26:37.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-19T23:26:37.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj39 2024-04-19T23:26:37.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-19T23:26:37.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj40 2024-04-19T23:26:37.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj41 2024-04-19T23:26:37.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj42 2024-04-19T23:26:37.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj43 2024-04-19T23:26:37.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-19T23:26:37.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj45 2024-04-19T23:26:37.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-19T23:26:37.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj47 2024-04-19T23:26:37.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj48 2024-04-19T23:26:37.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj49 2024-04-19T23:26:37.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-19T23:26:37.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-19T23:26:37.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-19T23:26:37.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-19T23:26:37.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj9 2024-04-19T23:26:37.347 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.347 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.347 DEBUG:boto:path=/aktlji-16/?versions&max-keys=999 2024-04-19T23:26:37.347 DEBUG:boto:auth_path=/aktlji-16/?versions&max-keys=999 2024-04-19T23:26:37.347 DEBUG:boto:Method: GET 2024-04-19T23:26:37.347 DEBUG:boto:Path: /aktlji-16/?versions&max-keys=999 2024-04-19T23:26:37.347 DEBUG:boto:Data: 2024-04-19T23:26:37.347 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.347 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.347 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.347 DEBUG:boto:Params: {} 2024-04-19T23:26:37.348 DEBUG:boto:Token: None 2024-04-19T23:26:37.348 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/?versions 2024-04-19T23:26:37.348 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ= 2024-04-19T23:26:37.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ=', 'Content-Length': '0'} 2024-04-19T23:26:37.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb147473ab8424ae-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.351 DEBUG:boto:b'aktlji-16999falseobj1nulltrue2024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.353 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.353 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.353 DEBUG:boto:path=/aktlji-16/?versions&max-keys=999 2024-04-19T23:26:37.354 DEBUG:boto:auth_path=/aktlji-16/?versions&max-keys=999 2024-04-19T23:26:37.354 DEBUG:boto:Method: GET 2024-04-19T23:26:37.354 DEBUG:boto:Path: /aktlji-16/?versions&max-keys=999 2024-04-19T23:26:37.354 DEBUG:boto:Data: 2024-04-19T23:26:37.354 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.354 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.354 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.354 DEBUG:boto:Params: {} 2024-04-19T23:26:37.354 DEBUG:boto:Token: None 2024-04-19T23:26:37.355 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/?versions 2024-04-19T23:26:37.355 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ= 2024-04-19T23:26:37.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ=', 'Content-Length': '0'} 2024-04-19T23:26:37.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000136f8ce1b356b4ea-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.358 DEBUG:boto:b'aktlji-16999falseobj1nulltrue2024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-19T23:26:37.361 DEBUG:boto:path=/aktlji-16/obj1 2024-04-19T23:26:37.361 DEBUG:boto:auth_path=/aktlji-16/obj1 2024-04-19T23:26:37.361 DEBUG:boto:path=/aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.361 DEBUG:boto:auth_path=/aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.362 DEBUG:boto:Method: GET 2024-04-19T23:26:37.362 DEBUG:boto:Path: /aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.362 DEBUG:boto:Data: 2024-04-19T23:26:37.362 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.362 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.362 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.362 DEBUG:boto:Params: {} 2024-04-19T23:26:37.362 DEBUG:boto:Token: None 2024-04-19T23:26:37.362 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.363 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E= 2024-04-19T23:26:37.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E=', 'Content-Length': '0'} 2024-04-19T23:26:37.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eac4907f72c6c64d-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.406 DEBUG:boto:path=/aktlji-16/obj1 2024-04-19T23:26:37.406 DEBUG:boto:auth_path=/aktlji-16/obj1 2024-04-19T23:26:37.407 DEBUG:boto:path=/aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.407 DEBUG:boto:auth_path=/aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.407 DEBUG:boto:Method: GET 2024-04-19T23:26:37.407 DEBUG:boto:Path: /aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.407 DEBUG:boto:Data: 2024-04-19T23:26:37.407 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.407 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.407 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.408 DEBUG:boto:Params: {} 2024-04-19T23:26:37.408 DEBUG:boto:Token: None 2024-04-19T23:26:37.408 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.408 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E= 2024-04-19T23:26:37.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E=', 'Content-Length': '0'} 2024-04-19T23:26:37.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0fd8a3cc70aa3e7-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.450 DEBUG:boto:path=/aktlji-16/obj1 2024-04-19T23:26:37.451 DEBUG:boto:auth_path=/aktlji-16/obj1 2024-04-19T23:26:37.451 DEBUG:boto:path=/aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.451 DEBUG:boto:auth_path=/aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.451 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.451 DEBUG:boto:Path: /aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.451 DEBUG:boto:Data: 2024-04-19T23:26:37.451 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.451 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.451 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.451 DEBUG:boto:Params: {} 2024-04-19T23:26:37.452 DEBUG:boto:Token: None 2024-04-19T23:26:37.452 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.452 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WRHkkeevAGFZqmF8I9uvqI6rOnk= 2024-04-19T23:26:37.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WRHkkeevAGFZqmF8I9uvqI6rOnk=', 'Content-Length': '0'} 2024-04-19T23:26:37.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000028a1632e5468659-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.455 DEBUG:boto:path=/aktlji-16/obj1 2024-04-19T23:26:37.455 DEBUG:boto:auth_path=/aktlji-16/obj1 2024-04-19T23:26:37.455 DEBUG:boto:path=/aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.455 DEBUG:boto:auth_path=/aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.455 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.455 DEBUG:boto:Path: /aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.455 DEBUG:boto:Data: 2024-04-19T23:26:37.456 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.456 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.456 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.456 DEBUG:boto:Params: {} 2024-04-19T23:26:37.456 DEBUG:boto:Token: None 2024-04-19T23:26:37.456 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.456 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WRHkkeevAGFZqmF8I9uvqI6rOnk= 2024-04-19T23:26:37.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WRHkkeevAGFZqmF8I9uvqI6rOnk=', 'Content-Length': '0'} 2024-04-19T23:26:37.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ab3eecceb369d5b-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-19T23:26:37.459 DEBUG:boto:path=/aktlji-16/obj1 2024-04-19T23:26:37.459 DEBUG:boto:auth_path=/aktlji-16/obj1 2024-04-19T23:26:37.459 DEBUG:boto:path=/aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.459 DEBUG:boto:auth_path=/aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.459 DEBUG:boto:Method: GET 2024-04-19T23:26:37.459 DEBUG:boto:Path: /aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.459 DEBUG:boto:Data: 2024-04-19T23:26:37.459 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.459 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.459 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.460 DEBUG:boto:Params: {} 2024-04-19T23:26:37.460 DEBUG:boto:Token: None 2024-04-19T23:26:37.460 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E= 2024-04-19T23:26:37.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E=', 'Content-Length': '0'} 2024-04-19T23:26:37.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf9c97caa4f3e57a-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.462 DEBUG:boto:path=/aktlji-16/obj1 2024-04-19T23:26:37.462 DEBUG:boto:auth_path=/aktlji-16/obj1 2024-04-19T23:26:37.462 DEBUG:boto:path=/aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.463 DEBUG:boto:auth_path=/aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.463 DEBUG:boto:Method: GET 2024-04-19T23:26:37.463 DEBUG:boto:Path: /aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.463 DEBUG:boto:Data: 2024-04-19T23:26:37.463 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.463 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.463 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.463 DEBUG:boto:Params: {} 2024-04-19T23:26:37.463 DEBUG:boto:Token: None 2024-04-19T23:26:37.463 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj1?versionId=null 2024-04-19T23:26:37.464 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E= 2024-04-19T23:26:37.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E=', 'Content-Length': '0'} 2024-04-19T23:26:37.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc0b62e64ee9aa44-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.466 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.466 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.466 DEBUG:boto:path=/aktlji-16/?prefix=obj1 2024-04-19T23:26:37.466 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj1 2024-04-19T23:26:37.466 DEBUG:boto:Method: GET 2024-04-19T23:26:37.466 DEBUG:boto:Path: /aktlji-16/?prefix=obj1 2024-04-19T23:26:37.467 DEBUG:boto:Data: 2024-04-19T23:26:37.467 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.467 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.467 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.467 DEBUG:boto:Params: {} 2024-04-19T23:26:37.467 DEBUG:boto:Token: None 2024-04-19T23:26:37.467 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.467 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed8f7312935d80a8-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.470 DEBUG:boto:b'aktlji-16obj11000falseobj12024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.471 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.471 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.471 DEBUG:boto:path=/aktlji-16/?prefix=obj1 2024-04-19T23:26:37.471 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj1 2024-04-19T23:26:37.471 DEBUG:boto:Method: GET 2024-04-19T23:26:37.471 DEBUG:boto:Path: /aktlji-16/?prefix=obj1 2024-04-19T23:26:37.471 DEBUG:boto:Data: 2024-04-19T23:26:37.471 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.471 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.472 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.472 DEBUG:boto:Params: {} 2024-04-19T23:26:37.472 DEBUG:boto:Token: None 2024-04-19T23:26:37.472 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.472 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3ddcac8bce4e2e9-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.475 DEBUG:boto:b'aktlji-16obj11000falseobj12024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.476 DEBUG:boto:path=/aktlji-16/obj1 2024-04-19T23:26:37.476 DEBUG:boto:auth_path=/aktlji-16/obj1 2024-04-19T23:26:37.476 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.476 DEBUG:boto:Path: /aktlji-16/obj1 2024-04-19T23:26:37.476 DEBUG:boto:Data: 2024-04-19T23:26:37.476 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.476 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.476 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.476 DEBUG:boto:Params: {} 2024-04-19T23:26:37.476 DEBUG:boto:Token: None 2024-04-19T23:26:37.477 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj1 2024-04-19T23:26:37.477 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KCCU2H3Yrr8eZJYdTf21DQKSVfA= 2024-04-19T23:26:37.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KCCU2H3Yrr8eZJYdTf21DQKSVfA=', 'Content-Length': '0'} 2024-04-19T23:26:37.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090d9b8d8c888d9d9-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.479 DEBUG:boto:path=/aktlji-16/obj1 2024-04-19T23:26:37.479 DEBUG:boto:auth_path=/aktlji-16/obj1 2024-04-19T23:26:37.479 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.479 DEBUG:boto:Path: /aktlji-16/obj1 2024-04-19T23:26:37.479 DEBUG:boto:Data: 2024-04-19T23:26:37.479 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.479 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.479 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.479 DEBUG:boto:Params: {} 2024-04-19T23:26:37.480 DEBUG:boto:Token: None 2024-04-19T23:26:37.480 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj1 2024-04-19T23:26:37.480 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KCCU2H3Yrr8eZJYdTf21DQKSVfA= 2024-04-19T23:26:37.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KCCU2H3Yrr8eZJYdTf21DQKSVfA=', 'Content-Length': '0'} 2024-04-19T23:26:37.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000536b517f07bae7df-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-19T23:26:37.482 DEBUG:boto:path=/aktlji-16/obj1 2024-04-19T23:26:37.482 DEBUG:boto:auth_path=/aktlji-16/obj1 2024-04-19T23:26:37.482 DEBUG:boto:Method: GET 2024-04-19T23:26:37.482 DEBUG:boto:Path: /aktlji-16/obj1 2024-04-19T23:26:37.483 DEBUG:boto:Data: 2024-04-19T23:26:37.483 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.483 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.483 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.483 DEBUG:boto:Params: {} 2024-04-19T23:26:37.483 DEBUG:boto:Token: None 2024-04-19T23:26:37.483 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj1 2024-04-19T23:26:37.483 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Q/AU8UO+g40/05l5K/wr3OZfmEg= 2024-04-19T23:26:37.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q/AU8UO+g40/05l5K/wr3OZfmEg=', 'Content-Length': '0'} 2024-04-19T23:26:37.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a18f620c12a12e60-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.485 DEBUG:boto:path=/aktlji-16/obj1 2024-04-19T23:26:37.485 DEBUG:boto:auth_path=/aktlji-16/obj1 2024-04-19T23:26:37.486 DEBUG:boto:Method: GET 2024-04-19T23:26:37.486 DEBUG:boto:Path: /aktlji-16/obj1 2024-04-19T23:26:37.486 DEBUG:boto:Data: 2024-04-19T23:26:37.486 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.486 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.486 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.486 DEBUG:boto:Params: {} 2024-04-19T23:26:37.486 DEBUG:boto:Token: None 2024-04-19T23:26:37.486 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj1 2024-04-19T23:26:37.486 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Q/AU8UO+g40/05l5K/wr3OZfmEg= 2024-04-19T23:26:37.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q/AU8UO+g40/05l5K/wr3OZfmEg=', 'Content-Length': '0'} 2024-04-19T23:26:37.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083552fbc0e0f5299-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.530 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.531 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.531 DEBUG:boto:path=/aktlji-16/?prefix=obj1 2024-04-19T23:26:37.531 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj1 2024-04-19T23:26:37.531 DEBUG:boto:Method: GET 2024-04-19T23:26:37.531 DEBUG:boto:Path: /aktlji-16/?prefix=obj1 2024-04-19T23:26:37.531 DEBUG:boto:Data: 2024-04-19T23:26:37.531 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.531 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.531 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.531 DEBUG:boto:Params: {} 2024-04-19T23:26:37.532 DEBUG:boto:Token: None 2024-04-19T23:26:37.532 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.532 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028da5ab90b8b91ad-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.535 DEBUG:boto:b'aktlji-16obj11000falseobj12024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.536 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.536 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.536 DEBUG:boto:path=/aktlji-16/?prefix=obj1 2024-04-19T23:26:37.536 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj1 2024-04-19T23:26:37.536 DEBUG:boto:Method: GET 2024-04-19T23:26:37.536 DEBUG:boto:Path: /aktlji-16/?prefix=obj1 2024-04-19T23:26:37.536 DEBUG:boto:Data: 2024-04-19T23:26:37.537 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.537 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.537 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.537 DEBUG:boto:Params: {} 2024-04-19T23:26:37.537 DEBUG:boto:Token: None 2024-04-19T23:26:37.537 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.537 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001581ec14335d1323-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.540 DEBUG:boto:b'aktlji-16obj11000falseobj12024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj10 2024-04-19T23:26:37.541 DEBUG:boto:path=/aktlji-16/obj10 2024-04-19T23:26:37.541 DEBUG:boto:auth_path=/aktlji-16/obj10 2024-04-19T23:26:37.541 DEBUG:boto:path=/aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.541 DEBUG:boto:auth_path=/aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.541 DEBUG:boto:Method: GET 2024-04-19T23:26:37.541 DEBUG:boto:Path: /aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.541 DEBUG:boto:Data: 2024-04-19T23:26:37.542 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.542 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.542 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.542 DEBUG:boto:Params: {} 2024-04-19T23:26:37.542 DEBUG:boto:Token: None 2024-04-19T23:26:37.542 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.542 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8= 2024-04-19T23:26:37.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8=', 'Content-Length': '0'} 2024-04-19T23:26:37.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5e6a95c3d5a6413-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.544 DEBUG:boto:path=/aktlji-16/obj10 2024-04-19T23:26:37.544 DEBUG:boto:auth_path=/aktlji-16/obj10 2024-04-19T23:26:37.544 DEBUG:boto:path=/aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.545 DEBUG:boto:auth_path=/aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.545 DEBUG:boto:Method: GET 2024-04-19T23:26:37.545 DEBUG:boto:Path: /aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.545 DEBUG:boto:Data: 2024-04-19T23:26:37.545 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.545 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.545 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.545 DEBUG:boto:Params: {} 2024-04-19T23:26:37.545 DEBUG:boto:Token: None 2024-04-19T23:26:37.545 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.546 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8= 2024-04-19T23:26:37.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8=', 'Content-Length': '0'} 2024-04-19T23:26:37.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008816d51005a31dc2-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.548 DEBUG:boto:path=/aktlji-16/obj10 2024-04-19T23:26:37.548 DEBUG:boto:auth_path=/aktlji-16/obj10 2024-04-19T23:26:37.548 DEBUG:boto:path=/aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.548 DEBUG:boto:auth_path=/aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.548 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.548 DEBUG:boto:Path: /aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.548 DEBUG:boto:Data: 2024-04-19T23:26:37.548 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.549 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.549 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.549 DEBUG:boto:Params: {} 2024-04-19T23:26:37.549 DEBUG:boto:Token: None 2024-04-19T23:26:37.549 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.549 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AqtqIO0CfxAa2Xx/FqIwSqxBkDc= 2024-04-19T23:26:37.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqtqIO0CfxAa2Xx/FqIwSqxBkDc=', 'Content-Length': '0'} 2024-04-19T23:26:37.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7b64ce4e40f396b-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.551 DEBUG:boto:path=/aktlji-16/obj10 2024-04-19T23:26:37.551 DEBUG:boto:auth_path=/aktlji-16/obj10 2024-04-19T23:26:37.551 DEBUG:boto:path=/aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.552 DEBUG:boto:auth_path=/aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.552 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.552 DEBUG:boto:Path: /aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.552 DEBUG:boto:Data: 2024-04-19T23:26:37.552 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.552 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.552 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.552 DEBUG:boto:Params: {} 2024-04-19T23:26:37.552 DEBUG:boto:Token: None 2024-04-19T23:26:37.552 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.553 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AqtqIO0CfxAa2Xx/FqIwSqxBkDc= 2024-04-19T23:26:37.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqtqIO0CfxAa2Xx/FqIwSqxBkDc=', 'Content-Length': '0'} 2024-04-19T23:26:37.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb39abb4a18d1526-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj10 2024-04-19T23:26:37.555 DEBUG:boto:path=/aktlji-16/obj10 2024-04-19T23:26:37.555 DEBUG:boto:auth_path=/aktlji-16/obj10 2024-04-19T23:26:37.555 DEBUG:boto:path=/aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.555 DEBUG:boto:auth_path=/aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.555 DEBUG:boto:Method: GET 2024-04-19T23:26:37.555 DEBUG:boto:Path: /aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.555 DEBUG:boto:Data: 2024-04-19T23:26:37.555 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.555 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.555 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.555 DEBUG:boto:Params: {} 2024-04-19T23:26:37.556 DEBUG:boto:Token: None 2024-04-19T23:26:37.556 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.556 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8= 2024-04-19T23:26:37.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8=', 'Content-Length': '0'} 2024-04-19T23:26:37.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cb0401b9bc14b95-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.602 DEBUG:boto:path=/aktlji-16/obj10 2024-04-19T23:26:37.603 DEBUG:boto:auth_path=/aktlji-16/obj10 2024-04-19T23:26:37.603 DEBUG:boto:path=/aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.603 DEBUG:boto:auth_path=/aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.603 DEBUG:boto:Method: GET 2024-04-19T23:26:37.603 DEBUG:boto:Path: /aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.603 DEBUG:boto:Data: 2024-04-19T23:26:37.603 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.603 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.603 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.603 DEBUG:boto:Params: {} 2024-04-19T23:26:37.604 DEBUG:boto:Token: None 2024-04-19T23:26:37.604 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj10?versionId=null 2024-04-19T23:26:37.604 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8= 2024-04-19T23:26:37.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8=', 'Content-Length': '0'} 2024-04-19T23:26:37.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d9b9a806f0c33ba-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.606 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.606 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.606 DEBUG:boto:path=/aktlji-16/?prefix=obj10 2024-04-19T23:26:37.606 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj10 2024-04-19T23:26:37.607 DEBUG:boto:Method: GET 2024-04-19T23:26:37.607 DEBUG:boto:Path: /aktlji-16/?prefix=obj10 2024-04-19T23:26:37.607 DEBUG:boto:Data: 2024-04-19T23:26:37.607 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.607 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.607 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.607 DEBUG:boto:Params: {} 2024-04-19T23:26:37.607 DEBUG:boto:Token: None 2024-04-19T23:26:37.607 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.607 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066b25d90f571f07f-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.610 DEBUG:boto:b'aktlji-16obj101000falseobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.611 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.611 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.611 DEBUG:boto:path=/aktlji-16/?prefix=obj10 2024-04-19T23:26:37.611 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj10 2024-04-19T23:26:37.611 DEBUG:boto:Method: GET 2024-04-19T23:26:37.611 DEBUG:boto:Path: /aktlji-16/?prefix=obj10 2024-04-19T23:26:37.611 DEBUG:boto:Data: 2024-04-19T23:26:37.611 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.611 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.612 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.612 DEBUG:boto:Params: {} 2024-04-19T23:26:37.612 DEBUG:boto:Token: None 2024-04-19T23:26:37.612 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.612 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026c8cafaf5e9c8a2-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.615 DEBUG:boto:b'aktlji-16obj101000falseobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.615 DEBUG:boto:path=/aktlji-16/obj10 2024-04-19T23:26:37.615 DEBUG:boto:auth_path=/aktlji-16/obj10 2024-04-19T23:26:37.615 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.615 DEBUG:boto:Path: /aktlji-16/obj10 2024-04-19T23:26:37.615 DEBUG:boto:Data: 2024-04-19T23:26:37.615 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.616 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.616 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.616 DEBUG:boto:Params: {} 2024-04-19T23:26:37.616 DEBUG:boto:Token: None 2024-04-19T23:26:37.616 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj10 2024-04-19T23:26:37.616 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u5YRXT1T/TwTeQKn76ue0TUYwC0= 2024-04-19T23:26:37.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u5YRXT1T/TwTeQKn76ue0TUYwC0=', 'Content-Length': '0'} 2024-04-19T23:26:37.618 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb9b7013254be24c-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.618 DEBUG:boto:path=/aktlji-16/obj10 2024-04-19T23:26:37.618 DEBUG:boto:auth_path=/aktlji-16/obj10 2024-04-19T23:26:37.619 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.619 DEBUG:boto:Path: /aktlji-16/obj10 2024-04-19T23:26:37.619 DEBUG:boto:Data: 2024-04-19T23:26:37.619 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.619 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.619 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.619 DEBUG:boto:Params: {} 2024-04-19T23:26:37.619 DEBUG:boto:Token: None 2024-04-19T23:26:37.619 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj10 2024-04-19T23:26:37.619 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u5YRXT1T/TwTeQKn76ue0TUYwC0= 2024-04-19T23:26:37.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u5YRXT1T/TwTeQKn76ue0TUYwC0=', 'Content-Length': '0'} 2024-04-19T23:26:37.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012312477e7de46ff-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj10 2024-04-19T23:26:37.622 DEBUG:boto:path=/aktlji-16/obj10 2024-04-19T23:26:37.622 DEBUG:boto:auth_path=/aktlji-16/obj10 2024-04-19T23:26:37.622 DEBUG:boto:Method: GET 2024-04-19T23:26:37.622 DEBUG:boto:Path: /aktlji-16/obj10 2024-04-19T23:26:37.622 DEBUG:boto:Data: 2024-04-19T23:26:37.622 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.622 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.622 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.622 DEBUG:boto:Params: {} 2024-04-19T23:26:37.623 DEBUG:boto:Token: None 2024-04-19T23:26:37.623 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj10 2024-04-19T23:26:37.623 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WQe5mKX7mQLQNzmtEtLFuPAQyPA= 2024-04-19T23:26:37.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WQe5mKX7mQLQNzmtEtLFuPAQyPA=', 'Content-Length': '0'} 2024-04-19T23:26:37.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9c244a05b4967bf-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.625 DEBUG:boto:path=/aktlji-16/obj10 2024-04-19T23:26:37.625 DEBUG:boto:auth_path=/aktlji-16/obj10 2024-04-19T23:26:37.625 DEBUG:boto:Method: GET 2024-04-19T23:26:37.625 DEBUG:boto:Path: /aktlji-16/obj10 2024-04-19T23:26:37.625 DEBUG:boto:Data: 2024-04-19T23:26:37.625 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.625 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.626 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.626 DEBUG:boto:Params: {} 2024-04-19T23:26:37.626 DEBUG:boto:Token: None 2024-04-19T23:26:37.626 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj10 2024-04-19T23:26:37.626 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WQe5mKX7mQLQNzmtEtLFuPAQyPA= 2024-04-19T23:26:37.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WQe5mKX7mQLQNzmtEtLFuPAQyPA=', 'Content-Length': '0'} 2024-04-19T23:26:37.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000001884a692f7cd60-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.670 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.671 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.671 DEBUG:boto:path=/aktlji-16/?prefix=obj10 2024-04-19T23:26:37.671 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj10 2024-04-19T23:26:37.671 DEBUG:boto:Method: GET 2024-04-19T23:26:37.671 DEBUG:boto:Path: /aktlji-16/?prefix=obj10 2024-04-19T23:26:37.671 DEBUG:boto:Data: 2024-04-19T23:26:37.671 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.671 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.671 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.671 DEBUG:boto:Params: {} 2024-04-19T23:26:37.671 DEBUG:boto:Token: None 2024-04-19T23:26:37.672 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.672 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000327e412eb3da5a07-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.675 DEBUG:boto:b'aktlji-16obj101000falseobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.675 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.675 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.675 DEBUG:boto:path=/aktlji-16/?prefix=obj10 2024-04-19T23:26:37.675 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj10 2024-04-19T23:26:37.675 DEBUG:boto:Method: GET 2024-04-19T23:26:37.675 DEBUG:boto:Path: /aktlji-16/?prefix=obj10 2024-04-19T23:26:37.675 DEBUG:boto:Data: 2024-04-19T23:26:37.676 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.676 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.676 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.676 DEBUG:boto:Params: {} 2024-04-19T23:26:37.676 DEBUG:boto:Token: None 2024-04-19T23:26:37.676 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000701312c5a82038ca-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.679 DEBUG:boto:b'aktlji-16obj101000falseobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-19T23:26:37.679 DEBUG:boto:path=/aktlji-16/obj11 2024-04-19T23:26:37.680 DEBUG:boto:auth_path=/aktlji-16/obj11 2024-04-19T23:26:37.680 DEBUG:boto:path=/aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.680 DEBUG:boto:auth_path=/aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.680 DEBUG:boto:Method: GET 2024-04-19T23:26:37.680 DEBUG:boto:Path: /aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.680 DEBUG:boto:Data: 2024-04-19T23:26:37.680 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.680 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.680 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.680 DEBUG:boto:Params: {} 2024-04-19T23:26:37.680 DEBUG:boto:Token: None 2024-04-19T23:26:37.681 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.681 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI= 2024-04-19T23:26:37.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI=', 'Content-Length': '0'} 2024-04-19T23:26:37.683 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f34309d91f6eaf62-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.730 DEBUG:boto:path=/aktlji-16/obj11 2024-04-19T23:26:37.731 DEBUG:boto:auth_path=/aktlji-16/obj11 2024-04-19T23:26:37.731 DEBUG:boto:path=/aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.731 DEBUG:boto:auth_path=/aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.731 DEBUG:boto:Method: GET 2024-04-19T23:26:37.731 DEBUG:boto:Path: /aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.731 DEBUG:boto:Data: 2024-04-19T23:26:37.731 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.731 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.732 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.732 DEBUG:boto:Params: {} 2024-04-19T23:26:37.732 DEBUG:boto:Token: None 2024-04-19T23:26:37.732 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.732 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI= 2024-04-19T23:26:37.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI=', 'Content-Length': '0'} 2024-04-19T23:26:37.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b42c50fc4243647d-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.735 DEBUG:boto:path=/aktlji-16/obj11 2024-04-19T23:26:37.735 DEBUG:boto:auth_path=/aktlji-16/obj11 2024-04-19T23:26:37.735 DEBUG:boto:path=/aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.735 DEBUG:boto:auth_path=/aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.735 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.735 DEBUG:boto:Path: /aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.735 DEBUG:boto:Data: 2024-04-19T23:26:37.735 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.735 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.735 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.736 DEBUG:boto:Params: {} 2024-04-19T23:26:37.736 DEBUG:boto:Token: None 2024-04-19T23:26:37.736 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.736 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1RWwxselmKhavHETsSEAugeMNqQ= 2024-04-19T23:26:37.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1RWwxselmKhavHETsSEAugeMNqQ=', 'Content-Length': '0'} 2024-04-19T23:26:37.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096f4335698de80e2-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.738 DEBUG:boto:path=/aktlji-16/obj11 2024-04-19T23:26:37.738 DEBUG:boto:auth_path=/aktlji-16/obj11 2024-04-19T23:26:37.738 DEBUG:boto:path=/aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.738 DEBUG:boto:auth_path=/aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.738 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.739 DEBUG:boto:Path: /aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.739 DEBUG:boto:Data: 2024-04-19T23:26:37.739 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.739 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.739 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.739 DEBUG:boto:Params: {} 2024-04-19T23:26:37.739 DEBUG:boto:Token: None 2024-04-19T23:26:37.739 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.739 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1RWwxselmKhavHETsSEAugeMNqQ= 2024-04-19T23:26:37.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1RWwxselmKhavHETsSEAugeMNqQ=', 'Content-Length': '0'} 2024-04-19T23:26:37.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5f15399958c7eaf-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-19T23:26:37.742 DEBUG:boto:path=/aktlji-16/obj11 2024-04-19T23:26:37.742 DEBUG:boto:auth_path=/aktlji-16/obj11 2024-04-19T23:26:37.742 DEBUG:boto:path=/aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.742 DEBUG:boto:auth_path=/aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.742 DEBUG:boto:Method: GET 2024-04-19T23:26:37.742 DEBUG:boto:Path: /aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.742 DEBUG:boto:Data: 2024-04-19T23:26:37.742 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.742 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.742 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.742 DEBUG:boto:Params: {} 2024-04-19T23:26:37.743 DEBUG:boto:Token: None 2024-04-19T23:26:37.743 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.743 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI= 2024-04-19T23:26:37.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI=', 'Content-Length': '0'} 2024-04-19T23:26:37.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009cf8238059e97e6-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.745 DEBUG:boto:path=/aktlji-16/obj11 2024-04-19T23:26:37.745 DEBUG:boto:auth_path=/aktlji-16/obj11 2024-04-19T23:26:37.745 DEBUG:boto:path=/aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.745 DEBUG:boto:auth_path=/aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.745 DEBUG:boto:Method: GET 2024-04-19T23:26:37.746 DEBUG:boto:Path: /aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.746 DEBUG:boto:Data: 2024-04-19T23:26:37.746 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.746 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.746 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.746 DEBUG:boto:Params: {} 2024-04-19T23:26:37.746 DEBUG:boto:Token: None 2024-04-19T23:26:37.746 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj11?versionId=null 2024-04-19T23:26:37.746 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI= 2024-04-19T23:26:37.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI=', 'Content-Length': '0'} 2024-04-19T23:26:37.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000215844e24ec434d3-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.748 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.749 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.749 DEBUG:boto:path=/aktlji-16/?prefix=obj11 2024-04-19T23:26:37.749 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj11 2024-04-19T23:26:37.749 DEBUG:boto:Method: GET 2024-04-19T23:26:37.749 DEBUG:boto:Path: /aktlji-16/?prefix=obj11 2024-04-19T23:26:37.749 DEBUG:boto:Data: 2024-04-19T23:26:37.749 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.749 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.749 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.749 DEBUG:boto:Params: {} 2024-04-19T23:26:37.749 DEBUG:boto:Token: None 2024-04-19T23:26:37.750 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.750 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000269875472c15e668-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.753 DEBUG:boto:b'aktlji-16obj111000falseobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.753 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.753 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.753 DEBUG:boto:path=/aktlji-16/?prefix=obj11 2024-04-19T23:26:37.753 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj11 2024-04-19T23:26:37.753 DEBUG:boto:Method: GET 2024-04-19T23:26:37.753 DEBUG:boto:Path: /aktlji-16/?prefix=obj11 2024-04-19T23:26:37.754 DEBUG:boto:Data: 2024-04-19T23:26:37.754 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.754 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.754 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.754 DEBUG:boto:Params: {} 2024-04-19T23:26:37.754 DEBUG:boto:Token: None 2024-04-19T23:26:37.754 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.754 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e54ce20b1fe8b584-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.757 DEBUG:boto:b'aktlji-16obj111000falseobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.757 DEBUG:boto:path=/aktlji-16/obj11 2024-04-19T23:26:37.758 DEBUG:boto:auth_path=/aktlji-16/obj11 2024-04-19T23:26:37.758 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.758 DEBUG:boto:Path: /aktlji-16/obj11 2024-04-19T23:26:37.758 DEBUG:boto:Data: 2024-04-19T23:26:37.758 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.758 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.758 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.758 DEBUG:boto:Params: {} 2024-04-19T23:26:37.758 DEBUG:boto:Token: None 2024-04-19T23:26:37.759 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj11 2024-04-19T23:26:37.759 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LoIjMd+TJkqxIdEed9CjYa1F3aI= 2024-04-19T23:26:37.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LoIjMd+TJkqxIdEed9CjYa1F3aI=', 'Content-Length': '0'} 2024-04-19T23:26:37.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066467b2b21684c94-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.761 DEBUG:boto:path=/aktlji-16/obj11 2024-04-19T23:26:37.761 DEBUG:boto:auth_path=/aktlji-16/obj11 2024-04-19T23:26:37.761 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.761 DEBUG:boto:Path: /aktlji-16/obj11 2024-04-19T23:26:37.761 DEBUG:boto:Data: 2024-04-19T23:26:37.761 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.761 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.761 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.761 DEBUG:boto:Params: {} 2024-04-19T23:26:37.762 DEBUG:boto:Token: None 2024-04-19T23:26:37.762 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj11 2024-04-19T23:26:37.762 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LoIjMd+TJkqxIdEed9CjYa1F3aI= 2024-04-19T23:26:37.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LoIjMd+TJkqxIdEed9CjYa1F3aI=', 'Content-Length': '0'} 2024-04-19T23:26:37.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c56f7b609e5f6666-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-19T23:26:37.764 DEBUG:boto:path=/aktlji-16/obj11 2024-04-19T23:26:37.764 DEBUG:boto:auth_path=/aktlji-16/obj11 2024-04-19T23:26:37.764 DEBUG:boto:Method: GET 2024-04-19T23:26:37.764 DEBUG:boto:Path: /aktlji-16/obj11 2024-04-19T23:26:37.764 DEBUG:boto:Data: 2024-04-19T23:26:37.764 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.764 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.764 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.764 DEBUG:boto:Params: {} 2024-04-19T23:26:37.765 DEBUG:boto:Token: None 2024-04-19T23:26:37.765 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj11 2024-04-19T23:26:37.765 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b1UbXEXDcF07K6+NScReW1L5wFs= 2024-04-19T23:26:37.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b1UbXEXDcF07K6+NScReW1L5wFs=', 'Content-Length': '0'} 2024-04-19T23:26:37.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e0fb464d9db8d55-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.767 DEBUG:boto:path=/aktlji-16/obj11 2024-04-19T23:26:37.767 DEBUG:boto:auth_path=/aktlji-16/obj11 2024-04-19T23:26:37.767 DEBUG:boto:Method: GET 2024-04-19T23:26:37.767 DEBUG:boto:Path: /aktlji-16/obj11 2024-04-19T23:26:37.767 DEBUG:boto:Data: 2024-04-19T23:26:37.767 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.767 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.768 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.768 DEBUG:boto:Params: {} 2024-04-19T23:26:37.768 DEBUG:boto:Token: None 2024-04-19T23:26:37.768 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj11 2024-04-19T23:26:37.768 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b1UbXEXDcF07K6+NScReW1L5wFs= 2024-04-19T23:26:37.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b1UbXEXDcF07K6+NScReW1L5wFs=', 'Content-Length': '0'} 2024-04-19T23:26:37.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005afba0fa101dda2f-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.770 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.770 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.770 DEBUG:boto:path=/aktlji-16/?prefix=obj11 2024-04-19T23:26:37.770 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj11 2024-04-19T23:26:37.771 DEBUG:boto:Method: GET 2024-04-19T23:26:37.771 DEBUG:boto:Path: /aktlji-16/?prefix=obj11 2024-04-19T23:26:37.771 DEBUG:boto:Data: 2024-04-19T23:26:37.771 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.771 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.771 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.771 DEBUG:boto:Params: {} 2024-04-19T23:26:37.771 DEBUG:boto:Token: None 2024-04-19T23:26:37.771 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.771 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1b55db7cd6554b9-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.774 DEBUG:boto:b'aktlji-16obj111000falseobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.774 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.774 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.774 DEBUG:boto:path=/aktlji-16/?prefix=obj11 2024-04-19T23:26:37.775 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj11 2024-04-19T23:26:37.775 DEBUG:boto:Method: GET 2024-04-19T23:26:37.775 DEBUG:boto:Path: /aktlji-16/?prefix=obj11 2024-04-19T23:26:37.775 DEBUG:boto:Data: 2024-04-19T23:26:37.775 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.775 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.775 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.775 DEBUG:boto:Params: {} 2024-04-19T23:26:37.775 DEBUG:boto:Token: None 2024-04-19T23:26:37.775 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.775 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ba15a764df090cf-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.778 DEBUG:boto:b'aktlji-16obj111000falseobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj12 2024-04-19T23:26:37.778 DEBUG:boto:path=/aktlji-16/obj12 2024-04-19T23:26:37.779 DEBUG:boto:auth_path=/aktlji-16/obj12 2024-04-19T23:26:37.779 DEBUG:boto:path=/aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.779 DEBUG:boto:auth_path=/aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.779 DEBUG:boto:Method: GET 2024-04-19T23:26:37.779 DEBUG:boto:Path: /aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.779 DEBUG:boto:Data: 2024-04-19T23:26:37.779 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.779 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.779 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.779 DEBUG:boto:Params: {} 2024-04-19T23:26:37.779 DEBUG:boto:Token: None 2024-04-19T23:26:37.780 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.780 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc= 2024-04-19T23:26:37.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc=', 'Content-Length': '0'} 2024-04-19T23:26:37.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053cef70fa241b58a-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.782 DEBUG:boto:path=/aktlji-16/obj12 2024-04-19T23:26:37.782 DEBUG:boto:auth_path=/aktlji-16/obj12 2024-04-19T23:26:37.782 DEBUG:boto:path=/aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.782 DEBUG:boto:auth_path=/aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.782 DEBUG:boto:Method: GET 2024-04-19T23:26:37.782 DEBUG:boto:Path: /aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.782 DEBUG:boto:Data: 2024-04-19T23:26:37.782 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.782 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.783 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.783 DEBUG:boto:Params: {} 2024-04-19T23:26:37.783 DEBUG:boto:Token: None 2024-04-19T23:26:37.783 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.783 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc= 2024-04-19T23:26:37.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc=', 'Content-Length': '0'} 2024-04-19T23:26:37.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001732fdd64ca27cf3-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.785 DEBUG:boto:path=/aktlji-16/obj12 2024-04-19T23:26:37.785 DEBUG:boto:auth_path=/aktlji-16/obj12 2024-04-19T23:26:37.785 DEBUG:boto:path=/aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.785 DEBUG:boto:auth_path=/aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.785 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.786 DEBUG:boto:Path: /aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.786 DEBUG:boto:Data: 2024-04-19T23:26:37.786 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.786 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.786 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.786 DEBUG:boto:Params: {} 2024-04-19T23:26:37.786 DEBUG:boto:Token: None 2024-04-19T23:26:37.786 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.786 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u9m+LNwO0o7Sh5tUHttHsNZQEMw= 2024-04-19T23:26:37.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u9m+LNwO0o7Sh5tUHttHsNZQEMw=', 'Content-Length': '0'} 2024-04-19T23:26:37.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b82491bcf128a1b4-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.788 DEBUG:boto:path=/aktlji-16/obj12 2024-04-19T23:26:37.788 DEBUG:boto:auth_path=/aktlji-16/obj12 2024-04-19T23:26:37.788 DEBUG:boto:path=/aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.789 DEBUG:boto:auth_path=/aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.789 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.789 DEBUG:boto:Path: /aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.789 DEBUG:boto:Data: 2024-04-19T23:26:37.789 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.789 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.789 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.789 DEBUG:boto:Params: {} 2024-04-19T23:26:37.789 DEBUG:boto:Token: None 2024-04-19T23:26:37.789 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.789 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u9m+LNwO0o7Sh5tUHttHsNZQEMw= 2024-04-19T23:26:37.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u9m+LNwO0o7Sh5tUHttHsNZQEMw=', 'Content-Length': '0'} 2024-04-19T23:26:37.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af9e6178cfabd94e-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj12 2024-04-19T23:26:37.792 DEBUG:boto:path=/aktlji-16/obj12 2024-04-19T23:26:37.792 DEBUG:boto:auth_path=/aktlji-16/obj12 2024-04-19T23:26:37.792 DEBUG:boto:path=/aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.792 DEBUG:boto:auth_path=/aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.792 DEBUG:boto:Method: GET 2024-04-19T23:26:37.792 DEBUG:boto:Path: /aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.792 DEBUG:boto:Data: 2024-04-19T23:26:37.792 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.792 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.793 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.793 DEBUG:boto:Params: {} 2024-04-19T23:26:37.793 DEBUG:boto:Token: None 2024-04-19T23:26:37.793 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.793 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc= 2024-04-19T23:26:37.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc=', 'Content-Length': '0'} 2024-04-19T23:26:37.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d18e7209f78b7c54-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.795 DEBUG:boto:path=/aktlji-16/obj12 2024-04-19T23:26:37.795 DEBUG:boto:auth_path=/aktlji-16/obj12 2024-04-19T23:26:37.795 DEBUG:boto:path=/aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.795 DEBUG:boto:auth_path=/aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.796 DEBUG:boto:Method: GET 2024-04-19T23:26:37.796 DEBUG:boto:Path: /aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.796 DEBUG:boto:Data: 2024-04-19T23:26:37.796 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.796 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.796 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.796 DEBUG:boto:Params: {} 2024-04-19T23:26:37.796 DEBUG:boto:Token: None 2024-04-19T23:26:37.796 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj12?versionId=null 2024-04-19T23:26:37.796 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc= 2024-04-19T23:26:37.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc=', 'Content-Length': '0'} 2024-04-19T23:26:37.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001797a9976d9fe624-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.838 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.839 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.839 DEBUG:boto:path=/aktlji-16/?prefix=obj12 2024-04-19T23:26:37.839 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj12 2024-04-19T23:26:37.839 DEBUG:boto:Method: GET 2024-04-19T23:26:37.839 DEBUG:boto:Path: /aktlji-16/?prefix=obj12 2024-04-19T23:26:37.839 DEBUG:boto:Data: 2024-04-19T23:26:37.839 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.839 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.839 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.839 DEBUG:boto:Params: {} 2024-04-19T23:26:37.840 DEBUG:boto:Token: None 2024-04-19T23:26:37.840 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.840 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009722e4e80b3b49a1-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.843 DEBUG:boto:b'aktlji-16obj121000falseobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.843 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.843 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.844 DEBUG:boto:path=/aktlji-16/?prefix=obj12 2024-04-19T23:26:37.844 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj12 2024-04-19T23:26:37.844 DEBUG:boto:Method: GET 2024-04-19T23:26:37.844 DEBUG:boto:Path: /aktlji-16/?prefix=obj12 2024-04-19T23:26:37.844 DEBUG:boto:Data: 2024-04-19T23:26:37.844 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.844 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.844 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.844 DEBUG:boto:Params: {} 2024-04-19T23:26:37.844 DEBUG:boto:Token: None 2024-04-19T23:26:37.845 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.845 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a12535ce9f4cebac-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.848 DEBUG:boto:b'aktlji-16obj121000falseobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.848 DEBUG:boto:path=/aktlji-16/obj12 2024-04-19T23:26:37.848 DEBUG:boto:auth_path=/aktlji-16/obj12 2024-04-19T23:26:37.848 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.848 DEBUG:boto:Path: /aktlji-16/obj12 2024-04-19T23:26:37.848 DEBUG:boto:Data: 2024-04-19T23:26:37.848 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.848 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.849 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.849 DEBUG:boto:Params: {} 2024-04-19T23:26:37.849 DEBUG:boto:Token: None 2024-04-19T23:26:37.849 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj12 2024-04-19T23:26:37.849 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+3MdO0phVU+xitqxbbEoLmsbZaM= 2024-04-19T23:26:37.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+3MdO0phVU+xitqxbbEoLmsbZaM=', 'Content-Length': '0'} 2024-04-19T23:26:37.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000beebccdada001954-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.851 DEBUG:boto:path=/aktlji-16/obj12 2024-04-19T23:26:37.851 DEBUG:boto:auth_path=/aktlji-16/obj12 2024-04-19T23:26:37.851 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.851 DEBUG:boto:Path: /aktlji-16/obj12 2024-04-19T23:26:37.851 DEBUG:boto:Data: 2024-04-19T23:26:37.851 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.852 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.852 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.852 DEBUG:boto:Params: {} 2024-04-19T23:26:37.852 DEBUG:boto:Token: None 2024-04-19T23:26:37.852 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj12 2024-04-19T23:26:37.852 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+3MdO0phVU+xitqxbbEoLmsbZaM= 2024-04-19T23:26:37.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+3MdO0phVU+xitqxbbEoLmsbZaM=', 'Content-Length': '0'} 2024-04-19T23:26:37.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000937381bc3ac958c7-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj12 2024-04-19T23:26:37.854 DEBUG:boto:path=/aktlji-16/obj12 2024-04-19T23:26:37.854 DEBUG:boto:auth_path=/aktlji-16/obj12 2024-04-19T23:26:37.854 DEBUG:boto:Method: GET 2024-04-19T23:26:37.854 DEBUG:boto:Path: /aktlji-16/obj12 2024-04-19T23:26:37.855 DEBUG:boto:Data: 2024-04-19T23:26:37.855 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.855 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.855 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.855 DEBUG:boto:Params: {} 2024-04-19T23:26:37.855 DEBUG:boto:Token: None 2024-04-19T23:26:37.855 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj12 2024-04-19T23:26:37.855 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DUKfNDThPBNbWLt3E94KBuyAE4M= 2024-04-19T23:26:37.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DUKfNDThPBNbWLt3E94KBuyAE4M=', 'Content-Length': '0'} 2024-04-19T23:26:37.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e5121b7a7b4f5e0-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.898 DEBUG:boto:path=/aktlji-16/obj12 2024-04-19T23:26:37.899 DEBUG:boto:auth_path=/aktlji-16/obj12 2024-04-19T23:26:37.899 DEBUG:boto:Method: GET 2024-04-19T23:26:37.899 DEBUG:boto:Path: /aktlji-16/obj12 2024-04-19T23:26:37.899 DEBUG:boto:Data: 2024-04-19T23:26:37.899 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.899 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.899 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.899 DEBUG:boto:Params: {} 2024-04-19T23:26:37.899 DEBUG:boto:Token: None 2024-04-19T23:26:37.900 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj12 2024-04-19T23:26:37.900 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DUKfNDThPBNbWLt3E94KBuyAE4M= 2024-04-19T23:26:37.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DUKfNDThPBNbWLt3E94KBuyAE4M=', 'Content-Length': '0'} 2024-04-19T23:26:37.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ff32ba48ba123a7-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.902 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.902 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.902 DEBUG:boto:path=/aktlji-16/?prefix=obj12 2024-04-19T23:26:37.902 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj12 2024-04-19T23:26:37.902 DEBUG:boto:Method: GET 2024-04-19T23:26:37.902 DEBUG:boto:Path: /aktlji-16/?prefix=obj12 2024-04-19T23:26:37.903 DEBUG:boto:Data: 2024-04-19T23:26:37.903 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.903 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.903 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.903 DEBUG:boto:Params: {} 2024-04-19T23:26:37.903 DEBUG:boto:Token: None 2024-04-19T23:26:37.903 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.904 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000368fd1c4ddeb2d35-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.907 DEBUG:boto:b'aktlji-16obj121000falseobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.907 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.907 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.907 DEBUG:boto:path=/aktlji-16/?prefix=obj12 2024-04-19T23:26:37.907 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj12 2024-04-19T23:26:37.907 DEBUG:boto:Method: GET 2024-04-19T23:26:37.907 DEBUG:boto:Path: /aktlji-16/?prefix=obj12 2024-04-19T23:26:37.908 DEBUG:boto:Data: 2024-04-19T23:26:37.908 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.908 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.908 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.908 DEBUG:boto:Params: {} 2024-04-19T23:26:37.908 DEBUG:boto:Token: None 2024-04-19T23:26:37.908 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.908 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfff76ac5995f934-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.911 DEBUG:boto:b'aktlji-16obj121000falseobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj13 2024-04-19T23:26:37.911 DEBUG:boto:path=/aktlji-16/obj13 2024-04-19T23:26:37.912 DEBUG:boto:auth_path=/aktlji-16/obj13 2024-04-19T23:26:37.912 DEBUG:boto:path=/aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.912 DEBUG:boto:auth_path=/aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.912 DEBUG:boto:Method: GET 2024-04-19T23:26:37.912 DEBUG:boto:Path: /aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.912 DEBUG:boto:Data: 2024-04-19T23:26:37.912 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.912 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.912 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.912 DEBUG:boto:Params: {} 2024-04-19T23:26:37.912 DEBUG:boto:Token: None 2024-04-19T23:26:37.913 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.913 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k= 2024-04-19T23:26:37.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k=', 'Content-Length': '0'} 2024-04-19T23:26:37.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da0f77e172d47bd8-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.915 DEBUG:boto:path=/aktlji-16/obj13 2024-04-19T23:26:37.915 DEBUG:boto:auth_path=/aktlji-16/obj13 2024-04-19T23:26:37.915 DEBUG:boto:path=/aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.915 DEBUG:boto:auth_path=/aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.915 DEBUG:boto:Method: GET 2024-04-19T23:26:37.915 DEBUG:boto:Path: /aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.915 DEBUG:boto:Data: 2024-04-19T23:26:37.916 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.916 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.916 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.916 DEBUG:boto:Params: {} 2024-04-19T23:26:37.916 DEBUG:boto:Token: None 2024-04-19T23:26:37.916 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.916 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k= 2024-04-19T23:26:37.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k=', 'Content-Length': '0'} 2024-04-19T23:26:37.918 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d9a1b8a2aa59e9d-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.962 DEBUG:boto:path=/aktlji-16/obj13 2024-04-19T23:26:37.963 DEBUG:boto:auth_path=/aktlji-16/obj13 2024-04-19T23:26:37.963 DEBUG:boto:path=/aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.963 DEBUG:boto:auth_path=/aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.963 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.963 DEBUG:boto:Path: /aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.963 DEBUG:boto:Data: 2024-04-19T23:26:37.963 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.963 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.963 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.963 DEBUG:boto:Params: {} 2024-04-19T23:26:37.964 DEBUG:boto:Token: None 2024-04-19T23:26:37.964 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.964 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HBFjf2wUQV8eZNkku7iK3D81QIo= 2024-04-19T23:26:37.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HBFjf2wUQV8eZNkku7iK3D81QIo=', 'Content-Length': '0'} 2024-04-19T23:26:37.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9442cbff09dfab9-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.966 DEBUG:boto:path=/aktlji-16/obj13 2024-04-19T23:26:37.966 DEBUG:boto:auth_path=/aktlji-16/obj13 2024-04-19T23:26:37.966 DEBUG:boto:path=/aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.966 DEBUG:boto:auth_path=/aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.966 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.967 DEBUG:boto:Path: /aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.967 DEBUG:boto:Data: 2024-04-19T23:26:37.967 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.967 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.967 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.967 DEBUG:boto:Params: {} 2024-04-19T23:26:37.967 DEBUG:boto:Token: None 2024-04-19T23:26:37.967 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.967 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HBFjf2wUQV8eZNkku7iK3D81QIo= 2024-04-19T23:26:37.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HBFjf2wUQV8eZNkku7iK3D81QIo=', 'Content-Length': '0'} 2024-04-19T23:26:37.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f79f801f32aef7bd-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj13 2024-04-19T23:26:37.970 DEBUG:boto:path=/aktlji-16/obj13 2024-04-19T23:26:37.970 DEBUG:boto:auth_path=/aktlji-16/obj13 2024-04-19T23:26:37.970 DEBUG:boto:path=/aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.970 DEBUG:boto:auth_path=/aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.970 DEBUG:boto:Method: GET 2024-04-19T23:26:37.970 DEBUG:boto:Path: /aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.970 DEBUG:boto:Data: 2024-04-19T23:26:37.970 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.970 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.970 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.971 DEBUG:boto:Params: {} 2024-04-19T23:26:37.971 DEBUG:boto:Token: None 2024-04-19T23:26:37.971 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.971 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k= 2024-04-19T23:26:37.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k=', 'Content-Length': '0'} 2024-04-19T23:26:37.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000500e9a049d0a2b83-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.973 DEBUG:boto:path=/aktlji-16/obj13 2024-04-19T23:26:37.973 DEBUG:boto:auth_path=/aktlji-16/obj13 2024-04-19T23:26:37.973 DEBUG:boto:path=/aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.974 DEBUG:boto:auth_path=/aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.974 DEBUG:boto:Method: GET 2024-04-19T23:26:37.974 DEBUG:boto:Path: /aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.974 DEBUG:boto:Data: 2024-04-19T23:26:37.974 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.974 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.974 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.974 DEBUG:boto:Params: {} 2024-04-19T23:26:37.974 DEBUG:boto:Token: None 2024-04-19T23:26:37.974 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj13?versionId=null 2024-04-19T23:26:37.975 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k= 2024-04-19T23:26:37.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k=', 'Content-Length': '0'} 2024-04-19T23:26:37.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cdfe8fd0cf38175-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.977 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.977 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.977 DEBUG:boto:path=/aktlji-16/?prefix=obj13 2024-04-19T23:26:37.977 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj13 2024-04-19T23:26:37.977 DEBUG:boto:Method: GET 2024-04-19T23:26:37.977 DEBUG:boto:Path: /aktlji-16/?prefix=obj13 2024-04-19T23:26:37.977 DEBUG:boto:Data: 2024-04-19T23:26:37.977 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.977 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.978 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.978 DEBUG:boto:Params: {} 2024-04-19T23:26:37.978 DEBUG:boto:Token: None 2024-04-19T23:26:37.978 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.978 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f03a6e95579240d-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.982 DEBUG:boto:b'aktlji-16obj131000falseobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.982 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:37.982 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:37.982 DEBUG:boto:path=/aktlji-16/?prefix=obj13 2024-04-19T23:26:37.982 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj13 2024-04-19T23:26:37.983 DEBUG:boto:Method: GET 2024-04-19T23:26:37.983 DEBUG:boto:Path: /aktlji-16/?prefix=obj13 2024-04-19T23:26:37.983 DEBUG:boto:Data: 2024-04-19T23:26:37.983 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.983 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.983 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.983 DEBUG:boto:Params: {} 2024-04-19T23:26:37.983 DEBUG:boto:Token: None 2024-04-19T23:26:37.983 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/ 2024-04-19T23:26:37.983 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-19T23:26:37.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-19T23:26:37.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fffa5e9a1ccc40dd-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.986 DEBUG:boto:b'aktlji-16obj131000falseobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:37.986 DEBUG:boto:path=/aktlji-16/obj13 2024-04-19T23:26:37.987 DEBUG:boto:auth_path=/aktlji-16/obj13 2024-04-19T23:26:37.987 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.987 DEBUG:boto:Path: /aktlji-16/obj13 2024-04-19T23:26:37.987 DEBUG:boto:Data: 2024-04-19T23:26:37.987 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.987 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.987 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.987 DEBUG:boto:Params: {} 2024-04-19T23:26:37.987 DEBUG:boto:Token: None 2024-04-19T23:26:37.988 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj13 2024-04-19T23:26:37.988 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:43N/TZyO7/FbVNF9eR5Lvvr6JNk= 2024-04-19T23:26:37.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:43N/TZyO7/FbVNF9eR5Lvvr6JNk=', 'Content-Length': '0'} 2024-04-19T23:26:37.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6986e7ca89d6427-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.990 DEBUG:boto:path=/aktlji-16/obj13 2024-04-19T23:26:37.990 DEBUG:boto:auth_path=/aktlji-16/obj13 2024-04-19T23:26:37.990 DEBUG:boto:Method: HEAD 2024-04-19T23:26:37.990 DEBUG:boto:Path: /aktlji-16/obj13 2024-04-19T23:26:37.990 DEBUG:boto:Data: 2024-04-19T23:26:37.990 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.990 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.990 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.990 DEBUG:boto:Params: {} 2024-04-19T23:26:37.991 DEBUG:boto:Token: None 2024-04-19T23:26:37.991 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj13 2024-04-19T23:26:37.991 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:43N/TZyO7/FbVNF9eR5Lvvr6JNk= 2024-04-19T23:26:37.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:43N/TZyO7/FbVNF9eR5Lvvr6JNk=', 'Content-Length': '0'} 2024-04-19T23:26:37.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6da219bda440860-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj13 2024-04-19T23:26:37.993 DEBUG:boto:path=/aktlji-16/obj13 2024-04-19T23:26:37.993 DEBUG:boto:auth_path=/aktlji-16/obj13 2024-04-19T23:26:37.993 DEBUG:boto:Method: GET 2024-04-19T23:26:37.993 DEBUG:boto:Path: /aktlji-16/obj13 2024-04-19T23:26:37.993 DEBUG:boto:Data: 2024-04-19T23:26:37.993 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.993 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:37.993 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.994 DEBUG:boto:Params: {} 2024-04-19T23:26:37.994 DEBUG:boto:Token: None 2024-04-19T23:26:37.994 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj13 2024-04-19T23:26:37.994 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nZVfLDYF22fcG1Yp78UYlBG4yCo= 2024-04-19T23:26:37.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nZVfLDYF22fcG1Yp78UYlBG4yCo=', 'Content-Length': '0'} 2024-04-19T23:26:37.996 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbe45a9b2a6397f6-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:37.996 DEBUG:boto:path=/aktlji-16/obj13 2024-04-19T23:26:37.996 DEBUG:boto:auth_path=/aktlji-16/obj13 2024-04-19T23:26:37.996 DEBUG:boto:Method: GET 2024-04-19T23:26:37.997 DEBUG:boto:Path: /aktlji-16/obj13 2024-04-19T23:26:37.997 DEBUG:boto:Data: 2024-04-19T23:26:37.997 DEBUG:boto:Headers: {} 2024-04-19T23:26:37.997 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:37.997 DEBUG:boto:Port: 8000 2024-04-19T23:26:37.997 DEBUG:boto:Params: {} 2024-04-19T23:26:37.997 DEBUG:boto:Token: None 2024-04-19T23:26:37.997 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:37 GMT /aktlji-16/obj13 2024-04-19T23:26:37.998 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nZVfLDYF22fcG1Yp78UYlBG4yCo= 2024-04-19T23:26:37.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nZVfLDYF22fcG1Yp78UYlBG4yCo=', 'Content-Length': '0'} 2024-04-19T23:26:37.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a058b187bc97393-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.000 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.000 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.000 DEBUG:boto:path=/aktlji-16/?prefix=obj13 2024-04-19T23:26:38.000 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj13 2024-04-19T23:26:38.000 DEBUG:boto:Method: GET 2024-04-19T23:26:38.000 DEBUG:boto:Path: /aktlji-16/?prefix=obj13 2024-04-19T23:26:38.000 DEBUG:boto:Data: 2024-04-19T23:26:38.000 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.001 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.001 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.001 DEBUG:boto:Params: {} 2024-04-19T23:26:38.001 DEBUG:boto:Token: None 2024-04-19T23:26:38.001 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.001 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7cc2b26bb1c8a0e-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.004 DEBUG:boto:b'aktlji-16obj131000falseobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.004 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.004 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.004 DEBUG:boto:path=/aktlji-16/?prefix=obj13 2024-04-19T23:26:38.004 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj13 2024-04-19T23:26:38.004 DEBUG:boto:Method: GET 2024-04-19T23:26:38.005 DEBUG:boto:Path: /aktlji-16/?prefix=obj13 2024-04-19T23:26:38.005 DEBUG:boto:Data: 2024-04-19T23:26:38.005 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.005 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.005 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.005 DEBUG:boto:Params: {} 2024-04-19T23:26:38.005 DEBUG:boto:Token: None 2024-04-19T23:26:38.005 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.005 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ddb0809b66ed7c2-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.008 DEBUG:boto:b'aktlji-16obj131000falseobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj14 2024-04-19T23:26:38.009 DEBUG:boto:path=/aktlji-16/obj14 2024-04-19T23:26:38.009 DEBUG:boto:auth_path=/aktlji-16/obj14 2024-04-19T23:26:38.009 DEBUG:boto:path=/aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.009 DEBUG:boto:auth_path=/aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.009 DEBUG:boto:Method: GET 2024-04-19T23:26:38.009 DEBUG:boto:Path: /aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.009 DEBUG:boto:Data: 2024-04-19T23:26:38.009 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.009 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.010 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.010 DEBUG:boto:Params: {} 2024-04-19T23:26:38.010 DEBUG:boto:Token: None 2024-04-19T23:26:38.010 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.010 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY= 2024-04-19T23:26:38.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY=', 'Content-Length': '0'} 2024-04-19T23:26:38.012 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000045dc715bffbd2d7-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.058 DEBUG:boto:path=/aktlji-16/obj14 2024-04-19T23:26:38.059 DEBUG:boto:auth_path=/aktlji-16/obj14 2024-04-19T23:26:38.059 DEBUG:boto:path=/aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.059 DEBUG:boto:auth_path=/aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.059 DEBUG:boto:Method: GET 2024-04-19T23:26:38.059 DEBUG:boto:Path: /aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.059 DEBUG:boto:Data: 2024-04-19T23:26:38.059 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.059 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.059 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.060 DEBUG:boto:Params: {} 2024-04-19T23:26:38.060 DEBUG:boto:Token: None 2024-04-19T23:26:38.060 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.060 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY= 2024-04-19T23:26:38.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY=', 'Content-Length': '0'} 2024-04-19T23:26:38.062 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000392f016d9552301a-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.063 DEBUG:boto:path=/aktlji-16/obj14 2024-04-19T23:26:38.063 DEBUG:boto:auth_path=/aktlji-16/obj14 2024-04-19T23:26:38.063 DEBUG:boto:path=/aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.063 DEBUG:boto:auth_path=/aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.063 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.063 DEBUG:boto:Path: /aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.063 DEBUG:boto:Data: 2024-04-19T23:26:38.063 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.064 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.064 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.064 DEBUG:boto:Params: {} 2024-04-19T23:26:38.064 DEBUG:boto:Token: None 2024-04-19T23:26:38.064 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.064 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OPz2E/MXlCGEqkyYYAdjKy/MFyo= 2024-04-19T23:26:38.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OPz2E/MXlCGEqkyYYAdjKy/MFyo=', 'Content-Length': '0'} 2024-04-19T23:26:38.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000458c1718ca518483-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.066 DEBUG:boto:path=/aktlji-16/obj14 2024-04-19T23:26:38.066 DEBUG:boto:auth_path=/aktlji-16/obj14 2024-04-19T23:26:38.067 DEBUG:boto:path=/aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.067 DEBUG:boto:auth_path=/aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.067 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.067 DEBUG:boto:Path: /aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.067 DEBUG:boto:Data: 2024-04-19T23:26:38.067 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.067 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.067 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.067 DEBUG:boto:Params: {} 2024-04-19T23:26:38.067 DEBUG:boto:Token: None 2024-04-19T23:26:38.068 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.068 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OPz2E/MXlCGEqkyYYAdjKy/MFyo= 2024-04-19T23:26:38.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OPz2E/MXlCGEqkyYYAdjKy/MFyo=', 'Content-Length': '0'} 2024-04-19T23:26:38.069 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f443cab64c57c8f1-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj14 2024-04-19T23:26:38.070 DEBUG:boto:path=/aktlji-16/obj14 2024-04-19T23:26:38.070 DEBUG:boto:auth_path=/aktlji-16/obj14 2024-04-19T23:26:38.070 DEBUG:boto:path=/aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.070 DEBUG:boto:auth_path=/aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.070 DEBUG:boto:Method: GET 2024-04-19T23:26:38.070 DEBUG:boto:Path: /aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.070 DEBUG:boto:Data: 2024-04-19T23:26:38.070 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.070 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.070 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.070 DEBUG:boto:Params: {} 2024-04-19T23:26:38.071 DEBUG:boto:Token: None 2024-04-19T23:26:38.071 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.071 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY= 2024-04-19T23:26:38.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY=', 'Content-Length': '0'} 2024-04-19T23:26:38.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006841b7bcc406452b-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.073 DEBUG:boto:path=/aktlji-16/obj14 2024-04-19T23:26:38.073 DEBUG:boto:auth_path=/aktlji-16/obj14 2024-04-19T23:26:38.073 DEBUG:boto:path=/aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.073 DEBUG:boto:auth_path=/aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.073 DEBUG:boto:Method: GET 2024-04-19T23:26:38.074 DEBUG:boto:Path: /aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.074 DEBUG:boto:Data: 2024-04-19T23:26:38.074 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.074 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.074 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.074 DEBUG:boto:Params: {} 2024-04-19T23:26:38.074 DEBUG:boto:Token: None 2024-04-19T23:26:38.074 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj14?versionId=null 2024-04-19T23:26:38.074 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY= 2024-04-19T23:26:38.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY=', 'Content-Length': '0'} 2024-04-19T23:26:38.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c555a8f5603b90b8-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.122 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.123 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.123 DEBUG:boto:path=/aktlji-16/?prefix=obj14 2024-04-19T23:26:38.123 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj14 2024-04-19T23:26:38.123 DEBUG:boto:Method: GET 2024-04-19T23:26:38.123 DEBUG:boto:Path: /aktlji-16/?prefix=obj14 2024-04-19T23:26:38.123 DEBUG:boto:Data: 2024-04-19T23:26:38.123 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.123 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.123 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.123 DEBUG:boto:Params: {} 2024-04-19T23:26:38.124 DEBUG:boto:Token: None 2024-04-19T23:26:38.124 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.124 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065fd0ec3f9b64183-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.127 DEBUG:boto:b'aktlji-16obj141000falseobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.128 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.128 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.128 DEBUG:boto:path=/aktlji-16/?prefix=obj14 2024-04-19T23:26:38.128 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj14 2024-04-19T23:26:38.128 DEBUG:boto:Method: GET 2024-04-19T23:26:38.128 DEBUG:boto:Path: /aktlji-16/?prefix=obj14 2024-04-19T23:26:38.128 DEBUG:boto:Data: 2024-04-19T23:26:38.128 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.129 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.129 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.129 DEBUG:boto:Params: {} 2024-04-19T23:26:38.129 DEBUG:boto:Token: None 2024-04-19T23:26:38.129 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.129 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7cacc24471827b4-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.132 DEBUG:boto:b'aktlji-16obj141000falseobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.132 DEBUG:boto:path=/aktlji-16/obj14 2024-04-19T23:26:38.132 DEBUG:boto:auth_path=/aktlji-16/obj14 2024-04-19T23:26:38.132 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.132 DEBUG:boto:Path: /aktlji-16/obj14 2024-04-19T23:26:38.133 DEBUG:boto:Data: 2024-04-19T23:26:38.133 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.133 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.133 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.133 DEBUG:boto:Params: {} 2024-04-19T23:26:38.133 DEBUG:boto:Token: None 2024-04-19T23:26:38.133 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj14 2024-04-19T23:26:38.133 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0RKRSF4W85aEi6HmYWJTrSBpmiw= 2024-04-19T23:26:38.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0RKRSF4W85aEi6HmYWJTrSBpmiw=', 'Content-Length': '0'} 2024-04-19T23:26:38.135 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b535b7d05e999ad5-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.136 DEBUG:boto:path=/aktlji-16/obj14 2024-04-19T23:26:38.136 DEBUG:boto:auth_path=/aktlji-16/obj14 2024-04-19T23:26:38.136 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.136 DEBUG:boto:Path: /aktlji-16/obj14 2024-04-19T23:26:38.136 DEBUG:boto:Data: 2024-04-19T23:26:38.136 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.136 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.136 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.136 DEBUG:boto:Params: {} 2024-04-19T23:26:38.136 DEBUG:boto:Token: None 2024-04-19T23:26:38.137 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj14 2024-04-19T23:26:38.137 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0RKRSF4W85aEi6HmYWJTrSBpmiw= 2024-04-19T23:26:38.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0RKRSF4W85aEi6HmYWJTrSBpmiw=', 'Content-Length': '0'} 2024-04-19T23:26:38.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024e9e8505d7ec048-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj14 2024-04-19T23:26:38.139 DEBUG:boto:path=/aktlji-16/obj14 2024-04-19T23:26:38.139 DEBUG:boto:auth_path=/aktlji-16/obj14 2024-04-19T23:26:38.139 DEBUG:boto:Method: GET 2024-04-19T23:26:38.139 DEBUG:boto:Path: /aktlji-16/obj14 2024-04-19T23:26:38.139 DEBUG:boto:Data: 2024-04-19T23:26:38.140 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.140 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.140 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.140 DEBUG:boto:Params: {} 2024-04-19T23:26:38.140 DEBUG:boto:Token: None 2024-04-19T23:26:38.141 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj14 2024-04-19T23:26:38.141 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iE8fTNz96M7vStGnybiEI/cSqUw= 2024-04-19T23:26:38.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iE8fTNz96M7vStGnybiEI/cSqUw=', 'Content-Length': '0'} 2024-04-19T23:26:38.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e89aa8de612aa14-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.186 DEBUG:boto:path=/aktlji-16/obj14 2024-04-19T23:26:38.187 DEBUG:boto:auth_path=/aktlji-16/obj14 2024-04-19T23:26:38.187 DEBUG:boto:Method: GET 2024-04-19T23:26:38.187 DEBUG:boto:Path: /aktlji-16/obj14 2024-04-19T23:26:38.187 DEBUG:boto:Data: 2024-04-19T23:26:38.187 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.190 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.190 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.190 DEBUG:boto:Params: {} 2024-04-19T23:26:38.190 DEBUG:boto:Token: None 2024-04-19T23:26:38.190 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj14 2024-04-19T23:26:38.190 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iE8fTNz96M7vStGnybiEI/cSqUw= 2024-04-19T23:26:38.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iE8fTNz96M7vStGnybiEI/cSqUw=', 'Content-Length': '0'} 2024-04-19T23:26:38.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f90d616fd6c6c25-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.196 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.197 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.197 DEBUG:boto:path=/aktlji-16/?prefix=obj14 2024-04-19T23:26:38.197 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj14 2024-04-19T23:26:38.197 DEBUG:boto:Method: GET 2024-04-19T23:26:38.197 DEBUG:boto:Path: /aktlji-16/?prefix=obj14 2024-04-19T23:26:38.197 DEBUG:boto:Data: 2024-04-19T23:26:38.198 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.198 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.198 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.198 DEBUG:boto:Params: {} 2024-04-19T23:26:38.198 DEBUG:boto:Token: None 2024-04-19T23:26:38.199 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.199 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d50af4d7b078b974-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.202 DEBUG:boto:b'aktlji-16obj141000falseobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.202 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.203 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.203 DEBUG:boto:path=/aktlji-16/?prefix=obj14 2024-04-19T23:26:38.203 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj14 2024-04-19T23:26:38.203 DEBUG:boto:Method: GET 2024-04-19T23:26:38.203 DEBUG:boto:Path: /aktlji-16/?prefix=obj14 2024-04-19T23:26:38.204 DEBUG:boto:Data: 2024-04-19T23:26:38.204 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.204 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.204 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.204 DEBUG:boto:Params: {} 2024-04-19T23:26:38.205 DEBUG:boto:Token: None 2024-04-19T23:26:38.205 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.205 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a37acf56199ba5bc-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.250 DEBUG:boto:b'aktlji-16obj141000falseobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj15 2024-04-19T23:26:38.251 DEBUG:boto:path=/aktlji-16/obj15 2024-04-19T23:26:38.251 DEBUG:boto:auth_path=/aktlji-16/obj15 2024-04-19T23:26:38.251 DEBUG:boto:path=/aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.251 DEBUG:boto:auth_path=/aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.251 DEBUG:boto:Method: GET 2024-04-19T23:26:38.251 DEBUG:boto:Path: /aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.251 DEBUG:boto:Data: 2024-04-19T23:26:38.251 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.252 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.252 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.252 DEBUG:boto:Params: {} 2024-04-19T23:26:38.252 DEBUG:boto:Token: None 2024-04-19T23:26:38.252 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.252 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ= 2024-04-19T23:26:38.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ=', 'Content-Length': '0'} 2024-04-19T23:26:38.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c8847797bfa41b9-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.254 DEBUG:boto:path=/aktlji-16/obj15 2024-04-19T23:26:38.254 DEBUG:boto:auth_path=/aktlji-16/obj15 2024-04-19T23:26:38.254 DEBUG:boto:path=/aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.255 DEBUG:boto:auth_path=/aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.255 DEBUG:boto:Method: GET 2024-04-19T23:26:38.255 DEBUG:boto:Path: /aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.255 DEBUG:boto:Data: 2024-04-19T23:26:38.255 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.255 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.255 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.255 DEBUG:boto:Params: {} 2024-04-19T23:26:38.256 DEBUG:boto:Token: None 2024-04-19T23:26:38.256 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.256 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ= 2024-04-19T23:26:38.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ=', 'Content-Length': '0'} 2024-04-19T23:26:38.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e661f03cb64c675-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.258 DEBUG:boto:path=/aktlji-16/obj15 2024-04-19T23:26:38.258 DEBUG:boto:auth_path=/aktlji-16/obj15 2024-04-19T23:26:38.258 DEBUG:boto:path=/aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.258 DEBUG:boto:auth_path=/aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.258 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.258 DEBUG:boto:Path: /aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.258 DEBUG:boto:Data: 2024-04-19T23:26:38.258 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.259 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.259 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.259 DEBUG:boto:Params: {} 2024-04-19T23:26:38.259 DEBUG:boto:Token: None 2024-04-19T23:26:38.259 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.259 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dUu+9oc8+RqEeqTMvL45iN0qru8= 2024-04-19T23:26:38.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dUu+9oc8+RqEeqTMvL45iN0qru8=', 'Content-Length': '0'} 2024-04-19T23:26:38.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc40580297a66c47-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.262 DEBUG:boto:path=/aktlji-16/obj15 2024-04-19T23:26:38.262 DEBUG:boto:auth_path=/aktlji-16/obj15 2024-04-19T23:26:38.262 DEBUG:boto:path=/aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.263 DEBUG:boto:auth_path=/aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.263 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.263 DEBUG:boto:Path: /aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.264 DEBUG:boto:Data: 2024-04-19T23:26:38.264 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.264 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.264 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.264 DEBUG:boto:Params: {} 2024-04-19T23:26:38.264 DEBUG:boto:Token: None 2024-04-19T23:26:38.265 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.265 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dUu+9oc8+RqEeqTMvL45iN0qru8= 2024-04-19T23:26:38.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dUu+9oc8+RqEeqTMvL45iN0qru8=', 'Content-Length': '0'} 2024-04-19T23:26:38.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035349728ca8e66b9-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj15 2024-04-19T23:26:38.268 DEBUG:boto:path=/aktlji-16/obj15 2024-04-19T23:26:38.268 DEBUG:boto:auth_path=/aktlji-16/obj15 2024-04-19T23:26:38.268 DEBUG:boto:path=/aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.268 DEBUG:boto:auth_path=/aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.268 DEBUG:boto:Method: GET 2024-04-19T23:26:38.268 DEBUG:boto:Path: /aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.269 DEBUG:boto:Data: 2024-04-19T23:26:38.269 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.269 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.269 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.269 DEBUG:boto:Params: {} 2024-04-19T23:26:38.269 DEBUG:boto:Token: None 2024-04-19T23:26:38.270 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.270 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ= 2024-04-19T23:26:38.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ=', 'Content-Length': '0'} 2024-04-19T23:26:38.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eeebb1caac1fcb3d-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.273 DEBUG:boto:path=/aktlji-16/obj15 2024-04-19T23:26:38.273 DEBUG:boto:auth_path=/aktlji-16/obj15 2024-04-19T23:26:38.273 DEBUG:boto:path=/aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.273 DEBUG:boto:auth_path=/aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.273 DEBUG:boto:Method: GET 2024-04-19T23:26:38.274 DEBUG:boto:Path: /aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.274 DEBUG:boto:Data: 2024-04-19T23:26:38.274 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.274 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.274 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.275 DEBUG:boto:Params: {} 2024-04-19T23:26:38.275 DEBUG:boto:Token: None 2024-04-19T23:26:38.275 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj15?versionId=null 2024-04-19T23:26:38.275 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ= 2024-04-19T23:26:38.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ=', 'Content-Length': '0'} 2024-04-19T23:26:38.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c0a4457bbf1dc62-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.278 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.278 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.278 DEBUG:boto:path=/aktlji-16/?prefix=obj15 2024-04-19T23:26:38.278 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj15 2024-04-19T23:26:38.279 DEBUG:boto:Method: GET 2024-04-19T23:26:38.279 DEBUG:boto:Path: /aktlji-16/?prefix=obj15 2024-04-19T23:26:38.279 DEBUG:boto:Data: 2024-04-19T23:26:38.279 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.279 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.279 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.280 DEBUG:boto:Params: {} 2024-04-19T23:26:38.280 DEBUG:boto:Token: None 2024-04-19T23:26:38.280 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.280 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8f3055151fc0f8d-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.284 DEBUG:boto:b'aktlji-16obj151000falseobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.284 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.285 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.285 DEBUG:boto:path=/aktlji-16/?prefix=obj15 2024-04-19T23:26:38.285 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj15 2024-04-19T23:26:38.285 DEBUG:boto:Method: GET 2024-04-19T23:26:38.285 DEBUG:boto:Path: /aktlji-16/?prefix=obj15 2024-04-19T23:26:38.286 DEBUG:boto:Data: 2024-04-19T23:26:38.286 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.286 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.286 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.286 DEBUG:boto:Params: {} 2024-04-19T23:26:38.286 DEBUG:boto:Token: None 2024-04-19T23:26:38.287 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.287 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bcdeb3928e6325b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.298 DEBUG:boto:b'aktlji-16obj151000falseobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.298 DEBUG:boto:path=/aktlji-16/obj15 2024-04-19T23:26:38.298 DEBUG:boto:auth_path=/aktlji-16/obj15 2024-04-19T23:26:38.298 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.298 DEBUG:boto:Path: /aktlji-16/obj15 2024-04-19T23:26:38.298 DEBUG:boto:Data: 2024-04-19T23:26:38.298 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.299 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.299 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.299 DEBUG:boto:Params: {} 2024-04-19T23:26:38.299 DEBUG:boto:Token: None 2024-04-19T23:26:38.299 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj15 2024-04-19T23:26:38.299 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nBoeWzh+ibTCbREgF2bdO/qAJM8= 2024-04-19T23:26:38.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nBoeWzh+ibTCbREgF2bdO/qAJM8=', 'Content-Length': '0'} 2024-04-19T23:26:38.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000947774888b2becc9-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.301 DEBUG:boto:path=/aktlji-16/obj15 2024-04-19T23:26:38.301 DEBUG:boto:auth_path=/aktlji-16/obj15 2024-04-19T23:26:38.301 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.301 DEBUG:boto:Path: /aktlji-16/obj15 2024-04-19T23:26:38.302 DEBUG:boto:Data: 2024-04-19T23:26:38.302 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.302 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.302 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.302 DEBUG:boto:Params: {} 2024-04-19T23:26:38.302 DEBUG:boto:Token: None 2024-04-19T23:26:38.302 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj15 2024-04-19T23:26:38.302 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nBoeWzh+ibTCbREgF2bdO/qAJM8= 2024-04-19T23:26:38.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nBoeWzh+ibTCbREgF2bdO/qAJM8=', 'Content-Length': '0'} 2024-04-19T23:26:38.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ad1fa7013da011f-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj15 2024-04-19T23:26:38.320 DEBUG:boto:path=/aktlji-16/obj15 2024-04-19T23:26:38.320 DEBUG:boto:auth_path=/aktlji-16/obj15 2024-04-19T23:26:38.321 DEBUG:boto:Method: GET 2024-04-19T23:26:38.321 DEBUG:boto:Path: /aktlji-16/obj15 2024-04-19T23:26:38.321 DEBUG:boto:Data: 2024-04-19T23:26:38.321 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.321 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.321 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.321 DEBUG:boto:Params: {} 2024-04-19T23:26:38.321 DEBUG:boto:Token: None 2024-04-19T23:26:38.321 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj15 2024-04-19T23:26:38.321 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uzVZ/NJqjX1+rHsk4hVdCmFqPrs= 2024-04-19T23:26:38.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uzVZ/NJqjX1+rHsk4hVdCmFqPrs=', 'Content-Length': '0'} 2024-04-19T23:26:38.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a050f1e54330ccd3-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.324 DEBUG:boto:path=/aktlji-16/obj15 2024-04-19T23:26:38.324 DEBUG:boto:auth_path=/aktlji-16/obj15 2024-04-19T23:26:38.324 DEBUG:boto:Method: GET 2024-04-19T23:26:38.324 DEBUG:boto:Path: /aktlji-16/obj15 2024-04-19T23:26:38.324 DEBUG:boto:Data: 2024-04-19T23:26:38.324 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.324 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.324 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.324 DEBUG:boto:Params: {} 2024-04-19T23:26:38.325 DEBUG:boto:Token: None 2024-04-19T23:26:38.325 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj15 2024-04-19T23:26:38.325 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uzVZ/NJqjX1+rHsk4hVdCmFqPrs= 2024-04-19T23:26:38.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uzVZ/NJqjX1+rHsk4hVdCmFqPrs=', 'Content-Length': '0'} 2024-04-19T23:26:38.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000093bd408c8cbef13-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.370 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.371 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.371 DEBUG:boto:path=/aktlji-16/?prefix=obj15 2024-04-19T23:26:38.371 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj15 2024-04-19T23:26:38.371 DEBUG:boto:Method: GET 2024-04-19T23:26:38.371 DEBUG:boto:Path: /aktlji-16/?prefix=obj15 2024-04-19T23:26:38.371 DEBUG:boto:Data: 2024-04-19T23:26:38.371 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.371 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.372 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.372 DEBUG:boto:Params: {} 2024-04-19T23:26:38.372 DEBUG:boto:Token: None 2024-04-19T23:26:38.372 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.372 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c31c2d77516c6204-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.375 DEBUG:boto:b'aktlji-16obj151000falseobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.375 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.376 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.376 DEBUG:boto:path=/aktlji-16/?prefix=obj15 2024-04-19T23:26:38.376 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj15 2024-04-19T23:26:38.376 DEBUG:boto:Method: GET 2024-04-19T23:26:38.376 DEBUG:boto:Path: /aktlji-16/?prefix=obj15 2024-04-19T23:26:38.377 DEBUG:boto:Data: 2024-04-19T23:26:38.377 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.377 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.377 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.377 DEBUG:boto:Params: {} 2024-04-19T23:26:38.377 DEBUG:boto:Token: None 2024-04-19T23:26:38.377 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.377 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000027bf28e24f3d03-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.380 DEBUG:boto:b'aktlji-16obj151000falseobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj16 2024-04-19T23:26:38.381 DEBUG:boto:path=/aktlji-16/obj16 2024-04-19T23:26:38.381 DEBUG:boto:auth_path=/aktlji-16/obj16 2024-04-19T23:26:38.381 DEBUG:boto:path=/aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.381 DEBUG:boto:auth_path=/aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.381 DEBUG:boto:Method: GET 2024-04-19T23:26:38.381 DEBUG:boto:Path: /aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.381 DEBUG:boto:Data: 2024-04-19T23:26:38.381 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.381 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.381 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.381 DEBUG:boto:Params: {} 2024-04-19T23:26:38.382 DEBUG:boto:Token: None 2024-04-19T23:26:38.382 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.382 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo= 2024-04-19T23:26:38.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo=', 'Content-Length': '0'} 2024-04-19T23:26:38.384 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037a34fd4270f8d4b-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.430 DEBUG:boto:path=/aktlji-16/obj16 2024-04-19T23:26:38.431 DEBUG:boto:auth_path=/aktlji-16/obj16 2024-04-19T23:26:38.431 DEBUG:boto:path=/aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.431 DEBUG:boto:auth_path=/aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.431 DEBUG:boto:Method: GET 2024-04-19T23:26:38.431 DEBUG:boto:Path: /aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.431 DEBUG:boto:Data: 2024-04-19T23:26:38.431 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.431 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.432 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.432 DEBUG:boto:Params: {} 2024-04-19T23:26:38.432 DEBUG:boto:Token: None 2024-04-19T23:26:38.432 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.432 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo= 2024-04-19T23:26:38.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo=', 'Content-Length': '0'} 2024-04-19T23:26:38.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002de36c73243ce49b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.435 DEBUG:boto:path=/aktlji-16/obj16 2024-04-19T23:26:38.435 DEBUG:boto:auth_path=/aktlji-16/obj16 2024-04-19T23:26:38.435 DEBUG:boto:path=/aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.435 DEBUG:boto:auth_path=/aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.435 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.435 DEBUG:boto:Path: /aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.435 DEBUG:boto:Data: 2024-04-19T23:26:38.435 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.436 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.436 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.436 DEBUG:boto:Params: {} 2024-04-19T23:26:38.436 DEBUG:boto:Token: None 2024-04-19T23:26:38.436 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.436 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aoIXtyBa7q/maP1xYAuzNgk0IDU= 2024-04-19T23:26:38.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aoIXtyBa7q/maP1xYAuzNgk0IDU=', 'Content-Length': '0'} 2024-04-19T23:26:38.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a99ca535afd4dd21-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.438 DEBUG:boto:path=/aktlji-16/obj16 2024-04-19T23:26:38.439 DEBUG:boto:auth_path=/aktlji-16/obj16 2024-04-19T23:26:38.439 DEBUG:boto:path=/aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.439 DEBUG:boto:auth_path=/aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.439 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.439 DEBUG:boto:Path: /aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.439 DEBUG:boto:Data: 2024-04-19T23:26:38.439 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.439 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.440 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.440 DEBUG:boto:Params: {} 2024-04-19T23:26:38.440 DEBUG:boto:Token: None 2024-04-19T23:26:38.440 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.441 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aoIXtyBa7q/maP1xYAuzNgk0IDU= 2024-04-19T23:26:38.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aoIXtyBa7q/maP1xYAuzNgk0IDU=', 'Content-Length': '0'} 2024-04-19T23:26:38.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ad00f70380d725c-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj16 2024-04-19T23:26:38.443 DEBUG:boto:path=/aktlji-16/obj16 2024-04-19T23:26:38.443 DEBUG:boto:auth_path=/aktlji-16/obj16 2024-04-19T23:26:38.443 DEBUG:boto:path=/aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.443 DEBUG:boto:auth_path=/aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.444 DEBUG:boto:Method: GET 2024-04-19T23:26:38.444 DEBUG:boto:Path: /aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.444 DEBUG:boto:Data: 2024-04-19T23:26:38.444 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.444 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.444 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.444 DEBUG:boto:Params: {} 2024-04-19T23:26:38.445 DEBUG:boto:Token: None 2024-04-19T23:26:38.445 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.445 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo= 2024-04-19T23:26:38.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo=', 'Content-Length': '0'} 2024-04-19T23:26:38.447 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000661a0fe359ab7196-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.447 DEBUG:boto:path=/aktlji-16/obj16 2024-04-19T23:26:38.447 DEBUG:boto:auth_path=/aktlji-16/obj16 2024-04-19T23:26:38.447 DEBUG:boto:path=/aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.448 DEBUG:boto:auth_path=/aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.448 DEBUG:boto:Method: GET 2024-04-19T23:26:38.448 DEBUG:boto:Path: /aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.448 DEBUG:boto:Data: 2024-04-19T23:26:38.448 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.449 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.449 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.449 DEBUG:boto:Params: {} 2024-04-19T23:26:38.449 DEBUG:boto:Token: None 2024-04-19T23:26:38.449 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj16?versionId=null 2024-04-19T23:26:38.449 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo= 2024-04-19T23:26:38.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo=', 'Content-Length': '0'} 2024-04-19T23:26:38.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b9347fa366809c9-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.498 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.499 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.499 DEBUG:boto:path=/aktlji-16/?prefix=obj16 2024-04-19T23:26:38.499 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj16 2024-04-19T23:26:38.499 DEBUG:boto:Method: GET 2024-04-19T23:26:38.499 DEBUG:boto:Path: /aktlji-16/?prefix=obj16 2024-04-19T23:26:38.499 DEBUG:boto:Data: 2024-04-19T23:26:38.499 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.499 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.499 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.499 DEBUG:boto:Params: {} 2024-04-19T23:26:38.499 DEBUG:boto:Token: None 2024-04-19T23:26:38.500 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.500 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048691a09b6b5b41d-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.503 DEBUG:boto:b'aktlji-16obj161000falseobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.503 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.503 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.503 DEBUG:boto:path=/aktlji-16/?prefix=obj16 2024-04-19T23:26:38.503 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj16 2024-04-19T23:26:38.504 DEBUG:boto:Method: GET 2024-04-19T23:26:38.504 DEBUG:boto:Path: /aktlji-16/?prefix=obj16 2024-04-19T23:26:38.504 DEBUG:boto:Data: 2024-04-19T23:26:38.504 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.504 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.504 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.504 DEBUG:boto:Params: {} 2024-04-19T23:26:38.504 DEBUG:boto:Token: None 2024-04-19T23:26:38.504 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.504 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000882ce372d55fadd1-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.507 DEBUG:boto:b'aktlji-16obj161000falseobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.508 DEBUG:boto:path=/aktlji-16/obj16 2024-04-19T23:26:38.508 DEBUG:boto:auth_path=/aktlji-16/obj16 2024-04-19T23:26:38.508 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.508 DEBUG:boto:Path: /aktlji-16/obj16 2024-04-19T23:26:38.508 DEBUG:boto:Data: 2024-04-19T23:26:38.508 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.508 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.508 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.508 DEBUG:boto:Params: {} 2024-04-19T23:26:38.508 DEBUG:boto:Token: None 2024-04-19T23:26:38.509 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj16 2024-04-19T23:26:38.509 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8fbRVNpMUhODOz/bSabz8Wausro= 2024-04-19T23:26:38.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8fbRVNpMUhODOz/bSabz8Wausro=', 'Content-Length': '0'} 2024-04-19T23:26:38.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000804c75dbb66bb317-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.511 DEBUG:boto:path=/aktlji-16/obj16 2024-04-19T23:26:38.511 DEBUG:boto:auth_path=/aktlji-16/obj16 2024-04-19T23:26:38.511 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.511 DEBUG:boto:Path: /aktlji-16/obj16 2024-04-19T23:26:38.511 DEBUG:boto:Data: 2024-04-19T23:26:38.511 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.511 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.511 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.511 DEBUG:boto:Params: {} 2024-04-19T23:26:38.512 DEBUG:boto:Token: None 2024-04-19T23:26:38.512 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj16 2024-04-19T23:26:38.512 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8fbRVNpMUhODOz/bSabz8Wausro= 2024-04-19T23:26:38.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8fbRVNpMUhODOz/bSabz8Wausro=', 'Content-Length': '0'} 2024-04-19T23:26:38.513 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080860c6a3f1c2403-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj16 2024-04-19T23:26:38.514 DEBUG:boto:path=/aktlji-16/obj16 2024-04-19T23:26:38.514 DEBUG:boto:auth_path=/aktlji-16/obj16 2024-04-19T23:26:38.514 DEBUG:boto:Method: GET 2024-04-19T23:26:38.514 DEBUG:boto:Path: /aktlji-16/obj16 2024-04-19T23:26:38.514 DEBUG:boto:Data: 2024-04-19T23:26:38.514 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.515 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.515 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.515 DEBUG:boto:Params: {} 2024-04-19T23:26:38.515 DEBUG:boto:Token: None 2024-04-19T23:26:38.515 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj16 2024-04-19T23:26:38.515 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hBS11ASULHnosTZ9F5oFl7seqiU= 2024-04-19T23:26:38.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hBS11ASULHnosTZ9F5oFl7seqiU=', 'Content-Length': '0'} 2024-04-19T23:26:38.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000adbc9411b84bc93-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.558 DEBUG:boto:path=/aktlji-16/obj16 2024-04-19T23:26:38.559 DEBUG:boto:auth_path=/aktlji-16/obj16 2024-04-19T23:26:38.559 DEBUG:boto:Method: GET 2024-04-19T23:26:38.559 DEBUG:boto:Path: /aktlji-16/obj16 2024-04-19T23:26:38.559 DEBUG:boto:Data: 2024-04-19T23:26:38.559 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.559 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.559 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.560 DEBUG:boto:Params: {} 2024-04-19T23:26:38.560 DEBUG:boto:Token: None 2024-04-19T23:26:38.560 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj16 2024-04-19T23:26:38.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hBS11ASULHnosTZ9F5oFl7seqiU= 2024-04-19T23:26:38.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hBS11ASULHnosTZ9F5oFl7seqiU=', 'Content-Length': '0'} 2024-04-19T23:26:38.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f7e840bf8260a13-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.563 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.563 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.563 DEBUG:boto:path=/aktlji-16/?prefix=obj16 2024-04-19T23:26:38.563 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj16 2024-04-19T23:26:38.563 DEBUG:boto:Method: GET 2024-04-19T23:26:38.564 DEBUG:boto:Path: /aktlji-16/?prefix=obj16 2024-04-19T23:26:38.564 DEBUG:boto:Data: 2024-04-19T23:26:38.564 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.564 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.564 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.564 DEBUG:boto:Params: {} 2024-04-19T23:26:38.564 DEBUG:boto:Token: None 2024-04-19T23:26:38.564 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.564 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004833cb3a5ff8ff11-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.568 DEBUG:boto:b'aktlji-16obj161000falseobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.569 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.569 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.569 DEBUG:boto:path=/aktlji-16/?prefix=obj16 2024-04-19T23:26:38.569 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj16 2024-04-19T23:26:38.569 DEBUG:boto:Method: GET 2024-04-19T23:26:38.569 DEBUG:boto:Path: /aktlji-16/?prefix=obj16 2024-04-19T23:26:38.569 DEBUG:boto:Data: 2024-04-19T23:26:38.569 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.570 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.570 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.570 DEBUG:boto:Params: {} 2024-04-19T23:26:38.570 DEBUG:boto:Token: None 2024-04-19T23:26:38.570 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.570 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ab02ffabf7c611d-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.573 DEBUG:boto:b'aktlji-16obj161000falseobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-19T23:26:38.574 DEBUG:boto:path=/aktlji-16/obj17 2024-04-19T23:26:38.574 DEBUG:boto:auth_path=/aktlji-16/obj17 2024-04-19T23:26:38.574 DEBUG:boto:path=/aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.574 DEBUG:boto:auth_path=/aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.575 DEBUG:boto:Method: GET 2024-04-19T23:26:38.575 DEBUG:boto:Path: /aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.575 DEBUG:boto:Data: 2024-04-19T23:26:38.575 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.575 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.575 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.575 DEBUG:boto:Params: {} 2024-04-19T23:26:38.576 DEBUG:boto:Token: None 2024-04-19T23:26:38.576 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.576 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0= 2024-04-19T23:26:38.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0=', 'Content-Length': '0'} 2024-04-19T23:26:38.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5ca60effc1e9337-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.578 DEBUG:boto:path=/aktlji-16/obj17 2024-04-19T23:26:38.579 DEBUG:boto:auth_path=/aktlji-16/obj17 2024-04-19T23:26:38.579 DEBUG:boto:path=/aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.579 DEBUG:boto:auth_path=/aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.579 DEBUG:boto:Method: GET 2024-04-19T23:26:38.579 DEBUG:boto:Path: /aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.579 DEBUG:boto:Data: 2024-04-19T23:26:38.579 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.579 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.580 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.580 DEBUG:boto:Params: {} 2024-04-19T23:26:38.580 DEBUG:boto:Token: None 2024-04-19T23:26:38.580 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.580 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0= 2024-04-19T23:26:38.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0=', 'Content-Length': '0'} 2024-04-19T23:26:38.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a867ae27bc2d06c-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.582 DEBUG:boto:path=/aktlji-16/obj17 2024-04-19T23:26:38.583 DEBUG:boto:auth_path=/aktlji-16/obj17 2024-04-19T23:26:38.583 DEBUG:boto:path=/aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.583 DEBUG:boto:auth_path=/aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.583 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.583 DEBUG:boto:Path: /aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.583 DEBUG:boto:Data: 2024-04-19T23:26:38.583 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.583 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.583 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.583 DEBUG:boto:Params: {} 2024-04-19T23:26:38.584 DEBUG:boto:Token: None 2024-04-19T23:26:38.584 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.584 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fZ3Kr6/0qub8zWi+NPVf8BflQsI= 2024-04-19T23:26:38.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fZ3Kr6/0qub8zWi+NPVf8BflQsI=', 'Content-Length': '0'} 2024-04-19T23:26:38.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001eb6ce3aec602f35-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.586 DEBUG:boto:path=/aktlji-16/obj17 2024-04-19T23:26:38.586 DEBUG:boto:auth_path=/aktlji-16/obj17 2024-04-19T23:26:38.586 DEBUG:boto:path=/aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.586 DEBUG:boto:auth_path=/aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.586 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.586 DEBUG:boto:Path: /aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.587 DEBUG:boto:Data: 2024-04-19T23:26:38.587 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.587 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.587 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.587 DEBUG:boto:Params: {} 2024-04-19T23:26:38.587 DEBUG:boto:Token: None 2024-04-19T23:26:38.587 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.588 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fZ3Kr6/0qub8zWi+NPVf8BflQsI= 2024-04-19T23:26:38.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fZ3Kr6/0qub8zWi+NPVf8BflQsI=', 'Content-Length': '0'} 2024-04-19T23:26:38.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001655c45b5845ed82-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-19T23:26:38.590 DEBUG:boto:path=/aktlji-16/obj17 2024-04-19T23:26:38.590 DEBUG:boto:auth_path=/aktlji-16/obj17 2024-04-19T23:26:38.590 DEBUG:boto:path=/aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.590 DEBUG:boto:auth_path=/aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.590 DEBUG:boto:Method: GET 2024-04-19T23:26:38.590 DEBUG:boto:Path: /aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.590 DEBUG:boto:Data: 2024-04-19T23:26:38.590 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.590 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.591 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.591 DEBUG:boto:Params: {} 2024-04-19T23:26:38.591 DEBUG:boto:Token: None 2024-04-19T23:26:38.591 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.591 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0= 2024-04-19T23:26:38.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0=', 'Content-Length': '0'} 2024-04-19T23:26:38.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba1ff91da9bb28c7-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.593 DEBUG:boto:path=/aktlji-16/obj17 2024-04-19T23:26:38.593 DEBUG:boto:auth_path=/aktlji-16/obj17 2024-04-19T23:26:38.593 DEBUG:boto:path=/aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.593 DEBUG:boto:auth_path=/aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.593 DEBUG:boto:Method: GET 2024-04-19T23:26:38.593 DEBUG:boto:Path: /aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.594 DEBUG:boto:Data: 2024-04-19T23:26:38.594 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.594 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.594 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.594 DEBUG:boto:Params: {} 2024-04-19T23:26:38.594 DEBUG:boto:Token: None 2024-04-19T23:26:38.594 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj17?versionId=null 2024-04-19T23:26:38.594 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0= 2024-04-19T23:26:38.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0=', 'Content-Length': '0'} 2024-04-19T23:26:38.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049944a88a702b706-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.598 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.598 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.598 DEBUG:boto:path=/aktlji-16/?prefix=obj17 2024-04-19T23:26:38.598 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj17 2024-04-19T23:26:38.598 DEBUG:boto:Method: GET 2024-04-19T23:26:38.598 DEBUG:boto:Path: /aktlji-16/?prefix=obj17 2024-04-19T23:26:38.598 DEBUG:boto:Data: 2024-04-19T23:26:38.599 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.599 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.599 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.599 DEBUG:boto:Params: {} 2024-04-19T23:26:38.599 DEBUG:boto:Token: None 2024-04-19T23:26:38.599 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.599 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efdd72c3a7c0ff2f-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.602 DEBUG:boto:b'aktlji-16obj171000falseobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.602 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.602 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.602 DEBUG:boto:path=/aktlji-16/?prefix=obj17 2024-04-19T23:26:38.603 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj17 2024-04-19T23:26:38.603 DEBUG:boto:Method: GET 2024-04-19T23:26:38.603 DEBUG:boto:Path: /aktlji-16/?prefix=obj17 2024-04-19T23:26:38.603 DEBUG:boto:Data: 2024-04-19T23:26:38.603 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.603 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.603 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.603 DEBUG:boto:Params: {} 2024-04-19T23:26:38.603 DEBUG:boto:Token: None 2024-04-19T23:26:38.603 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.604 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050e3a5815f807580-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.606 DEBUG:boto:b'aktlji-16obj171000falseobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.606 DEBUG:boto:path=/aktlji-16/obj17 2024-04-19T23:26:38.607 DEBUG:boto:auth_path=/aktlji-16/obj17 2024-04-19T23:26:38.607 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.607 DEBUG:boto:Path: /aktlji-16/obj17 2024-04-19T23:26:38.607 DEBUG:boto:Data: 2024-04-19T23:26:38.607 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.607 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.607 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.607 DEBUG:boto:Params: {} 2024-04-19T23:26:38.607 DEBUG:boto:Token: None 2024-04-19T23:26:38.608 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj17 2024-04-19T23:26:38.608 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tH8rwQxe7MoJgK/KeqoEMToY12o= 2024-04-19T23:26:38.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tH8rwQxe7MoJgK/KeqoEMToY12o=', 'Content-Length': '0'} 2024-04-19T23:26:38.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b74f1acef798760d-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.610 DEBUG:boto:path=/aktlji-16/obj17 2024-04-19T23:26:38.610 DEBUG:boto:auth_path=/aktlji-16/obj17 2024-04-19T23:26:38.610 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.611 DEBUG:boto:Path: /aktlji-16/obj17 2024-04-19T23:26:38.611 DEBUG:boto:Data: 2024-04-19T23:26:38.611 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.611 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.611 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.611 DEBUG:boto:Params: {} 2024-04-19T23:26:38.611 DEBUG:boto:Token: None 2024-04-19T23:26:38.611 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj17 2024-04-19T23:26:38.611 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tH8rwQxe7MoJgK/KeqoEMToY12o= 2024-04-19T23:26:38.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tH8rwQxe7MoJgK/KeqoEMToY12o=', 'Content-Length': '0'} 2024-04-19T23:26:38.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008697d02e976c4e8a-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-19T23:26:38.613 DEBUG:boto:path=/aktlji-16/obj17 2024-04-19T23:26:38.613 DEBUG:boto:auth_path=/aktlji-16/obj17 2024-04-19T23:26:38.614 DEBUG:boto:Method: GET 2024-04-19T23:26:38.614 DEBUG:boto:Path: /aktlji-16/obj17 2024-04-19T23:26:38.614 DEBUG:boto:Data: 2024-04-19T23:26:38.614 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.614 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.614 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.614 DEBUG:boto:Params: {} 2024-04-19T23:26:38.614 DEBUG:boto:Token: None 2024-04-19T23:26:38.614 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj17 2024-04-19T23:26:38.615 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:djgMuTyHIaNInOJnGn1xwEYOtoA= 2024-04-19T23:26:38.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:djgMuTyHIaNInOJnGn1xwEYOtoA=', 'Content-Length': '0'} 2024-04-19T23:26:38.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0a3cfbf22ac30b5-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.616 DEBUG:boto:path=/aktlji-16/obj17 2024-04-19T23:26:38.617 DEBUG:boto:auth_path=/aktlji-16/obj17 2024-04-19T23:26:38.617 DEBUG:boto:Method: GET 2024-04-19T23:26:38.617 DEBUG:boto:Path: /aktlji-16/obj17 2024-04-19T23:26:38.617 DEBUG:boto:Data: 2024-04-19T23:26:38.617 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.617 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.617 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.617 DEBUG:boto:Params: {} 2024-04-19T23:26:38.617 DEBUG:boto:Token: None 2024-04-19T23:26:38.618 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj17 2024-04-19T23:26:38.618 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:djgMuTyHIaNInOJnGn1xwEYOtoA= 2024-04-19T23:26:38.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:djgMuTyHIaNInOJnGn1xwEYOtoA=', 'Content-Length': '0'} 2024-04-19T23:26:38.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e51a784fc53b329d-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.620 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.620 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.620 DEBUG:boto:path=/aktlji-16/?prefix=obj17 2024-04-19T23:26:38.620 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj17 2024-04-19T23:26:38.620 DEBUG:boto:Method: GET 2024-04-19T23:26:38.620 DEBUG:boto:Path: /aktlji-16/?prefix=obj17 2024-04-19T23:26:38.620 DEBUG:boto:Data: 2024-04-19T23:26:38.620 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.620 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.620 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.621 DEBUG:boto:Params: {} 2024-04-19T23:26:38.621 DEBUG:boto:Token: None 2024-04-19T23:26:38.621 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.621 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093892da2d803fcfe-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.624 DEBUG:boto:b'aktlji-16obj171000falseobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.624 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.624 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.624 DEBUG:boto:path=/aktlji-16/?prefix=obj17 2024-04-19T23:26:38.624 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj17 2024-04-19T23:26:38.624 DEBUG:boto:Method: GET 2024-04-19T23:26:38.624 DEBUG:boto:Path: /aktlji-16/?prefix=obj17 2024-04-19T23:26:38.625 DEBUG:boto:Data: 2024-04-19T23:26:38.625 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.625 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.625 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.625 DEBUG:boto:Params: {} 2024-04-19T23:26:38.625 DEBUG:boto:Token: None 2024-04-19T23:26:38.625 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.625 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf479447eb948634-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.628 DEBUG:boto:b'aktlji-16obj171000falseobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj18 2024-04-19T23:26:38.628 DEBUG:boto:path=/aktlji-16/obj18 2024-04-19T23:26:38.628 DEBUG:boto:auth_path=/aktlji-16/obj18 2024-04-19T23:26:38.629 DEBUG:boto:path=/aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.629 DEBUG:boto:auth_path=/aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.629 DEBUG:boto:Method: GET 2024-04-19T23:26:38.629 DEBUG:boto:Path: /aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.629 DEBUG:boto:Data: 2024-04-19T23:26:38.629 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.629 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.629 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.629 DEBUG:boto:Params: {} 2024-04-19T23:26:38.629 DEBUG:boto:Token: None 2024-04-19T23:26:38.630 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.630 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw= 2024-04-19T23:26:38.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw=', 'Content-Length': '0'} 2024-04-19T23:26:38.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e31a4b289a132e4a-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.674 DEBUG:boto:path=/aktlji-16/obj18 2024-04-19T23:26:38.675 DEBUG:boto:auth_path=/aktlji-16/obj18 2024-04-19T23:26:38.675 DEBUG:boto:path=/aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.675 DEBUG:boto:auth_path=/aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.675 DEBUG:boto:Method: GET 2024-04-19T23:26:38.675 DEBUG:boto:Path: /aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.675 DEBUG:boto:Data: 2024-04-19T23:26:38.675 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.675 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.675 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.675 DEBUG:boto:Params: {} 2024-04-19T23:26:38.676 DEBUG:boto:Token: None 2024-04-19T23:26:38.676 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw= 2024-04-19T23:26:38.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw=', 'Content-Length': '0'} 2024-04-19T23:26:38.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ee49123c4587647-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.678 DEBUG:boto:path=/aktlji-16/obj18 2024-04-19T23:26:38.678 DEBUG:boto:auth_path=/aktlji-16/obj18 2024-04-19T23:26:38.679 DEBUG:boto:path=/aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.679 DEBUG:boto:auth_path=/aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.679 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.679 DEBUG:boto:Path: /aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.679 DEBUG:boto:Data: 2024-04-19T23:26:38.679 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.679 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.679 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.679 DEBUG:boto:Params: {} 2024-04-19T23:26:38.680 DEBUG:boto:Token: None 2024-04-19T23:26:38.680 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.680 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9BlM1/0Aww32tf801X3f1RzUsdA= 2024-04-19T23:26:38.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9BlM1/0Aww32tf801X3f1RzUsdA=', 'Content-Length': '0'} 2024-04-19T23:26:38.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e199b8525fc4f73e-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.682 DEBUG:boto:path=/aktlji-16/obj18 2024-04-19T23:26:38.682 DEBUG:boto:auth_path=/aktlji-16/obj18 2024-04-19T23:26:38.682 DEBUG:boto:path=/aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.682 DEBUG:boto:auth_path=/aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.683 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.683 DEBUG:boto:Path: /aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.683 DEBUG:boto:Data: 2024-04-19T23:26:38.683 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.683 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.683 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.683 DEBUG:boto:Params: {} 2024-04-19T23:26:38.683 DEBUG:boto:Token: None 2024-04-19T23:26:38.683 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.683 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9BlM1/0Aww32tf801X3f1RzUsdA= 2024-04-19T23:26:38.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9BlM1/0Aww32tf801X3f1RzUsdA=', 'Content-Length': '0'} 2024-04-19T23:26:38.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5c26d4d6825bd10-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj18 2024-04-19T23:26:38.686 DEBUG:boto:path=/aktlji-16/obj18 2024-04-19T23:26:38.686 DEBUG:boto:auth_path=/aktlji-16/obj18 2024-04-19T23:26:38.686 DEBUG:boto:path=/aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.686 DEBUG:boto:auth_path=/aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.686 DEBUG:boto:Method: GET 2024-04-19T23:26:38.686 DEBUG:boto:Path: /aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.686 DEBUG:boto:Data: 2024-04-19T23:26:38.686 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.687 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.687 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.687 DEBUG:boto:Params: {} 2024-04-19T23:26:38.687 DEBUG:boto:Token: None 2024-04-19T23:26:38.687 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.687 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw= 2024-04-19T23:26:38.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw=', 'Content-Length': '0'} 2024-04-19T23:26:38.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000316102b46a6ddb9f-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.689 DEBUG:boto:path=/aktlji-16/obj18 2024-04-19T23:26:38.690 DEBUG:boto:auth_path=/aktlji-16/obj18 2024-04-19T23:26:38.690 DEBUG:boto:path=/aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.690 DEBUG:boto:auth_path=/aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.690 DEBUG:boto:Method: GET 2024-04-19T23:26:38.690 DEBUG:boto:Path: /aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.690 DEBUG:boto:Data: 2024-04-19T23:26:38.690 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.690 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.690 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.690 DEBUG:boto:Params: {} 2024-04-19T23:26:38.690 DEBUG:boto:Token: None 2024-04-19T23:26:38.691 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj18?versionId=null 2024-04-19T23:26:38.691 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw= 2024-04-19T23:26:38.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw=', 'Content-Length': '0'} 2024-04-19T23:26:38.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026546691c4e2c52b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.693 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.693 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.693 DEBUG:boto:path=/aktlji-16/?prefix=obj18 2024-04-19T23:26:38.693 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj18 2024-04-19T23:26:38.693 DEBUG:boto:Method: GET 2024-04-19T23:26:38.693 DEBUG:boto:Path: /aktlji-16/?prefix=obj18 2024-04-19T23:26:38.694 DEBUG:boto:Data: 2024-04-19T23:26:38.694 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.694 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.694 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.694 DEBUG:boto:Params: {} 2024-04-19T23:26:38.694 DEBUG:boto:Token: None 2024-04-19T23:26:38.694 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.695 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003590c9ba931579a6-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.697 DEBUG:boto:b'aktlji-16obj181000falseobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.698 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.698 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.698 DEBUG:boto:path=/aktlji-16/?prefix=obj18 2024-04-19T23:26:38.698 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj18 2024-04-19T23:26:38.698 DEBUG:boto:Method: GET 2024-04-19T23:26:38.699 DEBUG:boto:Path: /aktlji-16/?prefix=obj18 2024-04-19T23:26:38.699 DEBUG:boto:Data: 2024-04-19T23:26:38.699 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.699 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.699 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.699 DEBUG:boto:Params: {} 2024-04-19T23:26:38.700 DEBUG:boto:Token: None 2024-04-19T23:26:38.700 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.700 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004de727c2ce2b9ddf-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.702 DEBUG:boto:b'aktlji-16obj181000falseobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.703 DEBUG:boto:path=/aktlji-16/obj18 2024-04-19T23:26:38.703 DEBUG:boto:auth_path=/aktlji-16/obj18 2024-04-19T23:26:38.703 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.704 DEBUG:boto:Path: /aktlji-16/obj18 2024-04-19T23:26:38.704 DEBUG:boto:Data: 2024-04-19T23:26:38.704 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.704 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.704 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.704 DEBUG:boto:Params: {} 2024-04-19T23:26:38.704 DEBUG:boto:Token: None 2024-04-19T23:26:38.704 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj18 2024-04-19T23:26:38.705 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zzu+mNwHdffBtjX354X2GzVp8Z0= 2024-04-19T23:26:38.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zzu+mNwHdffBtjX354X2GzVp8Z0=', 'Content-Length': '0'} 2024-04-19T23:26:38.706 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcc346167d5294d0-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.707 DEBUG:boto:path=/aktlji-16/obj18 2024-04-19T23:26:38.707 DEBUG:boto:auth_path=/aktlji-16/obj18 2024-04-19T23:26:38.707 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.707 DEBUG:boto:Path: /aktlji-16/obj18 2024-04-19T23:26:38.707 DEBUG:boto:Data: 2024-04-19T23:26:38.707 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.707 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.707 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.707 DEBUG:boto:Params: {} 2024-04-19T23:26:38.708 DEBUG:boto:Token: None 2024-04-19T23:26:38.708 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj18 2024-04-19T23:26:38.708 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zzu+mNwHdffBtjX354X2GzVp8Z0= 2024-04-19T23:26:38.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zzu+mNwHdffBtjX354X2GzVp8Z0=', 'Content-Length': '0'} 2024-04-19T23:26:38.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e1c9d4158c06728-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj18 2024-04-19T23:26:38.710 DEBUG:boto:path=/aktlji-16/obj18 2024-04-19T23:26:38.711 DEBUG:boto:auth_path=/aktlji-16/obj18 2024-04-19T23:26:38.711 DEBUG:boto:Method: GET 2024-04-19T23:26:38.711 DEBUG:boto:Path: /aktlji-16/obj18 2024-04-19T23:26:38.711 DEBUG:boto:Data: 2024-04-19T23:26:38.711 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.711 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.711 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.711 DEBUG:boto:Params: {} 2024-04-19T23:26:38.711 DEBUG:boto:Token: None 2024-04-19T23:26:38.711 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj18 2024-04-19T23:26:38.712 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YpviMq0WuPNIyMfFeMd6+v9symU= 2024-04-19T23:26:38.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpviMq0WuPNIyMfFeMd6+v9symU=', 'Content-Length': '0'} 2024-04-19T23:26:38.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074edd147fed816fe-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.714 DEBUG:boto:path=/aktlji-16/obj18 2024-04-19T23:26:38.714 DEBUG:boto:auth_path=/aktlji-16/obj18 2024-04-19T23:26:38.714 DEBUG:boto:Method: GET 2024-04-19T23:26:38.714 DEBUG:boto:Path: /aktlji-16/obj18 2024-04-19T23:26:38.714 DEBUG:boto:Data: 2024-04-19T23:26:38.714 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.714 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.715 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.715 DEBUG:boto:Params: {} 2024-04-19T23:26:38.715 DEBUG:boto:Token: None 2024-04-19T23:26:38.715 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj18 2024-04-19T23:26:38.715 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YpviMq0WuPNIyMfFeMd6+v9symU= 2024-04-19T23:26:38.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpviMq0WuPNIyMfFeMd6+v9symU=', 'Content-Length': '0'} 2024-04-19T23:26:38.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe917ed409f93e2f-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.717 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.717 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.717 DEBUG:boto:path=/aktlji-16/?prefix=obj18 2024-04-19T23:26:38.718 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj18 2024-04-19T23:26:38.718 DEBUG:boto:Method: GET 2024-04-19T23:26:38.718 DEBUG:boto:Path: /aktlji-16/?prefix=obj18 2024-04-19T23:26:38.718 DEBUG:boto:Data: 2024-04-19T23:26:38.718 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.718 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.718 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.718 DEBUG:boto:Params: {} 2024-04-19T23:26:38.719 DEBUG:boto:Token: None 2024-04-19T23:26:38.719 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.719 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d41d56e9d3f847b4-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.722 DEBUG:boto:b'aktlji-16obj181000falseobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.722 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.722 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.722 DEBUG:boto:path=/aktlji-16/?prefix=obj18 2024-04-19T23:26:38.723 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj18 2024-04-19T23:26:38.723 DEBUG:boto:Method: GET 2024-04-19T23:26:38.723 DEBUG:boto:Path: /aktlji-16/?prefix=obj18 2024-04-19T23:26:38.723 DEBUG:boto:Data: 2024-04-19T23:26:38.723 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.723 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.723 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.723 DEBUG:boto:Params: {} 2024-04-19T23:26:38.723 DEBUG:boto:Token: None 2024-04-19T23:26:38.723 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.724 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9822f573ddf47dc-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.726 DEBUG:boto:b'aktlji-16obj181000falseobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-19T23:26:38.727 DEBUG:boto:path=/aktlji-16/obj19 2024-04-19T23:26:38.727 DEBUG:boto:auth_path=/aktlji-16/obj19 2024-04-19T23:26:38.727 DEBUG:boto:path=/aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.727 DEBUG:boto:auth_path=/aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.727 DEBUG:boto:Method: GET 2024-04-19T23:26:38.728 DEBUG:boto:Path: /aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.728 DEBUG:boto:Data: 2024-04-19T23:26:38.728 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.728 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.728 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.728 DEBUG:boto:Params: {} 2024-04-19T23:26:38.728 DEBUG:boto:Token: None 2024-04-19T23:26:38.728 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.728 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw= 2024-04-19T23:26:38.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw=', 'Content-Length': '0'} 2024-04-19T23:26:38.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d85c4a19c49c27ef-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.731 DEBUG:boto:path=/aktlji-16/obj19 2024-04-19T23:26:38.731 DEBUG:boto:auth_path=/aktlji-16/obj19 2024-04-19T23:26:38.731 DEBUG:boto:path=/aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.731 DEBUG:boto:auth_path=/aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.731 DEBUG:boto:Method: GET 2024-04-19T23:26:38.731 DEBUG:boto:Path: /aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.731 DEBUG:boto:Data: 2024-04-19T23:26:38.731 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.731 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.731 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.731 DEBUG:boto:Params: {} 2024-04-19T23:26:38.732 DEBUG:boto:Token: None 2024-04-19T23:26:38.732 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.732 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw= 2024-04-19T23:26:38.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw=', 'Content-Length': '0'} 2024-04-19T23:26:38.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f87121a9b4f18a1d-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.735 DEBUG:boto:path=/aktlji-16/obj19 2024-04-19T23:26:38.735 DEBUG:boto:auth_path=/aktlji-16/obj19 2024-04-19T23:26:38.735 DEBUG:boto:path=/aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.735 DEBUG:boto:auth_path=/aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.735 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.735 DEBUG:boto:Path: /aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.736 DEBUG:boto:Data: 2024-04-19T23:26:38.736 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.736 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.736 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.736 DEBUG:boto:Params: {} 2024-04-19T23:26:38.736 DEBUG:boto:Token: None 2024-04-19T23:26:38.736 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.737 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HazYyvj85FjqxT5y0Aid8Xb3cAY= 2024-04-19T23:26:38.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HazYyvj85FjqxT5y0Aid8Xb3cAY=', 'Content-Length': '0'} 2024-04-19T23:26:38.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d7e60da5218809f-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.739 DEBUG:boto:path=/aktlji-16/obj19 2024-04-19T23:26:38.739 DEBUG:boto:auth_path=/aktlji-16/obj19 2024-04-19T23:26:38.739 DEBUG:boto:path=/aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.739 DEBUG:boto:auth_path=/aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.739 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.739 DEBUG:boto:Path: /aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.739 DEBUG:boto:Data: 2024-04-19T23:26:38.739 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.739 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.740 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.740 DEBUG:boto:Params: {} 2024-04-19T23:26:38.740 DEBUG:boto:Token: None 2024-04-19T23:26:38.740 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.740 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HazYyvj85FjqxT5y0Aid8Xb3cAY= 2024-04-19T23:26:38.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HazYyvj85FjqxT5y0Aid8Xb3cAY=', 'Content-Length': '0'} 2024-04-19T23:26:38.742 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd1460779545206b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-19T23:26:38.742 DEBUG:boto:path=/aktlji-16/obj19 2024-04-19T23:26:38.742 DEBUG:boto:auth_path=/aktlji-16/obj19 2024-04-19T23:26:38.742 DEBUG:boto:path=/aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.743 DEBUG:boto:auth_path=/aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.743 DEBUG:boto:Method: GET 2024-04-19T23:26:38.743 DEBUG:boto:Path: /aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.743 DEBUG:boto:Data: 2024-04-19T23:26:38.743 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.743 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.743 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.743 DEBUG:boto:Params: {} 2024-04-19T23:26:38.743 DEBUG:boto:Token: None 2024-04-19T23:26:38.744 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.744 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw= 2024-04-19T23:26:38.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw=', 'Content-Length': '0'} 2024-04-19T23:26:38.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0355cb4bde0f233-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.786 DEBUG:boto:path=/aktlji-16/obj19 2024-04-19T23:26:38.787 DEBUG:boto:auth_path=/aktlji-16/obj19 2024-04-19T23:26:38.787 DEBUG:boto:path=/aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.787 DEBUG:boto:auth_path=/aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.787 DEBUG:boto:Method: GET 2024-04-19T23:26:38.787 DEBUG:boto:Path: /aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.787 DEBUG:boto:Data: 2024-04-19T23:26:38.787 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.787 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.787 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.787 DEBUG:boto:Params: {} 2024-04-19T23:26:38.787 DEBUG:boto:Token: None 2024-04-19T23:26:38.788 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj19?versionId=null 2024-04-19T23:26:38.788 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw= 2024-04-19T23:26:38.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw=', 'Content-Length': '0'} 2024-04-19T23:26:38.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000671423ab9a14c40c-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.790 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.790 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.790 DEBUG:boto:path=/aktlji-16/?prefix=obj19 2024-04-19T23:26:38.790 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj19 2024-04-19T23:26:38.790 DEBUG:boto:Method: GET 2024-04-19T23:26:38.791 DEBUG:boto:Path: /aktlji-16/?prefix=obj19 2024-04-19T23:26:38.791 DEBUG:boto:Data: 2024-04-19T23:26:38.791 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.791 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.791 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.791 DEBUG:boto:Params: {} 2024-04-19T23:26:38.791 DEBUG:boto:Token: None 2024-04-19T23:26:38.791 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.791 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b7cd68f53e2d86b-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.794 DEBUG:boto:b'aktlji-16obj191000falseobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.795 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.795 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.795 DEBUG:boto:path=/aktlji-16/?prefix=obj19 2024-04-19T23:26:38.795 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj19 2024-04-19T23:26:38.795 DEBUG:boto:Method: GET 2024-04-19T23:26:38.795 DEBUG:boto:Path: /aktlji-16/?prefix=obj19 2024-04-19T23:26:38.795 DEBUG:boto:Data: 2024-04-19T23:26:38.795 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.795 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.795 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.795 DEBUG:boto:Params: {} 2024-04-19T23:26:38.796 DEBUG:boto:Token: None 2024-04-19T23:26:38.796 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.796 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9d17abfc860353b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.802 DEBUG:boto:b'aktlji-16obj191000falseobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.802 DEBUG:boto:path=/aktlji-16/obj19 2024-04-19T23:26:38.802 DEBUG:boto:auth_path=/aktlji-16/obj19 2024-04-19T23:26:38.802 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.802 DEBUG:boto:Path: /aktlji-16/obj19 2024-04-19T23:26:38.802 DEBUG:boto:Data: 2024-04-19T23:26:38.803 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.803 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.803 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.803 DEBUG:boto:Params: {} 2024-04-19T23:26:38.803 DEBUG:boto:Token: None 2024-04-19T23:26:38.803 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj19 2024-04-19T23:26:38.803 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EQ4hjtqKLA1Z+IMZoj/GBja9M9s= 2024-04-19T23:26:38.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EQ4hjtqKLA1Z+IMZoj/GBja9M9s=', 'Content-Length': '0'} 2024-04-19T23:26:38.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000095dc486b0abeda2-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.805 DEBUG:boto:path=/aktlji-16/obj19 2024-04-19T23:26:38.806 DEBUG:boto:auth_path=/aktlji-16/obj19 2024-04-19T23:26:38.806 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.806 DEBUG:boto:Path: /aktlji-16/obj19 2024-04-19T23:26:38.806 DEBUG:boto:Data: 2024-04-19T23:26:38.806 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.806 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.806 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.806 DEBUG:boto:Params: {} 2024-04-19T23:26:38.806 DEBUG:boto:Token: None 2024-04-19T23:26:38.806 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj19 2024-04-19T23:26:38.807 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EQ4hjtqKLA1Z+IMZoj/GBja9M9s= 2024-04-19T23:26:38.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EQ4hjtqKLA1Z+IMZoj/GBja9M9s=', 'Content-Length': '0'} 2024-04-19T23:26:38.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c40885d10715d49-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-19T23:26:38.809 DEBUG:boto:path=/aktlji-16/obj19 2024-04-19T23:26:38.809 DEBUG:boto:auth_path=/aktlji-16/obj19 2024-04-19T23:26:38.810 DEBUG:boto:Method: GET 2024-04-19T23:26:38.810 DEBUG:boto:Path: /aktlji-16/obj19 2024-04-19T23:26:38.810 DEBUG:boto:Data: 2024-04-19T23:26:38.810 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.810 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.810 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.811 DEBUG:boto:Params: {} 2024-04-19T23:26:38.811 DEBUG:boto:Token: None 2024-04-19T23:26:38.811 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj19 2024-04-19T23:26:38.811 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Uf+9+w68gqXM94LysRb10TSG5DI= 2024-04-19T23:26:38.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Uf+9+w68gqXM94LysRb10TSG5DI=', 'Content-Length': '0'} 2024-04-19T23:26:38.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034ec301da8e5ba9b-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.814 DEBUG:boto:path=/aktlji-16/obj19 2024-04-19T23:26:38.814 DEBUG:boto:auth_path=/aktlji-16/obj19 2024-04-19T23:26:38.814 DEBUG:boto:Method: GET 2024-04-19T23:26:38.815 DEBUG:boto:Path: /aktlji-16/obj19 2024-04-19T23:26:38.815 DEBUG:boto:Data: 2024-04-19T23:26:38.815 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.815 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.815 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.816 DEBUG:boto:Params: {} 2024-04-19T23:26:38.816 DEBUG:boto:Token: None 2024-04-19T23:26:38.816 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj19 2024-04-19T23:26:38.816 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Uf+9+w68gqXM94LysRb10TSG5DI= 2024-04-19T23:26:38.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Uf+9+w68gqXM94LysRb10TSG5DI=', 'Content-Length': '0'} 2024-04-19T23:26:38.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000399949c5971021f7-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.818 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.818 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.818 DEBUG:boto:path=/aktlji-16/?prefix=obj19 2024-04-19T23:26:38.819 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj19 2024-04-19T23:26:38.819 DEBUG:boto:Method: GET 2024-04-19T23:26:38.819 DEBUG:boto:Path: /aktlji-16/?prefix=obj19 2024-04-19T23:26:38.819 DEBUG:boto:Data: 2024-04-19T23:26:38.819 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.819 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.819 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.819 DEBUG:boto:Params: {} 2024-04-19T23:26:38.819 DEBUG:boto:Token: None 2024-04-19T23:26:38.819 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.820 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000741c2bd3fa94ab9-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.822 DEBUG:boto:b'aktlji-16obj191000falseobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.823 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.823 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.823 DEBUG:boto:path=/aktlji-16/?prefix=obj19 2024-04-19T23:26:38.823 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj19 2024-04-19T23:26:38.823 DEBUG:boto:Method: GET 2024-04-19T23:26:38.823 DEBUG:boto:Path: /aktlji-16/?prefix=obj19 2024-04-19T23:26:38.823 DEBUG:boto:Data: 2024-04-19T23:26:38.823 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.823 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.823 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.824 DEBUG:boto:Params: {} 2024-04-19T23:26:38.824 DEBUG:boto:Token: None 2024-04-19T23:26:38.824 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b61666fe1939c2b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.827 DEBUG:boto:b'aktlji-16obj191000falseobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-19T23:26:38.827 DEBUG:boto:path=/aktlji-16/obj2 2024-04-19T23:26:38.828 DEBUG:boto:auth_path=/aktlji-16/obj2 2024-04-19T23:26:38.828 DEBUG:boto:path=/aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.828 DEBUG:boto:auth_path=/aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.828 DEBUG:boto:Method: GET 2024-04-19T23:26:38.829 DEBUG:boto:Path: /aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.829 DEBUG:boto:Data: 2024-04-19T23:26:38.829 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.829 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.830 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.830 DEBUG:boto:Params: {} 2024-04-19T23:26:38.830 DEBUG:boto:Token: None 2024-04-19T23:26:38.830 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.831 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA= 2024-04-19T23:26:38.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA=', 'Content-Length': '0'} 2024-04-19T23:26:38.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9d745ee4105bc32-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.833 DEBUG:boto:path=/aktlji-16/obj2 2024-04-19T23:26:38.833 DEBUG:boto:auth_path=/aktlji-16/obj2 2024-04-19T23:26:38.833 DEBUG:boto:path=/aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.833 DEBUG:boto:auth_path=/aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.834 DEBUG:boto:Method: GET 2024-04-19T23:26:38.834 DEBUG:boto:Path: /aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.834 DEBUG:boto:Data: 2024-04-19T23:26:38.834 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.834 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.834 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.834 DEBUG:boto:Params: {} 2024-04-19T23:26:38.834 DEBUG:boto:Token: None 2024-04-19T23:26:38.834 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.835 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA= 2024-04-19T23:26:38.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA=', 'Content-Length': '0'} 2024-04-19T23:26:38.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f83002a0cf3cf640-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.837 DEBUG:boto:path=/aktlji-16/obj2 2024-04-19T23:26:38.837 DEBUG:boto:auth_path=/aktlji-16/obj2 2024-04-19T23:26:38.837 DEBUG:boto:path=/aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.837 DEBUG:boto:auth_path=/aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.837 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.837 DEBUG:boto:Path: /aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.837 DEBUG:boto:Data: 2024-04-19T23:26:38.837 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.837 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.838 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.838 DEBUG:boto:Params: {} 2024-04-19T23:26:38.838 DEBUG:boto:Token: None 2024-04-19T23:26:38.838 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.838 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qOh0DBEhQvgnQZDOPdjtHRCzYlg= 2024-04-19T23:26:38.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qOh0DBEhQvgnQZDOPdjtHRCzYlg=', 'Content-Length': '0'} 2024-04-19T23:26:38.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae82722e32254b7c-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.840 DEBUG:boto:path=/aktlji-16/obj2 2024-04-19T23:26:38.840 DEBUG:boto:auth_path=/aktlji-16/obj2 2024-04-19T23:26:38.840 DEBUG:boto:path=/aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.840 DEBUG:boto:auth_path=/aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.841 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.841 DEBUG:boto:Path: /aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.841 DEBUG:boto:Data: 2024-04-19T23:26:38.841 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.841 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.842 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.842 DEBUG:boto:Params: {} 2024-04-19T23:26:38.842 DEBUG:boto:Token: None 2024-04-19T23:26:38.842 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.842 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qOh0DBEhQvgnQZDOPdjtHRCzYlg= 2024-04-19T23:26:38.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qOh0DBEhQvgnQZDOPdjtHRCzYlg=', 'Content-Length': '0'} 2024-04-19T23:26:38.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fcf88b244f33a27-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-19T23:26:38.845 DEBUG:boto:path=/aktlji-16/obj2 2024-04-19T23:26:38.845 DEBUG:boto:auth_path=/aktlji-16/obj2 2024-04-19T23:26:38.845 DEBUG:boto:path=/aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.845 DEBUG:boto:auth_path=/aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.845 DEBUG:boto:Method: GET 2024-04-19T23:26:38.845 DEBUG:boto:Path: /aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.845 DEBUG:boto:Data: 2024-04-19T23:26:38.845 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.845 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.845 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.845 DEBUG:boto:Params: {} 2024-04-19T23:26:38.846 DEBUG:boto:Token: None 2024-04-19T23:26:38.846 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.846 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA= 2024-04-19T23:26:38.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA=', 'Content-Length': '0'} 2024-04-19T23:26:38.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012d4eab4e5a6cc49-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.848 DEBUG:boto:path=/aktlji-16/obj2 2024-04-19T23:26:38.848 DEBUG:boto:auth_path=/aktlji-16/obj2 2024-04-19T23:26:38.849 DEBUG:boto:path=/aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.849 DEBUG:boto:auth_path=/aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.849 DEBUG:boto:Method: GET 2024-04-19T23:26:38.849 DEBUG:boto:Path: /aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.849 DEBUG:boto:Data: 2024-04-19T23:26:38.849 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.849 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.849 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.849 DEBUG:boto:Params: {} 2024-04-19T23:26:38.849 DEBUG:boto:Token: None 2024-04-19T23:26:38.850 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj2?versionId=null 2024-04-19T23:26:38.850 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA= 2024-04-19T23:26:38.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA=', 'Content-Length': '0'} 2024-04-19T23:26:38.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d2b17e8fbb183eb-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.852 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.852 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.852 DEBUG:boto:path=/aktlji-16/?prefix=obj2 2024-04-19T23:26:38.852 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj2 2024-04-19T23:26:38.852 DEBUG:boto:Method: GET 2024-04-19T23:26:38.852 DEBUG:boto:Path: /aktlji-16/?prefix=obj2 2024-04-19T23:26:38.853 DEBUG:boto:Data: 2024-04-19T23:26:38.853 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.853 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.853 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.853 DEBUG:boto:Params: {} 2024-04-19T23:26:38.853 DEBUG:boto:Token: None 2024-04-19T23:26:38.853 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.853 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000153be6504a2c1c33-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.856 DEBUG:boto:b'aktlji-16obj21000falseobj22024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.857 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.857 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.857 DEBUG:boto:path=/aktlji-16/?prefix=obj2 2024-04-19T23:26:38.857 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj2 2024-04-19T23:26:38.857 DEBUG:boto:Method: GET 2024-04-19T23:26:38.858 DEBUG:boto:Path: /aktlji-16/?prefix=obj2 2024-04-19T23:26:38.858 DEBUG:boto:Data: 2024-04-19T23:26:38.858 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.858 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.858 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.858 DEBUG:boto:Params: {} 2024-04-19T23:26:38.858 DEBUG:boto:Token: None 2024-04-19T23:26:38.858 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.858 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c00f627763d8b0e3-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.861 DEBUG:boto:b'aktlji-16obj21000falseobj22024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.862 DEBUG:boto:path=/aktlji-16/obj2 2024-04-19T23:26:38.862 DEBUG:boto:auth_path=/aktlji-16/obj2 2024-04-19T23:26:38.862 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.862 DEBUG:boto:Path: /aktlji-16/obj2 2024-04-19T23:26:38.862 DEBUG:boto:Data: 2024-04-19T23:26:38.862 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.862 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.862 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.862 DEBUG:boto:Params: {} 2024-04-19T23:26:38.863 DEBUG:boto:Token: None 2024-04-19T23:26:38.863 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj2 2024-04-19T23:26:38.863 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Mc2uk9IARTNO4v2AsoZfKq3hjzs= 2024-04-19T23:26:38.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Mc2uk9IARTNO4v2AsoZfKq3hjzs=', 'Content-Length': '0'} 2024-04-19T23:26:38.864 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d9be398748785a9-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.865 DEBUG:boto:path=/aktlji-16/obj2 2024-04-19T23:26:38.865 DEBUG:boto:auth_path=/aktlji-16/obj2 2024-04-19T23:26:38.865 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.865 DEBUG:boto:Path: /aktlji-16/obj2 2024-04-19T23:26:38.865 DEBUG:boto:Data: 2024-04-19T23:26:38.865 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.865 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.865 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.866 DEBUG:boto:Params: {} 2024-04-19T23:26:38.866 DEBUG:boto:Token: None 2024-04-19T23:26:38.866 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj2 2024-04-19T23:26:38.866 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Mc2uk9IARTNO4v2AsoZfKq3hjzs= 2024-04-19T23:26:38.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Mc2uk9IARTNO4v2AsoZfKq3hjzs=', 'Content-Length': '0'} 2024-04-19T23:26:38.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000653b0b2a79cb33c-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-19T23:26:38.868 DEBUG:boto:path=/aktlji-16/obj2 2024-04-19T23:26:38.868 DEBUG:boto:auth_path=/aktlji-16/obj2 2024-04-19T23:26:38.868 DEBUG:boto:Method: GET 2024-04-19T23:26:38.868 DEBUG:boto:Path: /aktlji-16/obj2 2024-04-19T23:26:38.868 DEBUG:boto:Data: 2024-04-19T23:26:38.869 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.869 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.869 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.869 DEBUG:boto:Params: {} 2024-04-19T23:26:38.869 DEBUG:boto:Token: None 2024-04-19T23:26:38.869 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj2 2024-04-19T23:26:38.869 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rmD/OVfx768/greWwHE8tVdkrC4= 2024-04-19T23:26:38.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rmD/OVfx768/greWwHE8tVdkrC4=', 'Content-Length': '0'} 2024-04-19T23:26:38.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afb53c2e0831b0b7-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.871 DEBUG:boto:path=/aktlji-16/obj2 2024-04-19T23:26:38.872 DEBUG:boto:auth_path=/aktlji-16/obj2 2024-04-19T23:26:38.872 DEBUG:boto:Method: GET 2024-04-19T23:26:38.872 DEBUG:boto:Path: /aktlji-16/obj2 2024-04-19T23:26:38.872 DEBUG:boto:Data: 2024-04-19T23:26:38.872 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.872 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.872 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.872 DEBUG:boto:Params: {} 2024-04-19T23:26:38.872 DEBUG:boto:Token: None 2024-04-19T23:26:38.873 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj2 2024-04-19T23:26:38.873 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rmD/OVfx768/greWwHE8tVdkrC4= 2024-04-19T23:26:38.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rmD/OVfx768/greWwHE8tVdkrC4=', 'Content-Length': '0'} 2024-04-19T23:26:38.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbe84b29afa00cab-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.875 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.875 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.875 DEBUG:boto:path=/aktlji-16/?prefix=obj2 2024-04-19T23:26:38.875 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj2 2024-04-19T23:26:38.875 DEBUG:boto:Method: GET 2024-04-19T23:26:38.875 DEBUG:boto:Path: /aktlji-16/?prefix=obj2 2024-04-19T23:26:38.875 DEBUG:boto:Data: 2024-04-19T23:26:38.875 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.875 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.875 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.876 DEBUG:boto:Params: {} 2024-04-19T23:26:38.876 DEBUG:boto:Token: None 2024-04-19T23:26:38.876 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.876 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d0616ff01878221-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.879 DEBUG:boto:b'aktlji-16obj21000falseobj22024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.880 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.880 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.880 DEBUG:boto:path=/aktlji-16/?prefix=obj2 2024-04-19T23:26:38.880 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj2 2024-04-19T23:26:38.880 DEBUG:boto:Method: GET 2024-04-19T23:26:38.880 DEBUG:boto:Path: /aktlji-16/?prefix=obj2 2024-04-19T23:26:38.880 DEBUG:boto:Data: 2024-04-19T23:26:38.881 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.881 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.881 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.881 DEBUG:boto:Params: {} 2024-04-19T23:26:38.881 DEBUG:boto:Token: None 2024-04-19T23:26:38.881 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.881 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a01c1504a6f23a1-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.885 DEBUG:boto:b'aktlji-16obj21000falseobj22024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj20 2024-04-19T23:26:38.885 DEBUG:boto:path=/aktlji-16/obj20 2024-04-19T23:26:38.886 DEBUG:boto:auth_path=/aktlji-16/obj20 2024-04-19T23:26:38.886 DEBUG:boto:path=/aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.886 DEBUG:boto:auth_path=/aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.886 DEBUG:boto:Method: GET 2024-04-19T23:26:38.886 DEBUG:boto:Path: /aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.886 DEBUG:boto:Data: 2024-04-19T23:26:38.887 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.887 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.887 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.887 DEBUG:boto:Params: {} 2024-04-19T23:26:38.887 DEBUG:boto:Token: None 2024-04-19T23:26:38.887 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.887 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14= 2024-04-19T23:26:38.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14=', 'Content-Length': '0'} 2024-04-19T23:26:38.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003694025d03ff0129-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.934 DEBUG:boto:path=/aktlji-16/obj20 2024-04-19T23:26:38.934 DEBUG:boto:auth_path=/aktlji-16/obj20 2024-04-19T23:26:38.935 DEBUG:boto:path=/aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.935 DEBUG:boto:auth_path=/aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.935 DEBUG:boto:Method: GET 2024-04-19T23:26:38.935 DEBUG:boto:Path: /aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.935 DEBUG:boto:Data: 2024-04-19T23:26:38.935 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.935 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.935 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.935 DEBUG:boto:Params: {} 2024-04-19T23:26:38.935 DEBUG:boto:Token: None 2024-04-19T23:26:38.936 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.936 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14= 2024-04-19T23:26:38.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14=', 'Content-Length': '0'} 2024-04-19T23:26:38.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072f9f38b23e2455b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.938 DEBUG:boto:path=/aktlji-16/obj20 2024-04-19T23:26:38.938 DEBUG:boto:auth_path=/aktlji-16/obj20 2024-04-19T23:26:38.938 DEBUG:boto:path=/aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.938 DEBUG:boto:auth_path=/aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.938 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.938 DEBUG:boto:Path: /aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.939 DEBUG:boto:Data: 2024-04-19T23:26:38.939 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.939 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.939 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.939 DEBUG:boto:Params: {} 2024-04-19T23:26:38.939 DEBUG:boto:Token: None 2024-04-19T23:26:38.939 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.939 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1DSkS2pdAHOnzIPj1WN4SsCr/es= 2024-04-19T23:26:38.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1DSkS2pdAHOnzIPj1WN4SsCr/es=', 'Content-Length': '0'} 2024-04-19T23:26:38.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006265432a42a5954f-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.941 DEBUG:boto:path=/aktlji-16/obj20 2024-04-19T23:26:38.941 DEBUG:boto:auth_path=/aktlji-16/obj20 2024-04-19T23:26:38.941 DEBUG:boto:path=/aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.941 DEBUG:boto:auth_path=/aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.942 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.942 DEBUG:boto:Path: /aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.942 DEBUG:boto:Data: 2024-04-19T23:26:38.942 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.942 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.942 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.942 DEBUG:boto:Params: {} 2024-04-19T23:26:38.942 DEBUG:boto:Token: None 2024-04-19T23:26:38.942 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.942 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1DSkS2pdAHOnzIPj1WN4SsCr/es= 2024-04-19T23:26:38.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1DSkS2pdAHOnzIPj1WN4SsCr/es=', 'Content-Length': '0'} 2024-04-19T23:26:38.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c14039f03a39e616-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj20 2024-04-19T23:26:38.945 DEBUG:boto:path=/aktlji-16/obj20 2024-04-19T23:26:38.945 DEBUG:boto:auth_path=/aktlji-16/obj20 2024-04-19T23:26:38.945 DEBUG:boto:path=/aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.945 DEBUG:boto:auth_path=/aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.945 DEBUG:boto:Method: GET 2024-04-19T23:26:38.945 DEBUG:boto:Path: /aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.945 DEBUG:boto:Data: 2024-04-19T23:26:38.945 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.945 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.946 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.946 DEBUG:boto:Params: {} 2024-04-19T23:26:38.946 DEBUG:boto:Token: None 2024-04-19T23:26:38.946 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.946 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14= 2024-04-19T23:26:38.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14=', 'Content-Length': '0'} 2024-04-19T23:26:38.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aee2efcd2e51f8d5-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.948 DEBUG:boto:path=/aktlji-16/obj20 2024-04-19T23:26:38.948 DEBUG:boto:auth_path=/aktlji-16/obj20 2024-04-19T23:26:38.949 DEBUG:boto:path=/aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.949 DEBUG:boto:auth_path=/aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.949 DEBUG:boto:Method: GET 2024-04-19T23:26:38.949 DEBUG:boto:Path: /aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.949 DEBUG:boto:Data: 2024-04-19T23:26:38.949 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.949 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.949 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.949 DEBUG:boto:Params: {} 2024-04-19T23:26:38.949 DEBUG:boto:Token: None 2024-04-19T23:26:38.949 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj20?versionId=null 2024-04-19T23:26:38.950 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14= 2024-04-19T23:26:38.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14=', 'Content-Length': '0'} 2024-04-19T23:26:38.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074d8d459bb59301d-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.952 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.952 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.952 DEBUG:boto:path=/aktlji-16/?prefix=obj20 2024-04-19T23:26:38.953 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj20 2024-04-19T23:26:38.953 DEBUG:boto:Method: GET 2024-04-19T23:26:38.953 DEBUG:boto:Path: /aktlji-16/?prefix=obj20 2024-04-19T23:26:38.953 DEBUG:boto:Data: 2024-04-19T23:26:38.953 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.953 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.953 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.953 DEBUG:boto:Params: {} 2024-04-19T23:26:38.954 DEBUG:boto:Token: None 2024-04-19T23:26:38.954 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.954 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5008239b4cc6bae-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.957 DEBUG:boto:b'aktlji-16obj201000falseobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.958 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.958 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.958 DEBUG:boto:path=/aktlji-16/?prefix=obj20 2024-04-19T23:26:38.958 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj20 2024-04-19T23:26:38.958 DEBUG:boto:Method: GET 2024-04-19T23:26:38.959 DEBUG:boto:Path: /aktlji-16/?prefix=obj20 2024-04-19T23:26:38.959 DEBUG:boto:Data: 2024-04-19T23:26:38.959 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.959 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.959 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.959 DEBUG:boto:Params: {} 2024-04-19T23:26:38.959 DEBUG:boto:Token: None 2024-04-19T23:26:38.960 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.960 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090d14a137569c439-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.963 DEBUG:boto:b'aktlji-16obj201000falseobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.963 DEBUG:boto:path=/aktlji-16/obj20 2024-04-19T23:26:38.963 DEBUG:boto:auth_path=/aktlji-16/obj20 2024-04-19T23:26:38.964 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.965 DEBUG:boto:Path: /aktlji-16/obj20 2024-04-19T23:26:38.965 DEBUG:boto:Data: 2024-04-19T23:26:38.965 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.965 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.966 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.966 DEBUG:boto:Params: {} 2024-04-19T23:26:38.966 DEBUG:boto:Token: None 2024-04-19T23:26:38.966 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj20 2024-04-19T23:26:38.967 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sVASCfz51ZyQzp/VefvI77+J/kw= 2024-04-19T23:26:38.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sVASCfz51ZyQzp/VefvI77+J/kw=', 'Content-Length': '0'} 2024-04-19T23:26:38.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f78f96fe09ec915-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.969 DEBUG:boto:path=/aktlji-16/obj20 2024-04-19T23:26:38.969 DEBUG:boto:auth_path=/aktlji-16/obj20 2024-04-19T23:26:38.969 DEBUG:boto:Method: HEAD 2024-04-19T23:26:38.969 DEBUG:boto:Path: /aktlji-16/obj20 2024-04-19T23:26:38.969 DEBUG:boto:Data: 2024-04-19T23:26:38.969 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.970 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.970 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.970 DEBUG:boto:Params: {} 2024-04-19T23:26:38.970 DEBUG:boto:Token: None 2024-04-19T23:26:38.970 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj20 2024-04-19T23:26:38.970 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sVASCfz51ZyQzp/VefvI77+J/kw= 2024-04-19T23:26:38.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sVASCfz51ZyQzp/VefvI77+J/kw=', 'Content-Length': '0'} 2024-04-19T23:26:38.972 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d8411e69d57c84b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj20 2024-04-19T23:26:38.972 DEBUG:boto:path=/aktlji-16/obj20 2024-04-19T23:26:38.972 DEBUG:boto:auth_path=/aktlji-16/obj20 2024-04-19T23:26:38.972 DEBUG:boto:Method: GET 2024-04-19T23:26:38.973 DEBUG:boto:Path: /aktlji-16/obj20 2024-04-19T23:26:38.973 DEBUG:boto:Data: 2024-04-19T23:26:38.973 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.973 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.973 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.973 DEBUG:boto:Params: {} 2024-04-19T23:26:38.973 DEBUG:boto:Token: None 2024-04-19T23:26:38.973 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj20 2024-04-19T23:26:38.973 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qeDPhB5yq7ZD0YA8az30o6cAW6w= 2024-04-19T23:26:38.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qeDPhB5yq7ZD0YA8az30o6cAW6w=', 'Content-Length': '0'} 2024-04-19T23:26:38.975 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e2df0f6db97c5a8-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.976 DEBUG:boto:path=/aktlji-16/obj20 2024-04-19T23:26:38.976 DEBUG:boto:auth_path=/aktlji-16/obj20 2024-04-19T23:26:38.976 DEBUG:boto:Method: GET 2024-04-19T23:26:38.976 DEBUG:boto:Path: /aktlji-16/obj20 2024-04-19T23:26:38.976 DEBUG:boto:Data: 2024-04-19T23:26:38.976 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.976 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.976 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.976 DEBUG:boto:Params: {} 2024-04-19T23:26:38.976 DEBUG:boto:Token: None 2024-04-19T23:26:38.977 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj20 2024-04-19T23:26:38.977 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qeDPhB5yq7ZD0YA8az30o6cAW6w= 2024-04-19T23:26:38.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qeDPhB5yq7ZD0YA8az30o6cAW6w=', 'Content-Length': '0'} 2024-04-19T23:26:38.979 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000424dea6409783783-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.979 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.979 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.979 DEBUG:boto:path=/aktlji-16/?prefix=obj20 2024-04-19T23:26:38.979 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj20 2024-04-19T23:26:38.979 DEBUG:boto:Method: GET 2024-04-19T23:26:38.979 DEBUG:boto:Path: /aktlji-16/?prefix=obj20 2024-04-19T23:26:38.979 DEBUG:boto:Data: 2024-04-19T23:26:38.979 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.980 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.980 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.980 DEBUG:boto:Params: {} 2024-04-19T23:26:38.980 DEBUG:boto:Token: None 2024-04-19T23:26:38.980 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.980 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000c777bbcbedd125-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.983 DEBUG:boto:b'aktlji-16obj201000falseobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.983 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:38.983 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:38.983 DEBUG:boto:path=/aktlji-16/?prefix=obj20 2024-04-19T23:26:38.983 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj20 2024-04-19T23:26:38.984 DEBUG:boto:Method: GET 2024-04-19T23:26:38.984 DEBUG:boto:Path: /aktlji-16/?prefix=obj20 2024-04-19T23:26:38.984 DEBUG:boto:Data: 2024-04-19T23:26:38.984 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.984 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:38.984 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.984 DEBUG:boto:Params: {} 2024-04-19T23:26:38.984 DEBUG:boto:Token: None 2024-04-19T23:26:38.984 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/ 2024-04-19T23:26:38.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-19T23:26:38.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-19T23:26:38.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040380aa8bda896ac-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:38.987 DEBUG:boto:b'aktlji-16obj201000falseobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:38.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj21 2024-04-19T23:26:38.988 DEBUG:boto:path=/aktlji-16/obj21 2024-04-19T23:26:38.988 DEBUG:boto:auth_path=/aktlji-16/obj21 2024-04-19T23:26:38.988 DEBUG:boto:path=/aktlji-16/obj21?versionId=null 2024-04-19T23:26:38.988 DEBUG:boto:auth_path=/aktlji-16/obj21?versionId=null 2024-04-19T23:26:38.988 DEBUG:boto:Method: GET 2024-04-19T23:26:38.988 DEBUG:boto:Path: /aktlji-16/obj21?versionId=null 2024-04-19T23:26:38.988 DEBUG:boto:Data: 2024-04-19T23:26:38.988 DEBUG:boto:Headers: {} 2024-04-19T23:26:38.989 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:38.989 DEBUG:boto:Port: 8000 2024-04-19T23:26:38.989 DEBUG:boto:Params: {} 2024-04-19T23:26:38.989 DEBUG:boto:Token: None 2024-04-19T23:26:38.989 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:38 GMT /aktlji-16/obj21?versionId=null 2024-04-19T23:26:38.989 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u488v733Fw92mSGAWfeXusrgYyg= 2024-04-19T23:26:38.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u488v733Fw92mSGAWfeXusrgYyg=', 'Content-Length': '0'} 2024-04-19T23:26:38.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df23d329556cbbde-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.038 DEBUG:boto:path=/aktlji-16/obj21 2024-04-19T23:26:39.038 DEBUG:boto:auth_path=/aktlji-16/obj21 2024-04-19T23:26:39.039 DEBUG:boto:path=/aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.039 DEBUG:boto:auth_path=/aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.039 DEBUG:boto:Method: GET 2024-04-19T23:26:39.039 DEBUG:boto:Path: /aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.039 DEBUG:boto:Data: 2024-04-19T23:26:39.039 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.039 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.039 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.039 DEBUG:boto:Params: {} 2024-04-19T23:26:39.039 DEBUG:boto:Token: None 2024-04-19T23:26:39.040 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.040 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ddoSZzT7/LHyKbw2oj5SllHeRdA= 2024-04-19T23:26:39.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ddoSZzT7/LHyKbw2oj5SllHeRdA=', 'Content-Length': '0'} 2024-04-19T23:26:39.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c1432c4d3d1c9fc-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.042 DEBUG:boto:path=/aktlji-16/obj21 2024-04-19T23:26:39.043 DEBUG:boto:auth_path=/aktlji-16/obj21 2024-04-19T23:26:39.043 DEBUG:boto:path=/aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.043 DEBUG:boto:auth_path=/aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.043 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.043 DEBUG:boto:Path: /aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.043 DEBUG:boto:Data: 2024-04-19T23:26:39.043 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.043 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.043 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.044 DEBUG:boto:Params: {} 2024-04-19T23:26:39.044 DEBUG:boto:Token: None 2024-04-19T23:26:39.044 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.044 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UMgmIzmz5t5zJlByBsJY0+qKg4I= 2024-04-19T23:26:39.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UMgmIzmz5t5zJlByBsJY0+qKg4I=', 'Content-Length': '0'} 2024-04-19T23:26:39.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000309a4113fbe4c0ee-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.047 DEBUG:boto:path=/aktlji-16/obj21 2024-04-19T23:26:39.047 DEBUG:boto:auth_path=/aktlji-16/obj21 2024-04-19T23:26:39.047 DEBUG:boto:path=/aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.047 DEBUG:boto:auth_path=/aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.047 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.047 DEBUG:boto:Path: /aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.047 DEBUG:boto:Data: 2024-04-19T23:26:39.047 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.047 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.048 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.048 DEBUG:boto:Params: {} 2024-04-19T23:26:39.048 DEBUG:boto:Token: None 2024-04-19T23:26:39.048 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.048 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UMgmIzmz5t5zJlByBsJY0+qKg4I= 2024-04-19T23:26:39.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UMgmIzmz5t5zJlByBsJY0+qKg4I=', 'Content-Length': '0'} 2024-04-19T23:26:39.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068ae06adcfba6a44-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj21 2024-04-19T23:26:39.050 DEBUG:boto:path=/aktlji-16/obj21 2024-04-19T23:26:39.050 DEBUG:boto:auth_path=/aktlji-16/obj21 2024-04-19T23:26:39.050 DEBUG:boto:path=/aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.050 DEBUG:boto:auth_path=/aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.050 DEBUG:boto:Method: GET 2024-04-19T23:26:39.051 DEBUG:boto:Path: /aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.051 DEBUG:boto:Data: 2024-04-19T23:26:39.051 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.051 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.051 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.051 DEBUG:boto:Params: {} 2024-04-19T23:26:39.051 DEBUG:boto:Token: None 2024-04-19T23:26:39.051 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.051 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ddoSZzT7/LHyKbw2oj5SllHeRdA= 2024-04-19T23:26:39.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ddoSZzT7/LHyKbw2oj5SllHeRdA=', 'Content-Length': '0'} 2024-04-19T23:26:39.053 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b29d4e8d20ada028-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.054 DEBUG:boto:path=/aktlji-16/obj21 2024-04-19T23:26:39.054 DEBUG:boto:auth_path=/aktlji-16/obj21 2024-04-19T23:26:39.054 DEBUG:boto:path=/aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.054 DEBUG:boto:auth_path=/aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.054 DEBUG:boto:Method: GET 2024-04-19T23:26:39.054 DEBUG:boto:Path: /aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.055 DEBUG:boto:Data: 2024-04-19T23:26:39.055 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.055 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.055 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.055 DEBUG:boto:Params: {} 2024-04-19T23:26:39.055 DEBUG:boto:Token: None 2024-04-19T23:26:39.055 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj21?versionId=null 2024-04-19T23:26:39.055 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ddoSZzT7/LHyKbw2oj5SllHeRdA= 2024-04-19T23:26:39.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ddoSZzT7/LHyKbw2oj5SllHeRdA=', 'Content-Length': '0'} 2024-04-19T23:26:39.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021bbaadec8afe5b6-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.098 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.099 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.099 DEBUG:boto:path=/aktlji-16/?prefix=obj21 2024-04-19T23:26:39.099 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj21 2024-04-19T23:26:39.099 DEBUG:boto:Method: GET 2024-04-19T23:26:39.099 DEBUG:boto:Path: /aktlji-16/?prefix=obj21 2024-04-19T23:26:39.099 DEBUG:boto:Data: 2024-04-19T23:26:39.099 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.099 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.100 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.100 DEBUG:boto:Params: {} 2024-04-19T23:26:39.100 DEBUG:boto:Token: None 2024-04-19T23:26:39.100 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.100 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000057a9d6e54011310-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.103 DEBUG:boto:b'aktlji-16obj211000falseobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.103 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.103 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.103 DEBUG:boto:path=/aktlji-16/?prefix=obj21 2024-04-19T23:26:39.103 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj21 2024-04-19T23:26:39.104 DEBUG:boto:Method: GET 2024-04-19T23:26:39.104 DEBUG:boto:Path: /aktlji-16/?prefix=obj21 2024-04-19T23:26:39.104 DEBUG:boto:Data: 2024-04-19T23:26:39.104 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.104 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.104 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.104 DEBUG:boto:Params: {} 2024-04-19T23:26:39.104 DEBUG:boto:Token: None 2024-04-19T23:26:39.104 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062cab148c7510608-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.107 DEBUG:boto:b'aktlji-16obj211000falseobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.108 DEBUG:boto:path=/aktlji-16/obj21 2024-04-19T23:26:39.108 DEBUG:boto:auth_path=/aktlji-16/obj21 2024-04-19T23:26:39.108 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.108 DEBUG:boto:Path: /aktlji-16/obj21 2024-04-19T23:26:39.108 DEBUG:boto:Data: 2024-04-19T23:26:39.108 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.108 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.108 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.108 DEBUG:boto:Params: {} 2024-04-19T23:26:39.108 DEBUG:boto:Token: None 2024-04-19T23:26:39.109 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj21 2024-04-19T23:26:39.109 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8pIl+ISpl7iql7XRcM0H1zseptE= 2024-04-19T23:26:39.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8pIl+ISpl7iql7XRcM0H1zseptE=', 'Content-Length': '0'} 2024-04-19T23:26:39.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004432b9442b78a634-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.111 DEBUG:boto:path=/aktlji-16/obj21 2024-04-19T23:26:39.111 DEBUG:boto:auth_path=/aktlji-16/obj21 2024-04-19T23:26:39.112 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.112 DEBUG:boto:Path: /aktlji-16/obj21 2024-04-19T23:26:39.112 DEBUG:boto:Data: 2024-04-19T23:26:39.112 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.112 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.112 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.112 DEBUG:boto:Params: {} 2024-04-19T23:26:39.112 DEBUG:boto:Token: None 2024-04-19T23:26:39.112 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj21 2024-04-19T23:26:39.112 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8pIl+ISpl7iql7XRcM0H1zseptE= 2024-04-19T23:26:39.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8pIl+ISpl7iql7XRcM0H1zseptE=', 'Content-Length': '0'} 2024-04-19T23:26:39.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d594960045b90936-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj21 2024-04-19T23:26:39.114 DEBUG:boto:path=/aktlji-16/obj21 2024-04-19T23:26:39.114 DEBUG:boto:auth_path=/aktlji-16/obj21 2024-04-19T23:26:39.115 DEBUG:boto:Method: GET 2024-04-19T23:26:39.115 DEBUG:boto:Path: /aktlji-16/obj21 2024-04-19T23:26:39.115 DEBUG:boto:Data: 2024-04-19T23:26:39.115 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.115 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.115 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.115 DEBUG:boto:Params: {} 2024-04-19T23:26:39.115 DEBUG:boto:Token: None 2024-04-19T23:26:39.115 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj21 2024-04-19T23:26:39.115 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mGQatwtgeRAtAUz/O2neLzPulfA= 2024-04-19T23:26:39.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mGQatwtgeRAtAUz/O2neLzPulfA=', 'Content-Length': '0'} 2024-04-19T23:26:39.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000833dbd9b87e1b39c-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.158 DEBUG:boto:path=/aktlji-16/obj21 2024-04-19T23:26:39.159 DEBUG:boto:auth_path=/aktlji-16/obj21 2024-04-19T23:26:39.159 DEBUG:boto:Method: GET 2024-04-19T23:26:39.159 DEBUG:boto:Path: /aktlji-16/obj21 2024-04-19T23:26:39.159 DEBUG:boto:Data: 2024-04-19T23:26:39.159 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.159 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.159 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.159 DEBUG:boto:Params: {} 2024-04-19T23:26:39.160 DEBUG:boto:Token: None 2024-04-19T23:26:39.160 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj21 2024-04-19T23:26:39.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mGQatwtgeRAtAUz/O2neLzPulfA= 2024-04-19T23:26:39.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mGQatwtgeRAtAUz/O2neLzPulfA=', 'Content-Length': '0'} 2024-04-19T23:26:39.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000464f5cbd6f6715fe-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.164 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.164 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.164 DEBUG:boto:path=/aktlji-16/?prefix=obj21 2024-04-19T23:26:39.164 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj21 2024-04-19T23:26:39.164 DEBUG:boto:Method: GET 2024-04-19T23:26:39.165 DEBUG:boto:Path: /aktlji-16/?prefix=obj21 2024-04-19T23:26:39.165 DEBUG:boto:Data: 2024-04-19T23:26:39.165 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.165 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.165 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.166 DEBUG:boto:Params: {} 2024-04-19T23:26:39.166 DEBUG:boto:Token: None 2024-04-19T23:26:39.166 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.166 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a35c30fae5b90ca9-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.170 DEBUG:boto:b'aktlji-16obj211000falseobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.170 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.170 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.170 DEBUG:boto:path=/aktlji-16/?prefix=obj21 2024-04-19T23:26:39.170 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj21 2024-04-19T23:26:39.171 DEBUG:boto:Method: GET 2024-04-19T23:26:39.171 DEBUG:boto:Path: /aktlji-16/?prefix=obj21 2024-04-19T23:26:39.171 DEBUG:boto:Data: 2024-04-19T23:26:39.171 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.171 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.171 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.171 DEBUG:boto:Params: {} 2024-04-19T23:26:39.171 DEBUG:boto:Token: None 2024-04-19T23:26:39.171 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.172 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000753f3a79bf3515c6-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.174 DEBUG:boto:b'aktlji-16obj211000falseobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-19T23:26:39.175 DEBUG:boto:path=/aktlji-16/obj22 2024-04-19T23:26:39.175 DEBUG:boto:auth_path=/aktlji-16/obj22 2024-04-19T23:26:39.175 DEBUG:boto:path=/aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.175 DEBUG:boto:auth_path=/aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.175 DEBUG:boto:Method: GET 2024-04-19T23:26:39.175 DEBUG:boto:Path: /aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.175 DEBUG:boto:Data: 2024-04-19T23:26:39.176 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.176 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.176 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.176 DEBUG:boto:Params: {} 2024-04-19T23:26:39.176 DEBUG:boto:Token: None 2024-04-19T23:26:39.176 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.176 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0= 2024-04-19T23:26:39.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0=', 'Content-Length': '0'} 2024-04-19T23:26:39.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d65593f3bf760cdf-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.178 DEBUG:boto:path=/aktlji-16/obj22 2024-04-19T23:26:39.178 DEBUG:boto:auth_path=/aktlji-16/obj22 2024-04-19T23:26:39.179 DEBUG:boto:path=/aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.179 DEBUG:boto:auth_path=/aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.179 DEBUG:boto:Method: GET 2024-04-19T23:26:39.179 DEBUG:boto:Path: /aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.179 DEBUG:boto:Data: 2024-04-19T23:26:39.179 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.179 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.179 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.179 DEBUG:boto:Params: {} 2024-04-19T23:26:39.179 DEBUG:boto:Token: None 2024-04-19T23:26:39.180 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0= 2024-04-19T23:26:39.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0=', 'Content-Length': '0'} 2024-04-19T23:26:39.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c824670dfb4551d4-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.226 DEBUG:boto:path=/aktlji-16/obj22 2024-04-19T23:26:39.227 DEBUG:boto:auth_path=/aktlji-16/obj22 2024-04-19T23:26:39.227 DEBUG:boto:path=/aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.227 DEBUG:boto:auth_path=/aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.227 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.227 DEBUG:boto:Path: /aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.228 DEBUG:boto:Data: 2024-04-19T23:26:39.228 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.228 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.228 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.228 DEBUG:boto:Params: {} 2024-04-19T23:26:39.228 DEBUG:boto:Token: None 2024-04-19T23:26:39.229 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.229 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fwJRdN5aQIT0A/zB1S6AtCwlSJc= 2024-04-19T23:26:39.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fwJRdN5aQIT0A/zB1S6AtCwlSJc=', 'Content-Length': '0'} 2024-04-19T23:26:39.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdc7984b1ea3ca3e-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.232 DEBUG:boto:path=/aktlji-16/obj22 2024-04-19T23:26:39.232 DEBUG:boto:auth_path=/aktlji-16/obj22 2024-04-19T23:26:39.232 DEBUG:boto:path=/aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.232 DEBUG:boto:auth_path=/aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.233 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.233 DEBUG:boto:Path: /aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.233 DEBUG:boto:Data: 2024-04-19T23:26:39.233 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.233 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.233 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.234 DEBUG:boto:Params: {} 2024-04-19T23:26:39.234 DEBUG:boto:Token: None 2024-04-19T23:26:39.234 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.234 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fwJRdN5aQIT0A/zB1S6AtCwlSJc= 2024-04-19T23:26:39.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fwJRdN5aQIT0A/zB1S6AtCwlSJc=', 'Content-Length': '0'} 2024-04-19T23:26:39.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb47f4cef51684b8-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-19T23:26:39.237 DEBUG:boto:path=/aktlji-16/obj22 2024-04-19T23:26:39.237 DEBUG:boto:auth_path=/aktlji-16/obj22 2024-04-19T23:26:39.237 DEBUG:boto:path=/aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.238 DEBUG:boto:auth_path=/aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.238 DEBUG:boto:Method: GET 2024-04-19T23:26:39.238 DEBUG:boto:Path: /aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.238 DEBUG:boto:Data: 2024-04-19T23:26:39.238 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.238 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.238 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.238 DEBUG:boto:Params: {} 2024-04-19T23:26:39.239 DEBUG:boto:Token: None 2024-04-19T23:26:39.239 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.239 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0= 2024-04-19T23:26:39.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0=', 'Content-Length': '0'} 2024-04-19T23:26:39.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad3c413068c96867-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.241 DEBUG:boto:path=/aktlji-16/obj22 2024-04-19T23:26:39.241 DEBUG:boto:auth_path=/aktlji-16/obj22 2024-04-19T23:26:39.241 DEBUG:boto:path=/aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.241 DEBUG:boto:auth_path=/aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.241 DEBUG:boto:Method: GET 2024-04-19T23:26:39.242 DEBUG:boto:Path: /aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.242 DEBUG:boto:Data: 2024-04-19T23:26:39.242 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.242 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.242 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.242 DEBUG:boto:Params: {} 2024-04-19T23:26:39.242 DEBUG:boto:Token: None 2024-04-19T23:26:39.243 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj22?versionId=null 2024-04-19T23:26:39.243 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0= 2024-04-19T23:26:39.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0=', 'Content-Length': '0'} 2024-04-19T23:26:39.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000158037c7d39bcf84-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.245 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.245 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.245 DEBUG:boto:path=/aktlji-16/?prefix=obj22 2024-04-19T23:26:39.245 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj22 2024-04-19T23:26:39.245 DEBUG:boto:Method: GET 2024-04-19T23:26:39.245 DEBUG:boto:Path: /aktlji-16/?prefix=obj22 2024-04-19T23:26:39.245 DEBUG:boto:Data: 2024-04-19T23:26:39.246 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.246 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.246 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.246 DEBUG:boto:Params: {} 2024-04-19T23:26:39.246 DEBUG:boto:Token: None 2024-04-19T23:26:39.246 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.246 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002e5cb08ed524e93-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.249 DEBUG:boto:b'aktlji-16obj221000falseobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.249 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.249 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.250 DEBUG:boto:path=/aktlji-16/?prefix=obj22 2024-04-19T23:26:39.250 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj22 2024-04-19T23:26:39.250 DEBUG:boto:Method: GET 2024-04-19T23:26:39.250 DEBUG:boto:Path: /aktlji-16/?prefix=obj22 2024-04-19T23:26:39.250 DEBUG:boto:Data: 2024-04-19T23:26:39.250 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.250 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.250 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.250 DEBUG:boto:Params: {} 2024-04-19T23:26:39.251 DEBUG:boto:Token: None 2024-04-19T23:26:39.251 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.251 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c7454df610f9800-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.253 DEBUG:boto:b'aktlji-16obj221000falseobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.254 DEBUG:boto:path=/aktlji-16/obj22 2024-04-19T23:26:39.254 DEBUG:boto:auth_path=/aktlji-16/obj22 2024-04-19T23:26:39.254 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.254 DEBUG:boto:Path: /aktlji-16/obj22 2024-04-19T23:26:39.254 DEBUG:boto:Data: 2024-04-19T23:26:39.254 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.254 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.255 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.255 DEBUG:boto:Params: {} 2024-04-19T23:26:39.255 DEBUG:boto:Token: None 2024-04-19T23:26:39.255 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj22 2024-04-19T23:26:39.255 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WL7RhfzDw1pLoroMDyP3lXgv70s= 2024-04-19T23:26:39.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WL7RhfzDw1pLoroMDyP3lXgv70s=', 'Content-Length': '0'} 2024-04-19T23:26:39.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000306ddaf0b6cb0ffd-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.257 DEBUG:boto:path=/aktlji-16/obj22 2024-04-19T23:26:39.257 DEBUG:boto:auth_path=/aktlji-16/obj22 2024-04-19T23:26:39.257 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.257 DEBUG:boto:Path: /aktlji-16/obj22 2024-04-19T23:26:39.257 DEBUG:boto:Data: 2024-04-19T23:26:39.257 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.258 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.258 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.258 DEBUG:boto:Params: {} 2024-04-19T23:26:39.258 DEBUG:boto:Token: None 2024-04-19T23:26:39.258 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj22 2024-04-19T23:26:39.258 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WL7RhfzDw1pLoroMDyP3lXgv70s= 2024-04-19T23:26:39.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WL7RhfzDw1pLoroMDyP3lXgv70s=', 'Content-Length': '0'} 2024-04-19T23:26:39.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000468951e1a8995ea1-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-19T23:26:39.260 DEBUG:boto:path=/aktlji-16/obj22 2024-04-19T23:26:39.260 DEBUG:boto:auth_path=/aktlji-16/obj22 2024-04-19T23:26:39.261 DEBUG:boto:Method: GET 2024-04-19T23:26:39.261 DEBUG:boto:Path: /aktlji-16/obj22 2024-04-19T23:26:39.261 DEBUG:boto:Data: 2024-04-19T23:26:39.261 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.261 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.261 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.261 DEBUG:boto:Params: {} 2024-04-19T23:26:39.261 DEBUG:boto:Token: None 2024-04-19T23:26:39.261 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj22 2024-04-19T23:26:39.262 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dJFa7gXb1tTrQudS/yeWYCzM+VA= 2024-04-19T23:26:39.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dJFa7gXb1tTrQudS/yeWYCzM+VA=', 'Content-Length': '0'} 2024-04-19T23:26:39.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044a4b1e79ea6306c-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.264 DEBUG:boto:path=/aktlji-16/obj22 2024-04-19T23:26:39.264 DEBUG:boto:auth_path=/aktlji-16/obj22 2024-04-19T23:26:39.264 DEBUG:boto:Method: GET 2024-04-19T23:26:39.264 DEBUG:boto:Path: /aktlji-16/obj22 2024-04-19T23:26:39.264 DEBUG:boto:Data: 2024-04-19T23:26:39.264 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.264 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.264 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.264 DEBUG:boto:Params: {} 2024-04-19T23:26:39.265 DEBUG:boto:Token: None 2024-04-19T23:26:39.265 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj22 2024-04-19T23:26:39.265 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dJFa7gXb1tTrQudS/yeWYCzM+VA= 2024-04-19T23:26:39.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dJFa7gXb1tTrQudS/yeWYCzM+VA=', 'Content-Length': '0'} 2024-04-19T23:26:39.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dd12092021f769e-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.267 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.267 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.267 DEBUG:boto:path=/aktlji-16/?prefix=obj22 2024-04-19T23:26:39.267 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj22 2024-04-19T23:26:39.267 DEBUG:boto:Method: GET 2024-04-19T23:26:39.267 DEBUG:boto:Path: /aktlji-16/?prefix=obj22 2024-04-19T23:26:39.268 DEBUG:boto:Data: 2024-04-19T23:26:39.268 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.268 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.268 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.268 DEBUG:boto:Params: {} 2024-04-19T23:26:39.268 DEBUG:boto:Token: None 2024-04-19T23:26:39.268 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.268 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000474b06f90042beb3-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.271 DEBUG:boto:b'aktlji-16obj221000falseobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.271 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.271 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.272 DEBUG:boto:path=/aktlji-16/?prefix=obj22 2024-04-19T23:26:39.272 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj22 2024-04-19T23:26:39.272 DEBUG:boto:Method: GET 2024-04-19T23:26:39.272 DEBUG:boto:Path: /aktlji-16/?prefix=obj22 2024-04-19T23:26:39.272 DEBUG:boto:Data: 2024-04-19T23:26:39.272 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.272 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.273 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.273 DEBUG:boto:Params: {} 2024-04-19T23:26:39.273 DEBUG:boto:Token: None 2024-04-19T23:26:39.273 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.273 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043101fdfa2f9e07d-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.276 DEBUG:boto:b'aktlji-16obj221000falseobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-19T23:26:39.277 DEBUG:boto:path=/aktlji-16/obj23 2024-04-19T23:26:39.277 DEBUG:boto:auth_path=/aktlji-16/obj23 2024-04-19T23:26:39.277 DEBUG:boto:path=/aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.277 DEBUG:boto:auth_path=/aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.277 DEBUG:boto:Method: GET 2024-04-19T23:26:39.277 DEBUG:boto:Path: /aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.277 DEBUG:boto:Data: 2024-04-19T23:26:39.277 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.277 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.277 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.278 DEBUG:boto:Params: {} 2024-04-19T23:26:39.278 DEBUG:boto:Token: None 2024-04-19T23:26:39.278 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.278 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34= 2024-04-19T23:26:39.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34=', 'Content-Length': '0'} 2024-04-19T23:26:39.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9a9840d29968428-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.326 DEBUG:boto:path=/aktlji-16/obj23 2024-04-19T23:26:39.327 DEBUG:boto:auth_path=/aktlji-16/obj23 2024-04-19T23:26:39.327 DEBUG:boto:path=/aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.327 DEBUG:boto:auth_path=/aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.327 DEBUG:boto:Method: GET 2024-04-19T23:26:39.327 DEBUG:boto:Path: /aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.327 DEBUG:boto:Data: 2024-04-19T23:26:39.327 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.327 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.327 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.327 DEBUG:boto:Params: {} 2024-04-19T23:26:39.328 DEBUG:boto:Token: None 2024-04-19T23:26:39.328 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.328 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34= 2024-04-19T23:26:39.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34=', 'Content-Length': '0'} 2024-04-19T23:26:39.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000348ed0d0c624089f-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.331 DEBUG:boto:path=/aktlji-16/obj23 2024-04-19T23:26:39.331 DEBUG:boto:auth_path=/aktlji-16/obj23 2024-04-19T23:26:39.331 DEBUG:boto:path=/aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.332 DEBUG:boto:auth_path=/aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.332 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.332 DEBUG:boto:Path: /aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.332 DEBUG:boto:Data: 2024-04-19T23:26:39.332 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.332 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.333 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.333 DEBUG:boto:Params: {} 2024-04-19T23:26:39.333 DEBUG:boto:Token: None 2024-04-19T23:26:39.333 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.333 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NlYF3RZ4OY9zuYTrxKjBE2x/fsg= 2024-04-19T23:26:39.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NlYF3RZ4OY9zuYTrxKjBE2x/fsg=', 'Content-Length': '0'} 2024-04-19T23:26:39.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b399f08d309a752b-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.336 DEBUG:boto:path=/aktlji-16/obj23 2024-04-19T23:26:39.336 DEBUG:boto:auth_path=/aktlji-16/obj23 2024-04-19T23:26:39.336 DEBUG:boto:path=/aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.336 DEBUG:boto:auth_path=/aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.336 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.337 DEBUG:boto:Path: /aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.337 DEBUG:boto:Data: 2024-04-19T23:26:39.337 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.337 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.337 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.337 DEBUG:boto:Params: {} 2024-04-19T23:26:39.337 DEBUG:boto:Token: None 2024-04-19T23:26:39.337 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.337 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NlYF3RZ4OY9zuYTrxKjBE2x/fsg= 2024-04-19T23:26:39.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NlYF3RZ4OY9zuYTrxKjBE2x/fsg=', 'Content-Length': '0'} 2024-04-19T23:26:39.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000431c274c4e408b66-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-19T23:26:39.340 DEBUG:boto:path=/aktlji-16/obj23 2024-04-19T23:26:39.340 DEBUG:boto:auth_path=/aktlji-16/obj23 2024-04-19T23:26:39.340 DEBUG:boto:path=/aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.340 DEBUG:boto:auth_path=/aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.340 DEBUG:boto:Method: GET 2024-04-19T23:26:39.340 DEBUG:boto:Path: /aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.340 DEBUG:boto:Data: 2024-04-19T23:26:39.340 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.340 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.340 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.340 DEBUG:boto:Params: {} 2024-04-19T23:26:39.341 DEBUG:boto:Token: None 2024-04-19T23:26:39.341 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.341 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34= 2024-04-19T23:26:39.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34=', 'Content-Length': '0'} 2024-04-19T23:26:39.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083dd3c70d3e35cd1-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.343 DEBUG:boto:path=/aktlji-16/obj23 2024-04-19T23:26:39.343 DEBUG:boto:auth_path=/aktlji-16/obj23 2024-04-19T23:26:39.343 DEBUG:boto:path=/aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.343 DEBUG:boto:auth_path=/aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.343 DEBUG:boto:Method: GET 2024-04-19T23:26:39.343 DEBUG:boto:Path: /aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.344 DEBUG:boto:Data: 2024-04-19T23:26:39.344 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.344 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.344 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.344 DEBUG:boto:Params: {} 2024-04-19T23:26:39.344 DEBUG:boto:Token: None 2024-04-19T23:26:39.344 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj23?versionId=null 2024-04-19T23:26:39.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34= 2024-04-19T23:26:39.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34=', 'Content-Length': '0'} 2024-04-19T23:26:39.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dc296d90d71fc15-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.386 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.386 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.387 DEBUG:boto:path=/aktlji-16/?prefix=obj23 2024-04-19T23:26:39.387 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj23 2024-04-19T23:26:39.387 DEBUG:boto:Method: GET 2024-04-19T23:26:39.387 DEBUG:boto:Path: /aktlji-16/?prefix=obj23 2024-04-19T23:26:39.387 DEBUG:boto:Data: 2024-04-19T23:26:39.387 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.387 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.387 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.387 DEBUG:boto:Params: {} 2024-04-19T23:26:39.387 DEBUG:boto:Token: None 2024-04-19T23:26:39.388 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.388 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053c2f8f37bc1ed66-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.390 DEBUG:boto:b'aktlji-16obj231000falseobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.391 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.391 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.391 DEBUG:boto:path=/aktlji-16/?prefix=obj23 2024-04-19T23:26:39.391 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj23 2024-04-19T23:26:39.391 DEBUG:boto:Method: GET 2024-04-19T23:26:39.391 DEBUG:boto:Path: /aktlji-16/?prefix=obj23 2024-04-19T23:26:39.391 DEBUG:boto:Data: 2024-04-19T23:26:39.391 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.392 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.392 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.392 DEBUG:boto:Params: {} 2024-04-19T23:26:39.392 DEBUG:boto:Token: None 2024-04-19T23:26:39.392 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.392 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d612fbc5986fa6d-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.395 DEBUG:boto:b'aktlji-16obj231000falseobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.395 DEBUG:boto:path=/aktlji-16/obj23 2024-04-19T23:26:39.395 DEBUG:boto:auth_path=/aktlji-16/obj23 2024-04-19T23:26:39.395 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.395 DEBUG:boto:Path: /aktlji-16/obj23 2024-04-19T23:26:39.395 DEBUG:boto:Data: 2024-04-19T23:26:39.395 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.396 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.396 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.396 DEBUG:boto:Params: {} 2024-04-19T23:26:39.396 DEBUG:boto:Token: None 2024-04-19T23:26:39.396 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj23 2024-04-19T23:26:39.396 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LPmhOcidz2Eki57bSq0cuvGHGMM= 2024-04-19T23:26:39.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LPmhOcidz2Eki57bSq0cuvGHGMM=', 'Content-Length': '0'} 2024-04-19T23:26:39.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000786c00e593af769a-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.398 DEBUG:boto:path=/aktlji-16/obj23 2024-04-19T23:26:39.399 DEBUG:boto:auth_path=/aktlji-16/obj23 2024-04-19T23:26:39.399 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.399 DEBUG:boto:Path: /aktlji-16/obj23 2024-04-19T23:26:39.399 DEBUG:boto:Data: 2024-04-19T23:26:39.399 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.399 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.400 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.400 DEBUG:boto:Params: {} 2024-04-19T23:26:39.400 DEBUG:boto:Token: None 2024-04-19T23:26:39.400 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj23 2024-04-19T23:26:39.401 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LPmhOcidz2Eki57bSq0cuvGHGMM= 2024-04-19T23:26:39.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LPmhOcidz2Eki57bSq0cuvGHGMM=', 'Content-Length': '0'} 2024-04-19T23:26:39.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9576f91d160c181-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-19T23:26:39.403 DEBUG:boto:path=/aktlji-16/obj23 2024-04-19T23:26:39.404 DEBUG:boto:auth_path=/aktlji-16/obj23 2024-04-19T23:26:39.404 DEBUG:boto:Method: GET 2024-04-19T23:26:39.404 DEBUG:boto:Path: /aktlji-16/obj23 2024-04-19T23:26:39.404 DEBUG:boto:Data: 2024-04-19T23:26:39.404 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.405 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.405 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.405 DEBUG:boto:Params: {} 2024-04-19T23:26:39.405 DEBUG:boto:Token: None 2024-04-19T23:26:39.405 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj23 2024-04-19T23:26:39.406 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Qj3Tv3Cxsnnq3vY75Q+SinroJ40= 2024-04-19T23:26:39.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Qj3Tv3Cxsnnq3vY75Q+SinroJ40=', 'Content-Length': '0'} 2024-04-19T23:26:39.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007cbf2aa96218d9b2-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.409 DEBUG:boto:path=/aktlji-16/obj23 2024-04-19T23:26:39.409 DEBUG:boto:auth_path=/aktlji-16/obj23 2024-04-19T23:26:39.409 DEBUG:boto:Method: GET 2024-04-19T23:26:39.409 DEBUG:boto:Path: /aktlji-16/obj23 2024-04-19T23:26:39.409 DEBUG:boto:Data: 2024-04-19T23:26:39.409 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.409 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.410 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.410 DEBUG:boto:Params: {} 2024-04-19T23:26:39.410 DEBUG:boto:Token: None 2024-04-19T23:26:39.410 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj23 2024-04-19T23:26:39.410 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Qj3Tv3Cxsnnq3vY75Q+SinroJ40= 2024-04-19T23:26:39.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Qj3Tv3Cxsnnq3vY75Q+SinroJ40=', 'Content-Length': '0'} 2024-04-19T23:26:39.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a27a2c177bebef8-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.458 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.459 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.459 DEBUG:boto:path=/aktlji-16/?prefix=obj23 2024-04-19T23:26:39.459 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj23 2024-04-19T23:26:39.459 DEBUG:boto:Method: GET 2024-04-19T23:26:39.459 DEBUG:boto:Path: /aktlji-16/?prefix=obj23 2024-04-19T23:26:39.459 DEBUG:boto:Data: 2024-04-19T23:26:39.459 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.459 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.459 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.459 DEBUG:boto:Params: {} 2024-04-19T23:26:39.460 DEBUG:boto:Token: None 2024-04-19T23:26:39.460 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015dcd4dc52c2b550-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.463 DEBUG:boto:b'aktlji-16obj231000falseobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.463 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.463 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.464 DEBUG:boto:path=/aktlji-16/?prefix=obj23 2024-04-19T23:26:39.464 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj23 2024-04-19T23:26:39.464 DEBUG:boto:Method: GET 2024-04-19T23:26:39.464 DEBUG:boto:Path: /aktlji-16/?prefix=obj23 2024-04-19T23:26:39.464 DEBUG:boto:Data: 2024-04-19T23:26:39.464 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.464 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.464 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.464 DEBUG:boto:Params: {} 2024-04-19T23:26:39.464 DEBUG:boto:Token: None 2024-04-19T23:26:39.465 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.465 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054a28fe5a6a1663f-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.467 DEBUG:boto:b'aktlji-16obj231000falseobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj24 2024-04-19T23:26:39.468 DEBUG:boto:path=/aktlji-16/obj24 2024-04-19T23:26:39.468 DEBUG:boto:auth_path=/aktlji-16/obj24 2024-04-19T23:26:39.468 DEBUG:boto:path=/aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.468 DEBUG:boto:auth_path=/aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.468 DEBUG:boto:Method: GET 2024-04-19T23:26:39.468 DEBUG:boto:Path: /aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.468 DEBUG:boto:Data: 2024-04-19T23:26:39.468 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.468 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.469 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.469 DEBUG:boto:Params: {} 2024-04-19T23:26:39.469 DEBUG:boto:Token: None 2024-04-19T23:26:39.469 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.469 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o= 2024-04-19T23:26:39.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o=', 'Content-Length': '0'} 2024-04-19T23:26:39.471 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f798c9ccc1720c5d-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.514 DEBUG:boto:path=/aktlji-16/obj24 2024-04-19T23:26:39.514 DEBUG:boto:auth_path=/aktlji-16/obj24 2024-04-19T23:26:39.514 DEBUG:boto:path=/aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.515 DEBUG:boto:auth_path=/aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.515 DEBUG:boto:Method: GET 2024-04-19T23:26:39.515 DEBUG:boto:Path: /aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.515 DEBUG:boto:Data: 2024-04-19T23:26:39.515 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.515 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.515 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.515 DEBUG:boto:Params: {} 2024-04-19T23:26:39.515 DEBUG:boto:Token: None 2024-04-19T23:26:39.516 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.516 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o= 2024-04-19T23:26:39.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o=', 'Content-Length': '0'} 2024-04-19T23:26:39.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba625e9f36ddc4fa-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.518 DEBUG:boto:path=/aktlji-16/obj24 2024-04-19T23:26:39.518 DEBUG:boto:auth_path=/aktlji-16/obj24 2024-04-19T23:26:39.518 DEBUG:boto:path=/aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.519 DEBUG:boto:auth_path=/aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.519 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.519 DEBUG:boto:Path: /aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.519 DEBUG:boto:Data: 2024-04-19T23:26:39.519 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.519 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.519 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.519 DEBUG:boto:Params: {} 2024-04-19T23:26:39.519 DEBUG:boto:Token: None 2024-04-19T23:26:39.519 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.520 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MyU+XMttDiay5XfzTxYphkMW6Kc= 2024-04-19T23:26:39.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MyU+XMttDiay5XfzTxYphkMW6Kc=', 'Content-Length': '0'} 2024-04-19T23:26:39.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e34ad0d0fc7fcfa-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.521 DEBUG:boto:path=/aktlji-16/obj24 2024-04-19T23:26:39.522 DEBUG:boto:auth_path=/aktlji-16/obj24 2024-04-19T23:26:39.522 DEBUG:boto:path=/aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.522 DEBUG:boto:auth_path=/aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.522 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.522 DEBUG:boto:Path: /aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.522 DEBUG:boto:Data: 2024-04-19T23:26:39.522 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.522 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.522 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.522 DEBUG:boto:Params: {} 2024-04-19T23:26:39.522 DEBUG:boto:Token: None 2024-04-19T23:26:39.523 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.523 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MyU+XMttDiay5XfzTxYphkMW6Kc= 2024-04-19T23:26:39.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MyU+XMttDiay5XfzTxYphkMW6Kc=', 'Content-Length': '0'} 2024-04-19T23:26:39.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000248777b7e16c2a4e-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj24 2024-04-19T23:26:39.525 DEBUG:boto:path=/aktlji-16/obj24 2024-04-19T23:26:39.525 DEBUG:boto:auth_path=/aktlji-16/obj24 2024-04-19T23:26:39.526 DEBUG:boto:path=/aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.526 DEBUG:boto:auth_path=/aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.526 DEBUG:boto:Method: GET 2024-04-19T23:26:39.526 DEBUG:boto:Path: /aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.526 DEBUG:boto:Data: 2024-04-19T23:26:39.526 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.526 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.526 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.526 DEBUG:boto:Params: {} 2024-04-19T23:26:39.526 DEBUG:boto:Token: None 2024-04-19T23:26:39.527 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.527 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o= 2024-04-19T23:26:39.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o=', 'Content-Length': '0'} 2024-04-19T23:26:39.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d7c15d4c50b10ef-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.529 DEBUG:boto:path=/aktlji-16/obj24 2024-04-19T23:26:39.529 DEBUG:boto:auth_path=/aktlji-16/obj24 2024-04-19T23:26:39.529 DEBUG:boto:path=/aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.529 DEBUG:boto:auth_path=/aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.529 DEBUG:boto:Method: GET 2024-04-19T23:26:39.529 DEBUG:boto:Path: /aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.529 DEBUG:boto:Data: 2024-04-19T23:26:39.529 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.529 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.530 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.530 DEBUG:boto:Params: {} 2024-04-19T23:26:39.530 DEBUG:boto:Token: None 2024-04-19T23:26:39.530 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj24?versionId=null 2024-04-19T23:26:39.530 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o= 2024-04-19T23:26:39.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o=', 'Content-Length': '0'} 2024-04-19T23:26:39.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006398ec760b366a83-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.532 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.532 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.532 DEBUG:boto:path=/aktlji-16/?prefix=obj24 2024-04-19T23:26:39.532 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj24 2024-04-19T23:26:39.533 DEBUG:boto:Method: GET 2024-04-19T23:26:39.533 DEBUG:boto:Path: /aktlji-16/?prefix=obj24 2024-04-19T23:26:39.533 DEBUG:boto:Data: 2024-04-19T23:26:39.533 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.533 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.533 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.533 DEBUG:boto:Params: {} 2024-04-19T23:26:39.533 DEBUG:boto:Token: None 2024-04-19T23:26:39.533 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.534 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bc5eb618eaab5e2-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.536 DEBUG:boto:b'aktlji-16obj241000falseobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.536 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.537 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.537 DEBUG:boto:path=/aktlji-16/?prefix=obj24 2024-04-19T23:26:39.537 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj24 2024-04-19T23:26:39.537 DEBUG:boto:Method: GET 2024-04-19T23:26:39.537 DEBUG:boto:Path: /aktlji-16/?prefix=obj24 2024-04-19T23:26:39.537 DEBUG:boto:Data: 2024-04-19T23:26:39.537 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.537 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.537 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.537 DEBUG:boto:Params: {} 2024-04-19T23:26:39.538 DEBUG:boto:Token: None 2024-04-19T23:26:39.538 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.538 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d13c800c711153c-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.540 DEBUG:boto:b'aktlji-16obj241000falseobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.541 DEBUG:boto:path=/aktlji-16/obj24 2024-04-19T23:26:39.541 DEBUG:boto:auth_path=/aktlji-16/obj24 2024-04-19T23:26:39.541 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.541 DEBUG:boto:Path: /aktlji-16/obj24 2024-04-19T23:26:39.541 DEBUG:boto:Data: 2024-04-19T23:26:39.541 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.541 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.541 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.542 DEBUG:boto:Params: {} 2024-04-19T23:26:39.542 DEBUG:boto:Token: None 2024-04-19T23:26:39.542 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj24 2024-04-19T23:26:39.542 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NKJ+ERI1zY8/zgHiCz8vydyPjlI= 2024-04-19T23:26:39.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NKJ+ERI1zY8/zgHiCz8vydyPjlI=', 'Content-Length': '0'} 2024-04-19T23:26:39.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eea0898b20cdaa15-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.544 DEBUG:boto:path=/aktlji-16/obj24 2024-04-19T23:26:39.544 DEBUG:boto:auth_path=/aktlji-16/obj24 2024-04-19T23:26:39.544 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.544 DEBUG:boto:Path: /aktlji-16/obj24 2024-04-19T23:26:39.544 DEBUG:boto:Data: 2024-04-19T23:26:39.545 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.545 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.545 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.545 DEBUG:boto:Params: {} 2024-04-19T23:26:39.545 DEBUG:boto:Token: None 2024-04-19T23:26:39.545 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj24 2024-04-19T23:26:39.545 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NKJ+ERI1zY8/zgHiCz8vydyPjlI= 2024-04-19T23:26:39.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NKJ+ERI1zY8/zgHiCz8vydyPjlI=', 'Content-Length': '0'} 2024-04-19T23:26:39.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016f49dd54f7a3969-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj24 2024-04-19T23:26:39.548 DEBUG:boto:path=/aktlji-16/obj24 2024-04-19T23:26:39.548 DEBUG:boto:auth_path=/aktlji-16/obj24 2024-04-19T23:26:39.548 DEBUG:boto:Method: GET 2024-04-19T23:26:39.548 DEBUG:boto:Path: /aktlji-16/obj24 2024-04-19T23:26:39.548 DEBUG:boto:Data: 2024-04-19T23:26:39.548 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.548 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.548 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.549 DEBUG:boto:Params: {} 2024-04-19T23:26:39.549 DEBUG:boto:Token: None 2024-04-19T23:26:39.549 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj24 2024-04-19T23:26:39.549 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+Fw0cXjT/uShfolYLVT4e4DFj3o= 2024-04-19T23:26:39.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+Fw0cXjT/uShfolYLVT4e4DFj3o=', 'Content-Length': '0'} 2024-04-19T23:26:39.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f3973514ca06883-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.552 DEBUG:boto:path=/aktlji-16/obj24 2024-04-19T23:26:39.552 DEBUG:boto:auth_path=/aktlji-16/obj24 2024-04-19T23:26:39.552 DEBUG:boto:Method: GET 2024-04-19T23:26:39.552 DEBUG:boto:Path: /aktlji-16/obj24 2024-04-19T23:26:39.552 DEBUG:boto:Data: 2024-04-19T23:26:39.552 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.552 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.553 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.553 DEBUG:boto:Params: {} 2024-04-19T23:26:39.553 DEBUG:boto:Token: None 2024-04-19T23:26:39.553 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj24 2024-04-19T23:26:39.553 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+Fw0cXjT/uShfolYLVT4e4DFj3o= 2024-04-19T23:26:39.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+Fw0cXjT/uShfolYLVT4e4DFj3o=', 'Content-Length': '0'} 2024-04-19T23:26:39.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef85b35ca771f60f-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.556 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.556 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.556 DEBUG:boto:path=/aktlji-16/?prefix=obj24 2024-04-19T23:26:39.556 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj24 2024-04-19T23:26:39.556 DEBUG:boto:Method: GET 2024-04-19T23:26:39.556 DEBUG:boto:Path: /aktlji-16/?prefix=obj24 2024-04-19T23:26:39.556 DEBUG:boto:Data: 2024-04-19T23:26:39.556 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.557 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.557 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.557 DEBUG:boto:Params: {} 2024-04-19T23:26:39.557 DEBUG:boto:Token: None 2024-04-19T23:26:39.557 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.557 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009356729953a40f8b-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.606 DEBUG:boto:b'aktlji-16obj241000falseobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.607 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.607 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.607 DEBUG:boto:path=/aktlji-16/?prefix=obj24 2024-04-19T23:26:39.607 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj24 2024-04-19T23:26:39.607 DEBUG:boto:Method: GET 2024-04-19T23:26:39.607 DEBUG:boto:Path: /aktlji-16/?prefix=obj24 2024-04-19T23:26:39.607 DEBUG:boto:Data: 2024-04-19T23:26:39.607 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.608 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.608 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.608 DEBUG:boto:Params: {} 2024-04-19T23:26:39.608 DEBUG:boto:Token: None 2024-04-19T23:26:39.608 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.608 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d0917c0abc910a2-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.611 DEBUG:boto:b'aktlji-16obj241000falseobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj25 2024-04-19T23:26:39.612 DEBUG:boto:path=/aktlji-16/obj25 2024-04-19T23:26:39.612 DEBUG:boto:auth_path=/aktlji-16/obj25 2024-04-19T23:26:39.612 DEBUG:boto:path=/aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.612 DEBUG:boto:auth_path=/aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.612 DEBUG:boto:Method: GET 2024-04-19T23:26:39.612 DEBUG:boto:Path: /aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.612 DEBUG:boto:Data: 2024-04-19T23:26:39.612 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.612 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.612 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.612 DEBUG:boto:Params: {} 2024-04-19T23:26:39.613 DEBUG:boto:Token: None 2024-04-19T23:26:39.613 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.613 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo= 2024-04-19T23:26:39.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo=', 'Content-Length': '0'} 2024-04-19T23:26:39.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c4462efca65d86e-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.615 DEBUG:boto:path=/aktlji-16/obj25 2024-04-19T23:26:39.615 DEBUG:boto:auth_path=/aktlji-16/obj25 2024-04-19T23:26:39.616 DEBUG:boto:path=/aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.616 DEBUG:boto:auth_path=/aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.616 DEBUG:boto:Method: GET 2024-04-19T23:26:39.617 DEBUG:boto:Path: /aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.617 DEBUG:boto:Data: 2024-04-19T23:26:39.617 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.617 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.617 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.617 DEBUG:boto:Params: {} 2024-04-19T23:26:39.617 DEBUG:boto:Token: None 2024-04-19T23:26:39.617 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.617 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo= 2024-04-19T23:26:39.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo=', 'Content-Length': '0'} 2024-04-19T23:26:39.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e281f691fad261d-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.662 DEBUG:boto:path=/aktlji-16/obj25 2024-04-19T23:26:39.663 DEBUG:boto:auth_path=/aktlji-16/obj25 2024-04-19T23:26:39.663 DEBUG:boto:path=/aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.663 DEBUG:boto:auth_path=/aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.663 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.663 DEBUG:boto:Path: /aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.663 DEBUG:boto:Data: 2024-04-19T23:26:39.664 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.664 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.664 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.664 DEBUG:boto:Params: {} 2024-04-19T23:26:39.664 DEBUG:boto:Token: None 2024-04-19T23:26:39.664 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.664 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wUInX1rIY1X4k/gjuL7c9jB+HTc= 2024-04-19T23:26:39.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wUInX1rIY1X4k/gjuL7c9jB+HTc=', 'Content-Length': '0'} 2024-04-19T23:26:39.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3f1b4deb37031d2-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.667 DEBUG:boto:path=/aktlji-16/obj25 2024-04-19T23:26:39.667 DEBUG:boto:auth_path=/aktlji-16/obj25 2024-04-19T23:26:39.667 DEBUG:boto:path=/aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.667 DEBUG:boto:auth_path=/aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.667 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.667 DEBUG:boto:Path: /aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.667 DEBUG:boto:Data: 2024-04-19T23:26:39.667 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.667 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.668 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.668 DEBUG:boto:Params: {} 2024-04-19T23:26:39.668 DEBUG:boto:Token: None 2024-04-19T23:26:39.668 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.668 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wUInX1rIY1X4k/gjuL7c9jB+HTc= 2024-04-19T23:26:39.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wUInX1rIY1X4k/gjuL7c9jB+HTc=', 'Content-Length': '0'} 2024-04-19T23:26:39.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c1a0fcfd5a156c8-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj25 2024-04-19T23:26:39.670 DEBUG:boto:path=/aktlji-16/obj25 2024-04-19T23:26:39.671 DEBUG:boto:auth_path=/aktlji-16/obj25 2024-04-19T23:26:39.671 DEBUG:boto:path=/aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.671 DEBUG:boto:auth_path=/aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.671 DEBUG:boto:Method: GET 2024-04-19T23:26:39.671 DEBUG:boto:Path: /aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.671 DEBUG:boto:Data: 2024-04-19T23:26:39.671 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.671 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.671 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.671 DEBUG:boto:Params: {} 2024-04-19T23:26:39.671 DEBUG:boto:Token: None 2024-04-19T23:26:39.672 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.672 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo= 2024-04-19T23:26:39.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo=', 'Content-Length': '0'} 2024-04-19T23:26:39.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058873134c543b9b9-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.674 DEBUG:boto:path=/aktlji-16/obj25 2024-04-19T23:26:39.674 DEBUG:boto:auth_path=/aktlji-16/obj25 2024-04-19T23:26:39.674 DEBUG:boto:path=/aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.674 DEBUG:boto:auth_path=/aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.674 DEBUG:boto:Method: GET 2024-04-19T23:26:39.674 DEBUG:boto:Path: /aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.674 DEBUG:boto:Data: 2024-04-19T23:26:39.674 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.675 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.675 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.675 DEBUG:boto:Params: {} 2024-04-19T23:26:39.675 DEBUG:boto:Token: None 2024-04-19T23:26:39.675 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj25?versionId=null 2024-04-19T23:26:39.675 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo= 2024-04-19T23:26:39.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo=', 'Content-Length': '0'} 2024-04-19T23:26:39.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000574c57e2cd0bf0c2-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.677 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.678 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.678 DEBUG:boto:path=/aktlji-16/?prefix=obj25 2024-04-19T23:26:39.678 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj25 2024-04-19T23:26:39.678 DEBUG:boto:Method: GET 2024-04-19T23:26:39.678 DEBUG:boto:Path: /aktlji-16/?prefix=obj25 2024-04-19T23:26:39.678 DEBUG:boto:Data: 2024-04-19T23:26:39.678 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.678 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.678 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.678 DEBUG:boto:Params: {} 2024-04-19T23:26:39.679 DEBUG:boto:Token: None 2024-04-19T23:26:39.679 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.679 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eed11fc20bc7e65-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.681 DEBUG:boto:b'aktlji-16obj251000falseobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.682 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.682 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.682 DEBUG:boto:path=/aktlji-16/?prefix=obj25 2024-04-19T23:26:39.682 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj25 2024-04-19T23:26:39.682 DEBUG:boto:Method: GET 2024-04-19T23:26:39.682 DEBUG:boto:Path: /aktlji-16/?prefix=obj25 2024-04-19T23:26:39.683 DEBUG:boto:Data: 2024-04-19T23:26:39.683 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.683 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.683 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.683 DEBUG:boto:Params: {} 2024-04-19T23:26:39.683 DEBUG:boto:Token: None 2024-04-19T23:26:39.683 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.683 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0c49b9f347b220f-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.686 DEBUG:boto:b'aktlji-16obj251000falseobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.686 DEBUG:boto:path=/aktlji-16/obj25 2024-04-19T23:26:39.686 DEBUG:boto:auth_path=/aktlji-16/obj25 2024-04-19T23:26:39.687 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.687 DEBUG:boto:Path: /aktlji-16/obj25 2024-04-19T23:26:39.687 DEBUG:boto:Data: 2024-04-19T23:26:39.687 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.687 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.687 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.687 DEBUG:boto:Params: {} 2024-04-19T23:26:39.687 DEBUG:boto:Token: None 2024-04-19T23:26:39.687 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj25 2024-04-19T23:26:39.688 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Goy24wK2sREjHvDks4fofHAbFGI= 2024-04-19T23:26:39.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Goy24wK2sREjHvDks4fofHAbFGI=', 'Content-Length': '0'} 2024-04-19T23:26:39.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036e6a42d64af7943-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.690 DEBUG:boto:path=/aktlji-16/obj25 2024-04-19T23:26:39.690 DEBUG:boto:auth_path=/aktlji-16/obj25 2024-04-19T23:26:39.690 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.690 DEBUG:boto:Path: /aktlji-16/obj25 2024-04-19T23:26:39.690 DEBUG:boto:Data: 2024-04-19T23:26:39.690 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.690 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.690 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.691 DEBUG:boto:Params: {} 2024-04-19T23:26:39.691 DEBUG:boto:Token: None 2024-04-19T23:26:39.691 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj25 2024-04-19T23:26:39.691 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Goy24wK2sREjHvDks4fofHAbFGI= 2024-04-19T23:26:39.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Goy24wK2sREjHvDks4fofHAbFGI=', 'Content-Length': '0'} 2024-04-19T23:26:39.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d92534ae4eb9f52a-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj25 2024-04-19T23:26:39.693 DEBUG:boto:path=/aktlji-16/obj25 2024-04-19T23:26:39.693 DEBUG:boto:auth_path=/aktlji-16/obj25 2024-04-19T23:26:39.694 DEBUG:boto:Method: GET 2024-04-19T23:26:39.694 DEBUG:boto:Path: /aktlji-16/obj25 2024-04-19T23:26:39.694 DEBUG:boto:Data: 2024-04-19T23:26:39.694 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.694 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.694 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.694 DEBUG:boto:Params: {} 2024-04-19T23:26:39.694 DEBUG:boto:Token: None 2024-04-19T23:26:39.695 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj25 2024-04-19T23:26:39.695 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Jr7pfIjN26yWqTnT/k2plX2mLc= 2024-04-19T23:26:39.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Jr7pfIjN26yWqTnT/k2plX2mLc=', 'Content-Length': '0'} 2024-04-19T23:26:39.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007632d82d2f538723-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.697 DEBUG:boto:path=/aktlji-16/obj25 2024-04-19T23:26:39.697 DEBUG:boto:auth_path=/aktlji-16/obj25 2024-04-19T23:26:39.697 DEBUG:boto:Method: GET 2024-04-19T23:26:39.697 DEBUG:boto:Path: /aktlji-16/obj25 2024-04-19T23:26:39.697 DEBUG:boto:Data: 2024-04-19T23:26:39.697 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.698 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.698 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.698 DEBUG:boto:Params: {} 2024-04-19T23:26:39.698 DEBUG:boto:Token: None 2024-04-19T23:26:39.698 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj25 2024-04-19T23:26:39.698 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Jr7pfIjN26yWqTnT/k2plX2mLc= 2024-04-19T23:26:39.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Jr7pfIjN26yWqTnT/k2plX2mLc=', 'Content-Length': '0'} 2024-04-19T23:26:39.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077456c84009aacbd-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.700 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.701 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.701 DEBUG:boto:path=/aktlji-16/?prefix=obj25 2024-04-19T23:26:39.701 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj25 2024-04-19T23:26:39.701 DEBUG:boto:Method: GET 2024-04-19T23:26:39.701 DEBUG:boto:Path: /aktlji-16/?prefix=obj25 2024-04-19T23:26:39.701 DEBUG:boto:Data: 2024-04-19T23:26:39.701 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.701 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.701 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.701 DEBUG:boto:Params: {} 2024-04-19T23:26:39.701 DEBUG:boto:Token: None 2024-04-19T23:26:39.702 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.702 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4471a28bef3f515-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.704 DEBUG:boto:b'aktlji-16obj251000falseobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.705 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.705 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.705 DEBUG:boto:path=/aktlji-16/?prefix=obj25 2024-04-19T23:26:39.705 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj25 2024-04-19T23:26:39.706 DEBUG:boto:Method: GET 2024-04-19T23:26:39.706 DEBUG:boto:Path: /aktlji-16/?prefix=obj25 2024-04-19T23:26:39.706 DEBUG:boto:Data: 2024-04-19T23:26:39.706 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.706 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.706 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.706 DEBUG:boto:Params: {} 2024-04-19T23:26:39.706 DEBUG:boto:Token: None 2024-04-19T23:26:39.706 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.706 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b6bd07a78d94046-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.709 DEBUG:boto:b'aktlji-16obj251000falseobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj26 2024-04-19T23:26:39.709 DEBUG:boto:path=/aktlji-16/obj26 2024-04-19T23:26:39.710 DEBUG:boto:auth_path=/aktlji-16/obj26 2024-04-19T23:26:39.710 DEBUG:boto:path=/aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.710 DEBUG:boto:auth_path=/aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.710 DEBUG:boto:Method: GET 2024-04-19T23:26:39.710 DEBUG:boto:Path: /aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.710 DEBUG:boto:Data: 2024-04-19T23:26:39.710 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.710 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.710 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.710 DEBUG:boto:Params: {} 2024-04-19T23:26:39.710 DEBUG:boto:Token: None 2024-04-19T23:26:39.711 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.711 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U= 2024-04-19T23:26:39.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U=', 'Content-Length': '0'} 2024-04-19T23:26:39.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa89023607c018ba-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.713 DEBUG:boto:path=/aktlji-16/obj26 2024-04-19T23:26:39.713 DEBUG:boto:auth_path=/aktlji-16/obj26 2024-04-19T23:26:39.713 DEBUG:boto:path=/aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.713 DEBUG:boto:auth_path=/aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.713 DEBUG:boto:Method: GET 2024-04-19T23:26:39.713 DEBUG:boto:Path: /aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.714 DEBUG:boto:Data: 2024-04-19T23:26:39.714 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.714 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.714 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.714 DEBUG:boto:Params: {} 2024-04-19T23:26:39.714 DEBUG:boto:Token: None 2024-04-19T23:26:39.714 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.714 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U= 2024-04-19T23:26:39.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U=', 'Content-Length': '0'} 2024-04-19T23:26:39.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bc0f0b8c0f5a530-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.716 DEBUG:boto:path=/aktlji-16/obj26 2024-04-19T23:26:39.716 DEBUG:boto:auth_path=/aktlji-16/obj26 2024-04-19T23:26:39.716 DEBUG:boto:path=/aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.717 DEBUG:boto:auth_path=/aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.717 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.717 DEBUG:boto:Path: /aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.717 DEBUG:boto:Data: 2024-04-19T23:26:39.717 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.717 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.717 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.717 DEBUG:boto:Params: {} 2024-04-19T23:26:39.717 DEBUG:boto:Token: None 2024-04-19T23:26:39.717 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.718 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rQYUECDsS/lLIF1MbR1BHq1A9Nc= 2024-04-19T23:26:39.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rQYUECDsS/lLIF1MbR1BHq1A9Nc=', 'Content-Length': '0'} 2024-04-19T23:26:39.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000815f5a76b0dd19ff-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.720 DEBUG:boto:path=/aktlji-16/obj26 2024-04-19T23:26:39.720 DEBUG:boto:auth_path=/aktlji-16/obj26 2024-04-19T23:26:39.720 DEBUG:boto:path=/aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.720 DEBUG:boto:auth_path=/aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.720 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.720 DEBUG:boto:Path: /aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.720 DEBUG:boto:Data: 2024-04-19T23:26:39.720 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.720 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.720 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.720 DEBUG:boto:Params: {} 2024-04-19T23:26:39.721 DEBUG:boto:Token: None 2024-04-19T23:26:39.721 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.721 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rQYUECDsS/lLIF1MbR1BHq1A9Nc= 2024-04-19T23:26:39.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rQYUECDsS/lLIF1MbR1BHq1A9Nc=', 'Content-Length': '0'} 2024-04-19T23:26:39.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010fc788b1aaa4c40-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj26 2024-04-19T23:26:39.723 DEBUG:boto:path=/aktlji-16/obj26 2024-04-19T23:26:39.723 DEBUG:boto:auth_path=/aktlji-16/obj26 2024-04-19T23:26:39.723 DEBUG:boto:path=/aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.723 DEBUG:boto:auth_path=/aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.723 DEBUG:boto:Method: GET 2024-04-19T23:26:39.724 DEBUG:boto:Path: /aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.724 DEBUG:boto:Data: 2024-04-19T23:26:39.724 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.724 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.724 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.724 DEBUG:boto:Params: {} 2024-04-19T23:26:39.724 DEBUG:boto:Token: None 2024-04-19T23:26:39.724 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.724 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U= 2024-04-19T23:26:39.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U=', 'Content-Length': '0'} 2024-04-19T23:26:39.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000264d00f379bf8b78-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.774 DEBUG:boto:path=/aktlji-16/obj26 2024-04-19T23:26:39.774 DEBUG:boto:auth_path=/aktlji-16/obj26 2024-04-19T23:26:39.775 DEBUG:boto:path=/aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.775 DEBUG:boto:auth_path=/aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.775 DEBUG:boto:Method: GET 2024-04-19T23:26:39.775 DEBUG:boto:Path: /aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.775 DEBUG:boto:Data: 2024-04-19T23:26:39.775 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.775 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.775 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.775 DEBUG:boto:Params: {} 2024-04-19T23:26:39.775 DEBUG:boto:Token: None 2024-04-19T23:26:39.776 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj26?versionId=null 2024-04-19T23:26:39.776 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U= 2024-04-19T23:26:39.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U=', 'Content-Length': '0'} 2024-04-19T23:26:39.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000356deee4b306f74c-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.778 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.778 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.778 DEBUG:boto:path=/aktlji-16/?prefix=obj26 2024-04-19T23:26:39.778 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj26 2024-04-19T23:26:39.778 DEBUG:boto:Method: GET 2024-04-19T23:26:39.778 DEBUG:boto:Path: /aktlji-16/?prefix=obj26 2024-04-19T23:26:39.779 DEBUG:boto:Data: 2024-04-19T23:26:39.779 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.779 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.779 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.779 DEBUG:boto:Params: {} 2024-04-19T23:26:39.779 DEBUG:boto:Token: None 2024-04-19T23:26:39.779 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.779 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000603e9701595ed440-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.782 DEBUG:boto:b'aktlji-16obj261000falseobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.783 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.783 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.783 DEBUG:boto:path=/aktlji-16/?prefix=obj26 2024-04-19T23:26:39.783 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj26 2024-04-19T23:26:39.783 DEBUG:boto:Method: GET 2024-04-19T23:26:39.783 DEBUG:boto:Path: /aktlji-16/?prefix=obj26 2024-04-19T23:26:39.783 DEBUG:boto:Data: 2024-04-19T23:26:39.783 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.783 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.783 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.784 DEBUG:boto:Params: {} 2024-04-19T23:26:39.784 DEBUG:boto:Token: None 2024-04-19T23:26:39.784 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aa3d5334975bad5-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.787 DEBUG:boto:b'aktlji-16obj261000falseobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.787 DEBUG:boto:path=/aktlji-16/obj26 2024-04-19T23:26:39.787 DEBUG:boto:auth_path=/aktlji-16/obj26 2024-04-19T23:26:39.787 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.787 DEBUG:boto:Path: /aktlji-16/obj26 2024-04-19T23:26:39.787 DEBUG:boto:Data: 2024-04-19T23:26:39.787 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.788 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.788 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.788 DEBUG:boto:Params: {} 2024-04-19T23:26:39.788 DEBUG:boto:Token: None 2024-04-19T23:26:39.788 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj26 2024-04-19T23:26:39.788 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Trq6kWHbkyFYZJQFK7T9s4ly3LI= 2024-04-19T23:26:39.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Trq6kWHbkyFYZJQFK7T9s4ly3LI=', 'Content-Length': '0'} 2024-04-19T23:26:39.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070207abc08336999-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.790 DEBUG:boto:path=/aktlji-16/obj26 2024-04-19T23:26:39.790 DEBUG:boto:auth_path=/aktlji-16/obj26 2024-04-19T23:26:39.790 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.790 DEBUG:boto:Path: /aktlji-16/obj26 2024-04-19T23:26:39.790 DEBUG:boto:Data: 2024-04-19T23:26:39.791 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.791 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.791 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.791 DEBUG:boto:Params: {} 2024-04-19T23:26:39.791 DEBUG:boto:Token: None 2024-04-19T23:26:39.791 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj26 2024-04-19T23:26:39.791 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Trq6kWHbkyFYZJQFK7T9s4ly3LI= 2024-04-19T23:26:39.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Trq6kWHbkyFYZJQFK7T9s4ly3LI=', 'Content-Length': '0'} 2024-04-19T23:26:39.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c571bdba46fae41-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj26 2024-04-19T23:26:39.793 DEBUG:boto:path=/aktlji-16/obj26 2024-04-19T23:26:39.793 DEBUG:boto:auth_path=/aktlji-16/obj26 2024-04-19T23:26:39.793 DEBUG:boto:Method: GET 2024-04-19T23:26:39.793 DEBUG:boto:Path: /aktlji-16/obj26 2024-04-19T23:26:39.794 DEBUG:boto:Data: 2024-04-19T23:26:39.794 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.794 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.794 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.794 DEBUG:boto:Params: {} 2024-04-19T23:26:39.794 DEBUG:boto:Token: None 2024-04-19T23:26:39.794 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj26 2024-04-19T23:26:39.794 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:B+15N8xHoAErXKIVmyjXDq/w5As= 2024-04-19T23:26:39.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B+15N8xHoAErXKIVmyjXDq/w5As=', 'Content-Length': '0'} 2024-04-19T23:26:39.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000405253a3b89a2055-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.797 DEBUG:boto:path=/aktlji-16/obj26 2024-04-19T23:26:39.797 DEBUG:boto:auth_path=/aktlji-16/obj26 2024-04-19T23:26:39.797 DEBUG:boto:Method: GET 2024-04-19T23:26:39.797 DEBUG:boto:Path: /aktlji-16/obj26 2024-04-19T23:26:39.797 DEBUG:boto:Data: 2024-04-19T23:26:39.797 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.797 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.797 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.797 DEBUG:boto:Params: {} 2024-04-19T23:26:39.797 DEBUG:boto:Token: None 2024-04-19T23:26:39.798 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj26 2024-04-19T23:26:39.798 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:B+15N8xHoAErXKIVmyjXDq/w5As= 2024-04-19T23:26:39.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B+15N8xHoAErXKIVmyjXDq/w5As=', 'Content-Length': '0'} 2024-04-19T23:26:39.799 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b89a6fd0e3df0ff5-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.800 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.800 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.800 DEBUG:boto:path=/aktlji-16/?prefix=obj26 2024-04-19T23:26:39.800 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj26 2024-04-19T23:26:39.800 DEBUG:boto:Method: GET 2024-04-19T23:26:39.800 DEBUG:boto:Path: /aktlji-16/?prefix=obj26 2024-04-19T23:26:39.800 DEBUG:boto:Data: 2024-04-19T23:26:39.800 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.800 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.801 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.801 DEBUG:boto:Params: {} 2024-04-19T23:26:39.801 DEBUG:boto:Token: None 2024-04-19T23:26:39.801 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.801 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075a049ea862a1e2e-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.804 DEBUG:boto:b'aktlji-16obj261000falseobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.804 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.805 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.805 DEBUG:boto:path=/aktlji-16/?prefix=obj26 2024-04-19T23:26:39.805 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj26 2024-04-19T23:26:39.805 DEBUG:boto:Method: GET 2024-04-19T23:26:39.805 DEBUG:boto:Path: /aktlji-16/?prefix=obj26 2024-04-19T23:26:39.805 DEBUG:boto:Data: 2024-04-19T23:26:39.805 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.805 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.806 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.806 DEBUG:boto:Params: {} 2024-04-19T23:26:39.806 DEBUG:boto:Token: None 2024-04-19T23:26:39.806 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.806 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076874c827f44a70b-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.809 DEBUG:boto:b'aktlji-16obj261000falseobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj27 2024-04-19T23:26:39.809 DEBUG:boto:path=/aktlji-16/obj27 2024-04-19T23:26:39.809 DEBUG:boto:auth_path=/aktlji-16/obj27 2024-04-19T23:26:39.809 DEBUG:boto:path=/aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.809 DEBUG:boto:auth_path=/aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.810 DEBUG:boto:Method: GET 2024-04-19T23:26:39.810 DEBUG:boto:Path: /aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.810 DEBUG:boto:Data: 2024-04-19T23:26:39.810 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.810 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.810 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.810 DEBUG:boto:Params: {} 2024-04-19T23:26:39.810 DEBUG:boto:Token: None 2024-04-19T23:26:39.810 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.810 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk= 2024-04-19T23:26:39.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk=', 'Content-Length': '0'} 2024-04-19T23:26:39.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ceb5d21a4b15afae-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.854 DEBUG:boto:path=/aktlji-16/obj27 2024-04-19T23:26:39.855 DEBUG:boto:auth_path=/aktlji-16/obj27 2024-04-19T23:26:39.855 DEBUG:boto:path=/aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.855 DEBUG:boto:auth_path=/aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.855 DEBUG:boto:Method: GET 2024-04-19T23:26:39.855 DEBUG:boto:Path: /aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.855 DEBUG:boto:Data: 2024-04-19T23:26:39.855 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.855 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.855 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.855 DEBUG:boto:Params: {} 2024-04-19T23:26:39.856 DEBUG:boto:Token: None 2024-04-19T23:26:39.856 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.856 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk= 2024-04-19T23:26:39.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk=', 'Content-Length': '0'} 2024-04-19T23:26:39.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009da1581ca0bcbc9d-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.858 DEBUG:boto:path=/aktlji-16/obj27 2024-04-19T23:26:39.859 DEBUG:boto:auth_path=/aktlji-16/obj27 2024-04-19T23:26:39.859 DEBUG:boto:path=/aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.859 DEBUG:boto:auth_path=/aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.859 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.859 DEBUG:boto:Path: /aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.859 DEBUG:boto:Data: 2024-04-19T23:26:39.859 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.859 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.859 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.859 DEBUG:boto:Params: {} 2024-04-19T23:26:39.859 DEBUG:boto:Token: None 2024-04-19T23:26:39.860 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.860 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1rzJsO7eXgE4NFwRisbZdejV5QU= 2024-04-19T23:26:39.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1rzJsO7eXgE4NFwRisbZdejV5QU=', 'Content-Length': '0'} 2024-04-19T23:26:39.862 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1b97ec40b805844-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.862 DEBUG:boto:path=/aktlji-16/obj27 2024-04-19T23:26:39.862 DEBUG:boto:auth_path=/aktlji-16/obj27 2024-04-19T23:26:39.862 DEBUG:boto:path=/aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.862 DEBUG:boto:auth_path=/aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.862 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.862 DEBUG:boto:Path: /aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.862 DEBUG:boto:Data: 2024-04-19T23:26:39.863 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.863 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.863 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.863 DEBUG:boto:Params: {} 2024-04-19T23:26:39.863 DEBUG:boto:Token: None 2024-04-19T23:26:39.863 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.863 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1rzJsO7eXgE4NFwRisbZdejV5QU= 2024-04-19T23:26:39.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1rzJsO7eXgE4NFwRisbZdejV5QU=', 'Content-Length': '0'} 2024-04-19T23:26:39.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3691f8f878d4961-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj27 2024-04-19T23:26:39.865 DEBUG:boto:path=/aktlji-16/obj27 2024-04-19T23:26:39.865 DEBUG:boto:auth_path=/aktlji-16/obj27 2024-04-19T23:26:39.866 DEBUG:boto:path=/aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.866 DEBUG:boto:auth_path=/aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.866 DEBUG:boto:Method: GET 2024-04-19T23:26:39.866 DEBUG:boto:Path: /aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.866 DEBUG:boto:Data: 2024-04-19T23:26:39.866 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.866 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.866 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.866 DEBUG:boto:Params: {} 2024-04-19T23:26:39.866 DEBUG:boto:Token: None 2024-04-19T23:26:39.867 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.867 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk= 2024-04-19T23:26:39.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk=', 'Content-Length': '0'} 2024-04-19T23:26:39.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000054b7e5ca97c2351-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.869 DEBUG:boto:path=/aktlji-16/obj27 2024-04-19T23:26:39.869 DEBUG:boto:auth_path=/aktlji-16/obj27 2024-04-19T23:26:39.869 DEBUG:boto:path=/aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.869 DEBUG:boto:auth_path=/aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.869 DEBUG:boto:Method: GET 2024-04-19T23:26:39.869 DEBUG:boto:Path: /aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.869 DEBUG:boto:Data: 2024-04-19T23:26:39.869 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.869 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.870 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.870 DEBUG:boto:Params: {} 2024-04-19T23:26:39.870 DEBUG:boto:Token: None 2024-04-19T23:26:39.870 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj27?versionId=null 2024-04-19T23:26:39.870 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk= 2024-04-19T23:26:39.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk=', 'Content-Length': '0'} 2024-04-19T23:26:39.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b06283d9e5f56d21-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.872 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.872 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.872 DEBUG:boto:path=/aktlji-16/?prefix=obj27 2024-04-19T23:26:39.873 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj27 2024-04-19T23:26:39.873 DEBUG:boto:Method: GET 2024-04-19T23:26:39.873 DEBUG:boto:Path: /aktlji-16/?prefix=obj27 2024-04-19T23:26:39.873 DEBUG:boto:Data: 2024-04-19T23:26:39.873 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.873 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.873 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.873 DEBUG:boto:Params: {} 2024-04-19T23:26:39.873 DEBUG:boto:Token: None 2024-04-19T23:26:39.873 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.873 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c46ef8a59156bc7-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.876 DEBUG:boto:b'aktlji-16obj271000falseobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.877 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.877 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.877 DEBUG:boto:path=/aktlji-16/?prefix=obj27 2024-04-19T23:26:39.877 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj27 2024-04-19T23:26:39.877 DEBUG:boto:Method: GET 2024-04-19T23:26:39.877 DEBUG:boto:Path: /aktlji-16/?prefix=obj27 2024-04-19T23:26:39.877 DEBUG:boto:Data: 2024-04-19T23:26:39.877 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.877 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.877 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.878 DEBUG:boto:Params: {} 2024-04-19T23:26:39.878 DEBUG:boto:Token: None 2024-04-19T23:26:39.878 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.878 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7c56b510526f7c-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.880 DEBUG:boto:b'aktlji-16obj271000falseobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.881 DEBUG:boto:path=/aktlji-16/obj27 2024-04-19T23:26:39.881 DEBUG:boto:auth_path=/aktlji-16/obj27 2024-04-19T23:26:39.881 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.881 DEBUG:boto:Path: /aktlji-16/obj27 2024-04-19T23:26:39.881 DEBUG:boto:Data: 2024-04-19T23:26:39.881 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.882 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.882 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.882 DEBUG:boto:Params: {} 2024-04-19T23:26:39.882 DEBUG:boto:Token: None 2024-04-19T23:26:39.883 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj27 2024-04-19T23:26:39.883 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KckFcHUeHICmaSOx9SH7U9zjDCo= 2024-04-19T23:26:39.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KckFcHUeHICmaSOx9SH7U9zjDCo=', 'Content-Length': '0'} 2024-04-19T23:26:39.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014c3ca2cb10dcef5-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.885 DEBUG:boto:path=/aktlji-16/obj27 2024-04-19T23:26:39.885 DEBUG:boto:auth_path=/aktlji-16/obj27 2024-04-19T23:26:39.885 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.885 DEBUG:boto:Path: /aktlji-16/obj27 2024-04-19T23:26:39.885 DEBUG:boto:Data: 2024-04-19T23:26:39.885 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.885 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.885 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.885 DEBUG:boto:Params: {} 2024-04-19T23:26:39.886 DEBUG:boto:Token: None 2024-04-19T23:26:39.886 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj27 2024-04-19T23:26:39.886 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KckFcHUeHICmaSOx9SH7U9zjDCo= 2024-04-19T23:26:39.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KckFcHUeHICmaSOx9SH7U9zjDCo=', 'Content-Length': '0'} 2024-04-19T23:26:39.888 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089bde4e1fd062828-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj27 2024-04-19T23:26:39.889 DEBUG:boto:path=/aktlji-16/obj27 2024-04-19T23:26:39.889 DEBUG:boto:auth_path=/aktlji-16/obj27 2024-04-19T23:26:39.889 DEBUG:boto:Method: GET 2024-04-19T23:26:39.889 DEBUG:boto:Path: /aktlji-16/obj27 2024-04-19T23:26:39.889 DEBUG:boto:Data: 2024-04-19T23:26:39.889 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.889 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.889 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.889 DEBUG:boto:Params: {} 2024-04-19T23:26:39.889 DEBUG:boto:Token: None 2024-04-19T23:26:39.890 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj27 2024-04-19T23:26:39.890 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nW0IbLU3TTagnEGGtPlA7mMDJSw= 2024-04-19T23:26:39.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nW0IbLU3TTagnEGGtPlA7mMDJSw=', 'Content-Length': '0'} 2024-04-19T23:26:39.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e71292b9cc771546-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.892 DEBUG:boto:path=/aktlji-16/obj27 2024-04-19T23:26:39.892 DEBUG:boto:auth_path=/aktlji-16/obj27 2024-04-19T23:26:39.892 DEBUG:boto:Method: GET 2024-04-19T23:26:39.892 DEBUG:boto:Path: /aktlji-16/obj27 2024-04-19T23:26:39.892 DEBUG:boto:Data: 2024-04-19T23:26:39.892 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.892 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.892 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.892 DEBUG:boto:Params: {} 2024-04-19T23:26:39.893 DEBUG:boto:Token: None 2024-04-19T23:26:39.893 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj27 2024-04-19T23:26:39.893 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nW0IbLU3TTagnEGGtPlA7mMDJSw= 2024-04-19T23:26:39.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nW0IbLU3TTagnEGGtPlA7mMDJSw=', 'Content-Length': '0'} 2024-04-19T23:26:39.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb05122f47c8af51-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.895 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.895 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.895 DEBUG:boto:path=/aktlji-16/?prefix=obj27 2024-04-19T23:26:39.895 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj27 2024-04-19T23:26:39.895 DEBUG:boto:Method: GET 2024-04-19T23:26:39.896 DEBUG:boto:Path: /aktlji-16/?prefix=obj27 2024-04-19T23:26:39.896 DEBUG:boto:Data: 2024-04-19T23:26:39.896 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.896 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.896 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.896 DEBUG:boto:Params: {} 2024-04-19T23:26:39.896 DEBUG:boto:Token: None 2024-04-19T23:26:39.896 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.896 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647b15f707d36db5-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.900 DEBUG:boto:b'aktlji-16obj271000falseobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.901 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.901 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.901 DEBUG:boto:path=/aktlji-16/?prefix=obj27 2024-04-19T23:26:39.901 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj27 2024-04-19T23:26:39.902 DEBUG:boto:Method: GET 2024-04-19T23:26:39.902 DEBUG:boto:Path: /aktlji-16/?prefix=obj27 2024-04-19T23:26:39.902 DEBUG:boto:Data: 2024-04-19T23:26:39.902 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.902 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.902 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.903 DEBUG:boto:Params: {} 2024-04-19T23:26:39.903 DEBUG:boto:Token: None 2024-04-19T23:26:39.903 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.903 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f0f4804ac46767d-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.954 DEBUG:boto:b'aktlji-16obj271000falseobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj28 2024-04-19T23:26:39.955 DEBUG:boto:path=/aktlji-16/obj28 2024-04-19T23:26:39.956 DEBUG:boto:auth_path=/aktlji-16/obj28 2024-04-19T23:26:39.956 DEBUG:boto:path=/aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.956 DEBUG:boto:auth_path=/aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.956 DEBUG:boto:Method: GET 2024-04-19T23:26:39.956 DEBUG:boto:Path: /aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.957 DEBUG:boto:Data: 2024-04-19T23:26:39.957 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.957 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.957 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.957 DEBUG:boto:Params: {} 2024-04-19T23:26:39.958 DEBUG:boto:Token: None 2024-04-19T23:26:39.958 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.958 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA= 2024-04-19T23:26:39.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA=', 'Content-Length': '0'} 2024-04-19T23:26:39.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000958c8655a35efa27-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.961 DEBUG:boto:path=/aktlji-16/obj28 2024-04-19T23:26:39.962 DEBUG:boto:auth_path=/aktlji-16/obj28 2024-04-19T23:26:39.962 DEBUG:boto:path=/aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.962 DEBUG:boto:auth_path=/aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.962 DEBUG:boto:Method: GET 2024-04-19T23:26:39.963 DEBUG:boto:Path: /aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.963 DEBUG:boto:Data: 2024-04-19T23:26:39.963 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.963 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.963 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.963 DEBUG:boto:Params: {} 2024-04-19T23:26:39.964 DEBUG:boto:Token: None 2024-04-19T23:26:39.964 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.964 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA= 2024-04-19T23:26:39.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA=', 'Content-Length': '0'} 2024-04-19T23:26:39.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077ace566115f447b-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.966 DEBUG:boto:path=/aktlji-16/obj28 2024-04-19T23:26:39.966 DEBUG:boto:auth_path=/aktlji-16/obj28 2024-04-19T23:26:39.967 DEBUG:boto:path=/aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.967 DEBUG:boto:auth_path=/aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.967 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.967 DEBUG:boto:Path: /aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.967 DEBUG:boto:Data: 2024-04-19T23:26:39.967 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.967 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.967 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.967 DEBUG:boto:Params: {} 2024-04-19T23:26:39.967 DEBUG:boto:Token: None 2024-04-19T23:26:39.968 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.968 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IB2EKtkPVLxkCY/OZ0jWyNXWN3A= 2024-04-19T23:26:39.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IB2EKtkPVLxkCY/OZ0jWyNXWN3A=', 'Content-Length': '0'} 2024-04-19T23:26:39.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091170bad3ba58905-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.970 DEBUG:boto:path=/aktlji-16/obj28 2024-04-19T23:26:39.970 DEBUG:boto:auth_path=/aktlji-16/obj28 2024-04-19T23:26:39.970 DEBUG:boto:path=/aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.970 DEBUG:boto:auth_path=/aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.970 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.970 DEBUG:boto:Path: /aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.970 DEBUG:boto:Data: 2024-04-19T23:26:39.971 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.971 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.971 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.971 DEBUG:boto:Params: {} 2024-04-19T23:26:39.971 DEBUG:boto:Token: None 2024-04-19T23:26:39.971 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.971 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IB2EKtkPVLxkCY/OZ0jWyNXWN3A= 2024-04-19T23:26:39.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IB2EKtkPVLxkCY/OZ0jWyNXWN3A=', 'Content-Length': '0'} 2024-04-19T23:26:39.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cce45ed216980d2-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj28 2024-04-19T23:26:39.974 DEBUG:boto:path=/aktlji-16/obj28 2024-04-19T23:26:39.974 DEBUG:boto:auth_path=/aktlji-16/obj28 2024-04-19T23:26:39.974 DEBUG:boto:path=/aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.974 DEBUG:boto:auth_path=/aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.974 DEBUG:boto:Method: GET 2024-04-19T23:26:39.974 DEBUG:boto:Path: /aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.974 DEBUG:boto:Data: 2024-04-19T23:26:39.975 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.975 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.975 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.975 DEBUG:boto:Params: {} 2024-04-19T23:26:39.975 DEBUG:boto:Token: None 2024-04-19T23:26:39.975 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.975 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA= 2024-04-19T23:26:39.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA=', 'Content-Length': '0'} 2024-04-19T23:26:39.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063edb07884045b13-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.977 DEBUG:boto:path=/aktlji-16/obj28 2024-04-19T23:26:39.977 DEBUG:boto:auth_path=/aktlji-16/obj28 2024-04-19T23:26:39.977 DEBUG:boto:path=/aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.978 DEBUG:boto:auth_path=/aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.978 DEBUG:boto:Method: GET 2024-04-19T23:26:39.978 DEBUG:boto:Path: /aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.978 DEBUG:boto:Data: 2024-04-19T23:26:39.978 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.978 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.978 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.978 DEBUG:boto:Params: {} 2024-04-19T23:26:39.978 DEBUG:boto:Token: None 2024-04-19T23:26:39.978 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj28?versionId=null 2024-04-19T23:26:39.979 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA= 2024-04-19T23:26:39.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA=', 'Content-Length': '0'} 2024-04-19T23:26:39.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081ebb1c02a18f24e-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.981 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.981 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.981 DEBUG:boto:path=/aktlji-16/?prefix=obj28 2024-04-19T23:26:39.981 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj28 2024-04-19T23:26:39.981 DEBUG:boto:Method: GET 2024-04-19T23:26:39.981 DEBUG:boto:Path: /aktlji-16/?prefix=obj28 2024-04-19T23:26:39.982 DEBUG:boto:Data: 2024-04-19T23:26:39.982 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.982 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.982 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.982 DEBUG:boto:Params: {} 2024-04-19T23:26:39.982 DEBUG:boto:Token: None 2024-04-19T23:26:39.982 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.982 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081e1ff8ab11a0e4d-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.985 DEBUG:boto:b'aktlji-16obj281000falseobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.985 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:39.986 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:39.986 DEBUG:boto:path=/aktlji-16/?prefix=obj28 2024-04-19T23:26:39.986 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj28 2024-04-19T23:26:39.986 DEBUG:boto:Method: GET 2024-04-19T23:26:39.986 DEBUG:boto:Path: /aktlji-16/?prefix=obj28 2024-04-19T23:26:39.986 DEBUG:boto:Data: 2024-04-19T23:26:39.986 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.986 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.986 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.986 DEBUG:boto:Params: {} 2024-04-19T23:26:39.987 DEBUG:boto:Token: None 2024-04-19T23:26:39.987 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/ 2024-04-19T23:26:39.987 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-19T23:26:39.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-19T23:26:39.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000950e7f308148f9f4-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.990 DEBUG:boto:b'aktlji-16obj281000falseobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:39.990 DEBUG:boto:path=/aktlji-16/obj28 2024-04-19T23:26:39.990 DEBUG:boto:auth_path=/aktlji-16/obj28 2024-04-19T23:26:39.990 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.990 DEBUG:boto:Path: /aktlji-16/obj28 2024-04-19T23:26:39.990 DEBUG:boto:Data: 2024-04-19T23:26:39.990 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.990 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.991 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.991 DEBUG:boto:Params: {} 2024-04-19T23:26:39.991 DEBUG:boto:Token: None 2024-04-19T23:26:39.991 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj28 2024-04-19T23:26:39.991 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z3bQ5a2MgHif541EjhFnN6Hip2Y= 2024-04-19T23:26:39.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z3bQ5a2MgHif541EjhFnN6Hip2Y=', 'Content-Length': '0'} 2024-04-19T23:26:39.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067023b426af9daa0-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.993 DEBUG:boto:path=/aktlji-16/obj28 2024-04-19T23:26:39.993 DEBUG:boto:auth_path=/aktlji-16/obj28 2024-04-19T23:26:39.993 DEBUG:boto:Method: HEAD 2024-04-19T23:26:39.993 DEBUG:boto:Path: /aktlji-16/obj28 2024-04-19T23:26:39.993 DEBUG:boto:Data: 2024-04-19T23:26:39.993 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.993 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:39.994 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.994 DEBUG:boto:Params: {} 2024-04-19T23:26:39.994 DEBUG:boto:Token: None 2024-04-19T23:26:39.994 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj28 2024-04-19T23:26:39.994 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z3bQ5a2MgHif541EjhFnN6Hip2Y= 2024-04-19T23:26:39.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z3bQ5a2MgHif541EjhFnN6Hip2Y=', 'Content-Length': '0'} 2024-04-19T23:26:39.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba2d8ec9213f2ee8-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj28 2024-04-19T23:26:39.996 DEBUG:boto:path=/aktlji-16/obj28 2024-04-19T23:26:39.996 DEBUG:boto:auth_path=/aktlji-16/obj28 2024-04-19T23:26:39.996 DEBUG:boto:Method: GET 2024-04-19T23:26:39.996 DEBUG:boto:Path: /aktlji-16/obj28 2024-04-19T23:26:39.996 DEBUG:boto:Data: 2024-04-19T23:26:39.996 DEBUG:boto:Headers: {} 2024-04-19T23:26:39.997 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:39.997 DEBUG:boto:Port: 8000 2024-04-19T23:26:39.997 DEBUG:boto:Params: {} 2024-04-19T23:26:39.997 DEBUG:boto:Token: None 2024-04-19T23:26:39.997 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:39 GMT /aktlji-16/obj28 2024-04-19T23:26:39.997 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1CBoBNGbZOQlZ5ITkhAfGOqXwxM= 2024-04-19T23:26:39.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1CBoBNGbZOQlZ5ITkhAfGOqXwxM=', 'Content-Length': '0'} 2024-04-19T23:26:39.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093733b5b16b11c17-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:39.999 DEBUG:boto:path=/aktlji-16/obj28 2024-04-19T23:26:39.999 DEBUG:boto:auth_path=/aktlji-16/obj28 2024-04-19T23:26:39.999 DEBUG:boto:Method: GET 2024-04-19T23:26:39.999 DEBUG:boto:Path: /aktlji-16/obj28 2024-04-19T23:26:39.999 DEBUG:boto:Data: 2024-04-19T23:26:40.000 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.000 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.000 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.000 DEBUG:boto:Params: {} 2024-04-19T23:26:40.000 DEBUG:boto:Token: None 2024-04-19T23:26:40.000 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj28 2024-04-19T23:26:40.000 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Wdeb8IGTFWWbxYQrL2XgENz7JPU= 2024-04-19T23:26:40.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Wdeb8IGTFWWbxYQrL2XgENz7JPU=', 'Content-Length': '0'} 2024-04-19T23:26:40.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000758f0a372d951b5f-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.003 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.003 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.003 DEBUG:boto:path=/aktlji-16/?prefix=obj28 2024-04-19T23:26:40.003 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj28 2024-04-19T23:26:40.003 DEBUG:boto:Method: GET 2024-04-19T23:26:40.003 DEBUG:boto:Path: /aktlji-16/?prefix=obj28 2024-04-19T23:26:40.003 DEBUG:boto:Data: 2024-04-19T23:26:40.003 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.003 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.003 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.003 DEBUG:boto:Params: {} 2024-04-19T23:26:40.004 DEBUG:boto:Token: None 2024-04-19T23:26:40.004 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.004 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f7f29892d03c034-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.054 DEBUG:boto:b'aktlji-16obj281000falseobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.055 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.055 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.055 DEBUG:boto:path=/aktlji-16/?prefix=obj28 2024-04-19T23:26:40.055 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj28 2024-04-19T23:26:40.055 DEBUG:boto:Method: GET 2024-04-19T23:26:40.055 DEBUG:boto:Path: /aktlji-16/?prefix=obj28 2024-04-19T23:26:40.055 DEBUG:boto:Data: 2024-04-19T23:26:40.056 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.056 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.056 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.056 DEBUG:boto:Params: {} 2024-04-19T23:26:40.056 DEBUG:boto:Token: None 2024-04-19T23:26:40.056 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.056 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050021151af770eb8-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.059 DEBUG:boto:b'aktlji-16obj281000falseobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-19T23:26:40.059 DEBUG:boto:path=/aktlji-16/obj29 2024-04-19T23:26:40.059 DEBUG:boto:auth_path=/aktlji-16/obj29 2024-04-19T23:26:40.059 DEBUG:boto:path=/aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.060 DEBUG:boto:auth_path=/aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.060 DEBUG:boto:Method: GET 2024-04-19T23:26:40.060 DEBUG:boto:Path: /aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.060 DEBUG:boto:Data: 2024-04-19T23:26:40.060 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.060 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.060 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.060 DEBUG:boto:Params: {} 2024-04-19T23:26:40.060 DEBUG:boto:Token: None 2024-04-19T23:26:40.061 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.061 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c= 2024-04-19T23:26:40.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c=', 'Content-Length': '0'} 2024-04-19T23:26:40.063 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009aff63fcfdffc8a9-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.063 DEBUG:boto:path=/aktlji-16/obj29 2024-04-19T23:26:40.063 DEBUG:boto:auth_path=/aktlji-16/obj29 2024-04-19T23:26:40.064 DEBUG:boto:path=/aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.064 DEBUG:boto:auth_path=/aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.064 DEBUG:boto:Method: GET 2024-04-19T23:26:40.064 DEBUG:boto:Path: /aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.064 DEBUG:boto:Data: 2024-04-19T23:26:40.064 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.065 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.065 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.065 DEBUG:boto:Params: {} 2024-04-19T23:26:40.065 DEBUG:boto:Token: None 2024-04-19T23:26:40.065 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.065 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c= 2024-04-19T23:26:40.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c=', 'Content-Length': '0'} 2024-04-19T23:26:40.067 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062d46e2b45c9870b-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.067 DEBUG:boto:path=/aktlji-16/obj29 2024-04-19T23:26:40.068 DEBUG:boto:auth_path=/aktlji-16/obj29 2024-04-19T23:26:40.068 DEBUG:boto:path=/aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.068 DEBUG:boto:auth_path=/aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.068 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.068 DEBUG:boto:Path: /aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.068 DEBUG:boto:Data: 2024-04-19T23:26:40.068 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.068 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.068 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.068 DEBUG:boto:Params: {} 2024-04-19T23:26:40.068 DEBUG:boto:Token: None 2024-04-19T23:26:40.069 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.069 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZiCisE2SNZ0UlZQirdJ/6liOffQ= 2024-04-19T23:26:40.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZiCisE2SNZ0UlZQirdJ/6liOffQ=', 'Content-Length': '0'} 2024-04-19T23:26:40.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000664c00c4765e7779-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.071 DEBUG:boto:path=/aktlji-16/obj29 2024-04-19T23:26:40.071 DEBUG:boto:auth_path=/aktlji-16/obj29 2024-04-19T23:26:40.071 DEBUG:boto:path=/aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.071 DEBUG:boto:auth_path=/aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.071 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.071 DEBUG:boto:Path: /aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.071 DEBUG:boto:Data: 2024-04-19T23:26:40.071 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.071 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.071 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.071 DEBUG:boto:Params: {} 2024-04-19T23:26:40.072 DEBUG:boto:Token: None 2024-04-19T23:26:40.072 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.072 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZiCisE2SNZ0UlZQirdJ/6liOffQ= 2024-04-19T23:26:40.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZiCisE2SNZ0UlZQirdJ/6liOffQ=', 'Content-Length': '0'} 2024-04-19T23:26:40.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007af55ba818d57e93-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-19T23:26:40.074 DEBUG:boto:path=/aktlji-16/obj29 2024-04-19T23:26:40.074 DEBUG:boto:auth_path=/aktlji-16/obj29 2024-04-19T23:26:40.074 DEBUG:boto:path=/aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.074 DEBUG:boto:auth_path=/aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.074 DEBUG:boto:Method: GET 2024-04-19T23:26:40.074 DEBUG:boto:Path: /aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.075 DEBUG:boto:Data: 2024-04-19T23:26:40.075 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.075 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.075 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.075 DEBUG:boto:Params: {} 2024-04-19T23:26:40.075 DEBUG:boto:Token: None 2024-04-19T23:26:40.075 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.075 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c= 2024-04-19T23:26:40.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c=', 'Content-Length': '0'} 2024-04-19T23:26:40.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d247bd9efec89524-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.118 DEBUG:boto:path=/aktlji-16/obj29 2024-04-19T23:26:40.119 DEBUG:boto:auth_path=/aktlji-16/obj29 2024-04-19T23:26:40.119 DEBUG:boto:path=/aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.119 DEBUG:boto:auth_path=/aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.119 DEBUG:boto:Method: GET 2024-04-19T23:26:40.119 DEBUG:boto:Path: /aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.119 DEBUG:boto:Data: 2024-04-19T23:26:40.119 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.119 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.119 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.119 DEBUG:boto:Params: {} 2024-04-19T23:26:40.119 DEBUG:boto:Token: None 2024-04-19T23:26:40.120 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj29?versionId=null 2024-04-19T23:26:40.120 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c= 2024-04-19T23:26:40.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c=', 'Content-Length': '0'} 2024-04-19T23:26:40.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f30a1c606d7c8ef-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.122 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.122 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.122 DEBUG:boto:path=/aktlji-16/?prefix=obj29 2024-04-19T23:26:40.122 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj29 2024-04-19T23:26:40.123 DEBUG:boto:Method: GET 2024-04-19T23:26:40.123 DEBUG:boto:Path: /aktlji-16/?prefix=obj29 2024-04-19T23:26:40.123 DEBUG:boto:Data: 2024-04-19T23:26:40.123 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.123 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.123 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.123 DEBUG:boto:Params: {} 2024-04-19T23:26:40.123 DEBUG:boto:Token: None 2024-04-19T23:26:40.124 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.124 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008507e2d7fd5e6e6f-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.126 DEBUG:boto:b'aktlji-16obj291000falseobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.127 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.127 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.127 DEBUG:boto:path=/aktlji-16/?prefix=obj29 2024-04-19T23:26:40.127 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj29 2024-04-19T23:26:40.127 DEBUG:boto:Method: GET 2024-04-19T23:26:40.127 DEBUG:boto:Path: /aktlji-16/?prefix=obj29 2024-04-19T23:26:40.127 DEBUG:boto:Data: 2024-04-19T23:26:40.127 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.127 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.128 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.128 DEBUG:boto:Params: {} 2024-04-19T23:26:40.128 DEBUG:boto:Token: None 2024-04-19T23:26:40.128 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.128 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026d71d27ce1a4f0d-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.131 DEBUG:boto:b'aktlji-16obj291000falseobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.131 DEBUG:boto:path=/aktlji-16/obj29 2024-04-19T23:26:40.131 DEBUG:boto:auth_path=/aktlji-16/obj29 2024-04-19T23:26:40.131 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.131 DEBUG:boto:Path: /aktlji-16/obj29 2024-04-19T23:26:40.131 DEBUG:boto:Data: 2024-04-19T23:26:40.132 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.132 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.132 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.132 DEBUG:boto:Params: {} 2024-04-19T23:26:40.132 DEBUG:boto:Token: None 2024-04-19T23:26:40.132 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj29 2024-04-19T23:26:40.132 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EraQY/yInyykUjTaYn0Sw8ZC+pY= 2024-04-19T23:26:40.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EraQY/yInyykUjTaYn0Sw8ZC+pY=', 'Content-Length': '0'} 2024-04-19T23:26:40.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000992c391367ffe2cb-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.135 DEBUG:boto:path=/aktlji-16/obj29 2024-04-19T23:26:40.135 DEBUG:boto:auth_path=/aktlji-16/obj29 2024-04-19T23:26:40.135 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.135 DEBUG:boto:Path: /aktlji-16/obj29 2024-04-19T23:26:40.135 DEBUG:boto:Data: 2024-04-19T23:26:40.135 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.137 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.137 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.137 DEBUG:boto:Params: {} 2024-04-19T23:26:40.137 DEBUG:boto:Token: None 2024-04-19T23:26:40.137 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj29 2024-04-19T23:26:40.137 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EraQY/yInyykUjTaYn0Sw8ZC+pY= 2024-04-19T23:26:40.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EraQY/yInyykUjTaYn0Sw8ZC+pY=', 'Content-Length': '0'} 2024-04-19T23:26:40.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfdfceee1d990082-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-19T23:26:40.139 DEBUG:boto:path=/aktlji-16/obj29 2024-04-19T23:26:40.140 DEBUG:boto:auth_path=/aktlji-16/obj29 2024-04-19T23:26:40.140 DEBUG:boto:Method: GET 2024-04-19T23:26:40.140 DEBUG:boto:Path: /aktlji-16/obj29 2024-04-19T23:26:40.140 DEBUG:boto:Data: 2024-04-19T23:26:40.140 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.140 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.140 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.140 DEBUG:boto:Params: {} 2024-04-19T23:26:40.140 DEBUG:boto:Token: None 2024-04-19T23:26:40.140 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj29 2024-04-19T23:26:40.140 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WVx1CcDA4XDcitlFhthPwEwadzA= 2024-04-19T23:26:40.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WVx1CcDA4XDcitlFhthPwEwadzA=', 'Content-Length': '0'} 2024-04-19T23:26:40.142 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a40c9035d989abe-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.142 DEBUG:boto:path=/aktlji-16/obj29 2024-04-19T23:26:40.143 DEBUG:boto:auth_path=/aktlji-16/obj29 2024-04-19T23:26:40.143 DEBUG:boto:Method: GET 2024-04-19T23:26:40.143 DEBUG:boto:Path: /aktlji-16/obj29 2024-04-19T23:26:40.143 DEBUG:boto:Data: 2024-04-19T23:26:40.143 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.143 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.143 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.143 DEBUG:boto:Params: {} 2024-04-19T23:26:40.143 DEBUG:boto:Token: None 2024-04-19T23:26:40.143 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj29 2024-04-19T23:26:40.144 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WVx1CcDA4XDcitlFhthPwEwadzA= 2024-04-19T23:26:40.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WVx1CcDA4XDcitlFhthPwEwadzA=', 'Content-Length': '0'} 2024-04-19T23:26:40.145 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000931aa794a977d0db-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.146 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.146 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.146 DEBUG:boto:path=/aktlji-16/?prefix=obj29 2024-04-19T23:26:40.146 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj29 2024-04-19T23:26:40.146 DEBUG:boto:Method: GET 2024-04-19T23:26:40.146 DEBUG:boto:Path: /aktlji-16/?prefix=obj29 2024-04-19T23:26:40.146 DEBUG:boto:Data: 2024-04-19T23:26:40.146 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.146 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.147 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.147 DEBUG:boto:Params: {} 2024-04-19T23:26:40.147 DEBUG:boto:Token: None 2024-04-19T23:26:40.147 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.147 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003104d41e23f1603f-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.190 DEBUG:boto:b'aktlji-16obj291000falseobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.191 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.191 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.191 DEBUG:boto:path=/aktlji-16/?prefix=obj29 2024-04-19T23:26:40.191 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj29 2024-04-19T23:26:40.191 DEBUG:boto:Method: GET 2024-04-19T23:26:40.191 DEBUG:boto:Path: /aktlji-16/?prefix=obj29 2024-04-19T23:26:40.191 DEBUG:boto:Data: 2024-04-19T23:26:40.191 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.191 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.191 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.192 DEBUG:boto:Params: {} 2024-04-19T23:26:40.192 DEBUG:boto:Token: None 2024-04-19T23:26:40.192 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.192 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddd951a8559e4841-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.195 DEBUG:boto:b'aktlji-16obj291000falseobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-19T23:26:40.195 DEBUG:boto:path=/aktlji-16/obj3 2024-04-19T23:26:40.195 DEBUG:boto:auth_path=/aktlji-16/obj3 2024-04-19T23:26:40.195 DEBUG:boto:path=/aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.195 DEBUG:boto:auth_path=/aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.196 DEBUG:boto:Method: GET 2024-04-19T23:26:40.196 DEBUG:boto:Path: /aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.196 DEBUG:boto:Data: 2024-04-19T23:26:40.196 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.196 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.196 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.196 DEBUG:boto:Params: {} 2024-04-19T23:26:40.196 DEBUG:boto:Token: None 2024-04-19T23:26:40.196 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.196 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss= 2024-04-19T23:26:40.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss=', 'Content-Length': '0'} 2024-04-19T23:26:40.198 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f7ce2d6e22ff5df-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.198 DEBUG:boto:path=/aktlji-16/obj3 2024-04-19T23:26:40.199 DEBUG:boto:auth_path=/aktlji-16/obj3 2024-04-19T23:26:40.199 DEBUG:boto:path=/aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.199 DEBUG:boto:auth_path=/aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.199 DEBUG:boto:Method: GET 2024-04-19T23:26:40.199 DEBUG:boto:Path: /aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.199 DEBUG:boto:Data: 2024-04-19T23:26:40.199 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.199 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.199 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.199 DEBUG:boto:Params: {} 2024-04-19T23:26:40.199 DEBUG:boto:Token: None 2024-04-19T23:26:40.200 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.200 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss= 2024-04-19T23:26:40.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss=', 'Content-Length': '0'} 2024-04-19T23:26:40.201 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d898ce304db1da2e-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.242 DEBUG:boto:path=/aktlji-16/obj3 2024-04-19T23:26:40.242 DEBUG:boto:auth_path=/aktlji-16/obj3 2024-04-19T23:26:40.243 DEBUG:boto:path=/aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.243 DEBUG:boto:auth_path=/aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.243 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.243 DEBUG:boto:Path: /aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.243 DEBUG:boto:Data: 2024-04-19T23:26:40.243 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.243 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.243 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.243 DEBUG:boto:Params: {} 2024-04-19T23:26:40.243 DEBUG:boto:Token: None 2024-04-19T23:26:40.244 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.244 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IH5YlwSv+zpCDEHX9Fg+DCgmhRo= 2024-04-19T23:26:40.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IH5YlwSv+zpCDEHX9Fg+DCgmhRo=', 'Content-Length': '0'} 2024-04-19T23:26:40.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006624818534d63075-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.246 DEBUG:boto:path=/aktlji-16/obj3 2024-04-19T23:26:40.246 DEBUG:boto:auth_path=/aktlji-16/obj3 2024-04-19T23:26:40.246 DEBUG:boto:path=/aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.246 DEBUG:boto:auth_path=/aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.246 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.246 DEBUG:boto:Path: /aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.247 DEBUG:boto:Data: 2024-04-19T23:26:40.247 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.247 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.247 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.247 DEBUG:boto:Params: {} 2024-04-19T23:26:40.247 DEBUG:boto:Token: None 2024-04-19T23:26:40.247 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.247 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IH5YlwSv+zpCDEHX9Fg+DCgmhRo= 2024-04-19T23:26:40.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IH5YlwSv+zpCDEHX9Fg+DCgmhRo=', 'Content-Length': '0'} 2024-04-19T23:26:40.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddbab46da24fe5b9-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-19T23:26:40.249 DEBUG:boto:path=/aktlji-16/obj3 2024-04-19T23:26:40.249 DEBUG:boto:auth_path=/aktlji-16/obj3 2024-04-19T23:26:40.250 DEBUG:boto:path=/aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.250 DEBUG:boto:auth_path=/aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.250 DEBUG:boto:Method: GET 2024-04-19T23:26:40.250 DEBUG:boto:Path: /aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.250 DEBUG:boto:Data: 2024-04-19T23:26:40.250 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.250 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.250 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.250 DEBUG:boto:Params: {} 2024-04-19T23:26:40.250 DEBUG:boto:Token: None 2024-04-19T23:26:40.251 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.251 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss= 2024-04-19T23:26:40.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss=', 'Content-Length': '0'} 2024-04-19T23:26:40.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7402aa7ae62c3a9-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.253 DEBUG:boto:path=/aktlji-16/obj3 2024-04-19T23:26:40.253 DEBUG:boto:auth_path=/aktlji-16/obj3 2024-04-19T23:26:40.253 DEBUG:boto:path=/aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.253 DEBUG:boto:auth_path=/aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.253 DEBUG:boto:Method: GET 2024-04-19T23:26:40.253 DEBUG:boto:Path: /aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.253 DEBUG:boto:Data: 2024-04-19T23:26:40.253 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.254 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.254 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.254 DEBUG:boto:Params: {} 2024-04-19T23:26:40.254 DEBUG:boto:Token: None 2024-04-19T23:26:40.254 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj3?versionId=null 2024-04-19T23:26:40.254 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss= 2024-04-19T23:26:40.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss=', 'Content-Length': '0'} 2024-04-19T23:26:40.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd257b72db045772-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.256 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.256 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.257 DEBUG:boto:path=/aktlji-16/?prefix=obj3 2024-04-19T23:26:40.257 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj3 2024-04-19T23:26:40.257 DEBUG:boto:Method: GET 2024-04-19T23:26:40.257 DEBUG:boto:Path: /aktlji-16/?prefix=obj3 2024-04-19T23:26:40.257 DEBUG:boto:Data: 2024-04-19T23:26:40.257 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.257 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.257 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.257 DEBUG:boto:Params: {} 2024-04-19T23:26:40.257 DEBUG:boto:Token: None 2024-04-19T23:26:40.257 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.258 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040ac69e56c081420-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.261 DEBUG:boto:b'aktlji-16obj31000falseobj32024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.261 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.261 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.262 DEBUG:boto:path=/aktlji-16/?prefix=obj3 2024-04-19T23:26:40.262 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj3 2024-04-19T23:26:40.262 DEBUG:boto:Method: GET 2024-04-19T23:26:40.262 DEBUG:boto:Path: /aktlji-16/?prefix=obj3 2024-04-19T23:26:40.262 DEBUG:boto:Data: 2024-04-19T23:26:40.262 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.262 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.262 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.262 DEBUG:boto:Params: {} 2024-04-19T23:26:40.263 DEBUG:boto:Token: None 2024-04-19T23:26:40.263 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.263 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083941db154047aab-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.266 DEBUG:boto:b'aktlji-16obj31000falseobj32024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.266 DEBUG:boto:path=/aktlji-16/obj3 2024-04-19T23:26:40.266 DEBUG:boto:auth_path=/aktlji-16/obj3 2024-04-19T23:26:40.266 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.267 DEBUG:boto:Path: /aktlji-16/obj3 2024-04-19T23:26:40.267 DEBUG:boto:Data: 2024-04-19T23:26:40.267 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.267 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.267 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.267 DEBUG:boto:Params: {} 2024-04-19T23:26:40.267 DEBUG:boto:Token: None 2024-04-19T23:26:40.267 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj3 2024-04-19T23:26:40.268 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XQI4Whz5Z/pkDMye3pDEF6AFjbI= 2024-04-19T23:26:40.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XQI4Whz5Z/pkDMye3pDEF6AFjbI=', 'Content-Length': '0'} 2024-04-19T23:26:40.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fc5b53b7ae489ff-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.269 DEBUG:boto:path=/aktlji-16/obj3 2024-04-19T23:26:40.270 DEBUG:boto:auth_path=/aktlji-16/obj3 2024-04-19T23:26:40.270 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.270 DEBUG:boto:Path: /aktlji-16/obj3 2024-04-19T23:26:40.270 DEBUG:boto:Data: 2024-04-19T23:26:40.270 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.270 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.270 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.270 DEBUG:boto:Params: {} 2024-04-19T23:26:40.270 DEBUG:boto:Token: None 2024-04-19T23:26:40.270 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj3 2024-04-19T23:26:40.271 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XQI4Whz5Z/pkDMye3pDEF6AFjbI= 2024-04-19T23:26:40.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XQI4Whz5Z/pkDMye3pDEF6AFjbI=', 'Content-Length': '0'} 2024-04-19T23:26:40.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b797ce35c074490-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-19T23:26:40.273 DEBUG:boto:path=/aktlji-16/obj3 2024-04-19T23:26:40.273 DEBUG:boto:auth_path=/aktlji-16/obj3 2024-04-19T23:26:40.273 DEBUG:boto:Method: GET 2024-04-19T23:26:40.273 DEBUG:boto:Path: /aktlji-16/obj3 2024-04-19T23:26:40.273 DEBUG:boto:Data: 2024-04-19T23:26:40.273 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.273 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.273 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.274 DEBUG:boto:Params: {} 2024-04-19T23:26:40.274 DEBUG:boto:Token: None 2024-04-19T23:26:40.274 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj3 2024-04-19T23:26:40.274 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3k17iRo2XJGQs0/nP7oZ57vXgsU= 2024-04-19T23:26:40.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3k17iRo2XJGQs0/nP7oZ57vXgsU=', 'Content-Length': '0'} 2024-04-19T23:26:40.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004394ecc8be3b298-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.276 DEBUG:boto:path=/aktlji-16/obj3 2024-04-19T23:26:40.276 DEBUG:boto:auth_path=/aktlji-16/obj3 2024-04-19T23:26:40.276 DEBUG:boto:Method: GET 2024-04-19T23:26:40.277 DEBUG:boto:Path: /aktlji-16/obj3 2024-04-19T23:26:40.277 DEBUG:boto:Data: 2024-04-19T23:26:40.277 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.277 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.277 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.277 DEBUG:boto:Params: {} 2024-04-19T23:26:40.277 DEBUG:boto:Token: None 2024-04-19T23:26:40.277 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj3 2024-04-19T23:26:40.277 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3k17iRo2XJGQs0/nP7oZ57vXgsU= 2024-04-19T23:26:40.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3k17iRo2XJGQs0/nP7oZ57vXgsU=', 'Content-Length': '0'} 2024-04-19T23:26:40.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcb5344ec6285a97-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.279 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.280 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.280 DEBUG:boto:path=/aktlji-16/?prefix=obj3 2024-04-19T23:26:40.280 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj3 2024-04-19T23:26:40.280 DEBUG:boto:Method: GET 2024-04-19T23:26:40.280 DEBUG:boto:Path: /aktlji-16/?prefix=obj3 2024-04-19T23:26:40.280 DEBUG:boto:Data: 2024-04-19T23:26:40.280 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.280 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.280 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.280 DEBUG:boto:Params: {} 2024-04-19T23:26:40.280 DEBUG:boto:Token: None 2024-04-19T23:26:40.281 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.281 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbded74056940ea2-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.284 DEBUG:boto:b'aktlji-16obj31000falseobj32024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.284 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.284 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.285 DEBUG:boto:path=/aktlji-16/?prefix=obj3 2024-04-19T23:26:40.285 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj3 2024-04-19T23:26:40.285 DEBUG:boto:Method: GET 2024-04-19T23:26:40.285 DEBUG:boto:Path: /aktlji-16/?prefix=obj3 2024-04-19T23:26:40.285 DEBUG:boto:Data: 2024-04-19T23:26:40.285 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.285 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.285 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.285 DEBUG:boto:Params: {} 2024-04-19T23:26:40.285 DEBUG:boto:Token: None 2024-04-19T23:26:40.285 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.286 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ee9530054eca798-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.288 DEBUG:boto:b'aktlji-16obj31000falseobj32024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj30 2024-04-19T23:26:40.289 DEBUG:boto:path=/aktlji-16/obj30 2024-04-19T23:26:40.289 DEBUG:boto:auth_path=/aktlji-16/obj30 2024-04-19T23:26:40.289 DEBUG:boto:path=/aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.289 DEBUG:boto:auth_path=/aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.290 DEBUG:boto:Method: GET 2024-04-19T23:26:40.290 DEBUG:boto:Path: /aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.290 DEBUG:boto:Data: 2024-04-19T23:26:40.290 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.290 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.290 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.290 DEBUG:boto:Params: {} 2024-04-19T23:26:40.290 DEBUG:boto:Token: None 2024-04-19T23:26:40.290 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.291 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk= 2024-04-19T23:26:40.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk=', 'Content-Length': '0'} 2024-04-19T23:26:40.292 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a087a07bce56480c-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.293 DEBUG:boto:path=/aktlji-16/obj30 2024-04-19T23:26:40.293 DEBUG:boto:auth_path=/aktlji-16/obj30 2024-04-19T23:26:40.293 DEBUG:boto:path=/aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.293 DEBUG:boto:auth_path=/aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.293 DEBUG:boto:Method: GET 2024-04-19T23:26:40.293 DEBUG:boto:Path: /aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.293 DEBUG:boto:Data: 2024-04-19T23:26:40.293 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.293 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.293 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.293 DEBUG:boto:Params: {} 2024-04-19T23:26:40.294 DEBUG:boto:Token: None 2024-04-19T23:26:40.294 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.294 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk= 2024-04-19T23:26:40.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk=', 'Content-Length': '0'} 2024-04-19T23:26:40.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000896e2f231bfba69-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.296 DEBUG:boto:path=/aktlji-16/obj30 2024-04-19T23:26:40.296 DEBUG:boto:auth_path=/aktlji-16/obj30 2024-04-19T23:26:40.296 DEBUG:boto:path=/aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.296 DEBUG:boto:auth_path=/aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.296 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.296 DEBUG:boto:Path: /aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.297 DEBUG:boto:Data: 2024-04-19T23:26:40.297 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.297 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.297 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.297 DEBUG:boto:Params: {} 2024-04-19T23:26:40.297 DEBUG:boto:Token: None 2024-04-19T23:26:40.297 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.297 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HkgMPSGyucZGSK8AwHwyquhDZYc= 2024-04-19T23:26:40.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HkgMPSGyucZGSK8AwHwyquhDZYc=', 'Content-Length': '0'} 2024-04-19T23:26:40.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d2739eddffde520-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.299 DEBUG:boto:path=/aktlji-16/obj30 2024-04-19T23:26:40.299 DEBUG:boto:auth_path=/aktlji-16/obj30 2024-04-19T23:26:40.299 DEBUG:boto:path=/aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.300 DEBUG:boto:auth_path=/aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.300 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.300 DEBUG:boto:Path: /aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.300 DEBUG:boto:Data: 2024-04-19T23:26:40.300 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.300 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.300 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.300 DEBUG:boto:Params: {} 2024-04-19T23:26:40.300 DEBUG:boto:Token: None 2024-04-19T23:26:40.300 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.300 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HkgMPSGyucZGSK8AwHwyquhDZYc= 2024-04-19T23:26:40.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HkgMPSGyucZGSK8AwHwyquhDZYc=', 'Content-Length': '0'} 2024-04-19T23:26:40.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e8fb23f9dc21090-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj30 2024-04-19T23:26:40.303 DEBUG:boto:path=/aktlji-16/obj30 2024-04-19T23:26:40.303 DEBUG:boto:auth_path=/aktlji-16/obj30 2024-04-19T23:26:40.303 DEBUG:boto:path=/aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.303 DEBUG:boto:auth_path=/aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.303 DEBUG:boto:Method: GET 2024-04-19T23:26:40.303 DEBUG:boto:Path: /aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.303 DEBUG:boto:Data: 2024-04-19T23:26:40.303 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.304 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.304 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.304 DEBUG:boto:Params: {} 2024-04-19T23:26:40.304 DEBUG:boto:Token: None 2024-04-19T23:26:40.304 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.304 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk= 2024-04-19T23:26:40.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk=', 'Content-Length': '0'} 2024-04-19T23:26:40.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f54a74d8015b8bf-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.346 DEBUG:boto:path=/aktlji-16/obj30 2024-04-19T23:26:40.347 DEBUG:boto:auth_path=/aktlji-16/obj30 2024-04-19T23:26:40.347 DEBUG:boto:path=/aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.347 DEBUG:boto:auth_path=/aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.347 DEBUG:boto:Method: GET 2024-04-19T23:26:40.347 DEBUG:boto:Path: /aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.347 DEBUG:boto:Data: 2024-04-19T23:26:40.347 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.347 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.347 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.348 DEBUG:boto:Params: {} 2024-04-19T23:26:40.348 DEBUG:boto:Token: None 2024-04-19T23:26:40.348 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj30?versionId=null 2024-04-19T23:26:40.348 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk= 2024-04-19T23:26:40.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk=', 'Content-Length': '0'} 2024-04-19T23:26:40.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008275ddf986ab27fb-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.351 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.351 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.351 DEBUG:boto:path=/aktlji-16/?prefix=obj30 2024-04-19T23:26:40.351 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj30 2024-04-19T23:26:40.351 DEBUG:boto:Method: GET 2024-04-19T23:26:40.351 DEBUG:boto:Path: /aktlji-16/?prefix=obj30 2024-04-19T23:26:40.351 DEBUG:boto:Data: 2024-04-19T23:26:40.351 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.351 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.352 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.352 DEBUG:boto:Params: {} 2024-04-19T23:26:40.352 DEBUG:boto:Token: None 2024-04-19T23:26:40.352 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.352 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f9a2c42e7386c9d-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.355 DEBUG:boto:b'aktlji-16obj301000falseobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.356 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.356 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.356 DEBUG:boto:path=/aktlji-16/?prefix=obj30 2024-04-19T23:26:40.356 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj30 2024-04-19T23:26:40.356 DEBUG:boto:Method: GET 2024-04-19T23:26:40.356 DEBUG:boto:Path: /aktlji-16/?prefix=obj30 2024-04-19T23:26:40.356 DEBUG:boto:Data: 2024-04-19T23:26:40.356 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.356 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.356 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.357 DEBUG:boto:Params: {} 2024-04-19T23:26:40.357 DEBUG:boto:Token: None 2024-04-19T23:26:40.357 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.357 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c754da078a1c399e-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.406 DEBUG:boto:b'aktlji-16obj301000falseobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.407 DEBUG:boto:path=/aktlji-16/obj30 2024-04-19T23:26:40.407 DEBUG:boto:auth_path=/aktlji-16/obj30 2024-04-19T23:26:40.407 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.407 DEBUG:boto:Path: /aktlji-16/obj30 2024-04-19T23:26:40.407 DEBUG:boto:Data: 2024-04-19T23:26:40.407 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.407 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.407 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.407 DEBUG:boto:Params: {} 2024-04-19T23:26:40.407 DEBUG:boto:Token: None 2024-04-19T23:26:40.408 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj30 2024-04-19T23:26:40.408 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8DJLqC+sf2h19QqWvrh0eDajWuo= 2024-04-19T23:26:40.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8DJLqC+sf2h19QqWvrh0eDajWuo=', 'Content-Length': '0'} 2024-04-19T23:26:40.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7c357b3545ba231-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.410 DEBUG:boto:path=/aktlji-16/obj30 2024-04-19T23:26:40.410 DEBUG:boto:auth_path=/aktlji-16/obj30 2024-04-19T23:26:40.410 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.410 DEBUG:boto:Path: /aktlji-16/obj30 2024-04-19T23:26:40.410 DEBUG:boto:Data: 2024-04-19T23:26:40.410 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.410 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.410 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.410 DEBUG:boto:Params: {} 2024-04-19T23:26:40.411 DEBUG:boto:Token: None 2024-04-19T23:26:40.411 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj30 2024-04-19T23:26:40.411 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8DJLqC+sf2h19QqWvrh0eDajWuo= 2024-04-19T23:26:40.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8DJLqC+sf2h19QqWvrh0eDajWuo=', 'Content-Length': '0'} 2024-04-19T23:26:40.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008925d0dc7ed18d75-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj30 2024-04-19T23:26:40.413 DEBUG:boto:path=/aktlji-16/obj30 2024-04-19T23:26:40.413 DEBUG:boto:auth_path=/aktlji-16/obj30 2024-04-19T23:26:40.413 DEBUG:boto:Method: GET 2024-04-19T23:26:40.413 DEBUG:boto:Path: /aktlji-16/obj30 2024-04-19T23:26:40.413 DEBUG:boto:Data: 2024-04-19T23:26:40.414 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.414 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.414 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.414 DEBUG:boto:Params: {} 2024-04-19T23:26:40.414 DEBUG:boto:Token: None 2024-04-19T23:26:40.414 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj30 2024-04-19T23:26:40.414 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ly0FbRvyVKIbRGccr88h92pm76s= 2024-04-19T23:26:40.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ly0FbRvyVKIbRGccr88h92pm76s=', 'Content-Length': '0'} 2024-04-19T23:26:40.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb56cdaa925986ba-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.416 DEBUG:boto:path=/aktlji-16/obj30 2024-04-19T23:26:40.416 DEBUG:boto:auth_path=/aktlji-16/obj30 2024-04-19T23:26:40.417 DEBUG:boto:Method: GET 2024-04-19T23:26:40.417 DEBUG:boto:Path: /aktlji-16/obj30 2024-04-19T23:26:40.417 DEBUG:boto:Data: 2024-04-19T23:26:40.417 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.417 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.417 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.417 DEBUG:boto:Params: {} 2024-04-19T23:26:40.417 DEBUG:boto:Token: None 2024-04-19T23:26:40.417 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj30 2024-04-19T23:26:40.417 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ly0FbRvyVKIbRGccr88h92pm76s= 2024-04-19T23:26:40.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ly0FbRvyVKIbRGccr88h92pm76s=', 'Content-Length': '0'} 2024-04-19T23:26:40.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d59caf2e6cf2da8a-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.419 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.420 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.420 DEBUG:boto:path=/aktlji-16/?prefix=obj30 2024-04-19T23:26:40.420 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj30 2024-04-19T23:26:40.420 DEBUG:boto:Method: GET 2024-04-19T23:26:40.420 DEBUG:boto:Path: /aktlji-16/?prefix=obj30 2024-04-19T23:26:40.420 DEBUG:boto:Data: 2024-04-19T23:26:40.420 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.420 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.420 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.420 DEBUG:boto:Params: {} 2024-04-19T23:26:40.420 DEBUG:boto:Token: None 2024-04-19T23:26:40.421 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.421 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015b00c6cdb3fb667-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.424 DEBUG:boto:b'aktlji-16obj301000falseobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.424 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.424 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.424 DEBUG:boto:path=/aktlji-16/?prefix=obj30 2024-04-19T23:26:40.424 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj30 2024-04-19T23:26:40.425 DEBUG:boto:Method: GET 2024-04-19T23:26:40.425 DEBUG:boto:Path: /aktlji-16/?prefix=obj30 2024-04-19T23:26:40.425 DEBUG:boto:Data: 2024-04-19T23:26:40.425 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.425 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.425 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.425 DEBUG:boto:Params: {} 2024-04-19T23:26:40.425 DEBUG:boto:Token: None 2024-04-19T23:26:40.425 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.425 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d95f65c0a8369a-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.428 DEBUG:boto:b'aktlji-16obj301000falseobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-19T23:26:40.429 DEBUG:boto:path=/aktlji-16/obj31 2024-04-19T23:26:40.429 DEBUG:boto:auth_path=/aktlji-16/obj31 2024-04-19T23:26:40.429 DEBUG:boto:path=/aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.429 DEBUG:boto:auth_path=/aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.429 DEBUG:boto:Method: GET 2024-04-19T23:26:40.429 DEBUG:boto:Path: /aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.429 DEBUG:boto:Data: 2024-04-19T23:26:40.429 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.429 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.429 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.429 DEBUG:boto:Params: {} 2024-04-19T23:26:40.430 DEBUG:boto:Token: None 2024-04-19T23:26:40.430 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.430 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8= 2024-04-19T23:26:40.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8=', 'Content-Length': '0'} 2024-04-19T23:26:40.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8adc7eecf8ee94d-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.432 DEBUG:boto:path=/aktlji-16/obj31 2024-04-19T23:26:40.432 DEBUG:boto:auth_path=/aktlji-16/obj31 2024-04-19T23:26:40.432 DEBUG:boto:path=/aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.432 DEBUG:boto:auth_path=/aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.432 DEBUG:boto:Method: GET 2024-04-19T23:26:40.433 DEBUG:boto:Path: /aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.433 DEBUG:boto:Data: 2024-04-19T23:26:40.433 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.433 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.433 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.433 DEBUG:boto:Params: {} 2024-04-19T23:26:40.433 DEBUG:boto:Token: None 2024-04-19T23:26:40.433 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.433 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8= 2024-04-19T23:26:40.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8=', 'Content-Length': '0'} 2024-04-19T23:26:40.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6d856b96eecb673-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.482 DEBUG:boto:path=/aktlji-16/obj31 2024-04-19T23:26:40.483 DEBUG:boto:auth_path=/aktlji-16/obj31 2024-04-19T23:26:40.483 DEBUG:boto:path=/aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.483 DEBUG:boto:auth_path=/aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.483 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.483 DEBUG:boto:Path: /aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.483 DEBUG:boto:Data: 2024-04-19T23:26:40.483 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.483 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.483 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.483 DEBUG:boto:Params: {} 2024-04-19T23:26:40.483 DEBUG:boto:Token: None 2024-04-19T23:26:40.484 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KtK5TniNDv+kZXvXSJ/fi5KIZlE= 2024-04-19T23:26:40.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KtK5TniNDv+kZXvXSJ/fi5KIZlE=', 'Content-Length': '0'} 2024-04-19T23:26:40.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f7c7cc32dd42ae6-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.486 DEBUG:boto:path=/aktlji-16/obj31 2024-04-19T23:26:40.486 DEBUG:boto:auth_path=/aktlji-16/obj31 2024-04-19T23:26:40.487 DEBUG:boto:path=/aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.487 DEBUG:boto:auth_path=/aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.487 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.487 DEBUG:boto:Path: /aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.487 DEBUG:boto:Data: 2024-04-19T23:26:40.487 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.487 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.487 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.487 DEBUG:boto:Params: {} 2024-04-19T23:26:40.487 DEBUG:boto:Token: None 2024-04-19T23:26:40.487 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KtK5TniNDv+kZXvXSJ/fi5KIZlE= 2024-04-19T23:26:40.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KtK5TniNDv+kZXvXSJ/fi5KIZlE=', 'Content-Length': '0'} 2024-04-19T23:26:40.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dc74625f4810f5c-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-19T23:26:40.490 DEBUG:boto:path=/aktlji-16/obj31 2024-04-19T23:26:40.490 DEBUG:boto:auth_path=/aktlji-16/obj31 2024-04-19T23:26:40.490 DEBUG:boto:path=/aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.490 DEBUG:boto:auth_path=/aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.490 DEBUG:boto:Method: GET 2024-04-19T23:26:40.490 DEBUG:boto:Path: /aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.490 DEBUG:boto:Data: 2024-04-19T23:26:40.491 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.491 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.491 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.491 DEBUG:boto:Params: {} 2024-04-19T23:26:40.491 DEBUG:boto:Token: None 2024-04-19T23:26:40.491 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.491 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8= 2024-04-19T23:26:40.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8=', 'Content-Length': '0'} 2024-04-19T23:26:40.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b189937cb83fd92-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.493 DEBUG:boto:path=/aktlji-16/obj31 2024-04-19T23:26:40.493 DEBUG:boto:auth_path=/aktlji-16/obj31 2024-04-19T23:26:40.493 DEBUG:boto:path=/aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.493 DEBUG:boto:auth_path=/aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.494 DEBUG:boto:Method: GET 2024-04-19T23:26:40.494 DEBUG:boto:Path: /aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.494 DEBUG:boto:Data: 2024-04-19T23:26:40.494 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.494 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.494 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.494 DEBUG:boto:Params: {} 2024-04-19T23:26:40.494 DEBUG:boto:Token: None 2024-04-19T23:26:40.494 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj31?versionId=null 2024-04-19T23:26:40.494 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8= 2024-04-19T23:26:40.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8=', 'Content-Length': '0'} 2024-04-19T23:26:40.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000568a36ebbe100395-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.497 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.497 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.497 DEBUG:boto:path=/aktlji-16/?prefix=obj31 2024-04-19T23:26:40.497 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj31 2024-04-19T23:26:40.497 DEBUG:boto:Method: GET 2024-04-19T23:26:40.497 DEBUG:boto:Path: /aktlji-16/?prefix=obj31 2024-04-19T23:26:40.497 DEBUG:boto:Data: 2024-04-19T23:26:40.497 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.497 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.497 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.498 DEBUG:boto:Params: {} 2024-04-19T23:26:40.498 DEBUG:boto:Token: None 2024-04-19T23:26:40.498 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.498 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d568379604953ebf-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.501 DEBUG:boto:b'aktlji-16obj311000falseobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.501 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.501 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.501 DEBUG:boto:path=/aktlji-16/?prefix=obj31 2024-04-19T23:26:40.502 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj31 2024-04-19T23:26:40.502 DEBUG:boto:Method: GET 2024-04-19T23:26:40.502 DEBUG:boto:Path: /aktlji-16/?prefix=obj31 2024-04-19T23:26:40.502 DEBUG:boto:Data: 2024-04-19T23:26:40.502 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.502 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.502 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.502 DEBUG:boto:Params: {} 2024-04-19T23:26:40.502 DEBUG:boto:Token: None 2024-04-19T23:26:40.502 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.503 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb703a2cfc399ba4-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.505 DEBUG:boto:b'aktlji-16obj311000falseobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.506 DEBUG:boto:path=/aktlji-16/obj31 2024-04-19T23:26:40.506 DEBUG:boto:auth_path=/aktlji-16/obj31 2024-04-19T23:26:40.506 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.506 DEBUG:boto:Path: /aktlji-16/obj31 2024-04-19T23:26:40.506 DEBUG:boto:Data: 2024-04-19T23:26:40.506 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.506 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.507 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.507 DEBUG:boto:Params: {} 2024-04-19T23:26:40.507 DEBUG:boto:Token: None 2024-04-19T23:26:40.507 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj31 2024-04-19T23:26:40.507 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0acb4s3hT61LXkPKQSHr97UHlyg= 2024-04-19T23:26:40.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0acb4s3hT61LXkPKQSHr97UHlyg=', 'Content-Length': '0'} 2024-04-19T23:26:40.509 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006713d0926f76f3d2-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.509 DEBUG:boto:path=/aktlji-16/obj31 2024-04-19T23:26:40.509 DEBUG:boto:auth_path=/aktlji-16/obj31 2024-04-19T23:26:40.509 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.510 DEBUG:boto:Path: /aktlji-16/obj31 2024-04-19T23:26:40.510 DEBUG:boto:Data: 2024-04-19T23:26:40.510 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.510 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.510 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.510 DEBUG:boto:Params: {} 2024-04-19T23:26:40.510 DEBUG:boto:Token: None 2024-04-19T23:26:40.510 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj31 2024-04-19T23:26:40.510 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0acb4s3hT61LXkPKQSHr97UHlyg= 2024-04-19T23:26:40.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0acb4s3hT61LXkPKQSHr97UHlyg=', 'Content-Length': '0'} 2024-04-19T23:26:40.512 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bda11d2c42830410-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-19T23:26:40.512 DEBUG:boto:path=/aktlji-16/obj31 2024-04-19T23:26:40.512 DEBUG:boto:auth_path=/aktlji-16/obj31 2024-04-19T23:26:40.513 DEBUG:boto:Method: GET 2024-04-19T23:26:40.513 DEBUG:boto:Path: /aktlji-16/obj31 2024-04-19T23:26:40.513 DEBUG:boto:Data: 2024-04-19T23:26:40.513 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.513 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.513 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.513 DEBUG:boto:Params: {} 2024-04-19T23:26:40.513 DEBUG:boto:Token: None 2024-04-19T23:26:40.513 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj31 2024-04-19T23:26:40.513 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z6aIj4PFM+iih7iSsQQ4M+XAbOc= 2024-04-19T23:26:40.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z6aIj4PFM+iih7iSsQQ4M+XAbOc=', 'Content-Length': '0'} 2024-04-19T23:26:40.515 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c379b53148e9cc28-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.516 DEBUG:boto:path=/aktlji-16/obj31 2024-04-19T23:26:40.516 DEBUG:boto:auth_path=/aktlji-16/obj31 2024-04-19T23:26:40.516 DEBUG:boto:Method: GET 2024-04-19T23:26:40.516 DEBUG:boto:Path: /aktlji-16/obj31 2024-04-19T23:26:40.516 DEBUG:boto:Data: 2024-04-19T23:26:40.516 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.516 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.516 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.516 DEBUG:boto:Params: {} 2024-04-19T23:26:40.517 DEBUG:boto:Token: None 2024-04-19T23:26:40.517 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj31 2024-04-19T23:26:40.517 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z6aIj4PFM+iih7iSsQQ4M+XAbOc= 2024-04-19T23:26:40.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z6aIj4PFM+iih7iSsQQ4M+XAbOc=', 'Content-Length': '0'} 2024-04-19T23:26:40.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b025778f9705bfd9-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.519 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.519 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.519 DEBUG:boto:path=/aktlji-16/?prefix=obj31 2024-04-19T23:26:40.519 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj31 2024-04-19T23:26:40.519 DEBUG:boto:Method: GET 2024-04-19T23:26:40.519 DEBUG:boto:Path: /aktlji-16/?prefix=obj31 2024-04-19T23:26:40.520 DEBUG:boto:Data: 2024-04-19T23:26:40.520 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.520 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.520 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.520 DEBUG:boto:Params: {} 2024-04-19T23:26:40.520 DEBUG:boto:Token: None 2024-04-19T23:26:40.520 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.520 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026e57ce3ad3f2f74-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.566 DEBUG:boto:b'aktlji-16obj311000falseobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.567 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.567 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.567 DEBUG:boto:path=/aktlji-16/?prefix=obj31 2024-04-19T23:26:40.567 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj31 2024-04-19T23:26:40.567 DEBUG:boto:Method: GET 2024-04-19T23:26:40.567 DEBUG:boto:Path: /aktlji-16/?prefix=obj31 2024-04-19T23:26:40.567 DEBUG:boto:Data: 2024-04-19T23:26:40.567 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.568 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.568 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.568 DEBUG:boto:Params: {} 2024-04-19T23:26:40.568 DEBUG:boto:Token: None 2024-04-19T23:26:40.568 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.568 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003b2f481a4d93760-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.571 DEBUG:boto:b'aktlji-16obj311000falseobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj32 2024-04-19T23:26:40.571 DEBUG:boto:path=/aktlji-16/obj32 2024-04-19T23:26:40.571 DEBUG:boto:auth_path=/aktlji-16/obj32 2024-04-19T23:26:40.571 DEBUG:boto:path=/aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.571 DEBUG:boto:auth_path=/aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.571 DEBUG:boto:Method: GET 2024-04-19T23:26:40.572 DEBUG:boto:Path: /aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.572 DEBUG:boto:Data: 2024-04-19T23:26:40.572 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.572 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.572 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.572 DEBUG:boto:Params: {} 2024-04-19T23:26:40.572 DEBUG:boto:Token: None 2024-04-19T23:26:40.572 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.572 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE= 2024-04-19T23:26:40.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE=', 'Content-Length': '0'} 2024-04-19T23:26:40.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003072ec9c504d2fbb-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.574 DEBUG:boto:path=/aktlji-16/obj32 2024-04-19T23:26:40.575 DEBUG:boto:auth_path=/aktlji-16/obj32 2024-04-19T23:26:40.575 DEBUG:boto:path=/aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.575 DEBUG:boto:auth_path=/aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.575 DEBUG:boto:Method: GET 2024-04-19T23:26:40.575 DEBUG:boto:Path: /aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.575 DEBUG:boto:Data: 2024-04-19T23:26:40.575 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.575 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.575 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.575 DEBUG:boto:Params: {} 2024-04-19T23:26:40.575 DEBUG:boto:Token: None 2024-04-19T23:26:40.576 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.576 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE= 2024-04-19T23:26:40.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE=', 'Content-Length': '0'} 2024-04-19T23:26:40.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be4b12f5ec6a2d7d-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.578 DEBUG:boto:path=/aktlji-16/obj32 2024-04-19T23:26:40.578 DEBUG:boto:auth_path=/aktlji-16/obj32 2024-04-19T23:26:40.578 DEBUG:boto:path=/aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.578 DEBUG:boto:auth_path=/aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.578 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.578 DEBUG:boto:Path: /aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.578 DEBUG:boto:Data: 2024-04-19T23:26:40.578 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.578 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.579 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.579 DEBUG:boto:Params: {} 2024-04-19T23:26:40.579 DEBUG:boto:Token: None 2024-04-19T23:26:40.579 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.579 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wqk4Wh3f2iaJUyPI0x8nlatsfb4= 2024-04-19T23:26:40.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wqk4Wh3f2iaJUyPI0x8nlatsfb4=', 'Content-Length': '0'} 2024-04-19T23:26:40.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9057cad8a6641bf-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.581 DEBUG:boto:path=/aktlji-16/obj32 2024-04-19T23:26:40.581 DEBUG:boto:auth_path=/aktlji-16/obj32 2024-04-19T23:26:40.581 DEBUG:boto:path=/aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.581 DEBUG:boto:auth_path=/aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.582 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.582 DEBUG:boto:Path: /aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.582 DEBUG:boto:Data: 2024-04-19T23:26:40.582 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.582 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.582 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.582 DEBUG:boto:Params: {} 2024-04-19T23:26:40.582 DEBUG:boto:Token: None 2024-04-19T23:26:40.583 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.583 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wqk4Wh3f2iaJUyPI0x8nlatsfb4= 2024-04-19T23:26:40.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wqk4Wh3f2iaJUyPI0x8nlatsfb4=', 'Content-Length': '0'} 2024-04-19T23:26:40.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c87dae9244ad0da-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj32 2024-04-19T23:26:40.585 DEBUG:boto:path=/aktlji-16/obj32 2024-04-19T23:26:40.585 DEBUG:boto:auth_path=/aktlji-16/obj32 2024-04-19T23:26:40.585 DEBUG:boto:path=/aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.585 DEBUG:boto:auth_path=/aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.585 DEBUG:boto:Method: GET 2024-04-19T23:26:40.585 DEBUG:boto:Path: /aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.585 DEBUG:boto:Data: 2024-04-19T23:26:40.586 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.586 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.586 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.586 DEBUG:boto:Params: {} 2024-04-19T23:26:40.586 DEBUG:boto:Token: None 2024-04-19T23:26:40.586 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.586 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE= 2024-04-19T23:26:40.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE=', 'Content-Length': '0'} 2024-04-19T23:26:40.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfb4d1ee31b53065-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.589 DEBUG:boto:path=/aktlji-16/obj32 2024-04-19T23:26:40.589 DEBUG:boto:auth_path=/aktlji-16/obj32 2024-04-19T23:26:40.589 DEBUG:boto:path=/aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.589 DEBUG:boto:auth_path=/aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.589 DEBUG:boto:Method: GET 2024-04-19T23:26:40.589 DEBUG:boto:Path: /aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.589 DEBUG:boto:Data: 2024-04-19T23:26:40.589 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.590 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.590 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.590 DEBUG:boto:Params: {} 2024-04-19T23:26:40.590 DEBUG:boto:Token: None 2024-04-19T23:26:40.590 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj32?versionId=null 2024-04-19T23:26:40.590 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE= 2024-04-19T23:26:40.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE=', 'Content-Length': '0'} 2024-04-19T23:26:40.592 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085a53b6baf2d8433-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.592 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.592 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.592 DEBUG:boto:path=/aktlji-16/?prefix=obj32 2024-04-19T23:26:40.593 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj32 2024-04-19T23:26:40.593 DEBUG:boto:Method: GET 2024-04-19T23:26:40.593 DEBUG:boto:Path: /aktlji-16/?prefix=obj32 2024-04-19T23:26:40.593 DEBUG:boto:Data: 2024-04-19T23:26:40.593 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.593 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.593 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.593 DEBUG:boto:Params: {} 2024-04-19T23:26:40.593 DEBUG:boto:Token: None 2024-04-19T23:26:40.593 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.594 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3b337d8476505a9-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.596 DEBUG:boto:b'aktlji-16obj321000falseobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.597 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.597 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.597 DEBUG:boto:path=/aktlji-16/?prefix=obj32 2024-04-19T23:26:40.597 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj32 2024-04-19T23:26:40.597 DEBUG:boto:Method: GET 2024-04-19T23:26:40.597 DEBUG:boto:Path: /aktlji-16/?prefix=obj32 2024-04-19T23:26:40.597 DEBUG:boto:Data: 2024-04-19T23:26:40.597 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.598 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.598 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.598 DEBUG:boto:Params: {} 2024-04-19T23:26:40.598 DEBUG:boto:Token: None 2024-04-19T23:26:40.598 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.598 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060bb425549145708-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.601 DEBUG:boto:b'aktlji-16obj321000falseobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.601 DEBUG:boto:path=/aktlji-16/obj32 2024-04-19T23:26:40.601 DEBUG:boto:auth_path=/aktlji-16/obj32 2024-04-19T23:26:40.602 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.602 DEBUG:boto:Path: /aktlji-16/obj32 2024-04-19T23:26:40.602 DEBUG:boto:Data: 2024-04-19T23:26:40.602 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.602 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.602 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.602 DEBUG:boto:Params: {} 2024-04-19T23:26:40.602 DEBUG:boto:Token: None 2024-04-19T23:26:40.602 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj32 2024-04-19T23:26:40.603 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZykiHAQAFoJjhK/R7fgSYYDlSw0= 2024-04-19T23:26:40.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZykiHAQAFoJjhK/R7fgSYYDlSw0=', 'Content-Length': '0'} 2024-04-19T23:26:40.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b39c9ef5109f93f-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.605 DEBUG:boto:path=/aktlji-16/obj32 2024-04-19T23:26:40.605 DEBUG:boto:auth_path=/aktlji-16/obj32 2024-04-19T23:26:40.605 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.605 DEBUG:boto:Path: /aktlji-16/obj32 2024-04-19T23:26:40.605 DEBUG:boto:Data: 2024-04-19T23:26:40.605 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.605 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.605 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.605 DEBUG:boto:Params: {} 2024-04-19T23:26:40.606 DEBUG:boto:Token: None 2024-04-19T23:26:40.606 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj32 2024-04-19T23:26:40.606 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZykiHAQAFoJjhK/R7fgSYYDlSw0= 2024-04-19T23:26:40.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZykiHAQAFoJjhK/R7fgSYYDlSw0=', 'Content-Length': '0'} 2024-04-19T23:26:40.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000943174b8a5088a61-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj32 2024-04-19T23:26:40.608 DEBUG:boto:path=/aktlji-16/obj32 2024-04-19T23:26:40.608 DEBUG:boto:auth_path=/aktlji-16/obj32 2024-04-19T23:26:40.608 DEBUG:boto:Method: GET 2024-04-19T23:26:40.608 DEBUG:boto:Path: /aktlji-16/obj32 2024-04-19T23:26:40.608 DEBUG:boto:Data: 2024-04-19T23:26:40.608 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.609 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.609 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.609 DEBUG:boto:Params: {} 2024-04-19T23:26:40.609 DEBUG:boto:Token: None 2024-04-19T23:26:40.609 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj32 2024-04-19T23:26:40.609 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ik74UYDwT5VQS665Fux86sJkqIQ= 2024-04-19T23:26:40.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ik74UYDwT5VQS665Fux86sJkqIQ=', 'Content-Length': '0'} 2024-04-19T23:26:40.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fe4b9dac3392cab-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.654 DEBUG:boto:path=/aktlji-16/obj32 2024-04-19T23:26:40.654 DEBUG:boto:auth_path=/aktlji-16/obj32 2024-04-19T23:26:40.654 DEBUG:boto:Method: GET 2024-04-19T23:26:40.655 DEBUG:boto:Path: /aktlji-16/obj32 2024-04-19T23:26:40.655 DEBUG:boto:Data: 2024-04-19T23:26:40.655 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.655 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.655 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.655 DEBUG:boto:Params: {} 2024-04-19T23:26:40.655 DEBUG:boto:Token: None 2024-04-19T23:26:40.655 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj32 2024-04-19T23:26:40.656 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ik74UYDwT5VQS665Fux86sJkqIQ= 2024-04-19T23:26:40.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ik74UYDwT5VQS665Fux86sJkqIQ=', 'Content-Length': '0'} 2024-04-19T23:26:40.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c1a7f202eb7c5f7-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.658 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.658 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.658 DEBUG:boto:path=/aktlji-16/?prefix=obj32 2024-04-19T23:26:40.658 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj32 2024-04-19T23:26:40.658 DEBUG:boto:Method: GET 2024-04-19T23:26:40.659 DEBUG:boto:Path: /aktlji-16/?prefix=obj32 2024-04-19T23:26:40.659 DEBUG:boto:Data: 2024-04-19T23:26:40.659 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.659 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.659 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.659 DEBUG:boto:Params: {} 2024-04-19T23:26:40.659 DEBUG:boto:Token: None 2024-04-19T23:26:40.659 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.659 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9e761c6a3ede467-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.662 DEBUG:boto:b'aktlji-16obj321000falseobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.662 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.663 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.663 DEBUG:boto:path=/aktlji-16/?prefix=obj32 2024-04-19T23:26:40.663 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj32 2024-04-19T23:26:40.663 DEBUG:boto:Method: GET 2024-04-19T23:26:40.663 DEBUG:boto:Path: /aktlji-16/?prefix=obj32 2024-04-19T23:26:40.663 DEBUG:boto:Data: 2024-04-19T23:26:40.663 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.663 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.663 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.663 DEBUG:boto:Params: {} 2024-04-19T23:26:40.664 DEBUG:boto:Token: None 2024-04-19T23:26:40.664 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.664 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b113af14c24b1de5-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.666 DEBUG:boto:b'aktlji-16obj321000falseobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-19T23:26:40.667 DEBUG:boto:path=/aktlji-16/obj33 2024-04-19T23:26:40.667 DEBUG:boto:auth_path=/aktlji-16/obj33 2024-04-19T23:26:40.667 DEBUG:boto:path=/aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.667 DEBUG:boto:auth_path=/aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.667 DEBUG:boto:Method: GET 2024-04-19T23:26:40.667 DEBUG:boto:Path: /aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.667 DEBUG:boto:Data: 2024-04-19T23:26:40.668 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.668 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.668 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.668 DEBUG:boto:Params: {} 2024-04-19T23:26:40.668 DEBUG:boto:Token: None 2024-04-19T23:26:40.668 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.668 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI= 2024-04-19T23:26:40.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI=', 'Content-Length': '0'} 2024-04-19T23:26:40.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f9bb4c530846832-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.670 DEBUG:boto:path=/aktlji-16/obj33 2024-04-19T23:26:40.671 DEBUG:boto:auth_path=/aktlji-16/obj33 2024-04-19T23:26:40.671 DEBUG:boto:path=/aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.671 DEBUG:boto:auth_path=/aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.671 DEBUG:boto:Method: GET 2024-04-19T23:26:40.671 DEBUG:boto:Path: /aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.671 DEBUG:boto:Data: 2024-04-19T23:26:40.671 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.671 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.671 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.671 DEBUG:boto:Params: {} 2024-04-19T23:26:40.672 DEBUG:boto:Token: None 2024-04-19T23:26:40.672 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.672 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI= 2024-04-19T23:26:40.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI=', 'Content-Length': '0'} 2024-04-19T23:26:40.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081eba2b5a26239c5-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.714 DEBUG:boto:path=/aktlji-16/obj33 2024-04-19T23:26:40.715 DEBUG:boto:auth_path=/aktlji-16/obj33 2024-04-19T23:26:40.715 DEBUG:boto:path=/aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.715 DEBUG:boto:auth_path=/aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.715 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.715 DEBUG:boto:Path: /aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.715 DEBUG:boto:Data: 2024-04-19T23:26:40.715 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.715 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.716 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.716 DEBUG:boto:Params: {} 2024-04-19T23:26:40.716 DEBUG:boto:Token: None 2024-04-19T23:26:40.716 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.716 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Likvd2hK7rp6bkk94ncOxfSdkbE= 2024-04-19T23:26:40.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Likvd2hK7rp6bkk94ncOxfSdkbE=', 'Content-Length': '0'} 2024-04-19T23:26:40.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e741147b2663e4b-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.719 DEBUG:boto:path=/aktlji-16/obj33 2024-04-19T23:26:40.719 DEBUG:boto:auth_path=/aktlji-16/obj33 2024-04-19T23:26:40.719 DEBUG:boto:path=/aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.719 DEBUG:boto:auth_path=/aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.719 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.719 DEBUG:boto:Path: /aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.719 DEBUG:boto:Data: 2024-04-19T23:26:40.719 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.719 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.720 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.720 DEBUG:boto:Params: {} 2024-04-19T23:26:40.720 DEBUG:boto:Token: None 2024-04-19T23:26:40.720 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.720 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Likvd2hK7rp6bkk94ncOxfSdkbE= 2024-04-19T23:26:40.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Likvd2hK7rp6bkk94ncOxfSdkbE=', 'Content-Length': '0'} 2024-04-19T23:26:40.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d6cb96964505de4-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-19T23:26:40.723 DEBUG:boto:path=/aktlji-16/obj33 2024-04-19T23:26:40.723 DEBUG:boto:auth_path=/aktlji-16/obj33 2024-04-19T23:26:40.723 DEBUG:boto:path=/aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.723 DEBUG:boto:auth_path=/aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.723 DEBUG:boto:Method: GET 2024-04-19T23:26:40.723 DEBUG:boto:Path: /aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.723 DEBUG:boto:Data: 2024-04-19T23:26:40.724 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.724 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.724 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.724 DEBUG:boto:Params: {} 2024-04-19T23:26:40.724 DEBUG:boto:Token: None 2024-04-19T23:26:40.724 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.724 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI= 2024-04-19T23:26:40.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI=', 'Content-Length': '0'} 2024-04-19T23:26:40.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000072ba07265b7419a-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.726 DEBUG:boto:path=/aktlji-16/obj33 2024-04-19T23:26:40.727 DEBUG:boto:auth_path=/aktlji-16/obj33 2024-04-19T23:26:40.727 DEBUG:boto:path=/aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.727 DEBUG:boto:auth_path=/aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.727 DEBUG:boto:Method: GET 2024-04-19T23:26:40.727 DEBUG:boto:Path: /aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.727 DEBUG:boto:Data: 2024-04-19T23:26:40.727 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.727 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.727 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.727 DEBUG:boto:Params: {} 2024-04-19T23:26:40.728 DEBUG:boto:Token: None 2024-04-19T23:26:40.728 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj33?versionId=null 2024-04-19T23:26:40.728 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI= 2024-04-19T23:26:40.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI=', 'Content-Length': '0'} 2024-04-19T23:26:40.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d1b436c7d62b46d-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.730 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.730 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.730 DEBUG:boto:path=/aktlji-16/?prefix=obj33 2024-04-19T23:26:40.731 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj33 2024-04-19T23:26:40.731 DEBUG:boto:Method: GET 2024-04-19T23:26:40.731 DEBUG:boto:Path: /aktlji-16/?prefix=obj33 2024-04-19T23:26:40.731 DEBUG:boto:Data: 2024-04-19T23:26:40.731 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.731 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.731 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.731 DEBUG:boto:Params: {} 2024-04-19T23:26:40.732 DEBUG:boto:Token: None 2024-04-19T23:26:40.732 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.732 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090ed0da9b2860fb1-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.734 DEBUG:boto:b'aktlji-16obj331000falseobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.735 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.735 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.735 DEBUG:boto:path=/aktlji-16/?prefix=obj33 2024-04-19T23:26:40.735 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj33 2024-04-19T23:26:40.735 DEBUG:boto:Method: GET 2024-04-19T23:26:40.735 DEBUG:boto:Path: /aktlji-16/?prefix=obj33 2024-04-19T23:26:40.735 DEBUG:boto:Data: 2024-04-19T23:26:40.735 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.736 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.736 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.736 DEBUG:boto:Params: {} 2024-04-19T23:26:40.736 DEBUG:boto:Token: None 2024-04-19T23:26:40.736 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.736 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dac6439131e66d74-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.739 DEBUG:boto:b'aktlji-16obj331000falseobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.739 DEBUG:boto:path=/aktlji-16/obj33 2024-04-19T23:26:40.739 DEBUG:boto:auth_path=/aktlji-16/obj33 2024-04-19T23:26:40.740 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.740 DEBUG:boto:Path: /aktlji-16/obj33 2024-04-19T23:26:40.740 DEBUG:boto:Data: 2024-04-19T23:26:40.740 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.740 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.740 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.740 DEBUG:boto:Params: {} 2024-04-19T23:26:40.740 DEBUG:boto:Token: None 2024-04-19T23:26:40.741 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj33 2024-04-19T23:26:40.741 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nCUGKwm4j/5vTHxonKFKXN9+uCM= 2024-04-19T23:26:40.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nCUGKwm4j/5vTHxonKFKXN9+uCM=', 'Content-Length': '0'} 2024-04-19T23:26:40.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e083b6d14ca383d7-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.743 DEBUG:boto:path=/aktlji-16/obj33 2024-04-19T23:26:40.743 DEBUG:boto:auth_path=/aktlji-16/obj33 2024-04-19T23:26:40.743 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.743 DEBUG:boto:Path: /aktlji-16/obj33 2024-04-19T23:26:40.744 DEBUG:boto:Data: 2024-04-19T23:26:40.744 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.744 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.744 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.744 DEBUG:boto:Params: {} 2024-04-19T23:26:40.744 DEBUG:boto:Token: None 2024-04-19T23:26:40.744 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj33 2024-04-19T23:26:40.744 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nCUGKwm4j/5vTHxonKFKXN9+uCM= 2024-04-19T23:26:40.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nCUGKwm4j/5vTHxonKFKXN9+uCM=', 'Content-Length': '0'} 2024-04-19T23:26:40.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090bc2c8921315a9a-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-19T23:26:40.746 DEBUG:boto:path=/aktlji-16/obj33 2024-04-19T23:26:40.747 DEBUG:boto:auth_path=/aktlji-16/obj33 2024-04-19T23:26:40.747 DEBUG:boto:Method: GET 2024-04-19T23:26:40.747 DEBUG:boto:Path: /aktlji-16/obj33 2024-04-19T23:26:40.747 DEBUG:boto:Data: 2024-04-19T23:26:40.747 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.747 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.747 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.747 DEBUG:boto:Params: {} 2024-04-19T23:26:40.747 DEBUG:boto:Token: None 2024-04-19T23:26:40.747 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj33 2024-04-19T23:26:40.748 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o2CJUTwM24L8Oq3lw5XY/0enNHU= 2024-04-19T23:26:40.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o2CJUTwM24L8Oq3lw5XY/0enNHU=', 'Content-Length': '0'} 2024-04-19T23:26:40.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b39be1222f463f3f-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.790 DEBUG:boto:path=/aktlji-16/obj33 2024-04-19T23:26:40.791 DEBUG:boto:auth_path=/aktlji-16/obj33 2024-04-19T23:26:40.791 DEBUG:boto:Method: GET 2024-04-19T23:26:40.791 DEBUG:boto:Path: /aktlji-16/obj33 2024-04-19T23:26:40.791 DEBUG:boto:Data: 2024-04-19T23:26:40.791 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.791 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.791 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.791 DEBUG:boto:Params: {} 2024-04-19T23:26:40.791 DEBUG:boto:Token: None 2024-04-19T23:26:40.792 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj33 2024-04-19T23:26:40.792 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o2CJUTwM24L8Oq3lw5XY/0enNHU= 2024-04-19T23:26:40.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o2CJUTwM24L8Oq3lw5XY/0enNHU=', 'Content-Length': '0'} 2024-04-19T23:26:40.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9385fbf434e2b91-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.794 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.794 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.794 DEBUG:boto:path=/aktlji-16/?prefix=obj33 2024-04-19T23:26:40.794 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj33 2024-04-19T23:26:40.794 DEBUG:boto:Method: GET 2024-04-19T23:26:40.795 DEBUG:boto:Path: /aktlji-16/?prefix=obj33 2024-04-19T23:26:40.795 DEBUG:boto:Data: 2024-04-19T23:26:40.795 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.795 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.795 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.795 DEBUG:boto:Params: {} 2024-04-19T23:26:40.795 DEBUG:boto:Token: None 2024-04-19T23:26:40.795 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.795 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008ec3684742dda9a-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.798 DEBUG:boto:b'aktlji-16obj331000falseobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.798 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.799 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.799 DEBUG:boto:path=/aktlji-16/?prefix=obj33 2024-04-19T23:26:40.799 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj33 2024-04-19T23:26:40.799 DEBUG:boto:Method: GET 2024-04-19T23:26:40.799 DEBUG:boto:Path: /aktlji-16/?prefix=obj33 2024-04-19T23:26:40.799 DEBUG:boto:Data: 2024-04-19T23:26:40.799 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.799 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.799 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.800 DEBUG:boto:Params: {} 2024-04-19T23:26:40.800 DEBUG:boto:Token: None 2024-04-19T23:26:40.800 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.800 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f46726d4eeb40566-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.803 DEBUG:boto:b'aktlji-16obj331000falseobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-19T23:26:40.803 DEBUG:boto:path=/aktlji-16/obj34 2024-04-19T23:26:40.803 DEBUG:boto:auth_path=/aktlji-16/obj34 2024-04-19T23:26:40.803 DEBUG:boto:path=/aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.803 DEBUG:boto:auth_path=/aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.803 DEBUG:boto:Method: GET 2024-04-19T23:26:40.803 DEBUG:boto:Path: /aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.804 DEBUG:boto:Data: 2024-04-19T23:26:40.804 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.804 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.804 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.804 DEBUG:boto:Params: {} 2024-04-19T23:26:40.804 DEBUG:boto:Token: None 2024-04-19T23:26:40.804 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0= 2024-04-19T23:26:40.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0=', 'Content-Length': '0'} 2024-04-19T23:26:40.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031abac1c34b2ca56-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.806 DEBUG:boto:path=/aktlji-16/obj34 2024-04-19T23:26:40.807 DEBUG:boto:auth_path=/aktlji-16/obj34 2024-04-19T23:26:40.807 DEBUG:boto:path=/aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.807 DEBUG:boto:auth_path=/aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.807 DEBUG:boto:Method: GET 2024-04-19T23:26:40.807 DEBUG:boto:Path: /aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.807 DEBUG:boto:Data: 2024-04-19T23:26:40.807 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.807 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.807 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.807 DEBUG:boto:Params: {} 2024-04-19T23:26:40.807 DEBUG:boto:Token: None 2024-04-19T23:26:40.808 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.808 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0= 2024-04-19T23:26:40.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0=', 'Content-Length': '0'} 2024-04-19T23:26:40.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e2bee4413a20907-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.850 DEBUG:boto:path=/aktlji-16/obj34 2024-04-19T23:26:40.854 DEBUG:boto:auth_path=/aktlji-16/obj34 2024-04-19T23:26:40.854 DEBUG:boto:path=/aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.854 DEBUG:boto:auth_path=/aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.854 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.854 DEBUG:boto:Path: /aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.854 DEBUG:boto:Data: 2024-04-19T23:26:40.854 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.854 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.855 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.855 DEBUG:boto:Params: {} 2024-04-19T23:26:40.855 DEBUG:boto:Token: None 2024-04-19T23:26:40.855 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.855 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k6LTzXxcaK3GNMYQjGNhXBwBes4= 2024-04-19T23:26:40.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k6LTzXxcaK3GNMYQjGNhXBwBes4=', 'Content-Length': '0'} 2024-04-19T23:26:40.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c27452c9bc0062b1-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.857 DEBUG:boto:path=/aktlji-16/obj34 2024-04-19T23:26:40.857 DEBUG:boto:auth_path=/aktlji-16/obj34 2024-04-19T23:26:40.858 DEBUG:boto:path=/aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.858 DEBUG:boto:auth_path=/aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.858 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.858 DEBUG:boto:Path: /aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.858 DEBUG:boto:Data: 2024-04-19T23:26:40.858 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.858 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.858 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.859 DEBUG:boto:Params: {} 2024-04-19T23:26:40.859 DEBUG:boto:Token: None 2024-04-19T23:26:40.859 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.859 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k6LTzXxcaK3GNMYQjGNhXBwBes4= 2024-04-19T23:26:40.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k6LTzXxcaK3GNMYQjGNhXBwBes4=', 'Content-Length': '0'} 2024-04-19T23:26:40.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce5ce4b7afe767b4-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-19T23:26:40.861 DEBUG:boto:path=/aktlji-16/obj34 2024-04-19T23:26:40.861 DEBUG:boto:auth_path=/aktlji-16/obj34 2024-04-19T23:26:40.862 DEBUG:boto:path=/aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.862 DEBUG:boto:auth_path=/aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.862 DEBUG:boto:Method: GET 2024-04-19T23:26:40.862 DEBUG:boto:Path: /aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.862 DEBUG:boto:Data: 2024-04-19T23:26:40.862 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.862 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.862 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.862 DEBUG:boto:Params: {} 2024-04-19T23:26:40.862 DEBUG:boto:Token: None 2024-04-19T23:26:40.863 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.863 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0= 2024-04-19T23:26:40.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0=', 'Content-Length': '0'} 2024-04-19T23:26:40.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092b3bab6e081ffeb-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.865 DEBUG:boto:path=/aktlji-16/obj34 2024-04-19T23:26:40.865 DEBUG:boto:auth_path=/aktlji-16/obj34 2024-04-19T23:26:40.865 DEBUG:boto:path=/aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.865 DEBUG:boto:auth_path=/aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.866 DEBUG:boto:Method: GET 2024-04-19T23:26:40.866 DEBUG:boto:Path: /aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.866 DEBUG:boto:Data: 2024-04-19T23:26:40.866 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.866 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.866 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.866 DEBUG:boto:Params: {} 2024-04-19T23:26:40.866 DEBUG:boto:Token: None 2024-04-19T23:26:40.866 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj34?versionId=null 2024-04-19T23:26:40.866 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0= 2024-04-19T23:26:40.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0=', 'Content-Length': '0'} 2024-04-19T23:26:40.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001089572b8f4ec7af-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.868 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.869 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.869 DEBUG:boto:path=/aktlji-16/?prefix=obj34 2024-04-19T23:26:40.869 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj34 2024-04-19T23:26:40.869 DEBUG:boto:Method: GET 2024-04-19T23:26:40.869 DEBUG:boto:Path: /aktlji-16/?prefix=obj34 2024-04-19T23:26:40.869 DEBUG:boto:Data: 2024-04-19T23:26:40.869 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.869 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.869 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.869 DEBUG:boto:Params: {} 2024-04-19T23:26:40.869 DEBUG:boto:Token: None 2024-04-19T23:26:40.870 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.870 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049fdf48c578da6f5-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.872 DEBUG:boto:b'aktlji-16obj341000falseobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.873 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.873 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.873 DEBUG:boto:path=/aktlji-16/?prefix=obj34 2024-04-19T23:26:40.873 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj34 2024-04-19T23:26:40.873 DEBUG:boto:Method: GET 2024-04-19T23:26:40.873 DEBUG:boto:Path: /aktlji-16/?prefix=obj34 2024-04-19T23:26:40.873 DEBUG:boto:Data: 2024-04-19T23:26:40.873 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.874 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.874 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.874 DEBUG:boto:Params: {} 2024-04-19T23:26:40.874 DEBUG:boto:Token: None 2024-04-19T23:26:40.874 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.874 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006059397b82367b45-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.877 DEBUG:boto:b'aktlji-16obj341000falseobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.877 DEBUG:boto:path=/aktlji-16/obj34 2024-04-19T23:26:40.877 DEBUG:boto:auth_path=/aktlji-16/obj34 2024-04-19T23:26:40.877 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.877 DEBUG:boto:Path: /aktlji-16/obj34 2024-04-19T23:26:40.877 DEBUG:boto:Data: 2024-04-19T23:26:40.878 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.878 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.878 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.878 DEBUG:boto:Params: {} 2024-04-19T23:26:40.878 DEBUG:boto:Token: None 2024-04-19T23:26:40.878 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj34 2024-04-19T23:26:40.878 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KNLYOGxyfURCYdY0b8btpDiskcw= 2024-04-19T23:26:40.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KNLYOGxyfURCYdY0b8btpDiskcw=', 'Content-Length': '0'} 2024-04-19T23:26:40.880 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000565e78694eee2a2f-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.880 DEBUG:boto:path=/aktlji-16/obj34 2024-04-19T23:26:40.881 DEBUG:boto:auth_path=/aktlji-16/obj34 2024-04-19T23:26:40.881 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.881 DEBUG:boto:Path: /aktlji-16/obj34 2024-04-19T23:26:40.881 DEBUG:boto:Data: 2024-04-19T23:26:40.881 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.881 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.881 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.881 DEBUG:boto:Params: {} 2024-04-19T23:26:40.881 DEBUG:boto:Token: None 2024-04-19T23:26:40.882 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj34 2024-04-19T23:26:40.882 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KNLYOGxyfURCYdY0b8btpDiskcw= 2024-04-19T23:26:40.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KNLYOGxyfURCYdY0b8btpDiskcw=', 'Content-Length': '0'} 2024-04-19T23:26:40.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee9a3791e3078d15-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-19T23:26:40.884 DEBUG:boto:path=/aktlji-16/obj34 2024-04-19T23:26:40.884 DEBUG:boto:auth_path=/aktlji-16/obj34 2024-04-19T23:26:40.884 DEBUG:boto:Method: GET 2024-04-19T23:26:40.884 DEBUG:boto:Path: /aktlji-16/obj34 2024-04-19T23:26:40.884 DEBUG:boto:Data: 2024-04-19T23:26:40.884 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.884 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.884 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.885 DEBUG:boto:Params: {} 2024-04-19T23:26:40.885 DEBUG:boto:Token: None 2024-04-19T23:26:40.885 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj34 2024-04-19T23:26:40.885 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wg0U/0Qy2oBzNuV3vjZfw/U9ijA= 2024-04-19T23:26:40.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wg0U/0Qy2oBzNuV3vjZfw/U9ijA=', 'Content-Length': '0'} 2024-04-19T23:26:40.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016c576c420daf9a9-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.887 DEBUG:boto:path=/aktlji-16/obj34 2024-04-19T23:26:40.887 DEBUG:boto:auth_path=/aktlji-16/obj34 2024-04-19T23:26:40.887 DEBUG:boto:Method: GET 2024-04-19T23:26:40.887 DEBUG:boto:Path: /aktlji-16/obj34 2024-04-19T23:26:40.888 DEBUG:boto:Data: 2024-04-19T23:26:40.888 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.888 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.888 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.888 DEBUG:boto:Params: {} 2024-04-19T23:26:40.888 DEBUG:boto:Token: None 2024-04-19T23:26:40.888 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj34 2024-04-19T23:26:40.888 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wg0U/0Qy2oBzNuV3vjZfw/U9ijA= 2024-04-19T23:26:40.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wg0U/0Qy2oBzNuV3vjZfw/U9ijA=', 'Content-Length': '0'} 2024-04-19T23:26:40.890 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c9aa85b48db89e1-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.890 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.890 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.890 DEBUG:boto:path=/aktlji-16/?prefix=obj34 2024-04-19T23:26:40.891 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj34 2024-04-19T23:26:40.891 DEBUG:boto:Method: GET 2024-04-19T23:26:40.891 DEBUG:boto:Path: /aktlji-16/?prefix=obj34 2024-04-19T23:26:40.891 DEBUG:boto:Data: 2024-04-19T23:26:40.891 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.891 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.891 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.891 DEBUG:boto:Params: {} 2024-04-19T23:26:40.891 DEBUG:boto:Token: None 2024-04-19T23:26:40.891 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.892 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010330d1f4e793bdf-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.894 DEBUG:boto:b'aktlji-16obj341000falseobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.895 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.895 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.895 DEBUG:boto:path=/aktlji-16/?prefix=obj34 2024-04-19T23:26:40.895 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj34 2024-04-19T23:26:40.895 DEBUG:boto:Method: GET 2024-04-19T23:26:40.895 DEBUG:boto:Path: /aktlji-16/?prefix=obj34 2024-04-19T23:26:40.895 DEBUG:boto:Data: 2024-04-19T23:26:40.895 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.895 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.896 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.896 DEBUG:boto:Params: {} 2024-04-19T23:26:40.896 DEBUG:boto:Token: None 2024-04-19T23:26:40.896 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.896 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004042a16a2bdd2f26-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.898 DEBUG:boto:b'aktlji-16obj341000falseobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj35 2024-04-19T23:26:40.899 DEBUG:boto:path=/aktlji-16/obj35 2024-04-19T23:26:40.899 DEBUG:boto:auth_path=/aktlji-16/obj35 2024-04-19T23:26:40.899 DEBUG:boto:path=/aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.899 DEBUG:boto:auth_path=/aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.899 DEBUG:boto:Method: GET 2024-04-19T23:26:40.899 DEBUG:boto:Path: /aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.899 DEBUG:boto:Data: 2024-04-19T23:26:40.899 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.900 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.900 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.900 DEBUG:boto:Params: {} 2024-04-19T23:26:40.900 DEBUG:boto:Token: None 2024-04-19T23:26:40.900 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.900 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA= 2024-04-19T23:26:40.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA=', 'Content-Length': '0'} 2024-04-19T23:26:40.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1ca6652f6e0d08c-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.946 DEBUG:boto:path=/aktlji-16/obj35 2024-04-19T23:26:40.947 DEBUG:boto:auth_path=/aktlji-16/obj35 2024-04-19T23:26:40.947 DEBUG:boto:path=/aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.947 DEBUG:boto:auth_path=/aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.947 DEBUG:boto:Method: GET 2024-04-19T23:26:40.947 DEBUG:boto:Path: /aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.947 DEBUG:boto:Data: 2024-04-19T23:26:40.947 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.947 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.948 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.948 DEBUG:boto:Params: {} 2024-04-19T23:26:40.948 DEBUG:boto:Token: None 2024-04-19T23:26:40.948 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.948 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA= 2024-04-19T23:26:40.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA=', 'Content-Length': '0'} 2024-04-19T23:26:40.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090665b171e35794e-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.951 DEBUG:boto:path=/aktlji-16/obj35 2024-04-19T23:26:40.951 DEBUG:boto:auth_path=/aktlji-16/obj35 2024-04-19T23:26:40.951 DEBUG:boto:path=/aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.951 DEBUG:boto:auth_path=/aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.951 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.951 DEBUG:boto:Path: /aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.951 DEBUG:boto:Data: 2024-04-19T23:26:40.951 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.952 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.952 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.952 DEBUG:boto:Params: {} 2024-04-19T23:26:40.952 DEBUG:boto:Token: None 2024-04-19T23:26:40.952 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.952 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QytyXqBo9BN4nxnHFScQ9vl3p9I= 2024-04-19T23:26:40.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QytyXqBo9BN4nxnHFScQ9vl3p9I=', 'Content-Length': '0'} 2024-04-19T23:26:40.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d460dab2f3ec3b13-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.954 DEBUG:boto:path=/aktlji-16/obj35 2024-04-19T23:26:40.954 DEBUG:boto:auth_path=/aktlji-16/obj35 2024-04-19T23:26:40.954 DEBUG:boto:path=/aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.954 DEBUG:boto:auth_path=/aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.954 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.955 DEBUG:boto:Path: /aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.955 DEBUG:boto:Data: 2024-04-19T23:26:40.955 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.955 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.955 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.955 DEBUG:boto:Params: {} 2024-04-19T23:26:40.955 DEBUG:boto:Token: None 2024-04-19T23:26:40.955 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.955 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QytyXqBo9BN4nxnHFScQ9vl3p9I= 2024-04-19T23:26:40.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QytyXqBo9BN4nxnHFScQ9vl3p9I=', 'Content-Length': '0'} 2024-04-19T23:26:40.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7b7214b90fcb014-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj35 2024-04-19T23:26:40.958 DEBUG:boto:path=/aktlji-16/obj35 2024-04-19T23:26:40.958 DEBUG:boto:auth_path=/aktlji-16/obj35 2024-04-19T23:26:40.958 DEBUG:boto:path=/aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.958 DEBUG:boto:auth_path=/aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.958 DEBUG:boto:Method: GET 2024-04-19T23:26:40.958 DEBUG:boto:Path: /aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.958 DEBUG:boto:Data: 2024-04-19T23:26:40.958 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.958 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.959 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.959 DEBUG:boto:Params: {} 2024-04-19T23:26:40.959 DEBUG:boto:Token: None 2024-04-19T23:26:40.959 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.959 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA= 2024-04-19T23:26:40.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA=', 'Content-Length': '0'} 2024-04-19T23:26:40.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f67fe97e3579090c-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.961 DEBUG:boto:path=/aktlji-16/obj35 2024-04-19T23:26:40.961 DEBUG:boto:auth_path=/aktlji-16/obj35 2024-04-19T23:26:40.961 DEBUG:boto:path=/aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.961 DEBUG:boto:auth_path=/aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.962 DEBUG:boto:Method: GET 2024-04-19T23:26:40.962 DEBUG:boto:Path: /aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.962 DEBUG:boto:Data: 2024-04-19T23:26:40.962 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.962 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.962 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.962 DEBUG:boto:Params: {} 2024-04-19T23:26:40.962 DEBUG:boto:Token: None 2024-04-19T23:26:40.962 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj35?versionId=null 2024-04-19T23:26:40.963 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA= 2024-04-19T23:26:40.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA=', 'Content-Length': '0'} 2024-04-19T23:26:40.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000498f15a17b71861-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.965 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.965 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.965 DEBUG:boto:path=/aktlji-16/?prefix=obj35 2024-04-19T23:26:40.965 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj35 2024-04-19T23:26:40.965 DEBUG:boto:Method: GET 2024-04-19T23:26:40.965 DEBUG:boto:Path: /aktlji-16/?prefix=obj35 2024-04-19T23:26:40.965 DEBUG:boto:Data: 2024-04-19T23:26:40.965 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.965 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.966 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.966 DEBUG:boto:Params: {} 2024-04-19T23:26:40.966 DEBUG:boto:Token: None 2024-04-19T23:26:40.966 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.966 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077de279d547ecf39-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.969 DEBUG:boto:b'aktlji-16obj351000falseobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.969 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.969 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.969 DEBUG:boto:path=/aktlji-16/?prefix=obj35 2024-04-19T23:26:40.969 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj35 2024-04-19T23:26:40.970 DEBUG:boto:Method: GET 2024-04-19T23:26:40.970 DEBUG:boto:Path: /aktlji-16/?prefix=obj35 2024-04-19T23:26:40.970 DEBUG:boto:Data: 2024-04-19T23:26:40.970 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.970 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.970 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.970 DEBUG:boto:Params: {} 2024-04-19T23:26:40.970 DEBUG:boto:Token: None 2024-04-19T23:26:40.970 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.970 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004457ce372fe90d98-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.973 DEBUG:boto:b'aktlji-16obj351000falseobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.973 DEBUG:boto:path=/aktlji-16/obj35 2024-04-19T23:26:40.973 DEBUG:boto:auth_path=/aktlji-16/obj35 2024-04-19T23:26:40.974 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.974 DEBUG:boto:Path: /aktlji-16/obj35 2024-04-19T23:26:40.974 DEBUG:boto:Data: 2024-04-19T23:26:40.974 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.974 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.974 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.974 DEBUG:boto:Params: {} 2024-04-19T23:26:40.974 DEBUG:boto:Token: None 2024-04-19T23:26:40.974 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj35 2024-04-19T23:26:40.975 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:n6aSx7JXg/+fIa4gYHUL9aJ1I5s= 2024-04-19T23:26:40.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:n6aSx7JXg/+fIa4gYHUL9aJ1I5s=', 'Content-Length': '0'} 2024-04-19T23:26:40.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b283a17575ae90c3-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.977 DEBUG:boto:path=/aktlji-16/obj35 2024-04-19T23:26:40.977 DEBUG:boto:auth_path=/aktlji-16/obj35 2024-04-19T23:26:40.977 DEBUG:boto:Method: HEAD 2024-04-19T23:26:40.977 DEBUG:boto:Path: /aktlji-16/obj35 2024-04-19T23:26:40.977 DEBUG:boto:Data: 2024-04-19T23:26:40.977 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.977 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.977 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.977 DEBUG:boto:Params: {} 2024-04-19T23:26:40.977 DEBUG:boto:Token: None 2024-04-19T23:26:40.978 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj35 2024-04-19T23:26:40.978 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:n6aSx7JXg/+fIa4gYHUL9aJ1I5s= 2024-04-19T23:26:40.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:n6aSx7JXg/+fIa4gYHUL9aJ1I5s=', 'Content-Length': '0'} 2024-04-19T23:26:40.979 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a54ca095fe55e41a-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj35 2024-04-19T23:26:40.980 DEBUG:boto:path=/aktlji-16/obj35 2024-04-19T23:26:40.980 DEBUG:boto:auth_path=/aktlji-16/obj35 2024-04-19T23:26:40.980 DEBUG:boto:Method: GET 2024-04-19T23:26:40.980 DEBUG:boto:Path: /aktlji-16/obj35 2024-04-19T23:26:40.980 DEBUG:boto:Data: 2024-04-19T23:26:40.981 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.981 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.981 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.981 DEBUG:boto:Params: {} 2024-04-19T23:26:40.981 DEBUG:boto:Token: None 2024-04-19T23:26:40.981 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj35 2024-04-19T23:26:40.981 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AmOun2s389rU3j1gilJUHKoZGbA= 2024-04-19T23:26:40.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AmOun2s389rU3j1gilJUHKoZGbA=', 'Content-Length': '0'} 2024-04-19T23:26:40.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7cfa6dae46c1ca1-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.984 DEBUG:boto:path=/aktlji-16/obj35 2024-04-19T23:26:40.984 DEBUG:boto:auth_path=/aktlji-16/obj35 2024-04-19T23:26:40.984 DEBUG:boto:Method: GET 2024-04-19T23:26:40.984 DEBUG:boto:Path: /aktlji-16/obj35 2024-04-19T23:26:40.984 DEBUG:boto:Data: 2024-04-19T23:26:40.984 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.984 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.984 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.984 DEBUG:boto:Params: {} 2024-04-19T23:26:40.985 DEBUG:boto:Token: None 2024-04-19T23:26:40.985 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/obj35 2024-04-19T23:26:40.985 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AmOun2s389rU3j1gilJUHKoZGbA= 2024-04-19T23:26:40.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AmOun2s389rU3j1gilJUHKoZGbA=', 'Content-Length': '0'} 2024-04-19T23:26:40.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0e68498988026d1-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.987 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.987 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.987 DEBUG:boto:path=/aktlji-16/?prefix=obj35 2024-04-19T23:26:40.987 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj35 2024-04-19T23:26:40.987 DEBUG:boto:Method: GET 2024-04-19T23:26:40.987 DEBUG:boto:Path: /aktlji-16/?prefix=obj35 2024-04-19T23:26:40.988 DEBUG:boto:Data: 2024-04-19T23:26:40.988 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.988 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:40.988 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.988 DEBUG:boto:Params: {} 2024-04-19T23:26:40.988 DEBUG:boto:Token: None 2024-04-19T23:26:40.988 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.988 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089cfea7c3b1994be-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:40.991 DEBUG:boto:b'aktlji-16obj351000falseobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:40.991 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:40.991 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:40.991 DEBUG:boto:path=/aktlji-16/?prefix=obj35 2024-04-19T23:26:40.992 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj35 2024-04-19T23:26:40.992 DEBUG:boto:Method: GET 2024-04-19T23:26:40.992 DEBUG:boto:Path: /aktlji-16/?prefix=obj35 2024-04-19T23:26:40.992 DEBUG:boto:Data: 2024-04-19T23:26:40.992 DEBUG:boto:Headers: {} 2024-04-19T23:26:40.992 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:40.992 DEBUG:boto:Port: 8000 2024-04-19T23:26:40.992 DEBUG:boto:Params: {} 2024-04-19T23:26:40.992 DEBUG:boto:Token: None 2024-04-19T23:26:40.992 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:40 GMT /aktlji-16/ 2024-04-19T23:26:40.993 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-19T23:26:40.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-19T23:26:40.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eb02fa96751d436-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.038 DEBUG:boto:b'aktlji-16obj351000falseobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj36 2024-04-19T23:26:41.039 DEBUG:boto:path=/aktlji-16/obj36 2024-04-19T23:26:41.039 DEBUG:boto:auth_path=/aktlji-16/obj36 2024-04-19T23:26:41.039 DEBUG:boto:path=/aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.039 DEBUG:boto:auth_path=/aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.039 DEBUG:boto:Method: GET 2024-04-19T23:26:41.039 DEBUG:boto:Path: /aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.040 DEBUG:boto:Data: 2024-04-19T23:26:41.040 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.040 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.040 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.040 DEBUG:boto:Params: {} 2024-04-19T23:26:41.040 DEBUG:boto:Token: None 2024-04-19T23:26:41.040 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.040 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0= 2024-04-19T23:26:41.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0=', 'Content-Length': '0'} 2024-04-19T23:26:41.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034bd8d293e07c3e4-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.043 DEBUG:boto:path=/aktlji-16/obj36 2024-04-19T23:26:41.043 DEBUG:boto:auth_path=/aktlji-16/obj36 2024-04-19T23:26:41.043 DEBUG:boto:path=/aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.043 DEBUG:boto:auth_path=/aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.043 DEBUG:boto:Method: GET 2024-04-19T23:26:41.043 DEBUG:boto:Path: /aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.043 DEBUG:boto:Data: 2024-04-19T23:26:41.043 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.043 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.043 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.044 DEBUG:boto:Params: {} 2024-04-19T23:26:41.044 DEBUG:boto:Token: None 2024-04-19T23:26:41.044 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.044 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0= 2024-04-19T23:26:41.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0=', 'Content-Length': '0'} 2024-04-19T23:26:41.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000237cdae0a618ce17-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.046 DEBUG:boto:path=/aktlji-16/obj36 2024-04-19T23:26:41.047 DEBUG:boto:auth_path=/aktlji-16/obj36 2024-04-19T23:26:41.047 DEBUG:boto:path=/aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.047 DEBUG:boto:auth_path=/aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.047 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.047 DEBUG:boto:Path: /aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.047 DEBUG:boto:Data: 2024-04-19T23:26:41.047 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.047 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.047 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.047 DEBUG:boto:Params: {} 2024-04-19T23:26:41.048 DEBUG:boto:Token: None 2024-04-19T23:26:41.048 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.048 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tN6fe4rQr+5rHkZP5UwBxUKqFyI= 2024-04-19T23:26:41.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tN6fe4rQr+5rHkZP5UwBxUKqFyI=', 'Content-Length': '0'} 2024-04-19T23:26:41.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3cfc3976b2827fc-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.050 DEBUG:boto:path=/aktlji-16/obj36 2024-04-19T23:26:41.050 DEBUG:boto:auth_path=/aktlji-16/obj36 2024-04-19T23:26:41.050 DEBUG:boto:path=/aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.050 DEBUG:boto:auth_path=/aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.050 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.050 DEBUG:boto:Path: /aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.050 DEBUG:boto:Data: 2024-04-19T23:26:41.051 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.051 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.051 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.051 DEBUG:boto:Params: {} 2024-04-19T23:26:41.051 DEBUG:boto:Token: None 2024-04-19T23:26:41.051 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.051 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tN6fe4rQr+5rHkZP5UwBxUKqFyI= 2024-04-19T23:26:41.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tN6fe4rQr+5rHkZP5UwBxUKqFyI=', 'Content-Length': '0'} 2024-04-19T23:26:41.053 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009514a1bca6bc9a7-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj36 2024-04-19T23:26:41.053 DEBUG:boto:path=/aktlji-16/obj36 2024-04-19T23:26:41.054 DEBUG:boto:auth_path=/aktlji-16/obj36 2024-04-19T23:26:41.054 DEBUG:boto:path=/aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.054 DEBUG:boto:auth_path=/aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.054 DEBUG:boto:Method: GET 2024-04-19T23:26:41.054 DEBUG:boto:Path: /aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.054 DEBUG:boto:Data: 2024-04-19T23:26:41.054 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.054 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.054 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.054 DEBUG:boto:Params: {} 2024-04-19T23:26:41.055 DEBUG:boto:Token: None 2024-04-19T23:26:41.055 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.055 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0= 2024-04-19T23:26:41.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0=', 'Content-Length': '0'} 2024-04-19T23:26:41.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f109ccc7b0b60b8-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.098 DEBUG:boto:path=/aktlji-16/obj36 2024-04-19T23:26:41.099 DEBUG:boto:auth_path=/aktlji-16/obj36 2024-04-19T23:26:41.099 DEBUG:boto:path=/aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.099 DEBUG:boto:auth_path=/aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.099 DEBUG:boto:Method: GET 2024-04-19T23:26:41.099 DEBUG:boto:Path: /aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.099 DEBUG:boto:Data: 2024-04-19T23:26:41.099 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.099 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.099 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.099 DEBUG:boto:Params: {} 2024-04-19T23:26:41.100 DEBUG:boto:Token: None 2024-04-19T23:26:41.100 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj36?versionId=null 2024-04-19T23:26:41.100 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0= 2024-04-19T23:26:41.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0=', 'Content-Length': '0'} 2024-04-19T23:26:41.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d63b6a54d499a7ce-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.102 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.102 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.102 DEBUG:boto:path=/aktlji-16/?prefix=obj36 2024-04-19T23:26:41.102 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj36 2024-04-19T23:26:41.102 DEBUG:boto:Method: GET 2024-04-19T23:26:41.103 DEBUG:boto:Path: /aktlji-16/?prefix=obj36 2024-04-19T23:26:41.103 DEBUG:boto:Data: 2024-04-19T23:26:41.103 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.103 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.103 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.103 DEBUG:boto:Params: {} 2024-04-19T23:26:41.103 DEBUG:boto:Token: None 2024-04-19T23:26:41.103 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.103 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2922c6aeafe5c49-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.106 DEBUG:boto:b'aktlji-16obj361000falseobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.106 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.107 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.107 DEBUG:boto:path=/aktlji-16/?prefix=obj36 2024-04-19T23:26:41.107 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj36 2024-04-19T23:26:41.107 DEBUG:boto:Method: GET 2024-04-19T23:26:41.107 DEBUG:boto:Path: /aktlji-16/?prefix=obj36 2024-04-19T23:26:41.107 DEBUG:boto:Data: 2024-04-19T23:26:41.107 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.107 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.107 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.107 DEBUG:boto:Params: {} 2024-04-19T23:26:41.108 DEBUG:boto:Token: None 2024-04-19T23:26:41.108 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.108 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7347d9a51752745-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.110 DEBUG:boto:b'aktlji-16obj361000falseobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.111 DEBUG:boto:path=/aktlji-16/obj36 2024-04-19T23:26:41.111 DEBUG:boto:auth_path=/aktlji-16/obj36 2024-04-19T23:26:41.111 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.111 DEBUG:boto:Path: /aktlji-16/obj36 2024-04-19T23:26:41.111 DEBUG:boto:Data: 2024-04-19T23:26:41.111 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.111 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.111 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.111 DEBUG:boto:Params: {} 2024-04-19T23:26:41.111 DEBUG:boto:Token: None 2024-04-19T23:26:41.112 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj36 2024-04-19T23:26:41.112 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dLf0AhMr014+dm+83tkZkdJnlk0= 2024-04-19T23:26:41.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dLf0AhMr014+dm+83tkZkdJnlk0=', 'Content-Length': '0'} 2024-04-19T23:26:41.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fd666940a791dbd-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.113 DEBUG:boto:path=/aktlji-16/obj36 2024-04-19T23:26:41.114 DEBUG:boto:auth_path=/aktlji-16/obj36 2024-04-19T23:26:41.114 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.114 DEBUG:boto:Path: /aktlji-16/obj36 2024-04-19T23:26:41.114 DEBUG:boto:Data: 2024-04-19T23:26:41.114 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.114 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.114 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.114 DEBUG:boto:Params: {} 2024-04-19T23:26:41.114 DEBUG:boto:Token: None 2024-04-19T23:26:41.115 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj36 2024-04-19T23:26:41.115 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dLf0AhMr014+dm+83tkZkdJnlk0= 2024-04-19T23:26:41.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dLf0AhMr014+dm+83tkZkdJnlk0=', 'Content-Length': '0'} 2024-04-19T23:26:41.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a1670f0a6496c14-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj36 2024-04-19T23:26:41.117 DEBUG:boto:path=/aktlji-16/obj36 2024-04-19T23:26:41.117 DEBUG:boto:auth_path=/aktlji-16/obj36 2024-04-19T23:26:41.117 DEBUG:boto:Method: GET 2024-04-19T23:26:41.117 DEBUG:boto:Path: /aktlji-16/obj36 2024-04-19T23:26:41.117 DEBUG:boto:Data: 2024-04-19T23:26:41.117 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.117 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.118 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.118 DEBUG:boto:Params: {} 2024-04-19T23:26:41.118 DEBUG:boto:Token: None 2024-04-19T23:26:41.118 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj36 2024-04-19T23:26:41.118 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3VKMbRYqwU6HhtoTdwvo3B1LW7M= 2024-04-19T23:26:41.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3VKMbRYqwU6HhtoTdwvo3B1LW7M=', 'Content-Length': '0'} 2024-04-19T23:26:41.120 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005928f09638f3f10e-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.120 DEBUG:boto:path=/aktlji-16/obj36 2024-04-19T23:26:41.120 DEBUG:boto:auth_path=/aktlji-16/obj36 2024-04-19T23:26:41.120 DEBUG:boto:Method: GET 2024-04-19T23:26:41.120 DEBUG:boto:Path: /aktlji-16/obj36 2024-04-19T23:26:41.120 DEBUG:boto:Data: 2024-04-19T23:26:41.121 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.121 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.121 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.121 DEBUG:boto:Params: {} 2024-04-19T23:26:41.121 DEBUG:boto:Token: None 2024-04-19T23:26:41.121 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj36 2024-04-19T23:26:41.121 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3VKMbRYqwU6HhtoTdwvo3B1LW7M= 2024-04-19T23:26:41.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3VKMbRYqwU6HhtoTdwvo3B1LW7M=', 'Content-Length': '0'} 2024-04-19T23:26:41.123 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f049c2b334183ab-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.123 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.124 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.124 DEBUG:boto:path=/aktlji-16/?prefix=obj36 2024-04-19T23:26:41.124 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj36 2024-04-19T23:26:41.124 DEBUG:boto:Method: GET 2024-04-19T23:26:41.124 DEBUG:boto:Path: /aktlji-16/?prefix=obj36 2024-04-19T23:26:41.124 DEBUG:boto:Data: 2024-04-19T23:26:41.124 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.124 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.124 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.124 DEBUG:boto:Params: {} 2024-04-19T23:26:41.124 DEBUG:boto:Token: None 2024-04-19T23:26:41.125 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.125 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b3fec427662ad19-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.127 DEBUG:boto:b'aktlji-16obj361000falseobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.128 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.128 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.128 DEBUG:boto:path=/aktlji-16/?prefix=obj36 2024-04-19T23:26:41.128 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj36 2024-04-19T23:26:41.128 DEBUG:boto:Method: GET 2024-04-19T23:26:41.128 DEBUG:boto:Path: /aktlji-16/?prefix=obj36 2024-04-19T23:26:41.128 DEBUG:boto:Data: 2024-04-19T23:26:41.128 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.128 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.128 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.129 DEBUG:boto:Params: {} 2024-04-19T23:26:41.129 DEBUG:boto:Token: None 2024-04-19T23:26:41.129 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.129 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037a6a8a65e7ab002-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.174 DEBUG:boto:b'aktlji-16obj361000falseobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-19T23:26:41.175 DEBUG:boto:path=/aktlji-16/obj37 2024-04-19T23:26:41.175 DEBUG:boto:auth_path=/aktlji-16/obj37 2024-04-19T23:26:41.175 DEBUG:boto:path=/aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.175 DEBUG:boto:auth_path=/aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.175 DEBUG:boto:Method: GET 2024-04-19T23:26:41.175 DEBUG:boto:Path: /aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.175 DEBUG:boto:Data: 2024-04-19T23:26:41.176 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.176 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.176 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.176 DEBUG:boto:Params: {} 2024-04-19T23:26:41.176 DEBUG:boto:Token: None 2024-04-19T23:26:41.176 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.176 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0= 2024-04-19T23:26:41.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0=', 'Content-Length': '0'} 2024-04-19T23:26:41.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000564fc71c45f482de-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.178 DEBUG:boto:path=/aktlji-16/obj37 2024-04-19T23:26:41.179 DEBUG:boto:auth_path=/aktlji-16/obj37 2024-04-19T23:26:41.179 DEBUG:boto:path=/aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.179 DEBUG:boto:auth_path=/aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.179 DEBUG:boto:Method: GET 2024-04-19T23:26:41.179 DEBUG:boto:Path: /aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.179 DEBUG:boto:Data: 2024-04-19T23:26:41.179 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.179 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.179 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.179 DEBUG:boto:Params: {} 2024-04-19T23:26:41.179 DEBUG:boto:Token: None 2024-04-19T23:26:41.180 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0= 2024-04-19T23:26:41.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0=', 'Content-Length': '0'} 2024-04-19T23:26:41.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b431c59d8e8efb0-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.182 DEBUG:boto:path=/aktlji-16/obj37 2024-04-19T23:26:41.182 DEBUG:boto:auth_path=/aktlji-16/obj37 2024-04-19T23:26:41.182 DEBUG:boto:path=/aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.182 DEBUG:boto:auth_path=/aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.182 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.182 DEBUG:boto:Path: /aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.182 DEBUG:boto:Data: 2024-04-19T23:26:41.182 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.182 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.183 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.183 DEBUG:boto:Params: {} 2024-04-19T23:26:41.183 DEBUG:boto:Token: None 2024-04-19T23:26:41.183 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.183 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:K6oDwdvPzN4jhQX5lvfdEyWtxTI= 2024-04-19T23:26:41.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K6oDwdvPzN4jhQX5lvfdEyWtxTI=', 'Content-Length': '0'} 2024-04-19T23:26:41.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f76e3596eb4b9295-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.185 DEBUG:boto:path=/aktlji-16/obj37 2024-04-19T23:26:41.185 DEBUG:boto:auth_path=/aktlji-16/obj37 2024-04-19T23:26:41.186 DEBUG:boto:path=/aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.186 DEBUG:boto:auth_path=/aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.186 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.186 DEBUG:boto:Path: /aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.187 DEBUG:boto:Data: 2024-04-19T23:26:41.187 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.187 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.187 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.187 DEBUG:boto:Params: {} 2024-04-19T23:26:41.188 DEBUG:boto:Token: None 2024-04-19T23:26:41.188 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.188 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:K6oDwdvPzN4jhQX5lvfdEyWtxTI= 2024-04-19T23:26:41.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K6oDwdvPzN4jhQX5lvfdEyWtxTI=', 'Content-Length': '0'} 2024-04-19T23:26:41.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bc93d67c298b8af-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-19T23:26:41.190 DEBUG:boto:path=/aktlji-16/obj37 2024-04-19T23:26:41.190 DEBUG:boto:auth_path=/aktlji-16/obj37 2024-04-19T23:26:41.190 DEBUG:boto:path=/aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.190 DEBUG:boto:auth_path=/aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.191 DEBUG:boto:Method: GET 2024-04-19T23:26:41.191 DEBUG:boto:Path: /aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.191 DEBUG:boto:Data: 2024-04-19T23:26:41.191 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.191 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.191 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.191 DEBUG:boto:Params: {} 2024-04-19T23:26:41.191 DEBUG:boto:Token: None 2024-04-19T23:26:41.191 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.191 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0= 2024-04-19T23:26:41.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0=', 'Content-Length': '0'} 2024-04-19T23:26:41.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca09dc9c72c13a77-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.193 DEBUG:boto:path=/aktlji-16/obj37 2024-04-19T23:26:41.194 DEBUG:boto:auth_path=/aktlji-16/obj37 2024-04-19T23:26:41.194 DEBUG:boto:path=/aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.194 DEBUG:boto:auth_path=/aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.194 DEBUG:boto:Method: GET 2024-04-19T23:26:41.194 DEBUG:boto:Path: /aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.194 DEBUG:boto:Data: 2024-04-19T23:26:41.194 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.194 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.194 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.194 DEBUG:boto:Params: {} 2024-04-19T23:26:41.194 DEBUG:boto:Token: None 2024-04-19T23:26:41.195 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj37?versionId=null 2024-04-19T23:26:41.195 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0= 2024-04-19T23:26:41.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0=', 'Content-Length': '0'} 2024-04-19T23:26:41.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f98751ab5fec3029-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.197 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.197 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.197 DEBUG:boto:path=/aktlji-16/?prefix=obj37 2024-04-19T23:26:41.197 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj37 2024-04-19T23:26:41.197 DEBUG:boto:Method: GET 2024-04-19T23:26:41.197 DEBUG:boto:Path: /aktlji-16/?prefix=obj37 2024-04-19T23:26:41.197 DEBUG:boto:Data: 2024-04-19T23:26:41.197 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.197 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.197 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.197 DEBUG:boto:Params: {} 2024-04-19T23:26:41.198 DEBUG:boto:Token: None 2024-04-19T23:26:41.198 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.198 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000015615b1effc5cd3-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.200 DEBUG:boto:b'aktlji-16obj371000falseobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.201 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.201 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.201 DEBUG:boto:path=/aktlji-16/?prefix=obj37 2024-04-19T23:26:41.201 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj37 2024-04-19T23:26:41.201 DEBUG:boto:Method: GET 2024-04-19T23:26:41.201 DEBUG:boto:Path: /aktlji-16/?prefix=obj37 2024-04-19T23:26:41.201 DEBUG:boto:Data: 2024-04-19T23:26:41.201 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.201 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.201 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.201 DEBUG:boto:Params: {} 2024-04-19T23:26:41.202 DEBUG:boto:Token: None 2024-04-19T23:26:41.202 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.202 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ef8dda4ad5d9d3e-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.205 DEBUG:boto:b'aktlji-16obj371000falseobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.205 DEBUG:boto:path=/aktlji-16/obj37 2024-04-19T23:26:41.205 DEBUG:boto:auth_path=/aktlji-16/obj37 2024-04-19T23:26:41.205 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.205 DEBUG:boto:Path: /aktlji-16/obj37 2024-04-19T23:26:41.205 DEBUG:boto:Data: 2024-04-19T23:26:41.205 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.205 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.206 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.206 DEBUG:boto:Params: {} 2024-04-19T23:26:41.206 DEBUG:boto:Token: None 2024-04-19T23:26:41.206 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj37 2024-04-19T23:26:41.206 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XlV5PnxnXqBMuScnZhB2+Fzdpik= 2024-04-19T23:26:41.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XlV5PnxnXqBMuScnZhB2+Fzdpik=', 'Content-Length': '0'} 2024-04-19T23:26:41.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c6fd09e30153133-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.208 DEBUG:boto:path=/aktlji-16/obj37 2024-04-19T23:26:41.208 DEBUG:boto:auth_path=/aktlji-16/obj37 2024-04-19T23:26:41.208 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.208 DEBUG:boto:Path: /aktlji-16/obj37 2024-04-19T23:26:41.208 DEBUG:boto:Data: 2024-04-19T23:26:41.208 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.208 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.209 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.209 DEBUG:boto:Params: {} 2024-04-19T23:26:41.209 DEBUG:boto:Token: None 2024-04-19T23:26:41.209 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj37 2024-04-19T23:26:41.209 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XlV5PnxnXqBMuScnZhB2+Fzdpik= 2024-04-19T23:26:41.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XlV5PnxnXqBMuScnZhB2+Fzdpik=', 'Content-Length': '0'} 2024-04-19T23:26:41.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050b6603aa7f719cb-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-19T23:26:41.211 DEBUG:boto:path=/aktlji-16/obj37 2024-04-19T23:26:41.212 DEBUG:boto:auth_path=/aktlji-16/obj37 2024-04-19T23:26:41.212 DEBUG:boto:Method: GET 2024-04-19T23:26:41.212 DEBUG:boto:Path: /aktlji-16/obj37 2024-04-19T23:26:41.212 DEBUG:boto:Data: 2024-04-19T23:26:41.212 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.212 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.212 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.212 DEBUG:boto:Params: {} 2024-04-19T23:26:41.212 DEBUG:boto:Token: None 2024-04-19T23:26:41.212 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj37 2024-04-19T23:26:41.213 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x5IQvbUZIZ9kFDN7OdLzAyHMBaA= 2024-04-19T23:26:41.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x5IQvbUZIZ9kFDN7OdLzAyHMBaA=', 'Content-Length': '0'} 2024-04-19T23:26:41.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca82b3eeebdeb04c-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.215 DEBUG:boto:path=/aktlji-16/obj37 2024-04-19T23:26:41.215 DEBUG:boto:auth_path=/aktlji-16/obj37 2024-04-19T23:26:41.215 DEBUG:boto:Method: GET 2024-04-19T23:26:41.215 DEBUG:boto:Path: /aktlji-16/obj37 2024-04-19T23:26:41.215 DEBUG:boto:Data: 2024-04-19T23:26:41.215 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.215 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.215 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.215 DEBUG:boto:Params: {} 2024-04-19T23:26:41.215 DEBUG:boto:Token: None 2024-04-19T23:26:41.216 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj37 2024-04-19T23:26:41.216 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x5IQvbUZIZ9kFDN7OdLzAyHMBaA= 2024-04-19T23:26:41.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x5IQvbUZIZ9kFDN7OdLzAyHMBaA=', 'Content-Length': '0'} 2024-04-19T23:26:41.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f95e6a5165d2774-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.218 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.218 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.218 DEBUG:boto:path=/aktlji-16/?prefix=obj37 2024-04-19T23:26:41.218 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj37 2024-04-19T23:26:41.218 DEBUG:boto:Method: GET 2024-04-19T23:26:41.218 DEBUG:boto:Path: /aktlji-16/?prefix=obj37 2024-04-19T23:26:41.218 DEBUG:boto:Data: 2024-04-19T23:26:41.219 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.219 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.219 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.219 DEBUG:boto:Params: {} 2024-04-19T23:26:41.219 DEBUG:boto:Token: None 2024-04-19T23:26:41.219 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.219 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d52be11d12f8d86-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.222 DEBUG:boto:b'aktlji-16obj371000falseobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.222 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.222 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.222 DEBUG:boto:path=/aktlji-16/?prefix=obj37 2024-04-19T23:26:41.223 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj37 2024-04-19T23:26:41.223 DEBUG:boto:Method: GET 2024-04-19T23:26:41.223 DEBUG:boto:Path: /aktlji-16/?prefix=obj37 2024-04-19T23:26:41.223 DEBUG:boto:Data: 2024-04-19T23:26:41.223 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.223 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.223 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.223 DEBUG:boto:Params: {} 2024-04-19T23:26:41.223 DEBUG:boto:Token: None 2024-04-19T23:26:41.223 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.223 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cbbc79d456e5faf-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.226 DEBUG:boto:b'aktlji-16obj371000falseobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-19T23:26:41.227 DEBUG:boto:path=/aktlji-16/obj38 2024-04-19T23:26:41.227 DEBUG:boto:auth_path=/aktlji-16/obj38 2024-04-19T23:26:41.227 DEBUG:boto:path=/aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.227 DEBUG:boto:auth_path=/aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.227 DEBUG:boto:Method: GET 2024-04-19T23:26:41.228 DEBUG:boto:Path: /aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.228 DEBUG:boto:Data: 2024-04-19T23:26:41.228 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.228 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.228 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.228 DEBUG:boto:Params: {} 2024-04-19T23:26:41.228 DEBUG:boto:Token: None 2024-04-19T23:26:41.228 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.228 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM= 2024-04-19T23:26:41.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM=', 'Content-Length': '0'} 2024-04-19T23:26:41.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000697513f3650f134d-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.231 DEBUG:boto:path=/aktlji-16/obj38 2024-04-19T23:26:41.231 DEBUG:boto:auth_path=/aktlji-16/obj38 2024-04-19T23:26:41.231 DEBUG:boto:path=/aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.231 DEBUG:boto:auth_path=/aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.231 DEBUG:boto:Method: GET 2024-04-19T23:26:41.231 DEBUG:boto:Path: /aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.231 DEBUG:boto:Data: 2024-04-19T23:26:41.231 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.231 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.231 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.232 DEBUG:boto:Params: {} 2024-04-19T23:26:41.232 DEBUG:boto:Token: None 2024-04-19T23:26:41.232 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.232 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM= 2024-04-19T23:26:41.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM=', 'Content-Length': '0'} 2024-04-19T23:26:41.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5bda94716750b21-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.234 DEBUG:boto:path=/aktlji-16/obj38 2024-04-19T23:26:41.234 DEBUG:boto:auth_path=/aktlji-16/obj38 2024-04-19T23:26:41.234 DEBUG:boto:path=/aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.234 DEBUG:boto:auth_path=/aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.235 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.235 DEBUG:boto:Path: /aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.235 DEBUG:boto:Data: 2024-04-19T23:26:41.235 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.235 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.235 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.235 DEBUG:boto:Params: {} 2024-04-19T23:26:41.235 DEBUG:boto:Token: None 2024-04-19T23:26:41.235 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.235 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XZuwX8PSqyPEVtEkwBkZmc1BmzA= 2024-04-19T23:26:41.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XZuwX8PSqyPEVtEkwBkZmc1BmzA=', 'Content-Length': '0'} 2024-04-19T23:26:41.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000449f06d43677c243-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.238 DEBUG:boto:path=/aktlji-16/obj38 2024-04-19T23:26:41.238 DEBUG:boto:auth_path=/aktlji-16/obj38 2024-04-19T23:26:41.238 DEBUG:boto:path=/aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.238 DEBUG:boto:auth_path=/aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.238 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.238 DEBUG:boto:Path: /aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.238 DEBUG:boto:Data: 2024-04-19T23:26:41.238 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.238 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.238 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.238 DEBUG:boto:Params: {} 2024-04-19T23:26:41.239 DEBUG:boto:Token: None 2024-04-19T23:26:41.239 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.239 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XZuwX8PSqyPEVtEkwBkZmc1BmzA= 2024-04-19T23:26:41.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XZuwX8PSqyPEVtEkwBkZmc1BmzA=', 'Content-Length': '0'} 2024-04-19T23:26:41.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be23a9713606df88-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-19T23:26:41.241 DEBUG:boto:path=/aktlji-16/obj38 2024-04-19T23:26:41.241 DEBUG:boto:auth_path=/aktlji-16/obj38 2024-04-19T23:26:41.241 DEBUG:boto:path=/aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.241 DEBUG:boto:auth_path=/aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.241 DEBUG:boto:Method: GET 2024-04-19T23:26:41.241 DEBUG:boto:Path: /aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.241 DEBUG:boto:Data: 2024-04-19T23:26:41.241 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.242 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.242 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.242 DEBUG:boto:Params: {} 2024-04-19T23:26:41.242 DEBUG:boto:Token: None 2024-04-19T23:26:41.242 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.242 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM= 2024-04-19T23:26:41.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM=', 'Content-Length': '0'} 2024-04-19T23:26:41.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8be4dea28e3fdea-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.286 DEBUG:boto:path=/aktlji-16/obj38 2024-04-19T23:26:41.286 DEBUG:boto:auth_path=/aktlji-16/obj38 2024-04-19T23:26:41.287 DEBUG:boto:path=/aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.287 DEBUG:boto:auth_path=/aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.287 DEBUG:boto:Method: GET 2024-04-19T23:26:41.287 DEBUG:boto:Path: /aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.287 DEBUG:boto:Data: 2024-04-19T23:26:41.287 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.287 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.287 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.287 DEBUG:boto:Params: {} 2024-04-19T23:26:41.287 DEBUG:boto:Token: None 2024-04-19T23:26:41.288 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj38?versionId=null 2024-04-19T23:26:41.288 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM= 2024-04-19T23:26:41.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM=', 'Content-Length': '0'} 2024-04-19T23:26:41.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c0d81f485691331-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.290 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.290 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.290 DEBUG:boto:path=/aktlji-16/?prefix=obj38 2024-04-19T23:26:41.290 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj38 2024-04-19T23:26:41.291 DEBUG:boto:Method: GET 2024-04-19T23:26:41.291 DEBUG:boto:Path: /aktlji-16/?prefix=obj38 2024-04-19T23:26:41.291 DEBUG:boto:Data: 2024-04-19T23:26:41.291 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.291 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.291 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.291 DEBUG:boto:Params: {} 2024-04-19T23:26:41.291 DEBUG:boto:Token: None 2024-04-19T23:26:41.291 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.291 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4d70369ece99509-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.294 DEBUG:boto:b'aktlji-16obj381000falseobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.295 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.295 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.295 DEBUG:boto:path=/aktlji-16/?prefix=obj38 2024-04-19T23:26:41.295 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj38 2024-04-19T23:26:41.295 DEBUG:boto:Method: GET 2024-04-19T23:26:41.295 DEBUG:boto:Path: /aktlji-16/?prefix=obj38 2024-04-19T23:26:41.295 DEBUG:boto:Data: 2024-04-19T23:26:41.295 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.295 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.296 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.296 DEBUG:boto:Params: {} 2024-04-19T23:26:41.296 DEBUG:boto:Token: None 2024-04-19T23:26:41.296 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.296 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037c7a13a473646ec-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.298 DEBUG:boto:b'aktlji-16obj381000falseobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.299 DEBUG:boto:path=/aktlji-16/obj38 2024-04-19T23:26:41.299 DEBUG:boto:auth_path=/aktlji-16/obj38 2024-04-19T23:26:41.299 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.299 DEBUG:boto:Path: /aktlji-16/obj38 2024-04-19T23:26:41.299 DEBUG:boto:Data: 2024-04-19T23:26:41.299 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.299 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.299 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.299 DEBUG:boto:Params: {} 2024-04-19T23:26:41.300 DEBUG:boto:Token: None 2024-04-19T23:26:41.300 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj38 2024-04-19T23:26:41.300 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hLv6OF+8hRhOpGalwHZ1ZwMRNCA= 2024-04-19T23:26:41.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hLv6OF+8hRhOpGalwHZ1ZwMRNCA=', 'Content-Length': '0'} 2024-04-19T23:26:41.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2f9101b4f3fd148-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.302 DEBUG:boto:path=/aktlji-16/obj38 2024-04-19T23:26:41.302 DEBUG:boto:auth_path=/aktlji-16/obj38 2024-04-19T23:26:41.302 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.302 DEBUG:boto:Path: /aktlji-16/obj38 2024-04-19T23:26:41.302 DEBUG:boto:Data: 2024-04-19T23:26:41.302 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.302 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.302 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.302 DEBUG:boto:Params: {} 2024-04-19T23:26:41.303 DEBUG:boto:Token: None 2024-04-19T23:26:41.303 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj38 2024-04-19T23:26:41.303 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hLv6OF+8hRhOpGalwHZ1ZwMRNCA= 2024-04-19T23:26:41.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hLv6OF+8hRhOpGalwHZ1ZwMRNCA=', 'Content-Length': '0'} 2024-04-19T23:26:41.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d8cd7981d09bb8b-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-19T23:26:41.305 DEBUG:boto:path=/aktlji-16/obj38 2024-04-19T23:26:41.305 DEBUG:boto:auth_path=/aktlji-16/obj38 2024-04-19T23:26:41.305 DEBUG:boto:Method: GET 2024-04-19T23:26:41.305 DEBUG:boto:Path: /aktlji-16/obj38 2024-04-19T23:26:41.306 DEBUG:boto:Data: 2024-04-19T23:26:41.306 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.306 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.306 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.306 DEBUG:boto:Params: {} 2024-04-19T23:26:41.306 DEBUG:boto:Token: None 2024-04-19T23:26:41.306 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj38 2024-04-19T23:26:41.306 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IQx/tf1NQ1MlyXoBsLeEYTCQS30= 2024-04-19T23:26:41.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IQx/tf1NQ1MlyXoBsLeEYTCQS30=', 'Content-Length': '0'} 2024-04-19T23:26:41.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ccf852d85242d72-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.308 DEBUG:boto:path=/aktlji-16/obj38 2024-04-19T23:26:41.308 DEBUG:boto:auth_path=/aktlji-16/obj38 2024-04-19T23:26:41.309 DEBUG:boto:Method: GET 2024-04-19T23:26:41.309 DEBUG:boto:Path: /aktlji-16/obj38 2024-04-19T23:26:41.309 DEBUG:boto:Data: 2024-04-19T23:26:41.309 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.309 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.309 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.309 DEBUG:boto:Params: {} 2024-04-19T23:26:41.309 DEBUG:boto:Token: None 2024-04-19T23:26:41.309 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj38 2024-04-19T23:26:41.309 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IQx/tf1NQ1MlyXoBsLeEYTCQS30= 2024-04-19T23:26:41.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IQx/tf1NQ1MlyXoBsLeEYTCQS30=', 'Content-Length': '0'} 2024-04-19T23:26:41.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e3cc8e5d8612bbe-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.354 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.354 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.354 DEBUG:boto:path=/aktlji-16/?prefix=obj38 2024-04-19T23:26:41.355 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj38 2024-04-19T23:26:41.355 DEBUG:boto:Method: GET 2024-04-19T23:26:41.355 DEBUG:boto:Path: /aktlji-16/?prefix=obj38 2024-04-19T23:26:41.355 DEBUG:boto:Data: 2024-04-19T23:26:41.355 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.355 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.355 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.355 DEBUG:boto:Params: {} 2024-04-19T23:26:41.355 DEBUG:boto:Token: None 2024-04-19T23:26:41.355 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.356 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004b361aaa6d4c141-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.358 DEBUG:boto:b'aktlji-16obj381000falseobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.359 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.359 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.359 DEBUG:boto:path=/aktlji-16/?prefix=obj38 2024-04-19T23:26:41.359 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj38 2024-04-19T23:26:41.359 DEBUG:boto:Method: GET 2024-04-19T23:26:41.359 DEBUG:boto:Path: /aktlji-16/?prefix=obj38 2024-04-19T23:26:41.359 DEBUG:boto:Data: 2024-04-19T23:26:41.359 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.359 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.359 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.360 DEBUG:boto:Params: {} 2024-04-19T23:26:41.360 DEBUG:boto:Token: None 2024-04-19T23:26:41.360 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.360 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008580cf57815a9ac-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.362 DEBUG:boto:b'aktlji-16obj381000falseobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj39 2024-04-19T23:26:41.363 DEBUG:boto:path=/aktlji-16/obj39 2024-04-19T23:26:41.363 DEBUG:boto:auth_path=/aktlji-16/obj39 2024-04-19T23:26:41.363 DEBUG:boto:path=/aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.363 DEBUG:boto:auth_path=/aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.363 DEBUG:boto:Method: GET 2024-04-19T23:26:41.363 DEBUG:boto:Path: /aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.364 DEBUG:boto:Data: 2024-04-19T23:26:41.364 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.364 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.364 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.364 DEBUG:boto:Params: {} 2024-04-19T23:26:41.364 DEBUG:boto:Token: None 2024-04-19T23:26:41.364 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.364 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ= 2024-04-19T23:26:41.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ=', 'Content-Length': '0'} 2024-04-19T23:26:41.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd14a080361af5c8-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.406 DEBUG:boto:path=/aktlji-16/obj39 2024-04-19T23:26:41.406 DEBUG:boto:auth_path=/aktlji-16/obj39 2024-04-19T23:26:41.407 DEBUG:boto:path=/aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.407 DEBUG:boto:auth_path=/aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.407 DEBUG:boto:Method: GET 2024-04-19T23:26:41.407 DEBUG:boto:Path: /aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.407 DEBUG:boto:Data: 2024-04-19T23:26:41.407 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.407 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.407 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.407 DEBUG:boto:Params: {} 2024-04-19T23:26:41.407 DEBUG:boto:Token: None 2024-04-19T23:26:41.408 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.408 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ= 2024-04-19T23:26:41.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ=', 'Content-Length': '0'} 2024-04-19T23:26:41.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff5b2bcdf7c8eb9c-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.410 DEBUG:boto:path=/aktlji-16/obj39 2024-04-19T23:26:41.410 DEBUG:boto:auth_path=/aktlji-16/obj39 2024-04-19T23:26:41.410 DEBUG:boto:path=/aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.411 DEBUG:boto:auth_path=/aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.411 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.411 DEBUG:boto:Path: /aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.411 DEBUG:boto:Data: 2024-04-19T23:26:41.411 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.411 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.411 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.411 DEBUG:boto:Params: {} 2024-04-19T23:26:41.411 DEBUG:boto:Token: None 2024-04-19T23:26:41.411 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.411 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LICDCeCJN6hsUn2y/KSfze85lgA= 2024-04-19T23:26:41.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LICDCeCJN6hsUn2y/KSfze85lgA=', 'Content-Length': '0'} 2024-04-19T23:26:41.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008532d74584dd1ec2-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.414 DEBUG:boto:path=/aktlji-16/obj39 2024-04-19T23:26:41.414 DEBUG:boto:auth_path=/aktlji-16/obj39 2024-04-19T23:26:41.414 DEBUG:boto:path=/aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.414 DEBUG:boto:auth_path=/aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.414 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.414 DEBUG:boto:Path: /aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.414 DEBUG:boto:Data: 2024-04-19T23:26:41.415 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.415 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.415 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.415 DEBUG:boto:Params: {} 2024-04-19T23:26:41.415 DEBUG:boto:Token: None 2024-04-19T23:26:41.415 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.415 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LICDCeCJN6hsUn2y/KSfze85lgA= 2024-04-19T23:26:41.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LICDCeCJN6hsUn2y/KSfze85lgA=', 'Content-Length': '0'} 2024-04-19T23:26:41.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe90dbedb66e340d-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj39 2024-04-19T23:26:41.417 DEBUG:boto:path=/aktlji-16/obj39 2024-04-19T23:26:41.417 DEBUG:boto:auth_path=/aktlji-16/obj39 2024-04-19T23:26:41.417 DEBUG:boto:path=/aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.418 DEBUG:boto:auth_path=/aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.418 DEBUG:boto:Method: GET 2024-04-19T23:26:41.418 DEBUG:boto:Path: /aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.418 DEBUG:boto:Data: 2024-04-19T23:26:41.418 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.418 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.418 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.418 DEBUG:boto:Params: {} 2024-04-19T23:26:41.418 DEBUG:boto:Token: None 2024-04-19T23:26:41.418 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.419 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ= 2024-04-19T23:26:41.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ=', 'Content-Length': '0'} 2024-04-19T23:26:41.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2a40f8df764dd10-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.421 DEBUG:boto:path=/aktlji-16/obj39 2024-04-19T23:26:41.421 DEBUG:boto:auth_path=/aktlji-16/obj39 2024-04-19T23:26:41.421 DEBUG:boto:path=/aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.421 DEBUG:boto:auth_path=/aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.421 DEBUG:boto:Method: GET 2024-04-19T23:26:41.422 DEBUG:boto:Path: /aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.422 DEBUG:boto:Data: 2024-04-19T23:26:41.422 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.422 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.422 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.422 DEBUG:boto:Params: {} 2024-04-19T23:26:41.422 DEBUG:boto:Token: None 2024-04-19T23:26:41.422 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj39?versionId=null 2024-04-19T23:26:41.422 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ= 2024-04-19T23:26:41.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ=', 'Content-Length': '0'} 2024-04-19T23:26:41.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fed09aae678a502-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.425 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.425 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.425 DEBUG:boto:path=/aktlji-16/?prefix=obj39 2024-04-19T23:26:41.425 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj39 2024-04-19T23:26:41.425 DEBUG:boto:Method: GET 2024-04-19T23:26:41.425 DEBUG:boto:Path: /aktlji-16/?prefix=obj39 2024-04-19T23:26:41.425 DEBUG:boto:Data: 2024-04-19T23:26:41.425 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.425 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.425 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.426 DEBUG:boto:Params: {} 2024-04-19T23:26:41.426 DEBUG:boto:Token: None 2024-04-19T23:26:41.426 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.426 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d4fa27d792bf2f6-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.429 DEBUG:boto:b'aktlji-16obj391000falseobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.429 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.429 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.429 DEBUG:boto:path=/aktlji-16/?prefix=obj39 2024-04-19T23:26:41.430 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj39 2024-04-19T23:26:41.430 DEBUG:boto:Method: GET 2024-04-19T23:26:41.430 DEBUG:boto:Path: /aktlji-16/?prefix=obj39 2024-04-19T23:26:41.430 DEBUG:boto:Data: 2024-04-19T23:26:41.430 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.430 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.430 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.431 DEBUG:boto:Params: {} 2024-04-19T23:26:41.431 DEBUG:boto:Token: None 2024-04-19T23:26:41.431 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.431 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000358d867c66bb4fb7-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.433 DEBUG:boto:b'aktlji-16obj391000falseobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.434 DEBUG:boto:path=/aktlji-16/obj39 2024-04-19T23:26:41.434 DEBUG:boto:auth_path=/aktlji-16/obj39 2024-04-19T23:26:41.434 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.434 DEBUG:boto:Path: /aktlji-16/obj39 2024-04-19T23:26:41.434 DEBUG:boto:Data: 2024-04-19T23:26:41.434 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.434 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.434 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.435 DEBUG:boto:Params: {} 2024-04-19T23:26:41.435 DEBUG:boto:Token: None 2024-04-19T23:26:41.435 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj39 2024-04-19T23:26:41.435 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tGUQWo75glRYv9UAklIrrhNeSUU= 2024-04-19T23:26:41.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGUQWo75glRYv9UAklIrrhNeSUU=', 'Content-Length': '0'} 2024-04-19T23:26:41.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bd583cfce2be853-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.437 DEBUG:boto:path=/aktlji-16/obj39 2024-04-19T23:26:41.437 DEBUG:boto:auth_path=/aktlji-16/obj39 2024-04-19T23:26:41.437 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.437 DEBUG:boto:Path: /aktlji-16/obj39 2024-04-19T23:26:41.437 DEBUG:boto:Data: 2024-04-19T23:26:41.437 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.437 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.437 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.438 DEBUG:boto:Params: {} 2024-04-19T23:26:41.438 DEBUG:boto:Token: None 2024-04-19T23:26:41.438 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj39 2024-04-19T23:26:41.438 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tGUQWo75glRYv9UAklIrrhNeSUU= 2024-04-19T23:26:41.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGUQWo75glRYv9UAklIrrhNeSUU=', 'Content-Length': '0'} 2024-04-19T23:26:41.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011676e39754d9c9f-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj39 2024-04-19T23:26:41.440 DEBUG:boto:path=/aktlji-16/obj39 2024-04-19T23:26:41.440 DEBUG:boto:auth_path=/aktlji-16/obj39 2024-04-19T23:26:41.440 DEBUG:boto:Method: GET 2024-04-19T23:26:41.440 DEBUG:boto:Path: /aktlji-16/obj39 2024-04-19T23:26:41.440 DEBUG:boto:Data: 2024-04-19T23:26:41.441 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.441 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.441 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.441 DEBUG:boto:Params: {} 2024-04-19T23:26:41.441 DEBUG:boto:Token: None 2024-04-19T23:26:41.441 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj39 2024-04-19T23:26:41.441 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:03aKqdiCeGD34jUoqUeQ0/zu+ps= 2024-04-19T23:26:41.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:03aKqdiCeGD34jUoqUeQ0/zu+ps=', 'Content-Length': '0'} 2024-04-19T23:26:41.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068c1c84b71843c05-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.443 DEBUG:boto:path=/aktlji-16/obj39 2024-04-19T23:26:41.443 DEBUG:boto:auth_path=/aktlji-16/obj39 2024-04-19T23:26:41.443 DEBUG:boto:Method: GET 2024-04-19T23:26:41.444 DEBUG:boto:Path: /aktlji-16/obj39 2024-04-19T23:26:41.444 DEBUG:boto:Data: 2024-04-19T23:26:41.444 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.444 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.444 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.444 DEBUG:boto:Params: {} 2024-04-19T23:26:41.444 DEBUG:boto:Token: None 2024-04-19T23:26:41.444 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj39 2024-04-19T23:26:41.444 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:03aKqdiCeGD34jUoqUeQ0/zu+ps= 2024-04-19T23:26:41.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:03aKqdiCeGD34jUoqUeQ0/zu+ps=', 'Content-Length': '0'} 2024-04-19T23:26:41.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcc08fed1953f88c-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.446 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.447 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.447 DEBUG:boto:path=/aktlji-16/?prefix=obj39 2024-04-19T23:26:41.447 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj39 2024-04-19T23:26:41.447 DEBUG:boto:Method: GET 2024-04-19T23:26:41.447 DEBUG:boto:Path: /aktlji-16/?prefix=obj39 2024-04-19T23:26:41.447 DEBUG:boto:Data: 2024-04-19T23:26:41.447 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.447 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.447 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.447 DEBUG:boto:Params: {} 2024-04-19T23:26:41.447 DEBUG:boto:Token: None 2024-04-19T23:26:41.448 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.448 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cdf163700dcfd59-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.450 DEBUG:boto:b'aktlji-16obj391000falseobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.451 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.451 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.451 DEBUG:boto:path=/aktlji-16/?prefix=obj39 2024-04-19T23:26:41.451 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj39 2024-04-19T23:26:41.451 DEBUG:boto:Method: GET 2024-04-19T23:26:41.451 DEBUG:boto:Path: /aktlji-16/?prefix=obj39 2024-04-19T23:26:41.451 DEBUG:boto:Data: 2024-04-19T23:26:41.451 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.451 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.451 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.452 DEBUG:boto:Params: {} 2024-04-19T23:26:41.452 DEBUG:boto:Token: None 2024-04-19T23:26:41.452 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.452 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8531f1130ef67c2-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.455 DEBUG:boto:b'aktlji-16obj391000falseobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-19T23:26:41.455 DEBUG:boto:path=/aktlji-16/obj4 2024-04-19T23:26:41.455 DEBUG:boto:auth_path=/aktlji-16/obj4 2024-04-19T23:26:41.455 DEBUG:boto:path=/aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.455 DEBUG:boto:auth_path=/aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.455 DEBUG:boto:Method: GET 2024-04-19T23:26:41.456 DEBUG:boto:Path: /aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.456 DEBUG:boto:Data: 2024-04-19T23:26:41.456 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.456 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.456 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.456 DEBUG:boto:Params: {} 2024-04-19T23:26:41.456 DEBUG:boto:Token: None 2024-04-19T23:26:41.456 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.456 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0= 2024-04-19T23:26:41.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0=', 'Content-Length': '0'} 2024-04-19T23:26:41.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a833ef7fb2943313-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.506 DEBUG:boto:path=/aktlji-16/obj4 2024-04-19T23:26:41.507 DEBUG:boto:auth_path=/aktlji-16/obj4 2024-04-19T23:26:41.507 DEBUG:boto:path=/aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.507 DEBUG:boto:auth_path=/aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.507 DEBUG:boto:Method: GET 2024-04-19T23:26:41.507 DEBUG:boto:Path: /aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.507 DEBUG:boto:Data: 2024-04-19T23:26:41.507 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.507 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.507 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.507 DEBUG:boto:Params: {} 2024-04-19T23:26:41.508 DEBUG:boto:Token: None 2024-04-19T23:26:41.508 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.508 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0= 2024-04-19T23:26:41.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0=', 'Content-Length': '0'} 2024-04-19T23:26:41.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f63fd7a3d9d56c2-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.510 DEBUG:boto:path=/aktlji-16/obj4 2024-04-19T23:26:41.510 DEBUG:boto:auth_path=/aktlji-16/obj4 2024-04-19T23:26:41.510 DEBUG:boto:path=/aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.510 DEBUG:boto:auth_path=/aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.511 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.511 DEBUG:boto:Path: /aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.511 DEBUG:boto:Data: 2024-04-19T23:26:41.511 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.511 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.511 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.511 DEBUG:boto:Params: {} 2024-04-19T23:26:41.511 DEBUG:boto:Token: None 2024-04-19T23:26:41.511 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.511 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HSUMhs6NTyS2N4jhNr1uLu5y43o= 2024-04-19T23:26:41.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HSUMhs6NTyS2N4jhNr1uLu5y43o=', 'Content-Length': '0'} 2024-04-19T23:26:41.513 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0f647032d8ad92a-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.513 DEBUG:boto:path=/aktlji-16/obj4 2024-04-19T23:26:41.514 DEBUG:boto:auth_path=/aktlji-16/obj4 2024-04-19T23:26:41.514 DEBUG:boto:path=/aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.514 DEBUG:boto:auth_path=/aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.514 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.514 DEBUG:boto:Path: /aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.514 DEBUG:boto:Data: 2024-04-19T23:26:41.514 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.514 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.514 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.514 DEBUG:boto:Params: {} 2024-04-19T23:26:41.514 DEBUG:boto:Token: None 2024-04-19T23:26:41.515 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.515 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HSUMhs6NTyS2N4jhNr1uLu5y43o= 2024-04-19T23:26:41.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HSUMhs6NTyS2N4jhNr1uLu5y43o=', 'Content-Length': '0'} 2024-04-19T23:26:41.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070e94dfa865975f8-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-19T23:26:41.517 DEBUG:boto:path=/aktlji-16/obj4 2024-04-19T23:26:41.517 DEBUG:boto:auth_path=/aktlji-16/obj4 2024-04-19T23:26:41.517 DEBUG:boto:path=/aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.518 DEBUG:boto:auth_path=/aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.518 DEBUG:boto:Method: GET 2024-04-19T23:26:41.518 DEBUG:boto:Path: /aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.518 DEBUG:boto:Data: 2024-04-19T23:26:41.518 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.518 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.518 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.518 DEBUG:boto:Params: {} 2024-04-19T23:26:41.518 DEBUG:boto:Token: None 2024-04-19T23:26:41.518 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.519 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0= 2024-04-19T23:26:41.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0=', 'Content-Length': '0'} 2024-04-19T23:26:41.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001094413ba83ca05a-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.521 DEBUG:boto:path=/aktlji-16/obj4 2024-04-19T23:26:41.521 DEBUG:boto:auth_path=/aktlji-16/obj4 2024-04-19T23:26:41.521 DEBUG:boto:path=/aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.521 DEBUG:boto:auth_path=/aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.522 DEBUG:boto:Method: GET 2024-04-19T23:26:41.522 DEBUG:boto:Path: /aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.522 DEBUG:boto:Data: 2024-04-19T23:26:41.522 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.522 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.522 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.522 DEBUG:boto:Params: {} 2024-04-19T23:26:41.522 DEBUG:boto:Token: None 2024-04-19T23:26:41.522 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj4?versionId=null 2024-04-19T23:26:41.522 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0= 2024-04-19T23:26:41.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0=', 'Content-Length': '0'} 2024-04-19T23:26:41.524 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005811f662a647428d-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.525 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.525 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.525 DEBUG:boto:path=/aktlji-16/?prefix=obj4 2024-04-19T23:26:41.525 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj4 2024-04-19T23:26:41.525 DEBUG:boto:Method: GET 2024-04-19T23:26:41.525 DEBUG:boto:Path: /aktlji-16/?prefix=obj4 2024-04-19T23:26:41.525 DEBUG:boto:Data: 2024-04-19T23:26:41.525 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.525 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.525 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.525 DEBUG:boto:Params: {} 2024-04-19T23:26:41.526 DEBUG:boto:Token: None 2024-04-19T23:26:41.526 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.526 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000147409facaa6aa2f-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.529 DEBUG:boto:b'aktlji-16obj41000falseobj42024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.529 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.529 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.530 DEBUG:boto:path=/aktlji-16/?prefix=obj4 2024-04-19T23:26:41.530 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj4 2024-04-19T23:26:41.530 DEBUG:boto:Method: GET 2024-04-19T23:26:41.530 DEBUG:boto:Path: /aktlji-16/?prefix=obj4 2024-04-19T23:26:41.530 DEBUG:boto:Data: 2024-04-19T23:26:41.530 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.530 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.530 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.530 DEBUG:boto:Params: {} 2024-04-19T23:26:41.531 DEBUG:boto:Token: None 2024-04-19T23:26:41.531 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.531 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097a4a55896367c2b-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.534 DEBUG:boto:b'aktlji-16obj41000falseobj42024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.534 DEBUG:boto:path=/aktlji-16/obj4 2024-04-19T23:26:41.534 DEBUG:boto:auth_path=/aktlji-16/obj4 2024-04-19T23:26:41.535 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.535 DEBUG:boto:Path: /aktlji-16/obj4 2024-04-19T23:26:41.535 DEBUG:boto:Data: 2024-04-19T23:26:41.535 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.535 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.535 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.535 DEBUG:boto:Params: {} 2024-04-19T23:26:41.535 DEBUG:boto:Token: None 2024-04-19T23:26:41.535 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj4 2024-04-19T23:26:41.535 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DTzQF6tNillgL7wYNLDuO76Ov9w= 2024-04-19T23:26:41.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DTzQF6tNillgL7wYNLDuO76Ov9w=', 'Content-Length': '0'} 2024-04-19T23:26:41.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ad3b4afd6aa111b-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.538 DEBUG:boto:path=/aktlji-16/obj4 2024-04-19T23:26:41.538 DEBUG:boto:auth_path=/aktlji-16/obj4 2024-04-19T23:26:41.538 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.538 DEBUG:boto:Path: /aktlji-16/obj4 2024-04-19T23:26:41.538 DEBUG:boto:Data: 2024-04-19T23:26:41.538 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.539 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.539 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.539 DEBUG:boto:Params: {} 2024-04-19T23:26:41.539 DEBUG:boto:Token: None 2024-04-19T23:26:41.539 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj4 2024-04-19T23:26:41.539 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DTzQF6tNillgL7wYNLDuO76Ov9w= 2024-04-19T23:26:41.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DTzQF6tNillgL7wYNLDuO76Ov9w=', 'Content-Length': '0'} 2024-04-19T23:26:41.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ee959a67855d7a3-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-19T23:26:41.541 DEBUG:boto:path=/aktlji-16/obj4 2024-04-19T23:26:41.541 DEBUG:boto:auth_path=/aktlji-16/obj4 2024-04-19T23:26:41.542 DEBUG:boto:Method: GET 2024-04-19T23:26:41.542 DEBUG:boto:Path: /aktlji-16/obj4 2024-04-19T23:26:41.542 DEBUG:boto:Data: 2024-04-19T23:26:41.542 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.542 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.542 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.542 DEBUG:boto:Params: {} 2024-04-19T23:26:41.542 DEBUG:boto:Token: None 2024-04-19T23:26:41.542 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj4 2024-04-19T23:26:41.543 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0a9JXPqLwgOS0y6Ci6Q7LOB3/no= 2024-04-19T23:26:41.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0a9JXPqLwgOS0y6Ci6Q7LOB3/no=', 'Content-Length': '0'} 2024-04-19T23:26:41.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014f4311483b40491-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.545 DEBUG:boto:path=/aktlji-16/obj4 2024-04-19T23:26:41.545 DEBUG:boto:auth_path=/aktlji-16/obj4 2024-04-19T23:26:41.545 DEBUG:boto:Method: GET 2024-04-19T23:26:41.545 DEBUG:boto:Path: /aktlji-16/obj4 2024-04-19T23:26:41.545 DEBUG:boto:Data: 2024-04-19T23:26:41.545 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.545 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.545 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.546 DEBUG:boto:Params: {} 2024-04-19T23:26:41.546 DEBUG:boto:Token: None 2024-04-19T23:26:41.546 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj4 2024-04-19T23:26:41.546 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0a9JXPqLwgOS0y6Ci6Q7LOB3/no= 2024-04-19T23:26:41.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0a9JXPqLwgOS0y6Ci6Q7LOB3/no=', 'Content-Length': '0'} 2024-04-19T23:26:41.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000271c6866d7a34c8a-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.548 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.548 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.548 DEBUG:boto:path=/aktlji-16/?prefix=obj4 2024-04-19T23:26:41.548 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj4 2024-04-19T23:26:41.548 DEBUG:boto:Method: GET 2024-04-19T23:26:41.548 DEBUG:boto:Path: /aktlji-16/?prefix=obj4 2024-04-19T23:26:41.548 DEBUG:boto:Data: 2024-04-19T23:26:41.548 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.549 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.549 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.549 DEBUG:boto:Params: {} 2024-04-19T23:26:41.549 DEBUG:boto:Token: None 2024-04-19T23:26:41.549 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.549 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ad01d019f0280dc-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.552 DEBUG:boto:b'aktlji-16obj41000falseobj42024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.553 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.553 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.553 DEBUG:boto:path=/aktlji-16/?prefix=obj4 2024-04-19T23:26:41.553 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj4 2024-04-19T23:26:41.553 DEBUG:boto:Method: GET 2024-04-19T23:26:41.553 DEBUG:boto:Path: /aktlji-16/?prefix=obj4 2024-04-19T23:26:41.553 DEBUG:boto:Data: 2024-04-19T23:26:41.553 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.553 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.554 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.554 DEBUG:boto:Params: {} 2024-04-19T23:26:41.554 DEBUG:boto:Token: None 2024-04-19T23:26:41.554 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.554 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb246f983da71f53-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.557 DEBUG:boto:b'aktlji-16obj41000falseobj42024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj40 2024-04-19T23:26:41.558 DEBUG:boto:path=/aktlji-16/obj40 2024-04-19T23:26:41.558 DEBUG:boto:auth_path=/aktlji-16/obj40 2024-04-19T23:26:41.558 DEBUG:boto:path=/aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.558 DEBUG:boto:auth_path=/aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.558 DEBUG:boto:Method: GET 2024-04-19T23:26:41.558 DEBUG:boto:Path: /aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.558 DEBUG:boto:Data: 2024-04-19T23:26:41.558 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.558 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.558 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.558 DEBUG:boto:Params: {} 2024-04-19T23:26:41.559 DEBUG:boto:Token: None 2024-04-19T23:26:41.559 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.559 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA= 2024-04-19T23:26:41.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA=', 'Content-Length': '0'} 2024-04-19T23:26:41.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001609f35a7667ffea-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.561 DEBUG:boto:path=/aktlji-16/obj40 2024-04-19T23:26:41.561 DEBUG:boto:auth_path=/aktlji-16/obj40 2024-04-19T23:26:41.561 DEBUG:boto:path=/aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.561 DEBUG:boto:auth_path=/aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.561 DEBUG:boto:Method: GET 2024-04-19T23:26:41.561 DEBUG:boto:Path: /aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.561 DEBUG:boto:Data: 2024-04-19T23:26:41.562 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.562 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.562 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.562 DEBUG:boto:Params: {} 2024-04-19T23:26:41.562 DEBUG:boto:Token: None 2024-04-19T23:26:41.562 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.562 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA= 2024-04-19T23:26:41.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA=', 'Content-Length': '0'} 2024-04-19T23:26:41.564 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf60424c13c3211b-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.564 DEBUG:boto:path=/aktlji-16/obj40 2024-04-19T23:26:41.564 DEBUG:boto:auth_path=/aktlji-16/obj40 2024-04-19T23:26:41.564 DEBUG:boto:path=/aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.565 DEBUG:boto:auth_path=/aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.565 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.565 DEBUG:boto:Path: /aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.565 DEBUG:boto:Data: 2024-04-19T23:26:41.565 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.565 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.565 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.565 DEBUG:boto:Params: {} 2024-04-19T23:26:41.565 DEBUG:boto:Token: None 2024-04-19T23:26:41.565 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.566 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:keYTqHzUCbUBjZWQvwNDq96nCGM= 2024-04-19T23:26:41.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:keYTqHzUCbUBjZWQvwNDq96nCGM=', 'Content-Length': '0'} 2024-04-19T23:26:41.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a87d48f2d093cd8-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.568 DEBUG:boto:path=/aktlji-16/obj40 2024-04-19T23:26:41.568 DEBUG:boto:auth_path=/aktlji-16/obj40 2024-04-19T23:26:41.568 DEBUG:boto:path=/aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.568 DEBUG:boto:auth_path=/aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.568 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.568 DEBUG:boto:Path: /aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.568 DEBUG:boto:Data: 2024-04-19T23:26:41.568 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.568 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.568 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.568 DEBUG:boto:Params: {} 2024-04-19T23:26:41.569 DEBUG:boto:Token: None 2024-04-19T23:26:41.569 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.569 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:keYTqHzUCbUBjZWQvwNDq96nCGM= 2024-04-19T23:26:41.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:keYTqHzUCbUBjZWQvwNDq96nCGM=', 'Content-Length': '0'} 2024-04-19T23:26:41.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040acb52c92a36d4c-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj40 2024-04-19T23:26:41.571 DEBUG:boto:path=/aktlji-16/obj40 2024-04-19T23:26:41.571 DEBUG:boto:auth_path=/aktlji-16/obj40 2024-04-19T23:26:41.571 DEBUG:boto:path=/aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.571 DEBUG:boto:auth_path=/aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.571 DEBUG:boto:Method: GET 2024-04-19T23:26:41.571 DEBUG:boto:Path: /aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.571 DEBUG:boto:Data: 2024-04-19T23:26:41.572 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.572 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.572 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.572 DEBUG:boto:Params: {} 2024-04-19T23:26:41.572 DEBUG:boto:Token: None 2024-04-19T23:26:41.572 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.572 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA= 2024-04-19T23:26:41.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA=', 'Content-Length': '0'} 2024-04-19T23:26:41.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c749952ec29f47d8-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.614 DEBUG:boto:path=/aktlji-16/obj40 2024-04-19T23:26:41.614 DEBUG:boto:auth_path=/aktlji-16/obj40 2024-04-19T23:26:41.615 DEBUG:boto:path=/aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.615 DEBUG:boto:auth_path=/aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.615 DEBUG:boto:Method: GET 2024-04-19T23:26:41.615 DEBUG:boto:Path: /aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.615 DEBUG:boto:Data: 2024-04-19T23:26:41.615 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.615 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.615 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.615 DEBUG:boto:Params: {} 2024-04-19T23:26:41.615 DEBUG:boto:Token: None 2024-04-19T23:26:41.616 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj40?versionId=null 2024-04-19T23:26:41.616 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA= 2024-04-19T23:26:41.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA=', 'Content-Length': '0'} 2024-04-19T23:26:41.618 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094707d317896e89c-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.618 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.618 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.618 DEBUG:boto:path=/aktlji-16/?prefix=obj40 2024-04-19T23:26:41.618 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj40 2024-04-19T23:26:41.619 DEBUG:boto:Method: GET 2024-04-19T23:26:41.619 DEBUG:boto:Path: /aktlji-16/?prefix=obj40 2024-04-19T23:26:41.619 DEBUG:boto:Data: 2024-04-19T23:26:41.619 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.619 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.619 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.619 DEBUG:boto:Params: {} 2024-04-19T23:26:41.619 DEBUG:boto:Token: None 2024-04-19T23:26:41.619 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.619 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000338e70b7cadb755e-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.622 DEBUG:boto:b'aktlji-16obj401000falseobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.622 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.623 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.623 DEBUG:boto:path=/aktlji-16/?prefix=obj40 2024-04-19T23:26:41.623 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj40 2024-04-19T23:26:41.623 DEBUG:boto:Method: GET 2024-04-19T23:26:41.623 DEBUG:boto:Path: /aktlji-16/?prefix=obj40 2024-04-19T23:26:41.623 DEBUG:boto:Data: 2024-04-19T23:26:41.623 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.623 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.623 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.623 DEBUG:boto:Params: {} 2024-04-19T23:26:41.623 DEBUG:boto:Token: None 2024-04-19T23:26:41.624 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.624 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008730f0c59f5cffcd-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.626 DEBUG:boto:b'aktlji-16obj401000falseobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.627 DEBUG:boto:path=/aktlji-16/obj40 2024-04-19T23:26:41.627 DEBUG:boto:auth_path=/aktlji-16/obj40 2024-04-19T23:26:41.627 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.627 DEBUG:boto:Path: /aktlji-16/obj40 2024-04-19T23:26:41.627 DEBUG:boto:Data: 2024-04-19T23:26:41.627 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.627 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.627 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.627 DEBUG:boto:Params: {} 2024-04-19T23:26:41.628 DEBUG:boto:Token: None 2024-04-19T23:26:41.628 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj40 2024-04-19T23:26:41.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:p8jeU/5J8oBCkTE3uE56bemHvVU= 2024-04-19T23:26:41.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:p8jeU/5J8oBCkTE3uE56bemHvVU=', 'Content-Length': '0'} 2024-04-19T23:26:41.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095008dd55abfa498-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.630 DEBUG:boto:path=/aktlji-16/obj40 2024-04-19T23:26:41.630 DEBUG:boto:auth_path=/aktlji-16/obj40 2024-04-19T23:26:41.630 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.630 DEBUG:boto:Path: /aktlji-16/obj40 2024-04-19T23:26:41.630 DEBUG:boto:Data: 2024-04-19T23:26:41.631 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.631 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.631 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.631 DEBUG:boto:Params: {} 2024-04-19T23:26:41.631 DEBUG:boto:Token: None 2024-04-19T23:26:41.631 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj40 2024-04-19T23:26:41.631 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:p8jeU/5J8oBCkTE3uE56bemHvVU= 2024-04-19T23:26:41.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:p8jeU/5J8oBCkTE3uE56bemHvVU=', 'Content-Length': '0'} 2024-04-19T23:26:41.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b1243508bb9f3c6-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj40 2024-04-19T23:26:41.633 DEBUG:boto:path=/aktlji-16/obj40 2024-04-19T23:26:41.634 DEBUG:boto:auth_path=/aktlji-16/obj40 2024-04-19T23:26:41.634 DEBUG:boto:Method: GET 2024-04-19T23:26:41.634 DEBUG:boto:Path: /aktlji-16/obj40 2024-04-19T23:26:41.634 DEBUG:boto:Data: 2024-04-19T23:26:41.634 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.634 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.634 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.634 DEBUG:boto:Params: {} 2024-04-19T23:26:41.634 DEBUG:boto:Token: None 2024-04-19T23:26:41.635 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj40 2024-04-19T23:26:41.635 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EknWK+Xky9UlBG9NtHCIQmxkSqk= 2024-04-19T23:26:41.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EknWK+Xky9UlBG9NtHCIQmxkSqk=', 'Content-Length': '0'} 2024-04-19T23:26:41.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ab6bcad9b61ac7a-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.637 DEBUG:boto:path=/aktlji-16/obj40 2024-04-19T23:26:41.637 DEBUG:boto:auth_path=/aktlji-16/obj40 2024-04-19T23:26:41.637 DEBUG:boto:Method: GET 2024-04-19T23:26:41.637 DEBUG:boto:Path: /aktlji-16/obj40 2024-04-19T23:26:41.637 DEBUG:boto:Data: 2024-04-19T23:26:41.637 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.637 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.637 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.638 DEBUG:boto:Params: {} 2024-04-19T23:26:41.638 DEBUG:boto:Token: None 2024-04-19T23:26:41.638 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj40 2024-04-19T23:26:41.638 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EknWK+Xky9UlBG9NtHCIQmxkSqk= 2024-04-19T23:26:41.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EknWK+Xky9UlBG9NtHCIQmxkSqk=', 'Content-Length': '0'} 2024-04-19T23:26:41.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b533d05ee49d9f84-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.682 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.683 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.683 DEBUG:boto:path=/aktlji-16/?prefix=obj40 2024-04-19T23:26:41.683 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj40 2024-04-19T23:26:41.683 DEBUG:boto:Method: GET 2024-04-19T23:26:41.683 DEBUG:boto:Path: /aktlji-16/?prefix=obj40 2024-04-19T23:26:41.683 DEBUG:boto:Data: 2024-04-19T23:26:41.683 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.683 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.683 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.683 DEBUG:boto:Params: {} 2024-04-19T23:26:41.684 DEBUG:boto:Token: None 2024-04-19T23:26:41.684 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.684 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006cee459d6a81700-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.687 DEBUG:boto:b'aktlji-16obj401000falseobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.687 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.687 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.687 DEBUG:boto:path=/aktlji-16/?prefix=obj40 2024-04-19T23:26:41.687 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj40 2024-04-19T23:26:41.688 DEBUG:boto:Method: GET 2024-04-19T23:26:41.688 DEBUG:boto:Path: /aktlji-16/?prefix=obj40 2024-04-19T23:26:41.688 DEBUG:boto:Data: 2024-04-19T23:26:41.688 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.688 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.688 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.688 DEBUG:boto:Params: {} 2024-04-19T23:26:41.688 DEBUG:boto:Token: None 2024-04-19T23:26:41.688 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.688 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000794eaac5d22273e8-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.691 DEBUG:boto:b'aktlji-16obj401000falseobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj41 2024-04-19T23:26:41.692 DEBUG:boto:path=/aktlji-16/obj41 2024-04-19T23:26:41.692 DEBUG:boto:auth_path=/aktlji-16/obj41 2024-04-19T23:26:41.692 DEBUG:boto:path=/aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.692 DEBUG:boto:auth_path=/aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.692 DEBUG:boto:Method: GET 2024-04-19T23:26:41.692 DEBUG:boto:Path: /aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.692 DEBUG:boto:Data: 2024-04-19T23:26:41.692 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.692 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.693 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.693 DEBUG:boto:Params: {} 2024-04-19T23:26:41.693 DEBUG:boto:Token: None 2024-04-19T23:26:41.693 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.693 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4= 2024-04-19T23:26:41.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4=', 'Content-Length': '0'} 2024-04-19T23:26:41.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d222b200ff215f52-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.695 DEBUG:boto:path=/aktlji-16/obj41 2024-04-19T23:26:41.696 DEBUG:boto:auth_path=/aktlji-16/obj41 2024-04-19T23:26:41.696 DEBUG:boto:path=/aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.696 DEBUG:boto:auth_path=/aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.696 DEBUG:boto:Method: GET 2024-04-19T23:26:41.696 DEBUG:boto:Path: /aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.696 DEBUG:boto:Data: 2024-04-19T23:26:41.696 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.696 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.696 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.696 DEBUG:boto:Params: {} 2024-04-19T23:26:41.696 DEBUG:boto:Token: None 2024-04-19T23:26:41.697 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.697 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4= 2024-04-19T23:26:41.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4=', 'Content-Length': '0'} 2024-04-19T23:26:41.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c144a46a7dfd5d04-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.699 DEBUG:boto:path=/aktlji-16/obj41 2024-04-19T23:26:41.699 DEBUG:boto:auth_path=/aktlji-16/obj41 2024-04-19T23:26:41.699 DEBUG:boto:path=/aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.699 DEBUG:boto:auth_path=/aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.699 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.699 DEBUG:boto:Path: /aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.699 DEBUG:boto:Data: 2024-04-19T23:26:41.699 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.699 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.699 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.700 DEBUG:boto:Params: {} 2024-04-19T23:26:41.700 DEBUG:boto:Token: None 2024-04-19T23:26:41.700 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.700 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rSOHZWGdgjhbfnt2sllqY9vfCek= 2024-04-19T23:26:41.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rSOHZWGdgjhbfnt2sllqY9vfCek=', 'Content-Length': '0'} 2024-04-19T23:26:41.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076bcfe9e88ee775a-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.702 DEBUG:boto:path=/aktlji-16/obj41 2024-04-19T23:26:41.702 DEBUG:boto:auth_path=/aktlji-16/obj41 2024-04-19T23:26:41.702 DEBUG:boto:path=/aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.702 DEBUG:boto:auth_path=/aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.702 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.703 DEBUG:boto:Path: /aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.703 DEBUG:boto:Data: 2024-04-19T23:26:41.703 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.703 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.703 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.703 DEBUG:boto:Params: {} 2024-04-19T23:26:41.703 DEBUG:boto:Token: None 2024-04-19T23:26:41.703 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.703 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rSOHZWGdgjhbfnt2sllqY9vfCek= 2024-04-19T23:26:41.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rSOHZWGdgjhbfnt2sllqY9vfCek=', 'Content-Length': '0'} 2024-04-19T23:26:41.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030aeb4d2f908f339-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj41 2024-04-19T23:26:41.705 DEBUG:boto:path=/aktlji-16/obj41 2024-04-19T23:26:41.705 DEBUG:boto:auth_path=/aktlji-16/obj41 2024-04-19T23:26:41.706 DEBUG:boto:path=/aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.706 DEBUG:boto:auth_path=/aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.706 DEBUG:boto:Method: GET 2024-04-19T23:26:41.706 DEBUG:boto:Path: /aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.706 DEBUG:boto:Data: 2024-04-19T23:26:41.706 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.706 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.706 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.706 DEBUG:boto:Params: {} 2024-04-19T23:26:41.706 DEBUG:boto:Token: None 2024-04-19T23:26:41.707 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.707 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4= 2024-04-19T23:26:41.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4=', 'Content-Length': '0'} 2024-04-19T23:26:41.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d96b290caf082f9-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.750 DEBUG:boto:path=/aktlji-16/obj41 2024-04-19T23:26:41.751 DEBUG:boto:auth_path=/aktlji-16/obj41 2024-04-19T23:26:41.751 DEBUG:boto:path=/aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.751 DEBUG:boto:auth_path=/aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.751 DEBUG:boto:Method: GET 2024-04-19T23:26:41.751 DEBUG:boto:Path: /aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.751 DEBUG:boto:Data: 2024-04-19T23:26:41.751 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.751 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.751 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.751 DEBUG:boto:Params: {} 2024-04-19T23:26:41.751 DEBUG:boto:Token: None 2024-04-19T23:26:41.752 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj41?versionId=null 2024-04-19T23:26:41.752 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4= 2024-04-19T23:26:41.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4=', 'Content-Length': '0'} 2024-04-19T23:26:41.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008671d36e9c1afee3-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.754 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.754 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.754 DEBUG:boto:path=/aktlji-16/?prefix=obj41 2024-04-19T23:26:41.754 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj41 2024-04-19T23:26:41.754 DEBUG:boto:Method: GET 2024-04-19T23:26:41.754 DEBUG:boto:Path: /aktlji-16/?prefix=obj41 2024-04-19T23:26:41.755 DEBUG:boto:Data: 2024-04-19T23:26:41.755 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.755 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.755 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.755 DEBUG:boto:Params: {} 2024-04-19T23:26:41.755 DEBUG:boto:Token: None 2024-04-19T23:26:41.755 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.756 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ae9c7b668e22d8-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.758 DEBUG:boto:b'aktlji-16obj411000falseobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.759 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.759 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.759 DEBUG:boto:path=/aktlji-16/?prefix=obj41 2024-04-19T23:26:41.759 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj41 2024-04-19T23:26:41.759 DEBUG:boto:Method: GET 2024-04-19T23:26:41.759 DEBUG:boto:Path: /aktlji-16/?prefix=obj41 2024-04-19T23:26:41.759 DEBUG:boto:Data: 2024-04-19T23:26:41.759 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.760 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.760 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.760 DEBUG:boto:Params: {} 2024-04-19T23:26:41.760 DEBUG:boto:Token: None 2024-04-19T23:26:41.760 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.760 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096a0bbfe4aabf91d-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.763 DEBUG:boto:b'aktlji-16obj411000falseobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.763 DEBUG:boto:path=/aktlji-16/obj41 2024-04-19T23:26:41.763 DEBUG:boto:auth_path=/aktlji-16/obj41 2024-04-19T23:26:41.763 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.764 DEBUG:boto:Path: /aktlji-16/obj41 2024-04-19T23:26:41.764 DEBUG:boto:Data: 2024-04-19T23:26:41.764 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.764 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.764 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.764 DEBUG:boto:Params: {} 2024-04-19T23:26:41.764 DEBUG:boto:Token: None 2024-04-19T23:26:41.765 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj41 2024-04-19T23:26:41.765 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vSgTRPFrdVZeygwMeBzQO1XIXho= 2024-04-19T23:26:41.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vSgTRPFrdVZeygwMeBzQO1XIXho=', 'Content-Length': '0'} 2024-04-19T23:26:41.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a778d7d73f64221e-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.767 DEBUG:boto:path=/aktlji-16/obj41 2024-04-19T23:26:41.767 DEBUG:boto:auth_path=/aktlji-16/obj41 2024-04-19T23:26:41.767 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.767 DEBUG:boto:Path: /aktlji-16/obj41 2024-04-19T23:26:41.767 DEBUG:boto:Data: 2024-04-19T23:26:41.767 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.768 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.768 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.768 DEBUG:boto:Params: {} 2024-04-19T23:26:41.768 DEBUG:boto:Token: None 2024-04-19T23:26:41.768 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj41 2024-04-19T23:26:41.768 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vSgTRPFrdVZeygwMeBzQO1XIXho= 2024-04-19T23:26:41.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vSgTRPFrdVZeygwMeBzQO1XIXho=', 'Content-Length': '0'} 2024-04-19T23:26:41.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027f8f125715c91a3-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj41 2024-04-19T23:26:41.770 DEBUG:boto:path=/aktlji-16/obj41 2024-04-19T23:26:41.770 DEBUG:boto:auth_path=/aktlji-16/obj41 2024-04-19T23:26:41.770 DEBUG:boto:Method: GET 2024-04-19T23:26:41.770 DEBUG:boto:Path: /aktlji-16/obj41 2024-04-19T23:26:41.770 DEBUG:boto:Data: 2024-04-19T23:26:41.771 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.771 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.771 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.771 DEBUG:boto:Params: {} 2024-04-19T23:26:41.771 DEBUG:boto:Token: None 2024-04-19T23:26:41.771 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj41 2024-04-19T23:26:41.771 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OW5WcKJu3+D4W+w4FB7GOOn935M= 2024-04-19T23:26:41.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OW5WcKJu3+D4W+w4FB7GOOn935M=', 'Content-Length': '0'} 2024-04-19T23:26:41.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ea6791d6c1b78e4-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.814 DEBUG:boto:path=/aktlji-16/obj41 2024-04-19T23:26:41.815 DEBUG:boto:auth_path=/aktlji-16/obj41 2024-04-19T23:26:41.815 DEBUG:boto:Method: GET 2024-04-19T23:26:41.815 DEBUG:boto:Path: /aktlji-16/obj41 2024-04-19T23:26:41.815 DEBUG:boto:Data: 2024-04-19T23:26:41.815 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.815 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.815 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.816 DEBUG:boto:Params: {} 2024-04-19T23:26:41.816 DEBUG:boto:Token: None 2024-04-19T23:26:41.816 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj41 2024-04-19T23:26:41.816 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OW5WcKJu3+D4W+w4FB7GOOn935M= 2024-04-19T23:26:41.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OW5WcKJu3+D4W+w4FB7GOOn935M=', 'Content-Length': '0'} 2024-04-19T23:26:41.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000634b10cb9d4cab6a-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.818 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.819 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.819 DEBUG:boto:path=/aktlji-16/?prefix=obj41 2024-04-19T23:26:41.819 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj41 2024-04-19T23:26:41.819 DEBUG:boto:Method: GET 2024-04-19T23:26:41.819 DEBUG:boto:Path: /aktlji-16/?prefix=obj41 2024-04-19T23:26:41.819 DEBUG:boto:Data: 2024-04-19T23:26:41.820 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.820 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.820 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.820 DEBUG:boto:Params: {} 2024-04-19T23:26:41.820 DEBUG:boto:Token: None 2024-04-19T23:26:41.820 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.820 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073c0dc0d265c01bb-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.823 DEBUG:boto:b'aktlji-16obj411000falseobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.824 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.824 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.824 DEBUG:boto:path=/aktlji-16/?prefix=obj41 2024-04-19T23:26:41.824 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj41 2024-04-19T23:26:41.824 DEBUG:boto:Method: GET 2024-04-19T23:26:41.824 DEBUG:boto:Path: /aktlji-16/?prefix=obj41 2024-04-19T23:26:41.824 DEBUG:boto:Data: 2024-04-19T23:26:41.824 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.824 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.824 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.825 DEBUG:boto:Params: {} 2024-04-19T23:26:41.825 DEBUG:boto:Token: None 2024-04-19T23:26:41.825 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.825 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6897833fc42130b-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.828 DEBUG:boto:b'aktlji-16obj411000falseobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj42 2024-04-19T23:26:41.828 DEBUG:boto:path=/aktlji-16/obj42 2024-04-19T23:26:41.829 DEBUG:boto:auth_path=/aktlji-16/obj42 2024-04-19T23:26:41.829 DEBUG:boto:path=/aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.829 DEBUG:boto:auth_path=/aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.829 DEBUG:boto:Method: GET 2024-04-19T23:26:41.829 DEBUG:boto:Path: /aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.829 DEBUG:boto:Data: 2024-04-19T23:26:41.829 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.829 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.830 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.830 DEBUG:boto:Params: {} 2024-04-19T23:26:41.830 DEBUG:boto:Token: None 2024-04-19T23:26:41.830 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.830 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s= 2024-04-19T23:26:41.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s=', 'Content-Length': '0'} 2024-04-19T23:26:41.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000922af8d1bdc1c42d-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.833 DEBUG:boto:path=/aktlji-16/obj42 2024-04-19T23:26:41.833 DEBUG:boto:auth_path=/aktlji-16/obj42 2024-04-19T23:26:41.833 DEBUG:boto:path=/aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.833 DEBUG:boto:auth_path=/aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.833 DEBUG:boto:Method: GET 2024-04-19T23:26:41.833 DEBUG:boto:Path: /aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.833 DEBUG:boto:Data: 2024-04-19T23:26:41.834 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.834 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.834 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.834 DEBUG:boto:Params: {} 2024-04-19T23:26:41.834 DEBUG:boto:Token: None 2024-04-19T23:26:41.834 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.835 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s= 2024-04-19T23:26:41.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s=', 'Content-Length': '0'} 2024-04-19T23:26:41.837 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a3b7bfae5a5a0e3-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.837 DEBUG:boto:path=/aktlji-16/obj42 2024-04-19T23:26:41.837 DEBUG:boto:auth_path=/aktlji-16/obj42 2024-04-19T23:26:41.837 DEBUG:boto:path=/aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.837 DEBUG:boto:auth_path=/aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.838 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.838 DEBUG:boto:Path: /aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.838 DEBUG:boto:Data: 2024-04-19T23:26:41.838 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.838 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.838 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.838 DEBUG:boto:Params: {} 2024-04-19T23:26:41.838 DEBUG:boto:Token: None 2024-04-19T23:26:41.838 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.839 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ra/uats+1ZnOz9YNrhNu3U/t8I8= 2024-04-19T23:26:41.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ra/uats+1ZnOz9YNrhNu3U/t8I8=', 'Content-Length': '0'} 2024-04-19T23:26:41.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a414936df569b11-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.841 DEBUG:boto:path=/aktlji-16/obj42 2024-04-19T23:26:41.841 DEBUG:boto:auth_path=/aktlji-16/obj42 2024-04-19T23:26:41.841 DEBUG:boto:path=/aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.841 DEBUG:boto:auth_path=/aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.841 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.841 DEBUG:boto:Path: /aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.841 DEBUG:boto:Data: 2024-04-19T23:26:41.841 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.841 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.842 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.842 DEBUG:boto:Params: {} 2024-04-19T23:26:41.842 DEBUG:boto:Token: None 2024-04-19T23:26:41.842 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.842 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ra/uats+1ZnOz9YNrhNu3U/t8I8= 2024-04-19T23:26:41.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ra/uats+1ZnOz9YNrhNu3U/t8I8=', 'Content-Length': '0'} 2024-04-19T23:26:41.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a92f0799f89b73d-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj42 2024-04-19T23:26:41.844 DEBUG:boto:path=/aktlji-16/obj42 2024-04-19T23:26:41.844 DEBUG:boto:auth_path=/aktlji-16/obj42 2024-04-19T23:26:41.844 DEBUG:boto:path=/aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.845 DEBUG:boto:auth_path=/aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.845 DEBUG:boto:Method: GET 2024-04-19T23:26:41.845 DEBUG:boto:Path: /aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.845 DEBUG:boto:Data: 2024-04-19T23:26:41.845 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.845 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.845 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.845 DEBUG:boto:Params: {} 2024-04-19T23:26:41.845 DEBUG:boto:Token: None 2024-04-19T23:26:41.845 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.846 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s= 2024-04-19T23:26:41.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s=', 'Content-Length': '0'} 2024-04-19T23:26:41.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0c79c8a7ed1862d-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.848 DEBUG:boto:path=/aktlji-16/obj42 2024-04-19T23:26:41.848 DEBUG:boto:auth_path=/aktlji-16/obj42 2024-04-19T23:26:41.848 DEBUG:boto:path=/aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.848 DEBUG:boto:auth_path=/aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.848 DEBUG:boto:Method: GET 2024-04-19T23:26:41.849 DEBUG:boto:Path: /aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.849 DEBUG:boto:Data: 2024-04-19T23:26:41.849 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.849 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.849 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.849 DEBUG:boto:Params: {} 2024-04-19T23:26:41.849 DEBUG:boto:Token: None 2024-04-19T23:26:41.849 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj42?versionId=null 2024-04-19T23:26:41.850 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s= 2024-04-19T23:26:41.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s=', 'Content-Length': '0'} 2024-04-19T23:26:41.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a22c66ad9967863-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.852 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.852 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.852 DEBUG:boto:path=/aktlji-16/?prefix=obj42 2024-04-19T23:26:41.852 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj42 2024-04-19T23:26:41.852 DEBUG:boto:Method: GET 2024-04-19T23:26:41.852 DEBUG:boto:Path: /aktlji-16/?prefix=obj42 2024-04-19T23:26:41.852 DEBUG:boto:Data: 2024-04-19T23:26:41.852 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.852 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.853 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.853 DEBUG:boto:Params: {} 2024-04-19T23:26:41.853 DEBUG:boto:Token: None 2024-04-19T23:26:41.853 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.853 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab45c0103387eec-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.856 DEBUG:boto:b'aktlji-16obj421000falseobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.858 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.858 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.858 DEBUG:boto:path=/aktlji-16/?prefix=obj42 2024-04-19T23:26:41.858 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj42 2024-04-19T23:26:41.859 DEBUG:boto:Method: GET 2024-04-19T23:26:41.859 DEBUG:boto:Path: /aktlji-16/?prefix=obj42 2024-04-19T23:26:41.859 DEBUG:boto:Data: 2024-04-19T23:26:41.859 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.859 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.859 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.859 DEBUG:boto:Params: {} 2024-04-19T23:26:41.859 DEBUG:boto:Token: None 2024-04-19T23:26:41.859 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.859 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020cfa761beadeba2-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.863 DEBUG:boto:b'aktlji-16obj421000falseobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.863 DEBUG:boto:path=/aktlji-16/obj42 2024-04-19T23:26:41.863 DEBUG:boto:auth_path=/aktlji-16/obj42 2024-04-19T23:26:41.863 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.863 DEBUG:boto:Path: /aktlji-16/obj42 2024-04-19T23:26:41.863 DEBUG:boto:Data: 2024-04-19T23:26:41.863 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.863 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.863 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.864 DEBUG:boto:Params: {} 2024-04-19T23:26:41.864 DEBUG:boto:Token: None 2024-04-19T23:26:41.864 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj42 2024-04-19T23:26:41.864 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Pxi+FiXd4nteQYhW+G2wcqt7aP4= 2024-04-19T23:26:41.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Pxi+FiXd4nteQYhW+G2wcqt7aP4=', 'Content-Length': '0'} 2024-04-19T23:26:41.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000322f2b2da4bb7a6c-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.866 DEBUG:boto:path=/aktlji-16/obj42 2024-04-19T23:26:41.866 DEBUG:boto:auth_path=/aktlji-16/obj42 2024-04-19T23:26:41.866 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.866 DEBUG:boto:Path: /aktlji-16/obj42 2024-04-19T23:26:41.866 DEBUG:boto:Data: 2024-04-19T23:26:41.866 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.867 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.867 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.867 DEBUG:boto:Params: {} 2024-04-19T23:26:41.867 DEBUG:boto:Token: None 2024-04-19T23:26:41.867 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj42 2024-04-19T23:26:41.867 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Pxi+FiXd4nteQYhW+G2wcqt7aP4= 2024-04-19T23:26:41.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Pxi+FiXd4nteQYhW+G2wcqt7aP4=', 'Content-Length': '0'} 2024-04-19T23:26:41.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f101cb1d157856ba-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj42 2024-04-19T23:26:41.869 DEBUG:boto:path=/aktlji-16/obj42 2024-04-19T23:26:41.869 DEBUG:boto:auth_path=/aktlji-16/obj42 2024-04-19T23:26:41.870 DEBUG:boto:Method: GET 2024-04-19T23:26:41.870 DEBUG:boto:Path: /aktlji-16/obj42 2024-04-19T23:26:41.870 DEBUG:boto:Data: 2024-04-19T23:26:41.870 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.870 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.870 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.871 DEBUG:boto:Params: {} 2024-04-19T23:26:41.871 DEBUG:boto:Token: None 2024-04-19T23:26:41.871 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj42 2024-04-19T23:26:41.871 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GJPzOPitAfDJEKzKQ7/CJGjB7KQ= 2024-04-19T23:26:41.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GJPzOPitAfDJEKzKQ7/CJGjB7KQ=', 'Content-Length': '0'} 2024-04-19T23:26:41.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cba9919f248bd65a-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.918 DEBUG:boto:path=/aktlji-16/obj42 2024-04-19T23:26:41.919 DEBUG:boto:auth_path=/aktlji-16/obj42 2024-04-19T23:26:41.919 DEBUG:boto:Method: GET 2024-04-19T23:26:41.919 DEBUG:boto:Path: /aktlji-16/obj42 2024-04-19T23:26:41.919 DEBUG:boto:Data: 2024-04-19T23:26:41.919 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.919 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.920 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.920 DEBUG:boto:Params: {} 2024-04-19T23:26:41.921 DEBUG:boto:Token: None 2024-04-19T23:26:41.921 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj42 2024-04-19T23:26:41.921 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GJPzOPitAfDJEKzKQ7/CJGjB7KQ= 2024-04-19T23:26:41.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GJPzOPitAfDJEKzKQ7/CJGjB7KQ=', 'Content-Length': '0'} 2024-04-19T23:26:41.923 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bac2253e55308c52-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.923 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.924 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.924 DEBUG:boto:path=/aktlji-16/?prefix=obj42 2024-04-19T23:26:41.924 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj42 2024-04-19T23:26:41.924 DEBUG:boto:Method: GET 2024-04-19T23:26:41.924 DEBUG:boto:Path: /aktlji-16/?prefix=obj42 2024-04-19T23:26:41.924 DEBUG:boto:Data: 2024-04-19T23:26:41.924 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.924 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.924 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.924 DEBUG:boto:Params: {} 2024-04-19T23:26:41.924 DEBUG:boto:Token: None 2024-04-19T23:26:41.925 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.925 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c144f99829aed52-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.927 DEBUG:boto:b'aktlji-16obj421000falseobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.928 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.928 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.928 DEBUG:boto:path=/aktlji-16/?prefix=obj42 2024-04-19T23:26:41.928 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj42 2024-04-19T23:26:41.928 DEBUG:boto:Method: GET 2024-04-19T23:26:41.928 DEBUG:boto:Path: /aktlji-16/?prefix=obj42 2024-04-19T23:26:41.928 DEBUG:boto:Data: 2024-04-19T23:26:41.928 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.928 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.929 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.929 DEBUG:boto:Params: {} 2024-04-19T23:26:41.929 DEBUG:boto:Token: None 2024-04-19T23:26:41.929 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.929 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:41.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:41.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd1ce911f5a36af7-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.932 DEBUG:boto:b'aktlji-16obj421000falseobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:41.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj43 2024-04-19T23:26:41.932 DEBUG:boto:path=/aktlji-16/obj43 2024-04-19T23:26:41.932 DEBUG:boto:auth_path=/aktlji-16/obj43 2024-04-19T23:26:41.932 DEBUG:boto:path=/aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.933 DEBUG:boto:auth_path=/aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.933 DEBUG:boto:Method: GET 2024-04-19T23:26:41.933 DEBUG:boto:Path: /aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.933 DEBUG:boto:Data: 2024-04-19T23:26:41.933 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.933 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.933 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.933 DEBUG:boto:Params: {} 2024-04-19T23:26:41.933 DEBUG:boto:Token: None 2024-04-19T23:26:41.933 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.934 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak= 2024-04-19T23:26:41.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak=', 'Content-Length': '0'} 2024-04-19T23:26:41.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064f4122a93daaf94-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.936 DEBUG:boto:path=/aktlji-16/obj43 2024-04-19T23:26:41.936 DEBUG:boto:auth_path=/aktlji-16/obj43 2024-04-19T23:26:41.936 DEBUG:boto:path=/aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.936 DEBUG:boto:auth_path=/aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.936 DEBUG:boto:Method: GET 2024-04-19T23:26:41.936 DEBUG:boto:Path: /aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.936 DEBUG:boto:Data: 2024-04-19T23:26:41.936 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.937 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.937 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.937 DEBUG:boto:Params: {} 2024-04-19T23:26:41.937 DEBUG:boto:Token: None 2024-04-19T23:26:41.937 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.937 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak= 2024-04-19T23:26:41.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak=', 'Content-Length': '0'} 2024-04-19T23:26:41.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001dd93ae6c9efddb-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.939 DEBUG:boto:path=/aktlji-16/obj43 2024-04-19T23:26:41.939 DEBUG:boto:auth_path=/aktlji-16/obj43 2024-04-19T23:26:41.939 DEBUG:boto:path=/aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.940 DEBUG:boto:auth_path=/aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.940 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.940 DEBUG:boto:Path: /aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.940 DEBUG:boto:Data: 2024-04-19T23:26:41.940 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.940 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.940 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.940 DEBUG:boto:Params: {} 2024-04-19T23:26:41.940 DEBUG:boto:Token: None 2024-04-19T23:26:41.940 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.941 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fk96L56YigQkhG2Aroo+77vxbqo= 2024-04-19T23:26:41.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fk96L56YigQkhG2Aroo+77vxbqo=', 'Content-Length': '0'} 2024-04-19T23:26:41.942 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7437e2f83179166-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.942 DEBUG:boto:path=/aktlji-16/obj43 2024-04-19T23:26:41.943 DEBUG:boto:auth_path=/aktlji-16/obj43 2024-04-19T23:26:41.943 DEBUG:boto:path=/aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.943 DEBUG:boto:auth_path=/aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.943 DEBUG:boto:Method: HEAD 2024-04-19T23:26:41.943 DEBUG:boto:Path: /aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.943 DEBUG:boto:Data: 2024-04-19T23:26:41.943 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.943 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.943 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.943 DEBUG:boto:Params: {} 2024-04-19T23:26:41.944 DEBUG:boto:Token: None 2024-04-19T23:26:41.944 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fk96L56YigQkhG2Aroo+77vxbqo= 2024-04-19T23:26:41.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fk96L56YigQkhG2Aroo+77vxbqo=', 'Content-Length': '0'} 2024-04-19T23:26:41.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011bd85f19c53a027-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj43 2024-04-19T23:26:41.946 DEBUG:boto:path=/aktlji-16/obj43 2024-04-19T23:26:41.946 DEBUG:boto:auth_path=/aktlji-16/obj43 2024-04-19T23:26:41.946 DEBUG:boto:path=/aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.947 DEBUG:boto:auth_path=/aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.947 DEBUG:boto:Method: GET 2024-04-19T23:26:41.947 DEBUG:boto:Path: /aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.947 DEBUG:boto:Data: 2024-04-19T23:26:41.947 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.947 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.947 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.947 DEBUG:boto:Params: {} 2024-04-19T23:26:41.947 DEBUG:boto:Token: None 2024-04-19T23:26:41.948 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.948 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak= 2024-04-19T23:26:41.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak=', 'Content-Length': '0'} 2024-04-19T23:26:41.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091319d63dbe40b61-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.994 DEBUG:boto:path=/aktlji-16/obj43 2024-04-19T23:26:41.995 DEBUG:boto:auth_path=/aktlji-16/obj43 2024-04-19T23:26:41.995 DEBUG:boto:path=/aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.995 DEBUG:boto:auth_path=/aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.995 DEBUG:boto:Method: GET 2024-04-19T23:26:41.995 DEBUG:boto:Path: /aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.995 DEBUG:boto:Data: 2024-04-19T23:26:41.995 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.995 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:41.995 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.995 DEBUG:boto:Params: {} 2024-04-19T23:26:41.996 DEBUG:boto:Token: None 2024-04-19T23:26:41.996 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/obj43?versionId=null 2024-04-19T23:26:41.996 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak= 2024-04-19T23:26:41.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak=', 'Content-Length': '0'} 2024-04-19T23:26:41.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f40cbff711f7a88-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:41.998 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:41.998 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:41.998 DEBUG:boto:path=/aktlji-16/?prefix=obj43 2024-04-19T23:26:41.998 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj43 2024-04-19T23:26:41.999 DEBUG:boto:Method: GET 2024-04-19T23:26:41.999 DEBUG:boto:Path: /aktlji-16/?prefix=obj43 2024-04-19T23:26:41.999 DEBUG:boto:Data: 2024-04-19T23:26:41.999 DEBUG:boto:Headers: {} 2024-04-19T23:26:41.999 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:41.999 DEBUG:boto:Port: 8000 2024-04-19T23:26:41.999 DEBUG:boto:Params: {} 2024-04-19T23:26:41.999 DEBUG:boto:Token: None 2024-04-19T23:26:41.999 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:41 GMT /aktlji-16/ 2024-04-19T23:26:41.999 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-19T23:26:42.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-19T23:26:42.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d03a9ba42eae56a-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.002 DEBUG:boto:b'aktlji-16obj431000falseobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.003 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.003 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.003 DEBUG:boto:path=/aktlji-16/?prefix=obj43 2024-04-19T23:26:42.003 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj43 2024-04-19T23:26:42.003 DEBUG:boto:Method: GET 2024-04-19T23:26:42.003 DEBUG:boto:Path: /aktlji-16/?prefix=obj43 2024-04-19T23:26:42.003 DEBUG:boto:Data: 2024-04-19T23:26:42.003 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.003 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.003 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.004 DEBUG:boto:Params: {} 2024-04-19T23:26:42.004 DEBUG:boto:Token: None 2024-04-19T23:26:42.004 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.004 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beef9ef74f9f6b15-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.007 DEBUG:boto:b'aktlji-16obj431000falseobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.008 DEBUG:boto:path=/aktlji-16/obj43 2024-04-19T23:26:42.008 DEBUG:boto:auth_path=/aktlji-16/obj43 2024-04-19T23:26:42.008 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.008 DEBUG:boto:Path: /aktlji-16/obj43 2024-04-19T23:26:42.008 DEBUG:boto:Data: 2024-04-19T23:26:42.008 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.008 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.008 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.008 DEBUG:boto:Params: {} 2024-04-19T23:26:42.009 DEBUG:boto:Token: None 2024-04-19T23:26:42.009 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj43 2024-04-19T23:26:42.009 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZwLhwpBmvtlavpfEqsph0RwnAjY= 2024-04-19T23:26:42.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZwLhwpBmvtlavpfEqsph0RwnAjY=', 'Content-Length': '0'} 2024-04-19T23:26:42.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfe0430759239c62-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.011 DEBUG:boto:path=/aktlji-16/obj43 2024-04-19T23:26:42.011 DEBUG:boto:auth_path=/aktlji-16/obj43 2024-04-19T23:26:42.011 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.011 DEBUG:boto:Path: /aktlji-16/obj43 2024-04-19T23:26:42.011 DEBUG:boto:Data: 2024-04-19T23:26:42.011 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.011 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.011 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.011 DEBUG:boto:Params: {} 2024-04-19T23:26:42.012 DEBUG:boto:Token: None 2024-04-19T23:26:42.012 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj43 2024-04-19T23:26:42.012 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZwLhwpBmvtlavpfEqsph0RwnAjY= 2024-04-19T23:26:42.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZwLhwpBmvtlavpfEqsph0RwnAjY=', 'Content-Length': '0'} 2024-04-19T23:26:42.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c232b2e594367e86-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj43 2024-04-19T23:26:42.014 DEBUG:boto:path=/aktlji-16/obj43 2024-04-19T23:26:42.014 DEBUG:boto:auth_path=/aktlji-16/obj43 2024-04-19T23:26:42.014 DEBUG:boto:Method: GET 2024-04-19T23:26:42.014 DEBUG:boto:Path: /aktlji-16/obj43 2024-04-19T23:26:42.014 DEBUG:boto:Data: 2024-04-19T23:26:42.014 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.014 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.015 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.015 DEBUG:boto:Params: {} 2024-04-19T23:26:42.015 DEBUG:boto:Token: None 2024-04-19T23:26:42.015 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj43 2024-04-19T23:26:42.015 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0o9p0D30IuytxplDVzrc7Vm5als= 2024-04-19T23:26:42.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0o9p0D30IuytxplDVzrc7Vm5als=', 'Content-Length': '0'} 2024-04-19T23:26:42.017 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b20615039285a14e-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.062 DEBUG:boto:path=/aktlji-16/obj43 2024-04-19T23:26:42.062 DEBUG:boto:auth_path=/aktlji-16/obj43 2024-04-19T23:26:42.063 DEBUG:boto:Method: GET 2024-04-19T23:26:42.063 DEBUG:boto:Path: /aktlji-16/obj43 2024-04-19T23:26:42.063 DEBUG:boto:Data: 2024-04-19T23:26:42.063 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.063 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.063 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.063 DEBUG:boto:Params: {} 2024-04-19T23:26:42.063 DEBUG:boto:Token: None 2024-04-19T23:26:42.063 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj43 2024-04-19T23:26:42.064 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0o9p0D30IuytxplDVzrc7Vm5als= 2024-04-19T23:26:42.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0o9p0D30IuytxplDVzrc7Vm5als=', 'Content-Length': '0'} 2024-04-19T23:26:42.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000997a166cd6c6ab7a-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.066 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.066 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.066 DEBUG:boto:path=/aktlji-16/?prefix=obj43 2024-04-19T23:26:42.066 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj43 2024-04-19T23:26:42.066 DEBUG:boto:Method: GET 2024-04-19T23:26:42.066 DEBUG:boto:Path: /aktlji-16/?prefix=obj43 2024-04-19T23:26:42.067 DEBUG:boto:Data: 2024-04-19T23:26:42.067 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.067 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.067 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.067 DEBUG:boto:Params: {} 2024-04-19T23:26:42.067 DEBUG:boto:Token: None 2024-04-19T23:26:42.067 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.067 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7abaa9c4fefde44-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.070 DEBUG:boto:b'aktlji-16obj431000falseobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.070 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.070 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.070 DEBUG:boto:path=/aktlji-16/?prefix=obj43 2024-04-19T23:26:42.071 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj43 2024-04-19T23:26:42.071 DEBUG:boto:Method: GET 2024-04-19T23:26:42.071 DEBUG:boto:Path: /aktlji-16/?prefix=obj43 2024-04-19T23:26:42.071 DEBUG:boto:Data: 2024-04-19T23:26:42.071 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.071 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.071 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.071 DEBUG:boto:Params: {} 2024-04-19T23:26:42.071 DEBUG:boto:Token: None 2024-04-19T23:26:42.072 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.072 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e58523bef9cacfc6-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.074 DEBUG:boto:b'aktlji-16obj431000falseobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-19T23:26:42.075 DEBUG:boto:path=/aktlji-16/obj44 2024-04-19T23:26:42.075 DEBUG:boto:auth_path=/aktlji-16/obj44 2024-04-19T23:26:42.075 DEBUG:boto:path=/aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.075 DEBUG:boto:auth_path=/aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.075 DEBUG:boto:Method: GET 2024-04-19T23:26:42.075 DEBUG:boto:Path: /aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.075 DEBUG:boto:Data: 2024-04-19T23:26:42.076 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.076 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.076 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.076 DEBUG:boto:Params: {} 2024-04-19T23:26:42.076 DEBUG:boto:Token: None 2024-04-19T23:26:42.076 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.076 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA= 2024-04-19T23:26:42.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA=', 'Content-Length': '0'} 2024-04-19T23:26:42.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ced12870502aa6a6-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.078 DEBUG:boto:path=/aktlji-16/obj44 2024-04-19T23:26:42.078 DEBUG:boto:auth_path=/aktlji-16/obj44 2024-04-19T23:26:42.079 DEBUG:boto:path=/aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.079 DEBUG:boto:auth_path=/aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.079 DEBUG:boto:Method: GET 2024-04-19T23:26:42.079 DEBUG:boto:Path: /aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.079 DEBUG:boto:Data: 2024-04-19T23:26:42.079 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.079 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.079 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.079 DEBUG:boto:Params: {} 2024-04-19T23:26:42.079 DEBUG:boto:Token: None 2024-04-19T23:26:42.080 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.080 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA= 2024-04-19T23:26:42.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA=', 'Content-Length': '0'} 2024-04-19T23:26:42.082 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2ddb6956888b07f-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.082 DEBUG:boto:path=/aktlji-16/obj44 2024-04-19T23:26:42.082 DEBUG:boto:auth_path=/aktlji-16/obj44 2024-04-19T23:26:42.082 DEBUG:boto:path=/aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.082 DEBUG:boto:auth_path=/aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.082 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.082 DEBUG:boto:Path: /aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.083 DEBUG:boto:Data: 2024-04-19T23:26:42.083 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.083 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.083 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.083 DEBUG:boto:Params: {} 2024-04-19T23:26:42.083 DEBUG:boto:Token: None 2024-04-19T23:26:42.083 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.085 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Qhc+M3g+f533oMy/asvV+KRpHjc= 2024-04-19T23:26:42.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Qhc+M3g+f533oMy/asvV+KRpHjc=', 'Content-Length': '0'} 2024-04-19T23:26:42.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9c2d47fe512bf53-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.087 DEBUG:boto:path=/aktlji-16/obj44 2024-04-19T23:26:42.087 DEBUG:boto:auth_path=/aktlji-16/obj44 2024-04-19T23:26:42.087 DEBUG:boto:path=/aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.087 DEBUG:boto:auth_path=/aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.088 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.088 DEBUG:boto:Path: /aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.088 DEBUG:boto:Data: 2024-04-19T23:26:42.088 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.088 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.088 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.088 DEBUG:boto:Params: {} 2024-04-19T23:26:42.088 DEBUG:boto:Token: None 2024-04-19T23:26:42.088 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.088 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Qhc+M3g+f533oMy/asvV+KRpHjc= 2024-04-19T23:26:42.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Qhc+M3g+f533oMy/asvV+KRpHjc=', 'Content-Length': '0'} 2024-04-19T23:26:42.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bed961fbe7fd17cb-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-19T23:26:42.091 DEBUG:boto:path=/aktlji-16/obj44 2024-04-19T23:26:42.091 DEBUG:boto:auth_path=/aktlji-16/obj44 2024-04-19T23:26:42.091 DEBUG:boto:path=/aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.091 DEBUG:boto:auth_path=/aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.091 DEBUG:boto:Method: GET 2024-04-19T23:26:42.091 DEBUG:boto:Path: /aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.091 DEBUG:boto:Data: 2024-04-19T23:26:42.091 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.091 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.091 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.092 DEBUG:boto:Params: {} 2024-04-19T23:26:42.092 DEBUG:boto:Token: None 2024-04-19T23:26:42.092 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.092 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA= 2024-04-19T23:26:42.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA=', 'Content-Length': '0'} 2024-04-19T23:26:42.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3d544383d91b98c-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.094 DEBUG:boto:path=/aktlji-16/obj44 2024-04-19T23:26:42.094 DEBUG:boto:auth_path=/aktlji-16/obj44 2024-04-19T23:26:42.095 DEBUG:boto:path=/aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.095 DEBUG:boto:auth_path=/aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.095 DEBUG:boto:Method: GET 2024-04-19T23:26:42.095 DEBUG:boto:Path: /aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.095 DEBUG:boto:Data: 2024-04-19T23:26:42.095 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.095 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.095 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.095 DEBUG:boto:Params: {} 2024-04-19T23:26:42.095 DEBUG:boto:Token: None 2024-04-19T23:26:42.096 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj44?versionId=null 2024-04-19T23:26:42.096 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA= 2024-04-19T23:26:42.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA=', 'Content-Length': '0'} 2024-04-19T23:26:42.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b650f35e200ce6c-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.098 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.098 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.098 DEBUG:boto:path=/aktlji-16/?prefix=obj44 2024-04-19T23:26:42.098 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj44 2024-04-19T23:26:42.098 DEBUG:boto:Method: GET 2024-04-19T23:26:42.098 DEBUG:boto:Path: /aktlji-16/?prefix=obj44 2024-04-19T23:26:42.098 DEBUG:boto:Data: 2024-04-19T23:26:42.098 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.098 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.099 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.099 DEBUG:boto:Params: {} 2024-04-19T23:26:42.099 DEBUG:boto:Token: None 2024-04-19T23:26:42.099 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.099 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095882cce684d041d-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.102 DEBUG:boto:b'aktlji-16obj441000falseobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.102 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.102 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.102 DEBUG:boto:path=/aktlji-16/?prefix=obj44 2024-04-19T23:26:42.102 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj44 2024-04-19T23:26:42.102 DEBUG:boto:Method: GET 2024-04-19T23:26:42.103 DEBUG:boto:Path: /aktlji-16/?prefix=obj44 2024-04-19T23:26:42.103 DEBUG:boto:Data: 2024-04-19T23:26:42.103 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.103 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.103 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.103 DEBUG:boto:Params: {} 2024-04-19T23:26:42.103 DEBUG:boto:Token: None 2024-04-19T23:26:42.103 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.103 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001780cfe6359f65e3-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.106 DEBUG:boto:b'aktlji-16obj441000falseobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.106 DEBUG:boto:path=/aktlji-16/obj44 2024-04-19T23:26:42.106 DEBUG:boto:auth_path=/aktlji-16/obj44 2024-04-19T23:26:42.107 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.107 DEBUG:boto:Path: /aktlji-16/obj44 2024-04-19T23:26:42.107 DEBUG:boto:Data: 2024-04-19T23:26:42.107 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.107 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.107 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.107 DEBUG:boto:Params: {} 2024-04-19T23:26:42.107 DEBUG:boto:Token: None 2024-04-19T23:26:42.107 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj44 2024-04-19T23:26:42.107 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DIGLAkFU51o24FhZlFy4U6gOBoU= 2024-04-19T23:26:42.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DIGLAkFU51o24FhZlFy4U6gOBoU=', 'Content-Length': '0'} 2024-04-19T23:26:42.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a34fd2778dc8b1bf-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.109 DEBUG:boto:path=/aktlji-16/obj44 2024-04-19T23:26:42.110 DEBUG:boto:auth_path=/aktlji-16/obj44 2024-04-19T23:26:42.110 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.110 DEBUG:boto:Path: /aktlji-16/obj44 2024-04-19T23:26:42.110 DEBUG:boto:Data: 2024-04-19T23:26:42.110 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.110 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.110 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.110 DEBUG:boto:Params: {} 2024-04-19T23:26:42.110 DEBUG:boto:Token: None 2024-04-19T23:26:42.111 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj44 2024-04-19T23:26:42.111 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DIGLAkFU51o24FhZlFy4U6gOBoU= 2024-04-19T23:26:42.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DIGLAkFU51o24FhZlFy4U6gOBoU=', 'Content-Length': '0'} 2024-04-19T23:26:42.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bebed758cb5f9b92-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-19T23:26:42.113 DEBUG:boto:path=/aktlji-16/obj44 2024-04-19T23:26:42.113 DEBUG:boto:auth_path=/aktlji-16/obj44 2024-04-19T23:26:42.113 DEBUG:boto:Method: GET 2024-04-19T23:26:42.113 DEBUG:boto:Path: /aktlji-16/obj44 2024-04-19T23:26:42.113 DEBUG:boto:Data: 2024-04-19T23:26:42.113 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.113 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.113 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.113 DEBUG:boto:Params: {} 2024-04-19T23:26:42.114 DEBUG:boto:Token: None 2024-04-19T23:26:42.114 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj44 2024-04-19T23:26:42.114 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SjA8iq123dcWVepe64XIr3saFIA= 2024-04-19T23:26:42.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SjA8iq123dcWVepe64XIr3saFIA=', 'Content-Length': '0'} 2024-04-19T23:26:42.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c7ad6a2b757e364-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.116 DEBUG:boto:path=/aktlji-16/obj44 2024-04-19T23:26:42.116 DEBUG:boto:auth_path=/aktlji-16/obj44 2024-04-19T23:26:42.116 DEBUG:boto:Method: GET 2024-04-19T23:26:42.116 DEBUG:boto:Path: /aktlji-16/obj44 2024-04-19T23:26:42.116 DEBUG:boto:Data: 2024-04-19T23:26:42.116 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.116 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.117 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.117 DEBUG:boto:Params: {} 2024-04-19T23:26:42.117 DEBUG:boto:Token: None 2024-04-19T23:26:42.117 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj44 2024-04-19T23:26:42.117 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SjA8iq123dcWVepe64XIr3saFIA= 2024-04-19T23:26:42.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SjA8iq123dcWVepe64XIr3saFIA=', 'Content-Length': '0'} 2024-04-19T23:26:42.119 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d86086fdf0fe6ddb-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.119 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.119 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.119 DEBUG:boto:path=/aktlji-16/?prefix=obj44 2024-04-19T23:26:42.119 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj44 2024-04-19T23:26:42.119 DEBUG:boto:Method: GET 2024-04-19T23:26:42.120 DEBUG:boto:Path: /aktlji-16/?prefix=obj44 2024-04-19T23:26:42.120 DEBUG:boto:Data: 2024-04-19T23:26:42.120 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.120 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.120 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.120 DEBUG:boto:Params: {} 2024-04-19T23:26:42.120 DEBUG:boto:Token: None 2024-04-19T23:26:42.120 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.120 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f803dd832b7c230c-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.123 DEBUG:boto:b'aktlji-16obj441000falseobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.124 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.124 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.124 DEBUG:boto:path=/aktlji-16/?prefix=obj44 2024-04-19T23:26:42.124 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj44 2024-04-19T23:26:42.124 DEBUG:boto:Method: GET 2024-04-19T23:26:42.124 DEBUG:boto:Path: /aktlji-16/?prefix=obj44 2024-04-19T23:26:42.124 DEBUG:boto:Data: 2024-04-19T23:26:42.124 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.124 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.124 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.124 DEBUG:boto:Params: {} 2024-04-19T23:26:42.125 DEBUG:boto:Token: None 2024-04-19T23:26:42.125 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.125 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebece6c8ec97553c-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.174 DEBUG:boto:b'aktlji-16obj441000falseobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj45 2024-04-19T23:26:42.175 DEBUG:boto:path=/aktlji-16/obj45 2024-04-19T23:26:42.175 DEBUG:boto:auth_path=/aktlji-16/obj45 2024-04-19T23:26:42.175 DEBUG:boto:path=/aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.175 DEBUG:boto:auth_path=/aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.175 DEBUG:boto:Method: GET 2024-04-19T23:26:42.175 DEBUG:boto:Path: /aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.175 DEBUG:boto:Data: 2024-04-19T23:26:42.176 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.176 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.176 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.176 DEBUG:boto:Params: {} 2024-04-19T23:26:42.176 DEBUG:boto:Token: None 2024-04-19T23:26:42.176 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.176 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0= 2024-04-19T23:26:42.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0=', 'Content-Length': '0'} 2024-04-19T23:26:42.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071d9652a279d7929-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.179 DEBUG:boto:path=/aktlji-16/obj45 2024-04-19T23:26:42.179 DEBUG:boto:auth_path=/aktlji-16/obj45 2024-04-19T23:26:42.179 DEBUG:boto:path=/aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.179 DEBUG:boto:auth_path=/aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.179 DEBUG:boto:Method: GET 2024-04-19T23:26:42.179 DEBUG:boto:Path: /aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.179 DEBUG:boto:Data: 2024-04-19T23:26:42.179 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.179 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.179 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.180 DEBUG:boto:Params: {} 2024-04-19T23:26:42.180 DEBUG:boto:Token: None 2024-04-19T23:26:42.180 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0= 2024-04-19T23:26:42.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0=', 'Content-Length': '0'} 2024-04-19T23:26:42.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b2bf5013916af95-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.182 DEBUG:boto:path=/aktlji-16/obj45 2024-04-19T23:26:42.183 DEBUG:boto:auth_path=/aktlji-16/obj45 2024-04-19T23:26:42.183 DEBUG:boto:path=/aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.183 DEBUG:boto:auth_path=/aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.183 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.183 DEBUG:boto:Path: /aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.183 DEBUG:boto:Data: 2024-04-19T23:26:42.183 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.183 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.183 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.183 DEBUG:boto:Params: {} 2024-04-19T23:26:42.183 DEBUG:boto:Token: None 2024-04-19T23:26:42.184 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.184 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zmmZypW4JBfLqspFvAQZyLugX48= 2024-04-19T23:26:42.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zmmZypW4JBfLqspFvAQZyLugX48=', 'Content-Length': '0'} 2024-04-19T23:26:42.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e2bd7dd0823d3a9-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.186 DEBUG:boto:path=/aktlji-16/obj45 2024-04-19T23:26:42.186 DEBUG:boto:auth_path=/aktlji-16/obj45 2024-04-19T23:26:42.186 DEBUG:boto:path=/aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.186 DEBUG:boto:auth_path=/aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.186 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.186 DEBUG:boto:Path: /aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.187 DEBUG:boto:Data: 2024-04-19T23:26:42.187 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.187 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.187 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.187 DEBUG:boto:Params: {} 2024-04-19T23:26:42.187 DEBUG:boto:Token: None 2024-04-19T23:26:42.187 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.187 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zmmZypW4JBfLqspFvAQZyLugX48= 2024-04-19T23:26:42.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zmmZypW4JBfLqspFvAQZyLugX48=', 'Content-Length': '0'} 2024-04-19T23:26:42.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0f37d0cbf63960c-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj45 2024-04-19T23:26:42.189 DEBUG:boto:path=/aktlji-16/obj45 2024-04-19T23:26:42.189 DEBUG:boto:auth_path=/aktlji-16/obj45 2024-04-19T23:26:42.190 DEBUG:boto:path=/aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.190 DEBUG:boto:auth_path=/aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.190 DEBUG:boto:Method: GET 2024-04-19T23:26:42.190 DEBUG:boto:Path: /aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.190 DEBUG:boto:Data: 2024-04-19T23:26:42.190 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.190 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.190 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.190 DEBUG:boto:Params: {} 2024-04-19T23:26:42.190 DEBUG:boto:Token: None 2024-04-19T23:26:42.191 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.191 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0= 2024-04-19T23:26:42.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0=', 'Content-Length': '0'} 2024-04-19T23:26:42.192 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5a1e6f239fe55a1-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.234 DEBUG:boto:path=/aktlji-16/obj45 2024-04-19T23:26:42.235 DEBUG:boto:auth_path=/aktlji-16/obj45 2024-04-19T23:26:42.235 DEBUG:boto:path=/aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.235 DEBUG:boto:auth_path=/aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.235 DEBUG:boto:Method: GET 2024-04-19T23:26:42.235 DEBUG:boto:Path: /aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.235 DEBUG:boto:Data: 2024-04-19T23:26:42.235 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.235 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.235 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.235 DEBUG:boto:Params: {} 2024-04-19T23:26:42.236 DEBUG:boto:Token: None 2024-04-19T23:26:42.236 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj45?versionId=null 2024-04-19T23:26:42.236 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0= 2024-04-19T23:26:42.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0=', 'Content-Length': '0'} 2024-04-19T23:26:42.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e410585349efbd0f-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.238 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.238 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.238 DEBUG:boto:path=/aktlji-16/?prefix=obj45 2024-04-19T23:26:42.238 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj45 2024-04-19T23:26:42.239 DEBUG:boto:Method: GET 2024-04-19T23:26:42.239 DEBUG:boto:Path: /aktlji-16/?prefix=obj45 2024-04-19T23:26:42.239 DEBUG:boto:Data: 2024-04-19T23:26:42.239 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.239 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.239 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.239 DEBUG:boto:Params: {} 2024-04-19T23:26:42.239 DEBUG:boto:Token: None 2024-04-19T23:26:42.239 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.239 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064ac8960642d8541-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.242 DEBUG:boto:b'aktlji-16obj451000falseobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.243 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.243 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.243 DEBUG:boto:path=/aktlji-16/?prefix=obj45 2024-04-19T23:26:42.243 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj45 2024-04-19T23:26:42.243 DEBUG:boto:Method: GET 2024-04-19T23:26:42.243 DEBUG:boto:Path: /aktlji-16/?prefix=obj45 2024-04-19T23:26:42.243 DEBUG:boto:Data: 2024-04-19T23:26:42.243 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.243 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.244 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.244 DEBUG:boto:Params: {} 2024-04-19T23:26:42.244 DEBUG:boto:Token: None 2024-04-19T23:26:42.244 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.244 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c088d5db2d003848-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.247 DEBUG:boto:b'aktlji-16obj451000falseobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.247 DEBUG:boto:path=/aktlji-16/obj45 2024-04-19T23:26:42.247 DEBUG:boto:auth_path=/aktlji-16/obj45 2024-04-19T23:26:42.248 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.248 DEBUG:boto:Path: /aktlji-16/obj45 2024-04-19T23:26:42.248 DEBUG:boto:Data: 2024-04-19T23:26:42.248 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.248 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.248 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.248 DEBUG:boto:Params: {} 2024-04-19T23:26:42.248 DEBUG:boto:Token: None 2024-04-19T23:26:42.248 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj45 2024-04-19T23:26:42.248 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bmwdV+rEIAbHM/+ysoHD041OCUw= 2024-04-19T23:26:42.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bmwdV+rEIAbHM/+ysoHD041OCUw=', 'Content-Length': '0'} 2024-04-19T23:26:42.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000515c46cebce9bee3-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.250 DEBUG:boto:path=/aktlji-16/obj45 2024-04-19T23:26:42.251 DEBUG:boto:auth_path=/aktlji-16/obj45 2024-04-19T23:26:42.251 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.251 DEBUG:boto:Path: /aktlji-16/obj45 2024-04-19T23:26:42.251 DEBUG:boto:Data: 2024-04-19T23:26:42.251 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.251 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.251 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.251 DEBUG:boto:Params: {} 2024-04-19T23:26:42.251 DEBUG:boto:Token: None 2024-04-19T23:26:42.251 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj45 2024-04-19T23:26:42.252 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bmwdV+rEIAbHM/+ysoHD041OCUw= 2024-04-19T23:26:42.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bmwdV+rEIAbHM/+ysoHD041OCUw=', 'Content-Length': '0'} 2024-04-19T23:26:42.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038d7ce295ce19219-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj45 2024-04-19T23:26:42.254 DEBUG:boto:path=/aktlji-16/obj45 2024-04-19T23:26:42.254 DEBUG:boto:auth_path=/aktlji-16/obj45 2024-04-19T23:26:42.254 DEBUG:boto:Method: GET 2024-04-19T23:26:42.254 DEBUG:boto:Path: /aktlji-16/obj45 2024-04-19T23:26:42.254 DEBUG:boto:Data: 2024-04-19T23:26:42.254 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.255 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.255 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.255 DEBUG:boto:Params: {} 2024-04-19T23:26:42.255 DEBUG:boto:Token: None 2024-04-19T23:26:42.255 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj45 2024-04-19T23:26:42.255 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/9En4I36Ej589rqc7/gxX19MVzg= 2024-04-19T23:26:42.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/9En4I36Ej589rqc7/gxX19MVzg=', 'Content-Length': '0'} 2024-04-19T23:26:42.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050969d20ddf2a7cc-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.258 DEBUG:boto:path=/aktlji-16/obj45 2024-04-19T23:26:42.258 DEBUG:boto:auth_path=/aktlji-16/obj45 2024-04-19T23:26:42.258 DEBUG:boto:Method: GET 2024-04-19T23:26:42.258 DEBUG:boto:Path: /aktlji-16/obj45 2024-04-19T23:26:42.258 DEBUG:boto:Data: 2024-04-19T23:26:42.258 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.258 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.259 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.259 DEBUG:boto:Params: {} 2024-04-19T23:26:42.259 DEBUG:boto:Token: None 2024-04-19T23:26:42.259 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj45 2024-04-19T23:26:42.259 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/9En4I36Ej589rqc7/gxX19MVzg= 2024-04-19T23:26:42.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/9En4I36Ej589rqc7/gxX19MVzg=', 'Content-Length': '0'} 2024-04-19T23:26:42.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ffaa6f101e445f3-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.302 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.303 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.303 DEBUG:boto:path=/aktlji-16/?prefix=obj45 2024-04-19T23:26:42.303 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj45 2024-04-19T23:26:42.303 DEBUG:boto:Method: GET 2024-04-19T23:26:42.303 DEBUG:boto:Path: /aktlji-16/?prefix=obj45 2024-04-19T23:26:42.303 DEBUG:boto:Data: 2024-04-19T23:26:42.303 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.303 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.303 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.303 DEBUG:boto:Params: {} 2024-04-19T23:26:42.304 DEBUG:boto:Token: None 2024-04-19T23:26:42.304 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.304 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009344a159b069d2f8-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.307 DEBUG:boto:b'aktlji-16obj451000falseobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.307 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.307 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.307 DEBUG:boto:path=/aktlji-16/?prefix=obj45 2024-04-19T23:26:42.307 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj45 2024-04-19T23:26:42.307 DEBUG:boto:Method: GET 2024-04-19T23:26:42.308 DEBUG:boto:Path: /aktlji-16/?prefix=obj45 2024-04-19T23:26:42.308 DEBUG:boto:Data: 2024-04-19T23:26:42.308 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.308 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.308 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.308 DEBUG:boto:Params: {} 2024-04-19T23:26:42.308 DEBUG:boto:Token: None 2024-04-19T23:26:42.308 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.308 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009836123faafa6e2c-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.311 DEBUG:boto:b'aktlji-16obj451000falseobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-19T23:26:42.312 DEBUG:boto:path=/aktlji-16/obj46 2024-04-19T23:26:42.312 DEBUG:boto:auth_path=/aktlji-16/obj46 2024-04-19T23:26:42.312 DEBUG:boto:path=/aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.312 DEBUG:boto:auth_path=/aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.312 DEBUG:boto:Method: GET 2024-04-19T23:26:42.312 DEBUG:boto:Path: /aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.313 DEBUG:boto:Data: 2024-04-19T23:26:42.313 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.313 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.313 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.313 DEBUG:boto:Params: {} 2024-04-19T23:26:42.313 DEBUG:boto:Token: None 2024-04-19T23:26:42.313 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.313 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY= 2024-04-19T23:26:42.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY=', 'Content-Length': '0'} 2024-04-19T23:26:42.315 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b94bda5ce0ad8123-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.315 DEBUG:boto:path=/aktlji-16/obj46 2024-04-19T23:26:42.316 DEBUG:boto:auth_path=/aktlji-16/obj46 2024-04-19T23:26:42.316 DEBUG:boto:path=/aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.316 DEBUG:boto:auth_path=/aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.316 DEBUG:boto:Method: GET 2024-04-19T23:26:42.316 DEBUG:boto:Path: /aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.316 DEBUG:boto:Data: 2024-04-19T23:26:42.316 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.316 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.316 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.316 DEBUG:boto:Params: {} 2024-04-19T23:26:42.316 DEBUG:boto:Token: None 2024-04-19T23:26:42.317 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.317 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY= 2024-04-19T23:26:42.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY=', 'Content-Length': '0'} 2024-04-19T23:26:42.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adc75fb8c19fd445-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.319 DEBUG:boto:path=/aktlji-16/obj46 2024-04-19T23:26:42.319 DEBUG:boto:auth_path=/aktlji-16/obj46 2024-04-19T23:26:42.319 DEBUG:boto:path=/aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.319 DEBUG:boto:auth_path=/aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.319 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.319 DEBUG:boto:Path: /aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.319 DEBUG:boto:Data: 2024-04-19T23:26:42.320 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.320 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.320 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.320 DEBUG:boto:Params: {} 2024-04-19T23:26:42.320 DEBUG:boto:Token: None 2024-04-19T23:26:42.320 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.320 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zAZD3tzKTHrXZZB9iFO8J+9lvpc= 2024-04-19T23:26:42.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zAZD3tzKTHrXZZB9iFO8J+9lvpc=', 'Content-Length': '0'} 2024-04-19T23:26:42.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e588260284d201cc-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.323 DEBUG:boto:path=/aktlji-16/obj46 2024-04-19T23:26:42.323 DEBUG:boto:auth_path=/aktlji-16/obj46 2024-04-19T23:26:42.323 DEBUG:boto:path=/aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.323 DEBUG:boto:auth_path=/aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.323 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.323 DEBUG:boto:Path: /aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.323 DEBUG:boto:Data: 2024-04-19T23:26:42.323 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.323 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.323 DEBUG:boto:Params: {} 2024-04-19T23:26:42.324 DEBUG:boto:Token: None 2024-04-19T23:26:42.324 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zAZD3tzKTHrXZZB9iFO8J+9lvpc= 2024-04-19T23:26:42.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zAZD3tzKTHrXZZB9iFO8J+9lvpc=', 'Content-Length': '0'} 2024-04-19T23:26:42.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa5a98a89a511622-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-19T23:26:42.326 DEBUG:boto:path=/aktlji-16/obj46 2024-04-19T23:26:42.326 DEBUG:boto:auth_path=/aktlji-16/obj46 2024-04-19T23:26:42.326 DEBUG:boto:path=/aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.326 DEBUG:boto:auth_path=/aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.327 DEBUG:boto:Method: GET 2024-04-19T23:26:42.327 DEBUG:boto:Path: /aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.327 DEBUG:boto:Data: 2024-04-19T23:26:42.327 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.327 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.327 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.327 DEBUG:boto:Params: {} 2024-04-19T23:26:42.327 DEBUG:boto:Token: None 2024-04-19T23:26:42.327 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.327 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY= 2024-04-19T23:26:42.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY=', 'Content-Length': '0'} 2024-04-19T23:26:42.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036c38b2364ba1e3b-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.374 DEBUG:boto:path=/aktlji-16/obj46 2024-04-19T23:26:42.374 DEBUG:boto:auth_path=/aktlji-16/obj46 2024-04-19T23:26:42.375 DEBUG:boto:path=/aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.375 DEBUG:boto:auth_path=/aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.375 DEBUG:boto:Method: GET 2024-04-19T23:26:42.375 DEBUG:boto:Path: /aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.375 DEBUG:boto:Data: 2024-04-19T23:26:42.375 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.375 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.375 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.375 DEBUG:boto:Params: {} 2024-04-19T23:26:42.375 DEBUG:boto:Token: None 2024-04-19T23:26:42.376 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj46?versionId=null 2024-04-19T23:26:42.376 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY= 2024-04-19T23:26:42.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY=', 'Content-Length': '0'} 2024-04-19T23:26:42.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4b7f97a61da9378-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.378 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.378 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.378 DEBUG:boto:path=/aktlji-16/?prefix=obj46 2024-04-19T23:26:42.378 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj46 2024-04-19T23:26:42.378 DEBUG:boto:Method: GET 2024-04-19T23:26:42.379 DEBUG:boto:Path: /aktlji-16/?prefix=obj46 2024-04-19T23:26:42.379 DEBUG:boto:Data: 2024-04-19T23:26:42.379 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.379 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.379 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.379 DEBUG:boto:Params: {} 2024-04-19T23:26:42.379 DEBUG:boto:Token: None 2024-04-19T23:26:42.379 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.379 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530e6b43630fe7cd-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.382 DEBUG:boto:b'aktlji-16obj461000falseobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.382 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.383 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.383 DEBUG:boto:path=/aktlji-16/?prefix=obj46 2024-04-19T23:26:42.383 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj46 2024-04-19T23:26:42.383 DEBUG:boto:Method: GET 2024-04-19T23:26:42.383 DEBUG:boto:Path: /aktlji-16/?prefix=obj46 2024-04-19T23:26:42.383 DEBUG:boto:Data: 2024-04-19T23:26:42.383 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.383 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.383 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.383 DEBUG:boto:Params: {} 2024-04-19T23:26:42.383 DEBUG:boto:Token: None 2024-04-19T23:26:42.384 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.384 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000031db4999a0105b7-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.386 DEBUG:boto:b'aktlji-16obj461000falseobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.387 DEBUG:boto:path=/aktlji-16/obj46 2024-04-19T23:26:42.387 DEBUG:boto:auth_path=/aktlji-16/obj46 2024-04-19T23:26:42.387 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.387 DEBUG:boto:Path: /aktlji-16/obj46 2024-04-19T23:26:42.387 DEBUG:boto:Data: 2024-04-19T23:26:42.387 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.387 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.388 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.388 DEBUG:boto:Params: {} 2024-04-19T23:26:42.388 DEBUG:boto:Token: None 2024-04-19T23:26:42.388 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj46 2024-04-19T23:26:42.388 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CvzPswguHImVzOaiXWqiQmv0kFU= 2024-04-19T23:26:42.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CvzPswguHImVzOaiXWqiQmv0kFU=', 'Content-Length': '0'} 2024-04-19T23:26:42.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a68e1458d997464-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.390 DEBUG:boto:path=/aktlji-16/obj46 2024-04-19T23:26:42.390 DEBUG:boto:auth_path=/aktlji-16/obj46 2024-04-19T23:26:42.390 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.390 DEBUG:boto:Path: /aktlji-16/obj46 2024-04-19T23:26:42.390 DEBUG:boto:Data: 2024-04-19T23:26:42.391 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.391 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.391 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.391 DEBUG:boto:Params: {} 2024-04-19T23:26:42.391 DEBUG:boto:Token: None 2024-04-19T23:26:42.391 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj46 2024-04-19T23:26:42.391 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CvzPswguHImVzOaiXWqiQmv0kFU= 2024-04-19T23:26:42.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CvzPswguHImVzOaiXWqiQmv0kFU=', 'Content-Length': '0'} 2024-04-19T23:26:42.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000460dbb1dc6b97d7a-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-19T23:26:42.394 DEBUG:boto:path=/aktlji-16/obj46 2024-04-19T23:26:42.394 DEBUG:boto:auth_path=/aktlji-16/obj46 2024-04-19T23:26:42.394 DEBUG:boto:Method: GET 2024-04-19T23:26:42.394 DEBUG:boto:Path: /aktlji-16/obj46 2024-04-19T23:26:42.394 DEBUG:boto:Data: 2024-04-19T23:26:42.394 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.394 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.394 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.394 DEBUG:boto:Params: {} 2024-04-19T23:26:42.394 DEBUG:boto:Token: None 2024-04-19T23:26:42.395 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj46 2024-04-19T23:26:42.395 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QT0tAxPcWAul5bzIOuOkXzXg5yo= 2024-04-19T23:26:42.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QT0tAxPcWAul5bzIOuOkXzXg5yo=', 'Content-Length': '0'} 2024-04-19T23:26:42.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082ada89ba37ca108-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.397 DEBUG:boto:path=/aktlji-16/obj46 2024-04-19T23:26:42.397 DEBUG:boto:auth_path=/aktlji-16/obj46 2024-04-19T23:26:42.397 DEBUG:boto:Method: GET 2024-04-19T23:26:42.397 DEBUG:boto:Path: /aktlji-16/obj46 2024-04-19T23:26:42.397 DEBUG:boto:Data: 2024-04-19T23:26:42.397 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.397 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.398 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.398 DEBUG:boto:Params: {} 2024-04-19T23:26:42.398 DEBUG:boto:Token: None 2024-04-19T23:26:42.398 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj46 2024-04-19T23:26:42.398 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QT0tAxPcWAul5bzIOuOkXzXg5yo= 2024-04-19T23:26:42.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QT0tAxPcWAul5bzIOuOkXzXg5yo=', 'Content-Length': '0'} 2024-04-19T23:26:42.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eefa112e0b17488f-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.400 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.400 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.400 DEBUG:boto:path=/aktlji-16/?prefix=obj46 2024-04-19T23:26:42.400 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj46 2024-04-19T23:26:42.400 DEBUG:boto:Method: GET 2024-04-19T23:26:42.400 DEBUG:boto:Path: /aktlji-16/?prefix=obj46 2024-04-19T23:26:42.401 DEBUG:boto:Data: 2024-04-19T23:26:42.401 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.401 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.401 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.401 DEBUG:boto:Params: {} 2024-04-19T23:26:42.401 DEBUG:boto:Token: None 2024-04-19T23:26:42.401 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.401 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d41927ce2ecab91-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.404 DEBUG:boto:b'aktlji-16obj461000falseobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.404 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.404 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.404 DEBUG:boto:path=/aktlji-16/?prefix=obj46 2024-04-19T23:26:42.404 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj46 2024-04-19T23:26:42.405 DEBUG:boto:Method: GET 2024-04-19T23:26:42.405 DEBUG:boto:Path: /aktlji-16/?prefix=obj46 2024-04-19T23:26:42.405 DEBUG:boto:Data: 2024-04-19T23:26:42.405 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.405 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.405 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.405 DEBUG:boto:Params: {} 2024-04-19T23:26:42.405 DEBUG:boto:Token: None 2024-04-19T23:26:42.405 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.405 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eccd71cb94a1cc5b-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.408 DEBUG:boto:b'aktlji-16obj461000falseobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj47 2024-04-19T23:26:42.409 DEBUG:boto:path=/aktlji-16/obj47 2024-04-19T23:26:42.409 DEBUG:boto:auth_path=/aktlji-16/obj47 2024-04-19T23:26:42.409 DEBUG:boto:path=/aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.409 DEBUG:boto:auth_path=/aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.409 DEBUG:boto:Method: GET 2024-04-19T23:26:42.409 DEBUG:boto:Path: /aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.409 DEBUG:boto:Data: 2024-04-19T23:26:42.409 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.409 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.410 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.410 DEBUG:boto:Params: {} 2024-04-19T23:26:42.410 DEBUG:boto:Token: None 2024-04-19T23:26:42.410 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.410 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY= 2024-04-19T23:26:42.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY=', 'Content-Length': '0'} 2024-04-19T23:26:42.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc11a14722a3a764-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.412 DEBUG:boto:path=/aktlji-16/obj47 2024-04-19T23:26:42.412 DEBUG:boto:auth_path=/aktlji-16/obj47 2024-04-19T23:26:42.413 DEBUG:boto:path=/aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.413 DEBUG:boto:auth_path=/aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.413 DEBUG:boto:Method: GET 2024-04-19T23:26:42.413 DEBUG:boto:Path: /aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.413 DEBUG:boto:Data: 2024-04-19T23:26:42.413 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.413 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.413 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.413 DEBUG:boto:Params: {} 2024-04-19T23:26:42.413 DEBUG:boto:Token: None 2024-04-19T23:26:42.414 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.414 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY= 2024-04-19T23:26:42.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY=', 'Content-Length': '0'} 2024-04-19T23:26:42.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070d3f40c13286b71-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.416 DEBUG:boto:path=/aktlji-16/obj47 2024-04-19T23:26:42.416 DEBUG:boto:auth_path=/aktlji-16/obj47 2024-04-19T23:26:42.416 DEBUG:boto:path=/aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.416 DEBUG:boto:auth_path=/aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.416 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.416 DEBUG:boto:Path: /aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.416 DEBUG:boto:Data: 2024-04-19T23:26:42.416 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.416 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.416 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.417 DEBUG:boto:Params: {} 2024-04-19T23:26:42.417 DEBUG:boto:Token: None 2024-04-19T23:26:42.417 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.417 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sPprbB2bcvgxEyejuDjZugxBERA= 2024-04-19T23:26:42.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sPprbB2bcvgxEyejuDjZugxBERA=', 'Content-Length': '0'} 2024-04-19T23:26:42.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a08148e10dace5dd-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.419 DEBUG:boto:path=/aktlji-16/obj47 2024-04-19T23:26:42.419 DEBUG:boto:auth_path=/aktlji-16/obj47 2024-04-19T23:26:42.419 DEBUG:boto:path=/aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.419 DEBUG:boto:auth_path=/aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.419 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.419 DEBUG:boto:Path: /aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.419 DEBUG:boto:Data: 2024-04-19T23:26:42.420 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.420 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.420 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.420 DEBUG:boto:Params: {} 2024-04-19T23:26:42.420 DEBUG:boto:Token: None 2024-04-19T23:26:42.420 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.420 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sPprbB2bcvgxEyejuDjZugxBERA= 2024-04-19T23:26:42.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sPprbB2bcvgxEyejuDjZugxBERA=', 'Content-Length': '0'} 2024-04-19T23:26:42.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f679cf61945e49e-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj47 2024-04-19T23:26:42.422 DEBUG:boto:path=/aktlji-16/obj47 2024-04-19T23:26:42.422 DEBUG:boto:auth_path=/aktlji-16/obj47 2024-04-19T23:26:42.422 DEBUG:boto:path=/aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.423 DEBUG:boto:auth_path=/aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.423 DEBUG:boto:Method: GET 2024-04-19T23:26:42.423 DEBUG:boto:Path: /aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.423 DEBUG:boto:Data: 2024-04-19T23:26:42.423 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.423 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.423 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.423 DEBUG:boto:Params: {} 2024-04-19T23:26:42.423 DEBUG:boto:Token: None 2024-04-19T23:26:42.423 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.424 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY= 2024-04-19T23:26:42.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY=', 'Content-Length': '0'} 2024-04-19T23:26:42.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006947a99e1a84544-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.470 DEBUG:boto:path=/aktlji-16/obj47 2024-04-19T23:26:42.470 DEBUG:boto:auth_path=/aktlji-16/obj47 2024-04-19T23:26:42.471 DEBUG:boto:path=/aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.471 DEBUG:boto:auth_path=/aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.471 DEBUG:boto:Method: GET 2024-04-19T23:26:42.471 DEBUG:boto:Path: /aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.471 DEBUG:boto:Data: 2024-04-19T23:26:42.471 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.471 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.471 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.471 DEBUG:boto:Params: {} 2024-04-19T23:26:42.471 DEBUG:boto:Token: None 2024-04-19T23:26:42.472 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj47?versionId=null 2024-04-19T23:26:42.472 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY= 2024-04-19T23:26:42.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY=', 'Content-Length': '0'} 2024-04-19T23:26:42.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dea9ad4930cd571-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.474 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.474 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.474 DEBUG:boto:path=/aktlji-16/?prefix=obj47 2024-04-19T23:26:42.474 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj47 2024-04-19T23:26:42.474 DEBUG:boto:Method: GET 2024-04-19T23:26:42.475 DEBUG:boto:Path: /aktlji-16/?prefix=obj47 2024-04-19T23:26:42.475 DEBUG:boto:Data: 2024-04-19T23:26:42.475 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.475 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.475 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.475 DEBUG:boto:Params: {} 2024-04-19T23:26:42.475 DEBUG:boto:Token: None 2024-04-19T23:26:42.475 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.475 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad8e005ad53d6a84-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.478 DEBUG:boto:b'aktlji-16obj471000falseobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.478 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.478 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.478 DEBUG:boto:path=/aktlji-16/?prefix=obj47 2024-04-19T23:26:42.479 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj47 2024-04-19T23:26:42.479 DEBUG:boto:Method: GET 2024-04-19T23:26:42.479 DEBUG:boto:Path: /aktlji-16/?prefix=obj47 2024-04-19T23:26:42.479 DEBUG:boto:Data: 2024-04-19T23:26:42.479 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.479 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.479 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.479 DEBUG:boto:Params: {} 2024-04-19T23:26:42.479 DEBUG:boto:Token: None 2024-04-19T23:26:42.479 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.479 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee4bbf8624eab54c-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.482 DEBUG:boto:b'aktlji-16obj471000falseobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.482 DEBUG:boto:path=/aktlji-16/obj47 2024-04-19T23:26:42.483 DEBUG:boto:auth_path=/aktlji-16/obj47 2024-04-19T23:26:42.483 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.483 DEBUG:boto:Path: /aktlji-16/obj47 2024-04-19T23:26:42.483 DEBUG:boto:Data: 2024-04-19T23:26:42.483 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.483 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.483 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.483 DEBUG:boto:Params: {} 2024-04-19T23:26:42.483 DEBUG:boto:Token: None 2024-04-19T23:26:42.483 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj47 2024-04-19T23:26:42.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xJNAZhPhZLDHWLEsg7/bZs0tWsw= 2024-04-19T23:26:42.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xJNAZhPhZLDHWLEsg7/bZs0tWsw=', 'Content-Length': '0'} 2024-04-19T23:26:42.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001413ea85fec4b2fa-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.486 DEBUG:boto:path=/aktlji-16/obj47 2024-04-19T23:26:42.486 DEBUG:boto:auth_path=/aktlji-16/obj47 2024-04-19T23:26:42.486 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.486 DEBUG:boto:Path: /aktlji-16/obj47 2024-04-19T23:26:42.486 DEBUG:boto:Data: 2024-04-19T23:26:42.486 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.486 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.486 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.486 DEBUG:boto:Params: {} 2024-04-19T23:26:42.487 DEBUG:boto:Token: None 2024-04-19T23:26:42.487 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj47 2024-04-19T23:26:42.487 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xJNAZhPhZLDHWLEsg7/bZs0tWsw= 2024-04-19T23:26:42.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xJNAZhPhZLDHWLEsg7/bZs0tWsw=', 'Content-Length': '0'} 2024-04-19T23:26:42.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f008cc73e32e3de1-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj47 2024-04-19T23:26:42.489 DEBUG:boto:path=/aktlji-16/obj47 2024-04-19T23:26:42.489 DEBUG:boto:auth_path=/aktlji-16/obj47 2024-04-19T23:26:42.489 DEBUG:boto:Method: GET 2024-04-19T23:26:42.489 DEBUG:boto:Path: /aktlji-16/obj47 2024-04-19T23:26:42.489 DEBUG:boto:Data: 2024-04-19T23:26:42.489 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.490 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.490 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.490 DEBUG:boto:Params: {} 2024-04-19T23:26:42.490 DEBUG:boto:Token: None 2024-04-19T23:26:42.490 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj47 2024-04-19T23:26:42.490 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3rIXwprxpB/nZlAehMuUPQDWF4w= 2024-04-19T23:26:42.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3rIXwprxpB/nZlAehMuUPQDWF4w=', 'Content-Length': '0'} 2024-04-19T23:26:42.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b689ac69b708dd11-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.492 DEBUG:boto:path=/aktlji-16/obj47 2024-04-19T23:26:42.492 DEBUG:boto:auth_path=/aktlji-16/obj47 2024-04-19T23:26:42.492 DEBUG:boto:Method: GET 2024-04-19T23:26:42.492 DEBUG:boto:Path: /aktlji-16/obj47 2024-04-19T23:26:42.492 DEBUG:boto:Data: 2024-04-19T23:26:42.493 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.493 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.493 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.493 DEBUG:boto:Params: {} 2024-04-19T23:26:42.493 DEBUG:boto:Token: None 2024-04-19T23:26:42.493 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj47 2024-04-19T23:26:42.493 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3rIXwprxpB/nZlAehMuUPQDWF4w= 2024-04-19T23:26:42.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3rIXwprxpB/nZlAehMuUPQDWF4w=', 'Content-Length': '0'} 2024-04-19T23:26:42.495 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b5f3dbec6210459-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.495 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.495 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.495 DEBUG:boto:path=/aktlji-16/?prefix=obj47 2024-04-19T23:26:42.496 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj47 2024-04-19T23:26:42.496 DEBUG:boto:Method: GET 2024-04-19T23:26:42.496 DEBUG:boto:Path: /aktlji-16/?prefix=obj47 2024-04-19T23:26:42.496 DEBUG:boto:Data: 2024-04-19T23:26:42.496 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.496 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.496 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.496 DEBUG:boto:Params: {} 2024-04-19T23:26:42.496 DEBUG:boto:Token: None 2024-04-19T23:26:42.496 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.496 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000249d2400fed99424-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.499 DEBUG:boto:b'aktlji-16obj471000falseobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.499 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.500 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.500 DEBUG:boto:path=/aktlji-16/?prefix=obj47 2024-04-19T23:26:42.500 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj47 2024-04-19T23:26:42.500 DEBUG:boto:Method: GET 2024-04-19T23:26:42.500 DEBUG:boto:Path: /aktlji-16/?prefix=obj47 2024-04-19T23:26:42.500 DEBUG:boto:Data: 2024-04-19T23:26:42.500 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.500 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.500 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.500 DEBUG:boto:Params: {} 2024-04-19T23:26:42.500 DEBUG:boto:Token: None 2024-04-19T23:26:42.501 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.501 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c0756cabb8b1135-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.546 DEBUG:boto:b'aktlji-16obj471000falseobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj48 2024-04-19T23:26:42.547 DEBUG:boto:path=/aktlji-16/obj48 2024-04-19T23:26:42.547 DEBUG:boto:auth_path=/aktlji-16/obj48 2024-04-19T23:26:42.547 DEBUG:boto:path=/aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.547 DEBUG:boto:auth_path=/aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.547 DEBUG:boto:Method: GET 2024-04-19T23:26:42.547 DEBUG:boto:Path: /aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.548 DEBUG:boto:Data: 2024-04-19T23:26:42.548 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.548 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.548 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.548 DEBUG:boto:Params: {} 2024-04-19T23:26:42.548 DEBUG:boto:Token: None 2024-04-19T23:26:42.548 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.548 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0= 2024-04-19T23:26:42.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0=', 'Content-Length': '0'} 2024-04-19T23:26:42.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5bc8c9f24795e46-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.551 DEBUG:boto:path=/aktlji-16/obj48 2024-04-19T23:26:42.551 DEBUG:boto:auth_path=/aktlji-16/obj48 2024-04-19T23:26:42.551 DEBUG:boto:path=/aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.551 DEBUG:boto:auth_path=/aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.551 DEBUG:boto:Method: GET 2024-04-19T23:26:42.551 DEBUG:boto:Path: /aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.551 DEBUG:boto:Data: 2024-04-19T23:26:42.551 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.551 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.551 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.552 DEBUG:boto:Params: {} 2024-04-19T23:26:42.552 DEBUG:boto:Token: None 2024-04-19T23:26:42.552 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.552 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0= 2024-04-19T23:26:42.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0=', 'Content-Length': '0'} 2024-04-19T23:26:42.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfc412b344421447-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.554 DEBUG:boto:path=/aktlji-16/obj48 2024-04-19T23:26:42.554 DEBUG:boto:auth_path=/aktlji-16/obj48 2024-04-19T23:26:42.554 DEBUG:boto:path=/aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.554 DEBUG:boto:auth_path=/aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.554 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.555 DEBUG:boto:Path: /aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.555 DEBUG:boto:Data: 2024-04-19T23:26:42.555 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.555 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.555 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.555 DEBUG:boto:Params: {} 2024-04-19T23:26:42.555 DEBUG:boto:Token: None 2024-04-19T23:26:42.555 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.555 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EO9SGck1cNQ37eBxu4ynRBtZXD4= 2024-04-19T23:26:42.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EO9SGck1cNQ37eBxu4ynRBtZXD4=', 'Content-Length': '0'} 2024-04-19T23:26:42.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d3dc31daeb330ba-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.557 DEBUG:boto:path=/aktlji-16/obj48 2024-04-19T23:26:42.558 DEBUG:boto:auth_path=/aktlji-16/obj48 2024-04-19T23:26:42.558 DEBUG:boto:path=/aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.558 DEBUG:boto:auth_path=/aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.558 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.558 DEBUG:boto:Path: /aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.558 DEBUG:boto:Data: 2024-04-19T23:26:42.558 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.558 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.558 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.558 DEBUG:boto:Params: {} 2024-04-19T23:26:42.559 DEBUG:boto:Token: None 2024-04-19T23:26:42.559 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.559 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EO9SGck1cNQ37eBxu4ynRBtZXD4= 2024-04-19T23:26:42.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EO9SGck1cNQ37eBxu4ynRBtZXD4=', 'Content-Length': '0'} 2024-04-19T23:26:42.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093b1388be12c5f81-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj48 2024-04-19T23:26:42.561 DEBUG:boto:path=/aktlji-16/obj48 2024-04-19T23:26:42.561 DEBUG:boto:auth_path=/aktlji-16/obj48 2024-04-19T23:26:42.561 DEBUG:boto:path=/aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.561 DEBUG:boto:auth_path=/aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.561 DEBUG:boto:Method: GET 2024-04-19T23:26:42.562 DEBUG:boto:Path: /aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.562 DEBUG:boto:Data: 2024-04-19T23:26:42.562 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.562 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.562 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.562 DEBUG:boto:Params: {} 2024-04-19T23:26:42.562 DEBUG:boto:Token: None 2024-04-19T23:26:42.562 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.562 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0= 2024-04-19T23:26:42.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0=', 'Content-Length': '0'} 2024-04-19T23:26:42.564 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024acd86a8cb5b733-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.565 DEBUG:boto:path=/aktlji-16/obj48 2024-04-19T23:26:42.565 DEBUG:boto:auth_path=/aktlji-16/obj48 2024-04-19T23:26:42.565 DEBUG:boto:path=/aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.565 DEBUG:boto:auth_path=/aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.565 DEBUG:boto:Method: GET 2024-04-19T23:26:42.565 DEBUG:boto:Path: /aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.565 DEBUG:boto:Data: 2024-04-19T23:26:42.565 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.565 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.565 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.565 DEBUG:boto:Params: {} 2024-04-19T23:26:42.566 DEBUG:boto:Token: None 2024-04-19T23:26:42.566 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj48?versionId=null 2024-04-19T23:26:42.566 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0= 2024-04-19T23:26:42.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0=', 'Content-Length': '0'} 2024-04-19T23:26:42.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1355a20268dc3eb-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.568 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.568 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.568 DEBUG:boto:path=/aktlji-16/?prefix=obj48 2024-04-19T23:26:42.568 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj48 2024-04-19T23:26:42.568 DEBUG:boto:Method: GET 2024-04-19T23:26:42.569 DEBUG:boto:Path: /aktlji-16/?prefix=obj48 2024-04-19T23:26:42.569 DEBUG:boto:Data: 2024-04-19T23:26:42.569 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.569 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.569 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.569 DEBUG:boto:Params: {} 2024-04-19T23:26:42.569 DEBUG:boto:Token: None 2024-04-19T23:26:42.569 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.569 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000342afb78931c1323-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.572 DEBUG:boto:b'aktlji-16obj481000falseobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.573 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.573 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.573 DEBUG:boto:path=/aktlji-16/?prefix=obj48 2024-04-19T23:26:42.573 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj48 2024-04-19T23:26:42.573 DEBUG:boto:Method: GET 2024-04-19T23:26:42.573 DEBUG:boto:Path: /aktlji-16/?prefix=obj48 2024-04-19T23:26:42.573 DEBUG:boto:Data: 2024-04-19T23:26:42.573 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.573 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.573 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.573 DEBUG:boto:Params: {} 2024-04-19T23:26:42.574 DEBUG:boto:Token: None 2024-04-19T23:26:42.574 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.574 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055bf177d12431fae-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.577 DEBUG:boto:b'aktlji-16obj481000falseobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.577 DEBUG:boto:path=/aktlji-16/obj48 2024-04-19T23:26:42.577 DEBUG:boto:auth_path=/aktlji-16/obj48 2024-04-19T23:26:42.577 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.577 DEBUG:boto:Path: /aktlji-16/obj48 2024-04-19T23:26:42.577 DEBUG:boto:Data: 2024-04-19T23:26:42.577 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.578 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.578 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.578 DEBUG:boto:Params: {} 2024-04-19T23:26:42.578 DEBUG:boto:Token: None 2024-04-19T23:26:42.578 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj48 2024-04-19T23:26:42.578 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ll75/inp3fKPjPJ97vHAirGCEPI= 2024-04-19T23:26:42.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ll75/inp3fKPjPJ97vHAirGCEPI=', 'Content-Length': '0'} 2024-04-19T23:26:42.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008645cd8b085de073-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.580 DEBUG:boto:path=/aktlji-16/obj48 2024-04-19T23:26:42.580 DEBUG:boto:auth_path=/aktlji-16/obj48 2024-04-19T23:26:42.580 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.580 DEBUG:boto:Path: /aktlji-16/obj48 2024-04-19T23:26:42.581 DEBUG:boto:Data: 2024-04-19T23:26:42.581 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.581 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.581 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.581 DEBUG:boto:Params: {} 2024-04-19T23:26:42.581 DEBUG:boto:Token: None 2024-04-19T23:26:42.581 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj48 2024-04-19T23:26:42.581 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ll75/inp3fKPjPJ97vHAirGCEPI= 2024-04-19T23:26:42.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ll75/inp3fKPjPJ97vHAirGCEPI=', 'Content-Length': '0'} 2024-04-19T23:26:42.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8c109210f7b40d3-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj48 2024-04-19T23:26:42.584 DEBUG:boto:path=/aktlji-16/obj48 2024-04-19T23:26:42.584 DEBUG:boto:auth_path=/aktlji-16/obj48 2024-04-19T23:26:42.584 DEBUG:boto:Method: GET 2024-04-19T23:26:42.584 DEBUG:boto:Path: /aktlji-16/obj48 2024-04-19T23:26:42.584 DEBUG:boto:Data: 2024-04-19T23:26:42.584 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.584 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.584 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.584 DEBUG:boto:Params: {} 2024-04-19T23:26:42.585 DEBUG:boto:Token: None 2024-04-19T23:26:42.585 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj48 2024-04-19T23:26:42.585 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ynuqLU9p/NIFuaNGKgeL2/UssvI= 2024-04-19T23:26:42.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ynuqLU9p/NIFuaNGKgeL2/UssvI=', 'Content-Length': '0'} 2024-04-19T23:26:42.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bf8929ca8e0c4aa-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.630 DEBUG:boto:path=/aktlji-16/obj48 2024-04-19T23:26:42.630 DEBUG:boto:auth_path=/aktlji-16/obj48 2024-04-19T23:26:42.631 DEBUG:boto:Method: GET 2024-04-19T23:26:42.631 DEBUG:boto:Path: /aktlji-16/obj48 2024-04-19T23:26:42.631 DEBUG:boto:Data: 2024-04-19T23:26:42.631 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.631 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.631 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.631 DEBUG:boto:Params: {} 2024-04-19T23:26:42.631 DEBUG:boto:Token: None 2024-04-19T23:26:42.631 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj48 2024-04-19T23:26:42.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ynuqLU9p/NIFuaNGKgeL2/UssvI= 2024-04-19T23:26:42.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ynuqLU9p/NIFuaNGKgeL2/UssvI=', 'Content-Length': '0'} 2024-04-19T23:26:42.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c237533273a4e6f-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.634 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.634 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.634 DEBUG:boto:path=/aktlji-16/?prefix=obj48 2024-04-19T23:26:42.634 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj48 2024-04-19T23:26:42.634 DEBUG:boto:Method: GET 2024-04-19T23:26:42.634 DEBUG:boto:Path: /aktlji-16/?prefix=obj48 2024-04-19T23:26:42.634 DEBUG:boto:Data: 2024-04-19T23:26:42.634 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.635 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.635 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.635 DEBUG:boto:Params: {} 2024-04-19T23:26:42.635 DEBUG:boto:Token: None 2024-04-19T23:26:42.635 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.635 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd8dac8eca44e48d-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.638 DEBUG:boto:b'aktlji-16obj481000falseobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.638 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.639 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.639 DEBUG:boto:path=/aktlji-16/?prefix=obj48 2024-04-19T23:26:42.639 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj48 2024-04-19T23:26:42.639 DEBUG:boto:Method: GET 2024-04-19T23:26:42.639 DEBUG:boto:Path: /aktlji-16/?prefix=obj48 2024-04-19T23:26:42.639 DEBUG:boto:Data: 2024-04-19T23:26:42.639 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.639 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.639 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.639 DEBUG:boto:Params: {} 2024-04-19T23:26:42.640 DEBUG:boto:Token: None 2024-04-19T23:26:42.640 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.640 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003923766e29db8559-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.642 DEBUG:boto:b'aktlji-16obj481000falseobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj49 2024-04-19T23:26:42.643 DEBUG:boto:path=/aktlji-16/obj49 2024-04-19T23:26:42.643 DEBUG:boto:auth_path=/aktlji-16/obj49 2024-04-19T23:26:42.643 DEBUG:boto:path=/aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.643 DEBUG:boto:auth_path=/aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.644 DEBUG:boto:Method: GET 2024-04-19T23:26:42.644 DEBUG:boto:Path: /aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.644 DEBUG:boto:Data: 2024-04-19T23:26:42.644 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.644 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.644 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.644 DEBUG:boto:Params: {} 2024-04-19T23:26:42.644 DEBUG:boto:Token: None 2024-04-19T23:26:42.644 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.644 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI= 2024-04-19T23:26:42.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI=', 'Content-Length': '0'} 2024-04-19T23:26:42.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8481b08eab0d9f0-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.647 DEBUG:boto:path=/aktlji-16/obj49 2024-04-19T23:26:42.647 DEBUG:boto:auth_path=/aktlji-16/obj49 2024-04-19T23:26:42.647 DEBUG:boto:path=/aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.647 DEBUG:boto:auth_path=/aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.647 DEBUG:boto:Method: GET 2024-04-19T23:26:42.647 DEBUG:boto:Path: /aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.647 DEBUG:boto:Data: 2024-04-19T23:26:42.647 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.647 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.647 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.647 DEBUG:boto:Params: {} 2024-04-19T23:26:42.648 DEBUG:boto:Token: None 2024-04-19T23:26:42.648 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.648 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI= 2024-04-19T23:26:42.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI=', 'Content-Length': '0'} 2024-04-19T23:26:42.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6d319101edbdb77-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.650 DEBUG:boto:path=/aktlji-16/obj49 2024-04-19T23:26:42.650 DEBUG:boto:auth_path=/aktlji-16/obj49 2024-04-19T23:26:42.650 DEBUG:boto:path=/aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.650 DEBUG:boto:auth_path=/aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.650 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.650 DEBUG:boto:Path: /aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.651 DEBUG:boto:Data: 2024-04-19T23:26:42.651 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.651 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.651 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.651 DEBUG:boto:Params: {} 2024-04-19T23:26:42.651 DEBUG:boto:Token: None 2024-04-19T23:26:42.651 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.651 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eAfIWoNF+elD5FILfnHDAvQ9EpY= 2024-04-19T23:26:42.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eAfIWoNF+elD5FILfnHDAvQ9EpY=', 'Content-Length': '0'} 2024-04-19T23:26:42.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e882f9f4bbc81ece-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.653 DEBUG:boto:path=/aktlji-16/obj49 2024-04-19T23:26:42.653 DEBUG:boto:auth_path=/aktlji-16/obj49 2024-04-19T23:26:42.653 DEBUG:boto:path=/aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.654 DEBUG:boto:auth_path=/aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.654 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.654 DEBUG:boto:Path: /aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.654 DEBUG:boto:Data: 2024-04-19T23:26:42.654 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.654 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.654 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.654 DEBUG:boto:Params: {} 2024-04-19T23:26:42.654 DEBUG:boto:Token: None 2024-04-19T23:26:42.654 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.655 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eAfIWoNF+elD5FILfnHDAvQ9EpY= 2024-04-19T23:26:42.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eAfIWoNF+elD5FILfnHDAvQ9EpY=', 'Content-Length': '0'} 2024-04-19T23:26:42.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000399d4226a04fd2ab-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj49 2024-04-19T23:26:42.657 DEBUG:boto:path=/aktlji-16/obj49 2024-04-19T23:26:42.657 DEBUG:boto:auth_path=/aktlji-16/obj49 2024-04-19T23:26:42.657 DEBUG:boto:path=/aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.657 DEBUG:boto:auth_path=/aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.657 DEBUG:boto:Method: GET 2024-04-19T23:26:42.657 DEBUG:boto:Path: /aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.657 DEBUG:boto:Data: 2024-04-19T23:26:42.657 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.657 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.657 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.657 DEBUG:boto:Params: {} 2024-04-19T23:26:42.657 DEBUG:boto:Token: None 2024-04-19T23:26:42.658 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.658 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI= 2024-04-19T23:26:42.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI=', 'Content-Length': '0'} 2024-04-19T23:26:42.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5fb0ebaf31eaa7f-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.660 DEBUG:boto:path=/aktlji-16/obj49 2024-04-19T23:26:42.660 DEBUG:boto:auth_path=/aktlji-16/obj49 2024-04-19T23:26:42.661 DEBUG:boto:path=/aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.661 DEBUG:boto:auth_path=/aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.661 DEBUG:boto:Method: GET 2024-04-19T23:26:42.661 DEBUG:boto:Path: /aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.661 DEBUG:boto:Data: 2024-04-19T23:26:42.661 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.661 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.661 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.661 DEBUG:boto:Params: {} 2024-04-19T23:26:42.661 DEBUG:boto:Token: None 2024-04-19T23:26:42.661 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj49?versionId=null 2024-04-19T23:26:42.662 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI= 2024-04-19T23:26:42.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI=', 'Content-Length': '0'} 2024-04-19T23:26:42.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d092bab75ff945f9-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.667 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.668 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.668 DEBUG:boto:path=/aktlji-16/?prefix=obj49 2024-04-19T23:26:42.668 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj49 2024-04-19T23:26:42.668 DEBUG:boto:Method: GET 2024-04-19T23:26:42.668 DEBUG:boto:Path: /aktlji-16/?prefix=obj49 2024-04-19T23:26:42.668 DEBUG:boto:Data: 2024-04-19T23:26:42.668 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.668 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.668 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.668 DEBUG:boto:Params: {} 2024-04-19T23:26:42.668 DEBUG:boto:Token: None 2024-04-19T23:26:42.669 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.669 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e23056429e20b0de-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.671 DEBUG:boto:b'aktlji-16obj491000falseobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.672 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.672 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.672 DEBUG:boto:path=/aktlji-16/?prefix=obj49 2024-04-19T23:26:42.672 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj49 2024-04-19T23:26:42.672 DEBUG:boto:Method: GET 2024-04-19T23:26:42.672 DEBUG:boto:Path: /aktlji-16/?prefix=obj49 2024-04-19T23:26:42.672 DEBUG:boto:Data: 2024-04-19T23:26:42.672 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.673 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.673 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.673 DEBUG:boto:Params: {} 2024-04-19T23:26:42.673 DEBUG:boto:Token: None 2024-04-19T23:26:42.673 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.673 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a972ee1a4b12dc83-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.719 DEBUG:boto:b'aktlji-16obj491000falseobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.719 DEBUG:boto:path=/aktlji-16/obj49 2024-04-19T23:26:42.720 DEBUG:boto:auth_path=/aktlji-16/obj49 2024-04-19T23:26:42.720 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.720 DEBUG:boto:Path: /aktlji-16/obj49 2024-04-19T23:26:42.720 DEBUG:boto:Data: 2024-04-19T23:26:42.720 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.720 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.720 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.720 DEBUG:boto:Params: {} 2024-04-19T23:26:42.720 DEBUG:boto:Token: None 2024-04-19T23:26:42.721 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj49 2024-04-19T23:26:42.721 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:W6+QcL41hgR9811tBstfuxigVYs= 2024-04-19T23:26:42.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W6+QcL41hgR9811tBstfuxigVYs=', 'Content-Length': '0'} 2024-04-19T23:26:42.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea43a2df53e8f3e3-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.723 DEBUG:boto:path=/aktlji-16/obj49 2024-04-19T23:26:42.723 DEBUG:boto:auth_path=/aktlji-16/obj49 2024-04-19T23:26:42.723 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.723 DEBUG:boto:Path: /aktlji-16/obj49 2024-04-19T23:26:42.723 DEBUG:boto:Data: 2024-04-19T23:26:42.723 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.723 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.723 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.724 DEBUG:boto:Params: {} 2024-04-19T23:26:42.724 DEBUG:boto:Token: None 2024-04-19T23:26:42.724 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj49 2024-04-19T23:26:42.724 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:W6+QcL41hgR9811tBstfuxigVYs= 2024-04-19T23:26:42.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W6+QcL41hgR9811tBstfuxigVYs=', 'Content-Length': '0'} 2024-04-19T23:26:42.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005518c2e844e6ae97-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj49 2024-04-19T23:26:42.726 DEBUG:boto:path=/aktlji-16/obj49 2024-04-19T23:26:42.726 DEBUG:boto:auth_path=/aktlji-16/obj49 2024-04-19T23:26:42.726 DEBUG:boto:Method: GET 2024-04-19T23:26:42.726 DEBUG:boto:Path: /aktlji-16/obj49 2024-04-19T23:26:42.726 DEBUG:boto:Data: 2024-04-19T23:26:42.726 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.726 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.727 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.727 DEBUG:boto:Params: {} 2024-04-19T23:26:42.727 DEBUG:boto:Token: None 2024-04-19T23:26:42.727 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj49 2024-04-19T23:26:42.727 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kYOhBOx+bUcfAKVXRExL41Y39pY= 2024-04-19T23:26:42.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kYOhBOx+bUcfAKVXRExL41Y39pY=', 'Content-Length': '0'} 2024-04-19T23:26:42.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2abdd0517ca36e7-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.774 DEBUG:boto:path=/aktlji-16/obj49 2024-04-19T23:26:42.775 DEBUG:boto:auth_path=/aktlji-16/obj49 2024-04-19T23:26:42.775 DEBUG:boto:Method: GET 2024-04-19T23:26:42.775 DEBUG:boto:Path: /aktlji-16/obj49 2024-04-19T23:26:42.775 DEBUG:boto:Data: 2024-04-19T23:26:42.775 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.775 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.775 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.775 DEBUG:boto:Params: {} 2024-04-19T23:26:42.775 DEBUG:boto:Token: None 2024-04-19T23:26:42.775 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj49 2024-04-19T23:26:42.776 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kYOhBOx+bUcfAKVXRExL41Y39pY= 2024-04-19T23:26:42.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kYOhBOx+bUcfAKVXRExL41Y39pY=', 'Content-Length': '0'} 2024-04-19T23:26:42.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030bad422308f6d1b-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.778 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.778 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.778 DEBUG:boto:path=/aktlji-16/?prefix=obj49 2024-04-19T23:26:42.778 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj49 2024-04-19T23:26:42.778 DEBUG:boto:Method: GET 2024-04-19T23:26:42.778 DEBUG:boto:Path: /aktlji-16/?prefix=obj49 2024-04-19T23:26:42.778 DEBUG:boto:Data: 2024-04-19T23:26:42.778 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.778 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.779 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.779 DEBUG:boto:Params: {} 2024-04-19T23:26:42.779 DEBUG:boto:Token: None 2024-04-19T23:26:42.779 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.779 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d820b3bd35715b75-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.782 DEBUG:boto:b'aktlji-16obj491000falseobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.782 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.783 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.783 DEBUG:boto:path=/aktlji-16/?prefix=obj49 2024-04-19T23:26:42.783 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj49 2024-04-19T23:26:42.783 DEBUG:boto:Method: GET 2024-04-19T23:26:42.783 DEBUG:boto:Path: /aktlji-16/?prefix=obj49 2024-04-19T23:26:42.783 DEBUG:boto:Data: 2024-04-19T23:26:42.783 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.783 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.783 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.783 DEBUG:boto:Params: {} 2024-04-19T23:26:42.783 DEBUG:boto:Token: None 2024-04-19T23:26:42.784 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd44fa9c82315611-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.786 DEBUG:boto:b'aktlji-16obj491000falseobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-19T23:26:42.787 DEBUG:boto:path=/aktlji-16/obj5 2024-04-19T23:26:42.787 DEBUG:boto:auth_path=/aktlji-16/obj5 2024-04-19T23:26:42.787 DEBUG:boto:path=/aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.787 DEBUG:boto:auth_path=/aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.787 DEBUG:boto:Method: GET 2024-04-19T23:26:42.787 DEBUG:boto:Path: /aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.787 DEBUG:boto:Data: 2024-04-19T23:26:42.787 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.787 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.788 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.788 DEBUG:boto:Params: {} 2024-04-19T23:26:42.788 DEBUG:boto:Token: None 2024-04-19T23:26:42.788 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.788 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U= 2024-04-19T23:26:42.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U=', 'Content-Length': '0'} 2024-04-19T23:26:42.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c04162890856f2e-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.790 DEBUG:boto:path=/aktlji-16/obj5 2024-04-19T23:26:42.790 DEBUG:boto:auth_path=/aktlji-16/obj5 2024-04-19T23:26:42.790 DEBUG:boto:path=/aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.791 DEBUG:boto:auth_path=/aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.791 DEBUG:boto:Method: GET 2024-04-19T23:26:42.791 DEBUG:boto:Path: /aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.791 DEBUG:boto:Data: 2024-04-19T23:26:42.791 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.791 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.791 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.791 DEBUG:boto:Params: {} 2024-04-19T23:26:42.791 DEBUG:boto:Token: None 2024-04-19T23:26:42.791 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.791 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U= 2024-04-19T23:26:42.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U=', 'Content-Length': '0'} 2024-04-19T23:26:42.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079bff67ceca0fc42-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.794 DEBUG:boto:path=/aktlji-16/obj5 2024-04-19T23:26:42.794 DEBUG:boto:auth_path=/aktlji-16/obj5 2024-04-19T23:26:42.794 DEBUG:boto:path=/aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.794 DEBUG:boto:auth_path=/aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.794 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.794 DEBUG:boto:Path: /aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.794 DEBUG:boto:Data: 2024-04-19T23:26:42.794 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.794 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.795 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.795 DEBUG:boto:Params: {} 2024-04-19T23:26:42.795 DEBUG:boto:Token: None 2024-04-19T23:26:42.795 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.795 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:264yvT9MNgk3+Dpz1XP7Wo1CKkM= 2024-04-19T23:26:42.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:264yvT9MNgk3+Dpz1XP7Wo1CKkM=', 'Content-Length': '0'} 2024-04-19T23:26:42.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000554de72105cc10a2-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.797 DEBUG:boto:path=/aktlji-16/obj5 2024-04-19T23:26:42.797 DEBUG:boto:auth_path=/aktlji-16/obj5 2024-04-19T23:26:42.798 DEBUG:boto:path=/aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.798 DEBUG:boto:auth_path=/aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.798 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.798 DEBUG:boto:Path: /aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.798 DEBUG:boto:Data: 2024-04-19T23:26:42.798 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.798 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.798 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.798 DEBUG:boto:Params: {} 2024-04-19T23:26:42.798 DEBUG:boto:Token: None 2024-04-19T23:26:42.799 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.799 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:264yvT9MNgk3+Dpz1XP7Wo1CKkM= 2024-04-19T23:26:42.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:264yvT9MNgk3+Dpz1XP7Wo1CKkM=', 'Content-Length': '0'} 2024-04-19T23:26:42.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5c7205bd120e42e-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-19T23:26:42.801 DEBUG:boto:path=/aktlji-16/obj5 2024-04-19T23:26:42.801 DEBUG:boto:auth_path=/aktlji-16/obj5 2024-04-19T23:26:42.801 DEBUG:boto:path=/aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.801 DEBUG:boto:auth_path=/aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.801 DEBUG:boto:Method: GET 2024-04-19T23:26:42.801 DEBUG:boto:Path: /aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.801 DEBUG:boto:Data: 2024-04-19T23:26:42.801 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.801 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.802 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.802 DEBUG:boto:Params: {} 2024-04-19T23:26:42.802 DEBUG:boto:Token: None 2024-04-19T23:26:42.802 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.802 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U= 2024-04-19T23:26:42.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U=', 'Content-Length': '0'} 2024-04-19T23:26:42.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ed9f45e130e771e-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.804 DEBUG:boto:path=/aktlji-16/obj5 2024-04-19T23:26:42.805 DEBUG:boto:auth_path=/aktlji-16/obj5 2024-04-19T23:26:42.805 DEBUG:boto:path=/aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.805 DEBUG:boto:auth_path=/aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.805 DEBUG:boto:Method: GET 2024-04-19T23:26:42.805 DEBUG:boto:Path: /aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.805 DEBUG:boto:Data: 2024-04-19T23:26:42.805 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.805 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.805 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.805 DEBUG:boto:Params: {} 2024-04-19T23:26:42.805 DEBUG:boto:Token: None 2024-04-19T23:26:42.806 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj5?versionId=null 2024-04-19T23:26:42.806 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U= 2024-04-19T23:26:42.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U=', 'Content-Length': '0'} 2024-04-19T23:26:42.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d5c8a223771b05e-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.808 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.808 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.808 DEBUG:boto:path=/aktlji-16/?prefix=obj5 2024-04-19T23:26:42.808 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj5 2024-04-19T23:26:42.808 DEBUG:boto:Method: GET 2024-04-19T23:26:42.808 DEBUG:boto:Path: /aktlji-16/?prefix=obj5 2024-04-19T23:26:42.808 DEBUG:boto:Data: 2024-04-19T23:26:42.809 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.809 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.809 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.809 DEBUG:boto:Params: {} 2024-04-19T23:26:42.809 DEBUG:boto:Token: None 2024-04-19T23:26:42.809 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.809 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a91cce509ff2891-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.812 DEBUG:boto:b'aktlji-16obj51000falseobj52024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.812 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.812 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.813 DEBUG:boto:path=/aktlji-16/?prefix=obj5 2024-04-19T23:26:42.813 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj5 2024-04-19T23:26:42.813 DEBUG:boto:Method: GET 2024-04-19T23:26:42.813 DEBUG:boto:Path: /aktlji-16/?prefix=obj5 2024-04-19T23:26:42.813 DEBUG:boto:Data: 2024-04-19T23:26:42.813 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.813 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.813 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.813 DEBUG:boto:Params: {} 2024-04-19T23:26:42.813 DEBUG:boto:Token: None 2024-04-19T23:26:42.813 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.814 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc3185633d95477d-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.816 DEBUG:boto:b'aktlji-16obj51000falseobj52024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.816 DEBUG:boto:path=/aktlji-16/obj5 2024-04-19T23:26:42.817 DEBUG:boto:auth_path=/aktlji-16/obj5 2024-04-19T23:26:42.817 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.817 DEBUG:boto:Path: /aktlji-16/obj5 2024-04-19T23:26:42.817 DEBUG:boto:Data: 2024-04-19T23:26:42.817 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.817 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.817 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.817 DEBUG:boto:Params: {} 2024-04-19T23:26:42.817 DEBUG:boto:Token: None 2024-04-19T23:26:42.818 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj5 2024-04-19T23:26:42.818 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VnwqfiYQs0bdtOKknVcHZsm3slk= 2024-04-19T23:26:42.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VnwqfiYQs0bdtOKknVcHZsm3slk=', 'Content-Length': '0'} 2024-04-19T23:26:42.819 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e46e15b25cf017e5-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.820 DEBUG:boto:path=/aktlji-16/obj5 2024-04-19T23:26:42.820 DEBUG:boto:auth_path=/aktlji-16/obj5 2024-04-19T23:26:42.820 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.820 DEBUG:boto:Path: /aktlji-16/obj5 2024-04-19T23:26:42.820 DEBUG:boto:Data: 2024-04-19T23:26:42.820 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.820 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.820 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.820 DEBUG:boto:Params: {} 2024-04-19T23:26:42.820 DEBUG:boto:Token: None 2024-04-19T23:26:42.821 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj5 2024-04-19T23:26:42.821 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VnwqfiYQs0bdtOKknVcHZsm3slk= 2024-04-19T23:26:42.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VnwqfiYQs0bdtOKknVcHZsm3slk=', 'Content-Length': '0'} 2024-04-19T23:26:42.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b94e6a88b131b1de-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-19T23:26:42.823 DEBUG:boto:path=/aktlji-16/obj5 2024-04-19T23:26:42.823 DEBUG:boto:auth_path=/aktlji-16/obj5 2024-04-19T23:26:42.823 DEBUG:boto:Method: GET 2024-04-19T23:26:42.823 DEBUG:boto:Path: /aktlji-16/obj5 2024-04-19T23:26:42.823 DEBUG:boto:Data: 2024-04-19T23:26:42.823 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.823 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.823 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.824 DEBUG:boto:Params: {} 2024-04-19T23:26:42.824 DEBUG:boto:Token: None 2024-04-19T23:26:42.824 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj5 2024-04-19T23:26:42.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2TjVUMKpM9v7YV9X8qLRfI4HUVE= 2024-04-19T23:26:42.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2TjVUMKpM9v7YV9X8qLRfI4HUVE=', 'Content-Length': '0'} 2024-04-19T23:26:42.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000091afed372342ea0-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.826 DEBUG:boto:path=/aktlji-16/obj5 2024-04-19T23:26:42.826 DEBUG:boto:auth_path=/aktlji-16/obj5 2024-04-19T23:26:42.826 DEBUG:boto:Method: GET 2024-04-19T23:26:42.826 DEBUG:boto:Path: /aktlji-16/obj5 2024-04-19T23:26:42.826 DEBUG:boto:Data: 2024-04-19T23:26:42.826 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.826 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.826 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.827 DEBUG:boto:Params: {} 2024-04-19T23:26:42.827 DEBUG:boto:Token: None 2024-04-19T23:26:42.827 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj5 2024-04-19T23:26:42.827 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2TjVUMKpM9v7YV9X8qLRfI4HUVE= 2024-04-19T23:26:42.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2TjVUMKpM9v7YV9X8qLRfI4HUVE=', 'Content-Length': '0'} 2024-04-19T23:26:42.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006dcf75f2e7552a1f-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.870 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.871 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.871 DEBUG:boto:path=/aktlji-16/?prefix=obj5 2024-04-19T23:26:42.871 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj5 2024-04-19T23:26:42.871 DEBUG:boto:Method: GET 2024-04-19T23:26:42.871 DEBUG:boto:Path: /aktlji-16/?prefix=obj5 2024-04-19T23:26:42.871 DEBUG:boto:Data: 2024-04-19T23:26:42.871 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.871 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.871 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.871 DEBUG:boto:Params: {} 2024-04-19T23:26:42.871 DEBUG:boto:Token: None 2024-04-19T23:26:42.872 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.872 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4cefb886e8ef9bb-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.874 DEBUG:boto:b'aktlji-16obj51000falseobj52024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.875 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.875 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.875 DEBUG:boto:path=/aktlji-16/?prefix=obj5 2024-04-19T23:26:42.875 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj5 2024-04-19T23:26:42.875 DEBUG:boto:Method: GET 2024-04-19T23:26:42.875 DEBUG:boto:Path: /aktlji-16/?prefix=obj5 2024-04-19T23:26:42.875 DEBUG:boto:Data: 2024-04-19T23:26:42.875 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.876 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.876 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.876 DEBUG:boto:Params: {} 2024-04-19T23:26:42.876 DEBUG:boto:Token: None 2024-04-19T23:26:42.876 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.876 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ec49e74a1ed2509-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.879 DEBUG:boto:b'aktlji-16obj51000falseobj52024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-19T23:26:42.879 DEBUG:boto:path=/aktlji-16/obj6 2024-04-19T23:26:42.879 DEBUG:boto:auth_path=/aktlji-16/obj6 2024-04-19T23:26:42.879 DEBUG:boto:path=/aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.879 DEBUG:boto:auth_path=/aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.880 DEBUG:boto:Method: GET 2024-04-19T23:26:42.880 DEBUG:boto:Path: /aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.880 DEBUG:boto:Data: 2024-04-19T23:26:42.880 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.880 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.880 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.880 DEBUG:boto:Params: {} 2024-04-19T23:26:42.880 DEBUG:boto:Token: None 2024-04-19T23:26:42.880 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0= 2024-04-19T23:26:42.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0=', 'Content-Length': '0'} 2024-04-19T23:26:42.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005119ea733477f806-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.926 DEBUG:boto:path=/aktlji-16/obj6 2024-04-19T23:26:42.927 DEBUG:boto:auth_path=/aktlji-16/obj6 2024-04-19T23:26:42.927 DEBUG:boto:path=/aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.927 DEBUG:boto:auth_path=/aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.927 DEBUG:boto:Method: GET 2024-04-19T23:26:42.927 DEBUG:boto:Path: /aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.927 DEBUG:boto:Data: 2024-04-19T23:26:42.927 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.927 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.927 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.927 DEBUG:boto:Params: {} 2024-04-19T23:26:42.927 DEBUG:boto:Token: None 2024-04-19T23:26:42.928 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.928 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0= 2024-04-19T23:26:42.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0=', 'Content-Length': '0'} 2024-04-19T23:26:42.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022680fc9b2295355-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.930 DEBUG:boto:path=/aktlji-16/obj6 2024-04-19T23:26:42.930 DEBUG:boto:auth_path=/aktlji-16/obj6 2024-04-19T23:26:42.930 DEBUG:boto:path=/aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.930 DEBUG:boto:auth_path=/aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.930 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.930 DEBUG:boto:Path: /aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.931 DEBUG:boto:Data: 2024-04-19T23:26:42.931 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.931 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.931 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.931 DEBUG:boto:Params: {} 2024-04-19T23:26:42.931 DEBUG:boto:Token: None 2024-04-19T23:26:42.931 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.931 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4VFW70TiVsYBjtSQh5whwu73Lcs= 2024-04-19T23:26:42.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4VFW70TiVsYBjtSQh5whwu73Lcs=', 'Content-Length': '0'} 2024-04-19T23:26:42.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f675d6cd8aff0ed7-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.933 DEBUG:boto:path=/aktlji-16/obj6 2024-04-19T23:26:42.933 DEBUG:boto:auth_path=/aktlji-16/obj6 2024-04-19T23:26:42.934 DEBUG:boto:path=/aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.934 DEBUG:boto:auth_path=/aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.934 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.934 DEBUG:boto:Path: /aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.934 DEBUG:boto:Data: 2024-04-19T23:26:42.934 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.934 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.934 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.934 DEBUG:boto:Params: {} 2024-04-19T23:26:42.934 DEBUG:boto:Token: None 2024-04-19T23:26:42.935 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.935 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4VFW70TiVsYBjtSQh5whwu73Lcs= 2024-04-19T23:26:42.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4VFW70TiVsYBjtSQh5whwu73Lcs=', 'Content-Length': '0'} 2024-04-19T23:26:42.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a869f93ae9450a5-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-19T23:26:42.937 DEBUG:boto:path=/aktlji-16/obj6 2024-04-19T23:26:42.937 DEBUG:boto:auth_path=/aktlji-16/obj6 2024-04-19T23:26:42.937 DEBUG:boto:path=/aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.937 DEBUG:boto:auth_path=/aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.937 DEBUG:boto:Method: GET 2024-04-19T23:26:42.937 DEBUG:boto:Path: /aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.937 DEBUG:boto:Data: 2024-04-19T23:26:42.937 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.937 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.938 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.938 DEBUG:boto:Params: {} 2024-04-19T23:26:42.938 DEBUG:boto:Token: None 2024-04-19T23:26:42.938 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.938 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0= 2024-04-19T23:26:42.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0=', 'Content-Length': '0'} 2024-04-19T23:26:42.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee945ad2e16000d9-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.940 DEBUG:boto:path=/aktlji-16/obj6 2024-04-19T23:26:42.940 DEBUG:boto:auth_path=/aktlji-16/obj6 2024-04-19T23:26:42.940 DEBUG:boto:path=/aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.941 DEBUG:boto:auth_path=/aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.941 DEBUG:boto:Method: GET 2024-04-19T23:26:42.941 DEBUG:boto:Path: /aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.941 DEBUG:boto:Data: 2024-04-19T23:26:42.941 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.941 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.941 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.941 DEBUG:boto:Params: {} 2024-04-19T23:26:42.941 DEBUG:boto:Token: None 2024-04-19T23:26:42.942 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj6?versionId=null 2024-04-19T23:26:42.942 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0= 2024-04-19T23:26:42.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0=', 'Content-Length': '0'} 2024-04-19T23:26:42.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001032021d458a074a-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.944 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.944 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.944 DEBUG:boto:path=/aktlji-16/?prefix=obj6 2024-04-19T23:26:42.944 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj6 2024-04-19T23:26:42.944 DEBUG:boto:Method: GET 2024-04-19T23:26:42.944 DEBUG:boto:Path: /aktlji-16/?prefix=obj6 2024-04-19T23:26:42.944 DEBUG:boto:Data: 2024-04-19T23:26:42.944 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.944 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.945 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.945 DEBUG:boto:Params: {} 2024-04-19T23:26:42.945 DEBUG:boto:Token: None 2024-04-19T23:26:42.945 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.945 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c27f280b4a428c2b-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.947 DEBUG:boto:b'aktlji-16obj61000falseobj62024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.948 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:42.948 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:42.948 DEBUG:boto:path=/aktlji-16/?prefix=obj6 2024-04-19T23:26:42.948 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj6 2024-04-19T23:26:42.948 DEBUG:boto:Method: GET 2024-04-19T23:26:42.948 DEBUG:boto:Path: /aktlji-16/?prefix=obj6 2024-04-19T23:26:42.948 DEBUG:boto:Data: 2024-04-19T23:26:42.949 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.949 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.949 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.949 DEBUG:boto:Params: {} 2024-04-19T23:26:42.949 DEBUG:boto:Token: None 2024-04-19T23:26:42.949 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/ 2024-04-19T23:26:42.949 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-19T23:26:42.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-19T23:26:42.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8bf69adda6c6795-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.952 DEBUG:boto:b'aktlji-16obj61000falseobj62024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:42.952 DEBUG:boto:path=/aktlji-16/obj6 2024-04-19T23:26:42.952 DEBUG:boto:auth_path=/aktlji-16/obj6 2024-04-19T23:26:42.952 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.952 DEBUG:boto:Path: /aktlji-16/obj6 2024-04-19T23:26:42.952 DEBUG:boto:Data: 2024-04-19T23:26:42.953 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.953 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.953 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.953 DEBUG:boto:Params: {} 2024-04-19T23:26:42.953 DEBUG:boto:Token: None 2024-04-19T23:26:42.953 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj6 2024-04-19T23:26:42.953 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rkGclTUbeGG4nnCEB3O2j+WG7BI= 2024-04-19T23:26:42.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rkGclTUbeGG4nnCEB3O2j+WG7BI=', 'Content-Length': '0'} 2024-04-19T23:26:42.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e1b5b3e539b14c0-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.956 DEBUG:boto:path=/aktlji-16/obj6 2024-04-19T23:26:42.956 DEBUG:boto:auth_path=/aktlji-16/obj6 2024-04-19T23:26:42.956 DEBUG:boto:Method: HEAD 2024-04-19T23:26:42.956 DEBUG:boto:Path: /aktlji-16/obj6 2024-04-19T23:26:42.956 DEBUG:boto:Data: 2024-04-19T23:26:42.956 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.956 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:42.956 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.956 DEBUG:boto:Params: {} 2024-04-19T23:26:42.956 DEBUG:boto:Token: None 2024-04-19T23:26:42.957 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj6 2024-04-19T23:26:42.957 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rkGclTUbeGG4nnCEB3O2j+WG7BI= 2024-04-19T23:26:42.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rkGclTUbeGG4nnCEB3O2j+WG7BI=', 'Content-Length': '0'} 2024-04-19T23:26:42.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006abea39868d40951-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:42.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-19T23:26:42.959 DEBUG:boto:path=/aktlji-16/obj6 2024-04-19T23:26:42.959 DEBUG:boto:auth_path=/aktlji-16/obj6 2024-04-19T23:26:42.959 DEBUG:boto:Method: GET 2024-04-19T23:26:42.959 DEBUG:boto:Path: /aktlji-16/obj6 2024-04-19T23:26:42.959 DEBUG:boto:Data: 2024-04-19T23:26:42.959 DEBUG:boto:Headers: {} 2024-04-19T23:26:42.959 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:42.959 DEBUG:boto:Port: 8000 2024-04-19T23:26:42.960 DEBUG:boto:Params: {} 2024-04-19T23:26:42.960 DEBUG:boto:Token: None 2024-04-19T23:26:42.960 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:42 GMT /aktlji-16/obj6 2024-04-19T23:26:42.960 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CNeqFsfuArjmeq/mFiyz3NzSZM0= 2024-04-19T23:26:42.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CNeqFsfuArjmeq/mFiyz3NzSZM0=', 'Content-Length': '0'} 2024-04-19T23:26:42.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e467df76ad63f82c-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.006 DEBUG:boto:path=/aktlji-16/obj6 2024-04-19T23:26:43.006 DEBUG:boto:auth_path=/aktlji-16/obj6 2024-04-19T23:26:43.007 DEBUG:boto:Method: GET 2024-04-19T23:26:43.007 DEBUG:boto:Path: /aktlji-16/obj6 2024-04-19T23:26:43.007 DEBUG:boto:Data: 2024-04-19T23:26:43.007 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.007 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.007 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.007 DEBUG:boto:Params: {} 2024-04-19T23:26:43.007 DEBUG:boto:Token: None 2024-04-19T23:26:43.007 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj6 2024-04-19T23:26:43.007 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aVNBVYJQVgQIIKRDLy8E6hapi1E= 2024-04-19T23:26:43.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aVNBVYJQVgQIIKRDLy8E6hapi1E=', 'Content-Length': '0'} 2024-04-19T23:26:43.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064eb63da20251ac6-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.010 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.010 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.010 DEBUG:boto:path=/aktlji-16/?prefix=obj6 2024-04-19T23:26:43.010 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj6 2024-04-19T23:26:43.010 DEBUG:boto:Method: GET 2024-04-19T23:26:43.010 DEBUG:boto:Path: /aktlji-16/?prefix=obj6 2024-04-19T23:26:43.010 DEBUG:boto:Data: 2024-04-19T23:26:43.010 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.011 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.011 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.011 DEBUG:boto:Params: {} 2024-04-19T23:26:43.011 DEBUG:boto:Token: None 2024-04-19T23:26:43.011 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.011 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee9114de33a16903-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.014 DEBUG:boto:b'aktlji-16obj61000falseobj62024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.014 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.015 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.015 DEBUG:boto:path=/aktlji-16/?prefix=obj6 2024-04-19T23:26:43.015 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj6 2024-04-19T23:26:43.015 DEBUG:boto:Method: GET 2024-04-19T23:26:43.015 DEBUG:boto:Path: /aktlji-16/?prefix=obj6 2024-04-19T23:26:43.015 DEBUG:boto:Data: 2024-04-19T23:26:43.015 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.015 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.015 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.015 DEBUG:boto:Params: {} 2024-04-19T23:26:43.015 DEBUG:boto:Token: None 2024-04-19T23:26:43.016 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.016 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ab5524241828e3d-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.019 DEBUG:boto:b'aktlji-16obj61000falseobj62024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-19T23:26:43.019 DEBUG:boto:path=/aktlji-16/obj7 2024-04-19T23:26:43.019 DEBUG:boto:auth_path=/aktlji-16/obj7 2024-04-19T23:26:43.019 DEBUG:boto:path=/aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.019 DEBUG:boto:auth_path=/aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.019 DEBUG:boto:Method: GET 2024-04-19T23:26:43.020 DEBUG:boto:Path: /aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.020 DEBUG:boto:Data: 2024-04-19T23:26:43.020 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.020 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.020 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.020 DEBUG:boto:Params: {} 2024-04-19T23:26:43.020 DEBUG:boto:Token: None 2024-04-19T23:26:43.020 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.020 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40= 2024-04-19T23:26:43.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40=', 'Content-Length': '0'} 2024-04-19T23:26:43.022 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006866a629b4b4f7d3-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.023 DEBUG:boto:path=/aktlji-16/obj7 2024-04-19T23:26:43.023 DEBUG:boto:auth_path=/aktlji-16/obj7 2024-04-19T23:26:43.023 DEBUG:boto:path=/aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.023 DEBUG:boto:auth_path=/aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.023 DEBUG:boto:Method: GET 2024-04-19T23:26:43.023 DEBUG:boto:Path: /aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.023 DEBUG:boto:Data: 2024-04-19T23:26:43.023 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.023 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.023 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.024 DEBUG:boto:Params: {} 2024-04-19T23:26:43.024 DEBUG:boto:Token: None 2024-04-19T23:26:43.024 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.024 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40= 2024-04-19T23:26:43.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40=', 'Content-Length': '0'} 2024-04-19T23:26:43.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e56132450ad5945d-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.070 DEBUG:boto:path=/aktlji-16/obj7 2024-04-19T23:26:43.070 DEBUG:boto:auth_path=/aktlji-16/obj7 2024-04-19T23:26:43.071 DEBUG:boto:path=/aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.071 DEBUG:boto:auth_path=/aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.071 DEBUG:boto:Method: HEAD 2024-04-19T23:26:43.071 DEBUG:boto:Path: /aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.071 DEBUG:boto:Data: 2024-04-19T23:26:43.071 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.071 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.071 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.071 DEBUG:boto:Params: {} 2024-04-19T23:26:43.071 DEBUG:boto:Token: None 2024-04-19T23:26:43.072 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.072 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:p/2EIeiW/CHijxd3EhN+u8YOLC4= 2024-04-19T23:26:43.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:p/2EIeiW/CHijxd3EhN+u8YOLC4=', 'Content-Length': '0'} 2024-04-19T23:26:43.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9c1427f16e80807-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.074 DEBUG:boto:path=/aktlji-16/obj7 2024-04-19T23:26:43.074 DEBUG:boto:auth_path=/aktlji-16/obj7 2024-04-19T23:26:43.074 DEBUG:boto:path=/aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.074 DEBUG:boto:auth_path=/aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.074 DEBUG:boto:Method: HEAD 2024-04-19T23:26:43.074 DEBUG:boto:Path: /aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.075 DEBUG:boto:Data: 2024-04-19T23:26:43.075 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.075 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.075 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.075 DEBUG:boto:Params: {} 2024-04-19T23:26:43.075 DEBUG:boto:Token: None 2024-04-19T23:26:43.075 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.075 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:p/2EIeiW/CHijxd3EhN+u8YOLC4= 2024-04-19T23:26:43.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:p/2EIeiW/CHijxd3EhN+u8YOLC4=', 'Content-Length': '0'} 2024-04-19T23:26:43.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe749f83bb5a7f4c-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-19T23:26:43.077 DEBUG:boto:path=/aktlji-16/obj7 2024-04-19T23:26:43.077 DEBUG:boto:auth_path=/aktlji-16/obj7 2024-04-19T23:26:43.078 DEBUG:boto:path=/aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.078 DEBUG:boto:auth_path=/aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.078 DEBUG:boto:Method: GET 2024-04-19T23:26:43.078 DEBUG:boto:Path: /aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.078 DEBUG:boto:Data: 2024-04-19T23:26:43.078 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.078 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.078 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.078 DEBUG:boto:Params: {} 2024-04-19T23:26:43.079 DEBUG:boto:Token: None 2024-04-19T23:26:43.079 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.079 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40= 2024-04-19T23:26:43.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40=', 'Content-Length': '0'} 2024-04-19T23:26:43.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c38b13bd3150c4b2-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.081 DEBUG:boto:path=/aktlji-16/obj7 2024-04-19T23:26:43.081 DEBUG:boto:auth_path=/aktlji-16/obj7 2024-04-19T23:26:43.081 DEBUG:boto:path=/aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.081 DEBUG:boto:auth_path=/aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.081 DEBUG:boto:Method: GET 2024-04-19T23:26:43.082 DEBUG:boto:Path: /aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.082 DEBUG:boto:Data: 2024-04-19T23:26:43.082 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.082 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.082 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.082 DEBUG:boto:Params: {} 2024-04-19T23:26:43.082 DEBUG:boto:Token: None 2024-04-19T23:26:43.082 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj7?versionId=null 2024-04-19T23:26:43.082 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40= 2024-04-19T23:26:43.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40=', 'Content-Length': '0'} 2024-04-19T23:26:43.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038861d19f7a974ca-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.085 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.085 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.085 DEBUG:boto:path=/aktlji-16/?prefix=obj7 2024-04-19T23:26:43.085 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj7 2024-04-19T23:26:43.085 DEBUG:boto:Method: GET 2024-04-19T23:26:43.085 DEBUG:boto:Path: /aktlji-16/?prefix=obj7 2024-04-19T23:26:43.085 DEBUG:boto:Data: 2024-04-19T23:26:43.085 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.085 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.086 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.086 DEBUG:boto:Params: {} 2024-04-19T23:26:43.086 DEBUG:boto:Token: None 2024-04-19T23:26:43.086 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.086 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fdb1ff116b19bb3-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.089 DEBUG:boto:b'aktlji-16obj71000falseobj72024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.089 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.089 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.089 DEBUG:boto:path=/aktlji-16/?prefix=obj7 2024-04-19T23:26:43.089 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj7 2024-04-19T23:26:43.089 DEBUG:boto:Method: GET 2024-04-19T23:26:43.089 DEBUG:boto:Path: /aktlji-16/?prefix=obj7 2024-04-19T23:26:43.090 DEBUG:boto:Data: 2024-04-19T23:26:43.090 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.090 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.090 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.090 DEBUG:boto:Params: {} 2024-04-19T23:26:43.090 DEBUG:boto:Token: None 2024-04-19T23:26:43.090 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.090 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ea3d04fcef4e0db-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.093 DEBUG:boto:b'aktlji-16obj71000falseobj72024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.094 DEBUG:boto:path=/aktlji-16/obj7 2024-04-19T23:26:43.094 DEBUG:boto:auth_path=/aktlji-16/obj7 2024-04-19T23:26:43.094 DEBUG:boto:Method: HEAD 2024-04-19T23:26:43.094 DEBUG:boto:Path: /aktlji-16/obj7 2024-04-19T23:26:43.094 DEBUG:boto:Data: 2024-04-19T23:26:43.094 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.094 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.094 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.094 DEBUG:boto:Params: {} 2024-04-19T23:26:43.094 DEBUG:boto:Token: None 2024-04-19T23:26:43.095 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj7 2024-04-19T23:26:43.095 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ErVvsYFBdx8o7Ofj44LmRetub9U= 2024-04-19T23:26:43.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ErVvsYFBdx8o7Ofj44LmRetub9U=', 'Content-Length': '0'} 2024-04-19T23:26:43.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a139a6381abb66c0-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.097 DEBUG:boto:path=/aktlji-16/obj7 2024-04-19T23:26:43.097 DEBUG:boto:auth_path=/aktlji-16/obj7 2024-04-19T23:26:43.097 DEBUG:boto:Method: HEAD 2024-04-19T23:26:43.097 DEBUG:boto:Path: /aktlji-16/obj7 2024-04-19T23:26:43.097 DEBUG:boto:Data: 2024-04-19T23:26:43.097 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.097 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.097 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.097 DEBUG:boto:Params: {} 2024-04-19T23:26:43.098 DEBUG:boto:Token: None 2024-04-19T23:26:43.098 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj7 2024-04-19T23:26:43.098 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ErVvsYFBdx8o7Ofj44LmRetub9U= 2024-04-19T23:26:43.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ErVvsYFBdx8o7Ofj44LmRetub9U=', 'Content-Length': '0'} 2024-04-19T23:26:43.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f605169fb81038bd-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-19T23:26:43.100 DEBUG:boto:path=/aktlji-16/obj7 2024-04-19T23:26:43.100 DEBUG:boto:auth_path=/aktlji-16/obj7 2024-04-19T23:26:43.100 DEBUG:boto:Method: GET 2024-04-19T23:26:43.100 DEBUG:boto:Path: /aktlji-16/obj7 2024-04-19T23:26:43.100 DEBUG:boto:Data: 2024-04-19T23:26:43.100 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.100 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.100 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.101 DEBUG:boto:Params: {} 2024-04-19T23:26:43.101 DEBUG:boto:Token: None 2024-04-19T23:26:43.101 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj7 2024-04-19T23:26:43.101 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l4qKxijHU/gu/8uauJx02dUGLks= 2024-04-19T23:26:43.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l4qKxijHU/gu/8uauJx02dUGLks=', 'Content-Length': '0'} 2024-04-19T23:26:43.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024e83fae83b95d18-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.103 DEBUG:boto:path=/aktlji-16/obj7 2024-04-19T23:26:43.103 DEBUG:boto:auth_path=/aktlji-16/obj7 2024-04-19T23:26:43.103 DEBUG:boto:Method: GET 2024-04-19T23:26:43.103 DEBUG:boto:Path: /aktlji-16/obj7 2024-04-19T23:26:43.104 DEBUG:boto:Data: 2024-04-19T23:26:43.104 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.104 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.104 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.104 DEBUG:boto:Params: {} 2024-04-19T23:26:43.104 DEBUG:boto:Token: None 2024-04-19T23:26:43.104 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj7 2024-04-19T23:26:43.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l4qKxijHU/gu/8uauJx02dUGLks= 2024-04-19T23:26:43.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l4qKxijHU/gu/8uauJx02dUGLks=', 'Content-Length': '0'} 2024-04-19T23:26:43.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c5afc38c1607f66-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.106 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.107 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.107 DEBUG:boto:path=/aktlji-16/?prefix=obj7 2024-04-19T23:26:43.107 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj7 2024-04-19T23:26:43.107 DEBUG:boto:Method: GET 2024-04-19T23:26:43.107 DEBUG:boto:Path: /aktlji-16/?prefix=obj7 2024-04-19T23:26:43.107 DEBUG:boto:Data: 2024-04-19T23:26:43.107 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.107 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.107 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.107 DEBUG:boto:Params: {} 2024-04-19T23:26:43.107 DEBUG:boto:Token: None 2024-04-19T23:26:43.107 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.108 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de4f57095c7c79c4-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.110 DEBUG:boto:b'aktlji-16obj71000falseobj72024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.114 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.114 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.114 DEBUG:boto:path=/aktlji-16/?prefix=obj7 2024-04-19T23:26:43.114 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj7 2024-04-19T23:26:43.115 DEBUG:boto:Method: GET 2024-04-19T23:26:43.115 DEBUG:boto:Path: /aktlji-16/?prefix=obj7 2024-04-19T23:26:43.115 DEBUG:boto:Data: 2024-04-19T23:26:43.115 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.115 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.115 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.115 DEBUG:boto:Params: {} 2024-04-19T23:26:43.115 DEBUG:boto:Token: None 2024-04-19T23:26:43.115 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.115 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f17f3a53e9b867a-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.118 DEBUG:boto:b'aktlji-16obj71000falseobj72024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-19T23:26:43.119 DEBUG:boto:path=/aktlji-16/obj8 2024-04-19T23:26:43.119 DEBUG:boto:auth_path=/aktlji-16/obj8 2024-04-19T23:26:43.119 DEBUG:boto:path=/aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.119 DEBUG:boto:auth_path=/aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.119 DEBUG:boto:Method: GET 2024-04-19T23:26:43.119 DEBUG:boto:Path: /aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.119 DEBUG:boto:Data: 2024-04-19T23:26:43.119 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.120 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.120 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.120 DEBUG:boto:Params: {} 2024-04-19T23:26:43.120 DEBUG:boto:Token: None 2024-04-19T23:26:43.120 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.120 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0= 2024-04-19T23:26:43.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0=', 'Content-Length': '0'} 2024-04-19T23:26:43.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068624c53f97abf98-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.166 DEBUG:boto:path=/aktlji-16/obj8 2024-04-19T23:26:43.167 DEBUG:boto:auth_path=/aktlji-16/obj8 2024-04-19T23:26:43.167 DEBUG:boto:path=/aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.167 DEBUG:boto:auth_path=/aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.167 DEBUG:boto:Method: GET 2024-04-19T23:26:43.167 DEBUG:boto:Path: /aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.167 DEBUG:boto:Data: 2024-04-19T23:26:43.167 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.167 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.167 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.167 DEBUG:boto:Params: {} 2024-04-19T23:26:43.168 DEBUG:boto:Token: None 2024-04-19T23:26:43.168 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.168 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0= 2024-04-19T23:26:43.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0=', 'Content-Length': '0'} 2024-04-19T23:26:43.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4c2bd71473e4bda-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.170 DEBUG:boto:path=/aktlji-16/obj8 2024-04-19T23:26:43.170 DEBUG:boto:auth_path=/aktlji-16/obj8 2024-04-19T23:26:43.170 DEBUG:boto:path=/aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.170 DEBUG:boto:auth_path=/aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.171 DEBUG:boto:Method: HEAD 2024-04-19T23:26:43.171 DEBUG:boto:Path: /aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.171 DEBUG:boto:Data: 2024-04-19T23:26:43.171 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.171 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.171 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.171 DEBUG:boto:Params: {} 2024-04-19T23:26:43.171 DEBUG:boto:Token: None 2024-04-19T23:26:43.171 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.171 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qsknR8i4CcVYoCg+Qvr8fg71cIU= 2024-04-19T23:26:43.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qsknR8i4CcVYoCg+Qvr8fg71cIU=', 'Content-Length': '0'} 2024-04-19T23:26:43.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b567e8ec9c43070-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.174 DEBUG:boto:path=/aktlji-16/obj8 2024-04-19T23:26:43.174 DEBUG:boto:auth_path=/aktlji-16/obj8 2024-04-19T23:26:43.174 DEBUG:boto:path=/aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.174 DEBUG:boto:auth_path=/aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.174 DEBUG:boto:Method: HEAD 2024-04-19T23:26:43.174 DEBUG:boto:Path: /aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.174 DEBUG:boto:Data: 2024-04-19T23:26:43.174 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.174 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.174 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.174 DEBUG:boto:Params: {} 2024-04-19T23:26:43.175 DEBUG:boto:Token: None 2024-04-19T23:26:43.175 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.175 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qsknR8i4CcVYoCg+Qvr8fg71cIU= 2024-04-19T23:26:43.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qsknR8i4CcVYoCg+Qvr8fg71cIU=', 'Content-Length': '0'} 2024-04-19T23:26:43.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fc8b7eea6253ccf-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-19T23:26:43.177 DEBUG:boto:path=/aktlji-16/obj8 2024-04-19T23:26:43.177 DEBUG:boto:auth_path=/aktlji-16/obj8 2024-04-19T23:26:43.177 DEBUG:boto:path=/aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.177 DEBUG:boto:auth_path=/aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.177 DEBUG:boto:Method: GET 2024-04-19T23:26:43.177 DEBUG:boto:Path: /aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.177 DEBUG:boto:Data: 2024-04-19T23:26:43.177 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.178 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.178 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.178 DEBUG:boto:Params: {} 2024-04-19T23:26:43.178 DEBUG:boto:Token: None 2024-04-19T23:26:43.178 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.178 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0= 2024-04-19T23:26:43.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0=', 'Content-Length': '0'} 2024-04-19T23:26:43.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002516b05754cc277c-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.180 DEBUG:boto:path=/aktlji-16/obj8 2024-04-19T23:26:43.180 DEBUG:boto:auth_path=/aktlji-16/obj8 2024-04-19T23:26:43.180 DEBUG:boto:path=/aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.180 DEBUG:boto:auth_path=/aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.181 DEBUG:boto:Method: GET 2024-04-19T23:26:43.181 DEBUG:boto:Path: /aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.181 DEBUG:boto:Data: 2024-04-19T23:26:43.181 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.181 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.181 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.181 DEBUG:boto:Params: {} 2024-04-19T23:26:43.181 DEBUG:boto:Token: None 2024-04-19T23:26:43.181 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj8?versionId=null 2024-04-19T23:26:43.181 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0= 2024-04-19T23:26:43.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0=', 'Content-Length': '0'} 2024-04-19T23:26:43.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c41bff5503c7c19-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.184 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.184 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.184 DEBUG:boto:path=/aktlji-16/?prefix=obj8 2024-04-19T23:26:43.184 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj8 2024-04-19T23:26:43.184 DEBUG:boto:Method: GET 2024-04-19T23:26:43.184 DEBUG:boto:Path: /aktlji-16/?prefix=obj8 2024-04-19T23:26:43.184 DEBUG:boto:Data: 2024-04-19T23:26:43.184 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.184 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.184 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.184 DEBUG:boto:Params: {} 2024-04-19T23:26:43.185 DEBUG:boto:Token: None 2024-04-19T23:26:43.185 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.185 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004afecc669040dca6-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.187 DEBUG:boto:b'aktlji-16obj81000falseobj82024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.188 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.188 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.188 DEBUG:boto:path=/aktlji-16/?prefix=obj8 2024-04-19T23:26:43.188 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj8 2024-04-19T23:26:43.188 DEBUG:boto:Method: GET 2024-04-19T23:26:43.188 DEBUG:boto:Path: /aktlji-16/?prefix=obj8 2024-04-19T23:26:43.188 DEBUG:boto:Data: 2024-04-19T23:26:43.189 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.189 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.189 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.189 DEBUG:boto:Params: {} 2024-04-19T23:26:43.189 DEBUG:boto:Token: None 2024-04-19T23:26:43.189 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.189 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043cf67e63c2006a2-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.234 DEBUG:boto:b'aktlji-16obj81000falseobj82024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.235 DEBUG:boto:path=/aktlji-16/obj8 2024-04-19T23:26:43.235 DEBUG:boto:auth_path=/aktlji-16/obj8 2024-04-19T23:26:43.235 DEBUG:boto:Method: HEAD 2024-04-19T23:26:43.235 DEBUG:boto:Path: /aktlji-16/obj8 2024-04-19T23:26:43.235 DEBUG:boto:Data: 2024-04-19T23:26:43.235 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.235 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.235 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.236 DEBUG:boto:Params: {} 2024-04-19T23:26:43.236 DEBUG:boto:Token: None 2024-04-19T23:26:43.236 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj8 2024-04-19T23:26:43.236 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q1gH94jCOB5lpLe70qwI9uQZdNI= 2024-04-19T23:26:43.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q1gH94jCOB5lpLe70qwI9uQZdNI=', 'Content-Length': '0'} 2024-04-19T23:26:43.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f59d5ebe06fc6afe-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.238 DEBUG:boto:path=/aktlji-16/obj8 2024-04-19T23:26:43.238 DEBUG:boto:auth_path=/aktlji-16/obj8 2024-04-19T23:26:43.238 DEBUG:boto:Method: HEAD 2024-04-19T23:26:43.238 DEBUG:boto:Path: /aktlji-16/obj8 2024-04-19T23:26:43.238 DEBUG:boto:Data: 2024-04-19T23:26:43.239 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.239 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.239 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.239 DEBUG:boto:Params: {} 2024-04-19T23:26:43.239 DEBUG:boto:Token: None 2024-04-19T23:26:43.239 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj8 2024-04-19T23:26:43.239 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q1gH94jCOB5lpLe70qwI9uQZdNI= 2024-04-19T23:26:43.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q1gH94jCOB5lpLe70qwI9uQZdNI=', 'Content-Length': '0'} 2024-04-19T23:26:43.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086bfa682059cfd56-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-19T23:26:43.241 DEBUG:boto:path=/aktlji-16/obj8 2024-04-19T23:26:43.241 DEBUG:boto:auth_path=/aktlji-16/obj8 2024-04-19T23:26:43.241 DEBUG:boto:Method: GET 2024-04-19T23:26:43.242 DEBUG:boto:Path: /aktlji-16/obj8 2024-04-19T23:26:43.242 DEBUG:boto:Data: 2024-04-19T23:26:43.242 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.242 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.242 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.242 DEBUG:boto:Params: {} 2024-04-19T23:26:43.242 DEBUG:boto:Token: None 2024-04-19T23:26:43.242 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj8 2024-04-19T23:26:43.242 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iUKSjOyGIIQ9ntimhnYKBENOk5Y= 2024-04-19T23:26:43.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iUKSjOyGIIQ9ntimhnYKBENOk5Y=', 'Content-Length': '0'} 2024-04-19T23:26:43.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000871c41e003e300e1-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.245 DEBUG:boto:path=/aktlji-16/obj8 2024-04-19T23:26:43.245 DEBUG:boto:auth_path=/aktlji-16/obj8 2024-04-19T23:26:43.245 DEBUG:boto:Method: GET 2024-04-19T23:26:43.245 DEBUG:boto:Path: /aktlji-16/obj8 2024-04-19T23:26:43.245 DEBUG:boto:Data: 2024-04-19T23:26:43.245 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.245 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.245 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.246 DEBUG:boto:Params: {} 2024-04-19T23:26:43.246 DEBUG:boto:Token: None 2024-04-19T23:26:43.246 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj8 2024-04-19T23:26:43.246 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iUKSjOyGIIQ9ntimhnYKBENOk5Y= 2024-04-19T23:26:43.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iUKSjOyGIIQ9ntimhnYKBENOk5Y=', 'Content-Length': '0'} 2024-04-19T23:26:43.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005468491bc38fd345-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.248 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.248 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.248 DEBUG:boto:path=/aktlji-16/?prefix=obj8 2024-04-19T23:26:43.248 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj8 2024-04-19T23:26:43.248 DEBUG:boto:Method: GET 2024-04-19T23:26:43.248 DEBUG:boto:Path: /aktlji-16/?prefix=obj8 2024-04-19T23:26:43.249 DEBUG:boto:Data: 2024-04-19T23:26:43.249 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.249 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.249 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.249 DEBUG:boto:Params: {} 2024-04-19T23:26:43.249 DEBUG:boto:Token: None 2024-04-19T23:26:43.249 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.249 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078d9572d26713fdb-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.252 DEBUG:boto:b'aktlji-16obj81000falseobj82024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.252 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.252 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.252 DEBUG:boto:path=/aktlji-16/?prefix=obj8 2024-04-19T23:26:43.253 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj8 2024-04-19T23:26:43.253 DEBUG:boto:Method: GET 2024-04-19T23:26:43.253 DEBUG:boto:Path: /aktlji-16/?prefix=obj8 2024-04-19T23:26:43.253 DEBUG:boto:Data: 2024-04-19T23:26:43.253 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.253 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.253 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.253 DEBUG:boto:Params: {} 2024-04-19T23:26:43.253 DEBUG:boto:Token: None 2024-04-19T23:26:43.253 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.254 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5a9650271aae5df-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.257 DEBUG:boto:b'aktlji-16obj81000falseobj82024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj9 2024-04-19T23:26:43.257 DEBUG:boto:path=/aktlji-16/obj9 2024-04-19T23:26:43.257 DEBUG:boto:auth_path=/aktlji-16/obj9 2024-04-19T23:26:43.257 DEBUG:boto:path=/aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.257 DEBUG:boto:auth_path=/aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.257 DEBUG:boto:Method: GET 2024-04-19T23:26:43.258 DEBUG:boto:Path: /aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.258 DEBUG:boto:Data: 2024-04-19T23:26:43.258 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.258 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.258 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.258 DEBUG:boto:Params: {} 2024-04-19T23:26:43.258 DEBUG:boto:Token: None 2024-04-19T23:26:43.258 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.258 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw= 2024-04-19T23:26:43.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw=', 'Content-Length': '0'} 2024-04-19T23:26:43.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbcab50f83431963-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.261 DEBUG:boto:path=/aktlji-16/obj9 2024-04-19T23:26:43.261 DEBUG:boto:auth_path=/aktlji-16/obj9 2024-04-19T23:26:43.261 DEBUG:boto:path=/aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.261 DEBUG:boto:auth_path=/aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.261 DEBUG:boto:Method: GET 2024-04-19T23:26:43.261 DEBUG:boto:Path: /aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.261 DEBUG:boto:Data: 2024-04-19T23:26:43.261 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.261 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.261 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.261 DEBUG:boto:Params: {} 2024-04-19T23:26:43.262 DEBUG:boto:Token: None 2024-04-19T23:26:43.262 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.262 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw= 2024-04-19T23:26:43.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw=', 'Content-Length': '0'} 2024-04-19T23:26:43.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb81a6a68bc42d26-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.264 DEBUG:boto:path=/aktlji-16/obj9 2024-04-19T23:26:43.264 DEBUG:boto:auth_path=/aktlji-16/obj9 2024-04-19T23:26:43.264 DEBUG:boto:path=/aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.264 DEBUG:boto:auth_path=/aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.264 DEBUG:boto:Method: HEAD 2024-04-19T23:26:43.264 DEBUG:boto:Path: /aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.265 DEBUG:boto:Data: 2024-04-19T23:26:43.265 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.265 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.265 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.265 DEBUG:boto:Params: {} 2024-04-19T23:26:43.265 DEBUG:boto:Token: None 2024-04-19T23:26:43.265 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.265 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hQGa/VY4gsSwFnBDoR3hY/60WQ0= 2024-04-19T23:26:43.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hQGa/VY4gsSwFnBDoR3hY/60WQ0=', 'Content-Length': '0'} 2024-04-19T23:26:43.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc0813979c5f9210-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.268 DEBUG:boto:path=/aktlji-16/obj9 2024-04-19T23:26:43.268 DEBUG:boto:auth_path=/aktlji-16/obj9 2024-04-19T23:26:43.268 DEBUG:boto:path=/aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.268 DEBUG:boto:auth_path=/aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.268 DEBUG:boto:Method: HEAD 2024-04-19T23:26:43.268 DEBUG:boto:Path: /aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.268 DEBUG:boto:Data: 2024-04-19T23:26:43.268 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.268 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.268 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.268 DEBUG:boto:Params: {} 2024-04-19T23:26:43.269 DEBUG:boto:Token: None 2024-04-19T23:26:43.269 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.269 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hQGa/VY4gsSwFnBDoR3hY/60WQ0= 2024-04-19T23:26:43.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hQGa/VY4gsSwFnBDoR3hY/60WQ0=', 'Content-Length': '0'} 2024-04-19T23:26:43.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9196269deb671e0-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj9 2024-04-19T23:26:43.271 DEBUG:boto:path=/aktlji-16/obj9 2024-04-19T23:26:43.271 DEBUG:boto:auth_path=/aktlji-16/obj9 2024-04-19T23:26:43.271 DEBUG:boto:path=/aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.271 DEBUG:boto:auth_path=/aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.271 DEBUG:boto:Method: GET 2024-04-19T23:26:43.271 DEBUG:boto:Path: /aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.271 DEBUG:boto:Data: 2024-04-19T23:26:43.271 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.271 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.272 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.272 DEBUG:boto:Params: {} 2024-04-19T23:26:43.272 DEBUG:boto:Token: None 2024-04-19T23:26:43.272 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.272 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw= 2024-04-19T23:26:43.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw=', 'Content-Length': '0'} 2024-04-19T23:26:43.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009179a24acb37ab71-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.318 DEBUG:boto:path=/aktlji-16/obj9 2024-04-19T23:26:43.319 DEBUG:boto:auth_path=/aktlji-16/obj9 2024-04-19T23:26:43.319 DEBUG:boto:path=/aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.319 DEBUG:boto:auth_path=/aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.319 DEBUG:boto:Method: GET 2024-04-19T23:26:43.319 DEBUG:boto:Path: /aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.319 DEBUG:boto:Data: 2024-04-19T23:26:43.319 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.319 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.319 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.319 DEBUG:boto:Params: {} 2024-04-19T23:26:43.320 DEBUG:boto:Token: None 2024-04-19T23:26:43.320 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj9?versionId=null 2024-04-19T23:26:43.320 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw= 2024-04-19T23:26:43.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw=', 'Content-Length': '0'} 2024-04-19T23:26:43.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000328fdad63a91bcaa-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.322 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.322 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.322 DEBUG:boto:path=/aktlji-16/?prefix=obj9 2024-04-19T23:26:43.322 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj9 2024-04-19T23:26:43.323 DEBUG:boto:Method: GET 2024-04-19T23:26:43.323 DEBUG:boto:Path: /aktlji-16/?prefix=obj9 2024-04-19T23:26:43.323 DEBUG:boto:Data: 2024-04-19T23:26:43.323 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.323 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.323 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.323 DEBUG:boto:Params: {} 2024-04-19T23:26:43.323 DEBUG:boto:Token: None 2024-04-19T23:26:43.323 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.323 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000388b8e71c23c4040-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.326 DEBUG:boto:b'aktlji-16obj91000falseobj92024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.327 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.327 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.327 DEBUG:boto:path=/aktlji-16/?prefix=obj9 2024-04-19T23:26:43.327 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj9 2024-04-19T23:26:43.327 DEBUG:boto:Method: GET 2024-04-19T23:26:43.327 DEBUG:boto:Path: /aktlji-16/?prefix=obj9 2024-04-19T23:26:43.327 DEBUG:boto:Data: 2024-04-19T23:26:43.327 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.327 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.328 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.328 DEBUG:boto:Params: {} 2024-04-19T23:26:43.328 DEBUG:boto:Token: None 2024-04-19T23:26:43.328 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.328 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000201dd4858c606354-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.330 DEBUG:boto:b'aktlji-16obj91000falseobj92024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.331 DEBUG:boto:path=/aktlji-16/obj9 2024-04-19T23:26:43.331 DEBUG:boto:auth_path=/aktlji-16/obj9 2024-04-19T23:26:43.331 DEBUG:boto:Method: HEAD 2024-04-19T23:26:43.331 DEBUG:boto:Path: /aktlji-16/obj9 2024-04-19T23:26:43.331 DEBUG:boto:Data: 2024-04-19T23:26:43.331 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.331 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.331 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.331 DEBUG:boto:Params: {} 2024-04-19T23:26:43.332 DEBUG:boto:Token: None 2024-04-19T23:26:43.332 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj9 2024-04-19T23:26:43.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6AZk+2qK4sQrxulwLPzUVx11IfE= 2024-04-19T23:26:43.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6AZk+2qK4sQrxulwLPzUVx11IfE=', 'Content-Length': '0'} 2024-04-19T23:26:43.333 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000218147c6d719f88a-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.334 DEBUG:boto:path=/aktlji-16/obj9 2024-04-19T23:26:43.334 DEBUG:boto:auth_path=/aktlji-16/obj9 2024-04-19T23:26:43.334 DEBUG:boto:Method: HEAD 2024-04-19T23:26:43.334 DEBUG:boto:Path: /aktlji-16/obj9 2024-04-19T23:26:43.334 DEBUG:boto:Data: 2024-04-19T23:26:43.334 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.334 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.334 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.334 DEBUG:boto:Params: {} 2024-04-19T23:26:43.335 DEBUG:boto:Token: None 2024-04-19T23:26:43.335 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj9 2024-04-19T23:26:43.335 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6AZk+2qK4sQrxulwLPzUVx11IfE= 2024-04-19T23:26:43.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6AZk+2qK4sQrxulwLPzUVx11IfE=', 'Content-Length': '0'} 2024-04-19T23:26:43.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc3d48c105f53d2f-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj9 2024-04-19T23:26:43.337 DEBUG:boto:path=/aktlji-16/obj9 2024-04-19T23:26:43.337 DEBUG:boto:auth_path=/aktlji-16/obj9 2024-04-19T23:26:43.337 DEBUG:boto:Method: GET 2024-04-19T23:26:43.337 DEBUG:boto:Path: /aktlji-16/obj9 2024-04-19T23:26:43.337 DEBUG:boto:Data: 2024-04-19T23:26:43.337 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.337 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.337 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.338 DEBUG:boto:Params: {} 2024-04-19T23:26:43.338 DEBUG:boto:Token: None 2024-04-19T23:26:43.338 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj9 2024-04-19T23:26:43.338 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+GHMYnL65H2K11TsBzfy9X5QAPU= 2024-04-19T23:26:43.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+GHMYnL65H2K11TsBzfy9X5QAPU=', 'Content-Length': '0'} 2024-04-19T23:26:43.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bd538a807a2b389-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.340 DEBUG:boto:path=/aktlji-16/obj9 2024-04-19T23:26:43.340 DEBUG:boto:auth_path=/aktlji-16/obj9 2024-04-19T23:26:43.340 DEBUG:boto:Method: GET 2024-04-19T23:26:43.341 DEBUG:boto:Path: /aktlji-16/obj9 2024-04-19T23:26:43.341 DEBUG:boto:Data: 2024-04-19T23:26:43.341 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.341 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.341 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.341 DEBUG:boto:Params: {} 2024-04-19T23:26:43.341 DEBUG:boto:Token: None 2024-04-19T23:26:43.341 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/obj9 2024-04-19T23:26:43.341 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+GHMYnL65H2K11TsBzfy9X5QAPU= 2024-04-19T23:26:43.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+GHMYnL65H2K11TsBzfy9X5QAPU=', 'Content-Length': '0'} 2024-04-19T23:26:43.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae722fd95b80e17f-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.386 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.386 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.387 DEBUG:boto:path=/aktlji-16/?prefix=obj9 2024-04-19T23:26:43.387 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj9 2024-04-19T23:26:43.387 DEBUG:boto:Method: GET 2024-04-19T23:26:43.387 DEBUG:boto:Path: /aktlji-16/?prefix=obj9 2024-04-19T23:26:43.387 DEBUG:boto:Data: 2024-04-19T23:26:43.387 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.387 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:26:43.387 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.387 DEBUG:boto:Params: {} 2024-04-19T23:26:43.387 DEBUG:boto:Token: None 2024-04-19T23:26:43.388 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.388 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b0b6cc0c9f5a72f-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.391 DEBUG:boto:b'aktlji-16obj91000falseobj92024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.391 DEBUG:boto:path=/aktlji-16/ 2024-04-19T23:26:43.391 DEBUG:boto:auth_path=/aktlji-16/ 2024-04-19T23:26:43.391 DEBUG:boto:path=/aktlji-16/?prefix=obj9 2024-04-19T23:26:43.391 DEBUG:boto:auth_path=/aktlji-16/?prefix=obj9 2024-04-19T23:26:43.391 DEBUG:boto:Method: GET 2024-04-19T23:26:43.391 DEBUG:boto:Path: /aktlji-16/?prefix=obj9 2024-04-19T23:26:43.392 DEBUG:boto:Data: 2024-04-19T23:26:43.392 DEBUG:boto:Headers: {} 2024-04-19T23:26:43.392 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:26:43.392 DEBUG:boto:Port: 8000 2024-04-19T23:26:43.392 DEBUG:boto:Params: {} 2024-04-19T23:26:43.392 DEBUG:boto:Token: None 2024-04-19T23:26:43.392 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:26:43 GMT /aktlji-16/ 2024-04-19T23:26:43.392 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-19T23:26:43.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-19T23:26:43.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073cb91d1a4781a13-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:26:43.395 DEBUG:boto:b'aktlji-16obj91000falseobj92024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:26:43.396 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-16 zones={test-zone2, test-zone1} 2024-04-19T23:26:43.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:26:43.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:26:43.397 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:26:43.464 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:26:43.464 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:26:43.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:26:43.511+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:26:48.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:26:48.512+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:26:53.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:26:53.512+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:26:58.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:26:58.513+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:27:03.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:27:03.514+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:27:08.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:27:08.514+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:27:13.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:27:13.515+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:27:18.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:27:18.516+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:27:23.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:27:23.517+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:27:28.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:27:28.518+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:27:33.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:27:33.519+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:27:38.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:27:38.520+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:27:43.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:27:43.520+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:27:48.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:27:48.521+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:27:53.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:27:53.521+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:27:58.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:27:58.521+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:28:03.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:28:03.522+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:28:08.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:28:08.522+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:28:13.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:28:13.523+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:28:18.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:28:18.524+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:28:23.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:28:23.525+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:28:28.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:28:28.525+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:28:33.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:28:33.526+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:28:38.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:28:38.527+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:28:43.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:28:43.528+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:28:48.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:28:48.528+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:28:53.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:28:53.529+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:28:54.486 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-19T23:28:54.490+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-19T23:28:58.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:28:58.530+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:29:03.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:29:03.530+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:29:08.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:29:08.531+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:29:13.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:29:13.533+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:29:18.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:29:18.533+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:29:23.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:29:23.534+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:29:28.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:29:28.535+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:29:33.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:29:33.535+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:29:38.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:29:38.536+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:29:43.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:29:43.536+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:29:48.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:29:48.536+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:29:53.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:29:53.537+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:29:58.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:29:58.538+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:30:03.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:30:03.538+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:30:08.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:30:08.539+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:30:13.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:30:13.540+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:30:18.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:30:18.541+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:30:23.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:30:23.541+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:30:28.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:30:28.542+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:30:33.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:30:33.543+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:30:38.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:30:38.544+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:30:43.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:30:43.544+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:30:48.541 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:30:48.545+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:30:53.541 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:30:53.547+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:30:58.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:30:58.547+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:31:03.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:03.548+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:31:08.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:08.549+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:31:13.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:13.549+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:31:18.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:18.550+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:31:23.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:23.549+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:31:28.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:28.550+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:31:33.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:33.551+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-19T23:31:38.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:38.552+0000 7f1ae9ded780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-19T23:31:38.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:38.552+0000 7f1ae9ded780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-19T23:31:38.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:38.552+0000 7f1ae9ded780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-19T23:31:38.563 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-19T23:31:38.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_object_delete ... FAIL 2024-04-19T23:31:38.577 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-19 2024-04-19T23:31:38.577 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:31:38.577 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:31:38.577 DEBUG:boto:Method: PUT 2024-04-19T23:31:38.577 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:31:38.578 DEBUG:boto:Data: 2024-04-19T23:31:38.578 DEBUG:boto:Headers: {} 2024-04-19T23:31:38.578 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:31:38.578 DEBUG:boto:Port: 8000 2024-04-19T23:31:38.578 DEBUG:boto:Params: {} 2024-04-19T23:31:38.578 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:31:38.579 DEBUG:boto:Token: None 2024-04-19T23:31:38.579 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:31:38 GMT /aktlji-19/ 2024-04-19T23:31:38.579 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fwdFW7cdTOz79wMJwRCg2okj+Mc= 2024-04-19T23:31:38.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:31:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fwdFW7cdTOz79wMJwRCg2okj+Mc=', 'Content-Length': '0'} 2024-04-19T23:31:38.602 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d0ce9faf97745de6-006622feda-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:31:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:31:38.602 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-20 2024-04-19T23:31:38.603 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:31:38.603 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:31:38.603 DEBUG:boto:Method: PUT 2024-04-19T23:31:38.603 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:31:38.603 DEBUG:boto:Data: 2024-04-19T23:31:38.603 DEBUG:boto:Headers: {} 2024-04-19T23:31:38.603 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:31:38.603 DEBUG:boto:Port: 8000 2024-04-19T23:31:38.603 DEBUG:boto:Params: {} 2024-04-19T23:31:38.604 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:31:38.604 DEBUG:boto:Token: None 2024-04-19T23:31:38.604 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:31:38 GMT /aktlji-20/ 2024-04-19T23:31:38.604 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:D1VDomuwvf851qQrYFRMVwzELts= 2024-04-19T23:31:38.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:31:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:D1VDomuwvf851qQrYFRMVwzELts=', 'Content-Length': '0'} 2024-04-19T23:31:38.637 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009c57bce297072e87-006622feda-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:31:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:31:38.637 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-21 2024-04-19T23:31:38.637 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:31:38.637 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:31:38.637 DEBUG:boto:Method: PUT 2024-04-19T23:31:38.638 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:31:38.638 DEBUG:boto:Data: 2024-04-19T23:31:38.638 DEBUG:boto:Headers: {} 2024-04-19T23:31:38.638 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:31:38.638 DEBUG:boto:Port: 8001 2024-04-19T23:31:38.638 DEBUG:boto:Params: {} 2024-04-19T23:31:38.638 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-19T23:31:38.638 DEBUG:boto:Token: None 2024-04-19T23:31:38.639 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:31:38 GMT /aktlji-21/ 2024-04-19T23:31:38.639 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:epdkkSyKMBzncAQWqmHnHWm7O2M= 2024-04-19T23:31:38.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:31:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:epdkkSyKMBzncAQWqmHnHWm7O2M=', 'Content-Length': '0'} 2024-04-19T23:31:38.674 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006ebabeb41192f5da-006622feda-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:31:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:31:38.675 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:31:38.675 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:31:38.675 DEBUG:boto:path=/aktlji-19/?versioning 2024-04-19T23:31:38.675 DEBUG:boto:auth_path=/aktlji-19/?versioning 2024-04-19T23:31:38.675 DEBUG:boto:Method: PUT 2024-04-19T23:31:38.675 DEBUG:boto:Path: /aktlji-19/?versioning 2024-04-19T23:31:38.675 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:31:38.675 DEBUG:boto:Headers: {} 2024-04-19T23:31:38.675 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:31:38.676 DEBUG:boto:Port: 8000 2024-04-19T23:31:38.676 DEBUG:boto:Params: {} 2024-04-19T23:31:38.676 DEBUG:boto:Token: None 2024-04-19T23:31:38.676 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:31:38 GMT /aktlji-19/?versioning 2024-04-19T23:31:38.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:655UkLG4xbP91I0FreYF6VtLLj0= 2024-04-19T23:31:38.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:31:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:655UkLG4xbP91I0FreYF6VtLLj0=', 'Content-Length': '228'} 2024-04-19T23:31:38.682 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003a1e93e9f2883e8d-006622feda-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:31:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:31:38.682 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:31:38.682 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:31:38.683 DEBUG:boto:path=/aktlji-20/?versioning 2024-04-19T23:31:38.683 DEBUG:boto:auth_path=/aktlji-20/?versioning 2024-04-19T23:31:38.683 DEBUG:boto:Method: PUT 2024-04-19T23:31:38.683 DEBUG:boto:Path: /aktlji-20/?versioning 2024-04-19T23:31:38.683 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:31:38.683 DEBUG:boto:Headers: {} 2024-04-19T23:31:38.683 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:31:38.683 DEBUG:boto:Port: 8000 2024-04-19T23:31:38.683 DEBUG:boto:Params: {} 2024-04-19T23:31:38.683 DEBUG:boto:Token: None 2024-04-19T23:31:38.684 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:31:38 GMT /aktlji-20/?versioning 2024-04-19T23:31:38.684 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KNP7wIpWr0IQTaouP1fUmlvdLn4= 2024-04-19T23:31:38.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:31:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KNP7wIpWr0IQTaouP1fUmlvdLn4=', 'Content-Length': '228'} 2024-04-19T23:31:38.693 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004e58e3aca2346c52-006622feda-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:31:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:31:38.693 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:31:38.693 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:31:38.694 DEBUG:boto:path=/aktlji-21/?versioning 2024-04-19T23:31:38.694 DEBUG:boto:auth_path=/aktlji-21/?versioning 2024-04-19T23:31:38.694 DEBUG:boto:Method: PUT 2024-04-19T23:31:38.694 DEBUG:boto:Path: /aktlji-21/?versioning 2024-04-19T23:31:38.694 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:31:38.694 DEBUG:boto:Headers: {} 2024-04-19T23:31:38.694 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:31:38.694 DEBUG:boto:Port: 8001 2024-04-19T23:31:38.694 DEBUG:boto:Params: {} 2024-04-19T23:31:38.694 DEBUG:boto:Token: None 2024-04-19T23:31:38.695 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:31:38 GMT /aktlji-21/?versioning 2024-04-19T23:31:38.695 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fyDspCIoRBYFRYpzHoYI0WsPujs= 2024-04-19T23:31:38.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:31:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fyDspCIoRBYFRYpzHoYI0WsPujs=', 'Content-Length': '228'} 2024-04-19T23:31:38.709 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000090905af8a849d528-006622feda-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:31:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:31:38.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:31:38.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:31:38.710 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:31:38.766 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:31:38.767 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:31:38.802 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:31:38.819 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:31:38.819 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:31:38.819 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:31:38.819 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:31:38.819 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:31:38.819 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:31:38.820 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:31:38.820 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:31:38.820 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:31:38.820 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:31:38.820 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713569498.706718_127.1", 2024-04-19T23:31:38.820 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:31:38.706718Z" 2024-04-19T23:31:38.820 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:31:38.820 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:31:38.821 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:31:38.821 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:31:38.821 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:31:38.821 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:31:38.821 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569498.706718_127.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:31:38.822 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:31:38.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:31:38.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:31:38.823 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:31:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:31:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:31:38.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:31:38.938+0000 7f06a1d29780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:31:38.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:31:38.939+0000 7f06a1d29780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:31:38.948 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:31:38.948 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:31:38.948 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:31:38.949 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:31:38.949 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:31:38.949 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:31:38.949 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:38.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:38.949 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:31:38.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:38.949 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:31:38.949 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:38.949 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:38.950 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:31:38.950 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:38.950 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:31:38.950 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:38.950 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:31:38.950 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:38.950 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:38.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:38.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:38.951 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:31:38.951 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:38.951 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:38.951 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:31:38.951 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:38.951 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:31:38.951 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:38.951 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:31:38.951 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:38.951 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:38.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:38.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:38.952 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:31:38.952 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:38.952 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:38.952 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713569498.597955_110.1", 2024-04-19T23:31:38.952 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:38.952 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:31:38.952 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:38.952 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:31:38.597955Z", 2024-04-19T23:31:38.952 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:38.953 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:38.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:38.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:38.953 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:31:38.953 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:38.953 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:38.953 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:31:38.953 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:38.953 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:31:38.954 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:38.954 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:31:38.954 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:38.954 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:38.954 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:38.954 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:31:38.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:38.954 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:31:38.954 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:31:38.954 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:31:38.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:38.955 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:31:38Z" 2024-04-19T23:31:38.955 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:31:38.956 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569498.597955_110.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:31:38.597955Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:31:38Z" } 2024-04-19T23:31:38.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569498.597955_110.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:31:38.597955Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:31:38.956 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569498.706718_127.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:31:38.956 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569498.597955_110.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:31:38.956 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713569498.706718_127.1 target=1_1713569498.597955_110.1 2024-04-19T23:31:43.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:31:43.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:31:43.958 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:31:44.017 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:31:44.017 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:31:44.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:31:44.057+0000 7f169373f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:31:44.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:31:44.058+0000 7f169373f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:31:44.068 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:31:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:31:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:31:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:31:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:31:44.068 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:31:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:31:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:31:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:44.069 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:31:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:31:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:31:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:44.070 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:31:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:31:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:31:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:31:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:44.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:31:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713569498.624399_115.1", 2024-04-19T23:31:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:31:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:31:38.624399Z", 2024-04-19T23:31:44.072 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:31:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:31:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:31:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:31:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:31:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:31:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:31:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:31:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:31:44Z" 2024-04-19T23:31:44.074 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:31:44.075 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569498.624399_115.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:31:38.624399Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:31:44Z" } 2024-04-19T23:31:44.076 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569498.624399_115.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:31:38.624399Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:31:44.076 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569498.706718_127.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:31:44.076 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569498.624399_115.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:31:44.076 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713569498.706718_127.1 target=1_1713569498.624399_115.1 2024-04-19T23:31:49.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:31:49.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:31:49.079 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:31:49.138 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:31:49.138 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:31:49.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:31:49.178+0000 7f3c0ed1f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:31:49.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:31:49.179+0000 7f3c0ed1f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:31:49.188 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:31:49.189 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:31:49.189 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:31:49.189 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:31:49.189 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:31:49.189 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:31:49.189 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:49.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:49.189 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:31:49.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:49.190 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:31:49.190 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:49.190 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:49.190 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:31:49.190 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:49.190 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:31:49.190 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:49.190 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:31:49.190 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:49.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:49.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:49.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:49.191 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:31:49.191 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:49.191 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:49.191 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:31:49.191 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:49.191 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:31:49.191 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:49.191 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:31:49.192 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:49.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:49.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:49.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:49.192 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:31:49.192 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:49.192 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:49.192 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713569498.621008_114.1", 2024-04-19T23:31:49.192 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:49.192 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:31:49.192 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:49.192 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:31:38.621008Z", 2024-04-19T23:31:49.193 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:49.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:49.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:49.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:49.193 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:31:49.193 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:49.193 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:49.193 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:31:49.194 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:49.194 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:31:49.194 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:49.194 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:31:49.194 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:49.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:49.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:49.194 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:31:49.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:49.195 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:31:49.195 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:31:49.195 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:31:49.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:49.195 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:31:49Z" 2024-04-19T23:31:49.195 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:31:49.196 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569498.621008_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:31:38.621008Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:31:49Z" } 2024-04-19T23:31:49.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569498.621008_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:31:38.621008Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:31:49.196 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569498.706718_127.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:31:49.196 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569498.621008_114.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:31:49.197 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713569498.706718_127.1 target=1_1713569498.621008_114.1 2024-04-19T23:31:54.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:31:54.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:31:54.198 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:31:54.257 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:31:54.257 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:31:54.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:31:54.297+0000 7fcaab181780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:31:54.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:31:54.299+0000 7fcaab181780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:31:54.307 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:31:54.308 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:31:54.308 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:31:54.308 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:31:54.308 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:31:54.308 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:31:54.308 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:54.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:54.309 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:31:54.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:54.309 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:31:54.309 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:54.309 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:54.309 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:31:54.309 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:54.309 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:31:54.309 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:54.310 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:31:54.310 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:54.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:54.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:54.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:54.310 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:31:54.310 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:54.310 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:54.310 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:31:54.310 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:54.311 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:31:54.311 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:54.311 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:31:54.311 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:54.311 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:54.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:54.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:54.311 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:31:54.311 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:54.311 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:54.312 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713569498.706718_127.1", 2024-04-19T23:31:54.312 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:54.312 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:31:54.312 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:54.312 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:31:38.706718Z", 2024-04-19T23:31:54.312 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:54.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:54.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:54.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:31:54.312 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:31:54.313 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:31:54.313 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:31:54.313 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:31:54.313 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:31:54.313 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:31:54.313 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:31:54.313 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:31:54.313 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:31:54.313 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:54.313 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:31:54.313 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:31:54.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:54.314 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:31:54.314 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:31:54.314 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:31:54.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:31:54.314 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:31:54Z" 2024-04-19T23:31:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:31:54.315 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569498.706718_127.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:31:38.706718Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:31:54Z" } 2024-04-19T23:31:54.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569498.706718_127.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:31:38.706718Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:31:54.316 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569498.706718_127.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:31:54.316 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569498.706718_127.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:31:54.316 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:31:54.316 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:31:54.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:31:54.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:31:54.317 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:31:54.378 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:31:54.378 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:31:54.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:54.417+0000 7f07b3f6e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:31:54.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:54.418+0000 7f07b3f6e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:31:54.429 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:31:54.429 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:31:54.429 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:31:54.429 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:31:54.429 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:31:54.429 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:31:54.429 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:31:54.429 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:31:54.430 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:31:54.430 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:31:54.430 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:31:54.430 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:31:54.430 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:31:54.430 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:31:54.430 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:31:54.430 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:31:54.430 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:31:54.430 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:31:54.431 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:31:54.431 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:31:54.431 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:31:54.431 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:31:54.431 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:31:54.431 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:31:54.431 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:31:54.431 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:31:54.431 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:31:54.432 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:31:54.432 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:31:54.432 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:31:54.432 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:31:54.432 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:31:54.432 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:31:54.432 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:31:54.432 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:31:54.432 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:31:54.432 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:31:54.433 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713569498.706718_127.1", 2024-04-19T23:31:54.433 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:31:54.433 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:31:54.433 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:31:54.433 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:31:38.706718Z", 2024-04-19T23:31:54.433 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:31:54.433 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:31:54.433 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:31:54.433 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:31:54.433 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:31:54.434 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:31:54.434 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:31:54.434 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:31:54.434 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:31:54.434 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:31:54.434 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:31:54.434 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:31:54.434 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:31:54.434 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:31:54.434 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:31:54.435 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:31:54.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:31:54.435 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:31:54.435 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:31:54.435 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:31:54.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:31:54.435 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:31:54Z" 2024-04-19T23:31:54.435 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:31:54.436 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569498.706718_127.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:31:38.706718Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:31:54Z" } 2024-04-19T23:31:54.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569498.706718_127.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:31:38.706718Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:31:54.436 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569498.706718_127.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:31:54.437 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569498.706718_127.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:31:54.437 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:31:54.437 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:31:54.437 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:31:54.437 DEBUG:boto:Method: HEAD 2024-04-19T23:31:54.437 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:31:54.437 DEBUG:boto:Data: 2024-04-19T23:31:54.438 DEBUG:boto:Headers: {} 2024-04-19T23:31:54.438 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:31:54.438 DEBUG:boto:Port: 8000 2024-04-19T23:31:54.438 DEBUG:boto:Params: {} 2024-04-19T23:31:54.438 DEBUG:boto:Token: None 2024-04-19T23:31:54.438 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:31:54 GMT /aktlji-19/ 2024-04-19T23:31:54.438 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KIxuzgb7xBFL2z4qb6lE8Wf9p04= 2024-04-19T23:31:54.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:31:54 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KIxuzgb7xBFL2z4qb6lE8Wf9p04=', 'Content-Length': '0'} 2024-04-19T23:31:54.441 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f6df8d47c206577-006622feea-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:31:54.441 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:31:54.441 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:31:54.442 DEBUG:boto:Method: PUT 2024-04-19T23:31:54.442 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:31:54.442 DEBUG:boto:Data: 2024-04-19T23:31:54.442 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:31:54.442 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:31:54.442 DEBUG:boto:Port: 8000 2024-04-19T23:31:54.442 DEBUG:boto:Params: {} 2024-04-19T23:31:54.442 DEBUG:boto:Token: None 2024-04-19T23:31:54.442 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:31:54 GMT /aktlji-19/dummy 2024-04-19T23:31:54.443 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KO962si0C1AYosNs+5odurJi+d8= 2024-04-19T23:31:54.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:31:54 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KO962si0C1AYosNs+5odurJi+d8='} 2024-04-19T23:31:54.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-amz-request-id', 'tx00000a1581addb939cf36-006622feea-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:31:54.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:31:54.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:31:54.457 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:31:54.526 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:31:54.527 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:31:54.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:31:54.578+0000 7fe7d4076780 1 waiting to reach incremental sync.. 2024-04-19T23:31:59.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:31:59.579+0000 7fe7d4076780 1 bucket sync caught up with source: 2024-04-19T23:31:59.575 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , 00000000002.31.11, , , , , ] 2024-04-19T23:31:59.575 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , 00000000002.31.11, , , , , ] 2024-04-19T23:31:59.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:31:59.579+0000 7fe7d4076780 0 bucket checkpoint complete 2024-04-19T23:31:59.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:31:59.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:31:59.592 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:31:59.689 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:31:59.689 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:31:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:59.733+0000 7fef5d11a780 1 bucket sync caught up with source: 2024-04-19T23:31:59.729 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000002.31.11, , , , , ] 2024-04-19T23:31:59.729 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000002.31.11, , , , , ] 2024-04-19T23:31:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:59.733+0000 7fef5d11a780 0 bucket checkpoint complete 2024-04-19T23:31:59.743 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:31:59.743 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:31:59.743 DEBUG:boto:Method: HEAD 2024-04-19T23:31:59.743 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:31:59.743 DEBUG:boto:Data: 2024-04-19T23:31:59.743 DEBUG:boto:Headers: {} 2024-04-19T23:31:59.743 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:31:59.743 DEBUG:boto:Port: 8000 2024-04-19T23:31:59.743 DEBUG:boto:Params: {} 2024-04-19T23:31:59.744 DEBUG:boto:Token: None 2024-04-19T23:31:59.744 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:31:59 GMT /aktlji-20/ 2024-04-19T23:31:59.744 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/oIjqdm77KJ2t64B8wfrIk5QvIM= 2024-04-19T23:31:59.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:31:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/oIjqdm77KJ2t64B8wfrIk5QvIM=', 'Content-Length': '0'} 2024-04-19T23:31:59.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bc1a8283f08ca55-006622feef-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:31:59.747 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:31:59.748 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:31:59.748 DEBUG:boto:Method: PUT 2024-04-19T23:31:59.748 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:31:59.748 DEBUG:boto:Data: 2024-04-19T23:31:59.748 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:31:59.748 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:31:59.748 DEBUG:boto:Port: 8000 2024-04-19T23:31:59.749 DEBUG:boto:Params: {} 2024-04-19T23:31:59.749 DEBUG:boto:Token: None 2024-04-19T23:31:59.749 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:31:59 GMT /aktlji-20/dummy 2024-04-19T23:31:59.749 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bk6H90iwkxDcJTAdiX5F5SUXIWA= 2024-04-19T23:31:59.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:31:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bk6H90iwkxDcJTAdiX5F5SUXIWA='} 2024-04-19T23:31:59.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-amz-request-id', 'tx00000ec8443220c999291-006622feef-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:31:59.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:31:59.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:31:59.763 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:31:59.826 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:31:59.826 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:31:59.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:31:59.869+0000 7fc04f8e3780 1 waiting to reach incremental sync.. 2024-04-19T23:32:04.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:04.870+0000 7fc04f8e3780 1 waiting to reach incremental sync.. 2024-04-19T23:32:09.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:09.870+0000 7fc04f8e3780 1 waiting to reach incremental sync.. 2024-04-19T23:32:14.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:14.872+0000 7fc04f8e3780 1 bucket sync caught up with source: 2024-04-19T23:32:14.868 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000002.44.11, , , , , ] 2024-04-19T23:32:14.869 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000002.44.11, , , , , ] 2024-04-19T23:32:14.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:14.872+0000 7fc04f8e3780 0 bucket checkpoint complete 2024-04-19T23:32:14.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:14.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:14.885 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:32:14.950 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:32:14.951 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:32:14.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:14.995+0000 7fc8661de780 1 waiting to reach incremental sync.. 2024-04-19T23:32:19.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:19.996+0000 7fc8661de780 1 bucket sync caught up with source: 2024-04-19T23:32:19.992 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000003.46.10, , , , , ] 2024-04-19T23:32:19.993 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000002.44.11, , , , , ] 2024-04-19T23:32:19.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:19.996+0000 7fc8661de780 0 bucket checkpoint complete 2024-04-19T23:32:20.009 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:32:20.009 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:32:20.009 DEBUG:boto:Method: HEAD 2024-04-19T23:32:20.010 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:32:20.010 DEBUG:boto:Data: 2024-04-19T23:32:20.010 DEBUG:boto:Headers: {} 2024-04-19T23:32:20.010 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:20.010 DEBUG:boto:Port: 8001 2024-04-19T23:32:20.010 DEBUG:boto:Params: {} 2024-04-19T23:32:20.010 DEBUG:boto:Token: None 2024-04-19T23:32:20.011 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:32:20 GMT /aktlji-21/ 2024-04-19T23:32:20.011 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CxcltmR+OxZ0kzi58oHLFz+nmZY= 2024-04-19T23:32:20.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CxcltmR+OxZ0kzi58oHLFz+nmZY=', 'Content-Length': '0'} 2024-04-19T23:32:20.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016dc16d23b4ca972-006622ff04-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:20.014 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:32:20.014 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:32:20.014 DEBUG:boto:Method: PUT 2024-04-19T23:32:20.014 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:32:20.015 DEBUG:boto:Data: 2024-04-19T23:32:20.015 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:32:20.015 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:20.015 DEBUG:boto:Port: 8001 2024-04-19T23:32:20.015 DEBUG:boto:Params: {} 2024-04-19T23:32:20.015 DEBUG:boto:Token: None 2024-04-19T23:32:20.015 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:32:20 GMT /aktlji-21/dummy 2024-04-19T23:32:20.015 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1LBxWkHDNosF90v8gCJEQ51e7AY= 2024-04-19T23:32:20.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1LBxWkHDNosF90v8gCJEQ51e7AY='} 2024-04-19T23:32:20.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-amz-request-id', 'tx000001712c82447d8e15e-006622ff04-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:20.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:20.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:20.029 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:32:20.094 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:32:20.094 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:32:20.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:20.140+0000 7fb2ddb20780 1 waiting to reach incremental sync.. 2024-04-19T23:32:25.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:25.140+0000 7fb2ddb20780 1 waiting to reach incremental sync.. 2024-04-19T23:32:30.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:30.142+0000 7fb2ddb20780 1 waiting to reach incremental sync.. 2024-04-19T23:32:35.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:35.144+0000 7fb2ddb20780 1 bucket sync caught up with source: 2024-04-19T23:32:35.140 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000002.27.11, , , , , ] 2024-04-19T23:32:35.140 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000002.27.11, , , , , ] 2024-04-19T23:32:35.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:35.144+0000 7fb2ddb20780 0 bucket checkpoint complete 2024-04-19T23:32:35.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:35.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:35.156 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:32:35.224 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:32:35.224 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:32:35.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:32:35.273+0000 7f90a2c17780 1 bucket sync caught up with source: 2024-04-19T23:32:35.269 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , 00000000002.27.11, , , , , ] 2024-04-19T23:32:35.269 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , 00000000002.27.11, , , , , ] 2024-04-19T23:32:35.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:32:35.273+0000 7f90a2c17780 0 bucket checkpoint complete 2024-04-19T23:32:35.283 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:32:35.283 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:32:35.283 DEBUG:boto:Method: HEAD 2024-04-19T23:32:35.284 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:32:35.284 DEBUG:boto:Data: 2024-04-19T23:32:35.284 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.284 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.284 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.284 DEBUG:boto:Params: {} 2024-04-19T23:32:35.284 DEBUG:boto:Token: None 2024-04-19T23:32:35.284 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/ 2024-04-19T23:32:35.285 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ho+aD0MtFu8zngtTK4+cyCohLkg= 2024-04-19T23:32:35.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ho+aD0MtFu8zngtTK4+cyCohLkg=', 'Content-Length': '0'} 2024-04-19T23:32:35.287 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030d0c2c67004963f-006622ff13-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.288 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:32:35.288 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:32:35.288 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.288 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:32:35.288 DEBUG:boto:Data: 2024-04-19T23:32:35.288 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.288 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.288 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.288 DEBUG:boto:Params: {} 2024-04-19T23:32:35.289 DEBUG:boto:Token: None 2024-04-19T23:32:35.289 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:32:35.289 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u25kokO8vzaMvwhszBM5h8nwDC4= 2024-04-19T23:32:35.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u25kokO8vzaMvwhszBM5h8nwDC4='} 2024-04-19T23:32:35.301 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-amz-request-id', 'tx00000a1a963aee8fbcef3-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.301 DEBUG:rgw_multi.tests:version1 id=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:32:35.302 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:32:35.302 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:32:35.302 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.302 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:32:35.302 DEBUG:boto:Data: 2024-04-19T23:32:35.302 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.302 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.302 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.303 DEBUG:boto:Params: {} 2024-04-19T23:32:35.303 DEBUG:boto:Token: None 2024-04-19T23:32:35.303 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:32:35.303 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rpjIjbRhNoC48V9OaDf1rCT0r/4= 2024-04-19T23:32:35.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rpjIjbRhNoC48V9OaDf1rCT0r/4='} 2024-04-19T23:32:35.313 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'c.mycN7iZxHaMud8AzYFtAHFR49hwr3'), ('x-amz-request-id', 'tx0000010a021e63fa312d5-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.313 DEBUG:rgw_multi.tests:version2 id=c.mycN7iZxHaMud8AzYFtAHFR49hwr3 2024-04-19T23:32:35.314 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:32:35.314 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:32:35.314 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=c.mycN7iZxHaMud8AzYFtAHFR49hwr3 2024-04-19T23:32:35.314 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=c.mycN7iZxHaMud8AzYFtAHFR49hwr3 2024-04-19T23:32:35.314 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.314 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=c.mycN7iZxHaMud8AzYFtAHFR49hwr3 2024-04-19T23:32:35.314 DEBUG:boto:Data: 2024-04-19T23:32:35.314 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.314 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.315 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.315 DEBUG:boto:Params: {} 2024-04-19T23:32:35.315 DEBUG:boto:Token: None 2024-04-19T23:32:35.315 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone1?versionId=c.mycN7iZxHaMud8AzYFtAHFR49hwr3 2024-04-19T23:32:35.315 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Fm4X5h0qI8LC28ntSu5k6Uq95f4= 2024-04-19T23:32:35.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Fm4X5h0qI8LC28ntSu5k6Uq95f4=', 'Content-Length': '0'} 2024-04-19T23:32:35.326 DEBUG:boto:Response headers: [('x-amz-version-id', 'c.mycN7iZxHaMud8AzYFtAHFR49hwr3'), ('x-amz-request-id', 'tx0000059360c0771c8b2c5-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.326 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:32:35.327 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:32:35.327 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.327 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:32:35.327 DEBUG:boto:Data: 2024-04-19T23:32:35.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.327 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.327 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.327 DEBUG:boto:Params: {} 2024-04-19T23:32:35.327 DEBUG:boto:Token: None 2024-04-19T23:32:35.328 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:32:35.328 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+Kfnag2RK2SyPyGlbXUmfoBpWpQ= 2024-04-19T23:32:35.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+Kfnag2RK2SyPyGlbXUmfoBpWpQ='} 2024-04-19T23:32:35.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'SJxAP8Z60t1h6QRFifnVpLrXTVsSAOY'), ('x-amz-request-id', 'tx000001b82f6a27810d057-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.371 DEBUG:rgw_multi.tests:version3 id=SJxAP8Z60t1h6QRFifnVpLrXTVsSAOY 2024-04-19T23:32:35.371 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:32:35.371 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:32:35.371 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=SJxAP8Z60t1h6QRFifnVpLrXTVsSAOY 2024-04-19T23:32:35.371 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=SJxAP8Z60t1h6QRFifnVpLrXTVsSAOY 2024-04-19T23:32:35.371 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.371 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=SJxAP8Z60t1h6QRFifnVpLrXTVsSAOY 2024-04-19T23:32:35.371 DEBUG:boto:Data: 2024-04-19T23:32:35.372 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.372 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.372 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.372 DEBUG:boto:Params: {} 2024-04-19T23:32:35.372 DEBUG:boto:Token: None 2024-04-19T23:32:35.372 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone1?versionId=SJxAP8Z60t1h6QRFifnVpLrXTVsSAOY 2024-04-19T23:32:35.372 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rFNwFE/HrKYVEvCfabRg0ZGqXjU= 2024-04-19T23:32:35.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rFNwFE/HrKYVEvCfabRg0ZGqXjU=', 'Content-Length': '0'} 2024-04-19T23:32:35.383 DEBUG:boto:Response headers: [('x-amz-version-id', 'SJxAP8Z60t1h6QRFifnVpLrXTVsSAOY'), ('x-amz-request-id', 'tx0000061f60a2538518a6b-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.384 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:32:35.384 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:32:35.384 DEBUG:boto:Method: HEAD 2024-04-19T23:32:35.384 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:32:35.384 DEBUG:boto:Data: 2024-04-19T23:32:35.384 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.384 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:35.384 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.384 DEBUG:boto:Params: {} 2024-04-19T23:32:35.385 DEBUG:boto:Token: None 2024-04-19T23:32:35.385 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/ 2024-04-19T23:32:35.385 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ho+aD0MtFu8zngtTK4+cyCohLkg= 2024-04-19T23:32:35.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ho+aD0MtFu8zngtTK4+cyCohLkg=', 'Content-Length': '0'} 2024-04-19T23:32:35.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006550b6b696d5e46b-006622ff13-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.388 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:32:35.388 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:32:35.388 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.388 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:32:35.388 DEBUG:boto:Data: 2024-04-19T23:32:35.388 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.388 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:35.388 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.388 DEBUG:boto:Params: {} 2024-04-19T23:32:35.389 DEBUG:boto:Token: None 2024-04-19T23:32:35.389 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:32:35.389 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ujnqxyg7/Y9E7/btnZqOE7PBnZM= 2024-04-19T23:32:35.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ujnqxyg7/Y9E7/btnZqOE7PBnZM='} 2024-04-19T23:32:35.401 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-amz-request-id', 'tx000002193470891ec87b1-006622ff13-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.402 DEBUG:rgw_multi.tests:version1 id=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:32:35.402 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:32:35.402 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:32:35.402 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.402 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:32:35.402 DEBUG:boto:Data: 2024-04-19T23:32:35.402 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.402 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:35.403 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.403 DEBUG:boto:Params: {} 2024-04-19T23:32:35.403 DEBUG:boto:Token: None 2024-04-19T23:32:35.403 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:32:35.403 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cLEBEh2iqB0qrN/73EwSgcunTQk= 2024-04-19T23:32:35.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cLEBEh2iqB0qrN/73EwSgcunTQk='} 2024-04-19T23:32:35.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7FE4ixM8m5scRR9ycHl8Onj-MFjEC-q'), ('x-amz-request-id', 'tx00000925114c94915805f-006622ff13-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.451 DEBUG:rgw_multi.tests:version2 id=7FE4ixM8m5scRR9ycHl8Onj-MFjEC-q 2024-04-19T23:32:35.451 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:32:35.451 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:32:35.451 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=7FE4ixM8m5scRR9ycHl8Onj-MFjEC-q 2024-04-19T23:32:35.451 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=7FE4ixM8m5scRR9ycHl8Onj-MFjEC-q 2024-04-19T23:32:35.451 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.452 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=7FE4ixM8m5scRR9ycHl8Onj-MFjEC-q 2024-04-19T23:32:35.452 DEBUG:boto:Data: 2024-04-19T23:32:35.452 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.452 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:35.452 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.452 DEBUG:boto:Params: {} 2024-04-19T23:32:35.452 DEBUG:boto:Token: None 2024-04-19T23:32:35.452 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone2?versionId=7FE4ixM8m5scRR9ycHl8Onj-MFjEC-q 2024-04-19T23:32:35.453 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/09Q4FkiP8cVsuh9pNnG4LzB5AM= 2024-04-19T23:32:35.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/09Q4FkiP8cVsuh9pNnG4LzB5AM=', 'Content-Length': '0'} 2024-04-19T23:32:35.463 DEBUG:boto:Response headers: [('x-amz-version-id', '7FE4ixM8m5scRR9ycHl8Onj-MFjEC-q'), ('x-amz-request-id', 'tx00000833415f9144f1719-006622ff13-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.464 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:32:35.464 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:32:35.464 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.464 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:32:35.464 DEBUG:boto:Data: 2024-04-19T23:32:35.464 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.464 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:35.464 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.464 DEBUG:boto:Params: {} 2024-04-19T23:32:35.465 DEBUG:boto:Token: None 2024-04-19T23:32:35.465 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:32:35.465 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EONpgd1WZJqIiMbFzfQDIZUEe6g= 2024-04-19T23:32:35.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EONpgd1WZJqIiMbFzfQDIZUEe6g='} 2024-04-19T23:32:35.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xcB73iR9xNXmT2x1O2r9utWbw.S2Eu9'), ('x-amz-request-id', 'tx000001425113f4a8ad12b-006622ff13-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.475 DEBUG:rgw_multi.tests:version3 id=xcB73iR9xNXmT2x1O2r9utWbw.S2Eu9 2024-04-19T23:32:35.475 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:32:35.476 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:32:35.476 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=xcB73iR9xNXmT2x1O2r9utWbw.S2Eu9 2024-04-19T23:32:35.476 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=xcB73iR9xNXmT2x1O2r9utWbw.S2Eu9 2024-04-19T23:32:35.476 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.476 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=xcB73iR9xNXmT2x1O2r9utWbw.S2Eu9 2024-04-19T23:32:35.476 DEBUG:boto:Data: 2024-04-19T23:32:35.476 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.476 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:35.476 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.476 DEBUG:boto:Params: {} 2024-04-19T23:32:35.476 DEBUG:boto:Token: None 2024-04-19T23:32:35.477 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone2?versionId=xcB73iR9xNXmT2x1O2r9utWbw.S2Eu9 2024-04-19T23:32:35.477 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mfOQTa0H0gNOHTmyKi0UXXAPfqw= 2024-04-19T23:32:35.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mfOQTa0H0gNOHTmyKi0UXXAPfqw=', 'Content-Length': '0'} 2024-04-19T23:32:35.487 DEBUG:boto:Response headers: [('x-amz-version-id', 'xcB73iR9xNXmT2x1O2r9utWbw.S2Eu9'), ('x-amz-request-id', 'tx00000c8d63b03ae233801-006622ff13-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.487 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:32:35.488 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:32:35.488 DEBUG:boto:Method: HEAD 2024-04-19T23:32:35.488 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:32:35.488 DEBUG:boto:Data: 2024-04-19T23:32:35.488 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.488 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:35.488 DEBUG:boto:Port: 8001 2024-04-19T23:32:35.488 DEBUG:boto:Params: {} 2024-04-19T23:32:35.488 DEBUG:boto:Token: None 2024-04-19T23:32:35.488 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/ 2024-04-19T23:32:35.489 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ho+aD0MtFu8zngtTK4+cyCohLkg= 2024-04-19T23:32:35.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ho+aD0MtFu8zngtTK4+cyCohLkg=', 'Content-Length': '0'} 2024-04-19T23:32:35.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f337d94ee25c0973-006622ff13-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.491 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:32:35.492 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:32:35.492 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.492 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:32:35.492 DEBUG:boto:Data: 2024-04-19T23:32:35.492 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.492 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:35.492 DEBUG:boto:Port: 8001 2024-04-19T23:32:35.492 DEBUG:boto:Params: {} 2024-04-19T23:32:35.492 DEBUG:boto:Token: None 2024-04-19T23:32:35.493 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:32:35.493 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RYG0+kljb4PFebbX3x7mqpXnbCk= 2024-04-19T23:32:35.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RYG0+kljb4PFebbX3x7mqpXnbCk='} 2024-04-19T23:32:35.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-amz-request-id', 'tx000008cf22f11ac19ebac-006622ff13-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.505 DEBUG:rgw_multi.tests:version1 id=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:32:35.506 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:32:35.506 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:32:35.506 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.506 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:32:35.506 DEBUG:boto:Data: 2024-04-19T23:32:35.506 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.506 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:35.506 DEBUG:boto:Port: 8001 2024-04-19T23:32:35.506 DEBUG:boto:Params: {} 2024-04-19T23:32:35.507 DEBUG:boto:Token: None 2024-04-19T23:32:35.507 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:32:35.507 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Dc3y4z86pRwfTGq/t6IUDSeYb68= 2024-04-19T23:32:35.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Dc3y4z86pRwfTGq/t6IUDSeYb68='} 2024-04-19T23:32:35.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ZgseuRXCqceXhX5w91PJ84zglcCEZzU'), ('x-amz-request-id', 'tx00000d023b991f24cc030-006622ff13-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.555 DEBUG:rgw_multi.tests:version2 id=ZgseuRXCqceXhX5w91PJ84zglcCEZzU 2024-04-19T23:32:35.555 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:32:35.555 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:32:35.555 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=ZgseuRXCqceXhX5w91PJ84zglcCEZzU 2024-04-19T23:32:35.555 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=ZgseuRXCqceXhX5w91PJ84zglcCEZzU 2024-04-19T23:32:35.555 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.555 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=ZgseuRXCqceXhX5w91PJ84zglcCEZzU 2024-04-19T23:32:35.555 DEBUG:boto:Data: 2024-04-19T23:32:35.556 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.556 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:35.556 DEBUG:boto:Port: 8001 2024-04-19T23:32:35.556 DEBUG:boto:Params: {} 2024-04-19T23:32:35.556 DEBUG:boto:Token: None 2024-04-19T23:32:35.556 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone3?versionId=ZgseuRXCqceXhX5w91PJ84zglcCEZzU 2024-04-19T23:32:35.556 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hNj4be/iyi3gZYcRxYf7BqAWiSU= 2024-04-19T23:32:35.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hNj4be/iyi3gZYcRxYf7BqAWiSU=', 'Content-Length': '0'} 2024-04-19T23:32:35.568 DEBUG:boto:Response headers: [('x-amz-version-id', 'ZgseuRXCqceXhX5w91PJ84zglcCEZzU'), ('x-amz-request-id', 'tx00000fcc0a481a3e1e8f9-006622ff13-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.568 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:32:35.568 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:32:35.568 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.568 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:32:35.569 DEBUG:boto:Data: 2024-04-19T23:32:35.569 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.569 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:35.569 DEBUG:boto:Port: 8001 2024-04-19T23:32:35.569 DEBUG:boto:Params: {} 2024-04-19T23:32:35.569 DEBUG:boto:Token: None 2024-04-19T23:32:35.569 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:32:35.569 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2qjxYfvoPDD20fQh909UO6aellw= 2024-04-19T23:32:35.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2qjxYfvoPDD20fQh909UO6aellw='} 2024-04-19T23:32:35.580 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'rC.KcprZOTvbxy49TkCNacEjtTJ95Pi'), ('x-amz-request-id', 'tx0000041c3dc18bc3f72a2-006622ff13-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.580 DEBUG:rgw_multi.tests:version3 id=rC.KcprZOTvbxy49TkCNacEjtTJ95Pi 2024-04-19T23:32:35.580 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:32:35.581 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:32:35.581 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=rC.KcprZOTvbxy49TkCNacEjtTJ95Pi 2024-04-19T23:32:35.581 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=rC.KcprZOTvbxy49TkCNacEjtTJ95Pi 2024-04-19T23:32:35.581 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.581 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=rC.KcprZOTvbxy49TkCNacEjtTJ95Pi 2024-04-19T23:32:35.581 DEBUG:boto:Data: 2024-04-19T23:32:35.581 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.581 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:35.581 DEBUG:boto:Port: 8001 2024-04-19T23:32:35.581 DEBUG:boto:Params: {} 2024-04-19T23:32:35.582 DEBUG:boto:Token: None 2024-04-19T23:32:35.582 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-19/obj-test-zone3?versionId=rC.KcprZOTvbxy49TkCNacEjtTJ95Pi 2024-04-19T23:32:35.582 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+M4HEVVw4tkXkosF9vC+evt+Ot8= 2024-04-19T23:32:35.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+M4HEVVw4tkXkosF9vC+evt+Ot8=', 'Content-Length': '0'} 2024-04-19T23:32:35.592 DEBUG:boto:Response headers: [('x-amz-version-id', 'rC.KcprZOTvbxy49TkCNacEjtTJ95Pi'), ('x-amz-request-id', 'tx00000bd6b9d8e52be462d-006622ff13-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.593 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:32:35.593 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:32:35.593 DEBUG:boto:Method: HEAD 2024-04-19T23:32:35.593 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:32:35.593 DEBUG:boto:Data: 2024-04-19T23:32:35.593 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.593 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.593 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.593 DEBUG:boto:Params: {} 2024-04-19T23:32:35.594 DEBUG:boto:Token: None 2024-04-19T23:32:35.594 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/ 2024-04-19T23:32:35.594 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b5QIE/iBiTig2L1AT4YR5OXqAaA= 2024-04-19T23:32:35.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b5QIE/iBiTig2L1AT4YR5OXqAaA=', 'Content-Length': '0'} 2024-04-19T23:32:35.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5550b156f43fbe7-006622ff13-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.597 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:32:35.597 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:32:35.597 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.597 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:32:35.597 DEBUG:boto:Data: 2024-04-19T23:32:35.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.597 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.597 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.597 DEBUG:boto:Params: {} 2024-04-19T23:32:35.598 DEBUG:boto:Token: None 2024-04-19T23:32:35.598 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:32:35.598 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wtdjgpfcNs8voaLrjwNdDwMqJw8= 2024-04-19T23:32:35.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wtdjgpfcNs8voaLrjwNdDwMqJw8='} 2024-04-19T23:32:35.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-amz-request-id', 'tx0000030bc99b8192e4c79-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.610 DEBUG:rgw_multi.tests:version1 id=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:32:35.611 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:32:35.611 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:32:35.611 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.611 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:32:35.611 DEBUG:boto:Data: 2024-04-19T23:32:35.611 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.611 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.611 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.611 DEBUG:boto:Params: {} 2024-04-19T23:32:35.611 DEBUG:boto:Token: None 2024-04-19T23:32:35.612 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:32:35.612 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:E1ld63pPL1FdZAT5Em0NdLr2/yg= 2024-04-19T23:32:35.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E1ld63pPL1FdZAT5Em0NdLr2/yg='} 2024-04-19T23:32:35.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Mzxx04Zt5IYeJa.pG87lrmzpK61By78'), ('x-amz-request-id', 'tx000001016b8c8c7209741-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.622 DEBUG:rgw_multi.tests:version2 id=Mzxx04Zt5IYeJa.pG87lrmzpK61By78 2024-04-19T23:32:35.623 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:32:35.623 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:32:35.623 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=Mzxx04Zt5IYeJa.pG87lrmzpK61By78 2024-04-19T23:32:35.623 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=Mzxx04Zt5IYeJa.pG87lrmzpK61By78 2024-04-19T23:32:35.623 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.623 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=Mzxx04Zt5IYeJa.pG87lrmzpK61By78 2024-04-19T23:32:35.623 DEBUG:boto:Data: 2024-04-19T23:32:35.623 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.623 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.623 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.623 DEBUG:boto:Params: {} 2024-04-19T23:32:35.624 DEBUG:boto:Token: None 2024-04-19T23:32:35.624 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone1?versionId=Mzxx04Zt5IYeJa.pG87lrmzpK61By78 2024-04-19T23:32:35.624 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:52amSou8Vh8f7SFI0DLswgSEdNM= 2024-04-19T23:32:35.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:52amSou8Vh8f7SFI0DLswgSEdNM=', 'Content-Length': '0'} 2024-04-19T23:32:35.635 DEBUG:boto:Response headers: [('x-amz-version-id', 'Mzxx04Zt5IYeJa.pG87lrmzpK61By78'), ('x-amz-request-id', 'tx00000340b8b29738d2f2d-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.635 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:32:35.635 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:32:35.635 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.635 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:32:35.635 DEBUG:boto:Data: 2024-04-19T23:32:35.636 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.636 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.636 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.636 DEBUG:boto:Params: {} 2024-04-19T23:32:35.636 DEBUG:boto:Token: None 2024-04-19T23:32:35.636 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:32:35.636 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HjFfXZzD7WXcBk7zhNHwfTt70dE= 2024-04-19T23:32:35.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HjFfXZzD7WXcBk7zhNHwfTt70dE='} 2024-04-19T23:32:35.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6Rht93G2eDdEpUusgk3CpyvrmgjS5qo'), ('x-amz-request-id', 'tx0000053ecbbb1667b727d-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.683 DEBUG:rgw_multi.tests:version3 id=6Rht93G2eDdEpUusgk3CpyvrmgjS5qo 2024-04-19T23:32:35.683 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:32:35.683 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:32:35.683 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=6Rht93G2eDdEpUusgk3CpyvrmgjS5qo 2024-04-19T23:32:35.683 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=6Rht93G2eDdEpUusgk3CpyvrmgjS5qo 2024-04-19T23:32:35.683 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.683 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=6Rht93G2eDdEpUusgk3CpyvrmgjS5qo 2024-04-19T23:32:35.683 DEBUG:boto:Data: 2024-04-19T23:32:35.684 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.684 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.684 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.684 DEBUG:boto:Params: {} 2024-04-19T23:32:35.684 DEBUG:boto:Token: None 2024-04-19T23:32:35.684 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone1?versionId=6Rht93G2eDdEpUusgk3CpyvrmgjS5qo 2024-04-19T23:32:35.684 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/cEdx+gdrnb26u5+cw4YjOxSBLI= 2024-04-19T23:32:35.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/cEdx+gdrnb26u5+cw4YjOxSBLI=', 'Content-Length': '0'} 2024-04-19T23:32:35.696 DEBUG:boto:Response headers: [('x-amz-version-id', '6Rht93G2eDdEpUusgk3CpyvrmgjS5qo'), ('x-amz-request-id', 'tx00000ec61728e31cd8ba5-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.696 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:32:35.696 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:32:35.696 DEBUG:boto:Method: HEAD 2024-04-19T23:32:35.696 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:32:35.696 DEBUG:boto:Data: 2024-04-19T23:32:35.696 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.696 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:35.697 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.697 DEBUG:boto:Params: {} 2024-04-19T23:32:35.697 DEBUG:boto:Token: None 2024-04-19T23:32:35.697 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/ 2024-04-19T23:32:35.697 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b5QIE/iBiTig2L1AT4YR5OXqAaA= 2024-04-19T23:32:35.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b5QIE/iBiTig2L1AT4YR5OXqAaA=', 'Content-Length': '0'} 2024-04-19T23:32:35.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da15bd672a38269b-006622ff13-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.700 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:32:35.701 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:32:35.701 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.701 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:32:35.701 DEBUG:boto:Data: 2024-04-19T23:32:35.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.701 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:35.701 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.701 DEBUG:boto:Params: {} 2024-04-19T23:32:35.702 DEBUG:boto:Token: None 2024-04-19T23:32:35.702 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:32:35.702 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qyRprMaW7UPpMyEeEpFPR8hIYgI= 2024-04-19T23:32:35.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qyRprMaW7UPpMyEeEpFPR8hIYgI='} 2024-04-19T23:32:35.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-amz-request-id', 'tx0000036309e668b1cf802-006622ff13-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.715 DEBUG:rgw_multi.tests:version1 id=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:32:35.716 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:32:35.716 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:32:35.716 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.716 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:32:35.716 DEBUG:boto:Data: 2024-04-19T23:32:35.716 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.716 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:35.716 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.717 DEBUG:boto:Params: {} 2024-04-19T23:32:35.717 DEBUG:boto:Token: None 2024-04-19T23:32:35.717 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:32:35.717 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QnVqlNuoreRIdXQNQFw0aVU3JZw= 2024-04-19T23:32:35.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QnVqlNuoreRIdXQNQFw0aVU3JZw='} 2024-04-19T23:32:35.729 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'eOq3tUCY0-htsrV8SUWscR.SAbS.n2K'), ('x-amz-request-id', 'tx00000408ee79cc9d5518d-006622ff13-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.729 DEBUG:rgw_multi.tests:version2 id=eOq3tUCY0-htsrV8SUWscR.SAbS.n2K 2024-04-19T23:32:35.729 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:32:35.730 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:32:35.730 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=eOq3tUCY0-htsrV8SUWscR.SAbS.n2K 2024-04-19T23:32:35.730 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=eOq3tUCY0-htsrV8SUWscR.SAbS.n2K 2024-04-19T23:32:35.730 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.730 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=eOq3tUCY0-htsrV8SUWscR.SAbS.n2K 2024-04-19T23:32:35.730 DEBUG:boto:Data: 2024-04-19T23:32:35.730 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.730 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:35.730 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.730 DEBUG:boto:Params: {} 2024-04-19T23:32:35.731 DEBUG:boto:Token: None 2024-04-19T23:32:35.731 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone2?versionId=eOq3tUCY0-htsrV8SUWscR.SAbS.n2K 2024-04-19T23:32:35.731 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xaXjCJ4trW5iaoR6LNP1UnnGpks= 2024-04-19T23:32:35.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xaXjCJ4trW5iaoR6LNP1UnnGpks=', 'Content-Length': '0'} 2024-04-19T23:32:35.744 DEBUG:boto:Response headers: [('x-amz-version-id', 'eOq3tUCY0-htsrV8SUWscR.SAbS.n2K'), ('x-amz-request-id', 'tx0000063160012e6672ab9-006622ff13-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.744 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:32:35.744 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:32:35.744 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.745 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:32:35.745 DEBUG:boto:Data: 2024-04-19T23:32:35.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.745 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:35.745 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.745 DEBUG:boto:Params: {} 2024-04-19T23:32:35.745 DEBUG:boto:Token: None 2024-04-19T23:32:35.745 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:32:35.746 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rnwcLgE3dYE84+0hFcyrHD/Dq08= 2024-04-19T23:32:35.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rnwcLgE3dYE84+0hFcyrHD/Dq08='} 2024-04-19T23:32:35.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-ZHhsD1aAWu9widYFmb92FPYlLoZXwv'), ('x-amz-request-id', 'tx0000064a6fbd38b23b51f-006622ff13-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.795 DEBUG:rgw_multi.tests:version3 id=-ZHhsD1aAWu9widYFmb92FPYlLoZXwv 2024-04-19T23:32:35.795 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:32:35.795 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:32:35.795 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=-ZHhsD1aAWu9widYFmb92FPYlLoZXwv 2024-04-19T23:32:35.795 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=-ZHhsD1aAWu9widYFmb92FPYlLoZXwv 2024-04-19T23:32:35.795 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.795 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=-ZHhsD1aAWu9widYFmb92FPYlLoZXwv 2024-04-19T23:32:35.795 DEBUG:boto:Data: 2024-04-19T23:32:35.796 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.796 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:35.796 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.796 DEBUG:boto:Params: {} 2024-04-19T23:32:35.796 DEBUG:boto:Token: None 2024-04-19T23:32:35.796 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone2?versionId=-ZHhsD1aAWu9widYFmb92FPYlLoZXwv 2024-04-19T23:32:35.796 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZMg1ZQ12mQD8RWrBvPJTiRPkKYY= 2024-04-19T23:32:35.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZMg1ZQ12mQD8RWrBvPJTiRPkKYY=', 'Content-Length': '0'} 2024-04-19T23:32:35.807 DEBUG:boto:Response headers: [('x-amz-version-id', '-ZHhsD1aAWu9widYFmb92FPYlLoZXwv'), ('x-amz-request-id', 'tx000009d39b4e27a409dc9-006622ff13-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.808 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:32:35.808 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:32:35.808 DEBUG:boto:Method: HEAD 2024-04-19T23:32:35.808 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:32:35.808 DEBUG:boto:Data: 2024-04-19T23:32:35.808 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.808 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:35.808 DEBUG:boto:Port: 8001 2024-04-19T23:32:35.808 DEBUG:boto:Params: {} 2024-04-19T23:32:35.809 DEBUG:boto:Token: None 2024-04-19T23:32:35.809 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/ 2024-04-19T23:32:35.809 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b5QIE/iBiTig2L1AT4YR5OXqAaA= 2024-04-19T23:32:35.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b5QIE/iBiTig2L1AT4YR5OXqAaA=', 'Content-Length': '0'} 2024-04-19T23:32:35.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000448d0e687f26f09e-006622ff13-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.811 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:32:35.812 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:32:35.812 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.812 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:32:35.812 DEBUG:boto:Data: 2024-04-19T23:32:35.812 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.812 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:35.812 DEBUG:boto:Port: 8001 2024-04-19T23:32:35.812 DEBUG:boto:Params: {} 2024-04-19T23:32:35.812 DEBUG:boto:Token: None 2024-04-19T23:32:35.812 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:32:35.812 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7FwsnyEULBZ+reo1J8TFR4jmlNk= 2024-04-19T23:32:35.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7FwsnyEULBZ+reo1J8TFR4jmlNk='} 2024-04-19T23:32:35.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-amz-request-id', 'tx00000fe31b366692f2600-006622ff13-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.824 DEBUG:rgw_multi.tests:version1 id=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:32:35.824 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:32:35.824 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:32:35.824 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.824 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:32:35.824 DEBUG:boto:Data: 2024-04-19T23:32:35.824 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.825 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:35.825 DEBUG:boto:Port: 8001 2024-04-19T23:32:35.825 DEBUG:boto:Params: {} 2024-04-19T23:32:35.825 DEBUG:boto:Token: None 2024-04-19T23:32:35.825 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:32:35.825 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L7yYTdy61AGCP+0wU5cWC+DQx/c= 2024-04-19T23:32:35.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L7yYTdy61AGCP+0wU5cWC+DQx/c='} 2024-04-19T23:32:35.836 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'c4fRZxSxepGW8O6q9CXWemwnUdFeqt1'), ('x-amz-request-id', 'tx00000afbad41be68e6589-006622ff13-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.837 DEBUG:rgw_multi.tests:version2 id=c4fRZxSxepGW8O6q9CXWemwnUdFeqt1 2024-04-19T23:32:35.837 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:32:35.837 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:32:35.837 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=c4fRZxSxepGW8O6q9CXWemwnUdFeqt1 2024-04-19T23:32:35.837 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=c4fRZxSxepGW8O6q9CXWemwnUdFeqt1 2024-04-19T23:32:35.837 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.837 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=c4fRZxSxepGW8O6q9CXWemwnUdFeqt1 2024-04-19T23:32:35.837 DEBUG:boto:Data: 2024-04-19T23:32:35.837 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.837 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:35.837 DEBUG:boto:Port: 8001 2024-04-19T23:32:35.838 DEBUG:boto:Params: {} 2024-04-19T23:32:35.838 DEBUG:boto:Token: None 2024-04-19T23:32:35.838 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone3?versionId=c4fRZxSxepGW8O6q9CXWemwnUdFeqt1 2024-04-19T23:32:35.838 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x6R6WdFKFtkCy4mkSZbl6u7pHD0= 2024-04-19T23:32:35.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x6R6WdFKFtkCy4mkSZbl6u7pHD0=', 'Content-Length': '0'} 2024-04-19T23:32:35.849 DEBUG:boto:Response headers: [('x-amz-version-id', 'c4fRZxSxepGW8O6q9CXWemwnUdFeqt1'), ('x-amz-request-id', 'tx00000913ec8981cce2476-006622ff13-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.849 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:32:35.849 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:32:35.850 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.850 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:32:35.850 DEBUG:boto:Data: 2024-04-19T23:32:35.850 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.850 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:35.850 DEBUG:boto:Port: 8001 2024-04-19T23:32:35.850 DEBUG:boto:Params: {} 2024-04-19T23:32:35.850 DEBUG:boto:Token: None 2024-04-19T23:32:35.850 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:32:35.851 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3KO0uq1iinQpAyHgmN66zp/AmlA= 2024-04-19T23:32:35.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3KO0uq1iinQpAyHgmN66zp/AmlA='} 2024-04-19T23:32:35.895 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'yWf7BT3HUrBEpjRKnsDOQ1Kt2LgKD.Y'), ('x-amz-request-id', 'tx00000fc41a1fb00e01870-006622ff13-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.895 DEBUG:rgw_multi.tests:version3 id=yWf7BT3HUrBEpjRKnsDOQ1Kt2LgKD.Y 2024-04-19T23:32:35.895 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:32:35.895 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:32:35.895 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=yWf7BT3HUrBEpjRKnsDOQ1Kt2LgKD.Y 2024-04-19T23:32:35.895 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=yWf7BT3HUrBEpjRKnsDOQ1Kt2LgKD.Y 2024-04-19T23:32:35.895 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.896 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=yWf7BT3HUrBEpjRKnsDOQ1Kt2LgKD.Y 2024-04-19T23:32:35.896 DEBUG:boto:Data: 2024-04-19T23:32:35.896 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.896 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:35.896 DEBUG:boto:Port: 8001 2024-04-19T23:32:35.896 DEBUG:boto:Params: {} 2024-04-19T23:32:35.896 DEBUG:boto:Token: None 2024-04-19T23:32:35.896 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-20/obj-test-zone3?versionId=yWf7BT3HUrBEpjRKnsDOQ1Kt2LgKD.Y 2024-04-19T23:32:35.896 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o6/4D1ALvJquYJWdCP6WcJCl4V0= 2024-04-19T23:32:35.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o6/4D1ALvJquYJWdCP6WcJCl4V0=', 'Content-Length': '0'} 2024-04-19T23:32:35.908 DEBUG:boto:Response headers: [('x-amz-version-id', 'yWf7BT3HUrBEpjRKnsDOQ1Kt2LgKD.Y'), ('x-amz-request-id', 'tx000001a7893e95a8b9acc-006622ff13-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.908 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:32:35.908 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:32:35.908 DEBUG:boto:Method: HEAD 2024-04-19T23:32:35.908 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:32:35.908 DEBUG:boto:Data: 2024-04-19T23:32:35.908 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.908 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.909 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.909 DEBUG:boto:Params: {} 2024-04-19T23:32:35.909 DEBUG:boto:Token: None 2024-04-19T23:32:35.909 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:32:35 GMT /aktlji-21/ 2024-04-19T23:32:35.909 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nHsf/GYH0A73pIcIyEI5L7TK8VE= 2024-04-19T23:32:35.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nHsf/GYH0A73pIcIyEI5L7TK8VE=', 'Content-Length': '0'} 2024-04-19T23:32:35.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000164f3dcb7f8757db-006622ff13-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.912 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:32:35.912 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:32:35.912 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.912 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:32:35.913 DEBUG:boto:Data: 2024-04-19T23:32:35.913 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.913 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.913 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.913 DEBUG:boto:Params: {} 2024-04-19T23:32:35.913 DEBUG:boto:Token: None 2024-04-19T23:32:35.913 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:32:35.913 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nnyu/+Se9aNC4SPkZbZbZoW5x9M= 2024-04-19T23:32:35.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nnyu/+Se9aNC4SPkZbZbZoW5x9M='} 2024-04-19T23:32:35.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-amz-request-id', 'tx00000592731c9d3d0246f-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.927 DEBUG:rgw_multi.tests:version1 id=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:32:35.927 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:32:35.927 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:32:35.928 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.928 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:32:35.928 DEBUG:boto:Data: 2024-04-19T23:32:35.928 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.928 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.928 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.928 DEBUG:boto:Params: {} 2024-04-19T23:32:35.929 DEBUG:boto:Token: None 2024-04-19T23:32:35.929 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:32:35.929 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0EY6Ld2Ic3MoYgyUdguPXNMY70Q= 2024-04-19T23:32:35.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0EY6Ld2Ic3MoYgyUdguPXNMY70Q='} 2024-04-19T23:32:35.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'mDXJyBysxJjRHy6LPA2dpZPc8TPS6K5'), ('x-amz-request-id', 'tx00000ae70736cf794e4a0-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.979 DEBUG:rgw_multi.tests:version2 id=mDXJyBysxJjRHy6LPA2dpZPc8TPS6K5 2024-04-19T23:32:35.979 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:32:35.979 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:32:35.979 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=mDXJyBysxJjRHy6LPA2dpZPc8TPS6K5 2024-04-19T23:32:35.979 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=mDXJyBysxJjRHy6LPA2dpZPc8TPS6K5 2024-04-19T23:32:35.980 DEBUG:boto:Method: DELETE 2024-04-19T23:32:35.980 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=mDXJyBysxJjRHy6LPA2dpZPc8TPS6K5 2024-04-19T23:32:35.980 DEBUG:boto:Data: 2024-04-19T23:32:35.980 DEBUG:boto:Headers: {} 2024-04-19T23:32:35.980 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.980 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.980 DEBUG:boto:Params: {} 2024-04-19T23:32:35.980 DEBUG:boto:Token: None 2024-04-19T23:32:35.981 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:35 GMT /aktlji-21/obj-test-zone1?versionId=mDXJyBysxJjRHy6LPA2dpZPc8TPS6K5 2024-04-19T23:32:35.981 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:POerqxquWBVuU2NeAiArM3Xg04M= 2024-04-19T23:32:35.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:POerqxquWBVuU2NeAiArM3Xg04M=', 'Content-Length': '0'} 2024-04-19T23:32:35.992 DEBUG:boto:Response headers: [('x-amz-version-id', 'mDXJyBysxJjRHy6LPA2dpZPc8TPS6K5'), ('x-amz-request-id', 'tx0000027ed7ffe1326a5ac-006622ff13-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:35.993 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:32:35.993 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:32:35.993 DEBUG:boto:Method: PUT 2024-04-19T23:32:35.993 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:32:35.993 DEBUG:boto:Data: 2024-04-19T23:32:35.993 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:35.993 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:35.993 DEBUG:boto:Port: 8000 2024-04-19T23:32:35.994 DEBUG:boto:Params: {} 2024-04-19T23:32:35.994 DEBUG:boto:Token: None 2024-04-19T23:32:35.994 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 19 Apr 2024 23:32:35 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:32:35.994 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sHi2quiDxEsP1qWqcrXvpDajOZA= 2024-04-19T23:32:35.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sHi2quiDxEsP1qWqcrXvpDajOZA='} 2024-04-19T23:32:36.005 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xkqqAOSLDZ4Y0x4g48eto4pC.XRHDjU'), ('x-amz-request-id', 'tx0000004102986f1ec5bd7-006622ff14-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.005 DEBUG:rgw_multi.tests:version3 id=xkqqAOSLDZ4Y0x4g48eto4pC.XRHDjU 2024-04-19T23:32:36.005 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:32:36.005 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:32:36.005 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=xkqqAOSLDZ4Y0x4g48eto4pC.XRHDjU 2024-04-19T23:32:36.005 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=xkqqAOSLDZ4Y0x4g48eto4pC.XRHDjU 2024-04-19T23:32:36.006 DEBUG:boto:Method: DELETE 2024-04-19T23:32:36.006 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=xkqqAOSLDZ4Y0x4g48eto4pC.XRHDjU 2024-04-19T23:32:36.006 DEBUG:boto:Data: 2024-04-19T23:32:36.006 DEBUG:boto:Headers: {} 2024-04-19T23:32:36.006 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:32:36.006 DEBUG:boto:Port: 8000 2024-04-19T23:32:36.006 DEBUG:boto:Params: {} 2024-04-19T23:32:36.006 DEBUG:boto:Token: None 2024-04-19T23:32:36.006 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/obj-test-zone1?versionId=xkqqAOSLDZ4Y0x4g48eto4pC.XRHDjU 2024-04-19T23:32:36.007 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NKO7mjmsM48u7uHPsS+QD1aIGaI= 2024-04-19T23:32:36.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NKO7mjmsM48u7uHPsS+QD1aIGaI=', 'Content-Length': '0'} 2024-04-19T23:32:36.017 DEBUG:boto:Response headers: [('x-amz-version-id', 'xkqqAOSLDZ4Y0x4g48eto4pC.XRHDjU'), ('x-amz-request-id', 'tx0000081f8d4c166ce93a5-006622ff14-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.018 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:32:36.018 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:32:36.018 DEBUG:boto:Method: HEAD 2024-04-19T23:32:36.018 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:32:36.018 DEBUG:boto:Data: 2024-04-19T23:32:36.018 DEBUG:boto:Headers: {} 2024-04-19T23:32:36.018 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:36.018 DEBUG:boto:Port: 8000 2024-04-19T23:32:36.018 DEBUG:boto:Params: {} 2024-04-19T23:32:36.019 DEBUG:boto:Token: None 2024-04-19T23:32:36.019 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/ 2024-04-19T23:32:36.019 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qy5rVRwXSJZwHW+EJ27H0eRk3Ew= 2024-04-19T23:32:36.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qy5rVRwXSJZwHW+EJ27H0eRk3Ew=', 'Content-Length': '0'} 2024-04-19T23:32:36.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008bd78ffe3b00f467-006622ff14-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.022 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:32:36.022 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:32:36.022 DEBUG:boto:Method: PUT 2024-04-19T23:32:36.023 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:32:36.023 DEBUG:boto:Data: 2024-04-19T23:32:36.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:36.023 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:36.023 DEBUG:boto:Port: 8000 2024-04-19T23:32:36.023 DEBUG:boto:Params: {} 2024-04-19T23:32:36.023 DEBUG:boto:Token: None 2024-04-19T23:32:36.023 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:32:36.023 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hs5L9fq/MVr50IKi21TJ3r9Nwus= 2024-04-19T23:32:36.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hs5L9fq/MVr50IKi21TJ3r9Nwus='} 2024-04-19T23:32:36.037 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-amz-request-id', 'tx0000040837dacef52e221-006622ff14-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.038 DEBUG:rgw_multi.tests:version1 id=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:32:36.038 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:32:36.038 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:32:36.038 DEBUG:boto:Method: PUT 2024-04-19T23:32:36.038 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:32:36.038 DEBUG:boto:Data: 2024-04-19T23:32:36.038 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:36.038 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:36.039 DEBUG:boto:Port: 8000 2024-04-19T23:32:36.039 DEBUG:boto:Params: {} 2024-04-19T23:32:36.039 DEBUG:boto:Token: None 2024-04-19T23:32:36.039 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:32:36.039 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ubOHDDPilGu9uvUA5vEDuWfcvmQ= 2024-04-19T23:32:36.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ubOHDDPilGu9uvUA5vEDuWfcvmQ='} 2024-04-19T23:32:36.053 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'g-mb2Y.73f7bZ2Z6eLChJt7jU-3mk2w'), ('x-amz-request-id', 'tx00000e0129fd14d2ddd63-006622ff14-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.054 DEBUG:rgw_multi.tests:version2 id=g-mb2Y.73f7bZ2Z6eLChJt7jU-3mk2w 2024-04-19T23:32:36.054 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:32:36.054 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:32:36.054 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=g-mb2Y.73f7bZ2Z6eLChJt7jU-3mk2w 2024-04-19T23:32:36.054 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=g-mb2Y.73f7bZ2Z6eLChJt7jU-3mk2w 2024-04-19T23:32:36.054 DEBUG:boto:Method: DELETE 2024-04-19T23:32:36.054 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=g-mb2Y.73f7bZ2Z6eLChJt7jU-3mk2w 2024-04-19T23:32:36.054 DEBUG:boto:Data: 2024-04-19T23:32:36.054 DEBUG:boto:Headers: {} 2024-04-19T23:32:36.054 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:36.055 DEBUG:boto:Port: 8000 2024-04-19T23:32:36.055 DEBUG:boto:Params: {} 2024-04-19T23:32:36.055 DEBUG:boto:Token: None 2024-04-19T23:32:36.055 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/obj-test-zone2?versionId=g-mb2Y.73f7bZ2Z6eLChJt7jU-3mk2w 2024-04-19T23:32:36.055 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sDYCvDvtVPFeQnftIrhaNvmIUyw= 2024-04-19T23:32:36.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sDYCvDvtVPFeQnftIrhaNvmIUyw=', 'Content-Length': '0'} 2024-04-19T23:32:36.067 DEBUG:boto:Response headers: [('x-amz-version-id', 'g-mb2Y.73f7bZ2Z6eLChJt7jU-3mk2w'), ('x-amz-request-id', 'tx0000075359ec951c2c337-006622ff14-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.067 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:32:36.067 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:32:36.068 DEBUG:boto:Method: PUT 2024-04-19T23:32:36.068 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:32:36.068 DEBUG:boto:Data: 2024-04-19T23:32:36.068 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:36.068 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:36.068 DEBUG:boto:Port: 8000 2024-04-19T23:32:36.068 DEBUG:boto:Params: {} 2024-04-19T23:32:36.068 DEBUG:boto:Token: None 2024-04-19T23:32:36.068 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:32:36.069 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YHkq33bMszt1vpQtkUxLmKNsiM8= 2024-04-19T23:32:36.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YHkq33bMszt1vpQtkUxLmKNsiM8='} 2024-04-19T23:32:36.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'g-bQ2K.3xLKnRqWGh4Gf3Yk6MVQdiKi'), ('x-amz-request-id', 'tx00000f3755ccdee6e0968-006622ff14-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.111 DEBUG:rgw_multi.tests:version3 id=g-bQ2K.3xLKnRqWGh4Gf3Yk6MVQdiKi 2024-04-19T23:32:36.111 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:32:36.111 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:32:36.111 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=g-bQ2K.3xLKnRqWGh4Gf3Yk6MVQdiKi 2024-04-19T23:32:36.111 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=g-bQ2K.3xLKnRqWGh4Gf3Yk6MVQdiKi 2024-04-19T23:32:36.111 DEBUG:boto:Method: DELETE 2024-04-19T23:32:36.111 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=g-bQ2K.3xLKnRqWGh4Gf3Yk6MVQdiKi 2024-04-19T23:32:36.111 DEBUG:boto:Data: 2024-04-19T23:32:36.112 DEBUG:boto:Headers: {} 2024-04-19T23:32:36.112 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:32:36.112 DEBUG:boto:Port: 8000 2024-04-19T23:32:36.112 DEBUG:boto:Params: {} 2024-04-19T23:32:36.112 DEBUG:boto:Token: None 2024-04-19T23:32:36.112 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/obj-test-zone2?versionId=g-bQ2K.3xLKnRqWGh4Gf3Yk6MVQdiKi 2024-04-19T23:32:36.113 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HnojrmUqNsoSUEhVmd3HvdF53gM= 2024-04-19T23:32:36.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HnojrmUqNsoSUEhVmd3HvdF53gM=', 'Content-Length': '0'} 2024-04-19T23:32:36.124 DEBUG:boto:Response headers: [('x-amz-version-id', 'g-bQ2K.3xLKnRqWGh4Gf3Yk6MVQdiKi'), ('x-amz-request-id', 'tx00000d62db4954ca21fe5-006622ff14-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.124 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:32:36.124 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:32:36.125 DEBUG:boto:Method: HEAD 2024-04-19T23:32:36.125 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:32:36.125 DEBUG:boto:Data: 2024-04-19T23:32:36.125 DEBUG:boto:Headers: {} 2024-04-19T23:32:36.125 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:36.125 DEBUG:boto:Port: 8001 2024-04-19T23:32:36.125 DEBUG:boto:Params: {} 2024-04-19T23:32:36.125 DEBUG:boto:Token: None 2024-04-19T23:32:36.125 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/ 2024-04-19T23:32:36.125 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qy5rVRwXSJZwHW+EJ27H0eRk3Ew= 2024-04-19T23:32:36.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qy5rVRwXSJZwHW+EJ27H0eRk3Ew=', 'Content-Length': '0'} 2024-04-19T23:32:36.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b2d7fc503890b0f-006622ff14-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.128 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:32:36.128 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:32:36.129 DEBUG:boto:Method: PUT 2024-04-19T23:32:36.129 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:32:36.129 DEBUG:boto:Data: 2024-04-19T23:32:36.129 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:36.129 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:36.129 DEBUG:boto:Port: 8001 2024-04-19T23:32:36.129 DEBUG:boto:Params: {} 2024-04-19T23:32:36.129 DEBUG:boto:Token: None 2024-04-19T23:32:36.129 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:32:36.129 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tNAqfGc898A+KTlDSA7Jmc99SzA= 2024-04-19T23:32:36.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tNAqfGc898A+KTlDSA7Jmc99SzA='} 2024-04-19T23:32:36.140 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-amz-request-id', 'tx0000017eb534bd5a65227-006622ff14-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.141 DEBUG:rgw_multi.tests:version1 id=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:32:36.141 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:32:36.141 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:32:36.141 DEBUG:boto:Method: PUT 2024-04-19T23:32:36.141 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:32:36.141 DEBUG:boto:Data: 2024-04-19T23:32:36.141 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:36.141 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:36.142 DEBUG:boto:Port: 8001 2024-04-19T23:32:36.142 DEBUG:boto:Params: {} 2024-04-19T23:32:36.142 DEBUG:boto:Token: None 2024-04-19T23:32:36.142 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:32:36.142 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VFWRArjAMdEw75AFKzVChcdJktM= 2024-04-19T23:32:36.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VFWRArjAMdEw75AFKzVChcdJktM='} 2024-04-19T23:32:36.186 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'b-Wap-66d6lmpjoCiBYH2tzeTKq4oaW'), ('x-amz-request-id', 'tx00000e5b0c24dd88990a3-006622ff14-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.187 DEBUG:rgw_multi.tests:version2 id=b-Wap-66d6lmpjoCiBYH2tzeTKq4oaW 2024-04-19T23:32:36.187 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:32:36.187 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:32:36.187 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=b-Wap-66d6lmpjoCiBYH2tzeTKq4oaW 2024-04-19T23:32:36.187 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=b-Wap-66d6lmpjoCiBYH2tzeTKq4oaW 2024-04-19T23:32:36.187 DEBUG:boto:Method: DELETE 2024-04-19T23:32:36.187 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=b-Wap-66d6lmpjoCiBYH2tzeTKq4oaW 2024-04-19T23:32:36.187 DEBUG:boto:Data: 2024-04-19T23:32:36.188 DEBUG:boto:Headers: {} 2024-04-19T23:32:36.188 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:36.188 DEBUG:boto:Port: 8001 2024-04-19T23:32:36.188 DEBUG:boto:Params: {} 2024-04-19T23:32:36.188 DEBUG:boto:Token: None 2024-04-19T23:32:36.188 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/obj-test-zone3?versionId=b-Wap-66d6lmpjoCiBYH2tzeTKq4oaW 2024-04-19T23:32:36.188 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CyIs1+7DLOiPnPi50c1iXnOC8ps= 2024-04-19T23:32:36.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CyIs1+7DLOiPnPi50c1iXnOC8ps=', 'Content-Length': '0'} 2024-04-19T23:32:36.199 DEBUG:boto:Response headers: [('x-amz-version-id', 'b-Wap-66d6lmpjoCiBYH2tzeTKq4oaW'), ('x-amz-request-id', 'tx00000ca4caeb8b7964ad3-006622ff14-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.199 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:32:36.200 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:32:36.200 DEBUG:boto:Method: PUT 2024-04-19T23:32:36.200 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:32:36.200 DEBUG:boto:Data: 2024-04-19T23:32:36.200 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:32:36.200 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:36.200 DEBUG:boto:Port: 8001 2024-04-19T23:32:36.200 DEBUG:boto:Params: {} 2024-04-19T23:32:36.200 DEBUG:boto:Token: None 2024-04-19T23:32:36.201 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:32:36.201 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LCJY00aVMpf9Izlkq9SpAJ/IshM= 2024-04-19T23:32:36.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LCJY00aVMpf9Izlkq9SpAJ/IshM='} 2024-04-19T23:32:36.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'JcY2SZrza3yC-.E1m-kidxYAoIOi8lr'), ('x-amz-request-id', 'tx000004b8683e92a63b06a-006622ff14-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.212 DEBUG:rgw_multi.tests:version3 id=JcY2SZrza3yC-.E1m-kidxYAoIOi8lr 2024-04-19T23:32:36.212 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:32:36.212 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:32:36.212 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=JcY2SZrza3yC-.E1m-kidxYAoIOi8lr 2024-04-19T23:32:36.212 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=JcY2SZrza3yC-.E1m-kidxYAoIOi8lr 2024-04-19T23:32:36.212 DEBUG:boto:Method: DELETE 2024-04-19T23:32:36.213 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=JcY2SZrza3yC-.E1m-kidxYAoIOi8lr 2024-04-19T23:32:36.213 DEBUG:boto:Data: 2024-04-19T23:32:36.213 DEBUG:boto:Headers: {} 2024-04-19T23:32:36.213 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:32:36.213 DEBUG:boto:Port: 8001 2024-04-19T23:32:36.213 DEBUG:boto:Params: {} 2024-04-19T23:32:36.213 DEBUG:boto:Token: None 2024-04-19T23:32:36.213 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:32:36 GMT /aktlji-21/obj-test-zone3?versionId=JcY2SZrza3yC-.E1m-kidxYAoIOi8lr 2024-04-19T23:32:36.213 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OsNs5lWRKz3kN4FJqSO1ACAv1JM= 2024-04-19T23:32:36.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:32:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OsNs5lWRKz3kN4FJqSO1ACAv1JM=', 'Content-Length': '0'} 2024-04-19T23:32:36.225 DEBUG:boto:Response headers: [('x-amz-version-id', 'JcY2SZrza3yC-.E1m-kidxYAoIOi8lr'), ('x-amz-request-id', 'tx0000022a19267b09fef3c-006622ff14-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:32:36.225 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-19 2024-04-19T23:32:36.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:36.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:36.225 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:32:36.292 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:32:36.293 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:32:36.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:32:36.340+0000 7f59a1e11780 1 waiting for incremental sync to catch up: 2024-04-19T23:32:36.335 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , 00000000004.35.10, , , , , ] 2024-04-19T23:32:36.336 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000010.26.6, , , , 00000000004.35.10, , , , , ] 2024-04-19T23:32:41.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:32:41.341+0000 7f59a1e11780 1 bucket sync caught up with source: 2024-04-19T23:32:41.337 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000010.26.6, , , , 00000000004.35.10, , , , , ] 2024-04-19T23:32:41.337 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000010.26.6, , , , 00000000004.35.10, , , , , ] 2024-04-19T23:32:41.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:32:41.341+0000 7f59a1e11780 0 bucket checkpoint complete 2024-04-19T23:32:41.354 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-19 2024-04-19T23:32:41.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:41.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:41.354 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:32:41.422 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:32:41.422 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:32:41.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:41.470+0000 7fa211029780 1 bucket sync caught up with source: 2024-04-19T23:32:41.465 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000010.26.6, , , , 00000000004.35.10, , , , , ] 2024-04-19T23:32:41.465 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000010.26.6, , , , 00000000004.35.10, , , , , ] 2024-04-19T23:32:41.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:41.470+0000 7fa211029780 0 bucket checkpoint complete 2024-04-19T23:32:41.478 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-19 2024-04-19T23:32:41.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:41.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:41.479 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:32:41.541 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:32:41.541 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:32:41.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:41.586+0000 7f4276775780 1 waiting for incremental sync to catch up: 2024-04-19T23:32:41.581 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000002.88.11, , , , , ] 2024-04-19T23:32:41.581 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.53.11, , 00000000010.70.6, , 00000000004.92.10, , , , , ] 2024-04-19T23:32:46.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:46.587+0000 7f4276775780 1 waiting for incremental sync to catch up: 2024-04-19T23:32:46.582 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000002.88.11, , , , , ] 2024-04-19T23:32:46.583 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.53.11, , 00000000010.70.6, , 00000000004.92.10, , , , , ] 2024-04-19T23:32:51.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:51.589+0000 7f4276775780 1 waiting for incremental sync to catch up: 2024-04-19T23:32:51.584 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000002.88.11, , , , , ] 2024-04-19T23:32:51.584 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.53.11, , 00000000010.70.6, , 00000000004.92.10, , , , , ] 2024-04-19T23:32:56.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:56.590+0000 7f4276775780 1 bucket sync caught up with source: 2024-04-19T23:32:56.585 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000002.53.11, , 00000000010.70.6, , 00000000004.92.10, , , , , ] 2024-04-19T23:32:56.585 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000002.53.11, , 00000000010.70.6, , 00000000004.92.10, , , , , ] 2024-04-19T23:32:56.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:56.590+0000 7f4276775780 0 bucket checkpoint complete 2024-04-19T23:32:56.602 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-19 2024-04-19T23:32:56.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:56.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:32:56.603 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:32:56.668 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:32:56.668 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:32:56.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:32:56.715+0000 7f7d4dc92780 1 waiting for incremental sync to catch up: 2024-04-19T23:32:56.710 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000002.53.11, , 00000000010.70.6, , 00000000004.92.10, , , , , ] 2024-04-19T23:32:56.710 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.55.10, , 00000000010.70.6, , 00000000006.116.11, , , , , ] 2024-04-19T23:33:01.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:01.716+0000 7f7d4dc92780 1 bucket sync caught up with source: 2024-04-19T23:33:01.712 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.55.10, , 00000000010.70.6, , 00000000007.118.10, , , , , ] 2024-04-19T23:33:01.712 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.55.10, , 00000000010.70.6, , 00000000006.116.11, , , , , ] 2024-04-19T23:33:01.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:01.716+0000 7f7d4dc92780 0 bucket checkpoint complete 2024-04-19T23:33:01.728 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-19 2024-04-19T23:33:01.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:01.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:01.729 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:01.796 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:01.796 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:01.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:01.842+0000 7fbbd49c6780 1 waiting for incremental sync to catch up: 2024-04-19T23:33:01.838 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.32.10, , 00000000002.32.11, , 00000000014.70.6, , , , , ] 2024-04-19T23:33:01.838 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.32.10, , 00000000003.34.10, , 00000000014.70.6, , , , , ] 2024-04-19T23:33:06.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:06.843+0000 7fbbd49c6780 1 waiting for incremental sync to catch up: 2024-04-19T23:33:06.839 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.32.10, , 00000000002.32.11, , 00000000014.70.6, , , , , ] 2024-04-19T23:33:06.839 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.32.10, , 00000000003.34.10, , 00000000014.70.6, , , , , ] 2024-04-19T23:33:11.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:11.844+0000 7fbbd49c6780 1 waiting for incremental sync to catch up: 2024-04-19T23:33:11.841 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.32.10, , 00000000002.32.11, , 00000000014.70.6, , , , , ] 2024-04-19T23:33:11.841 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.32.10, , 00000000003.34.10, , 00000000014.70.6, , , , , ] 2024-04-19T23:33:16.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:16.846+0000 7fbbd49c6780 1 bucket sync caught up with source: 2024-04-19T23:33:16.842 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.32.10, , 00000000003.34.10, , 00000000014.70.6, , , , , ] 2024-04-19T23:33:16.842 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.32.10, , 00000000003.34.10, , 00000000014.70.6, , , , , ] 2024-04-19T23:33:16.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:16.846+0000 7fbbd49c6780 0 bucket checkpoint complete 2024-04-19T23:33:16.859 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-19 2024-04-19T23:33:16.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:16.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:16.859 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:16.928 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:16.928 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:16.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:16.975+0000 7f03d9543780 1 bucket sync caught up with source: 2024-04-19T23:33:16.971 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000003.32.10, , 00000000003.34.10, , 00000000014.70.6, , , , , ] 2024-04-19T23:33:16.971 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000003.32.10, , 00000000003.34.10, , 00000000014.70.6, , , , , ] 2024-04-19T23:33:16.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:16.975+0000 7f03d9543780 0 bucket checkpoint complete 2024-04-19T23:33:16.985 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-19 zones={test-zone2, test-zone1} 2024-04-19T23:33:16.986 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:16.986 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:16.986 DEBUG:boto:Method: HEAD 2024-04-19T23:33:16.986 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:33:16.986 DEBUG:boto:Data: 2024-04-19T23:33:16.986 DEBUG:boto:Headers: {} 2024-04-19T23:33:16.986 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:16.986 DEBUG:boto:Port: 8000 2024-04-19T23:33:16.986 DEBUG:boto:Params: {} 2024-04-19T23:33:16.987 DEBUG:boto:Token: None 2024-04-19T23:33:16.987 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:16 GMT /aktlji-19/ 2024-04-19T23:33:16.987 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PnjeJjqq0UvqOIjh1dXG57n1vao= 2024-04-19T23:33:16.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PnjeJjqq0UvqOIjh1dXG57n1vao=', 'Content-Length': '0'} 2024-04-19T23:33:16.990 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae72b3749eff3d4d-006622ff3c-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:16.993 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:16.994 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:16.994 DEBUG:boto:Method: HEAD 2024-04-19T23:33:16.995 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:33:16.995 DEBUG:boto:Data: 2024-04-19T23:33:16.995 DEBUG:boto:Headers: {} 2024-04-19T23:33:16.995 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:16.995 DEBUG:boto:Port: 8000 2024-04-19T23:33:16.995 DEBUG:boto:Params: {} 2024-04-19T23:33:16.995 DEBUG:boto:Token: None 2024-04-19T23:33:16.995 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:16 GMT /aktlji-19/ 2024-04-19T23:33:16.995 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PnjeJjqq0UvqOIjh1dXG57n1vao= 2024-04-19T23:33:16.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PnjeJjqq0UvqOIjh1dXG57n1vao=', 'Content-Length': '0'} 2024-04-19T23:33:16.998 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ff275424e195c54-006622ff3d-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:16.998 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:33:16.998 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:16.998 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:16.998 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:16.998 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:16.999 DEBUG:boto:Method: GET 2024-04-19T23:33:16.999 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:33:16.999 DEBUG:boto:Data: 2024-04-19T23:33:16.999 DEBUG:boto:Headers: {} 2024-04-19T23:33:16.999 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:16.999 DEBUG:boto:Port: 8000 2024-04-19T23:33:16.999 DEBUG:boto:Params: {} 2024-04-19T23:33:16.999 DEBUG:boto:Token: None 2024-04-19T23:33:16.999 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:16 GMT /aktlji-19/?versions 2024-04-19T23:33:16.999 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jBecRTCYX/UrIa8UvA0gZiV3KMs= 2024-04-19T23:33:17.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jBecRTCYX/UrIa8UvA0gZiV3KMs=', 'Content-Length': '0'} 2024-04-19T23:33:17.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064734b1308c01bc5-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.002 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.003 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:17.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:17.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:17.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:17.003 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:33:17.003 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.003 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.004 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.004 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.004 DEBUG:boto:Method: GET 2024-04-19T23:33:17.004 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.004 DEBUG:boto:Data: 2024-04-19T23:33:17.004 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.004 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.004 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.004 DEBUG:boto:Params: {} 2024-04-19T23:33:17.004 DEBUG:boto:Token: None 2024-04-19T23:33:17.004 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/?versions 2024-04-19T23:33:17.005 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I= 2024-04-19T23:33:17.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I=', 'Content-Length': '0'} 2024-04-19T23:33:17.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc4a1ffe5bcf4964-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.007 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.008 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:17.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:17.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:17.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:17.008 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.008 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.009 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.009 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.009 DEBUG:boto:Method: GET 2024-04-19T23:33:17.009 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.009 DEBUG:boto:Data: 2024-04-19T23:33:17.009 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.009 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.009 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.009 DEBUG:boto:Params: {} 2024-04-19T23:33:17.009 DEBUG:boto:Token: None 2024-04-19T23:33:17.009 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/?versions 2024-04-19T23:33:17.010 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I= 2024-04-19T23:33:17.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I=', 'Content-Length': '0'} 2024-04-19T23:33:17.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cf893027315152d-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.012 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.013 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.013 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.013 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.013 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.013 DEBUG:boto:Method: GET 2024-04-19T23:33:17.013 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.013 DEBUG:boto:Data: 2024-04-19T23:33:17.014 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.014 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.014 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.014 DEBUG:boto:Params: {} 2024-04-19T23:33:17.014 DEBUG:boto:Token: None 2024-04-19T23:33:17.014 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/?versions 2024-04-19T23:33:17.014 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I= 2024-04-19T23:33:17.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I=', 'Content-Length': '0'} 2024-04-19T23:33:17.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002049d58e9da04b1b-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.017 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:17.018 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.018 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.018 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.018 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.018 DEBUG:boto:Method: GET 2024-04-19T23:33:17.018 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.018 DEBUG:boto:Data: 2024-04-19T23:33:17.019 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.019 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.019 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.019 DEBUG:boto:Params: {} 2024-04-19T23:33:17.019 DEBUG:boto:Token: None 2024-04-19T23:33:17.019 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.019 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc= 2024-04-19T23:33:17.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc=', 'Content-Length': '0'} 2024-04-19T23:33:17.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000702ab824c895e4fe-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.021 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.021 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.021 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.022 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.022 DEBUG:boto:Method: GET 2024-04-19T23:33:17.022 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.022 DEBUG:boto:Data: 2024-04-19T23:33:17.022 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.022 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.022 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.022 DEBUG:boto:Params: {} 2024-04-19T23:33:17.022 DEBUG:boto:Token: None 2024-04-19T23:33:17.023 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.023 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc= 2024-04-19T23:33:17.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc=', 'Content-Length': '0'} 2024-04-19T23:33:17.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007aa78a77b04b34be-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.025 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.025 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.025 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.025 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.025 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.025 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.026 DEBUG:boto:Data: 2024-04-19T23:33:17.026 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.026 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.026 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.026 DEBUG:boto:Params: {} 2024-04-19T23:33:17.026 DEBUG:boto:Token: None 2024-04-19T23:33:17.026 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.026 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FUSW5/bEqdXRBCZ/gWueq/dlxCE= 2024-04-19T23:33:17.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FUSW5/bEqdXRBCZ/gWueq/dlxCE=', 'Content-Length': '0'} 2024-04-19T23:33:17.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061dc537008641aff-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.028 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.028 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.029 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.029 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.029 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.029 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.029 DEBUG:boto:Data: 2024-04-19T23:33:17.029 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.029 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.029 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.029 DEBUG:boto:Params: {} 2024-04-19T23:33:17.030 DEBUG:boto:Token: None 2024-04-19T23:33:17.030 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.030 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FUSW5/bEqdXRBCZ/gWueq/dlxCE= 2024-04-19T23:33:17.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FUSW5/bEqdXRBCZ/gWueq/dlxCE=', 'Content-Length': '0'} 2024-04-19T23:33:17.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2ff2d24b1e07506-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:17.032 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.032 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.032 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.032 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.033 DEBUG:boto:Method: GET 2024-04-19T23:33:17.033 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.033 DEBUG:boto:Data: 2024-04-19T23:33:17.033 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.033 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.033 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.033 DEBUG:boto:Params: {} 2024-04-19T23:33:17.033 DEBUG:boto:Token: None 2024-04-19T23:33:17.033 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.034 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc= 2024-04-19T23:33:17.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc=', 'Content-Length': '0'} 2024-04-19T23:33:17.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab8710c832a3eef1-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.036 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.036 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.036 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.036 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.036 DEBUG:boto:Method: GET 2024-04-19T23:33:17.036 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.036 DEBUG:boto:Data: 2024-04-19T23:33:17.037 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.037 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.037 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.037 DEBUG:boto:Params: {} 2024-04-19T23:33:17.037 DEBUG:boto:Token: None 2024-04-19T23:33:17.037 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.037 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc= 2024-04-19T23:33:17.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc=', 'Content-Length': '0'} 2024-04-19T23:33:17.039 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce95b64853912ed9-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.040 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.040 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.040 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.040 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.040 DEBUG:boto:Method: GET 2024-04-19T23:33:17.040 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:33:17.040 DEBUG:boto:Data: 2024-04-19T23:33:17.040 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.040 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.041 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.041 DEBUG:boto:Params: {} 2024-04-19T23:33:17.041 DEBUG:boto:Token: None 2024-04-19T23:33:17.041 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.041 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6a9db907877d5f2-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.044 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.044 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.044 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.045 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.045 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.045 DEBUG:boto:Method: GET 2024-04-19T23:33:17.045 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:33:17.045 DEBUG:boto:Data: 2024-04-19T23:33:17.045 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.045 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.045 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.045 DEBUG:boto:Params: {} 2024-04-19T23:33:17.046 DEBUG:boto:Token: None 2024-04-19T23:33:17.046 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.046 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a25811b1ff061f-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.049 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.049 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.050 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.050 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.050 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:33:17.050 DEBUG:boto:Data: 2024-04-19T23:33:17.050 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.050 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.050 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.050 DEBUG:boto:Params: {} 2024-04-19T23:33:17.050 DEBUG:boto:Token: None 2024-04-19T23:33:17.051 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy 2024-04-19T23:33:17.051 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OFm94Ex3wBKAP2XQmthjh0NvEbk= 2024-04-19T23:33:17.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OFm94Ex3wBKAP2XQmthjh0NvEbk=', 'Content-Length': '0'} 2024-04-19T23:33:17.053 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001dd9f7bcaac7b3e7-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.053 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.053 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.053 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.053 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:33:17.054 DEBUG:boto:Data: 2024-04-19T23:33:17.054 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.054 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.054 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.054 DEBUG:boto:Params: {} 2024-04-19T23:33:17.054 DEBUG:boto:Token: None 2024-04-19T23:33:17.054 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy 2024-04-19T23:33:17.054 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OFm94Ex3wBKAP2XQmthjh0NvEbk= 2024-04-19T23:33:17.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OFm94Ex3wBKAP2XQmthjh0NvEbk=', 'Content-Length': '0'} 2024-04-19T23:33:17.057 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b62ba99600698f68-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:17.057 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.057 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.058 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.058 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.058 DEBUG:boto:Method: GET 2024-04-19T23:33:17.058 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.058 DEBUG:boto:Data: 2024-04-19T23:33:17.058 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.058 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.058 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.058 DEBUG:boto:Params: {} 2024-04-19T23:33:17.058 DEBUG:boto:Token: None 2024-04-19T23:33:17.059 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.059 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc= 2024-04-19T23:33:17.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc=', 'Content-Length': '0'} 2024-04-19T23:33:17.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007428766248485f5d-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.061 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.061 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.061 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.061 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.061 DEBUG:boto:Method: GET 2024-04-19T23:33:17.061 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.061 DEBUG:boto:Data: 2024-04-19T23:33:17.061 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.062 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.062 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.062 DEBUG:boto:Params: {} 2024-04-19T23:33:17.062 DEBUG:boto:Token: None 2024-04-19T23:33:17.062 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.062 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc= 2024-04-19T23:33:17.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc=', 'Content-Length': '0'} 2024-04-19T23:33:17.064 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009e26d5b196964938-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.064 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.064 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.065 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.065 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.065 DEBUG:boto:Method: GET 2024-04-19T23:33:17.065 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:33:17.065 DEBUG:boto:Data: 2024-04-19T23:33:17.065 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.065 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.065 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.065 DEBUG:boto:Params: {} 2024-04-19T23:33:17.065 DEBUG:boto:Token: None 2024-04-19T23:33:17.066 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.066 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a0db5798bf8681-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.068 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.069 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.069 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.069 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.069 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.069 DEBUG:boto:Method: GET 2024-04-19T23:33:17.069 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:33:17.069 DEBUG:boto:Data: 2024-04-19T23:33:17.069 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.069 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.069 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.069 DEBUG:boto:Params: {} 2024-04-19T23:33:17.070 DEBUG:boto:Token: None 2024-04-19T23:33:17.070 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.070 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098b4ced52a1d7265-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.072 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:17.073 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.073 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.073 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.073 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.073 DEBUG:boto:Method: GET 2024-04-19T23:33:17.073 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.073 DEBUG:boto:Data: 2024-04-19T23:33:17.073 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.073 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.074 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.074 DEBUG:boto:Params: {} 2024-04-19T23:33:17.074 DEBUG:boto:Token: None 2024-04-19T23:33:17.074 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.074 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA= 2024-04-19T23:33:17.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA=', 'Content-Length': '0'} 2024-04-19T23:33:17.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004aca4ff225654543-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.077 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.077 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.077 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.077 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.077 DEBUG:boto:Method: GET 2024-04-19T23:33:17.077 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.077 DEBUG:boto:Data: 2024-04-19T23:33:17.077 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.078 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.078 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.078 DEBUG:boto:Params: {} 2024-04-19T23:33:17.078 DEBUG:boto:Token: None 2024-04-19T23:33:17.079 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.079 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA= 2024-04-19T23:33:17.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA=', 'Content-Length': '0'} 2024-04-19T23:33:17.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f418590704bebc1-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.082 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.082 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.082 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.082 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.083 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.083 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.083 DEBUG:boto:Data: 2024-04-19T23:33:17.083 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.083 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.083 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.084 DEBUG:boto:Params: {} 2024-04-19T23:33:17.084 DEBUG:boto:Token: None 2024-04-19T23:33:17.084 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ARZGI+JlLpYMqtvsALMS5Fjdq1w= 2024-04-19T23:33:17.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ARZGI+JlLpYMqtvsALMS5Fjdq1w=', 'Content-Length': '0'} 2024-04-19T23:33:17.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a3346968ff33097-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.087 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.087 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.087 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.088 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.088 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.088 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.088 DEBUG:boto:Data: 2024-04-19T23:33:17.088 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.088 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.089 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.089 DEBUG:boto:Params: {} 2024-04-19T23:33:17.089 DEBUG:boto:Token: None 2024-04-19T23:33:17.089 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.090 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ARZGI+JlLpYMqtvsALMS5Fjdq1w= 2024-04-19T23:33:17.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ARZGI+JlLpYMqtvsALMS5Fjdq1w=', 'Content-Length': '0'} 2024-04-19T23:33:17.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001012d85cca2a7405-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:17.092 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.093 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.093 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.093 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.093 DEBUG:boto:Method: GET 2024-04-19T23:33:17.093 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.094 DEBUG:boto:Data: 2024-04-19T23:33:17.094 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.094 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.094 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.094 DEBUG:boto:Params: {} 2024-04-19T23:33:17.095 DEBUG:boto:Token: None 2024-04-19T23:33:17.095 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.095 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA= 2024-04-19T23:33:17.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA=', 'Content-Length': '0'} 2024-04-19T23:33:17.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b30af874de5275a9-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.142 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.142 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.143 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.143 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.143 DEBUG:boto:Method: GET 2024-04-19T23:33:17.143 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.143 DEBUG:boto:Data: 2024-04-19T23:33:17.143 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.143 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.143 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.143 DEBUG:boto:Params: {} 2024-04-19T23:33:17.144 DEBUG:boto:Token: None 2024-04-19T23:33:17.144 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.144 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA= 2024-04-19T23:33:17.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA=', 'Content-Length': '0'} 2024-04-19T23:33:17.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ae3295fc3d7dc0b-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.147 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.147 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.147 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.147 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.147 DEBUG:boto:Method: GET 2024-04-19T23:33:17.147 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.147 DEBUG:boto:Data: 2024-04-19T23:33:17.147 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.147 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.148 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.148 DEBUG:boto:Params: {} 2024-04-19T23:33:17.148 DEBUG:boto:Token: None 2024-04-19T23:33:17.148 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.148 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5c35c6f22c8e06d-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.151 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.151 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.151 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.152 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.152 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.152 DEBUG:boto:Method: GET 2024-04-19T23:33:17.152 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.152 DEBUG:boto:Data: 2024-04-19T23:33:17.152 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.152 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.153 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.153 DEBUG:boto:Params: {} 2024-04-19T23:33:17.153 DEBUG:boto:Token: None 2024-04-19T23:33:17.153 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.154 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000498ec3dfd1814f26-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.198 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.199 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.199 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.199 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.199 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:33:17.199 DEBUG:boto:Data: 2024-04-19T23:33:17.199 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.199 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.199 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.199 DEBUG:boto:Params: {} 2024-04-19T23:33:17.199 DEBUG:boto:Token: None 2024-04-19T23:33:17.200 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:33:17.200 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vuc+bov9KDpffdO47VC8BeM6l58= 2024-04-19T23:33:17.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vuc+bov9KDpffdO47VC8BeM6l58=', 'Content-Length': '0'} 2024-04-19T23:33:17.202 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af16949caa95cb85-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.202 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.203 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.203 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.203 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:33:17.203 DEBUG:boto:Data: 2024-04-19T23:33:17.203 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.203 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.203 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.203 DEBUG:boto:Params: {} 2024-04-19T23:33:17.203 DEBUG:boto:Token: None 2024-04-19T23:33:17.203 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:33:17.204 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vuc+bov9KDpffdO47VC8BeM6l58= 2024-04-19T23:33:17.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vuc+bov9KDpffdO47VC8BeM6l58=', 'Content-Length': '0'} 2024-04-19T23:33:17.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077132b61babf72b6-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:17.206 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.207 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.207 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.207 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.207 DEBUG:boto:Method: GET 2024-04-19T23:33:17.207 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.207 DEBUG:boto:Data: 2024-04-19T23:33:17.207 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.207 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.207 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.207 DEBUG:boto:Params: {} 2024-04-19T23:33:17.207 DEBUG:boto:Token: None 2024-04-19T23:33:17.208 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.208 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA= 2024-04-19T23:33:17.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA=', 'Content-Length': '0'} 2024-04-19T23:33:17.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2b08514bb239fac-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.250 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.251 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.251 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.251 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.251 DEBUG:boto:Method: GET 2024-04-19T23:33:17.251 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.251 DEBUG:boto:Data: 2024-04-19T23:33:17.251 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.251 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.251 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.251 DEBUG:boto:Params: {} 2024-04-19T23:33:17.252 DEBUG:boto:Token: None 2024-04-19T23:33:17.252 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.252 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA= 2024-04-19T23:33:17.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA=', 'Content-Length': '0'} 2024-04-19T23:33:17.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041f03cd37e9319de-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.254 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.255 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.255 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.255 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.255 DEBUG:boto:Method: GET 2024-04-19T23:33:17.255 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.255 DEBUG:boto:Data: 2024-04-19T23:33:17.255 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.255 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.255 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.255 DEBUG:boto:Params: {} 2024-04-19T23:33:17.255 DEBUG:boto:Token: None 2024-04-19T23:33:17.256 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.256 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec218f406e052e7d-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.258 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.259 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.259 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.259 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.259 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.259 DEBUG:boto:Method: GET 2024-04-19T23:33:17.259 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.259 DEBUG:boto:Data: 2024-04-19T23:33:17.259 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.259 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.259 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.260 DEBUG:boto:Params: {} 2024-04-19T23:33:17.260 DEBUG:boto:Token: None 2024-04-19T23:33:17.260 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.260 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d682d70cf5e557a-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.263 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:17.263 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.263 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.263 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.263 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.264 DEBUG:boto:Method: GET 2024-04-19T23:33:17.264 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.264 DEBUG:boto:Data: 2024-04-19T23:33:17.264 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.264 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.264 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.264 DEBUG:boto:Params: {} 2024-04-19T23:33:17.264 DEBUG:boto:Token: None 2024-04-19T23:33:17.264 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.265 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s= 2024-04-19T23:33:17.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s=', 'Content-Length': '0'} 2024-04-19T23:33:17.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7a8ac7f670aeb86-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.267 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.267 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.267 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.267 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.267 DEBUG:boto:Method: GET 2024-04-19T23:33:17.268 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.268 DEBUG:boto:Data: 2024-04-19T23:33:17.268 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.268 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.268 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.268 DEBUG:boto:Params: {} 2024-04-19T23:33:17.268 DEBUG:boto:Token: None 2024-04-19T23:33:17.268 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.268 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s= 2024-04-19T23:33:17.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s=', 'Content-Length': '0'} 2024-04-19T23:33:17.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8b412d909b64ef7-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.270 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.271 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.271 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.271 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.271 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.271 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.271 DEBUG:boto:Data: 2024-04-19T23:33:17.271 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.271 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.271 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.271 DEBUG:boto:Params: {} 2024-04-19T23:33:17.272 DEBUG:boto:Token: None 2024-04-19T23:33:17.272 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.272 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XymHlss4Pn8LCxFhU7Tnm0/Sptg= 2024-04-19T23:33:17.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XymHlss4Pn8LCxFhU7Tnm0/Sptg=', 'Content-Length': '0'} 2024-04-19T23:33:17.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051c7b3785491e4d0-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.274 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.274 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.274 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.274 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.277 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.277 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.278 DEBUG:boto:Data: 2024-04-19T23:33:17.278 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.278 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.278 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.278 DEBUG:boto:Params: {} 2024-04-19T23:33:17.278 DEBUG:boto:Token: None 2024-04-19T23:33:17.278 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.278 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XymHlss4Pn8LCxFhU7Tnm0/Sptg= 2024-04-19T23:33:17.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XymHlss4Pn8LCxFhU7Tnm0/Sptg=', 'Content-Length': '0'} 2024-04-19T23:33:17.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023d0ae6823924a18-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:17.281 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.281 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.281 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.281 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.281 DEBUG:boto:Method: GET 2024-04-19T23:33:17.281 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.281 DEBUG:boto:Data: 2024-04-19T23:33:17.281 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.282 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.282 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.282 DEBUG:boto:Params: {} 2024-04-19T23:33:17.282 DEBUG:boto:Token: None 2024-04-19T23:33:17.282 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.282 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s= 2024-04-19T23:33:17.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s=', 'Content-Length': '0'} 2024-04-19T23:33:17.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be292e550a9ff576-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.326 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.326 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.327 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.327 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.327 DEBUG:boto:Method: GET 2024-04-19T23:33:17.327 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.327 DEBUG:boto:Data: 2024-04-19T23:33:17.327 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.327 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.327 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.327 DEBUG:boto:Params: {} 2024-04-19T23:33:17.327 DEBUG:boto:Token: None 2024-04-19T23:33:17.328 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.328 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s= 2024-04-19T23:33:17.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s=', 'Content-Length': '0'} 2024-04-19T23:33:17.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6ec0daa4b3d4dab-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.370 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.371 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.371 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:17.371 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:17.371 DEBUG:boto:Method: GET 2024-04-19T23:33:17.371 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:17.371 DEBUG:boto:Data: 2024-04-19T23:33:17.371 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.371 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.371 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.371 DEBUG:boto:Params: {} 2024-04-19T23:33:17.372 DEBUG:boto:Token: None 2024-04-19T23:33:17.372 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.372 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4b2bb2a11734e44-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.374 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.375 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.375 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.375 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:17.375 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:17.375 DEBUG:boto:Method: GET 2024-04-19T23:33:17.375 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:17.375 DEBUG:boto:Data: 2024-04-19T23:33:17.376 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.376 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.376 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.376 DEBUG:boto:Params: {} 2024-04-19T23:33:17.376 DEBUG:boto:Token: None 2024-04-19T23:33:17.376 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.376 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e38dcb0955e60725-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.422 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.423 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.423 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.423 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.423 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:33:17.423 DEBUG:boto:Data: 2024-04-19T23:33:17.423 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.423 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.423 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.424 DEBUG:boto:Params: {} 2024-04-19T23:33:17.424 DEBUG:boto:Token: None 2024-04-19T23:33:17.424 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:33:17.424 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9aXHf4dRRwfTrBmXGHVrqAORtaI= 2024-04-19T23:33:17.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9aXHf4dRRwfTrBmXGHVrqAORtaI=', 'Content-Length': '0'} 2024-04-19T23:33:17.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005cdeae48cb91b27-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.427 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.427 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.427 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.427 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:33:17.427 DEBUG:boto:Data: 2024-04-19T23:33:17.427 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.427 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.427 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.427 DEBUG:boto:Params: {} 2024-04-19T23:33:17.428 DEBUG:boto:Token: None 2024-04-19T23:33:17.428 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:33:17.428 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9aXHf4dRRwfTrBmXGHVrqAORtaI= 2024-04-19T23:33:17.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9aXHf4dRRwfTrBmXGHVrqAORtaI=', 'Content-Length': '0'} 2024-04-19T23:33:17.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa49d8452307e775-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:17.431 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.431 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.431 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.431 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.432 DEBUG:boto:Method: GET 2024-04-19T23:33:17.432 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.432 DEBUG:boto:Data: 2024-04-19T23:33:17.432 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.432 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.432 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.432 DEBUG:boto:Params: {} 2024-04-19T23:33:17.432 DEBUG:boto:Token: None 2024-04-19T23:33:17.432 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.432 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s= 2024-04-19T23:33:17.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s=', 'Content-Length': '0'} 2024-04-19T23:33:17.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ceb286631a62e61f-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.434 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.435 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.435 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.435 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.435 DEBUG:boto:Method: GET 2024-04-19T23:33:17.435 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.435 DEBUG:boto:Data: 2024-04-19T23:33:17.435 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.435 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.435 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.435 DEBUG:boto:Params: {} 2024-04-19T23:33:17.436 DEBUG:boto:Token: None 2024-04-19T23:33:17.436 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.436 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s= 2024-04-19T23:33:17.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s=', 'Content-Length': '0'} 2024-04-19T23:33:17.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad5dca6e1bb58cd0-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.438 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.438 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.438 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:17.438 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:17.439 DEBUG:boto:Method: GET 2024-04-19T23:33:17.439 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:17.439 DEBUG:boto:Data: 2024-04-19T23:33:17.439 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.439 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.439 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.439 DEBUG:boto:Params: {} 2024-04-19T23:33:17.439 DEBUG:boto:Token: None 2024-04-19T23:33:17.440 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.440 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000527da50da3c8f99c-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.482 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.483 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.483 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.483 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:17.483 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:17.483 DEBUG:boto:Method: GET 2024-04-19T23:33:17.483 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:17.483 DEBUG:boto:Data: 2024-04-19T23:33:17.483 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.483 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.484 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.484 DEBUG:boto:Params: {} 2024-04-19T23:33:17.484 DEBUG:boto:Token: None 2024-04-19T23:33:17.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068ef4eafca72411b-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.487 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:17.487 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.487 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.487 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.487 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.488 DEBUG:boto:Method: GET 2024-04-19T23:33:17.488 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.488 DEBUG:boto:Data: 2024-04-19T23:33:17.488 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.488 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.488 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.488 DEBUG:boto:Params: {} 2024-04-19T23:33:17.488 DEBUG:boto:Token: None 2024-04-19T23:33:17.488 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MyIf3HwXcS9k7uqRdw7GMa1x4NU= 2024-04-19T23:33:17.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MyIf3HwXcS9k7uqRdw7GMa1x4NU=', 'Content-Length': '0'} 2024-04-19T23:33:17.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1ab295211a23ed8-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.491 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.491 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.491 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.491 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.491 DEBUG:boto:Method: GET 2024-04-19T23:33:17.491 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.491 DEBUG:boto:Data: 2024-04-19T23:33:17.492 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.492 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.492 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.492 DEBUG:boto:Params: {} 2024-04-19T23:33:17.492 DEBUG:boto:Token: None 2024-04-19T23:33:17.492 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.492 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MyIf3HwXcS9k7uqRdw7GMa1x4NU= 2024-04-19T23:33:17.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MyIf3HwXcS9k7uqRdw7GMa1x4NU=', 'Content-Length': '0'} 2024-04-19T23:33:17.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000191299e2c41ffffc-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.534 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.534 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.534 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.534 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.535 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.535 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.535 DEBUG:boto:Data: 2024-04-19T23:33:17.535 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.535 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.535 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.535 DEBUG:boto:Params: {} 2024-04-19T23:33:17.535 DEBUG:boto:Token: None 2024-04-19T23:33:17.536 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.536 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TMScqvxyltbUxWJkd4VuD+eMLvM= 2024-04-19T23:33:17.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TMScqvxyltbUxWJkd4VuD+eMLvM=', 'Content-Length': '0'} 2024-04-19T23:33:17.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000407aec800a8f25bc-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.538 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.538 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.538 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.538 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.538 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.538 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.538 DEBUG:boto:Data: 2024-04-19T23:33:17.538 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.538 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.539 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.539 DEBUG:boto:Params: {} 2024-04-19T23:33:17.539 DEBUG:boto:Token: None 2024-04-19T23:33:17.539 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.539 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TMScqvxyltbUxWJkd4VuD+eMLvM= 2024-04-19T23:33:17.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TMScqvxyltbUxWJkd4VuD+eMLvM=', 'Content-Length': '0'} 2024-04-19T23:33:17.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001717c275d560f172-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:17.541 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.541 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.542 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.542 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.542 DEBUG:boto:Method: GET 2024-04-19T23:33:17.542 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.542 DEBUG:boto:Data: 2024-04-19T23:33:17.542 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.542 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.542 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.542 DEBUG:boto:Params: {} 2024-04-19T23:33:17.543 DEBUG:boto:Token: None 2024-04-19T23:33:17.543 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.543 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MyIf3HwXcS9k7uqRdw7GMa1x4NU= 2024-04-19T23:33:17.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MyIf3HwXcS9k7uqRdw7GMa1x4NU=', 'Content-Length': '0'} 2024-04-19T23:33:17.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d02867ced8e9f9a-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.586 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.587 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.587 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.587 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.587 DEBUG:boto:Method: GET 2024-04-19T23:33:17.587 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.587 DEBUG:boto:Data: 2024-04-19T23:33:17.587 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.587 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.587 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.587 DEBUG:boto:Params: {} 2024-04-19T23:33:17.588 DEBUG:boto:Token: None 2024-04-19T23:33:17.588 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.588 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MyIf3HwXcS9k7uqRdw7GMa1x4NU= 2024-04-19T23:33:17.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MyIf3HwXcS9k7uqRdw7GMa1x4NU=', 'Content-Length': '0'} 2024-04-19T23:33:17.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000593aeeb2f33184c-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.590 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.590 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.590 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:17.591 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:17.591 DEBUG:boto:Method: GET 2024-04-19T23:33:17.591 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:17.591 DEBUG:boto:Data: 2024-04-19T23:33:17.591 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.591 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.591 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.591 DEBUG:boto:Params: {} 2024-04-19T23:33:17.591 DEBUG:boto:Token: None 2024-04-19T23:33:17.591 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.592 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000308e879778966d65-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.595 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.595 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.595 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.595 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:17.595 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:17.595 DEBUG:boto:Method: GET 2024-04-19T23:33:17.596 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:17.596 DEBUG:boto:Data: 2024-04-19T23:33:17.596 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.596 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.596 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.596 DEBUG:boto:Params: {} 2024-04-19T23:33:17.596 DEBUG:boto:Token: None 2024-04-19T23:33:17.596 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.596 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e7e1e37da22229-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.599 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.599 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.599 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.599 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.600 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:33:17.600 DEBUG:boto:Data: 2024-04-19T23:33:17.600 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.600 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.600 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.600 DEBUG:boto:Params: {} 2024-04-19T23:33:17.600 DEBUG:boto:Token: None 2024-04-19T23:33:17.600 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:33:17.600 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iPmXv8IYjTTQQ0K4JHJV5QbZvjI= 2024-04-19T23:33:17.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iPmXv8IYjTTQQ0K4JHJV5QbZvjI=', 'Content-Length': '0'} 2024-04-19T23:33:17.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbbba017c72b5cb1-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.603 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.603 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.603 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.604 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:33:17.604 DEBUG:boto:Data: 2024-04-19T23:33:17.604 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.604 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.604 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.604 DEBUG:boto:Params: {} 2024-04-19T23:33:17.604 DEBUG:boto:Token: None 2024-04-19T23:33:17.604 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:33:17.604 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iPmXv8IYjTTQQ0K4JHJV5QbZvjI= 2024-04-19T23:33:17.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iPmXv8IYjTTQQ0K4JHJV5QbZvjI=', 'Content-Length': '0'} 2024-04-19T23:33:17.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c41a487a22b33e3-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:17.607 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.607 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.607 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.607 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.607 DEBUG:boto:Method: GET 2024-04-19T23:33:17.607 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.607 DEBUG:boto:Data: 2024-04-19T23:33:17.608 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.608 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.608 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.608 DEBUG:boto:Params: {} 2024-04-19T23:33:17.608 DEBUG:boto:Token: None 2024-04-19T23:33:17.608 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.608 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MyIf3HwXcS9k7uqRdw7GMa1x4NU= 2024-04-19T23:33:17.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MyIf3HwXcS9k7uqRdw7GMa1x4NU=', 'Content-Length': '0'} 2024-04-19T23:33:17.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047616dc17d550e3f-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.654 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.654 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:17.654 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.655 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.655 DEBUG:boto:Method: GET 2024-04-19T23:33:17.655 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.655 DEBUG:boto:Data: 2024-04-19T23:33:17.655 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.655 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.655 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.655 DEBUG:boto:Params: {} 2024-04-19T23:33:17.655 DEBUG:boto:Token: None 2024-04-19T23:33:17.656 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:17.656 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MyIf3HwXcS9k7uqRdw7GMa1x4NU= 2024-04-19T23:33:17.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MyIf3HwXcS9k7uqRdw7GMa1x4NU=', 'Content-Length': '0'} 2024-04-19T23:33:17.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000081ac07f083cebac-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.658 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.658 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.658 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:17.658 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:17.658 DEBUG:boto:Method: GET 2024-04-19T23:33:17.658 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:17.659 DEBUG:boto:Data: 2024-04-19T23:33:17.659 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.659 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:17.659 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.659 DEBUG:boto:Params: {} 2024-04-19T23:33:17.659 DEBUG:boto:Token: None 2024-04-19T23:33:17.659 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.659 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007875321cca87172f-006622ff3d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.662 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.662 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.662 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.663 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:17.663 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:17.663 DEBUG:boto:Method: GET 2024-04-19T23:33:17.663 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:17.663 DEBUG:boto:Data: 2024-04-19T23:33:17.663 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.663 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.663 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.663 DEBUG:boto:Params: {} 2024-04-19T23:33:17.663 DEBUG:boto:Token: None 2024-04-19T23:33:17.664 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.664 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb5e4722e7ab4d17-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.666 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.667 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-19 zones={test-zone2, test-zone1} 2024-04-19T23:33:17.667 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-19 zones={test-zone3, test-zone1} 2024-04-19T23:33:17.667 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.667 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.667 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.667 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:33:17.667 DEBUG:boto:Data: 2024-04-19T23:33:17.667 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.668 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.668 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.668 DEBUG:boto:Params: {} 2024-04-19T23:33:17.668 DEBUG:boto:Token: None 2024-04-19T23:33:17.668 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.668 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VyyiZFaTslRtWQThJi8MM/5WRW8= 2024-04-19T23:33:17.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VyyiZFaTslRtWQThJi8MM/5WRW8=', 'Content-Length': '0'} 2024-04-19T23:33:17.670 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c6e1f4d91dc628c-006622ff3d-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.671 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.671 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.671 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.671 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:33:17.671 DEBUG:boto:Data: 2024-04-19T23:33:17.671 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.671 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.671 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.671 DEBUG:boto:Params: {} 2024-04-19T23:33:17.671 DEBUG:boto:Token: None 2024-04-19T23:33:17.672 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.672 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VyyiZFaTslRtWQThJi8MM/5WRW8= 2024-04-19T23:33:17.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VyyiZFaTslRtWQThJi8MM/5WRW8=', 'Content-Length': '0'} 2024-04-19T23:33:17.674 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bb1cf77fe92cbee-006622ff3d-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.674 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:33:17.674 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.674 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.674 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.674 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.675 DEBUG:boto:Method: GET 2024-04-19T23:33:17.675 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.675 DEBUG:boto:Data: 2024-04-19T23:33:17.675 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.675 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.675 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.675 DEBUG:boto:Params: {} 2024-04-19T23:33:17.675 DEBUG:boto:Token: None 2024-04-19T23:33:17.675 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/?versions 2024-04-19T23:33:17.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I= 2024-04-19T23:33:17.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I=', 'Content-Length': '0'} 2024-04-19T23:33:17.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f9dffb1bd85ca99-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.678 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.679 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:17.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:17.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:17.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:17.679 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:33:17.679 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.679 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.680 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.680 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.680 DEBUG:boto:Method: GET 2024-04-19T23:33:17.680 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.680 DEBUG:boto:Data: 2024-04-19T23:33:17.680 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.680 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.680 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.680 DEBUG:boto:Params: {} 2024-04-19T23:33:17.680 DEBUG:boto:Token: None 2024-04-19T23:33:17.681 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/?versions 2024-04-19T23:33:17.681 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I= 2024-04-19T23:33:17.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I=', 'Content-Length': '0'} 2024-04-19T23:33:17.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f064c33c8d0c2db4-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.686 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.686 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:17.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:17.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:17.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:17.687 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.687 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.687 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.687 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.687 DEBUG:boto:Method: GET 2024-04-19T23:33:17.687 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.687 DEBUG:boto:Data: 2024-04-19T23:33:17.687 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.688 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.688 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.688 DEBUG:boto:Params: {} 2024-04-19T23:33:17.688 DEBUG:boto:Token: None 2024-04-19T23:33:17.688 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/?versions 2024-04-19T23:33:17.688 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I= 2024-04-19T23:33:17.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I=', 'Content-Length': '0'} 2024-04-19T23:33:17.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e58f4f8c31256012-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.691 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.691 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.691 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.691 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.692 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.692 DEBUG:boto:Method: GET 2024-04-19T23:33:17.692 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:33:17.692 DEBUG:boto:Data: 2024-04-19T23:33:17.692 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.692 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.692 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.692 DEBUG:boto:Params: {} 2024-04-19T23:33:17.692 DEBUG:boto:Token: None 2024-04-19T23:33:17.692 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/?versions 2024-04-19T23:33:17.693 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I= 2024-04-19T23:33:17.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shy+zDiQRRSPD+fqp1ZZZjlca9I=', 'Content-Length': '0'} 2024-04-19T23:33:17.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da235eef27e3a50-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.695 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:17.696 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.696 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.696 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.696 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.696 DEBUG:boto:Method: GET 2024-04-19T23:33:17.696 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.696 DEBUG:boto:Data: 2024-04-19T23:33:17.697 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.697 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.697 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.697 DEBUG:boto:Params: {} 2024-04-19T23:33:17.697 DEBUG:boto:Token: None 2024-04-19T23:33:17.697 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.697 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc= 2024-04-19T23:33:17.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc=', 'Content-Length': '0'} 2024-04-19T23:33:17.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b8126adea2085bf-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.699 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.700 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.700 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.700 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.700 DEBUG:boto:Method: GET 2024-04-19T23:33:17.700 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.700 DEBUG:boto:Data: 2024-04-19T23:33:17.700 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.700 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.700 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.700 DEBUG:boto:Params: {} 2024-04-19T23:33:17.701 DEBUG:boto:Token: None 2024-04-19T23:33:17.701 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.701 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc= 2024-04-19T23:33:17.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc=', 'Content-Length': '0'} 2024-04-19T23:33:17.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f2ad1f1d3cfae216-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.703 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.703 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.703 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.703 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.703 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.704 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.704 DEBUG:boto:Data: 2024-04-19T23:33:17.704 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.704 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.704 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.704 DEBUG:boto:Params: {} 2024-04-19T23:33:17.704 DEBUG:boto:Token: None 2024-04-19T23:33:17.704 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FUSW5/bEqdXRBCZ/gWueq/dlxCE= 2024-04-19T23:33:17.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FUSW5/bEqdXRBCZ/gWueq/dlxCE=', 'Content-Length': '0'} 2024-04-19T23:33:17.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000090725f3947cc52d4-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.706 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.707 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.707 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.707 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.707 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.707 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.707 DEBUG:boto:Data: 2024-04-19T23:33:17.707 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.707 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.707 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.707 DEBUG:boto:Params: {} 2024-04-19T23:33:17.708 DEBUG:boto:Token: None 2024-04-19T23:33:17.708 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.708 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FUSW5/bEqdXRBCZ/gWueq/dlxCE= 2024-04-19T23:33:17.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FUSW5/bEqdXRBCZ/gWueq/dlxCE=', 'Content-Length': '0'} 2024-04-19T23:33:17.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a0f9be17d479ab99-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:17.710 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.710 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.710 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.711 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.711 DEBUG:boto:Method: GET 2024-04-19T23:33:17.711 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.711 DEBUG:boto:Data: 2024-04-19T23:33:17.711 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.711 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.711 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.711 DEBUG:boto:Params: {} 2024-04-19T23:33:17.711 DEBUG:boto:Token: None 2024-04-19T23:33:17.711 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.712 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc= 2024-04-19T23:33:17.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc=', 'Content-Length': '0'} 2024-04-19T23:33:17.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046108813366a5cf4-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.714 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.714 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.714 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.714 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.714 DEBUG:boto:Method: GET 2024-04-19T23:33:17.715 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.715 DEBUG:boto:Data: 2024-04-19T23:33:17.715 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.715 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.715 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.715 DEBUG:boto:Params: {} 2024-04-19T23:33:17.715 DEBUG:boto:Token: None 2024-04-19T23:33:17.715 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.715 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc= 2024-04-19T23:33:17.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc=', 'Content-Length': '0'} 2024-04-19T23:33:17.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000089f64077642e2219-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.717 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.718 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.718 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.718 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.718 DEBUG:boto:Method: GET 2024-04-19T23:33:17.718 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:33:17.718 DEBUG:boto:Data: 2024-04-19T23:33:17.718 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.718 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.718 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.718 DEBUG:boto:Params: {} 2024-04-19T23:33:17.719 DEBUG:boto:Token: None 2024-04-19T23:33:17.719 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.719 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006874579fba79a5fd-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.762 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.763 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.763 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.763 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.763 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.763 DEBUG:boto:Method: GET 2024-04-19T23:33:17.763 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:33:17.763 DEBUG:boto:Data: 2024-04-19T23:33:17.763 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.763 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.764 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.764 DEBUG:boto:Params: {} 2024-04-19T23:33:17.764 DEBUG:boto:Token: None 2024-04-19T23:33:17.764 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.764 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a288d3bbffe91c40-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.767 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.767 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.767 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.767 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.767 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:33:17.767 DEBUG:boto:Data: 2024-04-19T23:33:17.767 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.767 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.768 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.768 DEBUG:boto:Params: {} 2024-04-19T23:33:17.768 DEBUG:boto:Token: None 2024-04-19T23:33:17.768 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy 2024-04-19T23:33:17.768 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OFm94Ex3wBKAP2XQmthjh0NvEbk= 2024-04-19T23:33:17.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OFm94Ex3wBKAP2XQmthjh0NvEbk=', 'Content-Length': '0'} 2024-04-19T23:33:17.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f56a064831f84ff-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.771 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.771 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.771 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.771 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:33:17.771 DEBUG:boto:Data: 2024-04-19T23:33:17.771 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.771 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.771 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.771 DEBUG:boto:Params: {} 2024-04-19T23:33:17.772 DEBUG:boto:Token: None 2024-04-19T23:33:17.772 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy 2024-04-19T23:33:17.772 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OFm94Ex3wBKAP2XQmthjh0NvEbk= 2024-04-19T23:33:17.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OFm94Ex3wBKAP2XQmthjh0NvEbk=', 'Content-Length': '0'} 2024-04-19T23:33:17.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000757a3fb903f3d738-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:17.775 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.775 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.775 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.775 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.775 DEBUG:boto:Method: GET 2024-04-19T23:33:17.775 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.775 DEBUG:boto:Data: 2024-04-19T23:33:17.775 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.775 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.775 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.776 DEBUG:boto:Params: {} 2024-04-19T23:33:17.776 DEBUG:boto:Token: None 2024-04-19T23:33:17.776 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.776 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc= 2024-04-19T23:33:17.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc=', 'Content-Length': '0'} 2024-04-19T23:33:17.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d327e3ec732d50f1-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.778 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:17.778 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:17.778 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.778 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.779 DEBUG:boto:Method: GET 2024-04-19T23:33:17.779 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.779 DEBUG:boto:Data: 2024-04-19T23:33:17.779 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.779 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.779 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.779 DEBUG:boto:Params: {} 2024-04-19T23:33:17.779 DEBUG:boto:Token: None 2024-04-19T23:33:17.780 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:17.780 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc= 2024-04-19T23:33:17.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:csAcRcFIlvXVYP1TyDiaiy94cTc=', 'Content-Length': '0'} 2024-04-19T23:33:17.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008a7fc3ed6a3384c2-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.782 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.782 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.782 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.782 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.783 DEBUG:boto:Method: GET 2024-04-19T23:33:17.783 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:33:17.783 DEBUG:boto:Data: 2024-04-19T23:33:17.783 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.783 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.783 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.783 DEBUG:boto:Params: {} 2024-04-19T23:33:17.783 DEBUG:boto:Token: None 2024-04-19T23:33:17.783 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.783 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f30ed650da0f814-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.786 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.786 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.786 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.786 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.787 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:17.787 DEBUG:boto:Method: GET 2024-04-19T23:33:17.787 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:33:17.787 DEBUG:boto:Data: 2024-04-19T23:33:17.787 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.787 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.787 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.787 DEBUG:boto:Params: {} 2024-04-19T23:33:17.787 DEBUG:boto:Token: None 2024-04-19T23:33:17.787 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.788 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6ffd6cab0e06e85-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.790 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:17.791 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.791 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.791 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.791 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.791 DEBUG:boto:Method: GET 2024-04-19T23:33:17.791 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.791 DEBUG:boto:Data: 2024-04-19T23:33:17.791 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.791 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.791 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.792 DEBUG:boto:Params: {} 2024-04-19T23:33:17.792 DEBUG:boto:Token: None 2024-04-19T23:33:17.792 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.792 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA= 2024-04-19T23:33:17.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA=', 'Content-Length': '0'} 2024-04-19T23:33:17.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034a4bc1183dc0ada-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.834 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.834 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.834 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.835 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.835 DEBUG:boto:Method: GET 2024-04-19T23:33:17.835 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.835 DEBUG:boto:Data: 2024-04-19T23:33:17.835 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.835 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.835 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.835 DEBUG:boto:Params: {} 2024-04-19T23:33:17.835 DEBUG:boto:Token: None 2024-04-19T23:33:17.835 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.836 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA= 2024-04-19T23:33:17.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA=', 'Content-Length': '0'} 2024-04-19T23:33:17.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045cfadb0f3eb165e-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.838 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.838 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.838 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.838 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.838 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.838 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.838 DEBUG:boto:Data: 2024-04-19T23:33:17.838 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.839 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.839 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.839 DEBUG:boto:Params: {} 2024-04-19T23:33:17.839 DEBUG:boto:Token: None 2024-04-19T23:33:17.839 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.839 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ARZGI+JlLpYMqtvsALMS5Fjdq1w= 2024-04-19T23:33:17.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ARZGI+JlLpYMqtvsALMS5Fjdq1w=', 'Content-Length': '0'} 2024-04-19T23:33:17.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee8dd05c86a15711-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.841 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.842 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.842 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.842 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.842 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.842 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.842 DEBUG:boto:Data: 2024-04-19T23:33:17.842 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.842 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.842 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.842 DEBUG:boto:Params: {} 2024-04-19T23:33:17.843 DEBUG:boto:Token: None 2024-04-19T23:33:17.843 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.843 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ARZGI+JlLpYMqtvsALMS5Fjdq1w= 2024-04-19T23:33:17.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ARZGI+JlLpYMqtvsALMS5Fjdq1w=', 'Content-Length': '0'} 2024-04-19T23:33:17.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccfe64c36f1c8dc5-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:17.845 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.845 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.845 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.845 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.845 DEBUG:boto:Method: GET 2024-04-19T23:33:17.846 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.846 DEBUG:boto:Data: 2024-04-19T23:33:17.846 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.846 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.846 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.846 DEBUG:boto:Params: {} 2024-04-19T23:33:17.846 DEBUG:boto:Token: None 2024-04-19T23:33:17.846 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.847 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA= 2024-04-19T23:33:17.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA=', 'Content-Length': '0'} 2024-04-19T23:33:17.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003275dda9e680c832-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.891 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.891 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.891 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.891 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.891 DEBUG:boto:Method: GET 2024-04-19T23:33:17.891 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.892 DEBUG:boto:Data: 2024-04-19T23:33:17.892 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.892 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.892 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.892 DEBUG:boto:Params: {} 2024-04-19T23:33:17.892 DEBUG:boto:Token: None 2024-04-19T23:33:17.892 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.892 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA= 2024-04-19T23:33:17.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA=', 'Content-Length': '0'} 2024-04-19T23:33:17.894 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000214a86eac297f39-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.895 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.895 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.895 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.895 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.895 DEBUG:boto:Method: GET 2024-04-19T23:33:17.895 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.895 DEBUG:boto:Data: 2024-04-19T23:33:17.895 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.895 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.895 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.895 DEBUG:boto:Params: {} 2024-04-19T23:33:17.896 DEBUG:boto:Token: None 2024-04-19T23:33:17.896 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.896 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b688a5eeb0800d1-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.898 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.899 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.899 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.899 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.899 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.899 DEBUG:boto:Method: GET 2024-04-19T23:33:17.899 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.899 DEBUG:boto:Data: 2024-04-19T23:33:17.900 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.900 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.900 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.900 DEBUG:boto:Params: {} 2024-04-19T23:33:17.900 DEBUG:boto:Token: None 2024-04-19T23:33:17.900 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.900 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008620b6ac02f126d7-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.903 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.903 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.903 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.903 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.903 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:33:17.904 DEBUG:boto:Data: 2024-04-19T23:33:17.904 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.904 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.904 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.904 DEBUG:boto:Params: {} 2024-04-19T23:33:17.904 DEBUG:boto:Token: None 2024-04-19T23:33:17.904 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:33:17.904 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vuc+bov9KDpffdO47VC8BeM6l58= 2024-04-19T23:33:17.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vuc+bov9KDpffdO47VC8BeM6l58=', 'Content-Length': '0'} 2024-04-19T23:33:17.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007afc37b59d6c0301-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.907 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.907 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.907 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.907 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:33:17.907 DEBUG:boto:Data: 2024-04-19T23:33:17.907 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.907 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.907 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.908 DEBUG:boto:Params: {} 2024-04-19T23:33:17.908 DEBUG:boto:Token: None 2024-04-19T23:33:17.908 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:33:17.908 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vuc+bov9KDpffdO47VC8BeM6l58= 2024-04-19T23:33:17.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vuc+bov9KDpffdO47VC8BeM6l58=', 'Content-Length': '0'} 2024-04-19T23:33:17.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000527f085459fd0124-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:17.911 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.911 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.911 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.911 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.911 DEBUG:boto:Method: GET 2024-04-19T23:33:17.912 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.912 DEBUG:boto:Data: 2024-04-19T23:33:17.912 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.912 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.912 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.912 DEBUG:boto:Params: {} 2024-04-19T23:33:17.913 DEBUG:boto:Token: None 2024-04-19T23:33:17.913 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.913 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA= 2024-04-19T23:33:17.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA=', 'Content-Length': '0'} 2024-04-19T23:33:17.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002181fcc033e9f52a-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.962 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.962 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:17.963 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.963 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.963 DEBUG:boto:Method: GET 2024-04-19T23:33:17.963 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.963 DEBUG:boto:Data: 2024-04-19T23:33:17.963 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.963 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.963 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.963 DEBUG:boto:Params: {} 2024-04-19T23:33:17.964 DEBUG:boto:Token: None 2024-04-19T23:33:17.964 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:17.964 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA= 2024-04-19T23:33:17.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mHHvIvDrmShEQw/gEskxC/EIUYA=', 'Content-Length': '0'} 2024-04-19T23:33:17.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a41c8f200d9b199-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.966 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.966 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.966 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.966 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.967 DEBUG:boto:Method: GET 2024-04-19T23:33:17.967 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.967 DEBUG:boto:Data: 2024-04-19T23:33:17.967 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.967 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.967 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.967 DEBUG:boto:Params: {} 2024-04-19T23:33:17.967 DEBUG:boto:Token: None 2024-04-19T23:33:17.967 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.967 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abc02b82fbd35a3a-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.970 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.971 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:17.971 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:17.971 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.971 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.971 DEBUG:boto:Method: GET 2024-04-19T23:33:17.971 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:17.971 DEBUG:boto:Data: 2024-04-19T23:33:17.971 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.971 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.971 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.971 DEBUG:boto:Params: {} 2024-04-19T23:33:17.972 DEBUG:boto:Token: None 2024-04-19T23:33:17.972 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/ 2024-04-19T23:33:17.972 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA= 2024-04-19T23:33:17.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h634dGxa0nzMtV3RhpYZ+mUwHHA=', 'Content-Length': '0'} 2024-04-19T23:33:17.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bad0e511e59d3a04-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.974 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:17.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:17.975 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.975 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.975 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.975 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.975 DEBUG:boto:Method: GET 2024-04-19T23:33:17.975 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.975 DEBUG:boto:Data: 2024-04-19T23:33:17.976 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.976 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.976 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.976 DEBUG:boto:Params: {} 2024-04-19T23:33:17.976 DEBUG:boto:Token: None 2024-04-19T23:33:17.976 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.976 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s= 2024-04-19T23:33:17.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s=', 'Content-Length': '0'} 2024-04-19T23:33:17.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c04d379199c4da4c-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.979 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.979 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.979 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.979 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.979 DEBUG:boto:Method: GET 2024-04-19T23:33:17.979 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.979 DEBUG:boto:Data: 2024-04-19T23:33:17.979 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.980 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.980 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.980 DEBUG:boto:Params: {} 2024-04-19T23:33:17.980 DEBUG:boto:Token: None 2024-04-19T23:33:17.980 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.980 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s= 2024-04-19T23:33:17.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s=', 'Content-Length': '0'} 2024-04-19T23:33:17.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000750c6f54b2926de0-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.983 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.983 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.983 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.983 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.983 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.983 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.983 DEBUG:boto:Data: 2024-04-19T23:33:17.983 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.983 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.984 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.984 DEBUG:boto:Params: {} 2024-04-19T23:33:17.984 DEBUG:boto:Token: None 2024-04-19T23:33:17.984 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XymHlss4Pn8LCxFhU7Tnm0/Sptg= 2024-04-19T23:33:17.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XymHlss4Pn8LCxFhU7Tnm0/Sptg=', 'Content-Length': '0'} 2024-04-19T23:33:17.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3e6c2f2f67a6f25-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.986 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.986 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.986 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.987 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.987 DEBUG:boto:Method: HEAD 2024-04-19T23:33:17.987 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.987 DEBUG:boto:Data: 2024-04-19T23:33:17.987 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.987 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:17.987 DEBUG:boto:Port: 8000 2024-04-19T23:33:17.987 DEBUG:boto:Params: {} 2024-04-19T23:33:17.987 DEBUG:boto:Token: None 2024-04-19T23:33:17.987 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.987 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XymHlss4Pn8LCxFhU7Tnm0/Sptg= 2024-04-19T23:33:17.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XymHlss4Pn8LCxFhU7Tnm0/Sptg=', 'Content-Length': '0'} 2024-04-19T23:33:17.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f85647f2efc5f61b-006622ff3d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:17.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:17.990 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.990 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:17.990 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.990 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.990 DEBUG:boto:Method: GET 2024-04-19T23:33:17.990 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.990 DEBUG:boto:Data: 2024-04-19T23:33:17.990 DEBUG:boto:Headers: {} 2024-04-19T23:33:17.991 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:17.991 DEBUG:boto:Port: 8001 2024-04-19T23:33:17.991 DEBUG:boto:Params: {} 2024-04-19T23:33:17.991 DEBUG:boto:Token: None 2024-04-19T23:33:17.991 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:17 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:17.991 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s= 2024-04-19T23:33:17.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BPEZVdQO8xFQ1jAu3oLdBMVNP3s=', 'Content-Length': '0'} 2024-04-19T23:33:17.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041297a9a8fbbc998-006622ff3d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.034 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:18.034 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:18.035 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:18.035 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:18.035 DEBUG:boto:Method: GET 2024-04-19T23:33:18.035 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:18.035 DEBUG:boto:Data: 2024-04-19T23:33:18.035 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.035 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:18.035 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.035 DEBUG:boto:Params: {} 2024-04-19T23:33:18.036 DEBUG:boto:Token: None 2024-04-19T23:33:18.036 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:18.036 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3DfTQ9wxib90RxR6C8Olo2h/HEQ= 2024-04-19T23:33:18.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3DfTQ9wxib90RxR6C8Olo2h/HEQ=', 'Content-Length': '0'} 2024-04-19T23:33:18.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005385a25406b3448c-006622ff3e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.082 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.082 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.083 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:18.083 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:18.083 DEBUG:boto:Method: GET 2024-04-19T23:33:18.083 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:18.083 DEBUG:boto:Data: 2024-04-19T23:33:18.083 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.083 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.083 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.083 DEBUG:boto:Params: {} 2024-04-19T23:33:18.083 DEBUG:boto:Token: None 2024-04-19T23:33:18.083 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f00213bb70b66f05-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.130 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.131 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.131 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.131 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:18.131 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:18.131 DEBUG:boto:Method: GET 2024-04-19T23:33:18.131 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:18.131 DEBUG:boto:Data: 2024-04-19T23:33:18.131 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.131 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:18.132 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.132 DEBUG:boto:Params: {} 2024-04-19T23:33:18.132 DEBUG:boto:Token: None 2024-04-19T23:33:18.132 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.132 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca5659fc2d0556e0-006622ff3e-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.178 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.179 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:18.179 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:18.179 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.179 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:33:18.179 DEBUG:boto:Data: 2024-04-19T23:33:18.179 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.179 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.179 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.179 DEBUG:boto:Params: {} 2024-04-19T23:33:18.180 DEBUG:boto:Token: None 2024-04-19T23:33:18.180 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:33:18.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1bwZnu9WwH+boW2SYHKQEOHtQBc= 2024-04-19T23:33:18.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1bwZnu9WwH+boW2SYHKQEOHtQBc=', 'Content-Length': '0'} 2024-04-19T23:33:18.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078d89db04750aab8-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.183 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:18.183 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:18.183 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.183 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:33:18.183 DEBUG:boto:Data: 2024-04-19T23:33:18.183 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.183 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:18.183 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.184 DEBUG:boto:Params: {} 2024-04-19T23:33:18.184 DEBUG:boto:Token: None 2024-04-19T23:33:18.184 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:33:18.184 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1bwZnu9WwH+boW2SYHKQEOHtQBc= 2024-04-19T23:33:18.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1bwZnu9WwH+boW2SYHKQEOHtQBc=', 'Content-Length': '0'} 2024-04-19T23:33:18.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085370b98acdab7c4-006622ff3e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:18.187 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:18.187 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:18.187 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:18.187 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:18.187 DEBUG:boto:Method: GET 2024-04-19T23:33:18.187 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:18.187 DEBUG:boto:Data: 2024-04-19T23:33:18.187 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.187 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.187 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.188 DEBUG:boto:Params: {} 2024-04-19T23:33:18.188 DEBUG:boto:Token: None 2024-04-19T23:33:18.188 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:18.188 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3DfTQ9wxib90RxR6C8Olo2h/HEQ= 2024-04-19T23:33:18.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3DfTQ9wxib90RxR6C8Olo2h/HEQ=', 'Content-Length': '0'} 2024-04-19T23:33:18.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000285f236e5aa23480-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.234 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:18.235 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:18.235 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:18.235 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:18.235 DEBUG:boto:Method: GET 2024-04-19T23:33:18.235 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:18.235 DEBUG:boto:Data: 2024-04-19T23:33:18.235 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.235 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:18.235 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.236 DEBUG:boto:Params: {} 2024-04-19T23:33:18.236 DEBUG:boto:Token: None 2024-04-19T23:33:18.236 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:18.236 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3DfTQ9wxib90RxR6C8Olo2h/HEQ= 2024-04-19T23:33:18.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3DfTQ9wxib90RxR6C8Olo2h/HEQ=', 'Content-Length': '0'} 2024-04-19T23:33:18.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004d51426eee80ee1-006622ff3e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.278 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.279 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.279 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:18.279 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:18.279 DEBUG:boto:Method: GET 2024-04-19T23:33:18.279 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:18.279 DEBUG:boto:Data: 2024-04-19T23:33:18.279 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.280 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.280 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.280 DEBUG:boto:Params: {} 2024-04-19T23:33:18.280 DEBUG:boto:Token: None 2024-04-19T23:33:18.280 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.280 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000562ec426d7e60745-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.330 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.331 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.331 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.331 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:18.331 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:18.331 DEBUG:boto:Method: GET 2024-04-19T23:33:18.331 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:18.331 DEBUG:boto:Data: 2024-04-19T23:33:18.332 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.332 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:18.332 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.332 DEBUG:boto:Params: {} 2024-04-19T23:33:18.332 DEBUG:boto:Token: None 2024-04-19T23:33:18.332 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a7976284e933e03-006622ff3e-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.336 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:18.337 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.337 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.337 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.337 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.337 DEBUG:boto:Method: GET 2024-04-19T23:33:18.337 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.337 DEBUG:boto:Data: 2024-04-19T23:33:18.337 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.338 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.338 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.338 DEBUG:boto:Params: {} 2024-04-19T23:33:18.338 DEBUG:boto:Token: None 2024-04-19T23:33:18.338 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.339 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:H91r2XwwNFfYJSYpL8QvMguZ8Uo= 2024-04-19T23:33:18.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:H91r2XwwNFfYJSYpL8QvMguZ8Uo=', 'Content-Length': '0'} 2024-04-19T23:33:18.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf436bca69a92d1f-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.341 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.341 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.341 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.341 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.341 DEBUG:boto:Method: GET 2024-04-19T23:33:18.341 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.341 DEBUG:boto:Data: 2024-04-19T23:33:18.341 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.342 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:18.342 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.342 DEBUG:boto:Params: {} 2024-04-19T23:33:18.342 DEBUG:boto:Token: None 2024-04-19T23:33:18.342 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.342 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:H91r2XwwNFfYJSYpL8QvMguZ8Uo= 2024-04-19T23:33:18.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:H91r2XwwNFfYJSYpL8QvMguZ8Uo=', 'Content-Length': '0'} 2024-04-19T23:33:18.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000655dcda2cd5948b0-006622ff3e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.390 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.390 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.391 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.391 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.391 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.391 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.391 DEBUG:boto:Data: 2024-04-19T23:33:18.391 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.391 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.391 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.391 DEBUG:boto:Params: {} 2024-04-19T23:33:18.391 DEBUG:boto:Token: None 2024-04-19T23:33:18.392 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.392 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Dhixpjc1L50ZGYpm/t+NAl6UWjE= 2024-04-19T23:33:18.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Dhixpjc1L50ZGYpm/t+NAl6UWjE=', 'Content-Length': '0'} 2024-04-19T23:33:18.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050686e882ec0e462-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.394 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.394 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.394 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.394 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.394 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.395 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.395 DEBUG:boto:Data: 2024-04-19T23:33:18.395 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.395 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:18.395 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.395 DEBUG:boto:Params: {} 2024-04-19T23:33:18.395 DEBUG:boto:Token: None 2024-04-19T23:33:18.395 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.395 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Dhixpjc1L50ZGYpm/t+NAl6UWjE= 2024-04-19T23:33:18.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Dhixpjc1L50ZGYpm/t+NAl6UWjE=', 'Content-Length': '0'} 2024-04-19T23:33:18.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000326e5360888f326b-006622ff3e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:18.398 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.398 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.398 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.398 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.398 DEBUG:boto:Method: GET 2024-04-19T23:33:18.398 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.398 DEBUG:boto:Data: 2024-04-19T23:33:18.398 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.398 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.398 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.398 DEBUG:boto:Params: {} 2024-04-19T23:33:18.399 DEBUG:boto:Token: None 2024-04-19T23:33:18.399 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.399 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:H91r2XwwNFfYJSYpL8QvMguZ8Uo= 2024-04-19T23:33:18.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:H91r2XwwNFfYJSYpL8QvMguZ8Uo=', 'Content-Length': '0'} 2024-04-19T23:33:18.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6ad99be03de69ac-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.401 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.401 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.402 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.402 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.402 DEBUG:boto:Method: GET 2024-04-19T23:33:18.402 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.402 DEBUG:boto:Data: 2024-04-19T23:33:18.402 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.402 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:18.402 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.402 DEBUG:boto:Params: {} 2024-04-19T23:33:18.403 DEBUG:boto:Token: None 2024-04-19T23:33:18.403 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.403 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:H91r2XwwNFfYJSYpL8QvMguZ8Uo= 2024-04-19T23:33:18.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:H91r2XwwNFfYJSYpL8QvMguZ8Uo=', 'Content-Length': '0'} 2024-04-19T23:33:18.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3dc4263b3a8292e-006622ff3e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.450 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.451 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.451 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:18.451 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:18.451 DEBUG:boto:Method: GET 2024-04-19T23:33:18.451 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:18.451 DEBUG:boto:Data: 2024-04-19T23:33:18.451 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.451 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.451 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.451 DEBUG:boto:Params: {} 2024-04-19T23:33:18.451 DEBUG:boto:Token: None 2024-04-19T23:33:18.452 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.452 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9e8cf813f7ec05a-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.455 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.455 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.455 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.455 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:18.455 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:18.455 DEBUG:boto:Method: GET 2024-04-19T23:33:18.456 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:18.456 DEBUG:boto:Data: 2024-04-19T23:33:18.456 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.456 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:18.456 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.456 DEBUG:boto:Params: {} 2024-04-19T23:33:18.456 DEBUG:boto:Token: None 2024-04-19T23:33:18.456 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.456 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce74d1c3751f1145-006622ff3e-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.459 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.459 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.459 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.459 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.460 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:33:18.460 DEBUG:boto:Data: 2024-04-19T23:33:18.460 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.460 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.460 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.460 DEBUG:boto:Params: {} 2024-04-19T23:33:18.460 DEBUG:boto:Token: None 2024-04-19T23:33:18.460 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:33:18.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T8dy0Qn6BIXmPCX+hSsRjm5q7gc= 2024-04-19T23:33:18.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T8dy0Qn6BIXmPCX+hSsRjm5q7gc=', 'Content-Length': '0'} 2024-04-19T23:33:18.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007176a4fc82baa31d-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.463 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.463 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.463 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.463 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:33:18.463 DEBUG:boto:Data: 2024-04-19T23:33:18.463 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.463 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:18.463 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.463 DEBUG:boto:Params: {} 2024-04-19T23:33:18.463 DEBUG:boto:Token: None 2024-04-19T23:33:18.464 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:33:18.464 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T8dy0Qn6BIXmPCX+hSsRjm5q7gc= 2024-04-19T23:33:18.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T8dy0Qn6BIXmPCX+hSsRjm5q7gc=', 'Content-Length': '0'} 2024-04-19T23:33:18.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026e869a92d40b10e-006622ff3e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:18.467 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.467 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.467 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.467 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.467 DEBUG:boto:Method: GET 2024-04-19T23:33:18.467 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.467 DEBUG:boto:Data: 2024-04-19T23:33:18.467 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.467 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.467 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.467 DEBUG:boto:Params: {} 2024-04-19T23:33:18.468 DEBUG:boto:Token: None 2024-04-19T23:33:18.468 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.468 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:H91r2XwwNFfYJSYpL8QvMguZ8Uo= 2024-04-19T23:33:18.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:H91r2XwwNFfYJSYpL8QvMguZ8Uo=', 'Content-Length': '0'} 2024-04-19T23:33:18.470 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbdb6fd0d7a9b76b-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.514 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.515 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:18.515 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.515 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.515 DEBUG:boto:Method: GET 2024-04-19T23:33:18.515 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.515 DEBUG:boto:Data: 2024-04-19T23:33:18.515 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.515 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:18.515 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.515 DEBUG:boto:Params: {} 2024-04-19T23:33:18.515 DEBUG:boto:Token: None 2024-04-19T23:33:18.516 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:18.516 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:H91r2XwwNFfYJSYpL8QvMguZ8Uo= 2024-04-19T23:33:18.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:H91r2XwwNFfYJSYpL8QvMguZ8Uo=', 'Content-Length': '0'} 2024-04-19T23:33:18.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fc9188c17b266f6-006622ff3e-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.558 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.558 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.559 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:18.559 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:18.559 DEBUG:boto:Method: GET 2024-04-19T23:33:18.559 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:18.559 DEBUG:boto:Data: 2024-04-19T23:33:18.559 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.559 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.559 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.559 DEBUG:boto:Params: {} 2024-04-19T23:33:18.559 DEBUG:boto:Token: None 2024-04-19T23:33:18.560 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a95408aef2beca76-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.562 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.563 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.563 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.563 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:18.563 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:18.563 DEBUG:boto:Method: GET 2024-04-19T23:33:18.563 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:18.563 DEBUG:boto:Data: 2024-04-19T23:33:18.563 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.564 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:18.564 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.564 DEBUG:boto:Params: {} 2024-04-19T23:33:18.564 DEBUG:boto:Token: None 2024-04-19T23:33:18.564 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.564 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e0f1e276cce54fe-006622ff3e-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.567 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.568 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-19 zones={test-zone3, test-zone1} 2024-04-19T23:33:18.568 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-19 zones={test-zone3, test-zone2} 2024-04-19T23:33:18.568 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.568 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.568 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.568 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:33:18.568 DEBUG:boto:Data: 2024-04-19T23:33:18.568 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.568 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.568 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.568 DEBUG:boto:Params: {} 2024-04-19T23:33:18.569 DEBUG:boto:Token: None 2024-04-19T23:33:18.569 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.569 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DGou/C6nAy+hN/EAieS3MaUZoNk= 2024-04-19T23:33:18.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DGou/C6nAy+hN/EAieS3MaUZoNk=', 'Content-Length': '0'} 2024-04-19T23:33:18.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000201ac1ecf8f8b816-006622ff3e-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.571 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.571 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.571 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.572 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:33:18.572 DEBUG:boto:Data: 2024-04-19T23:33:18.572 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.572 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.572 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.572 DEBUG:boto:Params: {} 2024-04-19T23:33:18.572 DEBUG:boto:Token: None 2024-04-19T23:33:18.572 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.572 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DGou/C6nAy+hN/EAieS3MaUZoNk= 2024-04-19T23:33:18.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DGou/C6nAy+hN/EAieS3MaUZoNk=', 'Content-Length': '0'} 2024-04-19T23:33:18.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d974b58b94b1533-006622ff3e-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.575 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:33:18.575 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.575 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.575 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:18.575 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:18.575 DEBUG:boto:Method: GET 2024-04-19T23:33:18.576 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:33:18.576 DEBUG:boto:Data: 2024-04-19T23:33:18.576 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.576 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.576 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.576 DEBUG:boto:Params: {} 2024-04-19T23:33:18.576 DEBUG:boto:Token: None 2024-04-19T23:33:18.576 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/?versions 2024-04-19T23:33:18.576 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xWk/Ctxupl6Q05s5L/QKfMP5ZWA= 2024-04-19T23:33:18.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xWk/Ctxupl6Q05s5L/QKfMP5ZWA=', 'Content-Length': '0'} 2024-04-19T23:33:18.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beb61e29d21706f3-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.579 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.580 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:18.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:18.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:18.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:18.580 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:33:18.580 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.580 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.580 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:18.580 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:18.581 DEBUG:boto:Method: GET 2024-04-19T23:33:18.581 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:33:18.581 DEBUG:boto:Data: 2024-04-19T23:33:18.581 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.581 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.581 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.581 DEBUG:boto:Params: {} 2024-04-19T23:33:18.581 DEBUG:boto:Token: None 2024-04-19T23:33:18.581 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/?versions 2024-04-19T23:33:18.582 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xWk/Ctxupl6Q05s5L/QKfMP5ZWA= 2024-04-19T23:33:18.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xWk/Ctxupl6Q05s5L/QKfMP5ZWA=', 'Content-Length': '0'} 2024-04-19T23:33:18.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdbdf8b60f321812-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.584 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.585 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:18.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:18.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:18.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:18.585 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.585 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.585 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:18.585 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:18.585 DEBUG:boto:Method: GET 2024-04-19T23:33:18.585 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:33:18.585 DEBUG:boto:Data: 2024-04-19T23:33:18.585 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.586 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.586 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.586 DEBUG:boto:Params: {} 2024-04-19T23:33:18.586 DEBUG:boto:Token: None 2024-04-19T23:33:18.586 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/?versions 2024-04-19T23:33:18.586 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xWk/Ctxupl6Q05s5L/QKfMP5ZWA= 2024-04-19T23:33:18.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xWk/Ctxupl6Q05s5L/QKfMP5ZWA=', 'Content-Length': '0'} 2024-04-19T23:33:18.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b12dc58ba370336-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.589 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.589 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.589 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.589 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:18.590 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:33:18.590 DEBUG:boto:Method: GET 2024-04-19T23:33:18.590 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:33:18.590 DEBUG:boto:Data: 2024-04-19T23:33:18.590 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.590 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.590 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.590 DEBUG:boto:Params: {} 2024-04-19T23:33:18.590 DEBUG:boto:Token: None 2024-04-19T23:33:18.590 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/?versions 2024-04-19T23:33:18.591 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xWk/Ctxupl6Q05s5L/QKfMP5ZWA= 2024-04-19T23:33:18.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xWk/Ctxupl6Q05s5L/QKfMP5ZWA=', 'Content-Length': '0'} 2024-04-19T23:33:18.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e0dcc3d3f82eff1-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.593 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:18.594 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:18.594 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:18.594 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.594 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.594 DEBUG:boto:Method: GET 2024-04-19T23:33:18.594 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.595 DEBUG:boto:Data: 2024-04-19T23:33:18.595 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.595 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.595 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.595 DEBUG:boto:Params: {} 2024-04-19T23:33:18.595 DEBUG:boto:Token: None 2024-04-19T23:33:18.595 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.595 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DpipO5Avya15zAy7YRGlvk9RiN4= 2024-04-19T23:33:18.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DpipO5Avya15zAy7YRGlvk9RiN4=', 'Content-Length': '0'} 2024-04-19T23:33:18.597 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ea9c4664493547b0-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.597 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:18.597 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:18.598 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.598 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.598 DEBUG:boto:Method: GET 2024-04-19T23:33:18.598 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.598 DEBUG:boto:Data: 2024-04-19T23:33:18.598 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.598 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.598 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.599 DEBUG:boto:Params: {} 2024-04-19T23:33:18.599 DEBUG:boto:Token: None 2024-04-19T23:33:18.599 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.599 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DpipO5Avya15zAy7YRGlvk9RiN4= 2024-04-19T23:33:18.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DpipO5Avya15zAy7YRGlvk9RiN4=', 'Content-Length': '0'} 2024-04-19T23:33:18.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019b3af00607e33a8-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.601 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:18.601 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:18.601 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.602 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.602 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.602 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.602 DEBUG:boto:Data: 2024-04-19T23:33:18.602 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.602 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.602 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.602 DEBUG:boto:Params: {} 2024-04-19T23:33:18.602 DEBUG:boto:Token: None 2024-04-19T23:33:18.602 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.603 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0nfFyL4qFueQyvYw82JrjazNTFs= 2024-04-19T23:33:18.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0nfFyL4qFueQyvYw82JrjazNTFs=', 'Content-Length': '0'} 2024-04-19T23:33:18.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000090272d92b7a78340-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.605 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:18.605 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:18.605 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.605 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.605 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.605 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.605 DEBUG:boto:Data: 2024-04-19T23:33:18.605 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.605 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.605 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.606 DEBUG:boto:Params: {} 2024-04-19T23:33:18.606 DEBUG:boto:Token: None 2024-04-19T23:33:18.606 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.606 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0nfFyL4qFueQyvYw82JrjazNTFs= 2024-04-19T23:33:18.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0nfFyL4qFueQyvYw82JrjazNTFs=', 'Content-Length': '0'} 2024-04-19T23:33:18.608 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ccd027f838a2d842-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:18.608 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:18.608 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:18.608 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.608 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.608 DEBUG:boto:Method: GET 2024-04-19T23:33:18.609 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.609 DEBUG:boto:Data: 2024-04-19T23:33:18.609 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.609 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.609 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.609 DEBUG:boto:Params: {} 2024-04-19T23:33:18.609 DEBUG:boto:Token: None 2024-04-19T23:33:18.609 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.609 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DpipO5Avya15zAy7YRGlvk9RiN4= 2024-04-19T23:33:18.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DpipO5Avya15zAy7YRGlvk9RiN4=', 'Content-Length': '0'} 2024-04-19T23:33:18.611 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b81054888ffd12f2-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.611 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:18.611 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:18.612 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.612 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.612 DEBUG:boto:Method: GET 2024-04-19T23:33:18.612 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.612 DEBUG:boto:Data: 2024-04-19T23:33:18.612 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.612 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.612 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.612 DEBUG:boto:Params: {} 2024-04-19T23:33:18.612 DEBUG:boto:Token: None 2024-04-19T23:33:18.612 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.613 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DpipO5Avya15zAy7YRGlvk9RiN4= 2024-04-19T23:33:18.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DpipO5Avya15zAy7YRGlvk9RiN4=', 'Content-Length': '0'} 2024-04-19T23:33:18.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046e8b397932bbf9f-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.614 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.615 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.615 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:18.615 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:18.615 DEBUG:boto:Method: GET 2024-04-19T23:33:18.615 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:33:18.615 DEBUG:boto:Data: 2024-04-19T23:33:18.615 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.615 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.615 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.615 DEBUG:boto:Params: {} 2024-04-19T23:33:18.615 DEBUG:boto:Token: None 2024-04-19T23:33:18.616 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.616 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099202961387b434b-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.662 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.663 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.663 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.663 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:18.663 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:18.663 DEBUG:boto:Method: GET 2024-04-19T23:33:18.663 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:33:18.663 DEBUG:boto:Data: 2024-04-19T23:33:18.663 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.664 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.664 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.664 DEBUG:boto:Params: {} 2024-04-19T23:33:18.664 DEBUG:boto:Token: None 2024-04-19T23:33:18.664 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.664 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccc8d0a7f4d5ab1b-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.667 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.667 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:18.667 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:18.668 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.668 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:33:18.668 DEBUG:boto:Data: 2024-04-19T23:33:18.668 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.668 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.668 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.668 DEBUG:boto:Params: {} 2024-04-19T23:33:18.668 DEBUG:boto:Token: None 2024-04-19T23:33:18.668 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/dummy 2024-04-19T23:33:18.668 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CGqeCz35LbZ0dvbrrjD9D3JmQLE= 2024-04-19T23:33:18.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CGqeCz35LbZ0dvbrrjD9D3JmQLE=', 'Content-Length': '0'} 2024-04-19T23:33:18.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000001cce94b609d142c-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.671 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:18.671 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:18.671 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.671 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:33:18.671 DEBUG:boto:Data: 2024-04-19T23:33:18.672 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.672 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.672 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.672 DEBUG:boto:Params: {} 2024-04-19T23:33:18.672 DEBUG:boto:Token: None 2024-04-19T23:33:18.672 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/dummy 2024-04-19T23:33:18.672 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CGqeCz35LbZ0dvbrrjD9D3JmQLE= 2024-04-19T23:33:18.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CGqeCz35LbZ0dvbrrjD9D3JmQLE=', 'Content-Length': '0'} 2024-04-19T23:33:18.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000090dfc22d610dc9cc-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:18.675 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:18.675 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:18.675 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.675 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.675 DEBUG:boto:Method: GET 2024-04-19T23:33:18.676 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.676 DEBUG:boto:Data: 2024-04-19T23:33:18.676 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.676 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.676 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.676 DEBUG:boto:Params: {} 2024-04-19T23:33:18.676 DEBUG:boto:Token: None 2024-04-19T23:33:18.676 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DpipO5Avya15zAy7YRGlvk9RiN4= 2024-04-19T23:33:18.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DpipO5Avya15zAy7YRGlvk9RiN4=', 'Content-Length': '0'} 2024-04-19T23:33:18.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e72edfced0171c75-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.678 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:33:18.678 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:33:18.679 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.679 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.679 DEBUG:boto:Method: GET 2024-04-19T23:33:18.679 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.679 DEBUG:boto:Data: 2024-04-19T23:33:18.679 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.679 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.679 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.679 DEBUG:boto:Params: {} 2024-04-19T23:33:18.679 DEBUG:boto:Token: None 2024-04-19T23:33:18.680 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:33:18.680 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DpipO5Avya15zAy7YRGlvk9RiN4= 2024-04-19T23:33:18.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DpipO5Avya15zAy7YRGlvk9RiN4=', 'Content-Length': '0'} 2024-04-19T23:33:18.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001d0e2bc5267f170b-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.682 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.682 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.682 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:18.682 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:18.682 DEBUG:boto:Method: GET 2024-04-19T23:33:18.682 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:33:18.682 DEBUG:boto:Data: 2024-04-19T23:33:18.683 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.683 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.683 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.683 DEBUG:boto:Params: {} 2024-04-19T23:33:18.683 DEBUG:boto:Token: None 2024-04-19T23:33:18.683 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.683 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000201b88d7defc7905-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.686 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.686 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.686 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.686 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:18.687 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:33:18.687 DEBUG:boto:Method: GET 2024-04-19T23:33:18.687 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:33:18.687 DEBUG:boto:Data: 2024-04-19T23:33:18.687 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.687 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.687 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.687 DEBUG:boto:Params: {} 2024-04-19T23:33:18.688 DEBUG:boto:Token: None 2024-04-19T23:33:18.688 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.688 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000125cf6dae1340fa-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.734 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:18.735 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.735 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.735 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.735 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.735 DEBUG:boto:Method: GET 2024-04-19T23:33:18.735 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.735 DEBUG:boto:Data: 2024-04-19T23:33:18.735 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.736 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.736 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.736 DEBUG:boto:Params: {} 2024-04-19T23:33:18.736 DEBUG:boto:Token: None 2024-04-19T23:33:18.736 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.736 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x2Oej9s63PXGrlAj14V8m9PGR0o= 2024-04-19T23:33:18.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x2Oej9s63PXGrlAj14V8m9PGR0o=', 'Content-Length': '0'} 2024-04-19T23:33:18.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e56ba35d65b98d57-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.782 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.783 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.783 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.783 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.783 DEBUG:boto:Method: GET 2024-04-19T23:33:18.783 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.783 DEBUG:boto:Data: 2024-04-19T23:33:18.783 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.783 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.783 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.783 DEBUG:boto:Params: {} 2024-04-19T23:33:18.783 DEBUG:boto:Token: None 2024-04-19T23:33:18.784 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x2Oej9s63PXGrlAj14V8m9PGR0o= 2024-04-19T23:33:18.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x2Oej9s63PXGrlAj14V8m9PGR0o=', 'Content-Length': '0'} 2024-04-19T23:33:18.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052f64aa483651cbe-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.786 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.786 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.786 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.787 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.787 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.787 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.787 DEBUG:boto:Data: 2024-04-19T23:33:18.787 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.787 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.787 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.787 DEBUG:boto:Params: {} 2024-04-19T23:33:18.787 DEBUG:boto:Token: None 2024-04-19T23:33:18.787 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.788 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tWqGHs8esYLwZcgkj0X8bOB+PAA= 2024-04-19T23:33:18.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tWqGHs8esYLwZcgkj0X8bOB+PAA=', 'Content-Length': '0'} 2024-04-19T23:33:18.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a41e6d9502e1a19a-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.790 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.790 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.790 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.790 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.790 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.790 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.790 DEBUG:boto:Data: 2024-04-19T23:33:18.790 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.791 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.791 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.791 DEBUG:boto:Params: {} 2024-04-19T23:33:18.791 DEBUG:boto:Token: None 2024-04-19T23:33:18.791 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.791 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tWqGHs8esYLwZcgkj0X8bOB+PAA= 2024-04-19T23:33:18.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tWqGHs8esYLwZcgkj0X8bOB+PAA=', 'Content-Length': '0'} 2024-04-19T23:33:18.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d243295325ae852-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:18.793 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.793 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.793 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.794 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.794 DEBUG:boto:Method: GET 2024-04-19T23:33:18.794 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.794 DEBUG:boto:Data: 2024-04-19T23:33:18.794 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.794 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.794 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.794 DEBUG:boto:Params: {} 2024-04-19T23:33:18.794 DEBUG:boto:Token: None 2024-04-19T23:33:18.794 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.795 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x2Oej9s63PXGrlAj14V8m9PGR0o= 2024-04-19T23:33:18.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x2Oej9s63PXGrlAj14V8m9PGR0o=', 'Content-Length': '0'} 2024-04-19T23:33:18.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dc5857ece6a4c8d-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.838 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.839 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.839 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.839 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.839 DEBUG:boto:Method: GET 2024-04-19T23:33:18.839 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.839 DEBUG:boto:Data: 2024-04-19T23:33:18.839 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.839 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.839 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.839 DEBUG:boto:Params: {} 2024-04-19T23:33:18.840 DEBUG:boto:Token: None 2024-04-19T23:33:18.840 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.840 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x2Oej9s63PXGrlAj14V8m9PGR0o= 2024-04-19T23:33:18.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x2Oej9s63PXGrlAj14V8m9PGR0o=', 'Content-Length': '0'} 2024-04-19T23:33:18.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1dac500fc0fcc9f-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.882 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.883 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.883 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:18.883 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:18.883 DEBUG:boto:Method: GET 2024-04-19T23:33:18.883 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:18.883 DEBUG:boto:Data: 2024-04-19T23:33:18.883 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.883 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.883 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.883 DEBUG:boto:Params: {} 2024-04-19T23:33:18.884 DEBUG:boto:Token: None 2024-04-19T23:33:18.884 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.884 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a58ba07e4d7f956a-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.887 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.887 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.887 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.887 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:18.887 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:18.887 DEBUG:boto:Method: GET 2024-04-19T23:33:18.888 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:18.888 DEBUG:boto:Data: 2024-04-19T23:33:18.888 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.888 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.888 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.888 DEBUG:boto:Params: {} 2024-04-19T23:33:18.888 DEBUG:boto:Token: None 2024-04-19T23:33:18.888 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.888 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000040cc09b323f439b-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.891 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:18.891 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.891 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.891 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.892 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:33:18.892 DEBUG:boto:Data: 2024-04-19T23:33:18.892 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.892 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.892 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.892 DEBUG:boto:Params: {} 2024-04-19T23:33:18.892 DEBUG:boto:Token: None 2024-04-19T23:33:18.892 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:33:18.892 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RRdduqeI/6NJcIpYNuMQMsrTe9A= 2024-04-19T23:33:18.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RRdduqeI/6NJcIpYNuMQMsrTe9A=', 'Content-Length': '0'} 2024-04-19T23:33:18.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000746bc3edcc6b2c75-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.895 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.895 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.895 DEBUG:boto:Method: HEAD 2024-04-19T23:33:18.895 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:33:18.895 DEBUG:boto:Data: 2024-04-19T23:33:18.895 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.896 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.896 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.896 DEBUG:boto:Params: {} 2024-04-19T23:33:18.896 DEBUG:boto:Token: None 2024-04-19T23:33:18.896 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:33:18.896 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RRdduqeI/6NJcIpYNuMQMsrTe9A= 2024-04-19T23:33:18.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RRdduqeI/6NJcIpYNuMQMsrTe9A=', 'Content-Length': '0'} 2024-04-19T23:33:18.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f187c4384ad2664-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:18.899 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.899 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.899 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.899 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.899 DEBUG:boto:Method: GET 2024-04-19T23:33:18.899 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.899 DEBUG:boto:Data: 2024-04-19T23:33:18.899 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.899 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.899 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.900 DEBUG:boto:Params: {} 2024-04-19T23:33:18.900 DEBUG:boto:Token: None 2024-04-19T23:33:18.900 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.900 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x2Oej9s63PXGrlAj14V8m9PGR0o= 2024-04-19T23:33:18.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x2Oej9s63PXGrlAj14V8m9PGR0o=', 'Content-Length': '0'} 2024-04-19T23:33:18.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bcc36f97929bd8c0-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.943 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.943 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:33:18.943 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.943 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.943 DEBUG:boto:Method: GET 2024-04-19T23:33:18.943 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.943 DEBUG:boto:Data: 2024-04-19T23:33:18.943 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.943 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:18.943 DEBUG:boto:Port: 8000 2024-04-19T23:33:18.944 DEBUG:boto:Params: {} 2024-04-19T23:33:18.944 DEBUG:boto:Token: None 2024-04-19T23:33:18.944 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:33:18.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x2Oej9s63PXGrlAj14V8m9PGR0o= 2024-04-19T23:33:18.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x2Oej9s63PXGrlAj14V8m9PGR0o=', 'Content-Length': '0'} 2024-04-19T23:33:18.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001afedeb839a8bf4-006622ff3e-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:18.990 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:18.991 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:18.991 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:18.991 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:18.991 DEBUG:boto:Method: GET 2024-04-19T23:33:18.991 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:18.991 DEBUG:boto:Data: 2024-04-19T23:33:18.991 DEBUG:boto:Headers: {} 2024-04-19T23:33:18.991 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:18.991 DEBUG:boto:Port: 8001 2024-04-19T23:33:18.991 DEBUG:boto:Params: {} 2024-04-19T23:33:18.991 DEBUG:boto:Token: None 2024-04-19T23:33:18.992 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:18 GMT /aktlji-19/ 2024-04-19T23:33:18.992 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI= 2024-04-19T23:33:18.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gcx+t4YxsrQQvTALdvXkZFMySzI=', 'Content-Length': '0'} 2024-04-19T23:33:18.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035bf05ebc94034a5-006622ff3e-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.038 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:19.039 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:19.039 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:19.039 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:19.039 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:19.039 DEBUG:boto:Method: GET 2024-04-19T23:33:19.039 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:33:19.039 DEBUG:boto:Data: 2024-04-19T23:33:19.040 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.040 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.040 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.040 DEBUG:boto:Params: {} 2024-04-19T23:33:19.040 DEBUG:boto:Token: None 2024-04-19T23:33:19.040 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/ 2024-04-19T23:33:19.040 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q= 2024-04-19T23:33:19.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009553c31e07112562-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.043 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:19.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:19.043 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.043 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.044 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.044 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.044 DEBUG:boto:Method: GET 2024-04-19T23:33:19.044 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.044 DEBUG:boto:Data: 2024-04-19T23:33:19.044 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.044 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.044 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.044 DEBUG:boto:Params: {} 2024-04-19T23:33:19.044 DEBUG:boto:Token: None 2024-04-19T23:33:19.044 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.045 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CPAQDkJUUzATey9Zmrf7DTuBH/Q= 2024-04-19T23:33:19.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CPAQDkJUUzATey9Zmrf7DTuBH/Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080f2248b2eb400fc-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.047 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.047 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.047 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.047 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.047 DEBUG:boto:Method: GET 2024-04-19T23:33:19.047 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.047 DEBUG:boto:Data: 2024-04-19T23:33:19.047 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.047 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.047 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.047 DEBUG:boto:Params: {} 2024-04-19T23:33:19.048 DEBUG:boto:Token: None 2024-04-19T23:33:19.048 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.048 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CPAQDkJUUzATey9Zmrf7DTuBH/Q= 2024-04-19T23:33:19.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CPAQDkJUUzATey9Zmrf7DTuBH/Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002407ff3401e80949-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.050 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.050 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.050 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.050 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.050 DEBUG:boto:Method: HEAD 2024-04-19T23:33:19.050 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.051 DEBUG:boto:Data: 2024-04-19T23:33:19.051 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.051 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.051 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.051 DEBUG:boto:Params: {} 2024-04-19T23:33:19.051 DEBUG:boto:Token: None 2024-04-19T23:33:19.051 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.051 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a87CBViSQunF4Uvey09OOt5QLiI= 2024-04-19T23:33:19.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a87CBViSQunF4Uvey09OOt5QLiI=', 'Content-Length': '0'} 2024-04-19T23:33:19.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3a775ae9fa31568-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.053 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.054 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.054 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.054 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.054 DEBUG:boto:Method: HEAD 2024-04-19T23:33:19.054 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.054 DEBUG:boto:Data: 2024-04-19T23:33:19.054 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.054 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.054 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.054 DEBUG:boto:Params: {} 2024-04-19T23:33:19.055 DEBUG:boto:Token: None 2024-04-19T23:33:19.055 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.055 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a87CBViSQunF4Uvey09OOt5QLiI= 2024-04-19T23:33:19.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a87CBViSQunF4Uvey09OOt5QLiI=', 'Content-Length': '0'} 2024-04-19T23:33:19.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b17e10f0d131f5e-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:19.057 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.057 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.057 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.057 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.057 DEBUG:boto:Method: GET 2024-04-19T23:33:19.057 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.058 DEBUG:boto:Data: 2024-04-19T23:33:19.058 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.058 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.058 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.058 DEBUG:boto:Params: {} 2024-04-19T23:33:19.058 DEBUG:boto:Token: None 2024-04-19T23:33:19.058 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.058 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CPAQDkJUUzATey9Zmrf7DTuBH/Q= 2024-04-19T23:33:19.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CPAQDkJUUzATey9Zmrf7DTuBH/Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048cba80341d96b21-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.102 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.102 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.103 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.103 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.103 DEBUG:boto:Method: GET 2024-04-19T23:33:19.103 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.103 DEBUG:boto:Data: 2024-04-19T23:33:19.103 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.103 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.103 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.103 DEBUG:boto:Params: {} 2024-04-19T23:33:19.103 DEBUG:boto:Token: None 2024-04-19T23:33:19.104 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CPAQDkJUUzATey9Zmrf7DTuBH/Q= 2024-04-19T23:33:19.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CPAQDkJUUzATey9Zmrf7DTuBH/Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ac5772cc912e44e-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.106 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:19.106 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:19.106 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:19.106 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:19.106 DEBUG:boto:Method: GET 2024-04-19T23:33:19.107 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:19.107 DEBUG:boto:Data: 2024-04-19T23:33:19.107 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.107 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.107 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.107 DEBUG:boto:Params: {} 2024-04-19T23:33:19.107 DEBUG:boto:Token: None 2024-04-19T23:33:19.107 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/ 2024-04-19T23:33:19.107 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q= 2024-04-19T23:33:19.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003275f19d70bc2715-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.154 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:19.155 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:19.155 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:19.155 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:19.155 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:19.155 DEBUG:boto:Method: GET 2024-04-19T23:33:19.155 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:19.155 DEBUG:boto:Data: 2024-04-19T23:33:19.155 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.155 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.155 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.155 DEBUG:boto:Params: {} 2024-04-19T23:33:19.156 DEBUG:boto:Token: None 2024-04-19T23:33:19.156 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/ 2024-04-19T23:33:19.156 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q= 2024-04-19T23:33:19.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096af6fa454d9d928-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.158 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:19.159 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.159 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.159 DEBUG:boto:Method: HEAD 2024-04-19T23:33:19.159 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:33:19.159 DEBUG:boto:Data: 2024-04-19T23:33:19.159 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.159 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.159 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.159 DEBUG:boto:Params: {} 2024-04-19T23:33:19.160 DEBUG:boto:Token: None 2024-04-19T23:33:19.160 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:33:19.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PdnuN2uRirSng8q9mExzsqJ7J6s= 2024-04-19T23:33:19.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PdnuN2uRirSng8q9mExzsqJ7J6s=', 'Content-Length': '0'} 2024-04-19T23:33:19.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006df244c6c13b2de5-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.162 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.163 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.163 DEBUG:boto:Method: HEAD 2024-04-19T23:33:19.163 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:33:19.163 DEBUG:boto:Data: 2024-04-19T23:33:19.163 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.163 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.163 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.163 DEBUG:boto:Params: {} 2024-04-19T23:33:19.163 DEBUG:boto:Token: None 2024-04-19T23:33:19.163 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:33:19.164 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PdnuN2uRirSng8q9mExzsqJ7J6s= 2024-04-19T23:33:19.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PdnuN2uRirSng8q9mExzsqJ7J6s=', 'Content-Length': '0'} 2024-04-19T23:33:19.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ad4899743da1287-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:19.166 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.166 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.166 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.166 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.167 DEBUG:boto:Method: GET 2024-04-19T23:33:19.167 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.167 DEBUG:boto:Data: 2024-04-19T23:33:19.167 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.167 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.167 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.167 DEBUG:boto:Params: {} 2024-04-19T23:33:19.167 DEBUG:boto:Token: None 2024-04-19T23:33:19.168 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.168 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CPAQDkJUUzATey9Zmrf7DTuBH/Q= 2024-04-19T23:33:19.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CPAQDkJUUzATey9Zmrf7DTuBH/Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd9c497548c72c25-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.210 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.210 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:33:19.211 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.211 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.211 DEBUG:boto:Method: GET 2024-04-19T23:33:19.211 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.211 DEBUG:boto:Data: 2024-04-19T23:33:19.211 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.211 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.211 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.211 DEBUG:boto:Params: {} 2024-04-19T23:33:19.211 DEBUG:boto:Token: None 2024-04-19T23:33:19.212 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:33:19.212 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CPAQDkJUUzATey9Zmrf7DTuBH/Q= 2024-04-19T23:33:19.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CPAQDkJUUzATey9Zmrf7DTuBH/Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebb5ea2cb96e3a46-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.214 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:19.214 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:19.214 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:19.215 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:19.215 DEBUG:boto:Method: GET 2024-04-19T23:33:19.215 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:19.215 DEBUG:boto:Data: 2024-04-19T23:33:19.215 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.215 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.215 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.215 DEBUG:boto:Params: {} 2024-04-19T23:33:19.215 DEBUG:boto:Token: None 2024-04-19T23:33:19.215 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/ 2024-04-19T23:33:19.216 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q= 2024-04-19T23:33:19.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7ac249fee83e3f6-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.262 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:19.263 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:19.263 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:19.263 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:19.263 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:19.263 DEBUG:boto:Method: GET 2024-04-19T23:33:19.263 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:33:19.263 DEBUG:boto:Data: 2024-04-19T23:33:19.263 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.264 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.264 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.264 DEBUG:boto:Params: {} 2024-04-19T23:33:19.264 DEBUG:boto:Token: None 2024-04-19T23:33:19.264 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/ 2024-04-19T23:33:19.264 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q= 2024-04-19T23:33:19.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c3496bc305676fb-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.267 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:19.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:19.267 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.267 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.267 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.267 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.267 DEBUG:boto:Method: GET 2024-04-19T23:33:19.268 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.268 DEBUG:boto:Data: 2024-04-19T23:33:19.268 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.268 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.268 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.268 DEBUG:boto:Params: {} 2024-04-19T23:33:19.268 DEBUG:boto:Token: None 2024-04-19T23:33:19.268 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.268 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ccfUoa4cHv5AnQFL+lDZzmhWmm8= 2024-04-19T23:33:19.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ccfUoa4cHv5AnQFL+lDZzmhWmm8=', 'Content-Length': '0'} 2024-04-19T23:33:19.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f4ca93866c28e3d-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.271 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.271 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.271 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.271 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.271 DEBUG:boto:Method: GET 2024-04-19T23:33:19.271 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.271 DEBUG:boto:Data: 2024-04-19T23:33:19.271 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.271 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.271 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.271 DEBUG:boto:Params: {} 2024-04-19T23:33:19.272 DEBUG:boto:Token: None 2024-04-19T23:33:19.272 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.272 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ccfUoa4cHv5AnQFL+lDZzmhWmm8= 2024-04-19T23:33:19.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ccfUoa4cHv5AnQFL+lDZzmhWmm8=', 'Content-Length': '0'} 2024-04-19T23:33:19.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008174be44291e1bfc-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.274 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.274 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.275 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.275 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.275 DEBUG:boto:Method: HEAD 2024-04-19T23:33:19.275 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.275 DEBUG:boto:Data: 2024-04-19T23:33:19.275 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.275 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.275 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.275 DEBUG:boto:Params: {} 2024-04-19T23:33:19.275 DEBUG:boto:Token: None 2024-04-19T23:33:19.275 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.276 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BVnt45Gmz5EADDJ8yB4iqQtEU1o= 2024-04-19T23:33:19.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BVnt45Gmz5EADDJ8yB4iqQtEU1o=', 'Content-Length': '0'} 2024-04-19T23:33:19.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070dd8cc6ff164289-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.278 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.278 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.278 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.278 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.278 DEBUG:boto:Method: HEAD 2024-04-19T23:33:19.278 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.278 DEBUG:boto:Data: 2024-04-19T23:33:19.278 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.278 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.278 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.279 DEBUG:boto:Params: {} 2024-04-19T23:33:19.279 DEBUG:boto:Token: None 2024-04-19T23:33:19.279 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.279 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BVnt45Gmz5EADDJ8yB4iqQtEU1o= 2024-04-19T23:33:19.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BVnt45Gmz5EADDJ8yB4iqQtEU1o=', 'Content-Length': '0'} 2024-04-19T23:33:19.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6968db5520b9f71-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:19.281 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.281 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.281 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.282 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.282 DEBUG:boto:Method: GET 2024-04-19T23:33:19.282 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.282 DEBUG:boto:Data: 2024-04-19T23:33:19.282 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.282 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.282 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.282 DEBUG:boto:Params: {} 2024-04-19T23:33:19.282 DEBUG:boto:Token: None 2024-04-19T23:33:19.283 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.283 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ccfUoa4cHv5AnQFL+lDZzmhWmm8= 2024-04-19T23:33:19.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ccfUoa4cHv5AnQFL+lDZzmhWmm8=', 'Content-Length': '0'} 2024-04-19T23:33:19.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6c2a384edf422b1-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.285 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.285 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.285 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.285 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.285 DEBUG:boto:Method: GET 2024-04-19T23:33:19.285 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.285 DEBUG:boto:Data: 2024-04-19T23:33:19.285 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.285 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.285 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.285 DEBUG:boto:Params: {} 2024-04-19T23:33:19.286 DEBUG:boto:Token: None 2024-04-19T23:33:19.286 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.286 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ccfUoa4cHv5AnQFL+lDZzmhWmm8= 2024-04-19T23:33:19.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ccfUoa4cHv5AnQFL+lDZzmhWmm8=', 'Content-Length': '0'} 2024-04-19T23:33:19.288 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e6f6d293112dfe6-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.330 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:19.331 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:19.331 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:19.331 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:19.331 DEBUG:boto:Method: GET 2024-04-19T23:33:19.331 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:19.331 DEBUG:boto:Data: 2024-04-19T23:33:19.331 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.332 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.332 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.332 DEBUG:boto:Params: {} 2024-04-19T23:33:19.332 DEBUG:boto:Token: None 2024-04-19T23:33:19.332 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/ 2024-04-19T23:33:19.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q= 2024-04-19T23:33:19.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a9e89337209b266-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.335 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:19.335 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:19.335 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:19.335 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:19.335 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:19.336 DEBUG:boto:Method: GET 2024-04-19T23:33:19.336 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:19.336 DEBUG:boto:Data: 2024-04-19T23:33:19.336 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.336 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.336 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.336 DEBUG:boto:Params: {} 2024-04-19T23:33:19.336 DEBUG:boto:Token: None 2024-04-19T23:33:19.337 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/ 2024-04-19T23:33:19.337 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q= 2024-04-19T23:33:19.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000023e89ee26a8ecbf-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.382 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:19.383 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.383 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.383 DEBUG:boto:Method: HEAD 2024-04-19T23:33:19.383 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:33:19.383 DEBUG:boto:Data: 2024-04-19T23:33:19.383 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.383 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.383 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.383 DEBUG:boto:Params: {} 2024-04-19T23:33:19.384 DEBUG:boto:Token: None 2024-04-19T23:33:19.384 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:33:19.384 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IwSmgeZ5Ac/4yQZLZocQBQ6rpp0= 2024-04-19T23:33:19.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IwSmgeZ5Ac/4yQZLZocQBQ6rpp0=', 'Content-Length': '0'} 2024-04-19T23:33:19.388 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066249eb943d03d7d-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.388 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.388 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.388 DEBUG:boto:Method: HEAD 2024-04-19T23:33:19.388 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:33:19.388 DEBUG:boto:Data: 2024-04-19T23:33:19.389 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.389 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.389 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.389 DEBUG:boto:Params: {} 2024-04-19T23:33:19.389 DEBUG:boto:Token: None 2024-04-19T23:33:19.389 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:33:19.389 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IwSmgeZ5Ac/4yQZLZocQBQ6rpp0= 2024-04-19T23:33:19.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IwSmgeZ5Ac/4yQZLZocQBQ6rpp0=', 'Content-Length': '0'} 2024-04-19T23:33:19.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059e4ce7c11ce86a5-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:19.392 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.392 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.392 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.392 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.392 DEBUG:boto:Method: GET 2024-04-19T23:33:19.392 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.393 DEBUG:boto:Data: 2024-04-19T23:33:19.393 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.393 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.393 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.393 DEBUG:boto:Params: {} 2024-04-19T23:33:19.393 DEBUG:boto:Token: None 2024-04-19T23:33:19.393 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.393 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ccfUoa4cHv5AnQFL+lDZzmhWmm8= 2024-04-19T23:33:19.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ccfUoa4cHv5AnQFL+lDZzmhWmm8=', 'Content-Length': '0'} 2024-04-19T23:33:19.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f35848ce84de003-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.438 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.438 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:33:19.438 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.439 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.439 DEBUG:boto:Method: GET 2024-04-19T23:33:19.439 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.439 DEBUG:boto:Data: 2024-04-19T23:33:19.439 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.439 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.439 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.439 DEBUG:boto:Params: {} 2024-04-19T23:33:19.439 DEBUG:boto:Token: None 2024-04-19T23:33:19.439 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:33:19.440 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ccfUoa4cHv5AnQFL+lDZzmhWmm8= 2024-04-19T23:33:19.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ccfUoa4cHv5AnQFL+lDZzmhWmm8=', 'Content-Length': '0'} 2024-04-19T23:33:19.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092c636765b2cc3d9-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.482 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:19.482 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:19.483 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:19.483 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:19.483 DEBUG:boto:Method: GET 2024-04-19T23:33:19.483 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:19.483 DEBUG:boto:Data: 2024-04-19T23:33:19.483 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.483 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:19.483 DEBUG:boto:Port: 8001 2024-04-19T23:33:19.483 DEBUG:boto:Params: {} 2024-04-19T23:33:19.483 DEBUG:boto:Token: None 2024-04-19T23:33:19.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/ 2024-04-19T23:33:19.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q= 2024-04-19T23:33:19.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ca3678bc539a607-006622ff3f-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.486 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:19.487 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:33:19.487 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:33:19.487 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:19.487 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:19.487 DEBUG:boto:Method: GET 2024-04-19T23:33:19.487 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:33:19.487 DEBUG:boto:Data: 2024-04-19T23:33:19.487 DEBUG:boto:Headers: {} 2024-04-19T23:33:19.487 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:19.488 DEBUG:boto:Port: 8000 2024-04-19T23:33:19.488 DEBUG:boto:Params: {} 2024-04-19T23:33:19.488 DEBUG:boto:Token: None 2024-04-19T23:33:19.488 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:19 GMT /aktlji-19/ 2024-04-19T23:33:19.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q= 2024-04-19T23:33:19.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ce6eDcfN3g3KBKmKesegixP+86Q=', 'Content-Length': '0'} 2024-04-19T23:33:19.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006588d9c1e0fea6d2-006622ff3f-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:19.490 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:19.491 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-19 zones={test-zone3, test-zone2} 2024-04-19T23:33:19.491 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-20 2024-04-19T23:33:19.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:19.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:19.491 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:19.560 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:19.560 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:19.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:19.608+0000 7f24b54be780 1 waiting for incremental sync to catch up: 2024-04-19T23:33:19.603 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000010.29.6, , 00000000003.10.10, , 00000000005.13.11, , , , , ] 2024-04-19T23:33:19.603 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000010.29.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:33:24.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:24.609+0000 7f24b54be780 1 waiting for incremental sync to catch up: 2024-04-19T23:33:24.605 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000010.29.6, , 00000000003.10.10, , 00000000005.13.11, , , , , ] 2024-04-19T23:33:24.605 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000010.29.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:33:29.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:29.611+0000 7f24b54be780 1 waiting for incremental sync to catch up: 2024-04-19T23:33:29.607 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000010.29.6, , 00000000003.10.10, , 00000000005.13.11, , , , , ] 2024-04-19T23:33:29.607 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000010.29.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:33:34.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:34.612+0000 7f24b54be780 1 waiting for incremental sync to catch up: 2024-04-19T23:33:34.608 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000010.29.6, , 00000000003.10.10, , 00000000005.13.11, , , , , ] 2024-04-19T23:33:34.608 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000010.29.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:33:39.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:39.614+0000 7f24b54be780 1 waiting for incremental sync to catch up: 2024-04-19T23:33:39.609 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000010.29.6, , 00000000003.10.10, , 00000000005.13.11, , , , , ] 2024-04-19T23:33:39.609 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000010.29.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:33:44.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:44.615+0000 7f24b54be780 1 bucket sync caught up with source: 2024-04-19T23:33:44.610 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000010.29.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:33:44.610 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000010.29.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:33:44.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:44.615+0000 7f24b54be780 0 bucket checkpoint complete 2024-04-19T23:33:44.628 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-20 2024-04-19T23:33:44.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:44.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:44.628 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:44.695 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:44.695 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:44.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:44.740+0000 7f573d77f780 1 bucket sync caught up with source: 2024-04-19T23:33:44.736 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000010.29.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:33:44.736 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000010.29.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:33:44.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:44.740+0000 7f573d77f780 0 bucket checkpoint complete 2024-04-19T23:33:44.751 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-20 2024-04-19T23:33:44.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:44.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:44.751 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:44.814 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:44.814 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:44.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:44.859+0000 7fdcef41a780 1 bucket sync caught up with source: 2024-04-19T23:33:44.855 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.65.10, , 00000000010.99.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:33:44.855 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.65.10, , 00000000010.99.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:33:44.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:44.859+0000 7fdcef41a780 0 bucket checkpoint complete 2024-04-19T23:33:44.869 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-20 2024-04-19T23:33:44.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:44.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:44.870 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:44.932 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:44.932 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:44.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:44.976+0000 7ff1d1846780 1 bucket sync caught up with source: 2024-04-19T23:33:44.972 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.65.10, , 00000000010.99.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:33:44.972 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.65.10, , 00000000010.99.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:33:44.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:44.976+0000 7ff1d1846780 0 bucket checkpoint complete 2024-04-19T23:33:44.986 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-20 2024-04-19T23:33:44.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:44.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:44.986 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:45.049 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:45.049 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:45.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:45.094+0000 7f1182d4e780 1 bucket sync caught up with source: 2024-04-19T23:33:45.089 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.34.10, , 00000000003.32.10, , 00000000013.37.6, , , , , ] 2024-04-19T23:33:45.089 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.34.10, , 00000000003.32.10, , 00000000013.37.6, , , , , ] 2024-04-19T23:33:45.090 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:45.094+0000 7f1182d4e780 0 bucket checkpoint complete 2024-04-19T23:33:45.104 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-20 2024-04-19T23:33:45.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:45.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:45.104 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:45.170 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:45.170 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:45.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:45.216+0000 7efd47ed2780 1 bucket sync caught up with source: 2024-04-19T23:33:45.211 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000003.34.10, , 00000000003.32.10, , 00000000013.37.6, , , , , ] 2024-04-19T23:33:45.211 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000003.34.10, , 00000000003.32.10, , 00000000013.37.6, , , , , ] 2024-04-19T23:33:45.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:45.216+0000 7efd47ed2780 0 bucket checkpoint complete 2024-04-19T23:33:45.226 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-20 zones={test-zone2, test-zone1} 2024-04-19T23:33:45.227 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.227 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.227 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.227 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:33:45.227 DEBUG:boto:Data: 2024-04-19T23:33:45.227 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.227 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.227 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.227 DEBUG:boto:Params: {} 2024-04-19T23:33:45.228 DEBUG:boto:Token: None 2024-04-19T23:33:45.228 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.228 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nJGWP/4bMyJv3heeovApYGXd1Sk= 2024-04-19T23:33:45.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nJGWP/4bMyJv3heeovApYGXd1Sk=', 'Content-Length': '0'} 2024-04-19T23:33:45.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b003558795bdd166-006622ff59-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.231 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.231 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.231 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.231 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:33:45.231 DEBUG:boto:Data: 2024-04-19T23:33:45.231 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.231 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.231 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.231 DEBUG:boto:Params: {} 2024-04-19T23:33:45.232 DEBUG:boto:Token: None 2024-04-19T23:33:45.232 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.232 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nJGWP/4bMyJv3heeovApYGXd1Sk= 2024-04-19T23:33:45.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nJGWP/4bMyJv3heeovApYGXd1Sk=', 'Content-Length': '0'} 2024-04-19T23:33:45.234 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2da4819e601f0c1-006622ff59-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.234 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:33:45.235 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.235 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.235 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:45.235 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:45.235 DEBUG:boto:Method: GET 2024-04-19T23:33:45.235 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:33:45.235 DEBUG:boto:Data: 2024-04-19T23:33:45.235 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.235 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.236 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.236 DEBUG:boto:Params: {} 2024-04-19T23:33:45.236 DEBUG:boto:Token: None 2024-04-19T23:33:45.236 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/?versions 2024-04-19T23:33:45.236 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BbTfiibk5juqA8+9R9ZPYwmSduo= 2024-04-19T23:33:45.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BbTfiibk5juqA8+9R9ZPYwmSduo=', 'Content-Length': '0'} 2024-04-19T23:33:45.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000681d71d629c51ce5-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.239 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.239 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:45.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:45.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:45.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:45.240 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:33:45.240 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.240 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.240 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:45.240 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:45.240 DEBUG:boto:Method: GET 2024-04-19T23:33:45.240 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:33:45.240 DEBUG:boto:Data: 2024-04-19T23:33:45.241 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.241 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.241 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.241 DEBUG:boto:Params: {} 2024-04-19T23:33:45.241 DEBUG:boto:Token: None 2024-04-19T23:33:45.241 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/?versions 2024-04-19T23:33:45.241 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BbTfiibk5juqA8+9R9ZPYwmSduo= 2024-04-19T23:33:45.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BbTfiibk5juqA8+9R9ZPYwmSduo=', 'Content-Length': '0'} 2024-04-19T23:33:45.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adf7333ac8aebfb0-006622ff59-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.244 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.244 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:45.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:45.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:45.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:45.245 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.245 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.245 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:45.245 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:45.245 DEBUG:boto:Method: GET 2024-04-19T23:33:45.245 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:33:45.245 DEBUG:boto:Data: 2024-04-19T23:33:45.245 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.246 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.246 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.246 DEBUG:boto:Params: {} 2024-04-19T23:33:45.246 DEBUG:boto:Token: None 2024-04-19T23:33:45.246 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/?versions 2024-04-19T23:33:45.246 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BbTfiibk5juqA8+9R9ZPYwmSduo= 2024-04-19T23:33:45.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BbTfiibk5juqA8+9R9ZPYwmSduo=', 'Content-Length': '0'} 2024-04-19T23:33:45.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2bb27034f138a91-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.249 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.249 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.249 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.249 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:45.250 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:45.250 DEBUG:boto:Method: GET 2024-04-19T23:33:45.250 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:33:45.250 DEBUG:boto:Data: 2024-04-19T23:33:45.250 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.250 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.250 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.250 DEBUG:boto:Params: {} 2024-04-19T23:33:45.250 DEBUG:boto:Token: None 2024-04-19T23:33:45.251 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/?versions 2024-04-19T23:33:45.251 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BbTfiibk5juqA8+9R9ZPYwmSduo= 2024-04-19T23:33:45.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BbTfiibk5juqA8+9R9ZPYwmSduo=', 'Content-Length': '0'} 2024-04-19T23:33:45.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad98d4bce30cf4e3-006622ff59-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.253 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:45.254 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:45.254 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:45.254 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.254 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.255 DEBUG:boto:Method: GET 2024-04-19T23:33:45.255 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.255 DEBUG:boto:Data: 2024-04-19T23:33:45.255 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.255 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.255 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.255 DEBUG:boto:Params: {} 2024-04-19T23:33:45.255 DEBUG:boto:Token: None 2024-04-19T23:33:45.255 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.255 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jM+vmo8hkjXn1p7yyDWfFPqb0wQ= 2024-04-19T23:33:45.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jM+vmo8hkjXn1p7yyDWfFPqb0wQ=', 'Content-Length': '0'} 2024-04-19T23:33:45.257 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009bcc43ff0fd9900f-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.257 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:45.258 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:45.258 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.258 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.258 DEBUG:boto:Method: GET 2024-04-19T23:33:45.258 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.258 DEBUG:boto:Data: 2024-04-19T23:33:45.258 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.258 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.258 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.258 DEBUG:boto:Params: {} 2024-04-19T23:33:45.259 DEBUG:boto:Token: None 2024-04-19T23:33:45.259 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.259 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jM+vmo8hkjXn1p7yyDWfFPqb0wQ= 2024-04-19T23:33:45.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jM+vmo8hkjXn1p7yyDWfFPqb0wQ=', 'Content-Length': '0'} 2024-04-19T23:33:45.261 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003057cc0266c6f51b-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.261 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:45.261 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:45.261 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.261 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.262 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.262 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.262 DEBUG:boto:Data: 2024-04-19T23:33:45.262 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.262 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.262 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.262 DEBUG:boto:Params: {} 2024-04-19T23:33:45.262 DEBUG:boto:Token: None 2024-04-19T23:33:45.262 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.262 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MAXbzitWLnCPSEKn/Ykn1lskmE8= 2024-04-19T23:33:45.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MAXbzitWLnCPSEKn/Ykn1lskmE8=', 'Content-Length': '0'} 2024-04-19T23:33:45.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000339e8c9d8dc9f645-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.264 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:45.265 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:45.265 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.265 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.265 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.265 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.265 DEBUG:boto:Data: 2024-04-19T23:33:45.265 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.265 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.265 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.265 DEBUG:boto:Params: {} 2024-04-19T23:33:45.265 DEBUG:boto:Token: None 2024-04-19T23:33:45.266 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.266 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MAXbzitWLnCPSEKn/Ykn1lskmE8= 2024-04-19T23:33:45.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MAXbzitWLnCPSEKn/Ykn1lskmE8=', 'Content-Length': '0'} 2024-04-19T23:33:45.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b6118cde1c2b1a1d-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:45.269 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:45.269 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:45.269 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.269 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.269 DEBUG:boto:Method: GET 2024-04-19T23:33:45.269 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.269 DEBUG:boto:Data: 2024-04-19T23:33:45.269 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.269 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.270 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.270 DEBUG:boto:Params: {} 2024-04-19T23:33:45.270 DEBUG:boto:Token: None 2024-04-19T23:33:45.270 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.271 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jM+vmo8hkjXn1p7yyDWfFPqb0wQ= 2024-04-19T23:33:45.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jM+vmo8hkjXn1p7yyDWfFPqb0wQ=', 'Content-Length': '0'} 2024-04-19T23:33:45.272 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001755100bc5cd210d-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.273 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:45.273 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:45.273 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.273 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.273 DEBUG:boto:Method: GET 2024-04-19T23:33:45.273 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.273 DEBUG:boto:Data: 2024-04-19T23:33:45.273 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.273 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.274 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.274 DEBUG:boto:Params: {} 2024-04-19T23:33:45.274 DEBUG:boto:Token: None 2024-04-19T23:33:45.274 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.274 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jM+vmo8hkjXn1p7yyDWfFPqb0wQ= 2024-04-19T23:33:45.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jM+vmo8hkjXn1p7yyDWfFPqb0wQ=', 'Content-Length': '0'} 2024-04-19T23:33:45.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005db9f028bda5cd96-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.276 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.277 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.277 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:45.277 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:45.277 DEBUG:boto:Method: GET 2024-04-19T23:33:45.277 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:33:45.277 DEBUG:boto:Data: 2024-04-19T23:33:45.278 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.278 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.278 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.278 DEBUG:boto:Params: {} 2024-04-19T23:33:45.278 DEBUG:boto:Token: None 2024-04-19T23:33:45.278 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.278 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc5d6a46c066e285-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.322 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.323 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.323 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.323 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:45.323 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:45.323 DEBUG:boto:Method: GET 2024-04-19T23:33:45.323 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:33:45.323 DEBUG:boto:Data: 2024-04-19T23:33:45.323 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.324 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.324 DEBUG:boto:Params: {} 2024-04-19T23:33:45.324 DEBUG:boto:Token: None 2024-04-19T23:33:45.324 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a7aef9db028917d-006622ff59-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.327 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.327 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:45.327 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:45.327 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.328 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:33:45.328 DEBUG:boto:Data: 2024-04-19T23:33:45.328 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.328 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.328 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.328 DEBUG:boto:Params: {} 2024-04-19T23:33:45.328 DEBUG:boto:Token: None 2024-04-19T23:33:45.328 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/dummy 2024-04-19T23:33:45.329 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+w1zl0x2L6od2X7fXk5gsd8b2/8= 2024-04-19T23:33:45.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+w1zl0x2L6od2X7fXk5gsd8b2/8=', 'Content-Length': '0'} 2024-04-19T23:33:45.331 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d19f5863224d9d1f-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.331 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:45.331 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:45.332 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.332 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:33:45.332 DEBUG:boto:Data: 2024-04-19T23:33:45.332 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.332 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.332 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.332 DEBUG:boto:Params: {} 2024-04-19T23:33:45.332 DEBUG:boto:Token: None 2024-04-19T23:33:45.332 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/dummy 2024-04-19T23:33:45.333 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+w1zl0x2L6od2X7fXk5gsd8b2/8= 2024-04-19T23:33:45.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+w1zl0x2L6od2X7fXk5gsd8b2/8=', 'Content-Length': '0'} 2024-04-19T23:33:45.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000644536cfe70fb592-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:45.335 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:45.335 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:45.335 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.335 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.336 DEBUG:boto:Method: GET 2024-04-19T23:33:45.336 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.336 DEBUG:boto:Data: 2024-04-19T23:33:45.336 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.336 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.336 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.336 DEBUG:boto:Params: {} 2024-04-19T23:33:45.336 DEBUG:boto:Token: None 2024-04-19T23:33:45.336 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.336 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jM+vmo8hkjXn1p7yyDWfFPqb0wQ= 2024-04-19T23:33:45.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jM+vmo8hkjXn1p7yyDWfFPqb0wQ=', 'Content-Length': '0'} 2024-04-19T23:33:45.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000051119c0571ef43d7-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.339 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:45.339 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:45.339 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.339 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.339 DEBUG:boto:Method: GET 2024-04-19T23:33:45.339 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.339 DEBUG:boto:Data: 2024-04-19T23:33:45.339 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.339 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.339 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.340 DEBUG:boto:Params: {} 2024-04-19T23:33:45.340 DEBUG:boto:Token: None 2024-04-19T23:33:45.340 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:45.340 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jM+vmo8hkjXn1p7yyDWfFPqb0wQ= 2024-04-19T23:33:45.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jM+vmo8hkjXn1p7yyDWfFPqb0wQ=', 'Content-Length': '0'} 2024-04-19T23:33:45.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057af14df704bc7a4-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.342 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.342 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.342 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:45.343 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:45.343 DEBUG:boto:Method: GET 2024-04-19T23:33:45.343 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:33:45.343 DEBUG:boto:Data: 2024-04-19T23:33:45.343 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.343 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.343 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.343 DEBUG:boto:Params: {} 2024-04-19T23:33:45.343 DEBUG:boto:Token: None 2024-04-19T23:33:45.343 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d9dd2559241915c-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.391 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.391 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.391 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.391 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:45.391 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:45.392 DEBUG:boto:Method: GET 2024-04-19T23:33:45.392 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:33:45.392 DEBUG:boto:Data: 2024-04-19T23:33:45.392 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.392 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.392 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.392 DEBUG:boto:Params: {} 2024-04-19T23:33:45.392 DEBUG:boto:Token: None 2024-04-19T23:33:45.393 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.393 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70fa8dfcdf4ee3f-006622ff59-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.396 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:45.397 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.397 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.397 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.397 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.397 DEBUG:boto:Method: GET 2024-04-19T23:33:45.397 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.397 DEBUG:boto:Data: 2024-04-19T23:33:45.397 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.397 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.397 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.397 DEBUG:boto:Params: {} 2024-04-19T23:33:45.398 DEBUG:boto:Token: None 2024-04-19T23:33:45.398 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.398 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7SpEuPWYTj5LerMPJoxHEbLTM74= 2024-04-19T23:33:45.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7SpEuPWYTj5LerMPJoxHEbLTM74=', 'Content-Length': '0'} 2024-04-19T23:33:45.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097cd607df9c0c5ac-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.400 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.400 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.401 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.401 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.401 DEBUG:boto:Method: GET 2024-04-19T23:33:45.401 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.401 DEBUG:boto:Data: 2024-04-19T23:33:45.401 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.401 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.401 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.401 DEBUG:boto:Params: {} 2024-04-19T23:33:45.401 DEBUG:boto:Token: None 2024-04-19T23:33:45.402 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.402 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7SpEuPWYTj5LerMPJoxHEbLTM74= 2024-04-19T23:33:45.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7SpEuPWYTj5LerMPJoxHEbLTM74=', 'Content-Length': '0'} 2024-04-19T23:33:45.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d38d2b3ae3f1198a-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.446 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.446 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.447 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.447 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.447 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.447 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.447 DEBUG:boto:Data: 2024-04-19T23:33:45.447 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.447 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.447 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.447 DEBUG:boto:Params: {} 2024-04-19T23:33:45.448 DEBUG:boto:Token: None 2024-04-19T23:33:45.448 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.448 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZMzDDWiaqtjixOjYLckI2sYsVVk= 2024-04-19T23:33:45.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZMzDDWiaqtjixOjYLckI2sYsVVk=', 'Content-Length': '0'} 2024-04-19T23:33:45.450 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db1b93747270ae69-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.450 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.450 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.450 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.450 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.451 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.451 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.451 DEBUG:boto:Data: 2024-04-19T23:33:45.451 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.451 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.451 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.451 DEBUG:boto:Params: {} 2024-04-19T23:33:45.451 DEBUG:boto:Token: None 2024-04-19T23:33:45.452 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.452 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZMzDDWiaqtjixOjYLckI2sYsVVk= 2024-04-19T23:33:45.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZMzDDWiaqtjixOjYLckI2sYsVVk=', 'Content-Length': '0'} 2024-04-19T23:33:45.453 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9f574183b5e9275-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:45.454 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.454 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.454 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.454 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.454 DEBUG:boto:Method: GET 2024-04-19T23:33:45.455 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.455 DEBUG:boto:Data: 2024-04-19T23:33:45.455 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.455 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.455 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.455 DEBUG:boto:Params: {} 2024-04-19T23:33:45.455 DEBUG:boto:Token: None 2024-04-19T23:33:45.455 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.455 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7SpEuPWYTj5LerMPJoxHEbLTM74= 2024-04-19T23:33:45.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7SpEuPWYTj5LerMPJoxHEbLTM74=', 'Content-Length': '0'} 2024-04-19T23:33:45.457 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d7d650b30f477b9-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.498 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.499 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.499 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.499 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.499 DEBUG:boto:Method: GET 2024-04-19T23:33:45.499 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.499 DEBUG:boto:Data: 2024-04-19T23:33:45.499 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.499 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.499 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.499 DEBUG:boto:Params: {} 2024-04-19T23:33:45.500 DEBUG:boto:Token: None 2024-04-19T23:33:45.500 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.500 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7SpEuPWYTj5LerMPJoxHEbLTM74= 2024-04-19T23:33:45.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7SpEuPWYTj5LerMPJoxHEbLTM74=', 'Content-Length': '0'} 2024-04-19T23:33:45.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094cfcf3a5b1f3a98-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.502 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.502 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.502 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:45.502 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:45.503 DEBUG:boto:Method: GET 2024-04-19T23:33:45.503 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:45.503 DEBUG:boto:Data: 2024-04-19T23:33:45.503 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.503 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.503 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.503 DEBUG:boto:Params: {} 2024-04-19T23:33:45.503 DEBUG:boto:Token: None 2024-04-19T23:33:45.503 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.504 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d855b8cfe65d4a84-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.506 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.507 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.507 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.507 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:45.507 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:45.507 DEBUG:boto:Method: GET 2024-04-19T23:33:45.507 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:45.507 DEBUG:boto:Data: 2024-04-19T23:33:45.507 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.507 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.507 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.507 DEBUG:boto:Params: {} 2024-04-19T23:33:45.508 DEBUG:boto:Token: None 2024-04-19T23:33:45.508 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.508 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009706960e06b2f080-006622ff59-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.510 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.511 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.511 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.511 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.511 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:33:45.511 DEBUG:boto:Data: 2024-04-19T23:33:45.511 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.511 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.511 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.511 DEBUG:boto:Params: {} 2024-04-19T23:33:45.511 DEBUG:boto:Token: None 2024-04-19T23:33:45.511 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:33:45.512 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:33A/limRNBMTCCJO/VlXU/8ZpsY= 2024-04-19T23:33:45.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:33A/limRNBMTCCJO/VlXU/8ZpsY=', 'Content-Length': '0'} 2024-04-19T23:33:45.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007178e0ac27eab25a-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.514 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.514 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.514 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.514 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:33:45.515 DEBUG:boto:Data: 2024-04-19T23:33:45.515 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.515 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.515 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.515 DEBUG:boto:Params: {} 2024-04-19T23:33:45.515 DEBUG:boto:Token: None 2024-04-19T23:33:45.515 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:33:45.515 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:33A/limRNBMTCCJO/VlXU/8ZpsY= 2024-04-19T23:33:45.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:33A/limRNBMTCCJO/VlXU/8ZpsY=', 'Content-Length': '0'} 2024-04-19T23:33:45.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de96eeff93f18acf-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:45.518 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.518 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.518 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.518 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.518 DEBUG:boto:Method: GET 2024-04-19T23:33:45.518 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.519 DEBUG:boto:Data: 2024-04-19T23:33:45.519 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.519 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.519 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.519 DEBUG:boto:Params: {} 2024-04-19T23:33:45.519 DEBUG:boto:Token: None 2024-04-19T23:33:45.519 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.519 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7SpEuPWYTj5LerMPJoxHEbLTM74= 2024-04-19T23:33:45.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7SpEuPWYTj5LerMPJoxHEbLTM74=', 'Content-Length': '0'} 2024-04-19T23:33:45.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da39ef68ba455faf-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.566 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.567 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:45.567 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.567 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.567 DEBUG:boto:Method: GET 2024-04-19T23:33:45.567 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.568 DEBUG:boto:Data: 2024-04-19T23:33:45.568 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.568 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.568 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.569 DEBUG:boto:Params: {} 2024-04-19T23:33:45.569 DEBUG:boto:Token: None 2024-04-19T23:33:45.569 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:45.569 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7SpEuPWYTj5LerMPJoxHEbLTM74= 2024-04-19T23:33:45.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7SpEuPWYTj5LerMPJoxHEbLTM74=', 'Content-Length': '0'} 2024-04-19T23:33:45.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b60f97c69b52dff-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.572 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.572 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.573 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:45.573 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:45.573 DEBUG:boto:Method: GET 2024-04-19T23:33:45.573 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:45.573 DEBUG:boto:Data: 2024-04-19T23:33:45.574 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.574 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.574 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.574 DEBUG:boto:Params: {} 2024-04-19T23:33:45.574 DEBUG:boto:Token: None 2024-04-19T23:33:45.574 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.574 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079b25c2b72baac25-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.577 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.577 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.577 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.577 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:45.578 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:45.578 DEBUG:boto:Method: GET 2024-04-19T23:33:45.578 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:45.578 DEBUG:boto:Data: 2024-04-19T23:33:45.578 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.578 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.578 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.579 DEBUG:boto:Params: {} 2024-04-19T23:33:45.579 DEBUG:boto:Token: None 2024-04-19T23:33:45.579 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.579 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085ac2e583f7ec217-006622ff59-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.622 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:45.623 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.623 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.624 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.624 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.624 DEBUG:boto:Method: GET 2024-04-19T23:33:45.624 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.624 DEBUG:boto:Data: 2024-04-19T23:33:45.624 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.624 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.625 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.625 DEBUG:boto:Params: {} 2024-04-19T23:33:45.625 DEBUG:boto:Token: None 2024-04-19T23:33:45.625 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.626 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UPevCqrtCMrFjfKfQUYQOOMMHv0= 2024-04-19T23:33:45.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UPevCqrtCMrFjfKfQUYQOOMMHv0=', 'Content-Length': '0'} 2024-04-19T23:33:45.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000760e10a65297c4f7-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.628 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.628 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.628 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.629 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.629 DEBUG:boto:Method: GET 2024-04-19T23:33:45.629 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.629 DEBUG:boto:Data: 2024-04-19T23:33:45.629 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.629 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.630 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.630 DEBUG:boto:Params: {} 2024-04-19T23:33:45.630 DEBUG:boto:Token: None 2024-04-19T23:33:45.630 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.630 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UPevCqrtCMrFjfKfQUYQOOMMHv0= 2024-04-19T23:33:45.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UPevCqrtCMrFjfKfQUYQOOMMHv0=', 'Content-Length': '0'} 2024-04-19T23:33:45.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058cc3157a847ad1e-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.633 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.634 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.634 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.634 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.634 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.635 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.635 DEBUG:boto:Data: 2024-04-19T23:33:45.635 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.635 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.635 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.635 DEBUG:boto:Params: {} 2024-04-19T23:33:45.635 DEBUG:boto:Token: None 2024-04-19T23:33:45.635 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.636 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ekvJwyvH1G481JXsGvCp01KbgbQ= 2024-04-19T23:33:45.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ekvJwyvH1G481JXsGvCp01KbgbQ=', 'Content-Length': '0'} 2024-04-19T23:33:45.638 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8d496c6169da199-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.638 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.638 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.639 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.639 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.639 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.639 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.639 DEBUG:boto:Data: 2024-04-19T23:33:45.639 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.639 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.640 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.640 DEBUG:boto:Params: {} 2024-04-19T23:33:45.640 DEBUG:boto:Token: None 2024-04-19T23:33:45.641 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.641 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ekvJwyvH1G481JXsGvCp01KbgbQ= 2024-04-19T23:33:45.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ekvJwyvH1G481JXsGvCp01KbgbQ=', 'Content-Length': '0'} 2024-04-19T23:33:45.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062cea235ccbdf18e-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:45.644 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.644 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.644 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.644 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.644 DEBUG:boto:Method: GET 2024-04-19T23:33:45.644 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.645 DEBUG:boto:Data: 2024-04-19T23:33:45.645 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.645 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.645 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.645 DEBUG:boto:Params: {} 2024-04-19T23:33:45.646 DEBUG:boto:Token: None 2024-04-19T23:33:45.646 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.646 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UPevCqrtCMrFjfKfQUYQOOMMHv0= 2024-04-19T23:33:45.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UPevCqrtCMrFjfKfQUYQOOMMHv0=', 'Content-Length': '0'} 2024-04-19T23:33:45.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000508553f8008386e0-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.648 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.648 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.648 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.649 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.649 DEBUG:boto:Method: GET 2024-04-19T23:33:45.649 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.649 DEBUG:boto:Data: 2024-04-19T23:33:45.649 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.649 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.649 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.650 DEBUG:boto:Params: {} 2024-04-19T23:33:45.650 DEBUG:boto:Token: None 2024-04-19T23:33:45.650 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.650 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UPevCqrtCMrFjfKfQUYQOOMMHv0= 2024-04-19T23:33:45.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UPevCqrtCMrFjfKfQUYQOOMMHv0=', 'Content-Length': '0'} 2024-04-19T23:33:45.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006758f971e75f2334-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.698 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.699 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.699 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:45.699 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:45.699 DEBUG:boto:Method: GET 2024-04-19T23:33:45.699 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:45.699 DEBUG:boto:Data: 2024-04-19T23:33:45.699 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.700 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.700 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.700 DEBUG:boto:Params: {} 2024-04-19T23:33:45.700 DEBUG:boto:Token: None 2024-04-19T23:33:45.701 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.701 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efae78050fdd086e-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.704 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.704 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.705 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.705 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:45.705 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:45.705 DEBUG:boto:Method: GET 2024-04-19T23:33:45.705 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:45.705 DEBUG:boto:Data: 2024-04-19T23:33:45.706 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.706 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.706 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.706 DEBUG:boto:Params: {} 2024-04-19T23:33:45.707 DEBUG:boto:Token: None 2024-04-19T23:33:45.707 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.707 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076838c0492ae889b-006622ff59-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.750 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.751 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.751 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.752 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.752 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:33:45.752 DEBUG:boto:Data: 2024-04-19T23:33:45.752 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.752 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.752 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.752 DEBUG:boto:Params: {} 2024-04-19T23:33:45.752 DEBUG:boto:Token: None 2024-04-19T23:33:45.753 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:33:45.753 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+uX/VgfE4nayjxuqsj0Zcvufeno= 2024-04-19T23:33:45.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+uX/VgfE4nayjxuqsj0Zcvufeno=', 'Content-Length': '0'} 2024-04-19T23:33:45.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005694533a6d785dd0-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.756 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.757 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.757 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.757 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:33:45.757 DEBUG:boto:Data: 2024-04-19T23:33:45.758 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.758 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.758 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.758 DEBUG:boto:Params: {} 2024-04-19T23:33:45.758 DEBUG:boto:Token: None 2024-04-19T23:33:45.758 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:33:45.758 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+uX/VgfE4nayjxuqsj0Zcvufeno= 2024-04-19T23:33:45.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+uX/VgfE4nayjxuqsj0Zcvufeno=', 'Content-Length': '0'} 2024-04-19T23:33:45.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000453af4faae684d84-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:45.761 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.761 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.761 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.762 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.762 DEBUG:boto:Method: GET 2024-04-19T23:33:45.762 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.762 DEBUG:boto:Data: 2024-04-19T23:33:45.762 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.763 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.763 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.763 DEBUG:boto:Params: {} 2024-04-19T23:33:45.763 DEBUG:boto:Token: None 2024-04-19T23:33:45.764 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.764 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UPevCqrtCMrFjfKfQUYQOOMMHv0= 2024-04-19T23:33:45.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UPevCqrtCMrFjfKfQUYQOOMMHv0=', 'Content-Length': '0'} 2024-04-19T23:33:45.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8ed92c986cc559f-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.806 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.807 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:45.807 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.807 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.807 DEBUG:boto:Method: GET 2024-04-19T23:33:45.807 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.807 DEBUG:boto:Data: 2024-04-19T23:33:45.807 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.808 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.808 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.808 DEBUG:boto:Params: {} 2024-04-19T23:33:45.808 DEBUG:boto:Token: None 2024-04-19T23:33:45.808 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:45.809 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UPevCqrtCMrFjfKfQUYQOOMMHv0= 2024-04-19T23:33:45.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UPevCqrtCMrFjfKfQUYQOOMMHv0=', 'Content-Length': '0'} 2024-04-19T23:33:45.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004403793ef00d9592-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.854 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.855 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.855 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:45.855 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:45.855 DEBUG:boto:Method: GET 2024-04-19T23:33:45.855 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:45.855 DEBUG:boto:Data: 2024-04-19T23:33:45.855 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.856 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.856 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.856 DEBUG:boto:Params: {} 2024-04-19T23:33:45.856 DEBUG:boto:Token: None 2024-04-19T23:33:45.856 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.856 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f845d5238044978f-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.860 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.860 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.861 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.861 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:45.861 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:45.861 DEBUG:boto:Method: GET 2024-04-19T23:33:45.861 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:45.861 DEBUG:boto:Data: 2024-04-19T23:33:45.861 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.861 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.862 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.862 DEBUG:boto:Params: {} 2024-04-19T23:33:45.862 DEBUG:boto:Token: None 2024-04-19T23:33:45.862 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.863 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dba7df9b65f6a73f-006622ff59-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.865 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:45.866 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.866 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.867 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.867 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.867 DEBUG:boto:Method: GET 2024-04-19T23:33:45.867 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.867 DEBUG:boto:Data: 2024-04-19T23:33:45.867 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.867 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.868 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.868 DEBUG:boto:Params: {} 2024-04-19T23:33:45.868 DEBUG:boto:Token: None 2024-04-19T23:33:45.868 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.868 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bLE6HCH+yKYjpfVFl2Oi4j9duA8= 2024-04-19T23:33:45.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bLE6HCH+yKYjpfVFl2Oi4j9duA8=', 'Content-Length': '0'} 2024-04-19T23:33:45.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cfd37b194b3a5d8-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.871 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.871 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.871 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.871 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.871 DEBUG:boto:Method: GET 2024-04-19T23:33:45.871 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.872 DEBUG:boto:Data: 2024-04-19T23:33:45.872 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.872 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.872 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.872 DEBUG:boto:Params: {} 2024-04-19T23:33:45.872 DEBUG:boto:Token: None 2024-04-19T23:33:45.873 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.873 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bLE6HCH+yKYjpfVFl2Oi4j9duA8= 2024-04-19T23:33:45.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bLE6HCH+yKYjpfVFl2Oi4j9duA8=', 'Content-Length': '0'} 2024-04-19T23:33:45.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da21d6f6decbf8a2-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.875 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.875 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.875 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.876 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.876 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.876 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.876 DEBUG:boto:Data: 2024-04-19T23:33:45.876 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.876 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.876 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.876 DEBUG:boto:Params: {} 2024-04-19T23:33:45.876 DEBUG:boto:Token: None 2024-04-19T23:33:45.876 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.877 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vmFEOcIJqKLYWq9b1xwwDBRKtkw= 2024-04-19T23:33:45.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vmFEOcIJqKLYWq9b1xwwDBRKtkw=', 'Content-Length': '0'} 2024-04-19T23:33:45.878 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003249b36ac01e54cf-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.879 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.879 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.879 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.879 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.879 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.879 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.879 DEBUG:boto:Data: 2024-04-19T23:33:45.879 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.880 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.880 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.880 DEBUG:boto:Params: {} 2024-04-19T23:33:45.880 DEBUG:boto:Token: None 2024-04-19T23:33:45.880 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vmFEOcIJqKLYWq9b1xwwDBRKtkw= 2024-04-19T23:33:45.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vmFEOcIJqKLYWq9b1xwwDBRKtkw=', 'Content-Length': '0'} 2024-04-19T23:33:45.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049943d8720573017-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:45.883 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.883 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.883 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.883 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.883 DEBUG:boto:Method: GET 2024-04-19T23:33:45.883 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.883 DEBUG:boto:Data: 2024-04-19T23:33:45.883 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.883 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.883 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.883 DEBUG:boto:Params: {} 2024-04-19T23:33:45.884 DEBUG:boto:Token: None 2024-04-19T23:33:45.884 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.884 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bLE6HCH+yKYjpfVFl2Oi4j9duA8= 2024-04-19T23:33:45.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bLE6HCH+yKYjpfVFl2Oi4j9duA8=', 'Content-Length': '0'} 2024-04-19T23:33:45.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000310fc076be0b3d05-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.926 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.926 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.926 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.927 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.927 DEBUG:boto:Method: GET 2024-04-19T23:33:45.927 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.927 DEBUG:boto:Data: 2024-04-19T23:33:45.927 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.927 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.927 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.927 DEBUG:boto:Params: {} 2024-04-19T23:33:45.928 DEBUG:boto:Token: None 2024-04-19T23:33:45.928 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.928 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bLE6HCH+yKYjpfVFl2Oi4j9duA8= 2024-04-19T23:33:45.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bLE6HCH+yKYjpfVFl2Oi4j9duA8=', 'Content-Length': '0'} 2024-04-19T23:33:45.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff48e8f71a4dd609-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.970 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.971 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.971 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:45.971 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:45.971 DEBUG:boto:Method: GET 2024-04-19T23:33:45.971 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:45.971 DEBUG:boto:Data: 2024-04-19T23:33:45.971 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.971 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.972 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.972 DEBUG:boto:Params: {} 2024-04-19T23:33:45.972 DEBUG:boto:Token: None 2024-04-19T23:33:45.972 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.972 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096164fefcd5d50de-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.975 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.976 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:45.976 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:45.976 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:45.976 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:45.976 DEBUG:boto:Method: GET 2024-04-19T23:33:45.976 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:45.976 DEBUG:boto:Data: 2024-04-19T23:33:45.976 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.976 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.976 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.977 DEBUG:boto:Params: {} 2024-04-19T23:33:45.977 DEBUG:boto:Token: None 2024-04-19T23:33:45.977 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/ 2024-04-19T23:33:45.977 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M= 2024-04-19T23:33:45.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZdYYe22zvfZ5RjWCS0EAPHBP1M=', 'Content-Length': '0'} 2024-04-19T23:33:45.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d616078ba83e3e8e-006622ff59-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.980 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:45.980 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.980 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.981 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.981 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:33:45.981 DEBUG:boto:Data: 2024-04-19T23:33:45.981 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.981 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.981 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.981 DEBUG:boto:Params: {} 2024-04-19T23:33:45.981 DEBUG:boto:Token: None 2024-04-19T23:33:45.981 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:33:45.982 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o7pgn8pl92nFTMY3s+jjKc+H8LI= 2024-04-19T23:33:45.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o7pgn8pl92nFTMY3s+jjKc+H8LI=', 'Content-Length': '0'} 2024-04-19T23:33:45.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c908a71f8a11c5f-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.985 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.985 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.985 DEBUG:boto:Method: HEAD 2024-04-19T23:33:45.985 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:33:45.986 DEBUG:boto:Data: 2024-04-19T23:33:45.986 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.986 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:45.986 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.986 DEBUG:boto:Params: {} 2024-04-19T23:33:45.986 DEBUG:boto:Token: None 2024-04-19T23:33:45.986 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:33:45.986 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o7pgn8pl92nFTMY3s+jjKc+H8LI= 2024-04-19T23:33:45.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o7pgn8pl92nFTMY3s+jjKc+H8LI=', 'Content-Length': '0'} 2024-04-19T23:33:45.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc802b3d93310f0f-006622ff59-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:45.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:45.989 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.989 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:45.989 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.989 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.990 DEBUG:boto:Method: GET 2024-04-19T23:33:45.990 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.990 DEBUG:boto:Data: 2024-04-19T23:33:45.990 DEBUG:boto:Headers: {} 2024-04-19T23:33:45.990 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:45.990 DEBUG:boto:Port: 8000 2024-04-19T23:33:45.990 DEBUG:boto:Params: {} 2024-04-19T23:33:45.990 DEBUG:boto:Token: None 2024-04-19T23:33:45.991 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:45 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:45.991 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bLE6HCH+yKYjpfVFl2Oi4j9duA8= 2024-04-19T23:33:45.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:45 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bLE6HCH+yKYjpfVFl2Oi4j9duA8=', 'Content-Length': '0'} 2024-04-19T23:33:45.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000950d38af59d695a9-006622ff59-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.038 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.038 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.038 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.039 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.039 DEBUG:boto:Method: GET 2024-04-19T23:33:46.039 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.039 DEBUG:boto:Data: 2024-04-19T23:33:46.039 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.039 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.039 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.039 DEBUG:boto:Params: {} 2024-04-19T23:33:46.039 DEBUG:boto:Token: None 2024-04-19T23:33:46.039 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.040 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Liz7IzWwEFHu2UjB+X/obQzipkk= 2024-04-19T23:33:46.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Liz7IzWwEFHu2UjB+X/obQzipkk=', 'Content-Length': '0'} 2024-04-19T23:33:46.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b1f26c4e7d8b379-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.086 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.086 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.087 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:46.087 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:46.087 DEBUG:boto:Method: GET 2024-04-19T23:33:46.087 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:46.087 DEBUG:boto:Data: 2024-04-19T23:33:46.087 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.087 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:46.087 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.088 DEBUG:boto:Params: {} 2024-04-19T23:33:46.088 DEBUG:boto:Token: None 2024-04-19T23:33:46.088 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.088 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5f3db9f21e344e8-006622ff5a-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.091 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.091 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.091 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.091 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:46.091 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:46.092 DEBUG:boto:Method: GET 2024-04-19T23:33:46.092 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:46.092 DEBUG:boto:Data: 2024-04-19T23:33:46.092 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.092 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.092 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.092 DEBUG:boto:Params: {} 2024-04-19T23:33:46.092 DEBUG:boto:Token: None 2024-04-19T23:33:46.092 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.092 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a84a7edbc475b7c2-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.142 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.143 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-20 zones={test-zone2, test-zone1} 2024-04-19T23:33:46.143 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-20 zones={test-zone3, test-zone1} 2024-04-19T23:33:46.143 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.143 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.143 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.143 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:33:46.143 DEBUG:boto:Data: 2024-04-19T23:33:46.143 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.144 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.144 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.144 DEBUG:boto:Params: {} 2024-04-19T23:33:46.144 DEBUG:boto:Token: None 2024-04-19T23:33:46.144 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.144 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:124UhsEKapytWWXTtTK7k23y0sg= 2024-04-19T23:33:46.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:124UhsEKapytWWXTtTK7k23y0sg=', 'Content-Length': '0'} 2024-04-19T23:33:46.147 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed2a3618a7e5f1fc-006622ff5a-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.147 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.147 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.147 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.147 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:33:46.147 DEBUG:boto:Data: 2024-04-19T23:33:46.147 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.147 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.147 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.148 DEBUG:boto:Params: {} 2024-04-19T23:33:46.148 DEBUG:boto:Token: None 2024-04-19T23:33:46.148 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.148 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:124UhsEKapytWWXTtTK7k23y0sg= 2024-04-19T23:33:46.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:124UhsEKapytWWXTtTK7k23y0sg=', 'Content-Length': '0'} 2024-04-19T23:33:46.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee63087795f95fb3-006622ff5a-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.150 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:33:46.151 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.151 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.151 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:46.151 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:46.151 DEBUG:boto:Method: GET 2024-04-19T23:33:46.151 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:33:46.151 DEBUG:boto:Data: 2024-04-19T23:33:46.151 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.152 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.152 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.152 DEBUG:boto:Params: {} 2024-04-19T23:33:46.152 DEBUG:boto:Token: None 2024-04-19T23:33:46.152 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/?versions 2024-04-19T23:33:46.152 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cPa6JiZE7x4b0c5UnKac4XDQahE= 2024-04-19T23:33:46.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cPa6JiZE7x4b0c5UnKac4XDQahE=', 'Content-Length': '0'} 2024-04-19T23:33:46.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000075644d2c427e585-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.155 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.155 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:46.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:46.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:46.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:46.156 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:33:46.156 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.156 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.156 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:46.156 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:46.156 DEBUG:boto:Method: GET 2024-04-19T23:33:46.157 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:33:46.157 DEBUG:boto:Data: 2024-04-19T23:33:46.157 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.157 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.157 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.157 DEBUG:boto:Params: {} 2024-04-19T23:33:46.157 DEBUG:boto:Token: None 2024-04-19T23:33:46.157 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/?versions 2024-04-19T23:33:46.158 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cPa6JiZE7x4b0c5UnKac4XDQahE= 2024-04-19T23:33:46.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cPa6JiZE7x4b0c5UnKac4XDQahE=', 'Content-Length': '0'} 2024-04-19T23:33:46.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a81e0e97fa2ab992-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.160 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.161 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:46.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:46.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:46.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:46.161 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.161 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.161 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:46.161 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:46.161 DEBUG:boto:Method: GET 2024-04-19T23:33:46.161 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:33:46.162 DEBUG:boto:Data: 2024-04-19T23:33:46.162 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.162 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.162 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.162 DEBUG:boto:Params: {} 2024-04-19T23:33:46.162 DEBUG:boto:Token: None 2024-04-19T23:33:46.162 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/?versions 2024-04-19T23:33:46.162 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cPa6JiZE7x4b0c5UnKac4XDQahE= 2024-04-19T23:33:46.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cPa6JiZE7x4b0c5UnKac4XDQahE=', 'Content-Length': '0'} 2024-04-19T23:33:46.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b288a38a0731a835-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.165 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.166 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.166 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.166 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:46.166 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:46.166 DEBUG:boto:Method: GET 2024-04-19T23:33:46.166 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:33:46.166 DEBUG:boto:Data: 2024-04-19T23:33:46.166 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.166 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.166 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.166 DEBUG:boto:Params: {} 2024-04-19T23:33:46.167 DEBUG:boto:Token: None 2024-04-19T23:33:46.167 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/?versions 2024-04-19T23:33:46.167 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cPa6JiZE7x4b0c5UnKac4XDQahE= 2024-04-19T23:33:46.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cPa6JiZE7x4b0c5UnKac4XDQahE=', 'Content-Length': '0'} 2024-04-19T23:33:46.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b5602c18a4786c6-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.169 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:46.170 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:46.170 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:46.170 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.170 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.170 DEBUG:boto:Method: GET 2024-04-19T23:33:46.170 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.171 DEBUG:boto:Data: 2024-04-19T23:33:46.171 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.171 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.171 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.171 DEBUG:boto:Params: {} 2024-04-19T23:33:46.171 DEBUG:boto:Token: None 2024-04-19T23:33:46.171 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.171 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mh2dzrSRh++YEk0RKBLzeg+LHsE= 2024-04-19T23:33:46.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mh2dzrSRh++YEk0RKBLzeg+LHsE=', 'Content-Length': '0'} 2024-04-19T23:33:46.173 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006987902fdff9f25d-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.173 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:46.173 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:46.173 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.173 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.173 DEBUG:boto:Method: GET 2024-04-19T23:33:46.174 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.174 DEBUG:boto:Data: 2024-04-19T23:33:46.174 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.174 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.174 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.174 DEBUG:boto:Params: {} 2024-04-19T23:33:46.174 DEBUG:boto:Token: None 2024-04-19T23:33:46.174 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.174 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mh2dzrSRh++YEk0RKBLzeg+LHsE= 2024-04-19T23:33:46.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mh2dzrSRh++YEk0RKBLzeg+LHsE=', 'Content-Length': '0'} 2024-04-19T23:33:46.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009256f47f4c424d01-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.176 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:46.176 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:46.177 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.177 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.177 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.177 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.177 DEBUG:boto:Data: 2024-04-19T23:33:46.177 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.177 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.177 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.177 DEBUG:boto:Params: {} 2024-04-19T23:33:46.178 DEBUG:boto:Token: None 2024-04-19T23:33:46.178 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.178 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QAngj5LFetA4k6UIYVCSM57uVYw= 2024-04-19T23:33:46.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QAngj5LFetA4k6UIYVCSM57uVYw=', 'Content-Length': '0'} 2024-04-19T23:33:46.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003f290924c18b57c7-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.180 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:46.180 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:46.180 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.180 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.180 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.180 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.180 DEBUG:boto:Data: 2024-04-19T23:33:46.181 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.181 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.181 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.181 DEBUG:boto:Params: {} 2024-04-19T23:33:46.181 DEBUG:boto:Token: None 2024-04-19T23:33:46.181 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.181 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QAngj5LFetA4k6UIYVCSM57uVYw= 2024-04-19T23:33:46.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QAngj5LFetA4k6UIYVCSM57uVYw=', 'Content-Length': '0'} 2024-04-19T23:33:46.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d8e034b86bcd15c7-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:46.184 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:46.184 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:46.184 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.184 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.184 DEBUG:boto:Method: GET 2024-04-19T23:33:46.184 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.184 DEBUG:boto:Data: 2024-04-19T23:33:46.184 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.185 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.185 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.185 DEBUG:boto:Params: {} 2024-04-19T23:33:46.185 DEBUG:boto:Token: None 2024-04-19T23:33:46.185 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.185 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mh2dzrSRh++YEk0RKBLzeg+LHsE= 2024-04-19T23:33:46.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mh2dzrSRh++YEk0RKBLzeg+LHsE=', 'Content-Length': '0'} 2024-04-19T23:33:46.187 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000071553c04605b634b-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.187 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:46.188 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:46.188 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.188 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.188 DEBUG:boto:Method: GET 2024-04-19T23:33:46.188 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.188 DEBUG:boto:Data: 2024-04-19T23:33:46.188 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.188 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.188 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.188 DEBUG:boto:Params: {} 2024-04-19T23:33:46.189 DEBUG:boto:Token: None 2024-04-19T23:33:46.189 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.189 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mh2dzrSRh++YEk0RKBLzeg+LHsE= 2024-04-19T23:33:46.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mh2dzrSRh++YEk0RKBLzeg+LHsE=', 'Content-Length': '0'} 2024-04-19T23:33:46.191 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ea804cbef05faee2-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.191 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.191 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.191 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:46.191 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:46.192 DEBUG:boto:Method: GET 2024-04-19T23:33:46.192 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:33:46.192 DEBUG:boto:Data: 2024-04-19T23:33:46.192 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.192 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.192 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.192 DEBUG:boto:Params: {} 2024-04-19T23:33:46.192 DEBUG:boto:Token: None 2024-04-19T23:33:46.192 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.193 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035148154d61c9b50-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.238 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.239 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.239 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.239 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:46.239 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:46.239 DEBUG:boto:Method: GET 2024-04-19T23:33:46.240 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:33:46.240 DEBUG:boto:Data: 2024-04-19T23:33:46.240 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.240 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.240 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.240 DEBUG:boto:Params: {} 2024-04-19T23:33:46.240 DEBUG:boto:Token: None 2024-04-19T23:33:46.240 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.241 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ddb45cb36195d8a-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.244 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.244 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:46.244 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:46.245 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.245 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:33:46.245 DEBUG:boto:Data: 2024-04-19T23:33:46.245 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.245 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.245 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.245 DEBUG:boto:Params: {} 2024-04-19T23:33:46.245 DEBUG:boto:Token: None 2024-04-19T23:33:46.245 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/dummy 2024-04-19T23:33:46.245 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OWu0OPtc8CFCcXJZryk3N0lWxlA= 2024-04-19T23:33:46.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OWu0OPtc8CFCcXJZryk3N0lWxlA=', 'Content-Length': '0'} 2024-04-19T23:33:46.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000064b7e56f3795241-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.248 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:46.248 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:46.248 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.249 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:33:46.249 DEBUG:boto:Data: 2024-04-19T23:33:46.249 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.249 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.249 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.249 DEBUG:boto:Params: {} 2024-04-19T23:33:46.249 DEBUG:boto:Token: None 2024-04-19T23:33:46.249 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/dummy 2024-04-19T23:33:46.249 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OWu0OPtc8CFCcXJZryk3N0lWxlA= 2024-04-19T23:33:46.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OWu0OPtc8CFCcXJZryk3N0lWxlA=', 'Content-Length': '0'} 2024-04-19T23:33:46.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f9acb10f0a89ad92-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:46.252 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:46.252 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:46.252 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.252 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.253 DEBUG:boto:Method: GET 2024-04-19T23:33:46.253 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.253 DEBUG:boto:Data: 2024-04-19T23:33:46.253 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.253 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.253 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.253 DEBUG:boto:Params: {} 2024-04-19T23:33:46.253 DEBUG:boto:Token: None 2024-04-19T23:33:46.253 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.254 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mh2dzrSRh++YEk0RKBLzeg+LHsE= 2024-04-19T23:33:46.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mh2dzrSRh++YEk0RKBLzeg+LHsE=', 'Content-Length': '0'} 2024-04-19T23:33:46.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bd8d71892bfefdde-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.256 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:46.256 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:46.256 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.256 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.256 DEBUG:boto:Method: GET 2024-04-19T23:33:46.256 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.256 DEBUG:boto:Data: 2024-04-19T23:33:46.256 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.257 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.257 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.257 DEBUG:boto:Params: {} 2024-04-19T23:33:46.257 DEBUG:boto:Token: None 2024-04-19T23:33:46.257 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:46.257 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mh2dzrSRh++YEk0RKBLzeg+LHsE= 2024-04-19T23:33:46.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mh2dzrSRh++YEk0RKBLzeg+LHsE=', 'Content-Length': '0'} 2024-04-19T23:33:46.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b3a71a32f0b87135-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.259 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.259 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.260 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:46.260 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:46.260 DEBUG:boto:Method: GET 2024-04-19T23:33:46.260 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:33:46.260 DEBUG:boto:Data: 2024-04-19T23:33:46.260 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.260 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.260 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.260 DEBUG:boto:Params: {} 2024-04-19T23:33:46.261 DEBUG:boto:Token: None 2024-04-19T23:33:46.261 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.261 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000074cb633dd694e85-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.263 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.264 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.264 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.264 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:46.264 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:46.264 DEBUG:boto:Method: GET 2024-04-19T23:33:46.264 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:33:46.264 DEBUG:boto:Data: 2024-04-19T23:33:46.265 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.265 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.265 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.265 DEBUG:boto:Params: {} 2024-04-19T23:33:46.265 DEBUG:boto:Token: None 2024-04-19T23:33:46.265 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.265 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b13cfa416fca77b-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.314 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:46.315 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.315 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.315 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.315 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.315 DEBUG:boto:Method: GET 2024-04-19T23:33:46.316 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.316 DEBUG:boto:Data: 2024-04-19T23:33:46.316 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.316 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.316 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.316 DEBUG:boto:Params: {} 2024-04-19T23:33:46.316 DEBUG:boto:Token: None 2024-04-19T23:33:46.317 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.317 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XGeNWg3tQGAT8dSztgdpkytXMgc= 2024-04-19T23:33:46.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XGeNWg3tQGAT8dSztgdpkytXMgc=', 'Content-Length': '0'} 2024-04-19T23:33:46.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e9ab3b182ad2d71-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.362 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.363 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.363 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.363 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.363 DEBUG:boto:Method: GET 2024-04-19T23:33:46.363 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.363 DEBUG:boto:Data: 2024-04-19T23:33:46.363 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.363 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.363 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.364 DEBUG:boto:Params: {} 2024-04-19T23:33:46.364 DEBUG:boto:Token: None 2024-04-19T23:33:46.364 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.364 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XGeNWg3tQGAT8dSztgdpkytXMgc= 2024-04-19T23:33:46.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XGeNWg3tQGAT8dSztgdpkytXMgc=', 'Content-Length': '0'} 2024-04-19T23:33:46.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046858abeb89ae57f-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.367 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.367 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.367 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.367 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.368 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.368 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.368 DEBUG:boto:Data: 2024-04-19T23:33:46.368 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.368 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.368 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.368 DEBUG:boto:Params: {} 2024-04-19T23:33:46.369 DEBUG:boto:Token: None 2024-04-19T23:33:46.369 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.369 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QqWyozTlXrW9L+xlMORVP8CGFq8= 2024-04-19T23:33:46.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QqWyozTlXrW9L+xlMORVP8CGFq8=', 'Content-Length': '0'} 2024-04-19T23:33:46.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002cc9de6fbf2e0bf7-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.371 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.371 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.372 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.372 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.372 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.372 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.372 DEBUG:boto:Data: 2024-04-19T23:33:46.372 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.372 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.372 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.372 DEBUG:boto:Params: {} 2024-04-19T23:33:46.373 DEBUG:boto:Token: None 2024-04-19T23:33:46.373 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.373 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QqWyozTlXrW9L+xlMORVP8CGFq8= 2024-04-19T23:33:46.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QqWyozTlXrW9L+xlMORVP8CGFq8=', 'Content-Length': '0'} 2024-04-19T23:33:46.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a84395cb39336ac-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:46.375 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.375 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.375 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.375 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.376 DEBUG:boto:Method: GET 2024-04-19T23:33:46.376 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.376 DEBUG:boto:Data: 2024-04-19T23:33:46.376 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.376 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.376 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.376 DEBUG:boto:Params: {} 2024-04-19T23:33:46.376 DEBUG:boto:Token: None 2024-04-19T23:33:46.377 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.377 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XGeNWg3tQGAT8dSztgdpkytXMgc= 2024-04-19T23:33:46.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XGeNWg3tQGAT8dSztgdpkytXMgc=', 'Content-Length': '0'} 2024-04-19T23:33:46.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fab7bf440000e8f6-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.422 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.423 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.423 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.423 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.423 DEBUG:boto:Method: GET 2024-04-19T23:33:46.423 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.423 DEBUG:boto:Data: 2024-04-19T23:33:46.423 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.423 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.423 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.423 DEBUG:boto:Params: {} 2024-04-19T23:33:46.424 DEBUG:boto:Token: None 2024-04-19T23:33:46.424 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.424 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XGeNWg3tQGAT8dSztgdpkytXMgc= 2024-04-19T23:33:46.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XGeNWg3tQGAT8dSztgdpkytXMgc=', 'Content-Length': '0'} 2024-04-19T23:33:46.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5c3a760e93870cb-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.466 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.466 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.467 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:46.467 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:46.467 DEBUG:boto:Method: GET 2024-04-19T23:33:46.467 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:46.467 DEBUG:boto:Data: 2024-04-19T23:33:46.467 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.467 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.467 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.467 DEBUG:boto:Params: {} 2024-04-19T23:33:46.468 DEBUG:boto:Token: None 2024-04-19T23:33:46.468 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.468 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a68048efc6833b9-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.514 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.515 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.515 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.515 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:46.515 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:46.515 DEBUG:boto:Method: GET 2024-04-19T23:33:46.515 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:46.515 DEBUG:boto:Data: 2024-04-19T23:33:46.515 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.516 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.516 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.516 DEBUG:boto:Params: {} 2024-04-19T23:33:46.516 DEBUG:boto:Token: None 2024-04-19T23:33:46.516 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.516 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1bf7f663f8fbf2d-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.519 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.519 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.519 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.520 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.520 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:33:46.520 DEBUG:boto:Data: 2024-04-19T23:33:46.520 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.520 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.520 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.520 DEBUG:boto:Params: {} 2024-04-19T23:33:46.520 DEBUG:boto:Token: None 2024-04-19T23:33:46.521 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:33:46.521 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b8fWXXekd/dm7VgARq82TJvfMuI= 2024-04-19T23:33:46.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b8fWXXekd/dm7VgARq82TJvfMuI=', 'Content-Length': '0'} 2024-04-19T23:33:46.523 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bd422d2f92664f6-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.523 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.524 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.524 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.524 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:33:46.524 DEBUG:boto:Data: 2024-04-19T23:33:46.524 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.524 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.524 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.524 DEBUG:boto:Params: {} 2024-04-19T23:33:46.524 DEBUG:boto:Token: None 2024-04-19T23:33:46.524 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:33:46.525 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b8fWXXekd/dm7VgARq82TJvfMuI= 2024-04-19T23:33:46.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b8fWXXekd/dm7VgARq82TJvfMuI=', 'Content-Length': '0'} 2024-04-19T23:33:46.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e33cedf7dd7f4cc-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:46.527 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.527 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.528 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.528 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.528 DEBUG:boto:Method: GET 2024-04-19T23:33:46.528 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.528 DEBUG:boto:Data: 2024-04-19T23:33:46.528 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.528 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.528 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.528 DEBUG:boto:Params: {} 2024-04-19T23:33:46.529 DEBUG:boto:Token: None 2024-04-19T23:33:46.529 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.529 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XGeNWg3tQGAT8dSztgdpkytXMgc= 2024-04-19T23:33:46.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XGeNWg3tQGAT8dSztgdpkytXMgc=', 'Content-Length': '0'} 2024-04-19T23:33:46.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d36abbb6bb10380-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.574 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.575 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:46.575 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.575 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.575 DEBUG:boto:Method: GET 2024-04-19T23:33:46.575 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.575 DEBUG:boto:Data: 2024-04-19T23:33:46.575 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.575 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.575 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.575 DEBUG:boto:Params: {} 2024-04-19T23:33:46.576 DEBUG:boto:Token: None 2024-04-19T23:33:46.576 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:46.576 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XGeNWg3tQGAT8dSztgdpkytXMgc= 2024-04-19T23:33:46.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XGeNWg3tQGAT8dSztgdpkytXMgc=', 'Content-Length': '0'} 2024-04-19T23:33:46.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043bebbbe2f35cb01-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.578 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.579 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.579 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:46.579 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:46.579 DEBUG:boto:Method: GET 2024-04-19T23:33:46.579 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:46.579 DEBUG:boto:Data: 2024-04-19T23:33:46.579 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.579 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.579 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.579 DEBUG:boto:Params: {} 2024-04-19T23:33:46.580 DEBUG:boto:Token: None 2024-04-19T23:33:46.580 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.580 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d6b3746de48b5f1-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.626 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.627 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.627 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.627 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:46.627 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:46.627 DEBUG:boto:Method: GET 2024-04-19T23:33:46.627 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:46.627 DEBUG:boto:Data: 2024-04-19T23:33:46.628 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.628 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.628 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.628 DEBUG:boto:Params: {} 2024-04-19T23:33:46.628 DEBUG:boto:Token: None 2024-04-19T23:33:46.628 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009070e261047b72bd-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.631 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:46.632 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.632 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.632 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.632 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.632 DEBUG:boto:Method: GET 2024-04-19T23:33:46.632 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.633 DEBUG:boto:Data: 2024-04-19T23:33:46.633 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.633 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.633 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.633 DEBUG:boto:Params: {} 2024-04-19T23:33:46.633 DEBUG:boto:Token: None 2024-04-19T23:33:46.633 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.633 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WSkXxu5s2VVDyrGYr7KBxs5vZFc= 2024-04-19T23:33:46.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WSkXxu5s2VVDyrGYr7KBxs5vZFc=', 'Content-Length': '0'} 2024-04-19T23:33:46.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000410d9c5544eff8e4-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.636 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.636 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.636 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.636 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.636 DEBUG:boto:Method: GET 2024-04-19T23:33:46.637 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.637 DEBUG:boto:Data: 2024-04-19T23:33:46.637 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.637 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.637 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.637 DEBUG:boto:Params: {} 2024-04-19T23:33:46.637 DEBUG:boto:Token: None 2024-04-19T23:33:46.637 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.638 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WSkXxu5s2VVDyrGYr7KBxs5vZFc= 2024-04-19T23:33:46.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WSkXxu5s2VVDyrGYr7KBxs5vZFc=', 'Content-Length': '0'} 2024-04-19T23:33:46.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c8ddde3d9903c08-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.640 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.640 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.640 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.640 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.640 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.640 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.640 DEBUG:boto:Data: 2024-04-19T23:33:46.640 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.640 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.641 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.641 DEBUG:boto:Params: {} 2024-04-19T23:33:46.641 DEBUG:boto:Token: None 2024-04-19T23:33:46.641 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.641 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBYSjEH9f9qycXADVD/I9uOA8F4= 2024-04-19T23:33:46.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBYSjEH9f9qycXADVD/I9uOA8F4=', 'Content-Length': '0'} 2024-04-19T23:33:46.643 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002902a45934ddba16-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.643 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.643 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.643 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.643 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.643 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.644 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.644 DEBUG:boto:Data: 2024-04-19T23:33:46.644 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.644 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.644 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.644 DEBUG:boto:Params: {} 2024-04-19T23:33:46.644 DEBUG:boto:Token: None 2024-04-19T23:33:46.644 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.644 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBYSjEH9f9qycXADVD/I9uOA8F4= 2024-04-19T23:33:46.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBYSjEH9f9qycXADVD/I9uOA8F4=', 'Content-Length': '0'} 2024-04-19T23:33:46.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077e5b5903451307f-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:46.646 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.647 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.647 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.647 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.647 DEBUG:boto:Method: GET 2024-04-19T23:33:46.647 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.647 DEBUG:boto:Data: 2024-04-19T23:33:46.647 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.647 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.647 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.647 DEBUG:boto:Params: {} 2024-04-19T23:33:46.647 DEBUG:boto:Token: None 2024-04-19T23:33:46.648 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.648 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WSkXxu5s2VVDyrGYr7KBxs5vZFc= 2024-04-19T23:33:46.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WSkXxu5s2VVDyrGYr7KBxs5vZFc=', 'Content-Length': '0'} 2024-04-19T23:33:46.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031f4e7cdc36a2216-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.690 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.690 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.691 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.691 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.691 DEBUG:boto:Method: GET 2024-04-19T23:33:46.691 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.691 DEBUG:boto:Data: 2024-04-19T23:33:46.691 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.691 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.691 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.691 DEBUG:boto:Params: {} 2024-04-19T23:33:46.692 DEBUG:boto:Token: None 2024-04-19T23:33:46.692 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.692 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WSkXxu5s2VVDyrGYr7KBxs5vZFc= 2024-04-19T23:33:46.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WSkXxu5s2VVDyrGYr7KBxs5vZFc=', 'Content-Length': '0'} 2024-04-19T23:33:46.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a54162eaba874dfb-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.734 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.735 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.735 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:46.735 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:46.735 DEBUG:boto:Method: GET 2024-04-19T23:33:46.735 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:46.735 DEBUG:boto:Data: 2024-04-19T23:33:46.735 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.735 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.735 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.735 DEBUG:boto:Params: {} 2024-04-19T23:33:46.735 DEBUG:boto:Token: None 2024-04-19T23:33:46.736 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.736 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6a93d70c75f0c28-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.739 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.739 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.739 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.740 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:46.740 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:46.740 DEBUG:boto:Method: GET 2024-04-19T23:33:46.740 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:46.740 DEBUG:boto:Data: 2024-04-19T23:33:46.740 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.740 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.740 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.740 DEBUG:boto:Params: {} 2024-04-19T23:33:46.740 DEBUG:boto:Token: None 2024-04-19T23:33:46.741 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.741 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049d1426b6b7b87e3-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.744 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.744 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.744 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.744 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.744 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:33:46.744 DEBUG:boto:Data: 2024-04-19T23:33:46.745 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.745 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.745 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.745 DEBUG:boto:Params: {} 2024-04-19T23:33:46.745 DEBUG:boto:Token: None 2024-04-19T23:33:46.745 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:33:46.745 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gpv0yu15F+I08FjFlcmRUtwfsKw= 2024-04-19T23:33:46.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gpv0yu15F+I08FjFlcmRUtwfsKw=', 'Content-Length': '0'} 2024-04-19T23:33:46.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090583b9560011951-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.748 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.748 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.748 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.748 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:33:46.748 DEBUG:boto:Data: 2024-04-19T23:33:46.748 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.748 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.748 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.749 DEBUG:boto:Params: {} 2024-04-19T23:33:46.749 DEBUG:boto:Token: None 2024-04-19T23:33:46.749 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:33:46.749 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gpv0yu15F+I08FjFlcmRUtwfsKw= 2024-04-19T23:33:46.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gpv0yu15F+I08FjFlcmRUtwfsKw=', 'Content-Length': '0'} 2024-04-19T23:33:46.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa6a95d9cd80595c-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:46.752 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.753 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.753 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.753 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.753 DEBUG:boto:Method: GET 2024-04-19T23:33:46.753 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.754 DEBUG:boto:Data: 2024-04-19T23:33:46.754 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.754 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.754 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.754 DEBUG:boto:Params: {} 2024-04-19T23:33:46.755 DEBUG:boto:Token: None 2024-04-19T23:33:46.755 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.755 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WSkXxu5s2VVDyrGYr7KBxs5vZFc= 2024-04-19T23:33:46.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WSkXxu5s2VVDyrGYr7KBxs5vZFc=', 'Content-Length': '0'} 2024-04-19T23:33:46.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab6b18765a7c72c6-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.798 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.799 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:46.799 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.799 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.799 DEBUG:boto:Method: GET 2024-04-19T23:33:46.799 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.799 DEBUG:boto:Data: 2024-04-19T23:33:46.799 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.799 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.799 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.800 DEBUG:boto:Params: {} 2024-04-19T23:33:46.800 DEBUG:boto:Token: None 2024-04-19T23:33:46.800 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:46.800 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WSkXxu5s2VVDyrGYr7KBxs5vZFc= 2024-04-19T23:33:46.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WSkXxu5s2VVDyrGYr7KBxs5vZFc=', 'Content-Length': '0'} 2024-04-19T23:33:46.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000314cb0a57e907634-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.842 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.842 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.843 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:46.843 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:46.843 DEBUG:boto:Method: GET 2024-04-19T23:33:46.843 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:46.843 DEBUG:boto:Data: 2024-04-19T23:33:46.843 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.843 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.843 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.843 DEBUG:boto:Params: {} 2024-04-19T23:33:46.844 DEBUG:boto:Token: None 2024-04-19T23:33:46.844 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.844 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095ed050b201a7b81-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.894 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.895 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:46.895 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:46.895 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:46.895 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:46.895 DEBUG:boto:Method: GET 2024-04-19T23:33:46.895 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:46.896 DEBUG:boto:Data: 2024-04-19T23:33:46.896 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.896 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.896 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.896 DEBUG:boto:Params: {} 2024-04-19T23:33:46.896 DEBUG:boto:Token: None 2024-04-19T23:33:46.896 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/ 2024-04-19T23:33:46.896 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4= 2024-04-19T23:33:46.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vLbGAhyZL1oGsoiAkBLbKY71Fg4=', 'Content-Length': '0'} 2024-04-19T23:33:46.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8330039b7de36d6-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.942 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:46.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:46.943 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.943 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.943 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.943 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.943 DEBUG:boto:Method: GET 2024-04-19T23:33:46.943 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.944 DEBUG:boto:Data: 2024-04-19T23:33:46.944 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.944 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.944 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.944 DEBUG:boto:Params: {} 2024-04-19T23:33:46.944 DEBUG:boto:Token: None 2024-04-19T23:33:46.944 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Liz7IzWwEFHu2UjB+X/obQzipkk= 2024-04-19T23:33:46.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Liz7IzWwEFHu2UjB+X/obQzipkk=', 'Content-Length': '0'} 2024-04-19T23:33:46.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ddfccea088d4985-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.947 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.947 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.947 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.947 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.947 DEBUG:boto:Method: GET 2024-04-19T23:33:46.948 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.948 DEBUG:boto:Data: 2024-04-19T23:33:46.948 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.948 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.948 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.948 DEBUG:boto:Params: {} 2024-04-19T23:33:46.948 DEBUG:boto:Token: None 2024-04-19T23:33:46.948 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.948 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Liz7IzWwEFHu2UjB+X/obQzipkk= 2024-04-19T23:33:46.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Liz7IzWwEFHu2UjB+X/obQzipkk=', 'Content-Length': '0'} 2024-04-19T23:33:46.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a14356a012cd9cda-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.951 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.951 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.951 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.951 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.951 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.951 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.952 DEBUG:boto:Data: 2024-04-19T23:33:46.952 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.952 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.952 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.952 DEBUG:boto:Params: {} 2024-04-19T23:33:46.952 DEBUG:boto:Token: None 2024-04-19T23:33:46.952 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.952 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ivK2itANs4m7SJVpsqFP0cnc1i4= 2024-04-19T23:33:46.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ivK2itANs4m7SJVpsqFP0cnc1i4=', 'Content-Length': '0'} 2024-04-19T23:33:46.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086f54bd108b63363-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.954 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.954 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.955 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.955 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.955 DEBUG:boto:Method: HEAD 2024-04-19T23:33:46.955 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.955 DEBUG:boto:Data: 2024-04-19T23:33:46.955 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.955 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.955 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.955 DEBUG:boto:Params: {} 2024-04-19T23:33:46.955 DEBUG:boto:Token: None 2024-04-19T23:33:46.956 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.956 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ivK2itANs4m7SJVpsqFP0cnc1i4= 2024-04-19T23:33:46.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ivK2itANs4m7SJVpsqFP0cnc1i4=', 'Content-Length': '0'} 2024-04-19T23:33:46.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aba3ca7bedf8b917-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:46.958 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.958 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.958 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.958 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.958 DEBUG:boto:Method: GET 2024-04-19T23:33:46.959 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.959 DEBUG:boto:Data: 2024-04-19T23:33:46.959 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.959 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:46.959 DEBUG:boto:Port: 8001 2024-04-19T23:33:46.959 DEBUG:boto:Params: {} 2024-04-19T23:33:46.959 DEBUG:boto:Token: None 2024-04-19T23:33:46.959 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.959 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Liz7IzWwEFHu2UjB+X/obQzipkk= 2024-04-19T23:33:46.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Liz7IzWwEFHu2UjB+X/obQzipkk=', 'Content-Length': '0'} 2024-04-19T23:33:46.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c709238ab5d7822e-006622ff5a-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:46.962 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.962 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:46.962 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.962 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.962 DEBUG:boto:Method: GET 2024-04-19T23:33:46.962 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.962 DEBUG:boto:Data: 2024-04-19T23:33:46.963 DEBUG:boto:Headers: {} 2024-04-19T23:33:46.963 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:46.963 DEBUG:boto:Port: 8000 2024-04-19T23:33:46.963 DEBUG:boto:Params: {} 2024-04-19T23:33:46.963 DEBUG:boto:Token: None 2024-04-19T23:33:46.963 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:46 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:46.963 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Liz7IzWwEFHu2UjB+X/obQzipkk= 2024-04-19T23:33:46.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:46 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Liz7IzWwEFHu2UjB+X/obQzipkk=', 'Content-Length': '0'} 2024-04-19T23:33:46.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f115b0f5c9ea917-006622ff5a-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.006 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.006 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.007 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.007 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.007 DEBUG:boto:Method: GET 2024-04-19T23:33:47.007 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.007 DEBUG:boto:Data: 2024-04-19T23:33:47.007 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.007 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.007 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.007 DEBUG:boto:Params: {} 2024-04-19T23:33:47.007 DEBUG:boto:Token: None 2024-04-19T23:33:47.008 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.008 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d199135ceb2b975d-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.011 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.011 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.011 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.011 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.012 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.012 DEBUG:boto:Method: GET 2024-04-19T23:33:47.012 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.012 DEBUG:boto:Data: 2024-04-19T23:33:47.012 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.012 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:47.012 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.012 DEBUG:boto:Params: {} 2024-04-19T23:33:47.012 DEBUG:boto:Token: None 2024-04-19T23:33:47.012 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.013 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027c664c504ea9c0a-006622ff5b-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.058 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.059 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.059 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.059 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.059 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:33:47.059 DEBUG:boto:Data: 2024-04-19T23:33:47.059 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.059 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.059 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.059 DEBUG:boto:Params: {} 2024-04-19T23:33:47.060 DEBUG:boto:Token: None 2024-04-19T23:33:47.060 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:33:47.060 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nN5nPWoVrQrZv9C7hmM7t3TP2pY= 2024-04-19T23:33:47.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nN5nPWoVrQrZv9C7hmM7t3TP2pY=', 'Content-Length': '0'} 2024-04-19T23:33:47.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053d45635e37fea31-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.063 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.063 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.063 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.063 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:33:47.063 DEBUG:boto:Data: 2024-04-19T23:33:47.063 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.063 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:47.063 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.063 DEBUG:boto:Params: {} 2024-04-19T23:33:47.063 DEBUG:boto:Token: None 2024-04-19T23:33:47.064 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:33:47.064 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nN5nPWoVrQrZv9C7hmM7t3TP2pY= 2024-04-19T23:33:47.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nN5nPWoVrQrZv9C7hmM7t3TP2pY=', 'Content-Length': '0'} 2024-04-19T23:33:47.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000546b2230b978525a-006622ff5b-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:47.067 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.067 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.067 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.067 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.067 DEBUG:boto:Method: GET 2024-04-19T23:33:47.067 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.067 DEBUG:boto:Data: 2024-04-19T23:33:47.067 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.067 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.067 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.067 DEBUG:boto:Params: {} 2024-04-19T23:33:47.068 DEBUG:boto:Token: None 2024-04-19T23:33:47.068 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.068 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM= 2024-04-19T23:33:47.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM=', 'Content-Length': '0'} 2024-04-19T23:33:47.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090f1b901978d9f26-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.110 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.110 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.111 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.111 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.111 DEBUG:boto:Method: GET 2024-04-19T23:33:47.111 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.111 DEBUG:boto:Data: 2024-04-19T23:33:47.111 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.111 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:47.111 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.111 DEBUG:boto:Params: {} 2024-04-19T23:33:47.112 DEBUG:boto:Token: None 2024-04-19T23:33:47.112 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.112 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM= 2024-04-19T23:33:47.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM=', 'Content-Length': '0'} 2024-04-19T23:33:47.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bf58185963f0f01-006622ff5b-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.154 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.155 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.155 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.155 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.155 DEBUG:boto:Method: GET 2024-04-19T23:33:47.155 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.155 DEBUG:boto:Data: 2024-04-19T23:33:47.155 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.155 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.156 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.156 DEBUG:boto:Params: {} 2024-04-19T23:33:47.156 DEBUG:boto:Token: None 2024-04-19T23:33:47.156 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.156 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c9fc3112701cd93-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.159 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.159 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.159 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.160 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.160 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.160 DEBUG:boto:Method: GET 2024-04-19T23:33:47.160 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.160 DEBUG:boto:Data: 2024-04-19T23:33:47.160 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.160 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:47.160 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.160 DEBUG:boto:Params: {} 2024-04-19T23:33:47.160 DEBUG:boto:Token: None 2024-04-19T23:33:47.160 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.161 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e78818f1921e89ef-006622ff5b-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.164 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.164 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-20 zones={test-zone3, test-zone1} 2024-04-19T23:33:47.164 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-20 zones={test-zone3, test-zone2} 2024-04-19T23:33:47.164 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.164 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.164 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.165 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:33:47.165 DEBUG:boto:Data: 2024-04-19T23:33:47.165 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.165 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.165 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.165 DEBUG:boto:Params: {} 2024-04-19T23:33:47.165 DEBUG:boto:Token: None 2024-04-19T23:33:47.165 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.165 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:A2VhzzI0yCAcboqMZvXi6RB8i24= 2024-04-19T23:33:47.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:A2VhzzI0yCAcboqMZvXi6RB8i24=', 'Content-Length': '0'} 2024-04-19T23:33:47.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000128edfa479bfb1a-006622ff5b-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.169 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.169 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.169 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.169 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:33:47.169 DEBUG:boto:Data: 2024-04-19T23:33:47.169 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.169 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.169 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.169 DEBUG:boto:Params: {} 2024-04-19T23:33:47.169 DEBUG:boto:Token: None 2024-04-19T23:33:47.170 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.170 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:A2VhzzI0yCAcboqMZvXi6RB8i24= 2024-04-19T23:33:47.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:A2VhzzI0yCAcboqMZvXi6RB8i24=', 'Content-Length': '0'} 2024-04-19T23:33:47.172 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f12504daa6330025-006622ff5b-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.173 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:33:47.173 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.173 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.173 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:47.173 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:47.173 DEBUG:boto:Method: GET 2024-04-19T23:33:47.173 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:33:47.173 DEBUG:boto:Data: 2024-04-19T23:33:47.174 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.174 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.174 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.174 DEBUG:boto:Params: {} 2024-04-19T23:33:47.174 DEBUG:boto:Token: None 2024-04-19T23:33:47.174 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/?versions 2024-04-19T23:33:47.174 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4gNxEQY5RDPtaj3kmV462Hy73v0= 2024-04-19T23:33:47.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4gNxEQY5RDPtaj3kmV462Hy73v0=', 'Content-Length': '0'} 2024-04-19T23:33:47.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fa138d0c4af1a29-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.177 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.178 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:47.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:47.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:47.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:47.178 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:33:47.178 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.178 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.179 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:47.179 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:47.179 DEBUG:boto:Method: GET 2024-04-19T23:33:47.179 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:33:47.179 DEBUG:boto:Data: 2024-04-19T23:33:47.179 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.179 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.179 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.179 DEBUG:boto:Params: {} 2024-04-19T23:33:47.179 DEBUG:boto:Token: None 2024-04-19T23:33:47.180 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/?versions 2024-04-19T23:33:47.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4gNxEQY5RDPtaj3kmV462Hy73v0= 2024-04-19T23:33:47.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4gNxEQY5RDPtaj3kmV462Hy73v0=', 'Content-Length': '0'} 2024-04-19T23:33:47.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000466b7f095d15a26d-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.182 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.183 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:47.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:47.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:47.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:47.183 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.183 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.183 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:47.183 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:47.183 DEBUG:boto:Method: GET 2024-04-19T23:33:47.184 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:33:47.184 DEBUG:boto:Data: 2024-04-19T23:33:47.184 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.184 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.184 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.184 DEBUG:boto:Params: {} 2024-04-19T23:33:47.184 DEBUG:boto:Token: None 2024-04-19T23:33:47.184 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/?versions 2024-04-19T23:33:47.185 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4gNxEQY5RDPtaj3kmV462Hy73v0= 2024-04-19T23:33:47.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4gNxEQY5RDPtaj3kmV462Hy73v0=', 'Content-Length': '0'} 2024-04-19T23:33:47.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007833fecc5a6c1027-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.187 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.188 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.188 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.188 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:47.188 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:33:47.188 DEBUG:boto:Method: GET 2024-04-19T23:33:47.188 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:33:47.188 DEBUG:boto:Data: 2024-04-19T23:33:47.188 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.189 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.189 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.189 DEBUG:boto:Params: {} 2024-04-19T23:33:47.189 DEBUG:boto:Token: None 2024-04-19T23:33:47.189 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/?versions 2024-04-19T23:33:47.189 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4gNxEQY5RDPtaj3kmV462Hy73v0= 2024-04-19T23:33:47.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4gNxEQY5RDPtaj3kmV462Hy73v0=', 'Content-Length': '0'} 2024-04-19T23:33:47.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e948ba8fa688ced-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.192 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:47.192 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:47.192 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:47.193 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.193 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.193 DEBUG:boto:Method: GET 2024-04-19T23:33:47.193 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.193 DEBUG:boto:Data: 2024-04-19T23:33:47.193 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.193 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.193 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.193 DEBUG:boto:Params: {} 2024-04-19T23:33:47.193 DEBUG:boto:Token: None 2024-04-19T23:33:47.194 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.194 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WZwuMjYU0upeUT3CkzEXa2QRemQ= 2024-04-19T23:33:47.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WZwuMjYU0upeUT3CkzEXa2QRemQ=', 'Content-Length': '0'} 2024-04-19T23:33:47.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000310b3b86ecb5ffa-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.196 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:47.196 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:47.196 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.196 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.196 DEBUG:boto:Method: GET 2024-04-19T23:33:47.196 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.197 DEBUG:boto:Data: 2024-04-19T23:33:47.197 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.197 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.197 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.197 DEBUG:boto:Params: {} 2024-04-19T23:33:47.197 DEBUG:boto:Token: None 2024-04-19T23:33:47.198 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.198 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WZwuMjYU0upeUT3CkzEXa2QRemQ= 2024-04-19T23:33:47.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WZwuMjYU0upeUT3CkzEXa2QRemQ=', 'Content-Length': '0'} 2024-04-19T23:33:47.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f6144eca86343495-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.200 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:47.200 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:47.200 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.200 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.200 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.200 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.201 DEBUG:boto:Data: 2024-04-19T23:33:47.201 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.201 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.201 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.201 DEBUG:boto:Params: {} 2024-04-19T23:33:47.201 DEBUG:boto:Token: None 2024-04-19T23:33:47.201 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.201 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aLJTQ1TCWH2KlY9ZhwGtkO5cI9Q= 2024-04-19T23:33:47.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aLJTQ1TCWH2KlY9ZhwGtkO5cI9Q=', 'Content-Length': '0'} 2024-04-19T23:33:47.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fcf2bd35d5012934-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.204 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:47.204 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:47.204 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.204 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.204 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.204 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.204 DEBUG:boto:Data: 2024-04-19T23:33:47.204 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.205 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.205 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.205 DEBUG:boto:Params: {} 2024-04-19T23:33:47.205 DEBUG:boto:Token: None 2024-04-19T23:33:47.205 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.205 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aLJTQ1TCWH2KlY9ZhwGtkO5cI9Q= 2024-04-19T23:33:47.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aLJTQ1TCWH2KlY9ZhwGtkO5cI9Q=', 'Content-Length': '0'} 2024-04-19T23:33:47.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002bc6fe4ffba92353-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:47.207 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:47.208 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:47.208 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.208 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.208 DEBUG:boto:Method: GET 2024-04-19T23:33:47.208 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.209 DEBUG:boto:Data: 2024-04-19T23:33:47.209 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.209 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.209 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.209 DEBUG:boto:Params: {} 2024-04-19T23:33:47.209 DEBUG:boto:Token: None 2024-04-19T23:33:47.210 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.210 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WZwuMjYU0upeUT3CkzEXa2QRemQ= 2024-04-19T23:33:47.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WZwuMjYU0upeUT3CkzEXa2QRemQ=', 'Content-Length': '0'} 2024-04-19T23:33:47.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000065094383f23bc009-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.212 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:47.212 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:47.212 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.212 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.212 DEBUG:boto:Method: GET 2024-04-19T23:33:47.212 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.212 DEBUG:boto:Data: 2024-04-19T23:33:47.212 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.213 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.213 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.213 DEBUG:boto:Params: {} 2024-04-19T23:33:47.213 DEBUG:boto:Token: None 2024-04-19T23:33:47.213 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.213 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WZwuMjYU0upeUT3CkzEXa2QRemQ= 2024-04-19T23:33:47.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WZwuMjYU0upeUT3CkzEXa2QRemQ=', 'Content-Length': '0'} 2024-04-19T23:33:47.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000080fe8a048a3682a8-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.215 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.215 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.216 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:47.216 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:47.216 DEBUG:boto:Method: GET 2024-04-19T23:33:47.216 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:33:47.216 DEBUG:boto:Data: 2024-04-19T23:33:47.216 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.216 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.216 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.216 DEBUG:boto:Params: {} 2024-04-19T23:33:47.216 DEBUG:boto:Token: None 2024-04-19T23:33:47.216 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.217 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecf4b9ca39d3d794-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.262 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.263 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.263 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.263 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:47.264 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:47.264 DEBUG:boto:Method: GET 2024-04-19T23:33:47.264 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:33:47.264 DEBUG:boto:Data: 2024-04-19T23:33:47.264 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.265 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.265 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.265 DEBUG:boto:Params: {} 2024-04-19T23:33:47.265 DEBUG:boto:Token: None 2024-04-19T23:33:47.265 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.265 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005349ff1470ec9574-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.268 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.269 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:47.269 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:47.269 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.269 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:33:47.269 DEBUG:boto:Data: 2024-04-19T23:33:47.270 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.270 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.270 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.270 DEBUG:boto:Params: {} 2024-04-19T23:33:47.270 DEBUG:boto:Token: None 2024-04-19T23:33:47.270 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/dummy 2024-04-19T23:33:47.271 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ezEQ4JC1YYMBjMXW2vZQvDB4yPw= 2024-04-19T23:33:47.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ezEQ4JC1YYMBjMXW2vZQvDB4yPw=', 'Content-Length': '0'} 2024-04-19T23:33:47.273 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000044efd13d2565fff0-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.273 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:47.274 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:47.274 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.274 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:33:47.274 DEBUG:boto:Data: 2024-04-19T23:33:47.274 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.274 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.274 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.274 DEBUG:boto:Params: {} 2024-04-19T23:33:47.274 DEBUG:boto:Token: None 2024-04-19T23:33:47.275 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/dummy 2024-04-19T23:33:47.275 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ezEQ4JC1YYMBjMXW2vZQvDB4yPw= 2024-04-19T23:33:47.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ezEQ4JC1YYMBjMXW2vZQvDB4yPw=', 'Content-Length': '0'} 2024-04-19T23:33:47.277 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dabac9d14f47fbe8-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:47.278 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:47.278 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:47.278 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.278 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.278 DEBUG:boto:Method: GET 2024-04-19T23:33:47.279 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.279 DEBUG:boto:Data: 2024-04-19T23:33:47.279 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.279 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.279 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.279 DEBUG:boto:Params: {} 2024-04-19T23:33:47.279 DEBUG:boto:Token: None 2024-04-19T23:33:47.280 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.280 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WZwuMjYU0upeUT3CkzEXa2QRemQ= 2024-04-19T23:33:47.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WZwuMjYU0upeUT3CkzEXa2QRemQ=', 'Content-Length': '0'} 2024-04-19T23:33:47.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000095c2389cad1f198a-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.282 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:33:47.283 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:33:47.283 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.283 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.283 DEBUG:boto:Method: GET 2024-04-19T23:33:47.283 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.283 DEBUG:boto:Data: 2024-04-19T23:33:47.283 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.283 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.283 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.283 DEBUG:boto:Params: {} 2024-04-19T23:33:47.283 DEBUG:boto:Token: None 2024-04-19T23:33:47.284 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:33:47.284 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WZwuMjYU0upeUT3CkzEXa2QRemQ= 2024-04-19T23:33:47.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WZwuMjYU0upeUT3CkzEXa2QRemQ=', 'Content-Length': '0'} 2024-04-19T23:33:47.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003b49a376d1efe582-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.286 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.286 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.286 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:47.286 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:47.287 DEBUG:boto:Method: GET 2024-04-19T23:33:47.287 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:33:47.287 DEBUG:boto:Data: 2024-04-19T23:33:47.287 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.287 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.287 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.287 DEBUG:boto:Params: {} 2024-04-19T23:33:47.287 DEBUG:boto:Token: None 2024-04-19T23:33:47.287 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.288 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025a637cef09bb8c0-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.290 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.290 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.291 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.291 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:47.291 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:33:47.291 DEBUG:boto:Method: GET 2024-04-19T23:33:47.291 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:33:47.291 DEBUG:boto:Data: 2024-04-19T23:33:47.291 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.291 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.291 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.291 DEBUG:boto:Params: {} 2024-04-19T23:33:47.292 DEBUG:boto:Token: None 2024-04-19T23:33:47.292 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.292 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ebfd57e2db3e3ab-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.295 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:47.296 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.296 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.296 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.296 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.296 DEBUG:boto:Method: GET 2024-04-19T23:33:47.296 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.296 DEBUG:boto:Data: 2024-04-19T23:33:47.296 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.297 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.297 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.297 DEBUG:boto:Params: {} 2024-04-19T23:33:47.297 DEBUG:boto:Token: None 2024-04-19T23:33:47.297 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.297 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7Ju9CeUuL7fYwFCGzEcnmulQB+8= 2024-04-19T23:33:47.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7Ju9CeUuL7fYwFCGzEcnmulQB+8=', 'Content-Length': '0'} 2024-04-19T23:33:47.299 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d06f0d1a93d1bf72-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.342 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.342 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.342 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.343 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.343 DEBUG:boto:Method: GET 2024-04-19T23:33:47.343 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.343 DEBUG:boto:Data: 2024-04-19T23:33:47.343 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.343 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.343 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.343 DEBUG:boto:Params: {} 2024-04-19T23:33:47.343 DEBUG:boto:Token: None 2024-04-19T23:33:47.343 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7Ju9CeUuL7fYwFCGzEcnmulQB+8= 2024-04-19T23:33:47.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7Ju9CeUuL7fYwFCGzEcnmulQB+8=', 'Content-Length': '0'} 2024-04-19T23:33:47.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db6edc7028dcbb97-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.346 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.346 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.346 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.346 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.346 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.346 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.346 DEBUG:boto:Data: 2024-04-19T23:33:47.347 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.347 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.347 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.347 DEBUG:boto:Params: {} 2024-04-19T23:33:47.347 DEBUG:boto:Token: None 2024-04-19T23:33:47.347 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.347 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nzQ0UgSZTroH/PUkx1Qy+POQBGY= 2024-04-19T23:33:47.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nzQ0UgSZTroH/PUkx1Qy+POQBGY=', 'Content-Length': '0'} 2024-04-19T23:33:47.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b41b494c06838be-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.349 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.349 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.349 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.350 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.350 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.350 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.350 DEBUG:boto:Data: 2024-04-19T23:33:47.350 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.350 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.350 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.350 DEBUG:boto:Params: {} 2024-04-19T23:33:47.351 DEBUG:boto:Token: None 2024-04-19T23:33:47.351 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.351 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nzQ0UgSZTroH/PUkx1Qy+POQBGY= 2024-04-19T23:33:47.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nzQ0UgSZTroH/PUkx1Qy+POQBGY=', 'Content-Length': '0'} 2024-04-19T23:33:47.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2fafb278b158ae0-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:47.353 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.353 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.353 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.353 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.353 DEBUG:boto:Method: GET 2024-04-19T23:33:47.353 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.353 DEBUG:boto:Data: 2024-04-19T23:33:47.353 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.354 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.354 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.354 DEBUG:boto:Params: {} 2024-04-19T23:33:47.354 DEBUG:boto:Token: None 2024-04-19T23:33:47.354 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.354 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7Ju9CeUuL7fYwFCGzEcnmulQB+8= 2024-04-19T23:33:47.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7Ju9CeUuL7fYwFCGzEcnmulQB+8=', 'Content-Length': '0'} 2024-04-19T23:33:47.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da5a9b84fbc973e8-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.398 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.398 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.398 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.399 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.399 DEBUG:boto:Method: GET 2024-04-19T23:33:47.399 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.399 DEBUG:boto:Data: 2024-04-19T23:33:47.399 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.399 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.399 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.399 DEBUG:boto:Params: {} 2024-04-19T23:33:47.399 DEBUG:boto:Token: None 2024-04-19T23:33:47.399 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.399 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7Ju9CeUuL7fYwFCGzEcnmulQB+8= 2024-04-19T23:33:47.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7Ju9CeUuL7fYwFCGzEcnmulQB+8=', 'Content-Length': '0'} 2024-04-19T23:33:47.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050e9e92715edb3dd-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.442 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.442 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.443 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:47.443 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:47.443 DEBUG:boto:Method: GET 2024-04-19T23:33:47.443 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:47.443 DEBUG:boto:Data: 2024-04-19T23:33:47.443 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.443 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.443 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.443 DEBUG:boto:Params: {} 2024-04-19T23:33:47.443 DEBUG:boto:Token: None 2024-04-19T23:33:47.443 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.444 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a074fbe43ebe4ea-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.490 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.491 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.491 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.491 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:47.491 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:47.491 DEBUG:boto:Method: GET 2024-04-19T23:33:47.491 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:47.491 DEBUG:boto:Data: 2024-04-19T23:33:47.491 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.491 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.491 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.491 DEBUG:boto:Params: {} 2024-04-19T23:33:47.492 DEBUG:boto:Token: None 2024-04-19T23:33:47.492 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.492 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb249d33550a9f21-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.494 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.495 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.495 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.495 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.495 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:33:47.495 DEBUG:boto:Data: 2024-04-19T23:33:47.495 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.495 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.495 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.495 DEBUG:boto:Params: {} 2024-04-19T23:33:47.496 DEBUG:boto:Token: None 2024-04-19T23:33:47.496 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:33:47.496 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wjdW2JrRnJbVcQQfbbGJPy8jBUE= 2024-04-19T23:33:47.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wjdW2JrRnJbVcQQfbbGJPy8jBUE=', 'Content-Length': '0'} 2024-04-19T23:33:47.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bad8bd8b6ba7f9ab-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.498 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.499 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.499 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.499 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:33:47.499 DEBUG:boto:Data: 2024-04-19T23:33:47.499 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.499 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.499 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.499 DEBUG:boto:Params: {} 2024-04-19T23:33:47.499 DEBUG:boto:Token: None 2024-04-19T23:33:47.499 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:33:47.499 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wjdW2JrRnJbVcQQfbbGJPy8jBUE= 2024-04-19T23:33:47.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wjdW2JrRnJbVcQQfbbGJPy8jBUE=', 'Content-Length': '0'} 2024-04-19T23:33:47.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059917e3857a6c84d-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:47.502 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.502 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.502 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.502 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.502 DEBUG:boto:Method: GET 2024-04-19T23:33:47.503 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.503 DEBUG:boto:Data: 2024-04-19T23:33:47.503 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.503 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.503 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.503 DEBUG:boto:Params: {} 2024-04-19T23:33:47.503 DEBUG:boto:Token: None 2024-04-19T23:33:47.503 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.503 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7Ju9CeUuL7fYwFCGzEcnmulQB+8= 2024-04-19T23:33:47.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7Ju9CeUuL7fYwFCGzEcnmulQB+8=', 'Content-Length': '0'} 2024-04-19T23:33:47.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098ed48cb20d995bb-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.550 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.550 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:33:47.551 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.551 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.551 DEBUG:boto:Method: GET 2024-04-19T23:33:47.551 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.551 DEBUG:boto:Data: 2024-04-19T23:33:47.551 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.551 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.551 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.551 DEBUG:boto:Params: {} 2024-04-19T23:33:47.551 DEBUG:boto:Token: None 2024-04-19T23:33:47.552 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:33:47.552 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7Ju9CeUuL7fYwFCGzEcnmulQB+8= 2024-04-19T23:33:47.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7Ju9CeUuL7fYwFCGzEcnmulQB+8=', 'Content-Length': '0'} 2024-04-19T23:33:47.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ecf6b43abda45ea-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.598 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.598 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.599 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:47.599 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:47.599 DEBUG:boto:Method: GET 2024-04-19T23:33:47.599 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:47.599 DEBUG:boto:Data: 2024-04-19T23:33:47.599 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.599 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.599 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.599 DEBUG:boto:Params: {} 2024-04-19T23:33:47.599 DEBUG:boto:Token: None 2024-04-19T23:33:47.599 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.600 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b17183f8615b1288-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.642 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.643 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.643 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.643 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:47.643 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:47.643 DEBUG:boto:Method: GET 2024-04-19T23:33:47.643 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:33:47.643 DEBUG:boto:Data: 2024-04-19T23:33:47.644 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.644 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.644 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.644 DEBUG:boto:Params: {} 2024-04-19T23:33:47.644 DEBUG:boto:Token: None 2024-04-19T23:33:47.644 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.644 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a906105cf6230c72-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.647 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:47.647 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.648 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.648 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.648 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.648 DEBUG:boto:Method: GET 2024-04-19T23:33:47.648 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.648 DEBUG:boto:Data: 2024-04-19T23:33:47.648 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.648 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.648 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.648 DEBUG:boto:Params: {} 2024-04-19T23:33:47.649 DEBUG:boto:Token: None 2024-04-19T23:33:47.649 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.649 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tafL/TuuH8FTfxUa5V4J7BA9ZU4= 2024-04-19T23:33:47.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tafL/TuuH8FTfxUa5V4J7BA9ZU4=', 'Content-Length': '0'} 2024-04-19T23:33:47.651 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c888d18bfad038e-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.651 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.651 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.652 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.652 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.652 DEBUG:boto:Method: GET 2024-04-19T23:33:47.652 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.652 DEBUG:boto:Data: 2024-04-19T23:33:47.652 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.652 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.652 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.652 DEBUG:boto:Params: {} 2024-04-19T23:33:47.652 DEBUG:boto:Token: None 2024-04-19T23:33:47.653 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.653 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tafL/TuuH8FTfxUa5V4J7BA9ZU4= 2024-04-19T23:33:47.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tafL/TuuH8FTfxUa5V4J7BA9ZU4=', 'Content-Length': '0'} 2024-04-19T23:33:47.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ea4da77177362a9-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.655 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.655 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.655 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.655 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.655 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.655 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.655 DEBUG:boto:Data: 2024-04-19T23:33:47.656 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.656 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.656 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.656 DEBUG:boto:Params: {} 2024-04-19T23:33:47.656 DEBUG:boto:Token: None 2024-04-19T23:33:47.656 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.656 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l34hEVfxhXeaqhUS+zmgSvYIGgY= 2024-04-19T23:33:47.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l34hEVfxhXeaqhUS+zmgSvYIGgY=', 'Content-Length': '0'} 2024-04-19T23:33:47.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b02fcbdd634b377-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.658 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.658 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.658 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.659 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.659 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.659 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.659 DEBUG:boto:Data: 2024-04-19T23:33:47.659 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.659 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.659 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.659 DEBUG:boto:Params: {} 2024-04-19T23:33:47.659 DEBUG:boto:Token: None 2024-04-19T23:33:47.659 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l34hEVfxhXeaqhUS+zmgSvYIGgY= 2024-04-19T23:33:47.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l34hEVfxhXeaqhUS+zmgSvYIGgY=', 'Content-Length': '0'} 2024-04-19T23:33:47.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032c3d527e2afccee-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:47.662 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.662 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.662 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.662 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.662 DEBUG:boto:Method: GET 2024-04-19T23:33:47.662 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.662 DEBUG:boto:Data: 2024-04-19T23:33:47.662 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.662 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.662 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.663 DEBUG:boto:Params: {} 2024-04-19T23:33:47.663 DEBUG:boto:Token: None 2024-04-19T23:33:47.663 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.663 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tafL/TuuH8FTfxUa5V4J7BA9ZU4= 2024-04-19T23:33:47.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tafL/TuuH8FTfxUa5V4J7BA9ZU4=', 'Content-Length': '0'} 2024-04-19T23:33:47.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d888eacca7be34db-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.706 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.706 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.707 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.707 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.707 DEBUG:boto:Method: GET 2024-04-19T23:33:47.707 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.707 DEBUG:boto:Data: 2024-04-19T23:33:47.707 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.707 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.707 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.707 DEBUG:boto:Params: {} 2024-04-19T23:33:47.707 DEBUG:boto:Token: None 2024-04-19T23:33:47.708 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.708 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tafL/TuuH8FTfxUa5V4J7BA9ZU4= 2024-04-19T23:33:47.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tafL/TuuH8FTfxUa5V4J7BA9ZU4=', 'Content-Length': '0'} 2024-04-19T23:33:47.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1c358015882f876-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.710 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.710 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.710 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:47.710 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:47.711 DEBUG:boto:Method: GET 2024-04-19T23:33:47.711 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:47.711 DEBUG:boto:Data: 2024-04-19T23:33:47.711 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.711 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.711 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.711 DEBUG:boto:Params: {} 2024-04-19T23:33:47.711 DEBUG:boto:Token: None 2024-04-19T23:33:47.711 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.712 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057827dce921a698c-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.754 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.755 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.755 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.755 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:47.755 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:47.755 DEBUG:boto:Method: GET 2024-04-19T23:33:47.755 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:47.755 DEBUG:boto:Data: 2024-04-19T23:33:47.755 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.755 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.756 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.756 DEBUG:boto:Params: {} 2024-04-19T23:33:47.756 DEBUG:boto:Token: None 2024-04-19T23:33:47.756 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.756 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b75b14729bd32fc-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.759 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.759 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.759 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.760 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.760 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:33:47.760 DEBUG:boto:Data: 2024-04-19T23:33:47.760 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.760 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.760 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.760 DEBUG:boto:Params: {} 2024-04-19T23:33:47.760 DEBUG:boto:Token: None 2024-04-19T23:33:47.760 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:33:47.761 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Y0RWpW+6kpGigQExwic2K3jkWEc= 2024-04-19T23:33:47.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y0RWpW+6kpGigQExwic2K3jkWEc=', 'Content-Length': '0'} 2024-04-19T23:33:47.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff32eb4cd3244329-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.763 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.763 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.763 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.764 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:33:47.764 DEBUG:boto:Data: 2024-04-19T23:33:47.764 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.764 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.764 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.764 DEBUG:boto:Params: {} 2024-04-19T23:33:47.764 DEBUG:boto:Token: None 2024-04-19T23:33:47.764 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:33:47.764 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Y0RWpW+6kpGigQExwic2K3jkWEc= 2024-04-19T23:33:47.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y0RWpW+6kpGigQExwic2K3jkWEc=', 'Content-Length': '0'} 2024-04-19T23:33:47.767 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000283bab2da9d87037-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:47.768 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.768 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.768 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.768 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.768 DEBUG:boto:Method: GET 2024-04-19T23:33:47.768 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.768 DEBUG:boto:Data: 2024-04-19T23:33:47.769 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.769 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.769 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.769 DEBUG:boto:Params: {} 2024-04-19T23:33:47.769 DEBUG:boto:Token: None 2024-04-19T23:33:47.769 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.769 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tafL/TuuH8FTfxUa5V4J7BA9ZU4= 2024-04-19T23:33:47.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tafL/TuuH8FTfxUa5V4J7BA9ZU4=', 'Content-Length': '0'} 2024-04-19T23:33:47.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000183f6df4779a40b2-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.818 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.818 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:33:47.818 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.819 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.819 DEBUG:boto:Method: GET 2024-04-19T23:33:47.819 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.819 DEBUG:boto:Data: 2024-04-19T23:33:47.819 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.819 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.819 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.819 DEBUG:boto:Params: {} 2024-04-19T23:33:47.819 DEBUG:boto:Token: None 2024-04-19T23:33:47.819 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:33:47.820 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tafL/TuuH8FTfxUa5V4J7BA9ZU4= 2024-04-19T23:33:47.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tafL/TuuH8FTfxUa5V4J7BA9ZU4=', 'Content-Length': '0'} 2024-04-19T23:33:47.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db281ea8e14cc923-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.822 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.822 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.822 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:47.822 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:47.822 DEBUG:boto:Method: GET 2024-04-19T23:33:47.822 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:47.823 DEBUG:boto:Data: 2024-04-19T23:33:47.823 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.823 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.823 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.823 DEBUG:boto:Params: {} 2024-04-19T23:33:47.823 DEBUG:boto:Token: None 2024-04-19T23:33:47.823 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.823 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fee0da64bccea80-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.870 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.871 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.871 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.871 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:47.871 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:47.871 DEBUG:boto:Method: GET 2024-04-19T23:33:47.871 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:33:47.871 DEBUG:boto:Data: 2024-04-19T23:33:47.871 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.871 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.871 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.872 DEBUG:boto:Params: {} 2024-04-19T23:33:47.872 DEBUG:boto:Token: None 2024-04-19T23:33:47.872 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.872 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003becc8d97d3a1c22-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.875 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:47.875 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.875 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.875 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.875 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.876 DEBUG:boto:Method: GET 2024-04-19T23:33:47.876 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.876 DEBUG:boto:Data: 2024-04-19T23:33:47.876 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.876 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.876 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.876 DEBUG:boto:Params: {} 2024-04-19T23:33:47.876 DEBUG:boto:Token: None 2024-04-19T23:33:47.876 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.876 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM= 2024-04-19T23:33:47.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM=', 'Content-Length': '0'} 2024-04-19T23:33:47.878 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a9b948a0dfc4d4d-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.879 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.879 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.879 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.879 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.879 DEBUG:boto:Method: GET 2024-04-19T23:33:47.879 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.879 DEBUG:boto:Data: 2024-04-19T23:33:47.879 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.880 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.880 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.880 DEBUG:boto:Params: {} 2024-04-19T23:33:47.880 DEBUG:boto:Token: None 2024-04-19T23:33:47.880 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM= 2024-04-19T23:33:47.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM=', 'Content-Length': '0'} 2024-04-19T23:33:47.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bd4dd8a78e74e53-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.882 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.883 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.883 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.883 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.883 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.883 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.883 DEBUG:boto:Data: 2024-04-19T23:33:47.883 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.883 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.883 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.883 DEBUG:boto:Params: {} 2024-04-19T23:33:47.884 DEBUG:boto:Token: None 2024-04-19T23:33:47.884 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.884 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OmJjjMpGeNPWsm08I/EY/4GSkc0= 2024-04-19T23:33:47.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OmJjjMpGeNPWsm08I/EY/4GSkc0=', 'Content-Length': '0'} 2024-04-19T23:33:47.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebeea5fdd1f0b990-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.886 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.886 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.886 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.887 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.887 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.887 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.887 DEBUG:boto:Data: 2024-04-19T23:33:47.887 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.887 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.887 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.887 DEBUG:boto:Params: {} 2024-04-19T23:33:47.887 DEBUG:boto:Token: None 2024-04-19T23:33:47.887 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.888 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OmJjjMpGeNPWsm08I/EY/4GSkc0= 2024-04-19T23:33:47.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OmJjjMpGeNPWsm08I/EY/4GSkc0=', 'Content-Length': '0'} 2024-04-19T23:33:47.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045d56473e217721c-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:47.890 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.890 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.890 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.890 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.890 DEBUG:boto:Method: GET 2024-04-19T23:33:47.890 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.891 DEBUG:boto:Data: 2024-04-19T23:33:47.891 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.891 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.891 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.891 DEBUG:boto:Params: {} 2024-04-19T23:33:47.891 DEBUG:boto:Token: None 2024-04-19T23:33:47.891 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.891 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM= 2024-04-19T23:33:47.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM=', 'Content-Length': '0'} 2024-04-19T23:33:47.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049385df7826221be-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.893 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.893 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.894 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.894 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.894 DEBUG:boto:Method: GET 2024-04-19T23:33:47.894 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.894 DEBUG:boto:Data: 2024-04-19T23:33:47.894 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.894 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.894 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.894 DEBUG:boto:Params: {} 2024-04-19T23:33:47.895 DEBUG:boto:Token: None 2024-04-19T23:33:47.895 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.895 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM= 2024-04-19T23:33:47.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM=', 'Content-Length': '0'} 2024-04-19T23:33:47.897 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000328bc1d5a096ff5b-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.942 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.942 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.943 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.943 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.943 DEBUG:boto:Method: GET 2024-04-19T23:33:47.943 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.943 DEBUG:boto:Data: 2024-04-19T23:33:47.943 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.943 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.943 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.943 DEBUG:boto:Params: {} 2024-04-19T23:33:47.943 DEBUG:boto:Token: None 2024-04-19T23:33:47.944 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000311cde84f837171e-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.947 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.947 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:47.947 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:47.947 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.947 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.947 DEBUG:boto:Method: GET 2024-04-19T23:33:47.947 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:47.948 DEBUG:boto:Data: 2024-04-19T23:33:47.948 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.948 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.948 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.948 DEBUG:boto:Params: {} 2024-04-19T23:33:47.948 DEBUG:boto:Token: None 2024-04-19T23:33:47.949 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/ 2024-04-19T23:33:47.949 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8= 2024-04-19T23:33:47.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rl0OsCTNOqMRdiJvCN0DMJgg7D8=', 'Content-Length': '0'} 2024-04-19T23:33:47.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec3f6cbc3efd66ce-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.951 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:47.952 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.952 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.952 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.952 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:33:47.952 DEBUG:boto:Data: 2024-04-19T23:33:47.952 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.952 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.952 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.952 DEBUG:boto:Params: {} 2024-04-19T23:33:47.953 DEBUG:boto:Token: None 2024-04-19T23:33:47.953 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:33:47.953 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nN5nPWoVrQrZv9C7hmM7t3TP2pY= 2024-04-19T23:33:47.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nN5nPWoVrQrZv9C7hmM7t3TP2pY=', 'Content-Length': '0'} 2024-04-19T23:33:47.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb230b5e06f636e8-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.955 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.955 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.956 DEBUG:boto:Method: HEAD 2024-04-19T23:33:47.956 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:33:47.956 DEBUG:boto:Data: 2024-04-19T23:33:47.956 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.956 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:47.956 DEBUG:boto:Port: 8000 2024-04-19T23:33:47.956 DEBUG:boto:Params: {} 2024-04-19T23:33:47.956 DEBUG:boto:Token: None 2024-04-19T23:33:47.956 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:33:47.957 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nN5nPWoVrQrZv9C7hmM7t3TP2pY= 2024-04-19T23:33:47.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nN5nPWoVrQrZv9C7hmM7t3TP2pY=', 'Content-Length': '0'} 2024-04-19T23:33:47.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087cafc6c1fda920c-006622ff5b-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:47.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:47.959 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.959 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:47.960 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.960 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.960 DEBUG:boto:Method: GET 2024-04-19T23:33:47.960 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.960 DEBUG:boto:Data: 2024-04-19T23:33:47.960 DEBUG:boto:Headers: {} 2024-04-19T23:33:47.960 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:47.960 DEBUG:boto:Port: 8001 2024-04-19T23:33:47.960 DEBUG:boto:Params: {} 2024-04-19T23:33:47.961 DEBUG:boto:Token: None 2024-04-19T23:33:47.961 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:47 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:47.961 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM= 2024-04-19T23:33:47.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ChjyCRnJYNqufBmw3zq+cngOyMM=', 'Content-Length': '0'} 2024-04-19T23:33:47.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea4d04e2a3c71370-006622ff5b-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:48.006 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:48.006 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:33:48.007 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:48.007 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:48.007 DEBUG:boto:Method: GET 2024-04-19T23:33:48.007 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:48.007 DEBUG:boto:Data: 2024-04-19T23:33:48.007 DEBUG:boto:Headers: {} 2024-04-19T23:33:48.007 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:48.007 DEBUG:boto:Port: 8000 2024-04-19T23:33:48.007 DEBUG:boto:Params: {} 2024-04-19T23:33:48.007 DEBUG:boto:Token: None 2024-04-19T23:33:48.008 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:48 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:33:48.008 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hkoMZ51n9F4tbvLipbxTKP7Yb6U= 2024-04-19T23:33:48.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:48 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hkoMZ51n9F4tbvLipbxTKP7Yb6U=', 'Content-Length': '0'} 2024-04-19T23:33:48.010 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fac6519af919bb2b-006622ff5c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:48.054 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:48.055 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:48.055 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:48.055 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:48.055 DEBUG:boto:Method: GET 2024-04-19T23:33:48.055 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:48.055 DEBUG:boto:Data: 2024-04-19T23:33:48.055 DEBUG:boto:Headers: {} 2024-04-19T23:33:48.055 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:48.055 DEBUG:boto:Port: 8001 2024-04-19T23:33:48.055 DEBUG:boto:Params: {} 2024-04-19T23:33:48.056 DEBUG:boto:Token: None 2024-04-19T23:33:48.056 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:48 GMT /aktlji-20/ 2024-04-19T23:33:48.056 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LraFlfMlUTf70vHqolEWthi++P4= 2024-04-19T23:33:48.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:48 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LraFlfMlUTf70vHqolEWthi++P4=', 'Content-Length': '0'} 2024-04-19T23:33:48.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030445d1eef6a7cc4-006622ff5c-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:48.058 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:48.059 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:33:48.059 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:33:48.059 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:48.059 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:48.059 DEBUG:boto:Method: GET 2024-04-19T23:33:48.059 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:33:48.059 DEBUG:boto:Data: 2024-04-19T23:33:48.060 DEBUG:boto:Headers: {} 2024-04-19T23:33:48.060 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:48.060 DEBUG:boto:Port: 8000 2024-04-19T23:33:48.060 DEBUG:boto:Params: {} 2024-04-19T23:33:48.060 DEBUG:boto:Token: None 2024-04-19T23:33:48.060 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:48 GMT /aktlji-20/ 2024-04-19T23:33:48.060 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LraFlfMlUTf70vHqolEWthi++P4= 2024-04-19T23:33:48.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:48 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LraFlfMlUTf70vHqolEWthi++P4=', 'Content-Length': '0'} 2024-04-19T23:33:48.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000234d224c2cd97579-006622ff5c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:48.106 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:48.107 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-20 zones={test-zone3, test-zone2} 2024-04-19T23:33:48.107 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-21 2024-04-19T23:33:48.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:48.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:48.108 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:48.174 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:48.174 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:48.223+0000 7fad88f77780 1 bucket sync caught up with source: 2024-04-19T23:33:48.219 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000010.28.6, , 00000000005.51.10, , 00000000010.23.10, , , , , ] 2024-04-19T23:33:48.219 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000010.28.6, , 00000000005.51.10, , 00000000010.23.10, , , , , ] 2024-04-19T23:33:48.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:48.223+0000 7fad88f77780 0 bucket checkpoint complete 2024-04-19T23:33:48.234 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-21 2024-04-19T23:33:48.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:48.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:48.234 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:48.302 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:48.302 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:48.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:48.348+0000 7f2900a21780 1 bucket sync caught up with source: 2024-04-19T23:33:48.343 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000010.28.6, , 00000000005.51.10, , 00000000010.23.10, , , , , ] 2024-04-19T23:33:48.343 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000010.28.6, , 00000000005.51.10, , 00000000010.23.10, , , , , ] 2024-04-19T23:33:48.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:48.348+0000 7f2900a21780 0 bucket checkpoint complete 2024-04-19T23:33:48.359 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-21 2024-04-19T23:33:48.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:48.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:48.359 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:48.422 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:48.422 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:48.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:48.466+0000 7f7da25a2780 1 bucket sync caught up with source: 2024-04-19T23:33:48.462 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.85.10, , 00000000010.112.6, , 00000000007.75.10, , , , , ] 2024-04-19T23:33:48.462 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.85.10, , 00000000010.112.6, , 00000000007.75.10, , , , , ] 2024-04-19T23:33:48.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:48.466+0000 7f7da25a2780 0 bucket checkpoint complete 2024-04-19T23:33:48.476 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-21 2024-04-19T23:33:48.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:48.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:48.476 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:48.538 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:48.538 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:48.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:48.582+0000 7f2730e18780 1 bucket sync caught up with source: 2024-04-19T23:33:48.578 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.85.10, , 00000000010.112.6, , 00000000007.75.10, , , , , ] 2024-04-19T23:33:48.578 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.85.10, , 00000000010.112.6, , 00000000007.75.10, , , , , ] 2024-04-19T23:33:48.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:48.582+0000 7f2730e18780 0 bucket checkpoint complete 2024-04-19T23:33:48.592 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-21 2024-04-19T23:33:48.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:48.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:48.593 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:48.655 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:48.655 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:48.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:48.699+0000 7fa382ed4780 1 waiting for incremental sync to catch up: 2024-04-19T23:33:48.694 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.45.10, , 00000000004.78.10, , 00000000017.57.10, , , , , ] 2024-04-19T23:33:48.695 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.45.10, , 00000000005.80.10, , 00000000018.60.10, , , , , ] 2024-04-19T23:33:53.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:53.700+0000 7fa382ed4780 1 waiting for incremental sync to catch up: 2024-04-19T23:33:53.696 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.45.10, , 00000000005.80.10, , 00000000017.57.10, , , , , ] 2024-04-19T23:33:53.696 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.45.10, , 00000000005.80.10, , 00000000018.60.10, , , , , ] 2024-04-19T23:33:58.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:58.701+0000 7fa382ed4780 1 bucket sync caught up with source: 2024-04-19T23:33:58.697 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.45.10, , 00000000005.80.10, , 00000000018.60.10, , , , , ] 2024-04-19T23:33:58.697 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.45.10, , 00000000005.80.10, , 00000000018.60.10, , , , , ] 2024-04-19T23:33:58.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:33:58.701+0000 7fa382ed4780 0 bucket checkpoint complete 2024-04-19T23:33:58.714 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-21 2024-04-19T23:33:58.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:58.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:33:58.715 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:33:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:33:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:33:58.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:58.831+0000 7f3f53820780 1 bucket sync caught up with source: 2024-04-19T23:33:58.827 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000003.45.10, , 00000000005.80.10, , 00000000018.60.10, , , , , ] 2024-04-19T23:33:58.827 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000003.45.10, , 00000000005.80.10, , 00000000018.60.10, , , , , ] 2024-04-19T23:33:58.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:33:58.831+0000 7f3f53820780 0 bucket checkpoint complete 2024-04-19T23:33:58.841 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-21 zones={test-zone2, test-zone1} 2024-04-19T23:33:58.842 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:58.842 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:58.842 DEBUG:boto:Method: HEAD 2024-04-19T23:33:58.842 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:33:58.842 DEBUG:boto:Data: 2024-04-19T23:33:58.842 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.842 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:58.842 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.843 DEBUG:boto:Params: {} 2024-04-19T23:33:58.843 DEBUG:boto:Token: None 2024-04-19T23:33:58.843 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/ 2024-04-19T23:33:58.843 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4nF83RSbh1uxa7lZGAWNBX5rxP4= 2024-04-19T23:33:58.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4nF83RSbh1uxa7lZGAWNBX5rxP4=', 'Content-Length': '0'} 2024-04-19T23:33:58.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5069374b15e45b2-006622ff66-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.846 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:58.846 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:58.846 DEBUG:boto:Method: HEAD 2024-04-19T23:33:58.846 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:33:58.846 DEBUG:boto:Data: 2024-04-19T23:33:58.846 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.846 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:58.846 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.847 DEBUG:boto:Params: {} 2024-04-19T23:33:58.847 DEBUG:boto:Token: None 2024-04-19T23:33:58.847 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/ 2024-04-19T23:33:58.847 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4nF83RSbh1uxa7lZGAWNBX5rxP4= 2024-04-19T23:33:58.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4nF83RSbh1uxa7lZGAWNBX5rxP4=', 'Content-Length': '0'} 2024-04-19T23:33:58.850 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f34ad6a19f8125b-006622ff66-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.850 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:33:58.850 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:58.850 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:58.850 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:58.850 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:58.850 DEBUG:boto:Method: GET 2024-04-19T23:33:58.850 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:33:58.850 DEBUG:boto:Data: 2024-04-19T23:33:58.851 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.851 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:58.851 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.851 DEBUG:boto:Params: {} 2024-04-19T23:33:58.851 DEBUG:boto:Token: None 2024-04-19T23:33:58.851 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/?versions 2024-04-19T23:33:58.851 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxExGA31zRrccpxvn/1YhU/Vzjs= 2024-04-19T23:33:58.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxExGA31zRrccpxvn/1YhU/Vzjs=', 'Content-Length': '0'} 2024-04-19T23:33:58.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d6b667456798a97-006622ff66-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.854 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:58.854 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:58.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:58.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:58.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:58.855 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:33:58.855 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:58.855 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:58.855 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:58.855 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:58.855 DEBUG:boto:Method: GET 2024-04-19T23:33:58.855 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:33:58.855 DEBUG:boto:Data: 2024-04-19T23:33:58.855 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.855 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:58.856 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.856 DEBUG:boto:Params: {} 2024-04-19T23:33:58.856 DEBUG:boto:Token: None 2024-04-19T23:33:58.856 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/?versions 2024-04-19T23:33:58.856 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxExGA31zRrccpxvn/1YhU/Vzjs= 2024-04-19T23:33:58.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxExGA31zRrccpxvn/1YhU/Vzjs=', 'Content-Length': '0'} 2024-04-19T23:33:58.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b82353e54e31fe7-006622ff66-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.859 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:58.859 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:58.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:58.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:58.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:58.860 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:58.860 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:58.860 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:58.860 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:58.860 DEBUG:boto:Method: GET 2024-04-19T23:33:58.860 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:33:58.860 DEBUG:boto:Data: 2024-04-19T23:33:58.860 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.860 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:58.860 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.860 DEBUG:boto:Params: {} 2024-04-19T23:33:58.861 DEBUG:boto:Token: None 2024-04-19T23:33:58.861 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/?versions 2024-04-19T23:33:58.861 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxExGA31zRrccpxvn/1YhU/Vzjs= 2024-04-19T23:33:58.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxExGA31zRrccpxvn/1YhU/Vzjs=', 'Content-Length': '0'} 2024-04-19T23:33:58.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b596e32cb3e07976-006622ff66-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.864 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:58.864 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:58.864 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:58.864 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:58.864 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:58.864 DEBUG:boto:Method: GET 2024-04-19T23:33:58.864 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:33:58.865 DEBUG:boto:Data: 2024-04-19T23:33:58.865 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.865 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:58.865 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.865 DEBUG:boto:Params: {} 2024-04-19T23:33:58.865 DEBUG:boto:Token: None 2024-04-19T23:33:58.865 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/?versions 2024-04-19T23:33:58.865 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxExGA31zRrccpxvn/1YhU/Vzjs= 2024-04-19T23:33:58.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxExGA31zRrccpxvn/1YhU/Vzjs=', 'Content-Length': '0'} 2024-04-19T23:33:58.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8e04c6f77f695ad-006622ff66-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.868 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:58.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:58.869 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:58.869 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:58.869 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.869 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.869 DEBUG:boto:Method: GET 2024-04-19T23:33:58.869 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.869 DEBUG:boto:Data: 2024-04-19T23:33:58.869 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.869 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:58.869 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.869 DEBUG:boto:Params: {} 2024-04-19T23:33:58.869 DEBUG:boto:Token: None 2024-04-19T23:33:58.870 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.870 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z/CMRvfgz8/A/Wx+mL09Z/F0bG8= 2024-04-19T23:33:58.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z/CMRvfgz8/A/Wx+mL09Z/F0bG8=', 'Content-Length': '0'} 2024-04-19T23:33:58.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008fa95e81b253f4d5-006622ff66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.872 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:58.872 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:58.872 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.872 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.872 DEBUG:boto:Method: GET 2024-04-19T23:33:58.872 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.872 DEBUG:boto:Data: 2024-04-19T23:33:58.872 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.872 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:58.873 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.873 DEBUG:boto:Params: {} 2024-04-19T23:33:58.873 DEBUG:boto:Token: None 2024-04-19T23:33:58.873 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.873 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z/CMRvfgz8/A/Wx+mL09Z/F0bG8= 2024-04-19T23:33:58.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z/CMRvfgz8/A/Wx+mL09Z/F0bG8=', 'Content-Length': '0'} 2024-04-19T23:33:58.875 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c37f305818845a74-006622ff66-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.875 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:58.875 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:58.875 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.875 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.875 DEBUG:boto:Method: HEAD 2024-04-19T23:33:58.876 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.876 DEBUG:boto:Data: 2024-04-19T23:33:58.876 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.876 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:58.876 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.876 DEBUG:boto:Params: {} 2024-04-19T23:33:58.876 DEBUG:boto:Token: None 2024-04-19T23:33:58.876 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.876 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L0olNZpjJtHE9VtuY2HGgDDepFs= 2024-04-19T23:33:58.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L0olNZpjJtHE9VtuY2HGgDDepFs=', 'Content-Length': '0'} 2024-04-19T23:33:58.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d0f69e4e9a57cb6-006622ff66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.878 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:58.878 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:58.878 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.879 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.879 DEBUG:boto:Method: HEAD 2024-04-19T23:33:58.879 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.879 DEBUG:boto:Data: 2024-04-19T23:33:58.879 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.879 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:58.879 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.879 DEBUG:boto:Params: {} 2024-04-19T23:33:58.879 DEBUG:boto:Token: None 2024-04-19T23:33:58.879 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L0olNZpjJtHE9VtuY2HGgDDepFs= 2024-04-19T23:33:58.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L0olNZpjJtHE9VtuY2HGgDDepFs=', 'Content-Length': '0'} 2024-04-19T23:33:58.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bb93e748681dc20d-006622ff66-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:58.882 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:58.882 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:58.882 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.882 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.882 DEBUG:boto:Method: GET 2024-04-19T23:33:58.882 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.882 DEBUG:boto:Data: 2024-04-19T23:33:58.882 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.882 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:58.882 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.882 DEBUG:boto:Params: {} 2024-04-19T23:33:58.883 DEBUG:boto:Token: None 2024-04-19T23:33:58.883 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.883 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z/CMRvfgz8/A/Wx+mL09Z/F0bG8= 2024-04-19T23:33:58.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z/CMRvfgz8/A/Wx+mL09Z/F0bG8=', 'Content-Length': '0'} 2024-04-19T23:33:58.885 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006cae8303377e1f96-006622ff66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.885 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:58.885 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:58.885 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.885 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.885 DEBUG:boto:Method: GET 2024-04-19T23:33:58.885 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.885 DEBUG:boto:Data: 2024-04-19T23:33:58.886 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.886 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:58.886 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.886 DEBUG:boto:Params: {} 2024-04-19T23:33:58.886 DEBUG:boto:Token: None 2024-04-19T23:33:58.886 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.886 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z/CMRvfgz8/A/Wx+mL09Z/F0bG8= 2024-04-19T23:33:58.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z/CMRvfgz8/A/Wx+mL09Z/F0bG8=', 'Content-Length': '0'} 2024-04-19T23:33:58.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad994f8ba2082014-006622ff66-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.888 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:58.888 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:58.888 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:58.888 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:58.889 DEBUG:boto:Method: GET 2024-04-19T23:33:58.889 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:33:58.889 DEBUG:boto:Data: 2024-04-19T23:33:58.889 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.889 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:58.889 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.889 DEBUG:boto:Params: {} 2024-04-19T23:33:58.889 DEBUG:boto:Token: None 2024-04-19T23:33:58.889 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/ 2024-04-19T23:33:58.889 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sLWLIcrZVOCG246g0NrTNBnwHwo= 2024-04-19T23:33:58.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sLWLIcrZVOCG246g0NrTNBnwHwo=', 'Content-Length': '0'} 2024-04-19T23:33:58.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c766b1e130c53a2-006622ff66-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.892 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:58.892 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:58.893 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:58.893 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:58.893 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:58.893 DEBUG:boto:Method: GET 2024-04-19T23:33:58.893 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:33:58.893 DEBUG:boto:Data: 2024-04-19T23:33:58.893 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.893 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:58.893 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.893 DEBUG:boto:Params: {} 2024-04-19T23:33:58.893 DEBUG:boto:Token: None 2024-04-19T23:33:58.894 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/ 2024-04-19T23:33:58.894 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sLWLIcrZVOCG246g0NrTNBnwHwo= 2024-04-19T23:33:58.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sLWLIcrZVOCG246g0NrTNBnwHwo=', 'Content-Length': '0'} 2024-04-19T23:33:58.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000414123ab321619f5-006622ff66-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.938 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:58.939 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:58.939 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:58.939 DEBUG:boto:Method: HEAD 2024-04-19T23:33:58.939 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:33:58.940 DEBUG:boto:Data: 2024-04-19T23:33:58.940 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.940 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:58.940 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.940 DEBUG:boto:Params: {} 2024-04-19T23:33:58.940 DEBUG:boto:Token: None 2024-04-19T23:33:58.940 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/dummy 2024-04-19T23:33:58.940 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:stxZnjl/80p0mdYpFoANNZ1RPGM= 2024-04-19T23:33:58.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:stxZnjl/80p0mdYpFoANNZ1RPGM=', 'Content-Length': '0'} 2024-04-19T23:33:58.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d85d3e5741de3a37-006622ff66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.943 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:58.943 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:58.943 DEBUG:boto:Method: HEAD 2024-04-19T23:33:58.943 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:33:58.943 DEBUG:boto:Data: 2024-04-19T23:33:58.943 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.944 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:58.944 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.944 DEBUG:boto:Params: {} 2024-04-19T23:33:58.944 DEBUG:boto:Token: None 2024-04-19T23:33:58.944 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/dummy 2024-04-19T23:33:58.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:stxZnjl/80p0mdYpFoANNZ1RPGM= 2024-04-19T23:33:58.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:stxZnjl/80p0mdYpFoANNZ1RPGM=', 'Content-Length': '0'} 2024-04-19T23:33:58.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2fd3d09f372d9bd-006622ff66-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:58.947 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:58.947 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:58.947 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.947 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.947 DEBUG:boto:Method: GET 2024-04-19T23:33:58.947 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.947 DEBUG:boto:Data: 2024-04-19T23:33:58.947 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.948 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:58.948 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.948 DEBUG:boto:Params: {} 2024-04-19T23:33:58.948 DEBUG:boto:Token: None 2024-04-19T23:33:58.948 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.948 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z/CMRvfgz8/A/Wx+mL09Z/F0bG8= 2024-04-19T23:33:58.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z/CMRvfgz8/A/Wx+mL09Z/F0bG8=', 'Content-Length': '0'} 2024-04-19T23:33:58.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000485cc96765e37f3d-006622ff66-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.950 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:58.950 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:58.950 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.950 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.951 DEBUG:boto:Method: GET 2024-04-19T23:33:58.951 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.951 DEBUG:boto:Data: 2024-04-19T23:33:58.951 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.951 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:58.951 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.951 DEBUG:boto:Params: {} 2024-04-19T23:33:58.951 DEBUG:boto:Token: None 2024-04-19T23:33:58.951 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:58.952 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z/CMRvfgz8/A/Wx+mL09Z/F0bG8= 2024-04-19T23:33:58.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z/CMRvfgz8/A/Wx+mL09Z/F0bG8=', 'Content-Length': '0'} 2024-04-19T23:33:58.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000838aa95dfac5b71-006622ff66-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.953 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:58.953 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:58.954 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:58.954 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:58.954 DEBUG:boto:Method: GET 2024-04-19T23:33:58.954 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:33:58.954 DEBUG:boto:Data: 2024-04-19T23:33:58.954 DEBUG:boto:Headers: {} 2024-04-19T23:33:58.954 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:58.954 DEBUG:boto:Port: 8000 2024-04-19T23:33:58.954 DEBUG:boto:Params: {} 2024-04-19T23:33:58.954 DEBUG:boto:Token: None 2024-04-19T23:33:58.955 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:58 GMT /aktlji-21/ 2024-04-19T23:33:58.955 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sLWLIcrZVOCG246g0NrTNBnwHwo= 2024-04-19T23:33:58.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sLWLIcrZVOCG246g0NrTNBnwHwo=', 'Content-Length': '0'} 2024-04-19T23:33:58.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000508cc0604f22569d-006622ff66-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:58.998 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:58.999 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:58.999 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:58.999 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:58.999 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:58.999 DEBUG:boto:Method: GET 2024-04-19T23:33:58.999 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:33:58.999 DEBUG:boto:Data: 2024-04-19T23:33:59.000 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.000 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.000 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.000 DEBUG:boto:Params: {} 2024-04-19T23:33:59.000 DEBUG:boto:Token: None 2024-04-19T23:33:59.000 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.000 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f0433d49dbe62a1-006622ff67-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.003 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:59.004 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.004 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.004 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.004 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.004 DEBUG:boto:Method: GET 2024-04-19T23:33:59.004 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.004 DEBUG:boto:Data: 2024-04-19T23:33:59.004 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.004 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.004 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.004 DEBUG:boto:Params: {} 2024-04-19T23:33:59.005 DEBUG:boto:Token: None 2024-04-19T23:33:59.005 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.005 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE= 2024-04-19T23:33:59.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE=', 'Content-Length': '0'} 2024-04-19T23:33:59.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4dd5276492fe496-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.007 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.007 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.007 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.008 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.008 DEBUG:boto:Method: GET 2024-04-19T23:33:59.008 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.008 DEBUG:boto:Data: 2024-04-19T23:33:59.008 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.008 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.008 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.008 DEBUG:boto:Params: {} 2024-04-19T23:33:59.009 DEBUG:boto:Token: None 2024-04-19T23:33:59.009 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.009 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE= 2024-04-19T23:33:59.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE=', 'Content-Length': '0'} 2024-04-19T23:33:59.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a78a76d4c8d7cff7-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.058 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.059 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.059 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.059 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.059 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.059 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.059 DEBUG:boto:Data: 2024-04-19T23:33:59.059 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.059 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.059 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.059 DEBUG:boto:Params: {} 2024-04-19T23:33:59.059 DEBUG:boto:Token: None 2024-04-19T23:33:59.060 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.060 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8DY6rasn9PtDZ/yP3QugQTYejgw= 2024-04-19T23:33:59.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8DY6rasn9PtDZ/yP3QugQTYejgw=', 'Content-Length': '0'} 2024-04-19T23:33:59.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bc79dae6d64a08e-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.062 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.062 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.062 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.063 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.063 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.063 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.063 DEBUG:boto:Data: 2024-04-19T23:33:59.063 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.063 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.063 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.063 DEBUG:boto:Params: {} 2024-04-19T23:33:59.063 DEBUG:boto:Token: None 2024-04-19T23:33:59.063 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.064 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8DY6rasn9PtDZ/yP3QugQTYejgw= 2024-04-19T23:33:59.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8DY6rasn9PtDZ/yP3QugQTYejgw=', 'Content-Length': '0'} 2024-04-19T23:33:59.066 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a51f7f9ed97ef18-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:59.067 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.067 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.067 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.067 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.067 DEBUG:boto:Method: GET 2024-04-19T23:33:59.067 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.068 DEBUG:boto:Data: 2024-04-19T23:33:59.068 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.068 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.068 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.068 DEBUG:boto:Params: {} 2024-04-19T23:33:59.068 DEBUG:boto:Token: None 2024-04-19T23:33:59.068 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.068 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE= 2024-04-19T23:33:59.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE=', 'Content-Length': '0'} 2024-04-19T23:33:59.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae1e0ef9365c7ec3-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.110 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.111 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.111 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.111 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.111 DEBUG:boto:Method: GET 2024-04-19T23:33:59.111 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.111 DEBUG:boto:Data: 2024-04-19T23:33:59.111 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.111 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.111 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.111 DEBUG:boto:Params: {} 2024-04-19T23:33:59.111 DEBUG:boto:Token: None 2024-04-19T23:33:59.112 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.112 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE= 2024-04-19T23:33:59.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE=', 'Content-Length': '0'} 2024-04-19T23:33:59.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9fc66570b5779d2-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.114 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.114 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.114 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:33:59.114 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:33:59.114 DEBUG:boto:Method: GET 2024-04-19T23:33:59.115 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:33:59.115 DEBUG:boto:Data: 2024-04-19T23:33:59.115 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.115 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.115 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.115 DEBUG:boto:Params: {} 2024-04-19T23:33:59.115 DEBUG:boto:Token: None 2024-04-19T23:33:59.115 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.115 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097c8406158d6e7ea-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.118 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.118 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.118 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.118 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:33:59.119 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:33:59.119 DEBUG:boto:Method: GET 2024-04-19T23:33:59.119 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:33:59.119 DEBUG:boto:Data: 2024-04-19T23:33:59.119 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.119 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.119 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.119 DEBUG:boto:Params: {} 2024-04-19T23:33:59.119 DEBUG:boto:Token: None 2024-04-19T23:33:59.119 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.119 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090625f889354d699-006622ff67-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.122 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.123 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.123 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.123 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.123 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:33:59.123 DEBUG:boto:Data: 2024-04-19T23:33:59.123 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.123 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.123 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.123 DEBUG:boto:Params: {} 2024-04-19T23:33:59.123 DEBUG:boto:Token: None 2024-04-19T23:33:59.124 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:33:59.124 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LfICJvSK3gsISLChXgt90GMAcpI= 2024-04-19T23:33:59.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LfICJvSK3gsISLChXgt90GMAcpI=', 'Content-Length': '0'} 2024-04-19T23:33:59.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dedbd4bee96cb14b-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.126 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.126 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.126 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.127 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:33:59.127 DEBUG:boto:Data: 2024-04-19T23:33:59.127 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.127 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.127 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.127 DEBUG:boto:Params: {} 2024-04-19T23:33:59.127 DEBUG:boto:Token: None 2024-04-19T23:33:59.127 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:33:59.127 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LfICJvSK3gsISLChXgt90GMAcpI= 2024-04-19T23:33:59.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LfICJvSK3gsISLChXgt90GMAcpI=', 'Content-Length': '0'} 2024-04-19T23:33:59.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ad59a9d0fdf72be-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:59.131 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.131 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.131 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.131 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.131 DEBUG:boto:Method: GET 2024-04-19T23:33:59.131 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.131 DEBUG:boto:Data: 2024-04-19T23:33:59.131 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.131 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.132 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.132 DEBUG:boto:Params: {} 2024-04-19T23:33:59.132 DEBUG:boto:Token: None 2024-04-19T23:33:59.132 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.132 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE= 2024-04-19T23:33:59.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE=', 'Content-Length': '0'} 2024-04-19T23:33:59.134 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099ea6a4872283be0-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.174 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.175 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.175 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.175 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.175 DEBUG:boto:Method: GET 2024-04-19T23:33:59.175 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.175 DEBUG:boto:Data: 2024-04-19T23:33:59.176 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.176 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.176 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.176 DEBUG:boto:Params: {} 2024-04-19T23:33:59.176 DEBUG:boto:Token: None 2024-04-19T23:33:59.177 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.177 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE= 2024-04-19T23:33:59.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE=', 'Content-Length': '0'} 2024-04-19T23:33:59.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bca25c511c2f32bd-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.180 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.180 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.180 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:33:59.180 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:33:59.180 DEBUG:boto:Method: GET 2024-04-19T23:33:59.180 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:33:59.180 DEBUG:boto:Data: 2024-04-19T23:33:59.180 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.180 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.180 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.180 DEBUG:boto:Params: {} 2024-04-19T23:33:59.181 DEBUG:boto:Token: None 2024-04-19T23:33:59.181 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.181 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000891ac3f754066292-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.230 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.231 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.231 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.232 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:33:59.232 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:33:59.232 DEBUG:boto:Method: GET 2024-04-19T23:33:59.232 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:33:59.232 DEBUG:boto:Data: 2024-04-19T23:33:59.232 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.233 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.233 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.233 DEBUG:boto:Params: {} 2024-04-19T23:33:59.233 DEBUG:boto:Token: None 2024-04-19T23:33:59.233 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.234 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066b9ea9354c8b91a-006622ff67-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.237 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:59.238 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.238 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.238 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.239 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.239 DEBUG:boto:Method: GET 2024-04-19T23:33:59.239 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.239 DEBUG:boto:Data: 2024-04-19T23:33:59.239 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.239 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.240 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.240 DEBUG:boto:Params: {} 2024-04-19T23:33:59.240 DEBUG:boto:Token: None 2024-04-19T23:33:59.241 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.241 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LndQcIPNlI7blTB7iX6un2aCK9M= 2024-04-19T23:33:59.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LndQcIPNlI7blTB7iX6un2aCK9M=', 'Content-Length': '0'} 2024-04-19T23:33:59.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2d47ea30e3d5c73-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.246 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.246 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.246 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.246 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.247 DEBUG:boto:Method: GET 2024-04-19T23:33:59.247 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.247 DEBUG:boto:Data: 2024-04-19T23:33:59.247 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.247 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.247 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.247 DEBUG:boto:Params: {} 2024-04-19T23:33:59.247 DEBUG:boto:Token: None 2024-04-19T23:33:59.247 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.248 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LndQcIPNlI7blTB7iX6un2aCK9M= 2024-04-19T23:33:59.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LndQcIPNlI7blTB7iX6un2aCK9M=', 'Content-Length': '0'} 2024-04-19T23:33:59.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000302c902d3aa968b7-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.290 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.290 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.291 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.291 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.291 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.291 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.291 DEBUG:boto:Data: 2024-04-19T23:33:59.291 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.291 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.291 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.291 DEBUG:boto:Params: {} 2024-04-19T23:33:59.292 DEBUG:boto:Token: None 2024-04-19T23:33:59.292 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.292 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dBlduUJ+kWkqgWTeg/qnqH9oXEw= 2024-04-19T23:33:59.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dBlduUJ+kWkqgWTeg/qnqH9oXEw=', 'Content-Length': '0'} 2024-04-19T23:33:59.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be45b78ead180cb2-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.294 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.294 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.294 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.294 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.294 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.295 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.295 DEBUG:boto:Data: 2024-04-19T23:33:59.295 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.295 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.295 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.295 DEBUG:boto:Params: {} 2024-04-19T23:33:59.295 DEBUG:boto:Token: None 2024-04-19T23:33:59.295 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.295 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dBlduUJ+kWkqgWTeg/qnqH9oXEw= 2024-04-19T23:33:59.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dBlduUJ+kWkqgWTeg/qnqH9oXEw=', 'Content-Length': '0'} 2024-04-19T23:33:59.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a969cd701cc1a8a-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:59.298 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.299 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.299 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.299 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.299 DEBUG:boto:Method: GET 2024-04-19T23:33:59.299 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.299 DEBUG:boto:Data: 2024-04-19T23:33:59.299 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.299 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.299 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.299 DEBUG:boto:Params: {} 2024-04-19T23:33:59.300 DEBUG:boto:Token: None 2024-04-19T23:33:59.300 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.300 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LndQcIPNlI7blTB7iX6un2aCK9M= 2024-04-19T23:33:59.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LndQcIPNlI7blTB7iX6un2aCK9M=', 'Content-Length': '0'} 2024-04-19T23:33:59.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e873c7b970af68b1-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.303 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.303 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.303 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.303 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.303 DEBUG:boto:Method: GET 2024-04-19T23:33:59.303 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.303 DEBUG:boto:Data: 2024-04-19T23:33:59.303 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.303 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.303 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.303 DEBUG:boto:Params: {} 2024-04-19T23:33:59.304 DEBUG:boto:Token: None 2024-04-19T23:33:59.304 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.304 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LndQcIPNlI7blTB7iX6un2aCK9M= 2024-04-19T23:33:59.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LndQcIPNlI7blTB7iX6un2aCK9M=', 'Content-Length': '0'} 2024-04-19T23:33:59.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087f4eff4a399d12c-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.350 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.350 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.351 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:33:59.351 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:33:59.351 DEBUG:boto:Method: GET 2024-04-19T23:33:59.351 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:33:59.351 DEBUG:boto:Data: 2024-04-19T23:33:59.351 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.351 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.351 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.351 DEBUG:boto:Params: {} 2024-04-19T23:33:59.351 DEBUG:boto:Token: None 2024-04-19T23:33:59.352 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.352 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc710f44d3828e05-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.356 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.356 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.356 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.356 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:33:59.356 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:33:59.356 DEBUG:boto:Method: GET 2024-04-19T23:33:59.357 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:33:59.357 DEBUG:boto:Data: 2024-04-19T23:33:59.357 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.357 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.357 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.357 DEBUG:boto:Params: {} 2024-04-19T23:33:59.357 DEBUG:boto:Token: None 2024-04-19T23:33:59.357 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.357 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045a9402a3ec35830-006622ff67-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.366 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.366 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.367 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.367 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.367 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:33:59.367 DEBUG:boto:Data: 2024-04-19T23:33:59.367 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.368 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.368 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.368 DEBUG:boto:Params: {} 2024-04-19T23:33:59.368 DEBUG:boto:Token: None 2024-04-19T23:33:59.368 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:33:59.369 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1ldW2/iaSuaSuBjCZAG+o7lYboo= 2024-04-19T23:33:59.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1ldW2/iaSuaSuBjCZAG+o7lYboo=', 'Content-Length': '0'} 2024-04-19T23:33:59.371 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2165a8a2ace31df-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.371 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.372 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.372 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.372 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:33:59.372 DEBUG:boto:Data: 2024-04-19T23:33:59.373 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.373 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.373 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.373 DEBUG:boto:Params: {} 2024-04-19T23:33:59.373 DEBUG:boto:Token: None 2024-04-19T23:33:59.373 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:33:59.373 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1ldW2/iaSuaSuBjCZAG+o7lYboo= 2024-04-19T23:33:59.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1ldW2/iaSuaSuBjCZAG+o7lYboo=', 'Content-Length': '0'} 2024-04-19T23:33:59.382 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3571e3debab3d11-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:33:59.383 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.383 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.383 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.383 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.383 DEBUG:boto:Method: GET 2024-04-19T23:33:59.383 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.383 DEBUG:boto:Data: 2024-04-19T23:33:59.383 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.383 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.383 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.384 DEBUG:boto:Params: {} 2024-04-19T23:33:59.384 DEBUG:boto:Token: None 2024-04-19T23:33:59.384 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.384 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LndQcIPNlI7blTB7iX6un2aCK9M= 2024-04-19T23:33:59.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LndQcIPNlI7blTB7iX6un2aCK9M=', 'Content-Length': '0'} 2024-04-19T23:33:59.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000078fbf083e4e99b7-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.430 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.431 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:33:59.431 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.431 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.431 DEBUG:boto:Method: GET 2024-04-19T23:33:59.431 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.431 DEBUG:boto:Data: 2024-04-19T23:33:59.431 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.431 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.431 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.431 DEBUG:boto:Params: {} 2024-04-19T23:33:59.431 DEBUG:boto:Token: None 2024-04-19T23:33:59.432 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:33:59.432 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LndQcIPNlI7blTB7iX6un2aCK9M= 2024-04-19T23:33:59.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LndQcIPNlI7blTB7iX6un2aCK9M=', 'Content-Length': '0'} 2024-04-19T23:33:59.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000574cbf60821f1389-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.474 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.475 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.475 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:33:59.475 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:33:59.475 DEBUG:boto:Method: GET 2024-04-19T23:33:59.475 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:33:59.475 DEBUG:boto:Data: 2024-04-19T23:33:59.475 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.475 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.475 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.475 DEBUG:boto:Params: {} 2024-04-19T23:33:59.476 DEBUG:boto:Token: None 2024-04-19T23:33:59.476 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.476 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070da4104ce5e1672-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.479 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.479 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.479 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.479 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:33:59.479 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:33:59.480 DEBUG:boto:Method: GET 2024-04-19T23:33:59.480 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:33:59.480 DEBUG:boto:Data: 2024-04-19T23:33:59.480 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.480 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.480 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.480 DEBUG:boto:Params: {} 2024-04-19T23:33:59.480 DEBUG:boto:Token: None 2024-04-19T23:33:59.481 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.481 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045d7d6c7a34d7a67-006622ff67-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.484 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:59.484 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.484 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.484 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.484 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.484 DEBUG:boto:Method: GET 2024-04-19T23:33:59.485 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.485 DEBUG:boto:Data: 2024-04-19T23:33:59.485 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.485 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.485 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.485 DEBUG:boto:Params: {} 2024-04-19T23:33:59.485 DEBUG:boto:Token: None 2024-04-19T23:33:59.485 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.485 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h+MX1UjttnMNpVW6RrldbEwi4VE= 2024-04-19T23:33:59.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h+MX1UjttnMNpVW6RrldbEwi4VE=', 'Content-Length': '0'} 2024-04-19T23:33:59.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ba79cbaa3a70b03-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.488 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.488 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.488 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.488 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.488 DEBUG:boto:Method: GET 2024-04-19T23:33:59.488 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.488 DEBUG:boto:Data: 2024-04-19T23:33:59.488 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.488 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.489 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.489 DEBUG:boto:Params: {} 2024-04-19T23:33:59.489 DEBUG:boto:Token: None 2024-04-19T23:33:59.489 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.489 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h+MX1UjttnMNpVW6RrldbEwi4VE= 2024-04-19T23:33:59.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h+MX1UjttnMNpVW6RrldbEwi4VE=', 'Content-Length': '0'} 2024-04-19T23:33:59.491 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032c9360624cf671f-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.492 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.492 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.492 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.492 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.492 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.492 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.492 DEBUG:boto:Data: 2024-04-19T23:33:59.492 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.492 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.492 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.492 DEBUG:boto:Params: {} 2024-04-19T23:33:59.493 DEBUG:boto:Token: None 2024-04-19T23:33:59.493 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.493 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fIedGTWViLyXBHLN+NtqyTVDWRg= 2024-04-19T23:33:59.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fIedGTWViLyXBHLN+NtqyTVDWRg=', 'Content-Length': '0'} 2024-04-19T23:33:59.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012e59779f2e18c4c-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.496 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.496 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.496 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.496 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.496 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.497 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.497 DEBUG:boto:Data: 2024-04-19T23:33:59.497 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.497 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.497 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.497 DEBUG:boto:Params: {} 2024-04-19T23:33:59.498 DEBUG:boto:Token: None 2024-04-19T23:33:59.498 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.498 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fIedGTWViLyXBHLN+NtqyTVDWRg= 2024-04-19T23:33:59.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fIedGTWViLyXBHLN+NtqyTVDWRg=', 'Content-Length': '0'} 2024-04-19T23:33:59.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007000eace586d4652-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:59.501 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.502 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.502 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.502 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.502 DEBUG:boto:Method: GET 2024-04-19T23:33:59.502 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.503 DEBUG:boto:Data: 2024-04-19T23:33:59.503 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.503 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.503 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.503 DEBUG:boto:Params: {} 2024-04-19T23:33:59.504 DEBUG:boto:Token: None 2024-04-19T23:33:59.504 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.504 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h+MX1UjttnMNpVW6RrldbEwi4VE= 2024-04-19T23:33:59.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h+MX1UjttnMNpVW6RrldbEwi4VE=', 'Content-Length': '0'} 2024-04-19T23:33:59.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e5c6cfcf27d4883-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.554 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.554 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.555 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.555 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.555 DEBUG:boto:Method: GET 2024-04-19T23:33:59.555 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.555 DEBUG:boto:Data: 2024-04-19T23:33:59.555 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.555 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.555 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.555 DEBUG:boto:Params: {} 2024-04-19T23:33:59.555 DEBUG:boto:Token: None 2024-04-19T23:33:59.556 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.556 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h+MX1UjttnMNpVW6RrldbEwi4VE= 2024-04-19T23:33:59.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h+MX1UjttnMNpVW6RrldbEwi4VE=', 'Content-Length': '0'} 2024-04-19T23:33:59.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9a8b2f09e582a5b-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.602 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.603 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.603 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:33:59.603 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:33:59.603 DEBUG:boto:Method: GET 2024-04-19T23:33:59.603 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:33:59.603 DEBUG:boto:Data: 2024-04-19T23:33:59.603 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.603 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.603 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.603 DEBUG:boto:Params: {} 2024-04-19T23:33:59.603 DEBUG:boto:Token: None 2024-04-19T23:33:59.604 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.604 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2962078f207dde-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.607 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.607 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.608 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.608 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:33:59.608 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:33:59.608 DEBUG:boto:Method: GET 2024-04-19T23:33:59.608 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:33:59.608 DEBUG:boto:Data: 2024-04-19T23:33:59.608 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.608 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.608 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.608 DEBUG:boto:Params: {} 2024-04-19T23:33:59.609 DEBUG:boto:Token: None 2024-04-19T23:33:59.609 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.609 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000614ad63eeb45959a-006622ff67-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.612 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.612 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.612 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.612 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.612 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:33:59.612 DEBUG:boto:Data: 2024-04-19T23:33:59.612 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.612 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.613 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.613 DEBUG:boto:Params: {} 2024-04-19T23:33:59.613 DEBUG:boto:Token: None 2024-04-19T23:33:59.613 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:33:59.613 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6WFVcmOTH1z949XZwVaTJJV3Eao= 2024-04-19T23:33:59.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6WFVcmOTH1z949XZwVaTJJV3Eao=', 'Content-Length': '0'} 2024-04-19T23:33:59.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076511573316c0f7e-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.616 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.616 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.616 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.616 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:33:59.616 DEBUG:boto:Data: 2024-04-19T23:33:59.616 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.616 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.616 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.616 DEBUG:boto:Params: {} 2024-04-19T23:33:59.617 DEBUG:boto:Token: None 2024-04-19T23:33:59.617 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:33:59.617 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6WFVcmOTH1z949XZwVaTJJV3Eao= 2024-04-19T23:33:59.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6WFVcmOTH1z949XZwVaTJJV3Eao=', 'Content-Length': '0'} 2024-04-19T23:33:59.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b22e21351c5c208-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:33:59.620 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.620 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.620 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.620 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.620 DEBUG:boto:Method: GET 2024-04-19T23:33:59.620 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.620 DEBUG:boto:Data: 2024-04-19T23:33:59.620 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.621 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.621 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.621 DEBUG:boto:Params: {} 2024-04-19T23:33:59.621 DEBUG:boto:Token: None 2024-04-19T23:33:59.621 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.621 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h+MX1UjttnMNpVW6RrldbEwi4VE= 2024-04-19T23:33:59.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h+MX1UjttnMNpVW6RrldbEwi4VE=', 'Content-Length': '0'} 2024-04-19T23:33:59.623 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9e1f2913a7412f3-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.666 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.667 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:33:59.667 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.667 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.667 DEBUG:boto:Method: GET 2024-04-19T23:33:59.667 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.667 DEBUG:boto:Data: 2024-04-19T23:33:59.667 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.667 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.667 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.667 DEBUG:boto:Params: {} 2024-04-19T23:33:59.668 DEBUG:boto:Token: None 2024-04-19T23:33:59.668 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:33:59.668 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:h+MX1UjttnMNpVW6RrldbEwi4VE= 2024-04-19T23:33:59.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:h+MX1UjttnMNpVW6RrldbEwi4VE=', 'Content-Length': '0'} 2024-04-19T23:33:59.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005b36866f4492a45-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.714 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.715 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.715 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:33:59.715 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:33:59.715 DEBUG:boto:Method: GET 2024-04-19T23:33:59.715 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:33:59.715 DEBUG:boto:Data: 2024-04-19T23:33:59.715 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.715 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:33:59.715 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.715 DEBUG:boto:Params: {} 2024-04-19T23:33:59.716 DEBUG:boto:Token: None 2024-04-19T23:33:59.716 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.716 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eaeea6900843088-006622ff67-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.766 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.767 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.767 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.767 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:33:59.767 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:33:59.767 DEBUG:boto:Method: GET 2024-04-19T23:33:59.767 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:33:59.767 DEBUG:boto:Data: 2024-04-19T23:33:59.767 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.768 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.768 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.768 DEBUG:boto:Params: {} 2024-04-19T23:33:59.768 DEBUG:boto:Token: None 2024-04-19T23:33:59.768 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.768 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009001c47e8811e752-006622ff67-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.814 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.815 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-21 zones={test-zone2, test-zone1} 2024-04-19T23:33:59.815 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-21 zones={test-zone3, test-zone1} 2024-04-19T23:33:59.815 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.815 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.815 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.815 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:33:59.816 DEBUG:boto:Data: 2024-04-19T23:33:59.816 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.816 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:59.816 DEBUG:boto:Port: 8001 2024-04-19T23:33:59.816 DEBUG:boto:Params: {} 2024-04-19T23:33:59.816 DEBUG:boto:Token: None 2024-04-19T23:33:59.816 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.816 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UtWTIqZcyDJNzfpj4ogR7zEnwdU= 2024-04-19T23:33:59.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UtWTIqZcyDJNzfpj4ogR7zEnwdU=', 'Content-Length': '0'} 2024-04-19T23:33:59.819 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000581995328ebf590f-006622ff67-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.819 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.819 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.819 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.819 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:33:59.820 DEBUG:boto:Data: 2024-04-19T23:33:59.820 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.820 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.820 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.820 DEBUG:boto:Params: {} 2024-04-19T23:33:59.821 DEBUG:boto:Token: None 2024-04-19T23:33:59.821 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.821 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UtWTIqZcyDJNzfpj4ogR7zEnwdU= 2024-04-19T23:33:59.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UtWTIqZcyDJNzfpj4ogR7zEnwdU=', 'Content-Length': '0'} 2024-04-19T23:33:59.823 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4fa41fe201f785e-006622ff67-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.824 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:33:59.824 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.824 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.824 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:59.824 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:59.824 DEBUG:boto:Method: GET 2024-04-19T23:33:59.824 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:33:59.824 DEBUG:boto:Data: 2024-04-19T23:33:59.824 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.824 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:59.824 DEBUG:boto:Port: 8001 2024-04-19T23:33:59.825 DEBUG:boto:Params: {} 2024-04-19T23:33:59.825 DEBUG:boto:Token: None 2024-04-19T23:33:59.825 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/?versions 2024-04-19T23:33:59.825 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:A2J35NV21CY+QHYBK4f9g78z7rc= 2024-04-19T23:33:59.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:A2J35NV21CY+QHYBK4f9g78z7rc=', 'Content-Length': '0'} 2024-04-19T23:33:59.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d17658019b419c46-006622ff67-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.828 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.828 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:59.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:59.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:59.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:59.829 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:33:59.829 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.829 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.829 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:59.829 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:59.829 DEBUG:boto:Method: GET 2024-04-19T23:33:59.829 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:33:59.830 DEBUG:boto:Data: 2024-04-19T23:33:59.830 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.830 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.830 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.830 DEBUG:boto:Params: {} 2024-04-19T23:33:59.830 DEBUG:boto:Token: None 2024-04-19T23:33:59.830 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/?versions 2024-04-19T23:33:59.831 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:A2J35NV21CY+QHYBK4f9g78z7rc= 2024-04-19T23:33:59.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:A2J35NV21CY+QHYBK4f9g78z7rc=', 'Content-Length': '0'} 2024-04-19T23:33:59.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036d29cbf1f17c7bd-006622ff67-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.834 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.834 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:33:59.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:33:59.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:33:59.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:33:59.835 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.835 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.835 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:59.835 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:59.835 DEBUG:boto:Method: GET 2024-04-19T23:33:59.835 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:33:59.835 DEBUG:boto:Data: 2024-04-19T23:33:59.835 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.835 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:59.835 DEBUG:boto:Port: 8001 2024-04-19T23:33:59.836 DEBUG:boto:Params: {} 2024-04-19T23:33:59.836 DEBUG:boto:Token: None 2024-04-19T23:33:59.836 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/?versions 2024-04-19T23:33:59.836 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:A2J35NV21CY+QHYBK4f9g78z7rc= 2024-04-19T23:33:59.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:A2J35NV21CY+QHYBK4f9g78z7rc=', 'Content-Length': '0'} 2024-04-19T23:33:59.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3cf3c8f05e773f6-006622ff67-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.839 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.840 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.840 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.840 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:59.840 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:33:59.840 DEBUG:boto:Method: GET 2024-04-19T23:33:59.840 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:33:59.840 DEBUG:boto:Data: 2024-04-19T23:33:59.840 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.840 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.840 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.841 DEBUG:boto:Params: {} 2024-04-19T23:33:59.841 DEBUG:boto:Token: None 2024-04-19T23:33:59.841 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/?versions 2024-04-19T23:33:59.841 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:A2J35NV21CY+QHYBK4f9g78z7rc= 2024-04-19T23:33:59.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:A2J35NV21CY+QHYBK4f9g78z7rc=', 'Content-Length': '0'} 2024-04-19T23:33:59.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd372fbdaf657a52-006622ff67-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.845 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:59.845 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:59.846 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:59.846 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.846 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.846 DEBUG:boto:Method: GET 2024-04-19T23:33:59.846 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.846 DEBUG:boto:Data: 2024-04-19T23:33:59.846 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.846 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:59.846 DEBUG:boto:Port: 8001 2024-04-19T23:33:59.847 DEBUG:boto:Params: {} 2024-04-19T23:33:59.847 DEBUG:boto:Token: None 2024-04-19T23:33:59.847 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.847 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9cJJqt24DAnUewC27KfvkmRoUG0= 2024-04-19T23:33:59.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9cJJqt24DAnUewC27KfvkmRoUG0=', 'Content-Length': '0'} 2024-04-19T23:33:59.849 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e9f1c2efd6a6a9d4-006622ff67-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.849 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:59.849 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:59.849 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.850 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.850 DEBUG:boto:Method: GET 2024-04-19T23:33:59.850 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.850 DEBUG:boto:Data: 2024-04-19T23:33:59.850 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.850 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.850 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.850 DEBUG:boto:Params: {} 2024-04-19T23:33:59.850 DEBUG:boto:Token: None 2024-04-19T23:33:59.851 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.851 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9cJJqt24DAnUewC27KfvkmRoUG0= 2024-04-19T23:33:59.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9cJJqt24DAnUewC27KfvkmRoUG0=', 'Content-Length': '0'} 2024-04-19T23:33:59.853 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004efa3acf606e8616-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.853 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:59.854 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:59.854 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.854 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.854 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.854 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.854 DEBUG:boto:Data: 2024-04-19T23:33:59.855 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.855 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:59.858 DEBUG:boto:Port: 8001 2024-04-19T23:33:59.858 DEBUG:boto:Params: {} 2024-04-19T23:33:59.858 DEBUG:boto:Token: None 2024-04-19T23:33:59.858 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.858 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iIAj0Mb1VLtdjFKqnJ7hF8ve5do= 2024-04-19T23:33:59.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iIAj0Mb1VLtdjFKqnJ7hF8ve5do=', 'Content-Length': '0'} 2024-04-19T23:33:59.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000055484f7ba7948a45-006622ff67-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.861 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:59.861 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:59.861 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.861 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.861 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.862 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.862 DEBUG:boto:Data: 2024-04-19T23:33:59.862 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.862 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.862 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.862 DEBUG:boto:Params: {} 2024-04-19T23:33:59.862 DEBUG:boto:Token: None 2024-04-19T23:33:59.862 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.863 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iIAj0Mb1VLtdjFKqnJ7hF8ve5do= 2024-04-19T23:33:59.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iIAj0Mb1VLtdjFKqnJ7hF8ve5do=', 'Content-Length': '0'} 2024-04-19T23:33:59.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007656c95a9e214549-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:59.865 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:59.865 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:59.865 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.865 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.865 DEBUG:boto:Method: GET 2024-04-19T23:33:59.866 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.866 DEBUG:boto:Data: 2024-04-19T23:33:59.866 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.866 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:59.866 DEBUG:boto:Port: 8001 2024-04-19T23:33:59.866 DEBUG:boto:Params: {} 2024-04-19T23:33:59.866 DEBUG:boto:Token: None 2024-04-19T23:33:59.866 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.866 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9cJJqt24DAnUewC27KfvkmRoUG0= 2024-04-19T23:33:59.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9cJJqt24DAnUewC27KfvkmRoUG0=', 'Content-Length': '0'} 2024-04-19T23:33:59.868 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000076b365071c8f8b05-006622ff67-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.868 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:59.868 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:59.869 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.869 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.869 DEBUG:boto:Method: GET 2024-04-19T23:33:59.869 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.869 DEBUG:boto:Data: 2024-04-19T23:33:59.869 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.869 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.869 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.869 DEBUG:boto:Params: {} 2024-04-19T23:33:59.869 DEBUG:boto:Token: None 2024-04-19T23:33:59.869 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.870 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9cJJqt24DAnUewC27KfvkmRoUG0= 2024-04-19T23:33:59.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9cJJqt24DAnUewC27KfvkmRoUG0=', 'Content-Length': '0'} 2024-04-19T23:33:59.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000090a19a000086e3e5-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.872 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.872 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.872 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:59.872 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:59.872 DEBUG:boto:Method: GET 2024-04-19T23:33:59.872 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:33:59.872 DEBUG:boto:Data: 2024-04-19T23:33:59.872 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.872 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:59.872 DEBUG:boto:Port: 8001 2024-04-19T23:33:59.872 DEBUG:boto:Params: {} 2024-04-19T23:33:59.872 DEBUG:boto:Token: None 2024-04-19T23:33:59.873 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.873 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9147db99c45a2dd-006622ff67-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.918 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.919 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.919 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.919 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:59.919 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:59.919 DEBUG:boto:Method: GET 2024-04-19T23:33:59.919 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:33:59.919 DEBUG:boto:Data: 2024-04-19T23:33:59.920 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.920 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.920 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.920 DEBUG:boto:Params: {} 2024-04-19T23:33:59.920 DEBUG:boto:Token: None 2024-04-19T23:33:59.920 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.920 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000704310e045077dca-006622ff67-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.970 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.971 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:59.971 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:59.971 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.971 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:33:59.971 DEBUG:boto:Data: 2024-04-19T23:33:59.971 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.971 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:59.971 DEBUG:boto:Port: 8001 2024-04-19T23:33:59.971 DEBUG:boto:Params: {} 2024-04-19T23:33:59.972 DEBUG:boto:Token: None 2024-04-19T23:33:59.972 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/dummy 2024-04-19T23:33:59.972 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HVyhQdm9JBcXIUORrmNmGFzA0S8= 2024-04-19T23:33:59.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HVyhQdm9JBcXIUORrmNmGFzA0S8=', 'Content-Length': '0'} 2024-04-19T23:33:59.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d10e4989bdc646d1-006622ff67-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.974 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:59.975 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:59.975 DEBUG:boto:Method: HEAD 2024-04-19T23:33:59.975 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:33:59.975 DEBUG:boto:Data: 2024-04-19T23:33:59.975 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.975 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.975 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.975 DEBUG:boto:Params: {} 2024-04-19T23:33:59.975 DEBUG:boto:Token: None 2024-04-19T23:33:59.975 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/dummy 2024-04-19T23:33:59.976 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HVyhQdm9JBcXIUORrmNmGFzA0S8= 2024-04-19T23:33:59.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HVyhQdm9JBcXIUORrmNmGFzA0S8=', 'Content-Length': '0'} 2024-04-19T23:33:59.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003f4e92558198d172-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:33:59.978 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:59.978 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:59.978 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.979 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.979 DEBUG:boto:Method: GET 2024-04-19T23:33:59.979 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.979 DEBUG:boto:Data: 2024-04-19T23:33:59.979 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.979 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:59.979 DEBUG:boto:Port: 8001 2024-04-19T23:33:59.979 DEBUG:boto:Params: {} 2024-04-19T23:33:59.979 DEBUG:boto:Token: None 2024-04-19T23:33:59.979 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.980 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9cJJqt24DAnUewC27KfvkmRoUG0= 2024-04-19T23:33:59.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9cJJqt24DAnUewC27KfvkmRoUG0=', 'Content-Length': '0'} 2024-04-19T23:33:59.981 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006273044a5072c89d-006622ff67-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.982 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:33:59.982 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:33:59.982 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.982 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.982 DEBUG:boto:Method: GET 2024-04-19T23:33:59.982 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.982 DEBUG:boto:Data: 2024-04-19T23:33:59.982 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.982 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.982 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.982 DEBUG:boto:Params: {} 2024-04-19T23:33:59.983 DEBUG:boto:Token: None 2024-04-19T23:33:59.983 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:33:59.983 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9cJJqt24DAnUewC27KfvkmRoUG0= 2024-04-19T23:33:59.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9cJJqt24DAnUewC27KfvkmRoUG0=', 'Content-Length': '0'} 2024-04-19T23:33:59.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000155b4c50e471d9f8-006622ff67-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.985 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.985 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.985 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:59.985 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:59.985 DEBUG:boto:Method: GET 2024-04-19T23:33:59.985 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:33:59.986 DEBUG:boto:Data: 2024-04-19T23:33:59.986 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.986 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:59.986 DEBUG:boto:Port: 8001 2024-04-19T23:33:59.986 DEBUG:boto:Params: {} 2024-04-19T23:33:59.986 DEBUG:boto:Token: None 2024-04-19T23:33:59.986 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.986 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a434fca8949df7d4-006622ff67-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.989 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.989 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:33:59.989 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:33:59.990 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:59.990 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:33:59.990 DEBUG:boto:Method: GET 2024-04-19T23:33:59.990 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:33:59.990 DEBUG:boto:Data: 2024-04-19T23:33:59.990 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.990 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.990 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.990 DEBUG:boto:Params: {} 2024-04-19T23:33:59.990 DEBUG:boto:Token: None 2024-04-19T23:33:59.991 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/ 2024-04-19T23:33:59.991 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0= 2024-04-19T23:33:59.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rjNZJIddXspaDm5Zy+UuHaCx8T0=', 'Content-Length': '0'} 2024-04-19T23:33:59.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e9637d636004f1f-006622ff67-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:33:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.993 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:33:59.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:33:59.994 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.994 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.994 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.994 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.994 DEBUG:boto:Method: GET 2024-04-19T23:33:59.994 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.994 DEBUG:boto:Data: 2024-04-19T23:33:59.994 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.995 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:33:59.995 DEBUG:boto:Port: 8001 2024-04-19T23:33:59.995 DEBUG:boto:Params: {} 2024-04-19T23:33:59.995 DEBUG:boto:Token: None 2024-04-19T23:33:59.995 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.995 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE= 2024-04-19T23:33:59.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE=', 'Content-Length': '0'} 2024-04-19T23:33:59.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053739496c47394d8-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:33:59.997 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.998 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:33:59.998 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.998 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.998 DEBUG:boto:Method: GET 2024-04-19T23:33:59.998 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.998 DEBUG:boto:Data: 2024-04-19T23:33:59.998 DEBUG:boto:Headers: {} 2024-04-19T23:33:59.998 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:33:59.998 DEBUG:boto:Port: 8000 2024-04-19T23:33:59.999 DEBUG:boto:Params: {} 2024-04-19T23:33:59.999 DEBUG:boto:Token: None 2024-04-19T23:33:59.999 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:33:59 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:33:59.999 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE= 2024-04-19T23:33:59.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:33:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40brDulKIYL6mmE+rK5gEkystTE=', 'Content-Length': '0'} 2024-04-19T23:34:00.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb12cdac32b28d09-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.001 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.001 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.001 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.001 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.001 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.001 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.002 DEBUG:boto:Data: 2024-04-19T23:34:00.002 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.002 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.002 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.002 DEBUG:boto:Params: {} 2024-04-19T23:34:00.002 DEBUG:boto:Token: None 2024-04-19T23:34:00.002 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.002 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qOBSFda/rUCJteOt6bzSZHanGtQ= 2024-04-19T23:34:00.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qOBSFda/rUCJteOt6bzSZHanGtQ=', 'Content-Length': '0'} 2024-04-19T23:34:00.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f872af7a80f639e3-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.005 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.005 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.005 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.005 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.005 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.005 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.005 DEBUG:boto:Data: 2024-04-19T23:34:00.005 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.006 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.006 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.006 DEBUG:boto:Params: {} 2024-04-19T23:34:00.006 DEBUG:boto:Token: None 2024-04-19T23:34:00.006 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.006 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qOBSFda/rUCJteOt6bzSZHanGtQ= 2024-04-19T23:34:00.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qOBSFda/rUCJteOt6bzSZHanGtQ=', 'Content-Length': '0'} 2024-04-19T23:34:00.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c202fe239d0347a-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:00.008 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.008 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.009 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.009 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.009 DEBUG:boto:Method: GET 2024-04-19T23:34:00.009 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.009 DEBUG:boto:Data: 2024-04-19T23:34:00.009 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.009 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.009 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.009 DEBUG:boto:Params: {} 2024-04-19T23:34:00.009 DEBUG:boto:Token: None 2024-04-19T23:34:00.010 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.010 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4= 2024-04-19T23:34:00.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4=', 'Content-Length': '0'} 2024-04-19T23:34:00.012 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cefa6862f7d03a69-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.058 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.059 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.059 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.059 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.059 DEBUG:boto:Method: GET 2024-04-19T23:34:00.059 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.059 DEBUG:boto:Data: 2024-04-19T23:34:00.059 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.059 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.059 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.059 DEBUG:boto:Params: {} 2024-04-19T23:34:00.060 DEBUG:boto:Token: None 2024-04-19T23:34:00.061 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.061 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4= 2024-04-19T23:34:00.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4=', 'Content-Length': '0'} 2024-04-19T23:34:00.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc792549191c8453-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.064 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.064 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.064 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.064 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.064 DEBUG:boto:Method: GET 2024-04-19T23:34:00.065 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.065 DEBUG:boto:Data: 2024-04-19T23:34:00.065 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.065 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.065 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.065 DEBUG:boto:Params: {} 2024-04-19T23:34:00.065 DEBUG:boto:Token: None 2024-04-19T23:34:00.065 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.065 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d7f28591869e377-006622ff68-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.068 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.068 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.069 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.069 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.069 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.069 DEBUG:boto:Method: GET 2024-04-19T23:34:00.069 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.069 DEBUG:boto:Data: 2024-04-19T23:34:00.069 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.069 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.069 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.069 DEBUG:boto:Params: {} 2024-04-19T23:34:00.069 DEBUG:boto:Token: None 2024-04-19T23:34:00.070 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.070 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fffe22f90b708e2-006622ff68-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.072 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.073 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.073 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.073 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.073 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:34:00.073 DEBUG:boto:Data: 2024-04-19T23:34:00.073 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.073 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.073 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.073 DEBUG:boto:Params: {} 2024-04-19T23:34:00.074 DEBUG:boto:Token: None 2024-04-19T23:34:00.074 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:34:00.074 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Y1Xw0avEV0Z1DT2ySG44lxOVIm0= 2024-04-19T23:34:00.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y1Xw0avEV0Z1DT2ySG44lxOVIm0=', 'Content-Length': '0'} 2024-04-19T23:34:00.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b1d2bb1a3cac2db-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.076 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.076 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.077 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.077 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:34:00.077 DEBUG:boto:Data: 2024-04-19T23:34:00.077 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.077 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.077 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.077 DEBUG:boto:Params: {} 2024-04-19T23:34:00.078 DEBUG:boto:Token: None 2024-04-19T23:34:00.078 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:34:00.078 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Y1Xw0avEV0Z1DT2ySG44lxOVIm0= 2024-04-19T23:34:00.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y1Xw0avEV0Z1DT2ySG44lxOVIm0=', 'Content-Length': '0'} 2024-04-19T23:34:00.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006840c6fec88f743e-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:00.082 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.082 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.082 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.082 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.082 DEBUG:boto:Method: GET 2024-04-19T23:34:00.082 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.082 DEBUG:boto:Data: 2024-04-19T23:34:00.082 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.082 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.082 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.082 DEBUG:boto:Params: {} 2024-04-19T23:34:00.083 DEBUG:boto:Token: None 2024-04-19T23:34:00.083 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.083 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4= 2024-04-19T23:34:00.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4=', 'Content-Length': '0'} 2024-04-19T23:34:00.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b25e4d3dd0e576e-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.126 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.126 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.127 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.127 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.127 DEBUG:boto:Method: GET 2024-04-19T23:34:00.127 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.127 DEBUG:boto:Data: 2024-04-19T23:34:00.127 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.127 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.127 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.127 DEBUG:boto:Params: {} 2024-04-19T23:34:00.127 DEBUG:boto:Token: None 2024-04-19T23:34:00.128 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.128 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4= 2024-04-19T23:34:00.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4=', 'Content-Length': '0'} 2024-04-19T23:34:00.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6521b29b7a626f0-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.130 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.130 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.131 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.131 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.131 DEBUG:boto:Method: GET 2024-04-19T23:34:00.131 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.131 DEBUG:boto:Data: 2024-04-19T23:34:00.131 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.131 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.131 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.131 DEBUG:boto:Params: {} 2024-04-19T23:34:00.131 DEBUG:boto:Token: None 2024-04-19T23:34:00.131 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.132 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071914d19ae76129a-006622ff68-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.178 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.179 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.179 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.179 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.179 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.179 DEBUG:boto:Method: GET 2024-04-19T23:34:00.179 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.179 DEBUG:boto:Data: 2024-04-19T23:34:00.179 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.180 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.180 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.180 DEBUG:boto:Params: {} 2024-04-19T23:34:00.180 DEBUG:boto:Token: None 2024-04-19T23:34:00.180 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088a0a42db2751556-006622ff68-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.183 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:00.184 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.184 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.184 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.184 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.184 DEBUG:boto:Method: GET 2024-04-19T23:34:00.184 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.184 DEBUG:boto:Data: 2024-04-19T23:34:00.184 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.184 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.184 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.184 DEBUG:boto:Params: {} 2024-04-19T23:34:00.185 DEBUG:boto:Token: None 2024-04-19T23:34:00.185 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.185 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5HY70X1bG1KBPQu/+dwwWjjWwQ8= 2024-04-19T23:34:00.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5HY70X1bG1KBPQu/+dwwWjjWwQ8=', 'Content-Length': '0'} 2024-04-19T23:34:00.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fb80f4cdb0a3c09-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.187 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.187 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.187 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.188 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.188 DEBUG:boto:Method: GET 2024-04-19T23:34:00.188 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.188 DEBUG:boto:Data: 2024-04-19T23:34:00.188 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.188 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.188 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.188 DEBUG:boto:Params: {} 2024-04-19T23:34:00.188 DEBUG:boto:Token: None 2024-04-19T23:34:00.188 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.189 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5HY70X1bG1KBPQu/+dwwWjjWwQ8= 2024-04-19T23:34:00.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5HY70X1bG1KBPQu/+dwwWjjWwQ8=', 'Content-Length': '0'} 2024-04-19T23:34:00.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009656c64cdc911f02-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.234 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.235 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.235 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.235 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.235 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.235 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.235 DEBUG:boto:Data: 2024-04-19T23:34:00.235 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.235 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.235 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.235 DEBUG:boto:Params: {} 2024-04-19T23:34:00.236 DEBUG:boto:Token: None 2024-04-19T23:34:00.236 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.236 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TQViT4CH6JuRIOZcmeI7yFVyNRA= 2024-04-19T23:34:00.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TQViT4CH6JuRIOZcmeI7yFVyNRA=', 'Content-Length': '0'} 2024-04-19T23:34:00.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016ce5a40afd5b2d5-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.238 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.238 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.238 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.238 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.238 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.239 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.239 DEBUG:boto:Data: 2024-04-19T23:34:00.239 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.239 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.239 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.239 DEBUG:boto:Params: {} 2024-04-19T23:34:00.239 DEBUG:boto:Token: None 2024-04-19T23:34:00.239 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.239 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TQViT4CH6JuRIOZcmeI7yFVyNRA= 2024-04-19T23:34:00.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TQViT4CH6JuRIOZcmeI7yFVyNRA=', 'Content-Length': '0'} 2024-04-19T23:34:00.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0961a663ae88564-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:00.242 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.242 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.242 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.242 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.242 DEBUG:boto:Method: GET 2024-04-19T23:34:00.242 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.242 DEBUG:boto:Data: 2024-04-19T23:34:00.243 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.243 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.243 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.243 DEBUG:boto:Params: {} 2024-04-19T23:34:00.244 DEBUG:boto:Token: None 2024-04-19T23:34:00.244 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.244 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5HY70X1bG1KBPQu/+dwwWjjWwQ8= 2024-04-19T23:34:00.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5HY70X1bG1KBPQu/+dwwWjjWwQ8=', 'Content-Length': '0'} 2024-04-19T23:34:00.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffa868c2144da819-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.286 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.286 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.287 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.287 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.287 DEBUG:boto:Method: GET 2024-04-19T23:34:00.287 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.287 DEBUG:boto:Data: 2024-04-19T23:34:00.287 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.287 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.287 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.287 DEBUG:boto:Params: {} 2024-04-19T23:34:00.287 DEBUG:boto:Token: None 2024-04-19T23:34:00.288 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.288 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5HY70X1bG1KBPQu/+dwwWjjWwQ8= 2024-04-19T23:34:00.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5HY70X1bG1KBPQu/+dwwWjjWwQ8=', 'Content-Length': '0'} 2024-04-19T23:34:00.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002d1a2c996e66f49-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.330 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.330 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.331 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:00.331 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:00.331 DEBUG:boto:Method: GET 2024-04-19T23:34:00.331 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:00.331 DEBUG:boto:Data: 2024-04-19T23:34:00.331 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.331 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.331 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.331 DEBUG:boto:Params: {} 2024-04-19T23:34:00.331 DEBUG:boto:Token: None 2024-04-19T23:34:00.331 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed22fea0f46dc2ea-006622ff68-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.334 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.335 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.335 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.335 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:00.335 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:00.335 DEBUG:boto:Method: GET 2024-04-19T23:34:00.335 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:00.335 DEBUG:boto:Data: 2024-04-19T23:34:00.335 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.336 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.336 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.336 DEBUG:boto:Params: {} 2024-04-19T23:34:00.336 DEBUG:boto:Token: None 2024-04-19T23:34:00.336 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.336 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a32e5eece5fd2a6e-006622ff68-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.339 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.340 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.340 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.340 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.340 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:34:00.340 DEBUG:boto:Data: 2024-04-19T23:34:00.340 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.340 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.340 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.340 DEBUG:boto:Params: {} 2024-04-19T23:34:00.340 DEBUG:boto:Token: None 2024-04-19T23:34:00.341 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:34:00.341 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xNpYnCu9e2ZszlFZATkKbxfgLXQ= 2024-04-19T23:34:00.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xNpYnCu9e2ZszlFZATkKbxfgLXQ=', 'Content-Length': '0'} 2024-04-19T23:34:00.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000380fac74b9c8759f-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.343 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.343 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.344 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.344 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:34:00.344 DEBUG:boto:Data: 2024-04-19T23:34:00.344 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.344 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.344 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.344 DEBUG:boto:Params: {} 2024-04-19T23:34:00.344 DEBUG:boto:Token: None 2024-04-19T23:34:00.344 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:34:00.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xNpYnCu9e2ZszlFZATkKbxfgLXQ= 2024-04-19T23:34:00.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xNpYnCu9e2ZszlFZATkKbxfgLXQ=', 'Content-Length': '0'} 2024-04-19T23:34:00.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000531dd517caac3cca-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:00.347 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.348 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.348 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.348 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.348 DEBUG:boto:Method: GET 2024-04-19T23:34:00.348 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.348 DEBUG:boto:Data: 2024-04-19T23:34:00.348 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.348 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.348 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.348 DEBUG:boto:Params: {} 2024-04-19T23:34:00.348 DEBUG:boto:Token: None 2024-04-19T23:34:00.349 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.349 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5HY70X1bG1KBPQu/+dwwWjjWwQ8= 2024-04-19T23:34:00.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5HY70X1bG1KBPQu/+dwwWjjWwQ8=', 'Content-Length': '0'} 2024-04-19T23:34:00.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000668f0213d3684593-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.394 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.394 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:00.395 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.395 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.395 DEBUG:boto:Method: GET 2024-04-19T23:34:00.395 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.395 DEBUG:boto:Data: 2024-04-19T23:34:00.395 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.395 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.395 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.395 DEBUG:boto:Params: {} 2024-04-19T23:34:00.395 DEBUG:boto:Token: None 2024-04-19T23:34:00.396 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:00.396 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5HY70X1bG1KBPQu/+dwwWjjWwQ8= 2024-04-19T23:34:00.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5HY70X1bG1KBPQu/+dwwWjjWwQ8=', 'Content-Length': '0'} 2024-04-19T23:34:00.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000367a3904ca82c59f-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.442 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.442 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.443 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:00.443 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:00.443 DEBUG:boto:Method: GET 2024-04-19T23:34:00.443 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:00.443 DEBUG:boto:Data: 2024-04-19T23:34:00.443 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.443 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.443 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.443 DEBUG:boto:Params: {} 2024-04-19T23:34:00.443 DEBUG:boto:Token: None 2024-04-19T23:34:00.444 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.444 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe5fc3e488f0a641-006622ff68-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.446 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.447 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.447 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.447 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:00.447 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:00.447 DEBUG:boto:Method: GET 2024-04-19T23:34:00.447 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:00.447 DEBUG:boto:Data: 2024-04-19T23:34:00.448 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.448 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.448 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.448 DEBUG:boto:Params: {} 2024-04-19T23:34:00.448 DEBUG:boto:Token: None 2024-04-19T23:34:00.448 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.448 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f18166ea0715477f-006622ff68-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.451 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:34:00.451 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.452 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.452 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.452 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.452 DEBUG:boto:Method: GET 2024-04-19T23:34:00.452 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.452 DEBUG:boto:Data: 2024-04-19T23:34:00.452 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.452 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.452 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.452 DEBUG:boto:Params: {} 2024-04-19T23:34:00.452 DEBUG:boto:Token: None 2024-04-19T23:34:00.453 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.453 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9jwIX97eNCasMhCKkRJrkrifsFw= 2024-04-19T23:34:00.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9jwIX97eNCasMhCKkRJrkrifsFw=', 'Content-Length': '0'} 2024-04-19T23:34:00.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7ffa62521ee5354-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.455 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.455 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.455 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.455 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.455 DEBUG:boto:Method: GET 2024-04-19T23:34:00.455 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.455 DEBUG:boto:Data: 2024-04-19T23:34:00.456 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.456 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.456 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.456 DEBUG:boto:Params: {} 2024-04-19T23:34:00.456 DEBUG:boto:Token: None 2024-04-19T23:34:00.456 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.456 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9jwIX97eNCasMhCKkRJrkrifsFw= 2024-04-19T23:34:00.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9jwIX97eNCasMhCKkRJrkrifsFw=', 'Content-Length': '0'} 2024-04-19T23:34:00.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000acde2527a56809-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.459 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.459 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.459 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.459 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.460 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.460 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.460 DEBUG:boto:Data: 2024-04-19T23:34:00.460 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.460 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.460 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.460 DEBUG:boto:Params: {} 2024-04-19T23:34:00.460 DEBUG:boto:Token: None 2024-04-19T23:34:00.460 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cHCWwHQujVxNVcP2Ey6XSDhz7XQ= 2024-04-19T23:34:00.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cHCWwHQujVxNVcP2Ey6XSDhz7XQ=', 'Content-Length': '0'} 2024-04-19T23:34:00.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fb609515165d56c-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.462 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.462 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.463 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.463 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.463 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.463 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.463 DEBUG:boto:Data: 2024-04-19T23:34:00.463 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.463 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.463 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.463 DEBUG:boto:Params: {} 2024-04-19T23:34:00.463 DEBUG:boto:Token: None 2024-04-19T23:34:00.464 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.464 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cHCWwHQujVxNVcP2Ey6XSDhz7XQ= 2024-04-19T23:34:00.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cHCWwHQujVxNVcP2Ey6XSDhz7XQ=', 'Content-Length': '0'} 2024-04-19T23:34:00.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf9e4c3364c33320-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:34:00.466 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.466 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.470 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.470 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.470 DEBUG:boto:Method: GET 2024-04-19T23:34:00.470 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.470 DEBUG:boto:Data: 2024-04-19T23:34:00.470 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.470 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.470 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.470 DEBUG:boto:Params: {} 2024-04-19T23:34:00.470 DEBUG:boto:Token: None 2024-04-19T23:34:00.471 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.471 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9jwIX97eNCasMhCKkRJrkrifsFw= 2024-04-19T23:34:00.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9jwIX97eNCasMhCKkRJrkrifsFw=', 'Content-Length': '0'} 2024-04-19T23:34:00.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5ca5cd5fa030f1b-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.473 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.473 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.473 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.473 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.473 DEBUG:boto:Method: GET 2024-04-19T23:34:00.473 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.473 DEBUG:boto:Data: 2024-04-19T23:34:00.473 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.473 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.473 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.474 DEBUG:boto:Params: {} 2024-04-19T23:34:00.474 DEBUG:boto:Token: None 2024-04-19T23:34:00.474 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.474 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9jwIX97eNCasMhCKkRJrkrifsFw= 2024-04-19T23:34:00.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9jwIX97eNCasMhCKkRJrkrifsFw=', 'Content-Length': '0'} 2024-04-19T23:34:00.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc11e92987817730-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.518 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.519 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.519 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:00.519 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:00.519 DEBUG:boto:Method: GET 2024-04-19T23:34:00.519 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:00.520 DEBUG:boto:Data: 2024-04-19T23:34:00.520 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.520 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.520 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.520 DEBUG:boto:Params: {} 2024-04-19T23:34:00.520 DEBUG:boto:Token: None 2024-04-19T23:34:00.520 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.520 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bef44da636811bd-006622ff68-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.523 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.523 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.523 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.524 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:00.524 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:00.524 DEBUG:boto:Method: GET 2024-04-19T23:34:00.524 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:00.524 DEBUG:boto:Data: 2024-04-19T23:34:00.524 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.524 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.524 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.524 DEBUG:boto:Params: {} 2024-04-19T23:34:00.524 DEBUG:boto:Token: None 2024-04-19T23:34:00.525 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.525 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003660188c6457b92f-006622ff68-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.527 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.528 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.528 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.528 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.528 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:34:00.528 DEBUG:boto:Data: 2024-04-19T23:34:00.528 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.528 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.528 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.528 DEBUG:boto:Params: {} 2024-04-19T23:34:00.529 DEBUG:boto:Token: None 2024-04-19T23:34:00.529 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:34:00.529 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brrucV8aQbdgWOp528ZDgZrnPZU= 2024-04-19T23:34:00.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brrucV8aQbdgWOp528ZDgZrnPZU=', 'Content-Length': '0'} 2024-04-19T23:34:00.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000440b3f646cb1cb4b-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.532 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.532 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.532 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.532 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:34:00.532 DEBUG:boto:Data: 2024-04-19T23:34:00.533 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.533 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.533 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.533 DEBUG:boto:Params: {} 2024-04-19T23:34:00.533 DEBUG:boto:Token: None 2024-04-19T23:34:00.533 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:34:00.533 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brrucV8aQbdgWOp528ZDgZrnPZU= 2024-04-19T23:34:00.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brrucV8aQbdgWOp528ZDgZrnPZU=', 'Content-Length': '0'} 2024-04-19T23:34:00.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5c204ebd29dfe70-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:34:00.536 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.536 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.536 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.536 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.536 DEBUG:boto:Method: GET 2024-04-19T23:34:00.536 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.536 DEBUG:boto:Data: 2024-04-19T23:34:00.537 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.537 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.537 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.537 DEBUG:boto:Params: {} 2024-04-19T23:34:00.537 DEBUG:boto:Token: None 2024-04-19T23:34:00.537 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.537 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9jwIX97eNCasMhCKkRJrkrifsFw= 2024-04-19T23:34:00.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9jwIX97eNCasMhCKkRJrkrifsFw=', 'Content-Length': '0'} 2024-04-19T23:34:00.539 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086ce6da199c7ab00-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.582 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.583 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:00.583 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.583 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.583 DEBUG:boto:Method: GET 2024-04-19T23:34:00.583 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.583 DEBUG:boto:Data: 2024-04-19T23:34:00.583 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.583 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.583 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.583 DEBUG:boto:Params: {} 2024-04-19T23:34:00.584 DEBUG:boto:Token: None 2024-04-19T23:34:00.584 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:00.584 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9jwIX97eNCasMhCKkRJrkrifsFw= 2024-04-19T23:34:00.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9jwIX97eNCasMhCKkRJrkrifsFw=', 'Content-Length': '0'} 2024-04-19T23:34:00.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c7894a46eec8a7f-006622ff68-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.626 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.627 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.627 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:00.627 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:00.627 DEBUG:boto:Method: GET 2024-04-19T23:34:00.627 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:00.627 DEBUG:boto:Data: 2024-04-19T23:34:00.627 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.627 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.627 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.627 DEBUG:boto:Params: {} 2024-04-19T23:34:00.627 DEBUG:boto:Token: None 2024-04-19T23:34:00.628 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099b8544cc26e2e7a-006622ff68-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.631 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.631 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.631 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.631 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:00.631 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:00.632 DEBUG:boto:Method: GET 2024-04-19T23:34:00.632 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:00.632 DEBUG:boto:Data: 2024-04-19T23:34:00.632 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.632 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:00.632 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.632 DEBUG:boto:Params: {} 2024-04-19T23:34:00.632 DEBUG:boto:Token: None 2024-04-19T23:34:00.632 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec4c9822800165fd-006622ff68-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.678 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.679 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-21 zones={test-zone3, test-zone1} 2024-04-19T23:34:00.679 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-21 zones={test-zone3, test-zone2} 2024-04-19T23:34:00.679 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.679 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.679 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.679 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:34:00.679 DEBUG:boto:Data: 2024-04-19T23:34:00.680 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.680 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.680 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.680 DEBUG:boto:Params: {} 2024-04-19T23:34:00.680 DEBUG:boto:Token: None 2024-04-19T23:34:00.680 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.680 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wCgO+I94eHGfqQqdbfPJE6W8UN4= 2024-04-19T23:34:00.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wCgO+I94eHGfqQqdbfPJE6W8UN4=', 'Content-Length': '0'} 2024-04-19T23:34:00.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb6d8e2c3d6fb266-006622ff68-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.683 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.683 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.683 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.683 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:34:00.683 DEBUG:boto:Data: 2024-04-19T23:34:00.683 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.683 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.684 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.684 DEBUG:boto:Params: {} 2024-04-19T23:34:00.684 DEBUG:boto:Token: None 2024-04-19T23:34:00.684 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.684 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wCgO+I94eHGfqQqdbfPJE6W8UN4= 2024-04-19T23:34:00.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wCgO+I94eHGfqQqdbfPJE6W8UN4=', 'Content-Length': '0'} 2024-04-19T23:34:00.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000081af02a20af341a-006622ff68-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.686 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:34:00.687 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.687 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.687 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:34:00.688 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:34:00.688 DEBUG:boto:Method: GET 2024-04-19T23:34:00.688 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:34:00.688 DEBUG:boto:Data: 2024-04-19T23:34:00.688 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.688 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.688 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.688 DEBUG:boto:Params: {} 2024-04-19T23:34:00.688 DEBUG:boto:Token: None 2024-04-19T23:34:00.689 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/?versions 2024-04-19T23:34:00.689 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lVffSPBCQo64dlrVVmWBiS3G1LM= 2024-04-19T23:34:00.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lVffSPBCQo64dlrVVmWBiS3G1LM=', 'Content-Length': '0'} 2024-04-19T23:34:00.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d24808cda57185d-006622ff68-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.691 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.692 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:34:00.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:34:00.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:34:00.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:34:00.692 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:34:00.692 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.692 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.692 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:34:00.693 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:34:00.693 DEBUG:boto:Method: GET 2024-04-19T23:34:00.693 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:34:00.693 DEBUG:boto:Data: 2024-04-19T23:34:00.693 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.693 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.693 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.693 DEBUG:boto:Params: {} 2024-04-19T23:34:00.693 DEBUG:boto:Token: None 2024-04-19T23:34:00.693 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/?versions 2024-04-19T23:34:00.694 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lVffSPBCQo64dlrVVmWBiS3G1LM= 2024-04-19T23:34:00.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lVffSPBCQo64dlrVVmWBiS3G1LM=', 'Content-Length': '0'} 2024-04-19T23:34:00.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4f1c1b6a405bcc1-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.696 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.697 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:34:00.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:34:00.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:34:00.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:34:00.697 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.697 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.698 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:34:00.698 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:34:00.698 DEBUG:boto:Method: GET 2024-04-19T23:34:00.698 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:34:00.698 DEBUG:boto:Data: 2024-04-19T23:34:00.698 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.698 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.698 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.698 DEBUG:boto:Params: {} 2024-04-19T23:34:00.698 DEBUG:boto:Token: None 2024-04-19T23:34:00.699 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/?versions 2024-04-19T23:34:00.699 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lVffSPBCQo64dlrVVmWBiS3G1LM= 2024-04-19T23:34:00.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lVffSPBCQo64dlrVVmWBiS3G1LM=', 'Content-Length': '0'} 2024-04-19T23:34:00.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089c3acd974059c39-006622ff68-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.702 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.703 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.703 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.703 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:34:00.703 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:34:00.703 DEBUG:boto:Method: GET 2024-04-19T23:34:00.703 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:34:00.703 DEBUG:boto:Data: 2024-04-19T23:34:00.703 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.704 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.704 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.704 DEBUG:boto:Params: {} 2024-04-19T23:34:00.704 DEBUG:boto:Token: None 2024-04-19T23:34:00.704 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/?versions 2024-04-19T23:34:00.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lVffSPBCQo64dlrVVmWBiS3G1LM= 2024-04-19T23:34:00.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lVffSPBCQo64dlrVVmWBiS3G1LM=', 'Content-Length': '0'} 2024-04-19T23:34:00.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000628a0f0d219dae13-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.707 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:34:00.707 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:34:00.707 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:34:00.707 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.707 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.708 DEBUG:boto:Method: GET 2024-04-19T23:34:00.708 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.708 DEBUG:boto:Data: 2024-04-19T23:34:00.708 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.708 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.708 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.708 DEBUG:boto:Params: {} 2024-04-19T23:34:00.708 DEBUG:boto:Token: None 2024-04-19T23:34:00.708 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.708 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wax3lnNbbb/LAaYg45kieQWVhk0= 2024-04-19T23:34:00.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wax3lnNbbb/LAaYg45kieQWVhk0=', 'Content-Length': '0'} 2024-04-19T23:34:00.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e5768437f9ee9db1-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.710 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:34:00.710 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:34:00.711 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.711 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.711 DEBUG:boto:Method: GET 2024-04-19T23:34:00.711 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.711 DEBUG:boto:Data: 2024-04-19T23:34:00.711 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.711 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.711 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.711 DEBUG:boto:Params: {} 2024-04-19T23:34:00.711 DEBUG:boto:Token: None 2024-04-19T23:34:00.711 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.712 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wax3lnNbbb/LAaYg45kieQWVhk0= 2024-04-19T23:34:00.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wax3lnNbbb/LAaYg45kieQWVhk0=', 'Content-Length': '0'} 2024-04-19T23:34:00.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7e18d569289eee6-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.714 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:34:00.714 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:34:00.714 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.714 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.714 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.714 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.714 DEBUG:boto:Data: 2024-04-19T23:34:00.715 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.715 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.715 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.715 DEBUG:boto:Params: {} 2024-04-19T23:34:00.715 DEBUG:boto:Token: None 2024-04-19T23:34:00.715 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.715 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:clDGdr2lpx+783xz2ku/Hp2VBuo= 2024-04-19T23:34:00.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:clDGdr2lpx+783xz2ku/Hp2VBuo=', 'Content-Length': '0'} 2024-04-19T23:34:00.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec77087fc3c5780a-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.717 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:34:00.717 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:34:00.717 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.717 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.718 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.718 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.718 DEBUG:boto:Data: 2024-04-19T23:34:00.718 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.718 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.718 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.718 DEBUG:boto:Params: {} 2024-04-19T23:34:00.718 DEBUG:boto:Token: None 2024-04-19T23:34:00.718 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.718 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:clDGdr2lpx+783xz2ku/Hp2VBuo= 2024-04-19T23:34:00.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:clDGdr2lpx+783xz2ku/Hp2VBuo=', 'Content-Length': '0'} 2024-04-19T23:34:00.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fcd5d2f000c39eda-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:34:00.721 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:34:00.721 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:34:00.721 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.721 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.721 DEBUG:boto:Method: GET 2024-04-19T23:34:00.722 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.722 DEBUG:boto:Data: 2024-04-19T23:34:00.722 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.722 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.722 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.722 DEBUG:boto:Params: {} 2024-04-19T23:34:00.723 DEBUG:boto:Token: None 2024-04-19T23:34:00.723 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.723 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wax3lnNbbb/LAaYg45kieQWVhk0= 2024-04-19T23:34:00.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wax3lnNbbb/LAaYg45kieQWVhk0=', 'Content-Length': '0'} 2024-04-19T23:34:00.729 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000990288e586cb6b54-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.730 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:34:00.730 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:34:00.730 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.730 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.730 DEBUG:boto:Method: GET 2024-04-19T23:34:00.730 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.730 DEBUG:boto:Data: 2024-04-19T23:34:00.730 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.730 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.730 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.730 DEBUG:boto:Params: {} 2024-04-19T23:34:00.731 DEBUG:boto:Token: None 2024-04-19T23:34:00.731 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.731 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wax3lnNbbb/LAaYg45kieQWVhk0= 2024-04-19T23:34:00.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wax3lnNbbb/LAaYg45kieQWVhk0=', 'Content-Length': '0'} 2024-04-19T23:34:00.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003bbb8e37bdfe84c1-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.733 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.733 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.733 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:34:00.733 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:34:00.733 DEBUG:boto:Method: GET 2024-04-19T23:34:00.734 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:34:00.734 DEBUG:boto:Data: 2024-04-19T23:34:00.734 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.734 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.734 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.734 DEBUG:boto:Params: {} 2024-04-19T23:34:00.734 DEBUG:boto:Token: None 2024-04-19T23:34:00.734 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.734 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026e30bdbecb3acc7-006622ff68-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.782 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.783 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.783 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.783 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:34:00.783 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:34:00.784 DEBUG:boto:Method: GET 2024-04-19T23:34:00.784 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:34:00.784 DEBUG:boto:Data: 2024-04-19T23:34:00.784 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.784 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.784 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.784 DEBUG:boto:Params: {} 2024-04-19T23:34:00.784 DEBUG:boto:Token: None 2024-04-19T23:34:00.784 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.785 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbe4c213df7323b3-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.787 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.788 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:34:00.788 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:34:00.788 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.788 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:34:00.788 DEBUG:boto:Data: 2024-04-19T23:34:00.788 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.789 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.789 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.789 DEBUG:boto:Params: {} 2024-04-19T23:34:00.789 DEBUG:boto:Token: None 2024-04-19T23:34:00.789 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/dummy 2024-04-19T23:34:00.789 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZwHIYnfFugRP0ZKOrZakaaWfe20= 2024-04-19T23:34:00.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZwHIYnfFugRP0ZKOrZakaaWfe20=', 'Content-Length': '0'} 2024-04-19T23:34:00.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce9d4c2a3774e68f-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.792 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:34:00.792 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:34:00.792 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.793 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:34:00.793 DEBUG:boto:Data: 2024-04-19T23:34:00.793 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.793 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.793 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.793 DEBUG:boto:Params: {} 2024-04-19T23:34:00.793 DEBUG:boto:Token: None 2024-04-19T23:34:00.793 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/dummy 2024-04-19T23:34:00.793 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZwHIYnfFugRP0ZKOrZakaaWfe20= 2024-04-19T23:34:00.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZwHIYnfFugRP0ZKOrZakaaWfe20=', 'Content-Length': '0'} 2024-04-19T23:34:00.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000002f2fb19b0bae68b-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:34:00.796 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:34:00.796 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:34:00.797 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.797 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.797 DEBUG:boto:Method: GET 2024-04-19T23:34:00.797 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.797 DEBUG:boto:Data: 2024-04-19T23:34:00.797 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.797 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.797 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.797 DEBUG:boto:Params: {} 2024-04-19T23:34:00.797 DEBUG:boto:Token: None 2024-04-19T23:34:00.798 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.798 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wax3lnNbbb/LAaYg45kieQWVhk0= 2024-04-19T23:34:00.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wax3lnNbbb/LAaYg45kieQWVhk0=', 'Content-Length': '0'} 2024-04-19T23:34:00.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000086f1a5192957da2f-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.799 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:34:00.800 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:34:00.800 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.800 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.800 DEBUG:boto:Method: GET 2024-04-19T23:34:00.800 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.800 DEBUG:boto:Data: 2024-04-19T23:34:00.800 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.800 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.800 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.800 DEBUG:boto:Params: {} 2024-04-19T23:34:00.800 DEBUG:boto:Token: None 2024-04-19T23:34:00.801 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:34:00.801 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wax3lnNbbb/LAaYg45kieQWVhk0= 2024-04-19T23:34:00.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wax3lnNbbb/LAaYg45kieQWVhk0=', 'Content-Length': '0'} 2024-04-19T23:34:00.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c4f6334a6b6d0be8-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.803 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.803 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.803 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:34:00.803 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:34:00.803 DEBUG:boto:Method: GET 2024-04-19T23:34:00.803 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:34:00.803 DEBUG:boto:Data: 2024-04-19T23:34:00.803 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.804 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.804 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.804 DEBUG:boto:Params: {} 2024-04-19T23:34:00.804 DEBUG:boto:Token: None 2024-04-19T23:34:00.804 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000916af487440e665f-006622ff68-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.806 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.807 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.807 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.807 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:34:00.807 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:34:00.807 DEBUG:boto:Method: GET 2024-04-19T23:34:00.807 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:34:00.807 DEBUG:boto:Data: 2024-04-19T23:34:00.807 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.808 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.808 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.808 DEBUG:boto:Params: {} 2024-04-19T23:34:00.808 DEBUG:boto:Token: None 2024-04-19T23:34:00.808 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.808 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067c9b1acb63a5410-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.858 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:00.859 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.859 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.859 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.859 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.859 DEBUG:boto:Method: GET 2024-04-19T23:34:00.860 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.860 DEBUG:boto:Data: 2024-04-19T23:34:00.860 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.860 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.860 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.860 DEBUG:boto:Params: {} 2024-04-19T23:34:00.860 DEBUG:boto:Token: None 2024-04-19T23:34:00.860 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.860 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4= 2024-04-19T23:34:00.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4=', 'Content-Length': '0'} 2024-04-19T23:34:00.862 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000daba2454fa69706a-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.863 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.863 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.863 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.863 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.863 DEBUG:boto:Method: GET 2024-04-19T23:34:00.863 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.863 DEBUG:boto:Data: 2024-04-19T23:34:00.863 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.863 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.863 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.864 DEBUG:boto:Params: {} 2024-04-19T23:34:00.864 DEBUG:boto:Token: None 2024-04-19T23:34:00.864 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.864 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4= 2024-04-19T23:34:00.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4=', 'Content-Length': '0'} 2024-04-19T23:34:00.866 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8d1a56f9445a1ad-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.866 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.866 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.867 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.867 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.867 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.867 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.867 DEBUG:boto:Data: 2024-04-19T23:34:00.867 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.867 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.867 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.867 DEBUG:boto:Params: {} 2024-04-19T23:34:00.867 DEBUG:boto:Token: None 2024-04-19T23:34:00.867 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.868 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qOBSFda/rUCJteOt6bzSZHanGtQ= 2024-04-19T23:34:00.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qOBSFda/rUCJteOt6bzSZHanGtQ=', 'Content-Length': '0'} 2024-04-19T23:34:00.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003112cad5cd3ad97e-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.870 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.870 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.870 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.870 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.870 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.870 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.870 DEBUG:boto:Data: 2024-04-19T23:34:00.870 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.870 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.871 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.871 DEBUG:boto:Params: {} 2024-04-19T23:34:00.871 DEBUG:boto:Token: None 2024-04-19T23:34:00.871 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.871 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qOBSFda/rUCJteOt6bzSZHanGtQ= 2024-04-19T23:34:00.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qOBSFda/rUCJteOt6bzSZHanGtQ=', 'Content-Length': '0'} 2024-04-19T23:34:00.873 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfe2df572fe179df-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:00.873 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.873 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.873 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.873 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.874 DEBUG:boto:Method: GET 2024-04-19T23:34:00.874 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.874 DEBUG:boto:Data: 2024-04-19T23:34:00.874 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.874 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.874 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.874 DEBUG:boto:Params: {} 2024-04-19T23:34:00.874 DEBUG:boto:Token: None 2024-04-19T23:34:00.874 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.874 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4= 2024-04-19T23:34:00.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4=', 'Content-Length': '0'} 2024-04-19T23:34:00.879 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb2c76babb9f84c5-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.918 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.919 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.919 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.919 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.919 DEBUG:boto:Method: GET 2024-04-19T23:34:00.919 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.919 DEBUG:boto:Data: 2024-04-19T23:34:00.919 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.919 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.919 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.919 DEBUG:boto:Params: {} 2024-04-19T23:34:00.920 DEBUG:boto:Token: None 2024-04-19T23:34:00.920 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.920 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4= 2024-04-19T23:34:00.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4=', 'Content-Length': '0'} 2024-04-19T23:34:00.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e6d44bb75c29407-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.963 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.963 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.963 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.963 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.963 DEBUG:boto:Method: GET 2024-04-19T23:34:00.963 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.963 DEBUG:boto:Data: 2024-04-19T23:34:00.963 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.963 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.963 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.964 DEBUG:boto:Params: {} 2024-04-19T23:34:00.964 DEBUG:boto:Token: None 2024-04-19T23:34:00.964 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.964 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004db0204f95735bc3-006622ff68-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.967 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.967 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:00.967 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:00.968 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.968 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.968 DEBUG:boto:Method: GET 2024-04-19T23:34:00.968 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:00.968 DEBUG:boto:Data: 2024-04-19T23:34:00.968 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.968 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.968 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.968 DEBUG:boto:Params: {} 2024-04-19T23:34:00.968 DEBUG:boto:Token: None 2024-04-19T23:34:00.969 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/ 2024-04-19T23:34:00.969 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA= 2024-04-19T23:34:00.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0GU8Zyc2DZUl6sY2mJK7sArGPA=', 'Content-Length': '0'} 2024-04-19T23:34:00.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e62dab7be7f7c415-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.971 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:00.972 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.972 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.972 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.972 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:34:00.972 DEBUG:boto:Data: 2024-04-19T23:34:00.972 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.972 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.972 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.972 DEBUG:boto:Params: {} 2024-04-19T23:34:00.972 DEBUG:boto:Token: None 2024-04-19T23:34:00.973 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:34:00.973 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Y1Xw0avEV0Z1DT2ySG44lxOVIm0= 2024-04-19T23:34:00.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y1Xw0avEV0Z1DT2ySG44lxOVIm0=', 'Content-Length': '0'} 2024-04-19T23:34:00.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004365e87b216d7048-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.975 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.975 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.975 DEBUG:boto:Method: HEAD 2024-04-19T23:34:00.975 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:34:00.975 DEBUG:boto:Data: 2024-04-19T23:34:00.976 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.976 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:00.976 DEBUG:boto:Port: 8000 2024-04-19T23:34:00.976 DEBUG:boto:Params: {} 2024-04-19T23:34:00.976 DEBUG:boto:Token: None 2024-04-19T23:34:00.976 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:34:00.976 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Y1Xw0avEV0Z1DT2ySG44lxOVIm0= 2024-04-19T23:34:00.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y1Xw0avEV0Z1DT2ySG44lxOVIm0=', 'Content-Length': '0'} 2024-04-19T23:34:00.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045aa37f8bdd9a562-006622ff68-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:00.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:00.979 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.979 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:00.979 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.979 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.979 DEBUG:boto:Method: GET 2024-04-19T23:34:00.980 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.980 DEBUG:boto:Data: 2024-04-19T23:34:00.980 DEBUG:boto:Headers: {} 2024-04-19T23:34:00.980 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:00.980 DEBUG:boto:Port: 8001 2024-04-19T23:34:00.980 DEBUG:boto:Params: {} 2024-04-19T23:34:00.980 DEBUG:boto:Token: None 2024-04-19T23:34:00.980 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:00 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:00.980 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4= 2024-04-19T23:34:00.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1J4V+mkAzEAhPuyrBxsLu8eJXm4=', 'Content-Length': '0'} 2024-04-19T23:34:00.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c66d9c7bf2c60bde-006622ff68-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.026 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:01.027 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:01.027 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:01.027 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:01.027 DEBUG:boto:Method: GET 2024-04-19T23:34:01.027 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:01.027 DEBUG:boto:Data: 2024-04-19T23:34:01.027 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.027 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.027 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.027 DEBUG:boto:Params: {} 2024-04-19T23:34:01.028 DEBUG:boto:Token: None 2024-04-19T23:34:01.028 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:34:01.028 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:os71FnIon+56txCKKHR8uxeYUmY= 2024-04-19T23:34:01.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:os71FnIon+56txCKKHR8uxeYUmY=', 'Content-Length': '0'} 2024-04-19T23:34:01.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5d7402b36129d61-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.070 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.070 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.071 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:01.071 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:01.071 DEBUG:boto:Method: GET 2024-04-19T23:34:01.071 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:01.071 DEBUG:boto:Data: 2024-04-19T23:34:01.071 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.071 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.071 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.071 DEBUG:boto:Params: {} 2024-04-19T23:34:01.071 DEBUG:boto:Token: None 2024-04-19T23:34:01.072 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.072 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o= 2024-04-19T23:34:01.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o=', 'Content-Length': '0'} 2024-04-19T23:34:01.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9ecf7408108460b-006622ff69-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.118 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.119 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.119 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.119 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:01.119 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:01.119 DEBUG:boto:Method: GET 2024-04-19T23:34:01.119 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:34:01.120 DEBUG:boto:Data: 2024-04-19T23:34:01.120 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.120 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.120 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.120 DEBUG:boto:Params: {} 2024-04-19T23:34:01.120 DEBUG:boto:Token: None 2024-04-19T23:34:01.120 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.120 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o= 2024-04-19T23:34:01.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o=', 'Content-Length': '0'} 2024-04-19T23:34:01.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009656b91b55ccfa33-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.123 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:01.124 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.124 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.124 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.124 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.124 DEBUG:boto:Method: GET 2024-04-19T23:34:01.124 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.124 DEBUG:boto:Data: 2024-04-19T23:34:01.124 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.124 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.124 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.124 DEBUG:boto:Params: {} 2024-04-19T23:34:01.125 DEBUG:boto:Token: None 2024-04-19T23:34:01.125 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.125 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yq42F7h2ZNmcG3hTKcOYlVzUDIg= 2024-04-19T23:34:01.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yq42F7h2ZNmcG3hTKcOYlVzUDIg=', 'Content-Length': '0'} 2024-04-19T23:34:01.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9b7ca1a5b8f5a8f-006622ff69-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.127 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.128 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.128 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.128 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.128 DEBUG:boto:Method: GET 2024-04-19T23:34:01.128 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.128 DEBUG:boto:Data: 2024-04-19T23:34:01.128 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.128 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.128 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.129 DEBUG:boto:Params: {} 2024-04-19T23:34:01.129 DEBUG:boto:Token: None 2024-04-19T23:34:01.129 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.129 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yq42F7h2ZNmcG3hTKcOYlVzUDIg= 2024-04-19T23:34:01.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yq42F7h2ZNmcG3hTKcOYlVzUDIg=', 'Content-Length': '0'} 2024-04-19T23:34:01.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000beecac084c74e4fc-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.131 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.132 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.132 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.132 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.132 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.132 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.133 DEBUG:boto:Data: 2024-04-19T23:34:01.133 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.133 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.133 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.133 DEBUG:boto:Params: {} 2024-04-19T23:34:01.133 DEBUG:boto:Token: None 2024-04-19T23:34:01.133 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.133 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WVyEH2y6pjejyJad6Fu/lgOcjhw= 2024-04-19T23:34:01.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WVyEH2y6pjejyJad6Fu/lgOcjhw=', 'Content-Length': '0'} 2024-04-19T23:34:01.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054a55585128f24dd-006622ff69-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.135 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.135 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.136 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.136 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.136 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.136 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.136 DEBUG:boto:Data: 2024-04-19T23:34:01.136 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.136 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.137 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.137 DEBUG:boto:Params: {} 2024-04-19T23:34:01.137 DEBUG:boto:Token: None 2024-04-19T23:34:01.137 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.137 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WVyEH2y6pjejyJad6Fu/lgOcjhw= 2024-04-19T23:34:01.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WVyEH2y6pjejyJad6Fu/lgOcjhw=', 'Content-Length': '0'} 2024-04-19T23:34:01.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004491e652fb0a59ec-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:01.139 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.139 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.139 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.139 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.139 DEBUG:boto:Method: GET 2024-04-19T23:34:01.140 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.140 DEBUG:boto:Data: 2024-04-19T23:34:01.140 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.140 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.140 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.140 DEBUG:boto:Params: {} 2024-04-19T23:34:01.140 DEBUG:boto:Token: None 2024-04-19T23:34:01.140 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.140 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yq42F7h2ZNmcG3hTKcOYlVzUDIg= 2024-04-19T23:34:01.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yq42F7h2ZNmcG3hTKcOYlVzUDIg=', 'Content-Length': '0'} 2024-04-19T23:34:01.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008090e84850c24c3a-006622ff69-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.187 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.187 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.187 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.188 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.188 DEBUG:boto:Method: GET 2024-04-19T23:34:01.188 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.188 DEBUG:boto:Data: 2024-04-19T23:34:01.188 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.188 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.188 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.188 DEBUG:boto:Params: {} 2024-04-19T23:34:01.189 DEBUG:boto:Token: None 2024-04-19T23:34:01.189 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.190 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yq42F7h2ZNmcG3hTKcOYlVzUDIg= 2024-04-19T23:34:01.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yq42F7h2ZNmcG3hTKcOYlVzUDIg=', 'Content-Length': '0'} 2024-04-19T23:34:01.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da29e7fe01d779b0-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.192 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.192 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.192 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:01.192 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:01.192 DEBUG:boto:Method: GET 2024-04-19T23:34:01.192 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:01.192 DEBUG:boto:Data: 2024-04-19T23:34:01.192 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.192 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.193 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.193 DEBUG:boto:Params: {} 2024-04-19T23:34:01.193 DEBUG:boto:Token: None 2024-04-19T23:34:01.193 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.193 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o= 2024-04-19T23:34:01.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o=', 'Content-Length': '0'} 2024-04-19T23:34:01.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002922737c050e8355-006622ff69-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.195 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.196 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.196 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.196 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:01.196 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:01.196 DEBUG:boto:Method: GET 2024-04-19T23:34:01.196 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:01.196 DEBUG:boto:Data: 2024-04-19T23:34:01.196 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.197 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.197 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.197 DEBUG:boto:Params: {} 2024-04-19T23:34:01.197 DEBUG:boto:Token: None 2024-04-19T23:34:01.197 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.197 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o= 2024-04-19T23:34:01.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o=', 'Content-Length': '0'} 2024-04-19T23:34:01.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e684fcd0afd3845b-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.200 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.200 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.200 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.200 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.200 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:34:01.200 DEBUG:boto:Data: 2024-04-19T23:34:01.200 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.200 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.201 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.201 DEBUG:boto:Params: {} 2024-04-19T23:34:01.201 DEBUG:boto:Token: None 2024-04-19T23:34:01.201 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:34:01.201 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1eTjcd9+2mFFiXDnBcgLOzZw/4U= 2024-04-19T23:34:01.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1eTjcd9+2mFFiXDnBcgLOzZw/4U=', 'Content-Length': '0'} 2024-04-19T23:34:01.203 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080e881433d5cf622-006622ff69-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.204 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.204 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.204 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.204 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:34:01.204 DEBUG:boto:Data: 2024-04-19T23:34:01.204 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.204 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.204 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.204 DEBUG:boto:Params: {} 2024-04-19T23:34:01.204 DEBUG:boto:Token: None 2024-04-19T23:34:01.205 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:34:01.205 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1eTjcd9+2mFFiXDnBcgLOzZw/4U= 2024-04-19T23:34:01.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1eTjcd9+2mFFiXDnBcgLOzZw/4U=', 'Content-Length': '0'} 2024-04-19T23:34:01.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000991204237755c550-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:01.207 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.207 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.207 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.207 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.208 DEBUG:boto:Method: GET 2024-04-19T23:34:01.208 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.208 DEBUG:boto:Data: 2024-04-19T23:34:01.208 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.208 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.208 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.208 DEBUG:boto:Params: {} 2024-04-19T23:34:01.208 DEBUG:boto:Token: None 2024-04-19T23:34:01.208 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.208 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yq42F7h2ZNmcG3hTKcOYlVzUDIg= 2024-04-19T23:34:01.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yq42F7h2ZNmcG3hTKcOYlVzUDIg=', 'Content-Length': '0'} 2024-04-19T23:34:01.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b997f3fc67e4f86-006622ff69-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.254 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.255 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.255 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.255 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.255 DEBUG:boto:Method: GET 2024-04-19T23:34:01.255 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.255 DEBUG:boto:Data: 2024-04-19T23:34:01.255 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.255 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.255 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.255 DEBUG:boto:Params: {} 2024-04-19T23:34:01.256 DEBUG:boto:Token: None 2024-04-19T23:34:01.256 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:34:01.256 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yq42F7h2ZNmcG3hTKcOYlVzUDIg= 2024-04-19T23:34:01.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yq42F7h2ZNmcG3hTKcOYlVzUDIg=', 'Content-Length': '0'} 2024-04-19T23:34:01.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b804a8649386e47e-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.258 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.259 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.259 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:01.259 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:01.259 DEBUG:boto:Method: GET 2024-04-19T23:34:01.259 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:01.259 DEBUG:boto:Data: 2024-04-19T23:34:01.259 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.259 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.259 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.259 DEBUG:boto:Params: {} 2024-04-19T23:34:01.260 DEBUG:boto:Token: None 2024-04-19T23:34:01.260 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.260 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o= 2024-04-19T23:34:01.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o=', 'Content-Length': '0'} 2024-04-19T23:34:01.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081f9e5def2df9ead-006622ff69-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.263 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.263 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.263 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.263 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:01.263 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:01.263 DEBUG:boto:Method: GET 2024-04-19T23:34:01.264 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:34:01.264 DEBUG:boto:Data: 2024-04-19T23:34:01.264 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.264 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.264 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.264 DEBUG:boto:Params: {} 2024-04-19T23:34:01.264 DEBUG:boto:Token: None 2024-04-19T23:34:01.264 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.264 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o= 2024-04-19T23:34:01.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o=', 'Content-Length': '0'} 2024-04-19T23:34:01.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bfab33b602fa00c-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.267 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:34:01.267 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.267 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.267 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.267 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.268 DEBUG:boto:Method: GET 2024-04-19T23:34:01.268 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.268 DEBUG:boto:Data: 2024-04-19T23:34:01.268 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.268 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.268 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.268 DEBUG:boto:Params: {} 2024-04-19T23:34:01.268 DEBUG:boto:Token: None 2024-04-19T23:34:01.268 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.268 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2mr4Bkon9wfF1zMhBy3ZFTtx/oc= 2024-04-19T23:34:01.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2mr4Bkon9wfF1zMhBy3ZFTtx/oc=', 'Content-Length': '0'} 2024-04-19T23:34:01.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000005104998cc622d8-006622ff69-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.270 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.271 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.271 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.271 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.271 DEBUG:boto:Method: GET 2024-04-19T23:34:01.271 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.271 DEBUG:boto:Data: 2024-04-19T23:34:01.271 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.271 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.271 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.271 DEBUG:boto:Params: {} 2024-04-19T23:34:01.272 DEBUG:boto:Token: None 2024-04-19T23:34:01.272 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.272 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2mr4Bkon9wfF1zMhBy3ZFTtx/oc= 2024-04-19T23:34:01.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2mr4Bkon9wfF1zMhBy3ZFTtx/oc=', 'Content-Length': '0'} 2024-04-19T23:34:01.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a0db84dcd47ef64-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.274 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.274 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.274 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.274 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.275 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.275 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.275 DEBUG:boto:Data: 2024-04-19T23:34:01.275 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.275 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.275 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.275 DEBUG:boto:Params: {} 2024-04-19T23:34:01.275 DEBUG:boto:Token: None 2024-04-19T23:34:01.275 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.276 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Q8mgo3fYOS0+sV04W07LTL3l6Qw= 2024-04-19T23:34:01.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q8mgo3fYOS0+sV04W07LTL3l6Qw=', 'Content-Length': '0'} 2024-04-19T23:34:01.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076c452ad4b1070b8-006622ff69-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.277 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.277 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.278 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.278 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.278 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.278 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.278 DEBUG:boto:Data: 2024-04-19T23:34:01.278 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.278 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.278 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.278 DEBUG:boto:Params: {} 2024-04-19T23:34:01.278 DEBUG:boto:Token: None 2024-04-19T23:34:01.279 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.279 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Q8mgo3fYOS0+sV04W07LTL3l6Qw= 2024-04-19T23:34:01.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q8mgo3fYOS0+sV04W07LTL3l6Qw=', 'Content-Length': '0'} 2024-04-19T23:34:01.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c51c843891ea7c81-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:34:01.281 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.281 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.281 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.281 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.281 DEBUG:boto:Method: GET 2024-04-19T23:34:01.282 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.282 DEBUG:boto:Data: 2024-04-19T23:34:01.282 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.282 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.282 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.282 DEBUG:boto:Params: {} 2024-04-19T23:34:01.282 DEBUG:boto:Token: None 2024-04-19T23:34:01.282 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.282 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2mr4Bkon9wfF1zMhBy3ZFTtx/oc= 2024-04-19T23:34:01.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2mr4Bkon9wfF1zMhBy3ZFTtx/oc=', 'Content-Length': '0'} 2024-04-19T23:34:01.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082bc61d730879eca-006622ff69-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.327 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.327 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.327 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.327 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.327 DEBUG:boto:Method: GET 2024-04-19T23:34:01.327 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.328 DEBUG:boto:Data: 2024-04-19T23:34:01.328 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.328 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.328 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.328 DEBUG:boto:Params: {} 2024-04-19T23:34:01.328 DEBUG:boto:Token: None 2024-04-19T23:34:01.328 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.328 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2mr4Bkon9wfF1zMhBy3ZFTtx/oc= 2024-04-19T23:34:01.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2mr4Bkon9wfF1zMhBy3ZFTtx/oc=', 'Content-Length': '0'} 2024-04-19T23:34:01.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eaca57ca696dc301-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.370 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.371 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.371 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:01.371 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:01.371 DEBUG:boto:Method: GET 2024-04-19T23:34:01.371 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:01.371 DEBUG:boto:Data: 2024-04-19T23:34:01.371 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.371 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.371 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.371 DEBUG:boto:Params: {} 2024-04-19T23:34:01.372 DEBUG:boto:Token: None 2024-04-19T23:34:01.372 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.372 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o= 2024-04-19T23:34:01.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o=', 'Content-Length': '0'} 2024-04-19T23:34:01.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009825380d86738401-006622ff69-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.374 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.375 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.375 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.375 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:01.375 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:01.375 DEBUG:boto:Method: GET 2024-04-19T23:34:01.375 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:01.375 DEBUG:boto:Data: 2024-04-19T23:34:01.376 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.376 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.376 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.376 DEBUG:boto:Params: {} 2024-04-19T23:34:01.376 DEBUG:boto:Token: None 2024-04-19T23:34:01.376 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.376 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o= 2024-04-19T23:34:01.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o=', 'Content-Length': '0'} 2024-04-19T23:34:01.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b4eb2c9ea3dcc85-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.422 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.423 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.423 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.423 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.423 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:34:01.423 DEBUG:boto:Data: 2024-04-19T23:34:01.423 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.424 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.424 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.424 DEBUG:boto:Params: {} 2024-04-19T23:34:01.424 DEBUG:boto:Token: None 2024-04-19T23:34:01.424 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:34:01.424 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tntwcEgAxcAYXuua+A+xiYVacvQ= 2024-04-19T23:34:01.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tntwcEgAxcAYXuua+A+xiYVacvQ=', 'Content-Length': '0'} 2024-04-19T23:34:01.428 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e206581d14f0095c-006622ff69-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.428 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.428 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.428 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.428 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:34:01.428 DEBUG:boto:Data: 2024-04-19T23:34:01.429 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.429 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.429 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.429 DEBUG:boto:Params: {} 2024-04-19T23:34:01.429 DEBUG:boto:Token: None 2024-04-19T23:34:01.429 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:34:01.429 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tntwcEgAxcAYXuua+A+xiYVacvQ= 2024-04-19T23:34:01.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tntwcEgAxcAYXuua+A+xiYVacvQ=', 'Content-Length': '0'} 2024-04-19T23:34:01.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b06f2d41e46b55d-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:34:01.432 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.432 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.432 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.432 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.432 DEBUG:boto:Method: GET 2024-04-19T23:34:01.433 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.433 DEBUG:boto:Data: 2024-04-19T23:34:01.433 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.433 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.433 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.433 DEBUG:boto:Params: {} 2024-04-19T23:34:01.433 DEBUG:boto:Token: None 2024-04-19T23:34:01.433 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.433 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2mr4Bkon9wfF1zMhBy3ZFTtx/oc= 2024-04-19T23:34:01.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2mr4Bkon9wfF1zMhBy3ZFTtx/oc=', 'Content-Length': '0'} 2024-04-19T23:34:01.435 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000669f4fb92fda67fb-006622ff69-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.436 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.436 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.436 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.436 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.436 DEBUG:boto:Method: GET 2024-04-19T23:34:01.436 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.436 DEBUG:boto:Data: 2024-04-19T23:34:01.436 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.436 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.436 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.436 DEBUG:boto:Params: {} 2024-04-19T23:34:01.437 DEBUG:boto:Token: None 2024-04-19T23:34:01.437 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:34:01.437 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2mr4Bkon9wfF1zMhBy3ZFTtx/oc= 2024-04-19T23:34:01.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2mr4Bkon9wfF1zMhBy3ZFTtx/oc=', 'Content-Length': '0'} 2024-04-19T23:34:01.439 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ca3136a1a132940-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.486 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.487 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.487 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:01.487 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:01.487 DEBUG:boto:Method: GET 2024-04-19T23:34:01.487 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:01.487 DEBUG:boto:Data: 2024-04-19T23:34:01.487 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.487 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.487 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.487 DEBUG:boto:Params: {} 2024-04-19T23:34:01.487 DEBUG:boto:Token: None 2024-04-19T23:34:01.488 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o= 2024-04-19T23:34:01.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o=', 'Content-Length': '0'} 2024-04-19T23:34:01.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015c2d3057c4d280f-006622ff69-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.491 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.491 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.491 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.491 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:01.491 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:01.491 DEBUG:boto:Method: GET 2024-04-19T23:34:01.491 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:34:01.492 DEBUG:boto:Data: 2024-04-19T23:34:01.492 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.492 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.492 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.492 DEBUG:boto:Params: {} 2024-04-19T23:34:01.492 DEBUG:boto:Token: None 2024-04-19T23:34:01.492 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.492 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o= 2024-04-19T23:34:01.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ViczlTseBvGhQuy3Eu8qQF3+6o=', 'Content-Length': '0'} 2024-04-19T23:34:01.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3f0a40a078444d5-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.538 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.539 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-21 zones={test-zone3, test-zone2} 2024-04-19T23:34:01.539 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:01.539 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:01.539 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.539 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:34:01.539 DEBUG:boto:Data: 2024-04-19T23:34:01.540 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.540 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:01.540 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.540 DEBUG:boto:Params: {} 2024-04-19T23:34:01.540 DEBUG:boto:Token: None 2024-04-19T23:34:01.540 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-19/ 2024-04-19T23:34:01.540 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cQ1QWJZQQKXk9SrWEu+SkEqonaM= 2024-04-19T23:34:01.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cQ1QWJZQQKXk9SrWEu+SkEqonaM=', 'Content-Length': '0'} 2024-04-19T23:34:01.543 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6052b5c42b1a622-006622ff69-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.543 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:01.543 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:01.543 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-19T23:34:01.543 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-19T23:34:01.543 DEBUG:boto:Method: GET 2024-04-19T23:34:01.544 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-19T23:34:01.544 DEBUG:boto:Data: 2024-04-19T23:34:01.544 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.544 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:01.544 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.544 DEBUG:boto:Params: {} 2024-04-19T23:34:01.544 DEBUG:boto:Token: None 2024-04-19T23:34:01.544 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-19/?versions 2024-04-19T23:34:01.544 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4c6scbiV4nOUrYaULtlmSsUl4qY= 2024-04-19T23:34:01.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4c6scbiV4nOUrYaULtlmSsUl4qY=', 'Content-Length': '0'} 2024-04-19T23:34:01.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037afeea81ffbb0fa-006622ff69-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.547 DEBUG:boto:b'aktlji-19obj-test-zone1999falseobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.547 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:01.548 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:01.548 DEBUG:boto:path=/aktlji-19/obj-test-zone1?acl 2024-04-19T23:34:01.548 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?acl 2024-04-19T23:34:01.548 DEBUG:boto:Method: PUT 2024-04-19T23:34:01.548 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?acl 2024-04-19T23:34:01.548 DEBUG:boto:Data: 2024-04-19T23:34:01.548 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-19T23:34:01.548 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:01.548 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.548 DEBUG:boto:Params: {} 2024-04-19T23:34:01.548 DEBUG:boto:Token: None 2024-04-19T23:34:01.549 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:34:01 GMT x-amz-acl:public-read /aktlji-19/obj-test-zone1?acl 2024-04-19T23:34:01.549 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dq9GywyH0paLsVAKWB9h/5qb1sE= 2024-04-19T23:34:01.549 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dq9GywyH0paLsVAKWB9h/5qb1sE=', 'Content-Length': '0'} 2024-04-19T23:34:01.556 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff1d1da4a499d883-006622ff69-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.556 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:01.556 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:01.556 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.556 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:34:01.557 DEBUG:boto:Data: 2024-04-19T23:34:01.557 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.557 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.557 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.557 DEBUG:boto:Params: {} 2024-04-19T23:34:01.557 DEBUG:boto:Token: None 2024-04-19T23:34:01.557 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-19/ 2024-04-19T23:34:01.557 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cQ1QWJZQQKXk9SrWEu+SkEqonaM= 2024-04-19T23:34:01.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cQ1QWJZQQKXk9SrWEu+SkEqonaM=', 'Content-Length': '0'} 2024-04-19T23:34:01.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003bb6f2cce121d6ce-006622ff69-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.560 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:01.560 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:01.560 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-19T23:34:01.560 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-19T23:34:01.560 DEBUG:boto:Method: GET 2024-04-19T23:34:01.560 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-19T23:34:01.561 DEBUG:boto:Data: 2024-04-19T23:34:01.561 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.561 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.561 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.561 DEBUG:boto:Params: {} 2024-04-19T23:34:01.561 DEBUG:boto:Token: None 2024-04-19T23:34:01.561 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-19/?versions 2024-04-19T23:34:01.561 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4c6scbiV4nOUrYaULtlmSsUl4qY= 2024-04-19T23:34:01.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4c6scbiV4nOUrYaULtlmSsUl4qY=', 'Content-Length': '0'} 2024-04-19T23:34:01.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bcb5570f57fd369-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.564 DEBUG:boto:b'aktlji-19obj-test-zone2999falseobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.564 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:01.564 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:01.564 DEBUG:boto:path=/aktlji-19/obj-test-zone2?acl 2024-04-19T23:34:01.564 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?acl 2024-04-19T23:34:01.564 DEBUG:boto:Method: PUT 2024-04-19T23:34:01.565 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?acl 2024-04-19T23:34:01.565 DEBUG:boto:Data: 2024-04-19T23:34:01.565 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-19T23:34:01.565 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.565 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.565 DEBUG:boto:Params: {} 2024-04-19T23:34:01.565 DEBUG:boto:Token: None 2024-04-19T23:34:01.565 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:34:01 GMT x-amz-acl:public-read /aktlji-19/obj-test-zone2?acl 2024-04-19T23:34:01.565 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rrrblEgoQvF05G9ZjXgh2wkJdXU= 2024-04-19T23:34:01.566 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rrrblEgoQvF05G9ZjXgh2wkJdXU=', 'Content-Length': '0'} 2024-04-19T23:34:01.573 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d8d90707bd6b893-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.573 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:01.573 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:01.573 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.573 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:34:01.574 DEBUG:boto:Data: 2024-04-19T23:34:01.574 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.574 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.574 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.574 DEBUG:boto:Params: {} 2024-04-19T23:34:01.574 DEBUG:boto:Token: None 2024-04-19T23:34:01.574 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-19/ 2024-04-19T23:34:01.574 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cQ1QWJZQQKXk9SrWEu+SkEqonaM= 2024-04-19T23:34:01.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cQ1QWJZQQKXk9SrWEu+SkEqonaM=', 'Content-Length': '0'} 2024-04-19T23:34:01.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b14f373c4a2cb00d-006622ff69-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.577 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:01.577 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:01.577 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999&prefix=obj-test-zone3 2024-04-19T23:34:01.577 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999&prefix=obj-test-zone3 2024-04-19T23:34:01.578 DEBUG:boto:Method: GET 2024-04-19T23:34:01.578 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999&prefix=obj-test-zone3 2024-04-19T23:34:01.578 DEBUG:boto:Data: 2024-04-19T23:34:01.578 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.578 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.578 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.578 DEBUG:boto:Params: {} 2024-04-19T23:34:01.578 DEBUG:boto:Token: None 2024-04-19T23:34:01.578 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-19/?versions 2024-04-19T23:34:01.579 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4c6scbiV4nOUrYaULtlmSsUl4qY= 2024-04-19T23:34:01.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4c6scbiV4nOUrYaULtlmSsUl4qY=', 'Content-Length': '0'} 2024-04-19T23:34:01.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0c4ca2f6c5bd4cd-006622ff69-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.622 DEBUG:boto:b'aktlji-19obj-test-zone3999falseobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.623 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:01.623 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:01.623 DEBUG:boto:path=/aktlji-19/obj-test-zone3?acl 2024-04-19T23:34:01.623 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?acl 2024-04-19T23:34:01.623 DEBUG:boto:Method: PUT 2024-04-19T23:34:01.623 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?acl 2024-04-19T23:34:01.623 DEBUG:boto:Data: 2024-04-19T23:34:01.623 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-19T23:34:01.624 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.624 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.624 DEBUG:boto:Params: {} 2024-04-19T23:34:01.624 DEBUG:boto:Token: None 2024-04-19T23:34:01.624 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:34:01 GMT x-amz-acl:public-read /aktlji-19/obj-test-zone3?acl 2024-04-19T23:34:01.624 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uyxhYcbsiSMl1jV+woAjbhURoyE= 2024-04-19T23:34:01.624 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uyxhYcbsiSMl1jV+woAjbhURoyE=', 'Content-Length': '0'} 2024-04-19T23:34:01.631 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006dc271113a121abd-006622ff69-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.631 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:34:01.631 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:34:01.631 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.632 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:34:01.632 DEBUG:boto:Data: 2024-04-19T23:34:01.632 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.632 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:01.632 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.632 DEBUG:boto:Params: {} 2024-04-19T23:34:01.632 DEBUG:boto:Token: None 2024-04-19T23:34:01.632 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-20/ 2024-04-19T23:34:01.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:94Zb3bkWhrEuk+gbcCNSiNw24GU= 2024-04-19T23:34:01.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:94Zb3bkWhrEuk+gbcCNSiNw24GU=', 'Content-Length': '0'} 2024-04-19T23:34:01.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000079d3e85aa72ac8c-006622ff69-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.635 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:34:01.635 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:34:01.635 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-19T23:34:01.635 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-19T23:34:01.635 DEBUG:boto:Method: GET 2024-04-19T23:34:01.635 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-19T23:34:01.636 DEBUG:boto:Data: 2024-04-19T23:34:01.636 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.636 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:01.636 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.636 DEBUG:boto:Params: {} 2024-04-19T23:34:01.636 DEBUG:boto:Token: None 2024-04-19T23:34:01.636 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-20/?versions 2024-04-19T23:34:01.636 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2n/LPDuPc7X7pQOkoI/MUWR0lLk= 2024-04-19T23:34:01.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2n/LPDuPc7X7pQOkoI/MUWR0lLk=', 'Content-Length': '0'} 2024-04-19T23:34:01.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000496362c31245a73d-006622ff69-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.682 DEBUG:boto:b'aktlji-20obj-test-zone1999falseobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.683 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:34:01.683 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:34:01.683 DEBUG:boto:path=/aktlji-20/obj-test-zone1?acl 2024-04-19T23:34:01.683 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?acl 2024-04-19T23:34:01.683 DEBUG:boto:Method: PUT 2024-04-19T23:34:01.683 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?acl 2024-04-19T23:34:01.683 DEBUG:boto:Data: 2024-04-19T23:34:01.683 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-19T23:34:01.684 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:01.684 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.684 DEBUG:boto:Params: {} 2024-04-19T23:34:01.684 DEBUG:boto:Token: None 2024-04-19T23:34:01.684 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:34:01 GMT x-amz-acl:public-read /aktlji-20/obj-test-zone1?acl 2024-04-19T23:34:01.684 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aARu1xjhbqr6pWi2oq6dxH7uPlg= 2024-04-19T23:34:01.684 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aARu1xjhbqr6pWi2oq6dxH7uPlg=', 'Content-Length': '0'} 2024-04-19T23:34:01.691 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000abf073463e80e1c9-006622ff69-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.692 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:34:01.692 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:34:01.692 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.692 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:34:01.692 DEBUG:boto:Data: 2024-04-19T23:34:01.692 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.692 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.692 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.692 DEBUG:boto:Params: {} 2024-04-19T23:34:01.692 DEBUG:boto:Token: None 2024-04-19T23:34:01.693 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-20/ 2024-04-19T23:34:01.693 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:94Zb3bkWhrEuk+gbcCNSiNw24GU= 2024-04-19T23:34:01.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:94Zb3bkWhrEuk+gbcCNSiNw24GU=', 'Content-Length': '0'} 2024-04-19T23:34:01.695 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4deccfd8b50649d-006622ff69-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.695 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:34:01.695 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:34:01.695 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-19T23:34:01.695 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-19T23:34:01.696 DEBUG:boto:Method: GET 2024-04-19T23:34:01.696 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-19T23:34:01.696 DEBUG:boto:Data: 2024-04-19T23:34:01.696 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.696 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.696 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.696 DEBUG:boto:Params: {} 2024-04-19T23:34:01.696 DEBUG:boto:Token: None 2024-04-19T23:34:01.696 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-20/?versions 2024-04-19T23:34:01.696 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2n/LPDuPc7X7pQOkoI/MUWR0lLk= 2024-04-19T23:34:01.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2n/LPDuPc7X7pQOkoI/MUWR0lLk=', 'Content-Length': '0'} 2024-04-19T23:34:01.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dc7ab73487510ad-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.702 DEBUG:boto:b'aktlji-20obj-test-zone2999falseobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.703 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:34:01.703 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:34:01.703 DEBUG:boto:path=/aktlji-20/obj-test-zone2?acl 2024-04-19T23:34:01.703 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?acl 2024-04-19T23:34:01.703 DEBUG:boto:Method: PUT 2024-04-19T23:34:01.703 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?acl 2024-04-19T23:34:01.703 DEBUG:boto:Data: 2024-04-19T23:34:01.703 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-19T23:34:01.703 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.704 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.704 DEBUG:boto:Params: {} 2024-04-19T23:34:01.704 DEBUG:boto:Token: None 2024-04-19T23:34:01.704 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:34:01 GMT x-amz-acl:public-read /aktlji-20/obj-test-zone2?acl 2024-04-19T23:34:01.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qTB2ieWfnNR+u/AbpHwXApxSSRg= 2024-04-19T23:34:01.704 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qTB2ieWfnNR+u/AbpHwXApxSSRg=', 'Content-Length': '0'} 2024-04-19T23:34:01.712 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007b0a72e669d2f094-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.712 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:34:01.712 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:34:01.712 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.712 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:34:01.712 DEBUG:boto:Data: 2024-04-19T23:34:01.713 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.713 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.713 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.713 DEBUG:boto:Params: {} 2024-04-19T23:34:01.713 DEBUG:boto:Token: None 2024-04-19T23:34:01.713 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-20/ 2024-04-19T23:34:01.713 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:94Zb3bkWhrEuk+gbcCNSiNw24GU= 2024-04-19T23:34:01.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:94Zb3bkWhrEuk+gbcCNSiNw24GU=', 'Content-Length': '0'} 2024-04-19T23:34:01.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bf68bb41ec1cbf3-006622ff69-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.716 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:34:01.716 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:34:01.716 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999&prefix=obj-test-zone3 2024-04-19T23:34:01.716 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999&prefix=obj-test-zone3 2024-04-19T23:34:01.716 DEBUG:boto:Method: GET 2024-04-19T23:34:01.717 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999&prefix=obj-test-zone3 2024-04-19T23:34:01.717 DEBUG:boto:Data: 2024-04-19T23:34:01.717 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.717 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.717 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.717 DEBUG:boto:Params: {} 2024-04-19T23:34:01.717 DEBUG:boto:Token: None 2024-04-19T23:34:01.717 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-20/?versions 2024-04-19T23:34:01.717 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2n/LPDuPc7X7pQOkoI/MUWR0lLk= 2024-04-19T23:34:01.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2n/LPDuPc7X7pQOkoI/MUWR0lLk=', 'Content-Length': '0'} 2024-04-19T23:34:01.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c4eafb2d2a7b104-006622ff69-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.720 DEBUG:boto:b'aktlji-20obj-test-zone3999falseobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.720 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:34:01.720 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:34:01.721 DEBUG:boto:path=/aktlji-20/obj-test-zone3?acl 2024-04-19T23:34:01.721 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?acl 2024-04-19T23:34:01.721 DEBUG:boto:Method: PUT 2024-04-19T23:34:01.721 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?acl 2024-04-19T23:34:01.721 DEBUG:boto:Data: 2024-04-19T23:34:01.721 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-19T23:34:01.721 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.721 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.721 DEBUG:boto:Params: {} 2024-04-19T23:34:01.721 DEBUG:boto:Token: None 2024-04-19T23:34:01.722 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:34:01 GMT x-amz-acl:public-read /aktlji-20/obj-test-zone3?acl 2024-04-19T23:34:01.722 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vJNcBwTosTA3VMkmuemZ03WlPpQ= 2024-04-19T23:34:01.722 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vJNcBwTosTA3VMkmuemZ03WlPpQ=', 'Content-Length': '0'} 2024-04-19T23:34:01.729 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000073164bffad5774a5-006622ff69-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.729 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.729 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.729 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.730 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:34:01.730 DEBUG:boto:Data: 2024-04-19T23:34:01.730 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.730 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:01.730 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.730 DEBUG:boto:Params: {} 2024-04-19T23:34:01.730 DEBUG:boto:Token: None 2024-04-19T23:34:01.730 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.730 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ynuXcg9/IlL/PWiZGM6gT651c6g= 2024-04-19T23:34:01.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ynuXcg9/IlL/PWiZGM6gT651c6g=', 'Content-Length': '0'} 2024-04-19T23:34:01.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f7eb8b6d3275b71-006622ff69-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.733 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.733 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.733 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-19T23:34:01.734 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-19T23:34:01.734 DEBUG:boto:Method: GET 2024-04-19T23:34:01.734 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999&prefix=obj-test-zone1 2024-04-19T23:34:01.734 DEBUG:boto:Data: 2024-04-19T23:34:01.734 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.734 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:01.734 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.734 DEBUG:boto:Params: {} 2024-04-19T23:34:01.734 DEBUG:boto:Token: None 2024-04-19T23:34:01.735 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/?versions 2024-04-19T23:34:01.735 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fWkw0/jE+Y0qVOXk0XOTOu0snto= 2024-04-19T23:34:01.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fWkw0/jE+Y0qVOXk0XOTOu0snto=', 'Content-Length': '0'} 2024-04-19T23:34:01.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccdf80e61794571e-006622ff69-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.737 DEBUG:boto:b'aktlji-21obj-test-zone1999falseobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.738 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:01.738 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:34:01.738 DEBUG:boto:path=/aktlji-21/obj-test-zone1?acl 2024-04-19T23:34:01.738 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?acl 2024-04-19T23:34:01.738 DEBUG:boto:Method: PUT 2024-04-19T23:34:01.738 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?acl 2024-04-19T23:34:01.738 DEBUG:boto:Data: 2024-04-19T23:34:01.739 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-19T23:34:01.739 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:01.739 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.739 DEBUG:boto:Params: {} 2024-04-19T23:34:01.739 DEBUG:boto:Token: None 2024-04-19T23:34:01.739 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:34:01 GMT x-amz-acl:public-read /aktlji-21/obj-test-zone1?acl 2024-04-19T23:34:01.739 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hYy7+8qFQupOytqWmEghevPOetY= 2024-04-19T23:34:01.739 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hYy7+8qFQupOytqWmEghevPOetY=', 'Content-Length': '0'} 2024-04-19T23:34:01.748 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e61361bfeca33fa2-006622ff69-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.748 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.748 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.748 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.748 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:34:01.749 DEBUG:boto:Data: 2024-04-19T23:34:01.749 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.749 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.749 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.749 DEBUG:boto:Params: {} 2024-04-19T23:34:01.749 DEBUG:boto:Token: None 2024-04-19T23:34:01.749 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.749 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ynuXcg9/IlL/PWiZGM6gT651c6g= 2024-04-19T23:34:01.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ynuXcg9/IlL/PWiZGM6gT651c6g=', 'Content-Length': '0'} 2024-04-19T23:34:01.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9b16cc777f653d4-006622ff69-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.752 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.752 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.752 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-19T23:34:01.752 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-19T23:34:01.752 DEBUG:boto:Method: GET 2024-04-19T23:34:01.752 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999&prefix=obj-test-zone2 2024-04-19T23:34:01.752 DEBUG:boto:Data: 2024-04-19T23:34:01.753 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.753 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.753 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.753 DEBUG:boto:Params: {} 2024-04-19T23:34:01.753 DEBUG:boto:Token: None 2024-04-19T23:34:01.753 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/?versions 2024-04-19T23:34:01.753 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fWkw0/jE+Y0qVOXk0XOTOu0snto= 2024-04-19T23:34:01.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fWkw0/jE+Y0qVOXk0XOTOu0snto=', 'Content-Length': '0'} 2024-04-19T23:34:01.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eca2fc2fce2d3abd-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.802 DEBUG:boto:b'aktlji-21obj-test-zone2999falseobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.803 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.803 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:34:01.803 DEBUG:boto:path=/aktlji-21/obj-test-zone2?acl 2024-04-19T23:34:01.803 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?acl 2024-04-19T23:34:01.803 DEBUG:boto:Method: PUT 2024-04-19T23:34:01.803 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?acl 2024-04-19T23:34:01.803 DEBUG:boto:Data: 2024-04-19T23:34:01.803 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-19T23:34:01.804 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:01.804 DEBUG:boto:Port: 8000 2024-04-19T23:34:01.804 DEBUG:boto:Params: {} 2024-04-19T23:34:01.804 DEBUG:boto:Token: None 2024-04-19T23:34:01.804 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:34:01 GMT x-amz-acl:public-read /aktlji-21/obj-test-zone2?acl 2024-04-19T23:34:01.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:v6WzQ+TU3+UcP1WSKLXAmiUPPc8= 2024-04-19T23:34:01.804 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:v6WzQ+TU3+UcP1WSKLXAmiUPPc8=', 'Content-Length': '0'} 2024-04-19T23:34:01.812 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000001eb2352e10bb0f6-006622ff69-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.812 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.812 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.812 DEBUG:boto:Method: HEAD 2024-04-19T23:34:01.812 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:34:01.812 DEBUG:boto:Data: 2024-04-19T23:34:01.812 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.812 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.812 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.813 DEBUG:boto:Params: {} 2024-04-19T23:34:01.813 DEBUG:boto:Token: None 2024-04-19T23:34:01.813 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/ 2024-04-19T23:34:01.813 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ynuXcg9/IlL/PWiZGM6gT651c6g= 2024-04-19T23:34:01.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ynuXcg9/IlL/PWiZGM6gT651c6g=', 'Content-Length': '0'} 2024-04-19T23:34:01.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c4be3c4637eb928-006622ff69-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.816 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:34:01.816 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:34:01.816 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999&prefix=obj-test-zone3 2024-04-19T23:34:01.816 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999&prefix=obj-test-zone3 2024-04-19T23:34:01.816 DEBUG:boto:Method: GET 2024-04-19T23:34:01.816 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999&prefix=obj-test-zone3 2024-04-19T23:34:01.816 DEBUG:boto:Data: 2024-04-19T23:34:01.816 DEBUG:boto:Headers: {} 2024-04-19T23:34:01.816 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.817 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.817 DEBUG:boto:Params: {} 2024-04-19T23:34:01.817 DEBUG:boto:Token: None 2024-04-19T23:34:01.817 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:01 GMT /aktlji-21/?versions 2024-04-19T23:34:01.817 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fWkw0/jE+Y0qVOXk0XOTOu0snto= 2024-04-19T23:34:01.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fWkw0/jE+Y0qVOXk0XOTOu0snto=', 'Content-Length': '0'} 2024-04-19T23:34:01.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ff12dea8638a479-006622ff69-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.819 DEBUG:boto:b'aktlji-21obj-test-zone3999falseobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:01.820 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.820 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:34:01.820 DEBUG:boto:path=/aktlji-21/obj-test-zone3?acl 2024-04-19T23:34:01.820 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?acl 2024-04-19T23:34:01.820 DEBUG:boto:Method: PUT 2024-04-19T23:34:01.820 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?acl 2024-04-19T23:34:01.820 DEBUG:boto:Data: 2024-04-19T23:34:01.821 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-19T23:34:01.821 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:01.821 DEBUG:boto:Port: 8001 2024-04-19T23:34:01.821 DEBUG:boto:Params: {} 2024-04-19T23:34:01.821 DEBUG:boto:Token: None 2024-04-19T23:34:01.821 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:34:01 GMT x-amz-acl:public-read /aktlji-21/obj-test-zone3?acl 2024-04-19T23:34:01.821 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iS5g5k081tm3ELjj2VrNMMbXKyU= 2024-04-19T23:34:01.821 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iS5g5k081tm3ELjj2VrNMMbXKyU=', 'Content-Length': '0'} 2024-04-19T23:34:01.828 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000045752ff90b6d96d1-006622ff69-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:01.828 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-19 2024-04-19T23:34:01.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:34:01.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:34:01.829 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:34:01.896 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:34:01.896 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:34:01.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:34:01.942+0000 7f8cead34780 1 waiting for incremental sync to catch up: 2024-04-19T23:34:01.937 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000010.26.6, , 00000000003.24.10, , 00000000007.49.11, , , , , ] 2024-04-19T23:34:01.937 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000011.29.6, , 00000000003.24.10, , 00000000007.49.11, , , , , ] 2024-04-19T23:34:06.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:34:06.942+0000 7f8cead34780 1 waiting for incremental sync to catch up: 2024-04-19T23:34:06.939 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000010.26.6, , 00000000003.24.10, , 00000000007.49.11, , , , , ] 2024-04-19T23:34:06.939 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000011.29.6, , 00000000003.24.10, , 00000000007.49.11, , , , , ] 2024-04-19T23:34:11.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:34:11.944+0000 7f8cead34780 1 waiting for incremental sync to catch up: 2024-04-19T23:34:11.940 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000010.26.6, , 00000000003.24.10, , 00000000007.49.11, , , , , ] 2024-04-19T23:34:11.941 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000011.29.6, , 00000000003.24.10, , 00000000007.49.11, , , , , ] 2024-04-19T23:34:14.493 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-19T23:34:14.498+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-19T23:34:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:34:16.946+0000 7f8cead34780 1 waiting for incremental sync to catch up: 2024-04-19T23:34:16.941 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000010.26.6, , 00000000003.24.10, , 00000000007.49.11, , , , , ] 2024-04-19T23:34:16.942 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000011.29.6, , 00000000003.24.10, , 00000000007.49.11, , , , , ] 2024-04-19T23:34:21.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:34:21.947+0000 7f8cead34780 1 bucket sync caught up with source: 2024-04-19T23:34:21.943 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000011.29.6, , 00000000003.24.10, , 00000000007.49.11, , , , , ] 2024-04-19T23:34:21.943 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000011.29.6, , 00000000003.24.10, , 00000000007.49.11, , , , , ] 2024-04-19T23:34:21.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:34:21.947+0000 7f8cead34780 0 bucket checkpoint complete 2024-04-19T23:34:21.960 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-19 2024-04-19T23:34:21.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:34:21.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:34:21.961 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:34:22.026 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:34:22.026 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:34:22.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:22.072+0000 7fe72d645780 1 bucket sync caught up with source: 2024-04-19T23:34:22.067 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000011.29.6, , 00000000003.24.10, , 00000000007.49.11, , , , , ] 2024-04-19T23:34:22.067 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000011.29.6, , 00000000003.24.10, , 00000000007.49.11, , , , , ] 2024-04-19T23:34:22.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:22.072+0000 7fe72d645780 0 bucket checkpoint complete 2024-04-19T23:34:22.081 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-19 2024-04-19T23:34:22.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:34:22.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:34:22.081 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:34:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:34:22.143 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:34:22.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:22.189+0000 7fb71319f780 1 waiting for incremental sync to catch up: 2024-04-19T23:34:22.186 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.55.10, , 00000000011.73.6, , 00000000005.115.6, , , , , ] 2024-04-19T23:34:22.187 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.55.10, , 00000000011.73.6, , 00000000007.118.10, , , , , ] 2024-04-19T23:34:27.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:27.191+0000 7fb71319f780 1 waiting for incremental sync to catch up: 2024-04-19T23:34:27.186 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.55.10, , 00000000011.73.6, , 00000000005.115.6, , , , , ] 2024-04-19T23:34:27.186 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.55.10, , 00000000011.73.6, , 00000000007.118.10, , , , , ] 2024-04-19T23:34:32.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:32.192+0000 7fb71319f780 1 waiting for incremental sync to catch up: 2024-04-19T23:34:32.187 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.55.10, , 00000000011.73.6, , 00000000005.115.6, , , , , ] 2024-04-19T23:34:32.187 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.55.10, , 00000000011.73.6, , 00000000007.118.10, , , , , ] 2024-04-19T23:34:37.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:37.193+0000 7fb71319f780 1 waiting for incremental sync to catch up: 2024-04-19T23:34:37.188 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.55.10, , 00000000011.73.6, , 00000000005.115.6, , , , , ] 2024-04-19T23:34:37.188 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.55.10, , 00000000011.73.6, , 00000000007.118.10, , , , , ] 2024-04-19T23:34:42.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:42.193+0000 7fb71319f780 1 waiting for incremental sync to catch up: 2024-04-19T23:34:42.189 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.55.10, , 00000000011.73.6, , 00000000005.115.6, , , , , ] 2024-04-19T23:34:42.189 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.55.10, , 00000000011.73.6, , 00000000007.118.10, , , , , ] 2024-04-19T23:34:47.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:47.195+0000 7fb71319f780 1 waiting for incremental sync to catch up: 2024-04-19T23:34:47.191 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.55.10, , 00000000011.73.6, , 00000000005.115.6, , , , , ] 2024-04-19T23:34:47.191 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.55.10, , 00000000011.73.6, , 00000000007.118.10, , , , , ] 2024-04-19T23:34:52.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:52.197+0000 7fb71319f780 1 waiting for incremental sync to catch up: 2024-04-19T23:34:52.193 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.55.10, , 00000000011.73.6, , 00000000005.115.6, , , , , ] 2024-04-19T23:34:52.193 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.55.10, , 00000000011.73.6, , 00000000007.118.10, , , , , ] 2024-04-19T23:34:57.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:57.198+0000 7fb71319f780 1 bucket sync caught up with source: 2024-04-19T23:34:57.194 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.55.10, , 00000000011.73.6, , 00000000007.118.10, , , , , ] 2024-04-19T23:34:57.194 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.55.10, , 00000000011.73.6, , 00000000007.118.10, , , , , ] 2024-04-19T23:34:57.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:57.198+0000 7fb71319f780 0 bucket checkpoint complete 2024-04-19T23:34:57.211 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-19 2024-04-19T23:34:57.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:34:57.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:34:57.212 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:34:57.278 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:34:57.278 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:34:57.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:57.323+0000 7f2f687c8780 1 bucket sync caught up with source: 2024-04-19T23:34:57.319 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.55.10, , 00000000011.73.6, , 00000000007.118.10, , , , , ] 2024-04-19T23:34:57.319 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.55.10, , 00000000011.73.6, , 00000000007.118.10, , , , , ] 2024-04-19T23:34:57.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:57.323+0000 7f2f687c8780 0 bucket checkpoint complete 2024-04-19T23:34:57.334 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-19 2024-04-19T23:34:57.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:34:57.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:34:57.334 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:34:57.397 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:34:57.397 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:34:57.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:57.443+0000 7fb23b7f0780 1 bucket sync caught up with source: 2024-04-19T23:34:57.439 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.32.10, , 00000000003.34.10, , 00000000015.83.6, , , , , ] 2024-04-19T23:34:57.439 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.32.10, , 00000000003.34.10, , 00000000015.83.6, , , , , ] 2024-04-19T23:34:57.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:34:57.443+0000 7fb23b7f0780 0 bucket checkpoint complete 2024-04-19T23:34:57.453 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-19 2024-04-19T23:34:57.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:34:57.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:34:57.454 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:34:57.522 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:34:57.522 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:34:57.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:34:57.572+0000 7f34bf007780 1 bucket sync caught up with source: 2024-04-19T23:34:57.567 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000003.32.10, , 00000000003.34.10, , 00000000015.83.6, , , , , ] 2024-04-19T23:34:57.568 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000003.32.10, , 00000000003.34.10, , 00000000015.83.6, , , , , ] 2024-04-19T23:34:57.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:34:57.572+0000 7f34bf007780 0 bucket checkpoint complete 2024-04-19T23:34:57.582 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-19 zones={test-zone2, test-zone1} 2024-04-19T23:34:57.583 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.583 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.583 DEBUG:boto:Method: HEAD 2024-04-19T23:34:57.583 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:34:57.583 DEBUG:boto:Data: 2024-04-19T23:34:57.583 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.583 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.583 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.583 DEBUG:boto:Params: {} 2024-04-19T23:34:57.584 DEBUG:boto:Token: None 2024-04-19T23:34:57.584 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/ 2024-04-19T23:34:57.584 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:biZt030t2HZ2Mkxpw6cg0kj9Ees= 2024-04-19T23:34:57.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:biZt030t2HZ2Mkxpw6cg0kj9Ees=', 'Content-Length': '0'} 2024-04-19T23:34:57.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094fd00c108875f27-006622ffa1-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.587 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.587 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.587 DEBUG:boto:Method: HEAD 2024-04-19T23:34:57.588 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:34:57.588 DEBUG:boto:Data: 2024-04-19T23:34:57.588 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.588 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.588 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.588 DEBUG:boto:Params: {} 2024-04-19T23:34:57.588 DEBUG:boto:Token: None 2024-04-19T23:34:57.588 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/ 2024-04-19T23:34:57.588 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:biZt030t2HZ2Mkxpw6cg0kj9Ees= 2024-04-19T23:34:57.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:biZt030t2HZ2Mkxpw6cg0kj9Ees=', 'Content-Length': '0'} 2024-04-19T23:34:57.591 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000732ef2d251d730a-006622ffa1-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.591 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:34:57.591 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.591 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.591 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:57.592 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:57.592 DEBUG:boto:Method: GET 2024-04-19T23:34:57.592 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:34:57.592 DEBUG:boto:Data: 2024-04-19T23:34:57.592 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.592 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.592 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.592 DEBUG:boto:Params: {} 2024-04-19T23:34:57.592 DEBUG:boto:Token: None 2024-04-19T23:34:57.592 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/?versions 2024-04-19T23:34:57.593 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tAvKAUj55DBRLFO88kuGw2iA6kw= 2024-04-19T23:34:57.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tAvKAUj55DBRLFO88kuGw2iA6kw=', 'Content-Length': '0'} 2024-04-19T23:34:57.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005375a650d41ee29e-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.595 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:57.596 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:34:57.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:34:57.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:34:57.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:34:57.596 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:34:57.596 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.596 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.597 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:57.597 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:57.597 DEBUG:boto:Method: GET 2024-04-19T23:34:57.597 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:34:57.597 DEBUG:boto:Data: 2024-04-19T23:34:57.597 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.597 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.597 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.597 DEBUG:boto:Params: {} 2024-04-19T23:34:57.597 DEBUG:boto:Token: None 2024-04-19T23:34:57.598 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/?versions 2024-04-19T23:34:57.598 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tAvKAUj55DBRLFO88kuGw2iA6kw= 2024-04-19T23:34:57.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tAvKAUj55DBRLFO88kuGw2iA6kw=', 'Content-Length': '0'} 2024-04-19T23:34:57.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000497c9a608105535e-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.601 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:57.601 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:34:57.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:34:57.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:34:57.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:34:57.602 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.602 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.602 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:57.602 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:57.602 DEBUG:boto:Method: GET 2024-04-19T23:34:57.602 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:34:57.602 DEBUG:boto:Data: 2024-04-19T23:34:57.602 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.602 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.603 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.603 DEBUG:boto:Params: {} 2024-04-19T23:34:57.603 DEBUG:boto:Token: None 2024-04-19T23:34:57.603 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/?versions 2024-04-19T23:34:57.603 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tAvKAUj55DBRLFO88kuGw2iA6kw= 2024-04-19T23:34:57.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tAvKAUj55DBRLFO88kuGw2iA6kw=', 'Content-Length': '0'} 2024-04-19T23:34:57.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7cdf247e4859a20-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.605 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:57.606 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.606 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.606 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:57.606 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:57.606 DEBUG:boto:Method: GET 2024-04-19T23:34:57.606 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:34:57.606 DEBUG:boto:Data: 2024-04-19T23:34:57.606 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.606 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.606 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.607 DEBUG:boto:Params: {} 2024-04-19T23:34:57.607 DEBUG:boto:Token: None 2024-04-19T23:34:57.607 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/?versions 2024-04-19T23:34:57.607 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tAvKAUj55DBRLFO88kuGw2iA6kw= 2024-04-19T23:34:57.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tAvKAUj55DBRLFO88kuGw2iA6kw=', 'Content-Length': '0'} 2024-04-19T23:34:57.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cca6739955485947-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.610 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:57.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:34:57.610 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:57.611 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:57.611 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.611 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.611 DEBUG:boto:Method: GET 2024-04-19T23:34:57.611 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.611 DEBUG:boto:Data: 2024-04-19T23:34:57.611 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.611 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.611 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.611 DEBUG:boto:Params: {} 2024-04-19T23:34:57.611 DEBUG:boto:Token: None 2024-04-19T23:34:57.612 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.612 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ii8Mm8wO4vKDLfr5kkTYvHoGXf4= 2024-04-19T23:34:57.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ii8Mm8wO4vKDLfr5kkTYvHoGXf4=', 'Content-Length': '0'} 2024-04-19T23:34:57.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000025b26d27b36f6de1-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.614 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:57.615 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:57.615 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.615 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.615 DEBUG:boto:Method: GET 2024-04-19T23:34:57.616 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.616 DEBUG:boto:Data: 2024-04-19T23:34:57.616 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.616 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.616 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.616 DEBUG:boto:Params: {} 2024-04-19T23:34:57.616 DEBUG:boto:Token: None 2024-04-19T23:34:57.617 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.617 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ii8Mm8wO4vKDLfr5kkTYvHoGXf4= 2024-04-19T23:34:57.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ii8Mm8wO4vKDLfr5kkTYvHoGXf4=', 'Content-Length': '0'} 2024-04-19T23:34:57.619 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000538b9e16184250a6-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.619 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:57.619 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:57.619 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.619 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.619 DEBUG:boto:Method: HEAD 2024-04-19T23:34:57.619 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.619 DEBUG:boto:Data: 2024-04-19T23:34:57.620 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.620 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.620 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.620 DEBUG:boto:Params: {} 2024-04-19T23:34:57.620 DEBUG:boto:Token: None 2024-04-19T23:34:57.620 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.620 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/9Hr1t+wrgjK3O2cSD9zy9zSBhM= 2024-04-19T23:34:57.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/9Hr1t+wrgjK3O2cSD9zy9zSBhM=', 'Content-Length': '0'} 2024-04-19T23:34:57.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000016ef16ec79d7ff09-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.622 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:57.622 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:57.622 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.623 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.623 DEBUG:boto:Method: HEAD 2024-04-19T23:34:57.623 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.623 DEBUG:boto:Data: 2024-04-19T23:34:57.623 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.623 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.623 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.623 DEBUG:boto:Params: {} 2024-04-19T23:34:57.623 DEBUG:boto:Token: None 2024-04-19T23:34:57.625 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.625 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/9Hr1t+wrgjK3O2cSD9zy9zSBhM= 2024-04-19T23:34:57.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/9Hr1t+wrgjK3O2cSD9zy9zSBhM=', 'Content-Length': '0'} 2024-04-19T23:34:57.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a169bdcc44b2f3b9-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:34:57.628 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:57.628 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:57.628 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.628 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.628 DEBUG:boto:Method: GET 2024-04-19T23:34:57.628 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.628 DEBUG:boto:Data: 2024-04-19T23:34:57.628 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.628 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.629 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.629 DEBUG:boto:Params: {} 2024-04-19T23:34:57.629 DEBUG:boto:Token: None 2024-04-19T23:34:57.629 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.629 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ii8Mm8wO4vKDLfr5kkTYvHoGXf4= 2024-04-19T23:34:57.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ii8Mm8wO4vKDLfr5kkTYvHoGXf4=', 'Content-Length': '0'} 2024-04-19T23:34:57.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007f7641ef5bee68b7-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.631 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:57.631 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:57.631 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.631 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.632 DEBUG:boto:Method: GET 2024-04-19T23:34:57.632 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.632 DEBUG:boto:Data: 2024-04-19T23:34:57.632 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.632 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.632 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.632 DEBUG:boto:Params: {} 2024-04-19T23:34:57.632 DEBUG:boto:Token: None 2024-04-19T23:34:57.632 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ii8Mm8wO4vKDLfr5kkTYvHoGXf4= 2024-04-19T23:34:57.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ii8Mm8wO4vKDLfr5kkTYvHoGXf4=', 'Content-Length': '0'} 2024-04-19T23:34:57.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006658ae8cf6f64cfb-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.634 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.634 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.635 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:57.635 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:57.635 DEBUG:boto:Method: GET 2024-04-19T23:34:57.635 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:34:57.635 DEBUG:boto:Data: 2024-04-19T23:34:57.635 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.635 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.635 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.635 DEBUG:boto:Params: {} 2024-04-19T23:34:57.635 DEBUG:boto:Token: None 2024-04-19T23:34:57.636 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/ 2024-04-19T23:34:57.636 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8= 2024-04-19T23:34:57.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8=', 'Content-Length': '0'} 2024-04-19T23:34:57.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090b596cb40aa194d-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.682 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:57.683 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.683 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.683 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:57.683 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:57.683 DEBUG:boto:Method: GET 2024-04-19T23:34:57.683 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:34:57.683 DEBUG:boto:Data: 2024-04-19T23:34:57.683 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.683 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.683 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.684 DEBUG:boto:Params: {} 2024-04-19T23:34:57.684 DEBUG:boto:Token: None 2024-04-19T23:34:57.684 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/ 2024-04-19T23:34:57.684 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8= 2024-04-19T23:34:57.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8=', 'Content-Length': '0'} 2024-04-19T23:34:57.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0c2d48aab17019f-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.687 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:57.687 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:57.687 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:57.687 DEBUG:boto:Method: HEAD 2024-04-19T23:34:57.687 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:34:57.688 DEBUG:boto:Data: 2024-04-19T23:34:57.688 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.688 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.688 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.688 DEBUG:boto:Params: {} 2024-04-19T23:34:57.688 DEBUG:boto:Token: None 2024-04-19T23:34:57.688 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/dummy 2024-04-19T23:34:57.688 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VTUTyX/4zy0QTPZirBNzvEO9CBo= 2024-04-19T23:34:57.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VTUTyX/4zy0QTPZirBNzvEO9CBo=', 'Content-Length': '0'} 2024-04-19T23:34:57.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c03e63bf7f18f51-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.691 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:57.691 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:57.691 DEBUG:boto:Method: HEAD 2024-04-19T23:34:57.691 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:34:57.691 DEBUG:boto:Data: 2024-04-19T23:34:57.691 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.691 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.691 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.691 DEBUG:boto:Params: {} 2024-04-19T23:34:57.692 DEBUG:boto:Token: None 2024-04-19T23:34:57.692 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/dummy 2024-04-19T23:34:57.692 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VTUTyX/4zy0QTPZirBNzvEO9CBo= 2024-04-19T23:34:57.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VTUTyX/4zy0QTPZirBNzvEO9CBo=', 'Content-Length': '0'} 2024-04-19T23:34:57.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000874dd90f7296034f-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:34:57.694 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:57.695 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:57.695 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.695 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.695 DEBUG:boto:Method: GET 2024-04-19T23:34:57.695 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.695 DEBUG:boto:Data: 2024-04-19T23:34:57.695 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.695 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.695 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.695 DEBUG:boto:Params: {} 2024-04-19T23:34:57.695 DEBUG:boto:Token: None 2024-04-19T23:34:57.696 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.696 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ii8Mm8wO4vKDLfr5kkTYvHoGXf4= 2024-04-19T23:34:57.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ii8Mm8wO4vKDLfr5kkTYvHoGXf4=', 'Content-Length': '0'} 2024-04-19T23:34:57.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046032288d795235d-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.698 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:57.698 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:57.698 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.698 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.698 DEBUG:boto:Method: GET 2024-04-19T23:34:57.698 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.698 DEBUG:boto:Data: 2024-04-19T23:34:57.699 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.699 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.699 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.699 DEBUG:boto:Params: {} 2024-04-19T23:34:57.699 DEBUG:boto:Token: None 2024-04-19T23:34:57.699 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:57.699 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ii8Mm8wO4vKDLfr5kkTYvHoGXf4= 2024-04-19T23:34:57.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ii8Mm8wO4vKDLfr5kkTYvHoGXf4=', 'Content-Length': '0'} 2024-04-19T23:34:57.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000530ff4c3acde26ce-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.701 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.701 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.701 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:57.701 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:57.701 DEBUG:boto:Method: GET 2024-04-19T23:34:57.702 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:34:57.702 DEBUG:boto:Data: 2024-04-19T23:34:57.702 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.702 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.702 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.702 DEBUG:boto:Params: {} 2024-04-19T23:34:57.702 DEBUG:boto:Token: None 2024-04-19T23:34:57.702 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/ 2024-04-19T23:34:57.702 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8= 2024-04-19T23:34:57.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8=', 'Content-Length': '0'} 2024-04-19T23:34:57.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a4ade35e97089a-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.705 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:57.705 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.705 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.706 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:57.706 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:57.706 DEBUG:boto:Method: GET 2024-04-19T23:34:57.706 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:34:57.706 DEBUG:boto:Data: 2024-04-19T23:34:57.706 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.706 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.706 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.706 DEBUG:boto:Params: {} 2024-04-19T23:34:57.706 DEBUG:boto:Token: None 2024-04-19T23:34:57.707 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/ 2024-04-19T23:34:57.707 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8= 2024-04-19T23:34:57.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8=', 'Content-Length': '0'} 2024-04-19T23:34:57.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003d06d66c23b58b0-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.709 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:57.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:57.710 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.710 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.710 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.710 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.710 DEBUG:boto:Method: GET 2024-04-19T23:34:57.710 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.710 DEBUG:boto:Data: 2024-04-19T23:34:57.710 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.710 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.711 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.711 DEBUG:boto:Params: {} 2024-04-19T23:34:57.711 DEBUG:boto:Token: None 2024-04-19T23:34:57.711 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.711 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AHTDYAxG9+6uIK7laxtr3cNOgnU= 2024-04-19T23:34:57.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AHTDYAxG9+6uIK7laxtr3cNOgnU=', 'Content-Length': '0'} 2024-04-19T23:34:57.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082191cd97507b2b6-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.713 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.713 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.713 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.714 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.714 DEBUG:boto:Method: GET 2024-04-19T23:34:57.714 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.714 DEBUG:boto:Data: 2024-04-19T23:34:57.714 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.714 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.714 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.714 DEBUG:boto:Params: {} 2024-04-19T23:34:57.714 DEBUG:boto:Token: None 2024-04-19T23:34:57.714 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.715 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AHTDYAxG9+6uIK7laxtr3cNOgnU= 2024-04-19T23:34:57.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AHTDYAxG9+6uIK7laxtr3cNOgnU=', 'Content-Length': '0'} 2024-04-19T23:34:57.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe297cfa32f626c0-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.717 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.717 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.717 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.717 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.717 DEBUG:boto:Method: HEAD 2024-04-19T23:34:57.717 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.717 DEBUG:boto:Data: 2024-04-19T23:34:57.717 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.718 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.718 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.718 DEBUG:boto:Params: {} 2024-04-19T23:34:57.718 DEBUG:boto:Token: None 2024-04-19T23:34:57.718 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.718 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o/D86tz2bR3go8N49YR5/onFTUc= 2024-04-19T23:34:57.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o/D86tz2bR3go8N49YR5/onFTUc=', 'Content-Length': '0'} 2024-04-19T23:34:57.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aea166ff125f0306-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.720 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.720 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.721 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.721 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.721 DEBUG:boto:Method: HEAD 2024-04-19T23:34:57.721 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.721 DEBUG:boto:Data: 2024-04-19T23:34:57.721 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.721 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.721 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.721 DEBUG:boto:Params: {} 2024-04-19T23:34:57.721 DEBUG:boto:Token: None 2024-04-19T23:34:57.721 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.722 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o/D86tz2bR3go8N49YR5/onFTUc= 2024-04-19T23:34:57.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o/D86tz2bR3go8N49YR5/onFTUc=', 'Content-Length': '0'} 2024-04-19T23:34:57.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085db0b35c80ff2ce-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:57.724 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.724 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.724 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.724 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.724 DEBUG:boto:Method: GET 2024-04-19T23:34:57.724 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.724 DEBUG:boto:Data: 2024-04-19T23:34:57.724 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.724 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.724 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.725 DEBUG:boto:Params: {} 2024-04-19T23:34:57.725 DEBUG:boto:Token: None 2024-04-19T23:34:57.725 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.725 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AHTDYAxG9+6uIK7laxtr3cNOgnU= 2024-04-19T23:34:57.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AHTDYAxG9+6uIK7laxtr3cNOgnU=', 'Content-Length': '0'} 2024-04-19T23:34:57.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b324c40318f0f6cf-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.770 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.771 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.771 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.771 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.771 DEBUG:boto:Method: GET 2024-04-19T23:34:57.771 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.771 DEBUG:boto:Data: 2024-04-19T23:34:57.771 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.772 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.772 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.772 DEBUG:boto:Params: {} 2024-04-19T23:34:57.772 DEBUG:boto:Token: None 2024-04-19T23:34:57.772 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.772 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AHTDYAxG9+6uIK7laxtr3cNOgnU= 2024-04-19T23:34:57.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AHTDYAxG9+6uIK7laxtr3cNOgnU=', 'Content-Length': '0'} 2024-04-19T23:34:57.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1e9fb580944694d-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.775 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.775 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.775 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:57.775 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:57.775 DEBUG:boto:Method: GET 2024-04-19T23:34:57.775 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:57.775 DEBUG:boto:Data: 2024-04-19T23:34:57.775 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.775 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.775 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.776 DEBUG:boto:Params: {} 2024-04-19T23:34:57.776 DEBUG:boto:Token: None 2024-04-19T23:34:57.776 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/ 2024-04-19T23:34:57.776 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8= 2024-04-19T23:34:57.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8=', 'Content-Length': '0'} 2024-04-19T23:34:57.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf792885d8c51a53-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.822 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:57.823 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.823 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.823 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:57.823 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:57.823 DEBUG:boto:Method: GET 2024-04-19T23:34:57.823 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:57.823 DEBUG:boto:Data: 2024-04-19T23:34:57.823 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.823 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.824 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.824 DEBUG:boto:Params: {} 2024-04-19T23:34:57.824 DEBUG:boto:Token: None 2024-04-19T23:34:57.824 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/ 2024-04-19T23:34:57.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8= 2024-04-19T23:34:57.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8=', 'Content-Length': '0'} 2024-04-19T23:34:57.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b252c90168cca309-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.827 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:57.827 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.827 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.827 DEBUG:boto:Method: HEAD 2024-04-19T23:34:57.827 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:34:57.828 DEBUG:boto:Data: 2024-04-19T23:34:57.828 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.828 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.828 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.828 DEBUG:boto:Params: {} 2024-04-19T23:34:57.828 DEBUG:boto:Token: None 2024-04-19T23:34:57.828 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:34:57.828 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ofe0y/VBzn7F8RTGonmRtVHiQRc= 2024-04-19T23:34:57.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ofe0y/VBzn7F8RTGonmRtVHiQRc=', 'Content-Length': '0'} 2024-04-19T23:34:57.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cfa2a10c3d8f66a-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.831 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.831 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.831 DEBUG:boto:Method: HEAD 2024-04-19T23:34:57.831 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:34:57.831 DEBUG:boto:Data: 2024-04-19T23:34:57.831 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.831 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.831 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.831 DEBUG:boto:Params: {} 2024-04-19T23:34:57.831 DEBUG:boto:Token: None 2024-04-19T23:34:57.832 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:34:57.832 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ofe0y/VBzn7F8RTGonmRtVHiQRc= 2024-04-19T23:34:57.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ofe0y/VBzn7F8RTGonmRtVHiQRc=', 'Content-Length': '0'} 2024-04-19T23:34:57.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000368f629024e2e8e2-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:57.835 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.835 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.835 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.835 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.835 DEBUG:boto:Method: GET 2024-04-19T23:34:57.835 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.835 DEBUG:boto:Data: 2024-04-19T23:34:57.835 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.835 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.835 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.835 DEBUG:boto:Params: {} 2024-04-19T23:34:57.836 DEBUG:boto:Token: None 2024-04-19T23:34:57.836 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.836 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AHTDYAxG9+6uIK7laxtr3cNOgnU= 2024-04-19T23:34:57.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AHTDYAxG9+6uIK7laxtr3cNOgnU=', 'Content-Length': '0'} 2024-04-19T23:34:57.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f40231e0282bd72c-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.878 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.878 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:57.879 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.879 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.879 DEBUG:boto:Method: GET 2024-04-19T23:34:57.879 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.879 DEBUG:boto:Data: 2024-04-19T23:34:57.879 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.879 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.879 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.879 DEBUG:boto:Params: {} 2024-04-19T23:34:57.879 DEBUG:boto:Token: None 2024-04-19T23:34:57.879 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:57.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AHTDYAxG9+6uIK7laxtr3cNOgnU= 2024-04-19T23:34:57.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AHTDYAxG9+6uIK7laxtr3cNOgnU=', 'Content-Length': '0'} 2024-04-19T23:34:57.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053537ec75004183c-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.882 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.882 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.882 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:57.882 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:57.882 DEBUG:boto:Method: GET 2024-04-19T23:34:57.882 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:57.882 DEBUG:boto:Data: 2024-04-19T23:34:57.882 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.883 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.883 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.883 DEBUG:boto:Params: {} 2024-04-19T23:34:57.883 DEBUG:boto:Token: None 2024-04-19T23:34:57.883 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/ 2024-04-19T23:34:57.883 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8= 2024-04-19T23:34:57.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8=', 'Content-Length': '0'} 2024-04-19T23:34:57.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b9bb0d4267e38e5-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.926 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:57.927 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:57.927 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:57.927 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:57.927 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:57.927 DEBUG:boto:Method: GET 2024-04-19T23:34:57.927 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:57.927 DEBUG:boto:Data: 2024-04-19T23:34:57.927 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.928 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.928 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.928 DEBUG:boto:Params: {} 2024-04-19T23:34:57.928 DEBUG:boto:Token: None 2024-04-19T23:34:57.928 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/ 2024-04-19T23:34:57.928 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8= 2024-04-19T23:34:57.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OiFF5l74Py5fpLhP88A150fW3Q8=', 'Content-Length': '0'} 2024-04-19T23:34:57.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000151f3c5b16ea5a9e-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.931 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:57.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:57.932 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:57.932 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:57.932 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.932 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.932 DEBUG:boto:Method: GET 2024-04-19T23:34:57.932 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.932 DEBUG:boto:Data: 2024-04-19T23:34:57.932 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.932 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.932 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.933 DEBUG:boto:Params: {} 2024-04-19T23:34:57.933 DEBUG:boto:Token: None 2024-04-19T23:34:57.933 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.933 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aArfl04WZSEcGSu+MDETw6ta9L8= 2024-04-19T23:34:57.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aArfl04WZSEcGSu+MDETw6ta9L8=', 'Content-Length': '0'} 2024-04-19T23:34:57.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b48a8205109fb00-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.935 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:57.935 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:57.936 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.936 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.936 DEBUG:boto:Method: GET 2024-04-19T23:34:57.936 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.936 DEBUG:boto:Data: 2024-04-19T23:34:57.936 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.936 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.936 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.936 DEBUG:boto:Params: {} 2024-04-19T23:34:57.936 DEBUG:boto:Token: None 2024-04-19T23:34:57.937 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.937 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aArfl04WZSEcGSu+MDETw6ta9L8= 2024-04-19T23:34:57.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aArfl04WZSEcGSu+MDETw6ta9L8=', 'Content-Length': '0'} 2024-04-19T23:34:57.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000995dd505d9240e97-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.982 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:57.982 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:57.983 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.983 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.983 DEBUG:boto:Method: HEAD 2024-04-19T23:34:57.983 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.983 DEBUG:boto:Data: 2024-04-19T23:34:57.983 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.983 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.983 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.983 DEBUG:boto:Params: {} 2024-04-19T23:34:57.983 DEBUG:boto:Token: None 2024-04-19T23:34:57.984 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:r99tfYWXGNxCLxiMPi52xv3qm4M= 2024-04-19T23:34:57.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r99tfYWXGNxCLxiMPi52xv3qm4M=', 'Content-Length': '0'} 2024-04-19T23:34:57.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c0b25d8ef97ac33-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.986 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:57.986 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:57.986 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.986 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.986 DEBUG:boto:Method: HEAD 2024-04-19T23:34:57.986 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.986 DEBUG:boto:Data: 2024-04-19T23:34:57.986 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.987 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.987 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.987 DEBUG:boto:Params: {} 2024-04-19T23:34:57.987 DEBUG:boto:Token: None 2024-04-19T23:34:57.987 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.987 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:r99tfYWXGNxCLxiMPi52xv3qm4M= 2024-04-19T23:34:57.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r99tfYWXGNxCLxiMPi52xv3qm4M=', 'Content-Length': '0'} 2024-04-19T23:34:57.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082d6e7bbcab64072-006622ffa1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:57.990 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:57.990 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:57.990 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.990 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.990 DEBUG:boto:Method: GET 2024-04-19T23:34:57.990 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.991 DEBUG:boto:Data: 2024-04-19T23:34:57.991 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.991 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:57.991 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.991 DEBUG:boto:Params: {} 2024-04-19T23:34:57.991 DEBUG:boto:Token: None 2024-04-19T23:34:57.991 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.992 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aArfl04WZSEcGSu+MDETw6ta9L8= 2024-04-19T23:34:57.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aArfl04WZSEcGSu+MDETw6ta9L8=', 'Content-Length': '0'} 2024-04-19T23:34:57.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee9f714611115722-006622ffa1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:57.994 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:57.994 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:57.994 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.994 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.994 DEBUG:boto:Method: GET 2024-04-19T23:34:57.994 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.994 DEBUG:boto:Data: 2024-04-19T23:34:57.994 DEBUG:boto:Headers: {} 2024-04-19T23:34:57.995 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:57.995 DEBUG:boto:Port: 8000 2024-04-19T23:34:57.995 DEBUG:boto:Params: {} 2024-04-19T23:34:57.995 DEBUG:boto:Token: None 2024-04-19T23:34:57.995 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:57 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:57.995 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aArfl04WZSEcGSu+MDETw6ta9L8= 2024-04-19T23:34:57.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:57 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aArfl04WZSEcGSu+MDETw6ta9L8=', 'Content-Length': '0'} 2024-04-19T23:34:57.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e035431410bc57e5-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.042 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.043 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.043 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.043 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.043 DEBUG:boto:Method: GET 2024-04-19T23:34:58.043 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.043 DEBUG:boto:Data: 2024-04-19T23:34:58.043 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.043 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:58.043 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.043 DEBUG:boto:Params: {} 2024-04-19T23:34:58.044 DEBUG:boto:Token: None 2024-04-19T23:34:58.044 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.044 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abb86f0b8425b927-006622ffa2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.047 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.047 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.047 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.047 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.047 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.047 DEBUG:boto:Method: GET 2024-04-19T23:34:58.048 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.048 DEBUG:boto:Data: 2024-04-19T23:34:58.048 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.048 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.048 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.048 DEBUG:boto:Params: {} 2024-04-19T23:34:58.048 DEBUG:boto:Token: None 2024-04-19T23:34:58.048 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.048 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeb8e94220d2f37c-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.051 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.052 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.052 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.052 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.052 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:34:58.053 DEBUG:boto:Data: 2024-04-19T23:34:58.053 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.053 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:58.053 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.054 DEBUG:boto:Params: {} 2024-04-19T23:34:58.054 DEBUG:boto:Token: None 2024-04-19T23:34:58.054 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:34:58.054 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SVIusA3St/sDpEmUQIXJtobhInI= 2024-04-19T23:34:58.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SVIusA3St/sDpEmUQIXJtobhInI=', 'Content-Length': '0'} 2024-04-19T23:34:58.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045cf086e9f2a49a4-006622ffa2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.057 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.057 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.057 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.057 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:34:58.057 DEBUG:boto:Data: 2024-04-19T23:34:58.058 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.058 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.058 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.058 DEBUG:boto:Params: {} 2024-04-19T23:34:58.058 DEBUG:boto:Token: None 2024-04-19T23:34:58.058 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:34:58.058 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SVIusA3St/sDpEmUQIXJtobhInI= 2024-04-19T23:34:58.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SVIusA3St/sDpEmUQIXJtobhInI=', 'Content-Length': '0'} 2024-04-19T23:34:58.060 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004eca2191359c3e53-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:58.061 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.061 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.061 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.061 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.061 DEBUG:boto:Method: GET 2024-04-19T23:34:58.062 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.062 DEBUG:boto:Data: 2024-04-19T23:34:58.062 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.062 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:58.062 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.062 DEBUG:boto:Params: {} 2024-04-19T23:34:58.062 DEBUG:boto:Token: None 2024-04-19T23:34:58.062 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.062 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54= 2024-04-19T23:34:58.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54=', 'Content-Length': '0'} 2024-04-19T23:34:58.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d2188f894ca691a-006622ffa2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.065 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.065 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.065 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.065 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.065 DEBUG:boto:Method: GET 2024-04-19T23:34:58.065 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.065 DEBUG:boto:Data: 2024-04-19T23:34:58.065 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.065 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.066 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.066 DEBUG:boto:Params: {} 2024-04-19T23:34:58.066 DEBUG:boto:Token: None 2024-04-19T23:34:58.066 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.066 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54= 2024-04-19T23:34:58.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54=', 'Content-Length': '0'} 2024-04-19T23:34:58.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018937d9793e72f26-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.114 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.115 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.115 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.115 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.115 DEBUG:boto:Method: GET 2024-04-19T23:34:58.115 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.115 DEBUG:boto:Data: 2024-04-19T23:34:58.115 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.115 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:58.115 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.116 DEBUG:boto:Params: {} 2024-04-19T23:34:58.116 DEBUG:boto:Token: None 2024-04-19T23:34:58.116 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.116 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d3e473832749a9b-006622ffa2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.119 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.119 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.119 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.119 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.120 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.120 DEBUG:boto:Method: GET 2024-04-19T23:34:58.120 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.120 DEBUG:boto:Data: 2024-04-19T23:34:58.120 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.120 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.120 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.120 DEBUG:boto:Params: {} 2024-04-19T23:34:58.120 DEBUG:boto:Token: None 2024-04-19T23:34:58.121 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.121 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000814c1a905199adca-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.123 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:34:58.124 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.124 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.124 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.124 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.124 DEBUG:boto:Method: GET 2024-04-19T23:34:58.124 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.125 DEBUG:boto:Data: 2024-04-19T23:34:58.125 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.125 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:58.125 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.125 DEBUG:boto:Params: {} 2024-04-19T23:34:58.125 DEBUG:boto:Token: None 2024-04-19T23:34:58.125 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.125 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KFOrU7hfuYT9VsQMAls+LwMzuHY= 2024-04-19T23:34:58.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KFOrU7hfuYT9VsQMAls+LwMzuHY=', 'Content-Length': '0'} 2024-04-19T23:34:58.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b877708ff1230407-006622ffa2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.170 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.171 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.171 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.171 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.171 DEBUG:boto:Method: GET 2024-04-19T23:34:58.171 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.171 DEBUG:boto:Data: 2024-04-19T23:34:58.171 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.171 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.172 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.172 DEBUG:boto:Params: {} 2024-04-19T23:34:58.172 DEBUG:boto:Token: None 2024-04-19T23:34:58.172 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.172 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KFOrU7hfuYT9VsQMAls+LwMzuHY= 2024-04-19T23:34:58.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KFOrU7hfuYT9VsQMAls+LwMzuHY=', 'Content-Length': '0'} 2024-04-19T23:34:58.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c327c355c279620-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.175 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.175 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.175 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.175 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.176 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.176 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.176 DEBUG:boto:Data: 2024-04-19T23:34:58.176 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.176 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:58.176 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.176 DEBUG:boto:Params: {} 2024-04-19T23:34:58.177 DEBUG:boto:Token: None 2024-04-19T23:34:58.177 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.177 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EHRzi5TtSVxey32B9OQplsDI9pk= 2024-04-19T23:34:58.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EHRzi5TtSVxey32B9OQplsDI9pk=', 'Content-Length': '0'} 2024-04-19T23:34:58.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b97046cc0ed79a5-006622ffa2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.179 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.179 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.179 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.179 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.179 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.179 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.179 DEBUG:boto:Data: 2024-04-19T23:34:58.180 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.180 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.180 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.180 DEBUG:boto:Params: {} 2024-04-19T23:34:58.180 DEBUG:boto:Token: None 2024-04-19T23:34:58.180 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.181 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EHRzi5TtSVxey32B9OQplsDI9pk= 2024-04-19T23:34:58.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EHRzi5TtSVxey32B9OQplsDI9pk=', 'Content-Length': '0'} 2024-04-19T23:34:58.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077007540222366ce-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:34:58.183 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.183 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.183 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.183 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.184 DEBUG:boto:Method: GET 2024-04-19T23:34:58.184 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.184 DEBUG:boto:Data: 2024-04-19T23:34:58.184 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.184 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:58.184 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.184 DEBUG:boto:Params: {} 2024-04-19T23:34:58.184 DEBUG:boto:Token: None 2024-04-19T23:34:58.184 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.184 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KFOrU7hfuYT9VsQMAls+LwMzuHY= 2024-04-19T23:34:58.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KFOrU7hfuYT9VsQMAls+LwMzuHY=', 'Content-Length': '0'} 2024-04-19T23:34:58.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000814e08c04f0d668-006622ffa2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.226 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.227 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.227 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.227 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.227 DEBUG:boto:Method: GET 2024-04-19T23:34:58.227 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.227 DEBUG:boto:Data: 2024-04-19T23:34:58.227 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.227 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.227 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.227 DEBUG:boto:Params: {} 2024-04-19T23:34:58.228 DEBUG:boto:Token: None 2024-04-19T23:34:58.228 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.228 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KFOrU7hfuYT9VsQMAls+LwMzuHY= 2024-04-19T23:34:58.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KFOrU7hfuYT9VsQMAls+LwMzuHY=', 'Content-Length': '0'} 2024-04-19T23:34:58.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060fc42a39f06997c-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.271 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.271 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.271 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:58.271 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:58.271 DEBUG:boto:Method: GET 2024-04-19T23:34:58.271 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:58.271 DEBUG:boto:Data: 2024-04-19T23:34:58.271 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.271 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:58.272 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.272 DEBUG:boto:Params: {} 2024-04-19T23:34:58.272 DEBUG:boto:Token: None 2024-04-19T23:34:58.272 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.272 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e62575e2322a7788-006622ffa2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.322 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.323 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.323 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.323 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:58.323 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:58.323 DEBUG:boto:Method: GET 2024-04-19T23:34:58.323 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:58.323 DEBUG:boto:Data: 2024-04-19T23:34:58.324 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.324 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.324 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.324 DEBUG:boto:Params: {} 2024-04-19T23:34:58.324 DEBUG:boto:Token: None 2024-04-19T23:34:58.324 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000277e644117f6b6-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.370 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.371 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.371 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.371 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.371 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:34:58.371 DEBUG:boto:Data: 2024-04-19T23:34:58.371 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.371 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:58.371 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.371 DEBUG:boto:Params: {} 2024-04-19T23:34:58.372 DEBUG:boto:Token: None 2024-04-19T23:34:58.372 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:34:58.372 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uV28NBK+7rFb/HsskP+08WE2dl0= 2024-04-19T23:34:58.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uV28NBK+7rFb/HsskP+08WE2dl0=', 'Content-Length': '0'} 2024-04-19T23:34:58.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094a8de8562b1071e-006622ffa2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.375 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.375 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.375 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.375 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:34:58.375 DEBUG:boto:Data: 2024-04-19T23:34:58.375 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.376 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.376 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.376 DEBUG:boto:Params: {} 2024-04-19T23:34:58.376 DEBUG:boto:Token: None 2024-04-19T23:34:58.376 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:34:58.376 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uV28NBK+7rFb/HsskP+08WE2dl0= 2024-04-19T23:34:58.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uV28NBK+7rFb/HsskP+08WE2dl0=', 'Content-Length': '0'} 2024-04-19T23:34:58.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db82140883dd959a-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:34:58.379 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.380 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.380 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.380 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.380 DEBUG:boto:Method: GET 2024-04-19T23:34:58.380 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.380 DEBUG:boto:Data: 2024-04-19T23:34:58.380 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.380 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:58.381 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.381 DEBUG:boto:Params: {} 2024-04-19T23:34:58.381 DEBUG:boto:Token: None 2024-04-19T23:34:58.381 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.381 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KFOrU7hfuYT9VsQMAls+LwMzuHY= 2024-04-19T23:34:58.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KFOrU7hfuYT9VsQMAls+LwMzuHY=', 'Content-Length': '0'} 2024-04-19T23:34:58.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce1915976950282f-006622ffa2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.427 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.427 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:58.427 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.427 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.427 DEBUG:boto:Method: GET 2024-04-19T23:34:58.427 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.427 DEBUG:boto:Data: 2024-04-19T23:34:58.427 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.427 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.428 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.428 DEBUG:boto:Params: {} 2024-04-19T23:34:58.428 DEBUG:boto:Token: None 2024-04-19T23:34:58.428 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:58.428 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KFOrU7hfuYT9VsQMAls+LwMzuHY= 2024-04-19T23:34:58.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KFOrU7hfuYT9VsQMAls+LwMzuHY=', 'Content-Length': '0'} 2024-04-19T23:34:58.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092934bc8be5e24fe-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.474 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.474 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.475 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:58.475 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:58.475 DEBUG:boto:Method: GET 2024-04-19T23:34:58.475 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:58.475 DEBUG:boto:Data: 2024-04-19T23:34:58.475 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.475 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:58.475 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.475 DEBUG:boto:Params: {} 2024-04-19T23:34:58.476 DEBUG:boto:Token: None 2024-04-19T23:34:58.476 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.476 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db25e9e9bfabcceb-006622ffa2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.518 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.519 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.519 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.519 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:58.519 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:58.519 DEBUG:boto:Method: GET 2024-04-19T23:34:58.519 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:58.519 DEBUG:boto:Data: 2024-04-19T23:34:58.520 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.520 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.520 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.520 DEBUG:boto:Params: {} 2024-04-19T23:34:58.520 DEBUG:boto:Token: None 2024-04-19T23:34:58.520 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.520 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb634990f95b4450-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.523 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.523 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-19 zones={test-zone2, test-zone1} 2024-04-19T23:34:58.523 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-19 zones={test-zone3, test-zone1} 2024-04-19T23:34:58.524 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.524 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.524 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.524 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:34:58.524 DEBUG:boto:Data: 2024-04-19T23:34:58.524 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.524 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.524 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.524 DEBUG:boto:Params: {} 2024-04-19T23:34:58.524 DEBUG:boto:Token: None 2024-04-19T23:34:58.524 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.525 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zFWiomOagNnMJ2KRNiDDBGTVSHY= 2024-04-19T23:34:58.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zFWiomOagNnMJ2KRNiDDBGTVSHY=', 'Content-Length': '0'} 2024-04-19T23:34:58.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000715416cac9fa342b-006622ffa2-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.527 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.528 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.528 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.528 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:34:58.528 DEBUG:boto:Data: 2024-04-19T23:34:58.528 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.528 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.528 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.528 DEBUG:boto:Params: {} 2024-04-19T23:34:58.528 DEBUG:boto:Token: None 2024-04-19T23:34:58.529 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.529 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zFWiomOagNnMJ2KRNiDDBGTVSHY= 2024-04-19T23:34:58.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zFWiomOagNnMJ2KRNiDDBGTVSHY=', 'Content-Length': '0'} 2024-04-19T23:34:58.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000654ff4abb3522c3c-006622ffa2-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.531 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:34:58.531 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.531 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.531 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:58.531 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:58.532 DEBUG:boto:Method: GET 2024-04-19T23:34:58.532 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:34:58.532 DEBUG:boto:Data: 2024-04-19T23:34:58.532 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.532 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.532 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.532 DEBUG:boto:Params: {} 2024-04-19T23:34:58.532 DEBUG:boto:Token: None 2024-04-19T23:34:58.532 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/?versions 2024-04-19T23:34:58.532 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:moCa7RgMh0KbvhLOdXbigWoRi/E= 2024-04-19T23:34:58.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:moCa7RgMh0KbvhLOdXbigWoRi/E=', 'Content-Length': '0'} 2024-04-19T23:34:58.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000596f2ca93dff07d4-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.535 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.536 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:34:58.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:34:58.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:34:58.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:34:58.536 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:34:58.536 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.536 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.536 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:58.536 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:58.537 DEBUG:boto:Method: GET 2024-04-19T23:34:58.537 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:34:58.537 DEBUG:boto:Data: 2024-04-19T23:34:58.537 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.537 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.537 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.537 DEBUG:boto:Params: {} 2024-04-19T23:34:58.537 DEBUG:boto:Token: None 2024-04-19T23:34:58.537 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/?versions 2024-04-19T23:34:58.538 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:moCa7RgMh0KbvhLOdXbigWoRi/E= 2024-04-19T23:34:58.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:moCa7RgMh0KbvhLOdXbigWoRi/E=', 'Content-Length': '0'} 2024-04-19T23:34:58.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007881b167b34743ea-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.540 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.541 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:34:58.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:34:58.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:34:58.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:34:58.541 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.541 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.542 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:58.542 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:58.542 DEBUG:boto:Method: GET 2024-04-19T23:34:58.542 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:34:58.542 DEBUG:boto:Data: 2024-04-19T23:34:58.542 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.542 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.542 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.542 DEBUG:boto:Params: {} 2024-04-19T23:34:58.542 DEBUG:boto:Token: None 2024-04-19T23:34:58.543 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/?versions 2024-04-19T23:34:58.543 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:moCa7RgMh0KbvhLOdXbigWoRi/E= 2024-04-19T23:34:58.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:moCa7RgMh0KbvhLOdXbigWoRi/E=', 'Content-Length': '0'} 2024-04-19T23:34:58.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6686fda2b6fecf4-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.546 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.546 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.546 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.546 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:58.547 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:58.547 DEBUG:boto:Method: GET 2024-04-19T23:34:58.547 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:34:58.547 DEBUG:boto:Data: 2024-04-19T23:34:58.547 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.547 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.547 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.547 DEBUG:boto:Params: {} 2024-04-19T23:34:58.547 DEBUG:boto:Token: None 2024-04-19T23:34:58.548 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/?versions 2024-04-19T23:34:58.548 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:moCa7RgMh0KbvhLOdXbigWoRi/E= 2024-04-19T23:34:58.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:moCa7RgMh0KbvhLOdXbigWoRi/E=', 'Content-Length': '0'} 2024-04-19T23:34:58.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0191a3c3fb25367-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.550 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:34:58.551 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:58.551 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:58.551 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.551 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.551 DEBUG:boto:Method: GET 2024-04-19T23:34:58.551 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.551 DEBUG:boto:Data: 2024-04-19T23:34:58.551 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.551 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.551 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.552 DEBUG:boto:Params: {} 2024-04-19T23:34:58.552 DEBUG:boto:Token: None 2024-04-19T23:34:58.552 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.552 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yr8DQQiYFl0QjQq+ezC0lDUUfF4= 2024-04-19T23:34:58.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yr8DQQiYFl0QjQq+ezC0lDUUfF4=', 'Content-Length': '0'} 2024-04-19T23:34:58.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006040cdc0fc06a61d-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.554 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:58.554 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:58.554 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.554 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.554 DEBUG:boto:Method: GET 2024-04-19T23:34:58.554 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.555 DEBUG:boto:Data: 2024-04-19T23:34:58.555 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.555 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.555 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.555 DEBUG:boto:Params: {} 2024-04-19T23:34:58.555 DEBUG:boto:Token: None 2024-04-19T23:34:58.555 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.555 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yr8DQQiYFl0QjQq+ezC0lDUUfF4= 2024-04-19T23:34:58.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yr8DQQiYFl0QjQq+ezC0lDUUfF4=', 'Content-Length': '0'} 2024-04-19T23:34:58.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b8dc803a4575213-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.557 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:58.557 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:58.558 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.558 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.558 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.558 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.558 DEBUG:boto:Data: 2024-04-19T23:34:58.558 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.558 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.558 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.558 DEBUG:boto:Params: {} 2024-04-19T23:34:58.558 DEBUG:boto:Token: None 2024-04-19T23:34:58.559 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.559 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QBI+j/uygooTIAA7p7vkNld4W38= 2024-04-19T23:34:58.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QBI+j/uygooTIAA7p7vkNld4W38=', 'Content-Length': '0'} 2024-04-19T23:34:58.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b6c79284f5ad321e-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.561 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:58.561 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:58.561 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.561 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.561 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.561 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.561 DEBUG:boto:Data: 2024-04-19T23:34:58.561 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.562 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.562 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.562 DEBUG:boto:Params: {} 2024-04-19T23:34:58.562 DEBUG:boto:Token: None 2024-04-19T23:34:58.562 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.562 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QBI+j/uygooTIAA7p7vkNld4W38= 2024-04-19T23:34:58.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QBI+j/uygooTIAA7p7vkNld4W38=', 'Content-Length': '0'} 2024-04-19T23:34:58.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006dddfe66db5bf7b6-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:34:58.565 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:58.565 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:58.565 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.565 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.565 DEBUG:boto:Method: GET 2024-04-19T23:34:58.565 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.565 DEBUG:boto:Data: 2024-04-19T23:34:58.565 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.565 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.566 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.566 DEBUG:boto:Params: {} 2024-04-19T23:34:58.566 DEBUG:boto:Token: None 2024-04-19T23:34:58.566 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.566 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yr8DQQiYFl0QjQq+ezC0lDUUfF4= 2024-04-19T23:34:58.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yr8DQQiYFl0QjQq+ezC0lDUUfF4=', 'Content-Length': '0'} 2024-04-19T23:34:58.568 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009323966f3e46ed33-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.568 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:58.568 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:58.568 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.568 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.568 DEBUG:boto:Method: GET 2024-04-19T23:34:58.569 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.569 DEBUG:boto:Data: 2024-04-19T23:34:58.569 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.569 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.569 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.569 DEBUG:boto:Params: {} 2024-04-19T23:34:58.569 DEBUG:boto:Token: None 2024-04-19T23:34:58.569 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.569 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yr8DQQiYFl0QjQq+ezC0lDUUfF4= 2024-04-19T23:34:58.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yr8DQQiYFl0QjQq+ezC0lDUUfF4=', 'Content-Length': '0'} 2024-04-19T23:34:58.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000286b1591fafe9ef5-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.571 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.572 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.572 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:58.572 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:58.572 DEBUG:boto:Method: GET 2024-04-19T23:34:58.572 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:34:58.572 DEBUG:boto:Data: 2024-04-19T23:34:58.572 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.572 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.572 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.572 DEBUG:boto:Params: {} 2024-04-19T23:34:58.572 DEBUG:boto:Token: None 2024-04-19T23:34:58.573 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.573 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052823e6f7f9a75f5-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.618 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.619 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.619 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.619 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:58.619 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:58.619 DEBUG:boto:Method: GET 2024-04-19T23:34:58.619 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:34:58.619 DEBUG:boto:Data: 2024-04-19T23:34:58.619 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.620 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.620 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.620 DEBUG:boto:Params: {} 2024-04-19T23:34:58.620 DEBUG:boto:Token: None 2024-04-19T23:34:58.620 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.620 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000046cfc760cb1a24c-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.623 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.623 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:58.624 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:58.624 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.624 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:34:58.624 DEBUG:boto:Data: 2024-04-19T23:34:58.624 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.624 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.624 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.624 DEBUG:boto:Params: {} 2024-04-19T23:34:58.624 DEBUG:boto:Token: None 2024-04-19T23:34:58.625 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/dummy 2024-04-19T23:34:58.625 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oqAx+NbX34KnAXlDZ57F5DMUCOI= 2024-04-19T23:34:58.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oqAx+NbX34KnAXlDZ57F5DMUCOI=', 'Content-Length': '0'} 2024-04-19T23:34:58.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aedde99f1ab4a584-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.627 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:58.627 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:58.628 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.628 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:34:58.628 DEBUG:boto:Data: 2024-04-19T23:34:58.628 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.628 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.628 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.628 DEBUG:boto:Params: {} 2024-04-19T23:34:58.628 DEBUG:boto:Token: None 2024-04-19T23:34:58.628 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/dummy 2024-04-19T23:34:58.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oqAx+NbX34KnAXlDZ57F5DMUCOI= 2024-04-19T23:34:58.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oqAx+NbX34KnAXlDZ57F5DMUCOI=', 'Content-Length': '0'} 2024-04-19T23:34:58.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d412c4c5e8219d41-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:34:58.631 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:58.631 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:58.631 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.631 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.631 DEBUG:boto:Method: GET 2024-04-19T23:34:58.631 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.632 DEBUG:boto:Data: 2024-04-19T23:34:58.632 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.632 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.632 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.632 DEBUG:boto:Params: {} 2024-04-19T23:34:58.632 DEBUG:boto:Token: None 2024-04-19T23:34:58.632 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yr8DQQiYFl0QjQq+ezC0lDUUfF4= 2024-04-19T23:34:58.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yr8DQQiYFl0QjQq+ezC0lDUUfF4=', 'Content-Length': '0'} 2024-04-19T23:34:58.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ca380a21217cc81-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.634 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:58.634 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:58.634 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.635 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.635 DEBUG:boto:Method: GET 2024-04-19T23:34:58.635 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.635 DEBUG:boto:Data: 2024-04-19T23:34:58.635 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.635 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.635 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.635 DEBUG:boto:Params: {} 2024-04-19T23:34:58.635 DEBUG:boto:Token: None 2024-04-19T23:34:58.636 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:58.636 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yr8DQQiYFl0QjQq+ezC0lDUUfF4= 2024-04-19T23:34:58.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yr8DQQiYFl0QjQq+ezC0lDUUfF4=', 'Content-Length': '0'} 2024-04-19T23:34:58.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a0681c30a487a93e-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.638 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.638 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.638 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:58.638 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:58.638 DEBUG:boto:Method: GET 2024-04-19T23:34:58.638 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:34:58.638 DEBUG:boto:Data: 2024-04-19T23:34:58.639 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.639 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.639 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.639 DEBUG:boto:Params: {} 2024-04-19T23:34:58.639 DEBUG:boto:Token: None 2024-04-19T23:34:58.639 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.639 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb04df780dc5cdb5-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.642 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.642 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.642 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.642 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:58.643 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:58.643 DEBUG:boto:Method: GET 2024-04-19T23:34:58.643 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:34:58.643 DEBUG:boto:Data: 2024-04-19T23:34:58.643 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.643 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.643 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.643 DEBUG:boto:Params: {} 2024-04-19T23:34:58.643 DEBUG:boto:Token: None 2024-04-19T23:34:58.644 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.645 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da08920c693a0d3f-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.648 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:58.648 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.648 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.648 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.648 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.649 DEBUG:boto:Method: GET 2024-04-19T23:34:58.649 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.649 DEBUG:boto:Data: 2024-04-19T23:34:58.649 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.649 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.649 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.649 DEBUG:boto:Params: {} 2024-04-19T23:34:58.649 DEBUG:boto:Token: None 2024-04-19T23:34:58.649 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.650 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sgvY1tFheFbAKWKz2fZ3VhLLmsQ= 2024-04-19T23:34:58.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sgvY1tFheFbAKWKz2fZ3VhLLmsQ=', 'Content-Length': '0'} 2024-04-19T23:34:58.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aec5301f6ca2c810-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.694 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.694 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.695 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.695 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.695 DEBUG:boto:Method: GET 2024-04-19T23:34:58.695 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.695 DEBUG:boto:Data: 2024-04-19T23:34:58.695 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.695 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.695 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.695 DEBUG:boto:Params: {} 2024-04-19T23:34:58.695 DEBUG:boto:Token: None 2024-04-19T23:34:58.696 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.696 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sgvY1tFheFbAKWKz2fZ3VhLLmsQ= 2024-04-19T23:34:58.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sgvY1tFheFbAKWKz2fZ3VhLLmsQ=', 'Content-Length': '0'} 2024-04-19T23:34:58.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c7e8fb86023507c-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.698 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.698 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.698 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.698 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.698 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.699 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.699 DEBUG:boto:Data: 2024-04-19T23:34:58.699 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.699 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.699 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.699 DEBUG:boto:Params: {} 2024-04-19T23:34:58.699 DEBUG:boto:Token: None 2024-04-19T23:34:58.699 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.699 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CUwR7SwKdQgnl/Cn85zY9zotHGc= 2024-04-19T23:34:58.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CUwR7SwKdQgnl/Cn85zY9zotHGc=', 'Content-Length': '0'} 2024-04-19T23:34:58.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5b300193ece2d80-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.702 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.702 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.702 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.702 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.702 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.702 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.702 DEBUG:boto:Data: 2024-04-19T23:34:58.702 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.702 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.702 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.703 DEBUG:boto:Params: {} 2024-04-19T23:34:58.703 DEBUG:boto:Token: None 2024-04-19T23:34:58.703 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.703 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CUwR7SwKdQgnl/Cn85zY9zotHGc= 2024-04-19T23:34:58.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CUwR7SwKdQgnl/Cn85zY9zotHGc=', 'Content-Length': '0'} 2024-04-19T23:34:58.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8dcd05f4e7792db-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:58.705 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.705 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.705 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.705 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.705 DEBUG:boto:Method: GET 2024-04-19T23:34:58.705 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.706 DEBUG:boto:Data: 2024-04-19T23:34:58.706 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.706 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.706 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.706 DEBUG:boto:Params: {} 2024-04-19T23:34:58.706 DEBUG:boto:Token: None 2024-04-19T23:34:58.706 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.706 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sgvY1tFheFbAKWKz2fZ3VhLLmsQ= 2024-04-19T23:34:58.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sgvY1tFheFbAKWKz2fZ3VhLLmsQ=', 'Content-Length': '0'} 2024-04-19T23:34:58.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a49657a9025387dd-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.750 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.750 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.750 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.751 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.751 DEBUG:boto:Method: GET 2024-04-19T23:34:58.751 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.751 DEBUG:boto:Data: 2024-04-19T23:34:58.751 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.751 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.751 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.751 DEBUG:boto:Params: {} 2024-04-19T23:34:58.751 DEBUG:boto:Token: None 2024-04-19T23:34:58.751 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.752 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sgvY1tFheFbAKWKz2fZ3VhLLmsQ= 2024-04-19T23:34:58.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sgvY1tFheFbAKWKz2fZ3VhLLmsQ=', 'Content-Length': '0'} 2024-04-19T23:34:58.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000420bc15e1b5a4983-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.754 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.754 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.754 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:58.754 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:58.754 DEBUG:boto:Method: GET 2024-04-19T23:34:58.754 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:58.754 DEBUG:boto:Data: 2024-04-19T23:34:58.754 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.755 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.755 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.755 DEBUG:boto:Params: {} 2024-04-19T23:34:58.755 DEBUG:boto:Token: None 2024-04-19T23:34:58.755 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.755 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000964d0eabe0caa76e-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.758 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.758 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.758 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.758 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:58.758 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:58.759 DEBUG:boto:Method: GET 2024-04-19T23:34:58.759 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:58.759 DEBUG:boto:Data: 2024-04-19T23:34:58.759 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.759 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.759 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.759 DEBUG:boto:Params: {} 2024-04-19T23:34:58.759 DEBUG:boto:Token: None 2024-04-19T23:34:58.759 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.759 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016ff3e0fc28a79dd-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.762 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.763 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.763 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.763 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.763 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:34:58.763 DEBUG:boto:Data: 2024-04-19T23:34:58.763 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.763 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.763 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.763 DEBUG:boto:Params: {} 2024-04-19T23:34:58.764 DEBUG:boto:Token: None 2024-04-19T23:34:58.764 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:34:58.764 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xYmTngEEiL+hMZTnuJgdjqJuzVg= 2024-04-19T23:34:58.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xYmTngEEiL+hMZTnuJgdjqJuzVg=', 'Content-Length': '0'} 2024-04-19T23:34:58.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063b2b6b34528173f-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.766 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.766 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.767 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.767 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:34:58.767 DEBUG:boto:Data: 2024-04-19T23:34:58.767 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.767 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.767 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.767 DEBUG:boto:Params: {} 2024-04-19T23:34:58.767 DEBUG:boto:Token: None 2024-04-19T23:34:58.767 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:34:58.767 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xYmTngEEiL+hMZTnuJgdjqJuzVg= 2024-04-19T23:34:58.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xYmTngEEiL+hMZTnuJgdjqJuzVg=', 'Content-Length': '0'} 2024-04-19T23:34:58.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000619f6f90ce66c5-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:58.770 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.770 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.770 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.770 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.771 DEBUG:boto:Method: GET 2024-04-19T23:34:58.771 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.771 DEBUG:boto:Data: 2024-04-19T23:34:58.771 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.771 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.771 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.771 DEBUG:boto:Params: {} 2024-04-19T23:34:58.771 DEBUG:boto:Token: None 2024-04-19T23:34:58.771 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.771 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sgvY1tFheFbAKWKz2fZ3VhLLmsQ= 2024-04-19T23:34:58.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sgvY1tFheFbAKWKz2fZ3VhLLmsQ=', 'Content-Length': '0'} 2024-04-19T23:34:58.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d5c8481948e9c69-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.814 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.814 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:58.814 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.815 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.815 DEBUG:boto:Method: GET 2024-04-19T23:34:58.815 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.815 DEBUG:boto:Data: 2024-04-19T23:34:58.815 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.815 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.815 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.815 DEBUG:boto:Params: {} 2024-04-19T23:34:58.815 DEBUG:boto:Token: None 2024-04-19T23:34:58.815 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:58.816 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sgvY1tFheFbAKWKz2fZ3VhLLmsQ= 2024-04-19T23:34:58.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sgvY1tFheFbAKWKz2fZ3VhLLmsQ=', 'Content-Length': '0'} 2024-04-19T23:34:58.818 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002cf317afe3649a47-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.818 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.818 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.818 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:58.819 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:58.819 DEBUG:boto:Method: GET 2024-04-19T23:34:58.819 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:58.819 DEBUG:boto:Data: 2024-04-19T23:34:58.819 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.819 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.819 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.819 DEBUG:boto:Params: {} 2024-04-19T23:34:58.819 DEBUG:boto:Token: None 2024-04-19T23:34:58.819 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.819 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ce4757ca22a00f4-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.822 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.822 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.823 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.823 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:58.823 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:58.823 DEBUG:boto:Method: GET 2024-04-19T23:34:58.823 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:58.823 DEBUG:boto:Data: 2024-04-19T23:34:58.823 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.823 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.823 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.823 DEBUG:boto:Params: {} 2024-04-19T23:34:58.824 DEBUG:boto:Token: None 2024-04-19T23:34:58.824 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c464a3d344efaded-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.826 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:58.827 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.827 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.827 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.827 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.827 DEBUG:boto:Method: GET 2024-04-19T23:34:58.827 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.828 DEBUG:boto:Data: 2024-04-19T23:34:58.828 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.828 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.828 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.828 DEBUG:boto:Params: {} 2024-04-19T23:34:58.828 DEBUG:boto:Token: None 2024-04-19T23:34:58.828 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.828 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54= 2024-04-19T23:34:58.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54=', 'Content-Length': '0'} 2024-04-19T23:34:58.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e1dd59458ed17be-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.830 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.831 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.831 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.831 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.831 DEBUG:boto:Method: GET 2024-04-19T23:34:58.831 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.831 DEBUG:boto:Data: 2024-04-19T23:34:58.831 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.831 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.831 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.831 DEBUG:boto:Params: {} 2024-04-19T23:34:58.832 DEBUG:boto:Token: None 2024-04-19T23:34:58.832 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.832 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54= 2024-04-19T23:34:58.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54=', 'Content-Length': '0'} 2024-04-19T23:34:58.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c01d822e222880d8-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.834 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.834 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.834 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.834 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.835 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.835 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.835 DEBUG:boto:Data: 2024-04-19T23:34:58.835 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.835 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.835 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.835 DEBUG:boto:Params: {} 2024-04-19T23:34:58.835 DEBUG:boto:Token: None 2024-04-19T23:34:58.835 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.835 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4r5B/cXFghjWu5BRkL1MIiPrEGI= 2024-04-19T23:34:58.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4r5B/cXFghjWu5BRkL1MIiPrEGI=', 'Content-Length': '0'} 2024-04-19T23:34:58.837 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000569abbc9b6d405d2-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.838 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.838 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.838 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.838 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.838 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.838 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.838 DEBUG:boto:Data: 2024-04-19T23:34:58.838 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.839 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.839 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.839 DEBUG:boto:Params: {} 2024-04-19T23:34:58.839 DEBUG:boto:Token: None 2024-04-19T23:34:58.839 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.839 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4r5B/cXFghjWu5BRkL1MIiPrEGI= 2024-04-19T23:34:58.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4r5B/cXFghjWu5BRkL1MIiPrEGI=', 'Content-Length': '0'} 2024-04-19T23:34:58.841 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8a218819168c8a1-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:58.842 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.842 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.842 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.842 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.842 DEBUG:boto:Method: GET 2024-04-19T23:34:58.842 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.843 DEBUG:boto:Data: 2024-04-19T23:34:58.843 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.843 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.843 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.843 DEBUG:boto:Params: {} 2024-04-19T23:34:58.843 DEBUG:boto:Token: None 2024-04-19T23:34:58.843 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.843 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54= 2024-04-19T23:34:58.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54=', 'Content-Length': '0'} 2024-04-19T23:34:58.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049ed788be355cee5-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.886 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.887 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.887 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.887 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.887 DEBUG:boto:Method: GET 2024-04-19T23:34:58.887 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.887 DEBUG:boto:Data: 2024-04-19T23:34:58.887 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.887 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.887 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.887 DEBUG:boto:Params: {} 2024-04-19T23:34:58.888 DEBUG:boto:Token: None 2024-04-19T23:34:58.888 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.888 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54= 2024-04-19T23:34:58.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54=', 'Content-Length': '0'} 2024-04-19T23:34:58.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b886ef38e36e1bf1-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.930 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.931 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.931 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.931 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.931 DEBUG:boto:Method: GET 2024-04-19T23:34:58.931 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.931 DEBUG:boto:Data: 2024-04-19T23:34:58.931 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.931 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.931 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.932 DEBUG:boto:Params: {} 2024-04-19T23:34:58.932 DEBUG:boto:Token: None 2024-04-19T23:34:58.932 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.932 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0f66f2a4221a47e-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.935 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.935 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:58.935 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:58.935 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.936 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.936 DEBUG:boto:Method: GET 2024-04-19T23:34:58.936 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:58.936 DEBUG:boto:Data: 2024-04-19T23:34:58.936 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.936 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.936 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.936 DEBUG:boto:Params: {} 2024-04-19T23:34:58.936 DEBUG:boto:Token: None 2024-04-19T23:34:58.936 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/ 2024-04-19T23:34:58.937 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU= 2024-04-19T23:34:58.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8xuWCNtyTOjIdTofY4mr5tDbJeU=', 'Content-Length': '0'} 2024-04-19T23:34:58.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebbc685e8de24ac1-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.939 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:58.940 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.940 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.940 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.940 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:34:58.940 DEBUG:boto:Data: 2024-04-19T23:34:58.940 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.940 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.940 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.940 DEBUG:boto:Params: {} 2024-04-19T23:34:58.941 DEBUG:boto:Token: None 2024-04-19T23:34:58.941 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:34:58.941 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SVIusA3St/sDpEmUQIXJtobhInI= 2024-04-19T23:34:58.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SVIusA3St/sDpEmUQIXJtobhInI=', 'Content-Length': '0'} 2024-04-19T23:34:58.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3a5c0505f55e04b-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.943 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.943 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.944 DEBUG:boto:Method: HEAD 2024-04-19T23:34:58.944 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:34:58.944 DEBUG:boto:Data: 2024-04-19T23:34:58.944 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.944 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.944 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.944 DEBUG:boto:Params: {} 2024-04-19T23:34:58.944 DEBUG:boto:Token: None 2024-04-19T23:34:58.944 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:34:58.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SVIusA3St/sDpEmUQIXJtobhInI= 2024-04-19T23:34:58.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SVIusA3St/sDpEmUQIXJtobhInI=', 'Content-Length': '0'} 2024-04-19T23:34:58.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5c7b1e85bc6806b-006622ffa2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:58.947 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.948 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.948 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.948 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.948 DEBUG:boto:Method: GET 2024-04-19T23:34:58.948 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.948 DEBUG:boto:Data: 2024-04-19T23:34:58.948 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.948 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:58.948 DEBUG:boto:Port: 8001 2024-04-19T23:34:58.948 DEBUG:boto:Params: {} 2024-04-19T23:34:58.949 DEBUG:boto:Token: None 2024-04-19T23:34:58.949 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:58 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.949 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54= 2024-04-19T23:34:58.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+N0ShEu6n78cOwWlO9jKD7WaH54=', 'Content-Length': '0'} 2024-04-19T23:34:58.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077a1e6efedfb5454-006622ffa2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:58.998 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.999 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:58.999 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.999 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.999 DEBUG:boto:Method: GET 2024-04-19T23:34:58.999 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:58.999 DEBUG:boto:Data: 2024-04-19T23:34:58.999 DEBUG:boto:Headers: {} 2024-04-19T23:34:58.999 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:58.999 DEBUG:boto:Port: 8000 2024-04-19T23:34:58.999 DEBUG:boto:Params: {} 2024-04-19T23:34:59.000 DEBUG:boto:Token: None 2024-04-19T23:34:59.000 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.000 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zrWGwxkQF2tF5SFPyZmV7WB24PA= 2024-04-19T23:34:59.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zrWGwxkQF2tF5SFPyZmV7WB24PA=', 'Content-Length': '0'} 2024-04-19T23:34:59.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c81f0d31bcbe478-006622ffa3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.042 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.043 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.043 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:59.043 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:59.043 DEBUG:boto:Method: GET 2024-04-19T23:34:59.043 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:59.043 DEBUG:boto:Data: 2024-04-19T23:34:59.043 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.043 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.043 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.044 DEBUG:boto:Params: {} 2024-04-19T23:34:59.044 DEBUG:boto:Token: None 2024-04-19T23:34:59.044 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.044 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f660fabfd85c8a5b-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.090 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.091 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.091 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.091 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:59.091 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:59.091 DEBUG:boto:Method: GET 2024-04-19T23:34:59.091 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:59.091 DEBUG:boto:Data: 2024-04-19T23:34:59.091 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.092 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:59.092 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.092 DEBUG:boto:Params: {} 2024-04-19T23:34:59.092 DEBUG:boto:Token: None 2024-04-19T23:34:59.092 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.092 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6acb7480fba918f-006622ffa3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.095 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:34:59.095 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.095 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.095 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.095 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.096 DEBUG:boto:Method: GET 2024-04-19T23:34:59.096 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.096 DEBUG:boto:Data: 2024-04-19T23:34:59.096 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.096 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.096 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.096 DEBUG:boto:Params: {} 2024-04-19T23:34:59.096 DEBUG:boto:Token: None 2024-04-19T23:34:59.096 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.096 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8A5GXF4ErSZZqqR5WjbMxUJ53Ps= 2024-04-19T23:34:59.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8A5GXF4ErSZZqqR5WjbMxUJ53Ps=', 'Content-Length': '0'} 2024-04-19T23:34:59.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015203811980180e0-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.098 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.099 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.099 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.099 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.099 DEBUG:boto:Method: GET 2024-04-19T23:34:59.099 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.099 DEBUG:boto:Data: 2024-04-19T23:34:59.099 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.099 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:59.099 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.099 DEBUG:boto:Params: {} 2024-04-19T23:34:59.100 DEBUG:boto:Token: None 2024-04-19T23:34:59.100 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.100 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8A5GXF4ErSZZqqR5WjbMxUJ53Ps= 2024-04-19T23:34:59.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8A5GXF4ErSZZqqR5WjbMxUJ53Ps=', 'Content-Length': '0'} 2024-04-19T23:34:59.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000004a15b0fe83d876-006622ffa3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.102 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.102 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.102 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.102 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.102 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.102 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.103 DEBUG:boto:Data: 2024-04-19T23:34:59.103 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.103 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.103 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.103 DEBUG:boto:Params: {} 2024-04-19T23:34:59.103 DEBUG:boto:Token: None 2024-04-19T23:34:59.103 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.103 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0eMPysxIg0qAOHtCaGFqpBu5yMM= 2024-04-19T23:34:59.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0eMPysxIg0qAOHtCaGFqpBu5yMM=', 'Content-Length': '0'} 2024-04-19T23:34:59.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e1b89fe2b932d8a-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.105 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.106 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.106 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.106 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.106 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.106 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.106 DEBUG:boto:Data: 2024-04-19T23:34:59.106 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.106 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:59.106 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.106 DEBUG:boto:Params: {} 2024-04-19T23:34:59.107 DEBUG:boto:Token: None 2024-04-19T23:34:59.107 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.107 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0eMPysxIg0qAOHtCaGFqpBu5yMM= 2024-04-19T23:34:59.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0eMPysxIg0qAOHtCaGFqpBu5yMM=', 'Content-Length': '0'} 2024-04-19T23:34:59.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000585c46e1c2f3848f-006622ffa3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:34:59.109 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.109 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.109 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.109 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.109 DEBUG:boto:Method: GET 2024-04-19T23:34:59.110 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.110 DEBUG:boto:Data: 2024-04-19T23:34:59.110 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.110 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.110 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.110 DEBUG:boto:Params: {} 2024-04-19T23:34:59.110 DEBUG:boto:Token: None 2024-04-19T23:34:59.110 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.110 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8A5GXF4ErSZZqqR5WjbMxUJ53Ps= 2024-04-19T23:34:59.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8A5GXF4ErSZZqqR5WjbMxUJ53Ps=', 'Content-Length': '0'} 2024-04-19T23:34:59.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000635559746373e5a9-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.113 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.113 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.113 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.113 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.113 DEBUG:boto:Method: GET 2024-04-19T23:34:59.113 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.113 DEBUG:boto:Data: 2024-04-19T23:34:59.113 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.113 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:59.113 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.113 DEBUG:boto:Params: {} 2024-04-19T23:34:59.114 DEBUG:boto:Token: None 2024-04-19T23:34:59.114 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.114 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8A5GXF4ErSZZqqR5WjbMxUJ53Ps= 2024-04-19T23:34:59.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8A5GXF4ErSZZqqR5WjbMxUJ53Ps=', 'Content-Length': '0'} 2024-04-19T23:34:59.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095cb4b2cd43e2733-006622ffa3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.158 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.158 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.159 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:59.159 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:59.159 DEBUG:boto:Method: GET 2024-04-19T23:34:59.159 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:59.159 DEBUG:boto:Data: 2024-04-19T23:34:59.159 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.159 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.159 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.159 DEBUG:boto:Params: {} 2024-04-19T23:34:59.159 DEBUG:boto:Token: None 2024-04-19T23:34:59.160 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000385cef4a59510825-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.162 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.163 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.163 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.163 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:59.163 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:59.163 DEBUG:boto:Method: GET 2024-04-19T23:34:59.163 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:59.163 DEBUG:boto:Data: 2024-04-19T23:34:59.164 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.164 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:59.164 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.164 DEBUG:boto:Params: {} 2024-04-19T23:34:59.164 DEBUG:boto:Token: None 2024-04-19T23:34:59.164 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.164 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003683bce9abc114db-006622ffa3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.210 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.211 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.211 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.211 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.211 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:34:59.211 DEBUG:boto:Data: 2024-04-19T23:34:59.211 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.211 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.211 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.212 DEBUG:boto:Params: {} 2024-04-19T23:34:59.212 DEBUG:boto:Token: None 2024-04-19T23:34:59.212 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:34:59.212 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JXRJlk7PRmXdAL0KhxaeVDZeJ3g= 2024-04-19T23:34:59.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JXRJlk7PRmXdAL0KhxaeVDZeJ3g=', 'Content-Length': '0'} 2024-04-19T23:34:59.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db190505a2d0f6f6-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.214 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.215 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.215 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.215 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:34:59.215 DEBUG:boto:Data: 2024-04-19T23:34:59.215 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.215 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:59.215 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.215 DEBUG:boto:Params: {} 2024-04-19T23:34:59.215 DEBUG:boto:Token: None 2024-04-19T23:34:59.215 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:34:59.216 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JXRJlk7PRmXdAL0KhxaeVDZeJ3g= 2024-04-19T23:34:59.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JXRJlk7PRmXdAL0KhxaeVDZeJ3g=', 'Content-Length': '0'} 2024-04-19T23:34:59.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4b83c19e4052204-006622ffa3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:34:59.218 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.218 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.219 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.219 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.219 DEBUG:boto:Method: GET 2024-04-19T23:34:59.219 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.219 DEBUG:boto:Data: 2024-04-19T23:34:59.219 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.219 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.219 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.219 DEBUG:boto:Params: {} 2024-04-19T23:34:59.219 DEBUG:boto:Token: None 2024-04-19T23:34:59.220 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.220 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8A5GXF4ErSZZqqR5WjbMxUJ53Ps= 2024-04-19T23:34:59.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8A5GXF4ErSZZqqR5WjbMxUJ53Ps=', 'Content-Length': '0'} 2024-04-19T23:34:59.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7d659e373f8c3bb-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.262 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.263 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:34:59.263 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.263 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.263 DEBUG:boto:Method: GET 2024-04-19T23:34:59.263 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.263 DEBUG:boto:Data: 2024-04-19T23:34:59.263 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.263 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:59.263 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.263 DEBUG:boto:Params: {} 2024-04-19T23:34:59.264 DEBUG:boto:Token: None 2024-04-19T23:34:59.264 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:34:59.264 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8A5GXF4ErSZZqqR5WjbMxUJ53Ps= 2024-04-19T23:34:59.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8A5GXF4ErSZZqqR5WjbMxUJ53Ps=', 'Content-Length': '0'} 2024-04-19T23:34:59.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed8279876beb3d6d-006622ffa3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.310 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.310 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.311 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:59.311 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:59.311 DEBUG:boto:Method: GET 2024-04-19T23:34:59.311 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:59.311 DEBUG:boto:Data: 2024-04-19T23:34:59.311 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.311 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.311 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.311 DEBUG:boto:Params: {} 2024-04-19T23:34:59.311 DEBUG:boto:Token: None 2024-04-19T23:34:59.312 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.312 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b5a46723a5f98e-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.315 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.316 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.316 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.316 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:59.316 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:59.316 DEBUG:boto:Method: GET 2024-04-19T23:34:59.316 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:34:59.316 DEBUG:boto:Data: 2024-04-19T23:34:59.316 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.317 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:34:59.317 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.317 DEBUG:boto:Params: {} 2024-04-19T23:34:59.317 DEBUG:boto:Token: None 2024-04-19T23:34:59.317 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.317 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001433f7ceeed4e535-006622ffa3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.320 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.320 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-19 zones={test-zone3, test-zone1} 2024-04-19T23:34:59.320 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-19 zones={test-zone3, test-zone2} 2024-04-19T23:34:59.320 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.320 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.321 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.321 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:34:59.321 DEBUG:boto:Data: 2024-04-19T23:34:59.321 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.321 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.321 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.321 DEBUG:boto:Params: {} 2024-04-19T23:34:59.321 DEBUG:boto:Token: None 2024-04-19T23:34:59.321 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.321 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NI5ZkMCNDYNBsIAFd1qEwPxKXVI= 2024-04-19T23:34:59.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NI5ZkMCNDYNBsIAFd1qEwPxKXVI=', 'Content-Length': '0'} 2024-04-19T23:34:59.328 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000beb15e236274318d-006622ffa3-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.329 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.329 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.329 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.329 DEBUG:boto:Path: /aktlji-19/ 2024-04-19T23:34:59.329 DEBUG:boto:Data: 2024-04-19T23:34:59.329 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.329 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.329 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.329 DEBUG:boto:Params: {} 2024-04-19T23:34:59.330 DEBUG:boto:Token: None 2024-04-19T23:34:59.330 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.330 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NI5ZkMCNDYNBsIAFd1qEwPxKXVI= 2024-04-19T23:34:59.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NI5ZkMCNDYNBsIAFd1qEwPxKXVI=', 'Content-Length': '0'} 2024-04-19T23:34:59.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff417714c8e00308-006622ffa3-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.332 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:34:59.332 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.333 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.333 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:59.333 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:59.333 DEBUG:boto:Method: GET 2024-04-19T23:34:59.333 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:34:59.333 DEBUG:boto:Data: 2024-04-19T23:34:59.333 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.333 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.333 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.333 DEBUG:boto:Params: {} 2024-04-19T23:34:59.333 DEBUG:boto:Token: None 2024-04-19T23:34:59.334 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/?versions 2024-04-19T23:34:59.334 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eKDs3LHCBkNYbJqG6dpe3qsuzPU= 2024-04-19T23:34:59.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eKDs3LHCBkNYbJqG6dpe3qsuzPU=', 'Content-Length': '0'} 2024-04-19T23:34:59.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c608fd09d81e42c-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.341 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.342 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:34:59.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:34:59.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:34:59.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:34:59.342 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:34:59.342 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.342 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.343 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:59.343 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:59.343 DEBUG:boto:Method: GET 2024-04-19T23:34:59.343 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:34:59.343 DEBUG:boto:Data: 2024-04-19T23:34:59.343 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.343 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.343 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.343 DEBUG:boto:Params: {} 2024-04-19T23:34:59.343 DEBUG:boto:Token: None 2024-04-19T23:34:59.343 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/?versions 2024-04-19T23:34:59.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eKDs3LHCBkNYbJqG6dpe3qsuzPU= 2024-04-19T23:34:59.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eKDs3LHCBkNYbJqG6dpe3qsuzPU=', 'Content-Length': '0'} 2024-04-19T23:34:59.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009372343b0fbfacd6-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.347 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.347 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:34:59.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:34:59.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:34:59.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:34:59.348 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.348 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.348 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:59.348 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:59.348 DEBUG:boto:Method: GET 2024-04-19T23:34:59.348 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:34:59.348 DEBUG:boto:Data: 2024-04-19T23:34:59.348 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.348 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.348 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.349 DEBUG:boto:Params: {} 2024-04-19T23:34:59.349 DEBUG:boto:Token: None 2024-04-19T23:34:59.349 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/?versions 2024-04-19T23:34:59.349 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eKDs3LHCBkNYbJqG6dpe3qsuzPU= 2024-04-19T23:34:59.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eKDs3LHCBkNYbJqG6dpe3qsuzPU=', 'Content-Length': '0'} 2024-04-19T23:34:59.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4f1f897a02a2915-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.360 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.360 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.361 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.361 DEBUG:boto:path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:59.361 DEBUG:boto:auth_path=/aktlji-19/?versions&max-keys=999 2024-04-19T23:34:59.361 DEBUG:boto:Method: GET 2024-04-19T23:34:59.361 DEBUG:boto:Path: /aktlji-19/?versions&max-keys=999 2024-04-19T23:34:59.361 DEBUG:boto:Data: 2024-04-19T23:34:59.361 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.361 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.361 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.361 DEBUG:boto:Params: {} 2024-04-19T23:34:59.361 DEBUG:boto:Token: None 2024-04-19T23:34:59.362 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/?versions 2024-04-19T23:34:59.362 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eKDs3LHCBkNYbJqG6dpe3qsuzPU= 2024-04-19T23:34:59.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eKDs3LHCBkNYbJqG6dpe3qsuzPU=', 'Content-Length': '0'} 2024-04-19T23:34:59.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006efc541b33fbdae3-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.364 DEBUG:boto:b'aktlji-19999falsedummyWqqT8T6GwFqmT0V6GqLrEzuk7sXzBYWtrue2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone18u7XEl.hcooeKX51sIY3frv2y4sjRMTtrue2024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone29bH.ePrHLUsvGAMlzHPsjIg2u6W.fQvtrue2024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3FJ.cw-LTzcm5kWtF-seTx9qRriifaimtrue2024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:34:59.365 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:59.365 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:59.365 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.365 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.365 DEBUG:boto:Method: GET 2024-04-19T23:34:59.365 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.366 DEBUG:boto:Data: 2024-04-19T23:34:59.366 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.366 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.366 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.366 DEBUG:boto:Params: {} 2024-04-19T23:34:59.366 DEBUG:boto:Token: None 2024-04-19T23:34:59.366 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.366 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IJlFLPcaKsGjSJxd7n8sWmSTVyg= 2024-04-19T23:34:59.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IJlFLPcaKsGjSJxd7n8sWmSTVyg=', 'Content-Length': '0'} 2024-04-19T23:34:59.369 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec21309e3d7cac03-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.369 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:59.369 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:59.369 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.369 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.369 DEBUG:boto:Method: GET 2024-04-19T23:34:59.369 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.369 DEBUG:boto:Data: 2024-04-19T23:34:59.370 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.370 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.370 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.370 DEBUG:boto:Params: {} 2024-04-19T23:34:59.370 DEBUG:boto:Token: None 2024-04-19T23:34:59.370 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.370 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IJlFLPcaKsGjSJxd7n8sWmSTVyg= 2024-04-19T23:34:59.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IJlFLPcaKsGjSJxd7n8sWmSTVyg=', 'Content-Length': '0'} 2024-04-19T23:34:59.372 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005cecb58afd540117-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.373 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:59.373 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:59.373 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.373 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.373 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.373 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.373 DEBUG:boto:Data: 2024-04-19T23:34:59.373 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.373 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.374 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.374 DEBUG:boto:Params: {} 2024-04-19T23:34:59.374 DEBUG:boto:Token: None 2024-04-19T23:34:59.374 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.374 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l7dufhJlxS66nTjlNjZFbImLRhY= 2024-04-19T23:34:59.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l7dufhJlxS66nTjlNjZFbImLRhY=', 'Content-Length': '0'} 2024-04-19T23:34:59.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000904f8f696d64d090-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.377 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:59.377 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:59.377 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.377 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.377 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.377 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.377 DEBUG:boto:Data: 2024-04-19T23:34:59.378 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.378 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.378 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.378 DEBUG:boto:Params: {} 2024-04-19T23:34:59.378 DEBUG:boto:Token: None 2024-04-19T23:34:59.378 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.378 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l7dufhJlxS66nTjlNjZFbImLRhY= 2024-04-19T23:34:59.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l7dufhJlxS66nTjlNjZFbImLRhY=', 'Content-Length': '0'} 2024-04-19T23:34:59.381 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a7bcc0ed134bc4c0-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:34:59.382 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:59.382 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:59.382 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.382 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.382 DEBUG:boto:Method: GET 2024-04-19T23:34:59.382 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.382 DEBUG:boto:Data: 2024-04-19T23:34:59.383 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.383 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.383 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.383 DEBUG:boto:Params: {} 2024-04-19T23:34:59.383 DEBUG:boto:Token: None 2024-04-19T23:34:59.383 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.383 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IJlFLPcaKsGjSJxd7n8sWmSTVyg= 2024-04-19T23:34:59.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IJlFLPcaKsGjSJxd7n8sWmSTVyg=', 'Content-Length': '0'} 2024-04-19T23:34:59.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009912aa3dcc85587a-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.389 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:59.389 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:59.389 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.389 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.389 DEBUG:boto:Method: GET 2024-04-19T23:34:59.389 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.389 DEBUG:boto:Data: 2024-04-19T23:34:59.389 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.389 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.390 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.390 DEBUG:boto:Params: {} 2024-04-19T23:34:59.390 DEBUG:boto:Token: None 2024-04-19T23:34:59.390 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.390 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IJlFLPcaKsGjSJxd7n8sWmSTVyg= 2024-04-19T23:34:59.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IJlFLPcaKsGjSJxd7n8sWmSTVyg=', 'Content-Length': '0'} 2024-04-19T23:34:59.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a9d2c6a672975b03-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.392 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.392 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.393 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:59.393 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:59.393 DEBUG:boto:Method: GET 2024-04-19T23:34:59.393 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:34:59.393 DEBUG:boto:Data: 2024-04-19T23:34:59.393 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.393 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.393 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.393 DEBUG:boto:Params: {} 2024-04-19T23:34:59.393 DEBUG:boto:Token: None 2024-04-19T23:34:59.393 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.394 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f706aab2d14ac043-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.442 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.447 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.447 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.447 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:59.447 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:59.447 DEBUG:boto:Method: GET 2024-04-19T23:34:59.447 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:34:59.447 DEBUG:boto:Data: 2024-04-19T23:34:59.447 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.447 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.448 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.448 DEBUG:boto:Params: {} 2024-04-19T23:34:59.448 DEBUG:boto:Token: None 2024-04-19T23:34:59.448 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.448 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029b37a1c655e8fcf-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.451 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.452 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:59.452 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:59.452 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.452 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:34:59.452 DEBUG:boto:Data: 2024-04-19T23:34:59.452 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.452 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.452 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.452 DEBUG:boto:Params: {} 2024-04-19T23:34:59.452 DEBUG:boto:Token: None 2024-04-19T23:34:59.453 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/dummy 2024-04-19T23:34:59.453 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3MkAmKiTpX6F6QuuxkEd6kvlj/g= 2024-04-19T23:34:59.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3MkAmKiTpX6F6QuuxkEd6kvlj/g=', 'Content-Length': '0'} 2024-04-19T23:34:59.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003bae340fe4fffd23-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.455 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:59.455 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:59.456 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.456 DEBUG:boto:Path: /aktlji-19/dummy 2024-04-19T23:34:59.456 DEBUG:boto:Data: 2024-04-19T23:34:59.456 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.456 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.456 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.456 DEBUG:boto:Params: {} 2024-04-19T23:34:59.456 DEBUG:boto:Token: None 2024-04-19T23:34:59.456 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/dummy 2024-04-19T23:34:59.457 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3MkAmKiTpX6F6QuuxkEd6kvlj/g= 2024-04-19T23:34:59.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3MkAmKiTpX6F6QuuxkEd6kvlj/g=', 'Content-Length': '0'} 2024-04-19T23:34:59.459 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8fd1607db71f5c9-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:34:59.459 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:59.459 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:59.460 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.460 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.460 DEBUG:boto:Method: GET 2024-04-19T23:34:59.460 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.460 DEBUG:boto:Data: 2024-04-19T23:34:59.460 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.460 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.460 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.460 DEBUG:boto:Params: {} 2024-04-19T23:34:59.460 DEBUG:boto:Token: None 2024-04-19T23:34:59.461 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.461 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IJlFLPcaKsGjSJxd7n8sWmSTVyg= 2024-04-19T23:34:59.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IJlFLPcaKsGjSJxd7n8sWmSTVyg=', 'Content-Length': '0'} 2024-04-19T23:34:59.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fac4e49abde9392a-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.466 DEBUG:boto:path=/aktlji-19/dummy 2024-04-19T23:34:59.466 DEBUG:boto:auth_path=/aktlji-19/dummy 2024-04-19T23:34:59.466 DEBUG:boto:path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.466 DEBUG:boto:auth_path=/aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.467 DEBUG:boto:Method: GET 2024-04-19T23:34:59.467 DEBUG:boto:Path: /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.467 DEBUG:boto:Data: 2024-04-19T23:34:59.467 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.467 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.467 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.467 DEBUG:boto:Params: {} 2024-04-19T23:34:59.467 DEBUG:boto:Token: None 2024-04-19T23:34:59.467 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/dummy?versionId=WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW 2024-04-19T23:34:59.468 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IJlFLPcaKsGjSJxd7n8sWmSTVyg= 2024-04-19T23:34:59.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IJlFLPcaKsGjSJxd7n8sWmSTVyg=', 'Content-Length': '0'} 2024-04-19T23:34:59.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:54 GMT'), ('x-amz-version-id', 'WqqT8T6GwFqmT0V6GqLrEzuk7sXzBYW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006fe6b7a8e8ae4ce0-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.469 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.470 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.470 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:59.470 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:59.470 DEBUG:boto:Method: GET 2024-04-19T23:34:59.470 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:34:59.470 DEBUG:boto:Data: 2024-04-19T23:34:59.470 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.470 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.470 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.470 DEBUG:boto:Params: {} 2024-04-19T23:34:59.471 DEBUG:boto:Token: None 2024-04-19T23:34:59.471 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.471 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006606c18772fdf890-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.514 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.515 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.515 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.515 DEBUG:boto:path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:59.515 DEBUG:boto:auth_path=/aktlji-19/?prefix=dummy 2024-04-19T23:34:59.515 DEBUG:boto:Method: GET 2024-04-19T23:34:59.515 DEBUG:boto:Path: /aktlji-19/?prefix=dummy 2024-04-19T23:34:59.515 DEBUG:boto:Data: 2024-04-19T23:34:59.515 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.516 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.516 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.516 DEBUG:boto:Params: {} 2024-04-19T23:34:59.516 DEBUG:boto:Token: None 2024-04-19T23:34:59.516 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.516 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d83d51d9629f63aa-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.519 DEBUG:boto:b'aktlji-19dummy1000falsedummy2024-04-19T23:31:54.450Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:59.519 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.519 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.519 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.520 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.520 DEBUG:boto:Method: GET 2024-04-19T23:34:59.520 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.520 DEBUG:boto:Data: 2024-04-19T23:34:59.520 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.520 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.520 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.520 DEBUG:boto:Params: {} 2024-04-19T23:34:59.520 DEBUG:boto:Token: None 2024-04-19T23:34:59.520 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.521 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:By3FsB/jeghB3oBgkRW2CT0jDLs= 2024-04-19T23:34:59.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:By3FsB/jeghB3oBgkRW2CT0jDLs=', 'Content-Length': '0'} 2024-04-19T23:34:59.523 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000eba4530133fe0a1-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.570 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.570 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.571 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.571 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.571 DEBUG:boto:Method: GET 2024-04-19T23:34:59.571 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.571 DEBUG:boto:Data: 2024-04-19T23:34:59.571 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.571 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.571 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.571 DEBUG:boto:Params: {} 2024-04-19T23:34:59.571 DEBUG:boto:Token: None 2024-04-19T23:34:59.572 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.572 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:By3FsB/jeghB3oBgkRW2CT0jDLs= 2024-04-19T23:34:59.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:By3FsB/jeghB3oBgkRW2CT0jDLs=', 'Content-Length': '0'} 2024-04-19T23:34:59.574 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000baaf4a5829b993c3-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.574 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.575 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.575 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.575 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.575 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.575 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.575 DEBUG:boto:Data: 2024-04-19T23:34:59.575 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.576 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.576 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.576 DEBUG:boto:Params: {} 2024-04-19T23:34:59.576 DEBUG:boto:Token: None 2024-04-19T23:34:59.576 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.576 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+8ta0HmDWLWVWLyeVJejQyDa3nI= 2024-04-19T23:34:59.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+8ta0HmDWLWVWLyeVJejQyDa3nI=', 'Content-Length': '0'} 2024-04-19T23:34:59.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b857ca9eb612b93d-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.579 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.579 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.579 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.579 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.579 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.579 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.579 DEBUG:boto:Data: 2024-04-19T23:34:59.579 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.579 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.580 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.580 DEBUG:boto:Params: {} 2024-04-19T23:34:59.580 DEBUG:boto:Token: None 2024-04-19T23:34:59.580 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.580 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+8ta0HmDWLWVWLyeVJejQyDa3nI= 2024-04-19T23:34:59.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+8ta0HmDWLWVWLyeVJejQyDa3nI=', 'Content-Length': '0'} 2024-04-19T23:34:59.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdf3c1fd24bea267-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:59.582 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.582 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.583 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.583 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.583 DEBUG:boto:Method: GET 2024-04-19T23:34:59.583 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.583 DEBUG:boto:Data: 2024-04-19T23:34:59.583 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.583 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.583 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.583 DEBUG:boto:Params: {} 2024-04-19T23:34:59.584 DEBUG:boto:Token: None 2024-04-19T23:34:59.584 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.584 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:By3FsB/jeghB3oBgkRW2CT0jDLs= 2024-04-19T23:34:59.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:By3FsB/jeghB3oBgkRW2CT0jDLs=', 'Content-Length': '0'} 2024-04-19T23:34:59.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e007f1f9d322568d-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.626 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.627 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.627 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.627 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.627 DEBUG:boto:Method: GET 2024-04-19T23:34:59.627 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.627 DEBUG:boto:Data: 2024-04-19T23:34:59.627 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.627 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.627 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.628 DEBUG:boto:Params: {} 2024-04-19T23:34:59.628 DEBUG:boto:Token: None 2024-04-19T23:34:59.628 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:By3FsB/jeghB3oBgkRW2CT0jDLs= 2024-04-19T23:34:59.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:By3FsB/jeghB3oBgkRW2CT0jDLs=', 'Content-Length': '0'} 2024-04-19T23:34:59.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058a90cc2a266dae8-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.670 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.670 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.671 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:59.671 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:59.671 DEBUG:boto:Method: GET 2024-04-19T23:34:59.671 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:59.671 DEBUG:boto:Data: 2024-04-19T23:34:59.671 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.671 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.671 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.671 DEBUG:boto:Params: {} 2024-04-19T23:34:59.671 DEBUG:boto:Token: None 2024-04-19T23:34:59.672 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.672 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d5e134f814a31f-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.674 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.675 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.675 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.675 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:59.675 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:59.675 DEBUG:boto:Method: GET 2024-04-19T23:34:59.675 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:59.675 DEBUG:boto:Data: 2024-04-19T23:34:59.676 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.676 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.676 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.676 DEBUG:boto:Params: {} 2024-04-19T23:34:59.676 DEBUG:boto:Token: None 2024-04-19T23:34:59.676 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d722a162689b7b22-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.679 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.680 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.680 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.680 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.680 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:34:59.680 DEBUG:boto:Data: 2024-04-19T23:34:59.680 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.680 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.680 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.681 DEBUG:boto:Params: {} 2024-04-19T23:34:59.681 DEBUG:boto:Token: None 2024-04-19T23:34:59.681 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:34:59.681 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PSUdBWYowh1AULB2TSm/MuPBGaQ= 2024-04-19T23:34:59.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PSUdBWYowh1AULB2TSm/MuPBGaQ=', 'Content-Length': '0'} 2024-04-19T23:34:59.683 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000325dddbb9799c1a8-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.684 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.684 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.684 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.684 DEBUG:boto:Path: /aktlji-19/obj-test-zone1 2024-04-19T23:34:59.684 DEBUG:boto:Data: 2024-04-19T23:34:59.684 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.684 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.684 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.684 DEBUG:boto:Params: {} 2024-04-19T23:34:59.685 DEBUG:boto:Token: None 2024-04-19T23:34:59.685 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone1 2024-04-19T23:34:59.685 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PSUdBWYowh1AULB2TSm/MuPBGaQ= 2024-04-19T23:34:59.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PSUdBWYowh1AULB2TSm/MuPBGaQ=', 'Content-Length': '0'} 2024-04-19T23:34:59.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1d85bf9ff9e3777-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:34:59.688 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.688 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.688 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.688 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.688 DEBUG:boto:Method: GET 2024-04-19T23:34:59.688 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.689 DEBUG:boto:Data: 2024-04-19T23:34:59.689 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.689 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.689 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.689 DEBUG:boto:Params: {} 2024-04-19T23:34:59.689 DEBUG:boto:Token: None 2024-04-19T23:34:59.689 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.689 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:By3FsB/jeghB3oBgkRW2CT0jDLs= 2024-04-19T23:34:59.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:By3FsB/jeghB3oBgkRW2CT0jDLs=', 'Content-Length': '0'} 2024-04-19T23:34:59.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d0242372a8b00f4-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.734 DEBUG:boto:path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.735 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1 2024-04-19T23:34:59.735 DEBUG:boto:path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.735 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.735 DEBUG:boto:Method: GET 2024-04-19T23:34:59.735 DEBUG:boto:Path: /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.735 DEBUG:boto:Data: 2024-04-19T23:34:59.735 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.735 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.735 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.735 DEBUG:boto:Params: {} 2024-04-19T23:34:59.735 DEBUG:boto:Token: None 2024-04-19T23:34:59.736 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone1?versionId=8u7XEl.hcooeKX51sIY3frv2y4sjRMT 2024-04-19T23:34:59.736 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:By3FsB/jeghB3oBgkRW2CT0jDLs= 2024-04-19T23:34:59.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:By3FsB/jeghB3oBgkRW2CT0jDLs=', 'Content-Length': '0'} 2024-04-19T23:34:59.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '8u7XEl.hcooeKX51sIY3frv2y4sjRMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026e263a4166e6496-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.782 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.783 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.783 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:59.783 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:59.783 DEBUG:boto:Method: GET 2024-04-19T23:34:59.783 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:59.783 DEBUG:boto:Data: 2024-04-19T23:34:59.783 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.783 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.783 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.783 DEBUG:boto:Params: {} 2024-04-19T23:34:59.784 DEBUG:boto:Token: None 2024-04-19T23:34:59.784 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000817bbd553fa989ed-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.830 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.831 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.831 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.831 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:59.831 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:59.831 DEBUG:boto:Method: GET 2024-04-19T23:34:59.831 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone1 2024-04-19T23:34:59.831 DEBUG:boto:Data: 2024-04-19T23:34:59.832 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.832 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.832 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.832 DEBUG:boto:Params: {} 2024-04-19T23:34:59.832 DEBUG:boto:Token: None 2024-04-19T23:34:59.832 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.832 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a98aba2cff769a4-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.835 DEBUG:boto:b'aktlji-19obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.296Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:59.835 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.835 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.835 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.836 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.836 DEBUG:boto:Method: GET 2024-04-19T23:34:59.836 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.836 DEBUG:boto:Data: 2024-04-19T23:34:59.836 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.836 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.836 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.836 DEBUG:boto:Params: {} 2024-04-19T23:34:59.836 DEBUG:boto:Token: None 2024-04-19T23:34:59.836 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.837 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zrWGwxkQF2tF5SFPyZmV7WB24PA= 2024-04-19T23:34:59.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zrWGwxkQF2tF5SFPyZmV7WB24PA=', 'Content-Length': '0'} 2024-04-19T23:34:59.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bda29fdf08d4232a-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.839 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.839 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.839 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.839 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.839 DEBUG:boto:Method: GET 2024-04-19T23:34:59.839 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.839 DEBUG:boto:Data: 2024-04-19T23:34:59.839 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.840 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.840 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.840 DEBUG:boto:Params: {} 2024-04-19T23:34:59.840 DEBUG:boto:Token: None 2024-04-19T23:34:59.840 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.840 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zrWGwxkQF2tF5SFPyZmV7WB24PA= 2024-04-19T23:34:59.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zrWGwxkQF2tF5SFPyZmV7WB24PA=', 'Content-Length': '0'} 2024-04-19T23:34:59.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082cc046c353edcec-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.842 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.842 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.842 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.843 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.843 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.843 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.843 DEBUG:boto:Data: 2024-04-19T23:34:59.843 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.843 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.843 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.843 DEBUG:boto:Params: {} 2024-04-19T23:34:59.843 DEBUG:boto:Token: None 2024-04-19T23:34:59.843 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.843 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UHCoVHVdbP7tq0P/PmAc9yJHgHs= 2024-04-19T23:34:59.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UHCoVHVdbP7tq0P/PmAc9yJHgHs=', 'Content-Length': '0'} 2024-04-19T23:34:59.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f989426f5ec9365-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.846 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.846 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.846 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.846 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.846 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.846 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.847 DEBUG:boto:Data: 2024-04-19T23:34:59.847 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.847 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.847 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.847 DEBUG:boto:Params: {} 2024-04-19T23:34:59.847 DEBUG:boto:Token: None 2024-04-19T23:34:59.847 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.847 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UHCoVHVdbP7tq0P/PmAc9yJHgHs= 2024-04-19T23:34:59.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UHCoVHVdbP7tq0P/PmAc9yJHgHs=', 'Content-Length': '0'} 2024-04-19T23:34:59.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9130b0e3e2d6f50-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:59.849 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.850 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.850 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.850 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.850 DEBUG:boto:Method: GET 2024-04-19T23:34:59.850 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.850 DEBUG:boto:Data: 2024-04-19T23:34:59.850 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.850 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.850 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.850 DEBUG:boto:Params: {} 2024-04-19T23:34:59.850 DEBUG:boto:Token: None 2024-04-19T23:34:59.851 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.851 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zrWGwxkQF2tF5SFPyZmV7WB24PA= 2024-04-19T23:34:59.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zrWGwxkQF2tF5SFPyZmV7WB24PA=', 'Content-Length': '0'} 2024-04-19T23:34:59.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d67f54eb29467b15-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.894 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.895 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.895 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.895 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.895 DEBUG:boto:Method: GET 2024-04-19T23:34:59.895 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.895 DEBUG:boto:Data: 2024-04-19T23:34:59.895 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.895 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.895 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.895 DEBUG:boto:Params: {} 2024-04-19T23:34:59.896 DEBUG:boto:Token: None 2024-04-19T23:34:59.896 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.896 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zrWGwxkQF2tF5SFPyZmV7WB24PA= 2024-04-19T23:34:59.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zrWGwxkQF2tF5SFPyZmV7WB24PA=', 'Content-Length': '0'} 2024-04-19T23:34:59.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a293ab468f71b39-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.898 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.898 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.898 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:59.898 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:59.899 DEBUG:boto:Method: GET 2024-04-19T23:34:59.899 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:59.899 DEBUG:boto:Data: 2024-04-19T23:34:59.899 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.899 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.899 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.899 DEBUG:boto:Params: {} 2024-04-19T23:34:59.899 DEBUG:boto:Token: None 2024-04-19T23:34:59.899 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.899 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fca4d8f5e2b06e64-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.942 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.943 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:34:59.943 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:34:59.943 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:59.943 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:59.944 DEBUG:boto:Method: GET 2024-04-19T23:34:59.944 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:34:59.944 DEBUG:boto:Data: 2024-04-19T23:34:59.944 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.944 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.944 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.944 DEBUG:boto:Params: {} 2024-04-19T23:34:59.944 DEBUG:boto:Token: None 2024-04-19T23:34:59.944 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/ 2024-04-19T23:34:59.945 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk= 2024-04-19T23:34:59.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b2PNoCAsv79bzwUpfnvSVAh+Ibk=', 'Content-Length': '0'} 2024-04-19T23:34:59.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041e5b5351821ca82-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.947 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:34:59.948 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.948 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.948 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.948 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:34:59.948 DEBUG:boto:Data: 2024-04-19T23:34:59.948 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.948 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.949 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.949 DEBUG:boto:Params: {} 2024-04-19T23:34:59.949 DEBUG:boto:Token: None 2024-04-19T23:34:59.949 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:34:59.949 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sxBxBXeZJokkRurooQlsl4f7Rug= 2024-04-19T23:34:59.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sxBxBXeZJokkRurooQlsl4f7Rug=', 'Content-Length': '0'} 2024-04-19T23:34:59.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b739e077949ad22c-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.952 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.952 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.952 DEBUG:boto:Method: HEAD 2024-04-19T23:34:59.952 DEBUG:boto:Path: /aktlji-19/obj-test-zone2 2024-04-19T23:34:59.952 DEBUG:boto:Data: 2024-04-19T23:34:59.952 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.952 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:34:59.952 DEBUG:boto:Port: 8000 2024-04-19T23:34:59.952 DEBUG:boto:Params: {} 2024-04-19T23:34:59.952 DEBUG:boto:Token: None 2024-04-19T23:34:59.953 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone2 2024-04-19T23:34:59.953 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sxBxBXeZJokkRurooQlsl4f7Rug= 2024-04-19T23:34:59.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sxBxBXeZJokkRurooQlsl4f7Rug=', 'Content-Length': '0'} 2024-04-19T23:34:59.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008484b195353ce78a-006622ffa3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:34:59.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:34:59.955 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.956 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:34:59.956 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.956 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.956 DEBUG:boto:Method: GET 2024-04-19T23:34:59.956 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.956 DEBUG:boto:Data: 2024-04-19T23:34:59.956 DEBUG:boto:Headers: {} 2024-04-19T23:34:59.956 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:34:59.956 DEBUG:boto:Port: 8001 2024-04-19T23:34:59.956 DEBUG:boto:Params: {} 2024-04-19T23:34:59.957 DEBUG:boto:Token: None 2024-04-19T23:34:59.957 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:34:59 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:34:59.957 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zrWGwxkQF2tF5SFPyZmV7WB24PA= 2024-04-19T23:34:59.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:34:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zrWGwxkQF2tF5SFPyZmV7WB24PA=', 'Content-Length': '0'} 2024-04-19T23:34:59.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d610f3330d5cd1ce-006622ffa3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:34:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.007 DEBUG:boto:path=/aktlji-19/obj-test-zone2 2024-04-19T23:35:00.007 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2 2024-04-19T23:35:00.008 DEBUG:boto:path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:35:00.008 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:35:00.008 DEBUG:boto:Method: GET 2024-04-19T23:35:00.008 DEBUG:boto:Path: /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:35:00.008 DEBUG:boto:Data: 2024-04-19T23:35:00.008 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.009 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:00.009 DEBUG:boto:Port: 8000 2024-04-19T23:35:00.010 DEBUG:boto:Params: {} 2024-04-19T23:35:00.010 DEBUG:boto:Token: None 2024-04-19T23:35:00.010 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/obj-test-zone2?versionId=9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv 2024-04-19T23:35:00.010 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GBRzI4rTGzFcuDrQcBqQ3p39qPI= 2024-04-19T23:35:00.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GBRzI4rTGzFcuDrQcBqQ3p39qPI=', 'Content-Length': '0'} 2024-04-19T23:35:00.012 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', '9bH.ePrHLUsvGAMlzHPsjIg2u6W.fQv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a35b71e903ced71-006622ffa4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.013 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:35:00.013 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:35:00.013 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:35:00.013 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:35:00.013 DEBUG:boto:Method: GET 2024-04-19T23:35:00.013 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:35:00.013 DEBUG:boto:Data: 2024-04-19T23:35:00.013 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.014 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:00.014 DEBUG:boto:Port: 8001 2024-04-19T23:35:00.014 DEBUG:boto:Params: {} 2024-04-19T23:35:00.014 DEBUG:boto:Token: None 2024-04-19T23:35:00.014 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/ 2024-04-19T23:35:00.014 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8WEbLBCTyhGl3BvEGa/nG6Te1ok= 2024-04-19T23:35:00.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8WEbLBCTyhGl3BvEGa/nG6Te1ok=', 'Content-Length': '0'} 2024-04-19T23:35:00.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030398867616f192e-006622ffa4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.058 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:00.059 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:35:00.059 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:35:00.059 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:35:00.059 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:35:00.059 DEBUG:boto:Method: GET 2024-04-19T23:35:00.059 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone2 2024-04-19T23:35:00.059 DEBUG:boto:Data: 2024-04-19T23:35:00.060 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.060 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:00.060 DEBUG:boto:Port: 8000 2024-04-19T23:35:00.060 DEBUG:boto:Params: {} 2024-04-19T23:35:00.060 DEBUG:boto:Token: None 2024-04-19T23:35:00.060 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/ 2024-04-19T23:35:00.060 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8WEbLBCTyhGl3BvEGa/nG6Te1ok= 2024-04-19T23:35:00.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8WEbLBCTyhGl3BvEGa/nG6Te1ok=', 'Content-Length': '0'} 2024-04-19T23:35:00.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c782befb138902b0-006622ffa4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.063 DEBUG:boto:b'aktlji-19obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.396Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:00.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:00.063 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.064 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.064 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.064 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.064 DEBUG:boto:Method: GET 2024-04-19T23:35:00.064 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.064 DEBUG:boto:Data: 2024-04-19T23:35:00.064 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.064 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:00.064 DEBUG:boto:Port: 8001 2024-04-19T23:35:00.064 DEBUG:boto:Params: {} 2024-04-19T23:35:00.064 DEBUG:boto:Token: None 2024-04-19T23:35:00.065 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.065 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yqCc3Cc7tx2ZHfn8eSTMW0rPLgU= 2024-04-19T23:35:00.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yqCc3Cc7tx2ZHfn8eSTMW0rPLgU=', 'Content-Length': '0'} 2024-04-19T23:35:00.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdab6fd1adc9ea82-006622ffa4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.067 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.067 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.067 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.067 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.067 DEBUG:boto:Method: GET 2024-04-19T23:35:00.067 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.068 DEBUG:boto:Data: 2024-04-19T23:35:00.068 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.068 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:00.068 DEBUG:boto:Port: 8000 2024-04-19T23:35:00.068 DEBUG:boto:Params: {} 2024-04-19T23:35:00.068 DEBUG:boto:Token: None 2024-04-19T23:35:00.068 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.068 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yqCc3Cc7tx2ZHfn8eSTMW0rPLgU= 2024-04-19T23:35:00.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yqCc3Cc7tx2ZHfn8eSTMW0rPLgU=', 'Content-Length': '0'} 2024-04-19T23:35:00.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4dea7f25850bc60-006622ffa4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.071 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.071 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.071 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.071 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.071 DEBUG:boto:Method: HEAD 2024-04-19T23:35:00.071 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.071 DEBUG:boto:Data: 2024-04-19T23:35:00.071 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.071 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:00.071 DEBUG:boto:Port: 8001 2024-04-19T23:35:00.071 DEBUG:boto:Params: {} 2024-04-19T23:35:00.072 DEBUG:boto:Token: None 2024-04-19T23:35:00.072 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.072 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:podSLwyWWSnKVAaP4+5y5FP5zwU= 2024-04-19T23:35:00.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:podSLwyWWSnKVAaP4+5y5FP5zwU=', 'Content-Length': '0'} 2024-04-19T23:35:00.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d086665b8d3dcf54-006622ffa4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.074 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.078 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.078 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.078 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.078 DEBUG:boto:Method: HEAD 2024-04-19T23:35:00.078 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.078 DEBUG:boto:Data: 2024-04-19T23:35:00.078 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.078 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:00.078 DEBUG:boto:Port: 8000 2024-04-19T23:35:00.078 DEBUG:boto:Params: {} 2024-04-19T23:35:00.079 DEBUG:boto:Token: None 2024-04-19T23:35:00.079 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.079 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:podSLwyWWSnKVAaP4+5y5FP5zwU= 2024-04-19T23:35:00.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:podSLwyWWSnKVAaP4+5y5FP5zwU=', 'Content-Length': '0'} 2024-04-19T23:35:00.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000ca4687bc8ffc20-006622ffa4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:00.081 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.081 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.081 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.081 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.081 DEBUG:boto:Method: GET 2024-04-19T23:35:00.081 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.082 DEBUG:boto:Data: 2024-04-19T23:35:00.082 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.082 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:00.082 DEBUG:boto:Port: 8001 2024-04-19T23:35:00.082 DEBUG:boto:Params: {} 2024-04-19T23:35:00.082 DEBUG:boto:Token: None 2024-04-19T23:35:00.082 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.082 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yqCc3Cc7tx2ZHfn8eSTMW0rPLgU= 2024-04-19T23:35:00.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yqCc3Cc7tx2ZHfn8eSTMW0rPLgU=', 'Content-Length': '0'} 2024-04-19T23:35:00.084 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075e6213f578adc08-006622ffa4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.085 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.085 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.085 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.085 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.085 DEBUG:boto:Method: GET 2024-04-19T23:35:00.085 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.085 DEBUG:boto:Data: 2024-04-19T23:35:00.085 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.085 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:00.085 DEBUG:boto:Port: 8000 2024-04-19T23:35:00.085 DEBUG:boto:Params: {} 2024-04-19T23:35:00.086 DEBUG:boto:Token: None 2024-04-19T23:35:00.086 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.086 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yqCc3Cc7tx2ZHfn8eSTMW0rPLgU= 2024-04-19T23:35:00.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yqCc3Cc7tx2ZHfn8eSTMW0rPLgU=', 'Content-Length': '0'} 2024-04-19T23:35:00.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009786bcfa3fa936e1-006622ffa4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.134 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:35:00.135 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:35:00.135 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:35:00.135 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:35:00.135 DEBUG:boto:Method: GET 2024-04-19T23:35:00.135 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:35:00.135 DEBUG:boto:Data: 2024-04-19T23:35:00.135 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.135 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:00.135 DEBUG:boto:Port: 8001 2024-04-19T23:35:00.135 DEBUG:boto:Params: {} 2024-04-19T23:35:00.135 DEBUG:boto:Token: None 2024-04-19T23:35:00.136 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/ 2024-04-19T23:35:00.136 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8WEbLBCTyhGl3BvEGa/nG6Te1ok= 2024-04-19T23:35:00.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8WEbLBCTyhGl3BvEGa/nG6Te1ok=', 'Content-Length': '0'} 2024-04-19T23:35:00.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007233dde08d89c1c7-006622ffa4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.138 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:00.139 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:35:00.139 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:35:00.139 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:35:00.139 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:35:00.139 DEBUG:boto:Method: GET 2024-04-19T23:35:00.139 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:35:00.140 DEBUG:boto:Data: 2024-04-19T23:35:00.140 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.140 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:00.140 DEBUG:boto:Port: 8000 2024-04-19T23:35:00.140 DEBUG:boto:Params: {} 2024-04-19T23:35:00.140 DEBUG:boto:Token: None 2024-04-19T23:35:00.140 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/ 2024-04-19T23:35:00.141 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8WEbLBCTyhGl3BvEGa/nG6Te1ok= 2024-04-19T23:35:00.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8WEbLBCTyhGl3BvEGa/nG6Te1ok=', 'Content-Length': '0'} 2024-04-19T23:35:00.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000132527d98c253a7-006622ffa4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.186 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:00.187 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.187 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.187 DEBUG:boto:Method: HEAD 2024-04-19T23:35:00.187 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:35:00.187 DEBUG:boto:Data: 2024-04-19T23:35:00.187 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.187 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:00.188 DEBUG:boto:Port: 8001 2024-04-19T23:35:00.188 DEBUG:boto:Params: {} 2024-04-19T23:35:00.188 DEBUG:boto:Token: None 2024-04-19T23:35:00.188 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:35:00.188 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gyfCB6r0A/YEikb113sS/BYV1jI= 2024-04-19T23:35:00.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gyfCB6r0A/YEikb113sS/BYV1jI=', 'Content-Length': '0'} 2024-04-19T23:35:00.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000755ddd987588d836-006622ffa4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.191 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.191 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.191 DEBUG:boto:Method: HEAD 2024-04-19T23:35:00.191 DEBUG:boto:Path: /aktlji-19/obj-test-zone3 2024-04-19T23:35:00.191 DEBUG:boto:Data: 2024-04-19T23:35:00.191 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.191 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:00.192 DEBUG:boto:Port: 8000 2024-04-19T23:35:00.192 DEBUG:boto:Params: {} 2024-04-19T23:35:00.192 DEBUG:boto:Token: None 2024-04-19T23:35:00.192 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/obj-test-zone3 2024-04-19T23:35:00.192 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gyfCB6r0A/YEikb113sS/BYV1jI= 2024-04-19T23:35:00.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gyfCB6r0A/YEikb113sS/BYV1jI=', 'Content-Length': '0'} 2024-04-19T23:35:00.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f63940b278b2015-006622ffa4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:00.195 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.195 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.195 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.195 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.195 DEBUG:boto:Method: GET 2024-04-19T23:35:00.195 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.195 DEBUG:boto:Data: 2024-04-19T23:35:00.195 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.195 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:00.196 DEBUG:boto:Port: 8001 2024-04-19T23:35:00.196 DEBUG:boto:Params: {} 2024-04-19T23:35:00.196 DEBUG:boto:Token: None 2024-04-19T23:35:00.196 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.196 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yqCc3Cc7tx2ZHfn8eSTMW0rPLgU= 2024-04-19T23:35:00.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yqCc3Cc7tx2ZHfn8eSTMW0rPLgU=', 'Content-Length': '0'} 2024-04-19T23:35:00.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c041480afac4ead-006622ffa4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.238 DEBUG:boto:path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.239 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3 2024-04-19T23:35:00.239 DEBUG:boto:path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.239 DEBUG:boto:auth_path=/aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.239 DEBUG:boto:Method: GET 2024-04-19T23:35:00.239 DEBUG:boto:Path: /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.239 DEBUG:boto:Data: 2024-04-19T23:35:00.239 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.239 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:00.239 DEBUG:boto:Port: 8000 2024-04-19T23:35:00.239 DEBUG:boto:Params: {} 2024-04-19T23:35:00.239 DEBUG:boto:Token: None 2024-04-19T23:35:00.240 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/obj-test-zone3?versionId=FJ.cw-LTzcm5kWtF-seTx9qRriifaim 2024-04-19T23:35:00.240 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yqCc3Cc7tx2ZHfn8eSTMW0rPLgU= 2024-04-19T23:35:00.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yqCc3Cc7tx2ZHfn8eSTMW0rPLgU=', 'Content-Length': '0'} 2024-04-19T23:35:00.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'FJ.cw-LTzcm5kWtF-seTx9qRriifaim'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ba74f639bff8a1c-006622ffa4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.282 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:35:00.283 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:35:00.283 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:35:00.283 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:35:00.283 DEBUG:boto:Method: GET 2024-04-19T23:35:00.283 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:35:00.283 DEBUG:boto:Data: 2024-04-19T23:35:00.283 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.283 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:00.283 DEBUG:boto:Port: 8001 2024-04-19T23:35:00.283 DEBUG:boto:Params: {} 2024-04-19T23:35:00.284 DEBUG:boto:Token: None 2024-04-19T23:35:00.284 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/ 2024-04-19T23:35:00.284 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8WEbLBCTyhGl3BvEGa/nG6Te1ok= 2024-04-19T23:35:00.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8WEbLBCTyhGl3BvEGa/nG6Te1ok=', 'Content-Length': '0'} 2024-04-19T23:35:00.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cd40aae96c46bdf-006622ffa4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.287 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:00.287 DEBUG:boto:path=/aktlji-19/ 2024-04-19T23:35:00.287 DEBUG:boto:auth_path=/aktlji-19/ 2024-04-19T23:35:00.288 DEBUG:boto:path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:35:00.288 DEBUG:boto:auth_path=/aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:35:00.288 DEBUG:boto:Method: GET 2024-04-19T23:35:00.288 DEBUG:boto:Path: /aktlji-19/?prefix=obj-test-zone3 2024-04-19T23:35:00.288 DEBUG:boto:Data: 2024-04-19T23:35:00.288 DEBUG:boto:Headers: {} 2024-04-19T23:35:00.288 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:00.288 DEBUG:boto:Port: 8000 2024-04-19T23:35:00.288 DEBUG:boto:Params: {} 2024-04-19T23:35:00.288 DEBUG:boto:Token: None 2024-04-19T23:35:00.288 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:00 GMT /aktlji-19/ 2024-04-19T23:35:00.289 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8WEbLBCTyhGl3BvEGa/nG6Te1ok= 2024-04-19T23:35:00.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8WEbLBCTyhGl3BvEGa/nG6Te1ok=', 'Content-Length': '0'} 2024-04-19T23:35:00.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8d5f4edca32161d-006622ffa4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:00.291 DEBUG:boto:b'aktlji-19obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:00.292 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-19 zones={test-zone3, test-zone2} 2024-04-19T23:35:00.292 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-20 2024-04-19T23:35:00.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:00.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:00.292 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:00.358 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:00.359 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:00.405+0000 7f3e74fb0780 1 bucket sync caught up with source: 2024-04-19T23:35:00.401 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000011.32.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:35:00.401 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000011.32.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:35:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:00.405+0000 7f3e74fb0780 0 bucket checkpoint complete 2024-04-19T23:35:00.416 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-20 2024-04-19T23:35:00.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:00.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:00.417 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:00.485 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:00.485 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:00.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:00.529+0000 7f63c00b6780 1 bucket sync caught up with source: 2024-04-19T23:35:00.524 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000011.32.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:35:00.525 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000011.32.6, , 00000000003.10.10, , 00000000007.16.10, , , , , ] 2024-04-19T23:35:00.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:00.529+0000 7f63c00b6780 0 bucket checkpoint complete 2024-04-19T23:35:00.539 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-20 2024-04-19T23:35:00.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:00.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:00.539 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:00.601 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:00.601 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:00.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:00.646+0000 7f1a60060780 1 waiting for incremental sync to catch up: 2024-04-19T23:35:00.642 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.65.10, , 00000000010.99.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:35:00.642 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.65.10, , 00000000011.102.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:35:05.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:05.647+0000 7f1a60060780 1 waiting for incremental sync to catch up: 2024-04-19T23:35:05.643 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.65.10, , 00000000010.99.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:35:05.643 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.65.10, , 00000000011.102.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:35:10.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:10.649+0000 7f1a60060780 1 waiting for incremental sync to catch up: 2024-04-19T23:35:10.645 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.65.10, , 00000000010.99.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:35:10.645 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.65.10, , 00000000011.102.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:35:15.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:15.650+0000 7f1a60060780 1 bucket sync caught up with source: 2024-04-19T23:35:15.646 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.65.10, , 00000000011.102.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:35:15.646 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.65.10, , 00000000011.102.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:35:15.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:15.650+0000 7f1a60060780 0 bucket checkpoint complete 2024-04-19T23:35:15.663 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-20 2024-04-19T23:35:15.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:15.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:15.664 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:15.730 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:15.730 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:15.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:15.775+0000 7fcc64b52780 1 bucket sync caught up with source: 2024-04-19T23:35:15.771 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.65.10, , 00000000011.102.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:35:15.771 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.65.10, , 00000000011.102.6, , 00000000008.56.10, , , , , ] 2024-04-19T23:35:15.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:15.775+0000 7fcc64b52780 0 bucket checkpoint complete 2024-04-19T23:35:15.785 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-20 2024-04-19T23:35:15.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:15.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:15.785 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:15.849 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:15.850 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:15.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:15.894+0000 7f2b5ba0f780 1 bucket sync caught up with source: 2024-04-19T23:35:15.889 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.34.10, , 00000000003.32.10, , 00000000014.40.6, , , , , ] 2024-04-19T23:35:15.890 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.34.10, , 00000000003.32.10, , 00000000014.40.6, , , , , ] 2024-04-19T23:35:15.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:15.894+0000 7f2b5ba0f780 0 bucket checkpoint complete 2024-04-19T23:35:15.904 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-20 2024-04-19T23:35:15.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:15.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:15.905 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:15.972 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:15.972 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:16.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:16.019+0000 7fcbe6ffb780 1 bucket sync caught up with source: 2024-04-19T23:35:16.015 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000003.34.10, , 00000000003.32.10, , 00000000014.40.6, , , , , ] 2024-04-19T23:35:16.015 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000003.34.10, , 00000000003.32.10, , 00000000014.40.6, , , , , ] 2024-04-19T23:35:16.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:16.019+0000 7fcbe6ffb780 0 bucket checkpoint complete 2024-04-19T23:35:16.030 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-20 zones={test-zone2, test-zone1} 2024-04-19T23:35:16.030 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.030 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.030 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.031 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:35:16.031 DEBUG:boto:Data: 2024-04-19T23:35:16.031 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.031 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.031 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.031 DEBUG:boto:Params: {} 2024-04-19T23:35:16.032 DEBUG:boto:Token: None 2024-04-19T23:35:16.032 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.032 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DoQq5YMuIp0ymYlzPYDvuLFR/2w= 2024-04-19T23:35:16.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DoQq5YMuIp0ymYlzPYDvuLFR/2w=', 'Content-Length': '0'} 2024-04-19T23:35:16.035 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1febc592e07b6ac-006622ffb4-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.035 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.035 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.035 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.036 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:35:16.036 DEBUG:boto:Data: 2024-04-19T23:35:16.036 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.036 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.036 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.036 DEBUG:boto:Params: {} 2024-04-19T23:35:16.036 DEBUG:boto:Token: None 2024-04-19T23:35:16.036 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.036 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DoQq5YMuIp0ymYlzPYDvuLFR/2w= 2024-04-19T23:35:16.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DoQq5YMuIp0ymYlzPYDvuLFR/2w=', 'Content-Length': '0'} 2024-04-19T23:35:16.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000074526047c7193e8-006622ffb4-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.043 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:35:16.043 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.044 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.044 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.044 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.044 DEBUG:boto:Method: GET 2024-04-19T23:35:16.044 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.044 DEBUG:boto:Data: 2024-04-19T23:35:16.044 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.044 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.044 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.044 DEBUG:boto:Params: {} 2024-04-19T23:35:16.045 DEBUG:boto:Token: None 2024-04-19T23:35:16.045 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/?versions 2024-04-19T23:35:16.045 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA= 2024-04-19T23:35:16.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA=', 'Content-Length': '0'} 2024-04-19T23:35:16.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000650b17f062766b42-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.048 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.048 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:35:16.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:35:16.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:35:16.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:35:16.049 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:35:16.049 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.049 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.050 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.050 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.050 DEBUG:boto:Method: GET 2024-04-19T23:35:16.050 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.051 DEBUG:boto:Data: 2024-04-19T23:35:16.051 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.051 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.051 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.051 DEBUG:boto:Params: {} 2024-04-19T23:35:16.051 DEBUG:boto:Token: None 2024-04-19T23:35:16.051 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/?versions 2024-04-19T23:35:16.051 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA= 2024-04-19T23:35:16.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA=', 'Content-Length': '0'} 2024-04-19T23:35:16.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073d4744332e9e511-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.054 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.055 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:35:16.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:35:16.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:35:16.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:35:16.055 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.055 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.055 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.055 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.056 DEBUG:boto:Method: GET 2024-04-19T23:35:16.056 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.056 DEBUG:boto:Data: 2024-04-19T23:35:16.056 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.056 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.056 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.056 DEBUG:boto:Params: {} 2024-04-19T23:35:16.056 DEBUG:boto:Token: None 2024-04-19T23:35:16.056 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/?versions 2024-04-19T23:35:16.056 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA= 2024-04-19T23:35:16.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA=', 'Content-Length': '0'} 2024-04-19T23:35:16.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000414961596c07f849-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.059 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.060 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.060 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.060 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.060 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.060 DEBUG:boto:Method: GET 2024-04-19T23:35:16.060 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.061 DEBUG:boto:Data: 2024-04-19T23:35:16.061 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.061 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.061 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.061 DEBUG:boto:Params: {} 2024-04-19T23:35:16.061 DEBUG:boto:Token: None 2024-04-19T23:35:16.061 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/?versions 2024-04-19T23:35:16.061 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA= 2024-04-19T23:35:16.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA=', 'Content-Length': '0'} 2024-04-19T23:35:16.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c873a2c31dd5e57-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.064 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:16.065 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:16.065 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:16.065 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.065 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.065 DEBUG:boto:Method: GET 2024-04-19T23:35:16.065 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.066 DEBUG:boto:Data: 2024-04-19T23:35:16.066 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.066 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.066 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.066 DEBUG:boto:Params: {} 2024-04-19T23:35:16.066 DEBUG:boto:Token: None 2024-04-19T23:35:16.067 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.067 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLuS5V4j/UQPUHSv1dMZwhE2cYo= 2024-04-19T23:35:16.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLuS5V4j/UQPUHSv1dMZwhE2cYo=', 'Content-Length': '0'} 2024-04-19T23:35:16.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d612d99806cf1a02-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.069 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:16.069 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:16.069 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.069 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.070 DEBUG:boto:Method: GET 2024-04-19T23:35:16.070 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.070 DEBUG:boto:Data: 2024-04-19T23:35:16.070 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.070 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.070 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.070 DEBUG:boto:Params: {} 2024-04-19T23:35:16.070 DEBUG:boto:Token: None 2024-04-19T23:35:16.070 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.070 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLuS5V4j/UQPUHSv1dMZwhE2cYo= 2024-04-19T23:35:16.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLuS5V4j/UQPUHSv1dMZwhE2cYo=', 'Content-Length': '0'} 2024-04-19T23:35:16.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003db264b09f1870bf-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.073 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:16.073 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:16.073 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.073 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.073 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.073 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.074 DEBUG:boto:Data: 2024-04-19T23:35:16.074 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.074 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.074 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.074 DEBUG:boto:Params: {} 2024-04-19T23:35:16.074 DEBUG:boto:Token: None 2024-04-19T23:35:16.074 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.074 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EhFGjiYQ3v1Gu/CrZlEdc/Xw8oA= 2024-04-19T23:35:16.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EhFGjiYQ3v1Gu/CrZlEdc/Xw8oA=', 'Content-Length': '0'} 2024-04-19T23:35:16.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000528943b19952c110-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.076 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:16.076 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:16.076 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.076 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.077 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.077 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.077 DEBUG:boto:Data: 2024-04-19T23:35:16.077 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.077 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.077 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.077 DEBUG:boto:Params: {} 2024-04-19T23:35:16.077 DEBUG:boto:Token: None 2024-04-19T23:35:16.077 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.078 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EhFGjiYQ3v1Gu/CrZlEdc/Xw8oA= 2024-04-19T23:35:16.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EhFGjiYQ3v1Gu/CrZlEdc/Xw8oA=', 'Content-Length': '0'} 2024-04-19T23:35:16.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f2d9295f530dba4-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:16.080 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:16.080 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:16.080 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.080 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.080 DEBUG:boto:Method: GET 2024-04-19T23:35:16.080 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.080 DEBUG:boto:Data: 2024-04-19T23:35:16.080 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.080 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.080 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.081 DEBUG:boto:Params: {} 2024-04-19T23:35:16.081 DEBUG:boto:Token: None 2024-04-19T23:35:16.081 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.081 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLuS5V4j/UQPUHSv1dMZwhE2cYo= 2024-04-19T23:35:16.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLuS5V4j/UQPUHSv1dMZwhE2cYo=', 'Content-Length': '0'} 2024-04-19T23:35:16.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003431be7e0adc4341-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.083 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:16.083 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:16.083 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.083 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.083 DEBUG:boto:Method: GET 2024-04-19T23:35:16.083 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.083 DEBUG:boto:Data: 2024-04-19T23:35:16.084 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.084 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.084 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.084 DEBUG:boto:Params: {} 2024-04-19T23:35:16.084 DEBUG:boto:Token: None 2024-04-19T23:35:16.084 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLuS5V4j/UQPUHSv1dMZwhE2cYo= 2024-04-19T23:35:16.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLuS5V4j/UQPUHSv1dMZwhE2cYo=', 'Content-Length': '0'} 2024-04-19T23:35:16.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a130663e5c6b9f4-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.086 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.086 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.086 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:16.087 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:16.087 DEBUG:boto:Method: GET 2024-04-19T23:35:16.087 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:35:16.087 DEBUG:boto:Data: 2024-04-19T23:35:16.087 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.087 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.087 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.087 DEBUG:boto:Params: {} 2024-04-19T23:35:16.087 DEBUG:boto:Token: None 2024-04-19T23:35:16.088 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.088 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091fb1052abef50a0-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.135 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.136 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.136 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.136 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:16.136 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:16.136 DEBUG:boto:Method: GET 2024-04-19T23:35:16.136 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:35:16.136 DEBUG:boto:Data: 2024-04-19T23:35:16.136 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.136 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.136 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.137 DEBUG:boto:Params: {} 2024-04-19T23:35:16.137 DEBUG:boto:Token: None 2024-04-19T23:35:16.137 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.137 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038285c83e303070e-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.140 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.140 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:16.140 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:16.140 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.140 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:35:16.141 DEBUG:boto:Data: 2024-04-19T23:35:16.141 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.141 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.141 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.141 DEBUG:boto:Params: {} 2024-04-19T23:35:16.141 DEBUG:boto:Token: None 2024-04-19T23:35:16.141 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/dummy 2024-04-19T23:35:16.141 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yF+2/kJK7YDPt+gV/K7sOOq8NKw= 2024-04-19T23:35:16.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yF+2/kJK7YDPt+gV/K7sOOq8NKw=', 'Content-Length': '0'} 2024-04-19T23:35:16.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008adb8408f8071f93-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.144 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:16.144 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:16.144 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.144 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:35:16.144 DEBUG:boto:Data: 2024-04-19T23:35:16.144 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.145 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.145 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.145 DEBUG:boto:Params: {} 2024-04-19T23:35:16.145 DEBUG:boto:Token: None 2024-04-19T23:35:16.145 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/dummy 2024-04-19T23:35:16.145 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yF+2/kJK7YDPt+gV/K7sOOq8NKw= 2024-04-19T23:35:16.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yF+2/kJK7YDPt+gV/K7sOOq8NKw=', 'Content-Length': '0'} 2024-04-19T23:35:16.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050c8c069947ef474-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:16.148 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:16.148 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:16.148 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.148 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.148 DEBUG:boto:Method: GET 2024-04-19T23:35:16.148 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.149 DEBUG:boto:Data: 2024-04-19T23:35:16.149 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.149 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.149 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.149 DEBUG:boto:Params: {} 2024-04-19T23:35:16.149 DEBUG:boto:Token: None 2024-04-19T23:35:16.149 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.149 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLuS5V4j/UQPUHSv1dMZwhE2cYo= 2024-04-19T23:35:16.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLuS5V4j/UQPUHSv1dMZwhE2cYo=', 'Content-Length': '0'} 2024-04-19T23:35:16.151 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000010097d73fdf0f1a1-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.151 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:16.151 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:16.151 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.151 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.152 DEBUG:boto:Method: GET 2024-04-19T23:35:16.152 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.152 DEBUG:boto:Data: 2024-04-19T23:35:16.152 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.152 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.152 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.152 DEBUG:boto:Params: {} 2024-04-19T23:35:16.152 DEBUG:boto:Token: None 2024-04-19T23:35:16.152 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:16.152 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLuS5V4j/UQPUHSv1dMZwhE2cYo= 2024-04-19T23:35:16.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLuS5V4j/UQPUHSv1dMZwhE2cYo=', 'Content-Length': '0'} 2024-04-19T23:35:16.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006e6795ad6b8ac624-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.155 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.155 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.155 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:16.155 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:16.155 DEBUG:boto:Method: GET 2024-04-19T23:35:16.155 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:35:16.155 DEBUG:boto:Data: 2024-04-19T23:35:16.155 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.155 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.155 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.155 DEBUG:boto:Params: {} 2024-04-19T23:35:16.155 DEBUG:boto:Token: None 2024-04-19T23:35:16.156 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.156 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000268120d935233484-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.198 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.199 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.199 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.199 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:16.199 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:16.199 DEBUG:boto:Method: GET 2024-04-19T23:35:16.199 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:35:16.199 DEBUG:boto:Data: 2024-04-19T23:35:16.199 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.199 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.200 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.200 DEBUG:boto:Params: {} 2024-04-19T23:35:16.200 DEBUG:boto:Token: None 2024-04-19T23:35:16.200 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.200 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb5cef8f97777189-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.203 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:16.203 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.203 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.204 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.204 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.204 DEBUG:boto:Method: GET 2024-04-19T23:35:16.204 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.204 DEBUG:boto:Data: 2024-04-19T23:35:16.204 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.204 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.204 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.204 DEBUG:boto:Params: {} 2024-04-19T23:35:16.204 DEBUG:boto:Token: None 2024-04-19T23:35:16.204 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.205 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shTCDH5bdPjzakXBtORUAsNE1j4= 2024-04-19T23:35:16.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shTCDH5bdPjzakXBtORUAsNE1j4=', 'Content-Length': '0'} 2024-04-19T23:35:16.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff299d598d9061e2-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.207 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.207 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.207 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.207 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.207 DEBUG:boto:Method: GET 2024-04-19T23:35:16.207 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.208 DEBUG:boto:Data: 2024-04-19T23:35:16.208 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.208 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.208 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.208 DEBUG:boto:Params: {} 2024-04-19T23:35:16.208 DEBUG:boto:Token: None 2024-04-19T23:35:16.208 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.208 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shTCDH5bdPjzakXBtORUAsNE1j4= 2024-04-19T23:35:16.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shTCDH5bdPjzakXBtORUAsNE1j4=', 'Content-Length': '0'} 2024-04-19T23:35:16.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b56e04c6be41fdd0-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.254 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.254 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.255 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.255 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.255 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.255 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.255 DEBUG:boto:Data: 2024-04-19T23:35:16.255 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.255 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.255 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.255 DEBUG:boto:Params: {} 2024-04-19T23:35:16.255 DEBUG:boto:Token: None 2024-04-19T23:35:16.256 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.256 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9CoIkZApGZybUROwNdYL7+xSK5Q= 2024-04-19T23:35:16.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9CoIkZApGZybUROwNdYL7+xSK5Q=', 'Content-Length': '0'} 2024-04-19T23:35:16.257 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d5feb7b933bec38-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.258 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.258 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.258 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.258 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.258 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.258 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.258 DEBUG:boto:Data: 2024-04-19T23:35:16.258 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.258 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.259 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.259 DEBUG:boto:Params: {} 2024-04-19T23:35:16.259 DEBUG:boto:Token: None 2024-04-19T23:35:16.259 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.259 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9CoIkZApGZybUROwNdYL7+xSK5Q= 2024-04-19T23:35:16.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9CoIkZApGZybUROwNdYL7+xSK5Q=', 'Content-Length': '0'} 2024-04-19T23:35:16.261 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b263ac7a92215c8-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:16.261 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.262 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.262 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.262 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.262 DEBUG:boto:Method: GET 2024-04-19T23:35:16.262 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.262 DEBUG:boto:Data: 2024-04-19T23:35:16.262 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.262 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.262 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.262 DEBUG:boto:Params: {} 2024-04-19T23:35:16.263 DEBUG:boto:Token: None 2024-04-19T23:35:16.263 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.263 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shTCDH5bdPjzakXBtORUAsNE1j4= 2024-04-19T23:35:16.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shTCDH5bdPjzakXBtORUAsNE1j4=', 'Content-Length': '0'} 2024-04-19T23:35:16.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e82ab39ff6468c87-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.306 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.306 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.307 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.307 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.307 DEBUG:boto:Method: GET 2024-04-19T23:35:16.307 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.307 DEBUG:boto:Data: 2024-04-19T23:35:16.307 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.307 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.307 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.307 DEBUG:boto:Params: {} 2024-04-19T23:35:16.307 DEBUG:boto:Token: None 2024-04-19T23:35:16.308 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.308 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shTCDH5bdPjzakXBtORUAsNE1j4= 2024-04-19T23:35:16.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shTCDH5bdPjzakXBtORUAsNE1j4=', 'Content-Length': '0'} 2024-04-19T23:35:16.310 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003a6f1edfc0efe31-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.310 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.311 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.311 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:16.311 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:16.311 DEBUG:boto:Method: GET 2024-04-19T23:35:16.311 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:16.311 DEBUG:boto:Data: 2024-04-19T23:35:16.311 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.311 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.311 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.311 DEBUG:boto:Params: {} 2024-04-19T23:35:16.311 DEBUG:boto:Token: None 2024-04-19T23:35:16.312 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.312 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50c5e9e4a018e47-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.314 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.314 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.315 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.315 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:16.315 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:16.315 DEBUG:boto:Method: GET 2024-04-19T23:35:16.315 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:16.315 DEBUG:boto:Data: 2024-04-19T23:35:16.315 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.315 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.315 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.315 DEBUG:boto:Params: {} 2024-04-19T23:35:16.316 DEBUG:boto:Token: None 2024-04-19T23:35:16.316 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.316 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000768e42fd1168d7a-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.318 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.318 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.319 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.319 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.319 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:35:16.319 DEBUG:boto:Data: 2024-04-19T23:35:16.319 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.319 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.319 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.319 DEBUG:boto:Params: {} 2024-04-19T23:35:16.319 DEBUG:boto:Token: None 2024-04-19T23:35:16.319 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:35:16.320 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jGmf25w7E92PJg2soOcCOm9Ra0E= 2024-04-19T23:35:16.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jGmf25w7E92PJg2soOcCOm9Ra0E=', 'Content-Length': '0'} 2024-04-19T23:35:16.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000102e69ebed9eb477-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.322 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.322 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.322 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.322 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:35:16.322 DEBUG:boto:Data: 2024-04-19T23:35:16.323 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.323 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.323 DEBUG:boto:Params: {} 2024-04-19T23:35:16.323 DEBUG:boto:Token: None 2024-04-19T23:35:16.323 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:35:16.323 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jGmf25w7E92PJg2soOcCOm9Ra0E= 2024-04-19T23:35:16.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jGmf25w7E92PJg2soOcCOm9Ra0E=', 'Content-Length': '0'} 2024-04-19T23:35:16.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4ae3ed00485f25b-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:16.326 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.326 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.326 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.326 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.326 DEBUG:boto:Method: GET 2024-04-19T23:35:16.327 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.327 DEBUG:boto:Data: 2024-04-19T23:35:16.327 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.327 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.327 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.327 DEBUG:boto:Params: {} 2024-04-19T23:35:16.327 DEBUG:boto:Token: None 2024-04-19T23:35:16.327 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.328 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shTCDH5bdPjzakXBtORUAsNE1j4= 2024-04-19T23:35:16.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shTCDH5bdPjzakXBtORUAsNE1j4=', 'Content-Length': '0'} 2024-04-19T23:35:16.330 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac7910184e3deeaf-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.370 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.370 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:16.370 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.371 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.371 DEBUG:boto:Method: GET 2024-04-19T23:35:16.371 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.371 DEBUG:boto:Data: 2024-04-19T23:35:16.371 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.371 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.371 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.371 DEBUG:boto:Params: {} 2024-04-19T23:35:16.371 DEBUG:boto:Token: None 2024-04-19T23:35:16.371 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:16.372 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:shTCDH5bdPjzakXBtORUAsNE1j4= 2024-04-19T23:35:16.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shTCDH5bdPjzakXBtORUAsNE1j4=', 'Content-Length': '0'} 2024-04-19T23:35:16.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b006a3c4c7bbe7b1-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.374 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.374 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.374 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:16.374 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:16.374 DEBUG:boto:Method: GET 2024-04-19T23:35:16.374 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:16.374 DEBUG:boto:Data: 2024-04-19T23:35:16.374 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.374 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.374 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.374 DEBUG:boto:Params: {} 2024-04-19T23:35:16.375 DEBUG:boto:Token: None 2024-04-19T23:35:16.375 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.375 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000167f19040a8a9fe1-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.426 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.427 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.427 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.427 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:16.427 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:16.427 DEBUG:boto:Method: GET 2024-04-19T23:35:16.427 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:16.427 DEBUG:boto:Data: 2024-04-19T23:35:16.427 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.428 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.428 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.428 DEBUG:boto:Params: {} 2024-04-19T23:35:16.428 DEBUG:boto:Token: None 2024-04-19T23:35:16.428 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.428 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000197d14c75fca73f6-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.431 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:16.431 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.432 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.432 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.432 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.432 DEBUG:boto:Method: GET 2024-04-19T23:35:16.432 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.432 DEBUG:boto:Data: 2024-04-19T23:35:16.432 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.432 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.432 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.432 DEBUG:boto:Params: {} 2024-04-19T23:35:16.432 DEBUG:boto:Token: None 2024-04-19T23:35:16.433 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.433 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ha99gEZKTDgySUa6RKpKAz6U40= 2024-04-19T23:35:16.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ha99gEZKTDgySUa6RKpKAz6U40=', 'Content-Length': '0'} 2024-04-19T23:35:16.435 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009507c5db8e80e85b-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.435 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.435 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.435 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.435 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.435 DEBUG:boto:Method: GET 2024-04-19T23:35:16.435 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.436 DEBUG:boto:Data: 2024-04-19T23:35:16.436 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.436 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.436 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.436 DEBUG:boto:Params: {} 2024-04-19T23:35:16.436 DEBUG:boto:Token: None 2024-04-19T23:35:16.436 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.436 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ha99gEZKTDgySUa6RKpKAz6U40= 2024-04-19T23:35:16.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ha99gEZKTDgySUa6RKpKAz6U40=', 'Content-Length': '0'} 2024-04-19T23:35:16.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e995b987233b77c1-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.438 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.438 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.438 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.438 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.439 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.439 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.439 DEBUG:boto:Data: 2024-04-19T23:35:16.439 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.439 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.439 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.439 DEBUG:boto:Params: {} 2024-04-19T23:35:16.439 DEBUG:boto:Token: None 2024-04-19T23:35:16.439 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.439 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ekUFiaooZaBqPYfKfwDpv85hSb8= 2024-04-19T23:35:16.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ekUFiaooZaBqPYfKfwDpv85hSb8=', 'Content-Length': '0'} 2024-04-19T23:35:16.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000579bdec5160e9467-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.441 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.442 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.442 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.442 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.442 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.442 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.442 DEBUG:boto:Data: 2024-04-19T23:35:16.442 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.442 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.442 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.442 DEBUG:boto:Params: {} 2024-04-19T23:35:16.442 DEBUG:boto:Token: None 2024-04-19T23:35:16.443 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.443 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ekUFiaooZaBqPYfKfwDpv85hSb8= 2024-04-19T23:35:16.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ekUFiaooZaBqPYfKfwDpv85hSb8=', 'Content-Length': '0'} 2024-04-19T23:35:16.444 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000135e201dc3a4f830-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:16.445 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.445 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.445 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.445 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.445 DEBUG:boto:Method: GET 2024-04-19T23:35:16.445 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.445 DEBUG:boto:Data: 2024-04-19T23:35:16.445 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.445 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.446 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.446 DEBUG:boto:Params: {} 2024-04-19T23:35:16.446 DEBUG:boto:Token: None 2024-04-19T23:35:16.446 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.446 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ha99gEZKTDgySUa6RKpKAz6U40= 2024-04-19T23:35:16.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ha99gEZKTDgySUa6RKpKAz6U40=', 'Content-Length': '0'} 2024-04-19T23:35:16.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b4aa13314125e38-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.448 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.448 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.448 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.448 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.448 DEBUG:boto:Method: GET 2024-04-19T23:35:16.449 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.449 DEBUG:boto:Data: 2024-04-19T23:35:16.449 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.449 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.449 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.449 DEBUG:boto:Params: {} 2024-04-19T23:35:16.449 DEBUG:boto:Token: None 2024-04-19T23:35:16.449 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.449 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ha99gEZKTDgySUa6RKpKAz6U40= 2024-04-19T23:35:16.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ha99gEZKTDgySUa6RKpKAz6U40=', 'Content-Length': '0'} 2024-04-19T23:35:16.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f56fc16e224d01af-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.494 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.495 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.495 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:16.495 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:16.495 DEBUG:boto:Method: GET 2024-04-19T23:35:16.495 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:16.495 DEBUG:boto:Data: 2024-04-19T23:35:16.495 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.495 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.495 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.495 DEBUG:boto:Params: {} 2024-04-19T23:35:16.496 DEBUG:boto:Token: None 2024-04-19T23:35:16.496 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.496 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a0189cfadf95d94-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.499 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.499 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.499 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.500 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:16.500 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:16.500 DEBUG:boto:Method: GET 2024-04-19T23:35:16.500 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:16.500 DEBUG:boto:Data: 2024-04-19T23:35:16.500 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.500 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.500 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.500 DEBUG:boto:Params: {} 2024-04-19T23:35:16.500 DEBUG:boto:Token: None 2024-04-19T23:35:16.500 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.501 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000929b5cdce17050ce-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.504 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.504 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.504 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.505 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.505 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:35:16.505 DEBUG:boto:Data: 2024-04-19T23:35:16.505 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.505 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.505 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.505 DEBUG:boto:Params: {} 2024-04-19T23:35:16.506 DEBUG:boto:Token: None 2024-04-19T23:35:16.506 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:35:16.506 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XvUS6NAF3VF7cfdI//wBstv9pFI= 2024-04-19T23:35:16.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XvUS6NAF3VF7cfdI//wBstv9pFI=', 'Content-Length': '0'} 2024-04-19T23:35:16.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fcf2f1a37009ca0-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.509 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.509 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.509 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.509 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:35:16.509 DEBUG:boto:Data: 2024-04-19T23:35:16.510 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.510 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.510 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.510 DEBUG:boto:Params: {} 2024-04-19T23:35:16.510 DEBUG:boto:Token: None 2024-04-19T23:35:16.510 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:35:16.510 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XvUS6NAF3VF7cfdI//wBstv9pFI= 2024-04-19T23:35:16.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XvUS6NAF3VF7cfdI//wBstv9pFI=', 'Content-Length': '0'} 2024-04-19T23:35:16.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069e865b5093aba32-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:16.513 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.513 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.514 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.514 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.514 DEBUG:boto:Method: GET 2024-04-19T23:35:16.514 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.514 DEBUG:boto:Data: 2024-04-19T23:35:16.514 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.514 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.514 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.514 DEBUG:boto:Params: {} 2024-04-19T23:35:16.514 DEBUG:boto:Token: None 2024-04-19T23:35:16.515 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.515 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ha99gEZKTDgySUa6RKpKAz6U40= 2024-04-19T23:35:16.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ha99gEZKTDgySUa6RKpKAz6U40=', 'Content-Length': '0'} 2024-04-19T23:35:16.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9c7477b0d3155e4-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.558 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.559 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:16.559 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.559 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.559 DEBUG:boto:Method: GET 2024-04-19T23:35:16.559 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.559 DEBUG:boto:Data: 2024-04-19T23:35:16.559 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.559 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.559 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.559 DEBUG:boto:Params: {} 2024-04-19T23:35:16.560 DEBUG:boto:Token: None 2024-04-19T23:35:16.560 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:16.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ha99gEZKTDgySUa6RKpKAz6U40= 2024-04-19T23:35:16.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ha99gEZKTDgySUa6RKpKAz6U40=', 'Content-Length': '0'} 2024-04-19T23:35:16.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fe80892f5771c76-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.602 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.602 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.603 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:16.603 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:16.603 DEBUG:boto:Method: GET 2024-04-19T23:35:16.603 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:16.603 DEBUG:boto:Data: 2024-04-19T23:35:16.603 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.603 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.603 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.603 DEBUG:boto:Params: {} 2024-04-19T23:35:16.603 DEBUG:boto:Token: None 2024-04-19T23:35:16.604 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.604 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a2bea100978351e-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.607 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.607 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.607 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.607 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:16.607 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:16.607 DEBUG:boto:Method: GET 2024-04-19T23:35:16.608 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:16.608 DEBUG:boto:Data: 2024-04-19T23:35:16.608 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.608 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.608 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.608 DEBUG:boto:Params: {} 2024-04-19T23:35:16.608 DEBUG:boto:Token: None 2024-04-19T23:35:16.608 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.608 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062c04acf71df3a5e-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.611 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:16.611 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.612 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.612 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.612 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.612 DEBUG:boto:Method: GET 2024-04-19T23:35:16.612 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.612 DEBUG:boto:Data: 2024-04-19T23:35:16.612 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.612 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.612 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.612 DEBUG:boto:Params: {} 2024-04-19T23:35:16.613 DEBUG:boto:Token: None 2024-04-19T23:35:16.613 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.613 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFfjHZiXm6PsACbkwJrnvg+N6EM= 2024-04-19T23:35:16.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFfjHZiXm6PsACbkwJrnvg+N6EM=', 'Content-Length': '0'} 2024-04-19T23:35:16.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce1f07107598e02f-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.615 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.615 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.615 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.615 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.615 DEBUG:boto:Method: GET 2024-04-19T23:35:16.615 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.616 DEBUG:boto:Data: 2024-04-19T23:35:16.616 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.616 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.616 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.616 DEBUG:boto:Params: {} 2024-04-19T23:35:16.616 DEBUG:boto:Token: None 2024-04-19T23:35:16.616 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.616 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFfjHZiXm6PsACbkwJrnvg+N6EM= 2024-04-19T23:35:16.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFfjHZiXm6PsACbkwJrnvg+N6EM=', 'Content-Length': '0'} 2024-04-19T23:35:16.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007eab709bd8302b44-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.618 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.618 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.619 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.619 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.619 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.619 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.619 DEBUG:boto:Data: 2024-04-19T23:35:16.619 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.619 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.619 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.619 DEBUG:boto:Params: {} 2024-04-19T23:35:16.619 DEBUG:boto:Token: None 2024-04-19T23:35:16.620 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.620 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lq94xQzhfuq5iIlLKXdXxPuRjSM= 2024-04-19T23:35:16.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lq94xQzhfuq5iIlLKXdXxPuRjSM=', 'Content-Length': '0'} 2024-04-19T23:35:16.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068191a2f6fa99340-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.622 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.622 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.622 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.622 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.623 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.623 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.623 DEBUG:boto:Data: 2024-04-19T23:35:16.623 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.623 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.623 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.623 DEBUG:boto:Params: {} 2024-04-19T23:35:16.623 DEBUG:boto:Token: None 2024-04-19T23:35:16.623 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.623 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lq94xQzhfuq5iIlLKXdXxPuRjSM= 2024-04-19T23:35:16.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lq94xQzhfuq5iIlLKXdXxPuRjSM=', 'Content-Length': '0'} 2024-04-19T23:35:16.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fe35d73908cc87c-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:16.626 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.626 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.626 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.626 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.626 DEBUG:boto:Method: GET 2024-04-19T23:35:16.626 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.626 DEBUG:boto:Data: 2024-04-19T23:35:16.626 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.626 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.626 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.627 DEBUG:boto:Params: {} 2024-04-19T23:35:16.627 DEBUG:boto:Token: None 2024-04-19T23:35:16.627 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.627 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFfjHZiXm6PsACbkwJrnvg+N6EM= 2024-04-19T23:35:16.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFfjHZiXm6PsACbkwJrnvg+N6EM=', 'Content-Length': '0'} 2024-04-19T23:35:16.629 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005329f98d04d6a5a8-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.674 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.675 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.675 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.675 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.675 DEBUG:boto:Method: GET 2024-04-19T23:35:16.675 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.675 DEBUG:boto:Data: 2024-04-19T23:35:16.675 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.675 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.675 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.676 DEBUG:boto:Params: {} 2024-04-19T23:35:16.676 DEBUG:boto:Token: None 2024-04-19T23:35:16.676 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFfjHZiXm6PsACbkwJrnvg+N6EM= 2024-04-19T23:35:16.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFfjHZiXm6PsACbkwJrnvg+N6EM=', 'Content-Length': '0'} 2024-04-19T23:35:16.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004037c8181b856767-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.718 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.719 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.719 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:16.719 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:16.719 DEBUG:boto:Method: GET 2024-04-19T23:35:16.719 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:16.719 DEBUG:boto:Data: 2024-04-19T23:35:16.719 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.719 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.719 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.720 DEBUG:boto:Params: {} 2024-04-19T23:35:16.720 DEBUG:boto:Token: None 2024-04-19T23:35:16.720 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.720 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca5c89e7d954f51f-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.766 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.767 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.767 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.767 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:16.767 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:16.767 DEBUG:boto:Method: GET 2024-04-19T23:35:16.767 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:16.768 DEBUG:boto:Data: 2024-04-19T23:35:16.768 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.768 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.768 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.768 DEBUG:boto:Params: {} 2024-04-19T23:35:16.768 DEBUG:boto:Token: None 2024-04-19T23:35:16.768 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.768 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ed33d2fc160039d-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.772 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.772 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.772 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.772 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.772 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:35:16.772 DEBUG:boto:Data: 2024-04-19T23:35:16.772 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.773 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.773 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.773 DEBUG:boto:Params: {} 2024-04-19T23:35:16.773 DEBUG:boto:Token: None 2024-04-19T23:35:16.773 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:35:16.773 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DPxJOTgE/w5Sz4lkNj1IrOPriDM= 2024-04-19T23:35:16.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DPxJOTgE/w5Sz4lkNj1IrOPriDM=', 'Content-Length': '0'} 2024-04-19T23:35:16.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3136847d1fc13c0-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.776 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.776 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.776 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.776 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:35:16.776 DEBUG:boto:Data: 2024-04-19T23:35:16.776 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.776 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.776 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.776 DEBUG:boto:Params: {} 2024-04-19T23:35:16.776 DEBUG:boto:Token: None 2024-04-19T23:35:16.777 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:35:16.777 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DPxJOTgE/w5Sz4lkNj1IrOPriDM= 2024-04-19T23:35:16.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DPxJOTgE/w5Sz4lkNj1IrOPriDM=', 'Content-Length': '0'} 2024-04-19T23:35:16.779 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc5ddeb77c1e1c8a-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:16.780 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.780 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.780 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.780 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.780 DEBUG:boto:Method: GET 2024-04-19T23:35:16.780 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.780 DEBUG:boto:Data: 2024-04-19T23:35:16.780 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.780 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.780 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.781 DEBUG:boto:Params: {} 2024-04-19T23:35:16.781 DEBUG:boto:Token: None 2024-04-19T23:35:16.781 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.781 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFfjHZiXm6PsACbkwJrnvg+N6EM= 2024-04-19T23:35:16.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFfjHZiXm6PsACbkwJrnvg+N6EM=', 'Content-Length': '0'} 2024-04-19T23:35:16.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000598ee45a8229b41e-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.830 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.830 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:16.831 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.831 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.831 DEBUG:boto:Method: GET 2024-04-19T23:35:16.831 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.831 DEBUG:boto:Data: 2024-04-19T23:35:16.831 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.831 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.831 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.831 DEBUG:boto:Params: {} 2024-04-19T23:35:16.832 DEBUG:boto:Token: None 2024-04-19T23:35:16.832 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:16.832 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFfjHZiXm6PsACbkwJrnvg+N6EM= 2024-04-19T23:35:16.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFfjHZiXm6PsACbkwJrnvg+N6EM=', 'Content-Length': '0'} 2024-04-19T23:35:16.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9a26b12a5c02609-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.878 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.878 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.879 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:16.879 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:16.879 DEBUG:boto:Method: GET 2024-04-19T23:35:16.879 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:16.879 DEBUG:boto:Data: 2024-04-19T23:35:16.879 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.879 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:16.879 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.879 DEBUG:boto:Params: {} 2024-04-19T23:35:16.879 DEBUG:boto:Token: None 2024-04-19T23:35:16.880 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008566341f5c394255-006622ffb4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.926 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.927 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.927 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.927 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:16.927 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:16.927 DEBUG:boto:Method: GET 2024-04-19T23:35:16.927 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:16.927 DEBUG:boto:Data: 2024-04-19T23:35:16.928 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.928 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.928 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.928 DEBUG:boto:Params: {} 2024-04-19T23:35:16.928 DEBUG:boto:Token: None 2024-04-19T23:35:16.928 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.928 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU= 2024-04-19T23:35:16.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KMOif4BDmZ43sZF75FDsA993mrU=', 'Content-Length': '0'} 2024-04-19T23:35:16.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d63b721d58978143-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.978 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.979 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-20 zones={test-zone2, test-zone1} 2024-04-19T23:35:16.979 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-20 zones={test-zone3, test-zone1} 2024-04-19T23:35:16.979 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.979 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.979 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.979 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:35:16.979 DEBUG:boto:Data: 2024-04-19T23:35:16.979 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.979 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:16.980 DEBUG:boto:Port: 8001 2024-04-19T23:35:16.980 DEBUG:boto:Params: {} 2024-04-19T23:35:16.980 DEBUG:boto:Token: None 2024-04-19T23:35:16.980 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.980 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DoQq5YMuIp0ymYlzPYDvuLFR/2w= 2024-04-19T23:35:16.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DoQq5YMuIp0ymYlzPYDvuLFR/2w=', 'Content-Length': '0'} 2024-04-19T23:35:16.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011ee96d9c6297d0f-006622ffb4-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.983 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.983 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.983 DEBUG:boto:Method: HEAD 2024-04-19T23:35:16.983 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:35:16.983 DEBUG:boto:Data: 2024-04-19T23:35:16.983 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.983 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.984 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.984 DEBUG:boto:Params: {} 2024-04-19T23:35:16.984 DEBUG:boto:Token: None 2024-04-19T23:35:16.984 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/ 2024-04-19T23:35:16.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DoQq5YMuIp0ymYlzPYDvuLFR/2w= 2024-04-19T23:35:16.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DoQq5YMuIp0ymYlzPYDvuLFR/2w=', 'Content-Length': '0'} 2024-04-19T23:35:16.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df36fc2da8f3e4bf-006622ffb4-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.986 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:35:16.987 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.987 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.987 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.987 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.987 DEBUG:boto:Method: GET 2024-04-19T23:35:16.987 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.987 DEBUG:boto:Data: 2024-04-19T23:35:16.987 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.987 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:16.987 DEBUG:boto:Port: 8001 2024-04-19T23:35:16.987 DEBUG:boto:Params: {} 2024-04-19T23:35:16.988 DEBUG:boto:Token: None 2024-04-19T23:35:16.988 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/?versions 2024-04-19T23:35:16.988 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA= 2024-04-19T23:35:16.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA=', 'Content-Length': '0'} 2024-04-19T23:35:16.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000783a60eda07b925d-006622ffb4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.990 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.991 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:35:16.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:35:16.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:35:16.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:35:16.992 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:35:16.992 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.992 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.992 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.992 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.992 DEBUG:boto:Method: GET 2024-04-19T23:35:16.992 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.992 DEBUG:boto:Data: 2024-04-19T23:35:16.992 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.993 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:16.993 DEBUG:boto:Port: 8000 2024-04-19T23:35:16.993 DEBUG:boto:Params: {} 2024-04-19T23:35:16.993 DEBUG:boto:Token: None 2024-04-19T23:35:16.993 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/?versions 2024-04-19T23:35:16.993 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA= 2024-04-19T23:35:16.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA=', 'Content-Length': '0'} 2024-04-19T23:35:16.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005905abb4a282a33b-006622ffb4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:16.996 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:16.996 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:35:16.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:35:16.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:35:16.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:35:16.997 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:16.997 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:16.997 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.997 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.997 DEBUG:boto:Method: GET 2024-04-19T23:35:16.997 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:35:16.998 DEBUG:boto:Data: 2024-04-19T23:35:16.998 DEBUG:boto:Headers: {} 2024-04-19T23:35:16.998 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:16.998 DEBUG:boto:Port: 8001 2024-04-19T23:35:16.998 DEBUG:boto:Params: {} 2024-04-19T23:35:16.998 DEBUG:boto:Token: None 2024-04-19T23:35:16.998 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:16 GMT /aktlji-20/?versions 2024-04-19T23:35:16.998 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA= 2024-04-19T23:35:16.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DFAlzISYw2qNA9qzcE1Bwc2IfYA=', 'Content-Length': '0'} 2024-04-19T23:35:17.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cb1dd7d6fc1efeb-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.001 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.001 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.002 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.002 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:17.002 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:17.002 DEBUG:boto:Method: GET 2024-04-19T23:35:17.002 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:35:17.002 DEBUG:boto:Data: 2024-04-19T23:35:17.003 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.003 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.003 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.003 DEBUG:boto:Params: {} 2024-04-19T23:35:17.003 DEBUG:boto:Token: None 2024-04-19T23:35:17.003 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/?versions 2024-04-19T23:35:17.003 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0IS7RABJoQ+pyrbVwIIt+zO+IaU= 2024-04-19T23:35:17.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0IS7RABJoQ+pyrbVwIIt+zO+IaU=', 'Content-Length': '0'} 2024-04-19T23:35:17.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f7b7f1d2bbfed81-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.006 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:17.007 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:17.007 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:17.007 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.007 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.007 DEBUG:boto:Method: GET 2024-04-19T23:35:17.007 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.008 DEBUG:boto:Data: 2024-04-19T23:35:17.008 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.008 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.008 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.008 DEBUG:boto:Params: {} 2024-04-19T23:35:17.008 DEBUG:boto:Token: None 2024-04-19T23:35:17.008 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.009 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+vOgoAN7HsgmO3SenRVGBbKlD/U= 2024-04-19T23:35:17.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+vOgoAN7HsgmO3SenRVGBbKlD/U=', 'Content-Length': '0'} 2024-04-19T23:35:17.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012982d02bf05e097-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.011 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:17.011 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:17.011 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.011 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.011 DEBUG:boto:Method: GET 2024-04-19T23:35:17.011 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.011 DEBUG:boto:Data: 2024-04-19T23:35:17.011 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.011 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.012 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.012 DEBUG:boto:Params: {} 2024-04-19T23:35:17.012 DEBUG:boto:Token: None 2024-04-19T23:35:17.012 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.012 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+vOgoAN7HsgmO3SenRVGBbKlD/U= 2024-04-19T23:35:17.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+vOgoAN7HsgmO3SenRVGBbKlD/U=', 'Content-Length': '0'} 2024-04-19T23:35:17.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000065476cfcd36c669d-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.014 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:17.014 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:17.014 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.014 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.014 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.014 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.014 DEBUG:boto:Data: 2024-04-19T23:35:17.015 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.015 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.015 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.015 DEBUG:boto:Params: {} 2024-04-19T23:35:17.015 DEBUG:boto:Token: None 2024-04-19T23:35:17.015 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.015 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xC2UVd9Tdaf9ra9iVOMuq6iJ1D0= 2024-04-19T23:35:17.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xC2UVd9Tdaf9ra9iVOMuq6iJ1D0=', 'Content-Length': '0'} 2024-04-19T23:35:17.017 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000022f92c069d8eb48c-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.017 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:17.017 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:17.017 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.017 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.018 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.018 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.018 DEBUG:boto:Data: 2024-04-19T23:35:17.018 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.018 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.018 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.018 DEBUG:boto:Params: {} 2024-04-19T23:35:17.018 DEBUG:boto:Token: None 2024-04-19T23:35:17.018 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.018 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xC2UVd9Tdaf9ra9iVOMuq6iJ1D0= 2024-04-19T23:35:17.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xC2UVd9Tdaf9ra9iVOMuq6iJ1D0=', 'Content-Length': '0'} 2024-04-19T23:35:17.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000acf6057463eaefc7-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:17.021 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:17.021 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:17.021 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.021 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.021 DEBUG:boto:Method: GET 2024-04-19T23:35:17.021 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.021 DEBUG:boto:Data: 2024-04-19T23:35:17.022 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.022 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.022 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.022 DEBUG:boto:Params: {} 2024-04-19T23:35:17.022 DEBUG:boto:Token: None 2024-04-19T23:35:17.022 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.022 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+vOgoAN7HsgmO3SenRVGBbKlD/U= 2024-04-19T23:35:17.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+vOgoAN7HsgmO3SenRVGBbKlD/U=', 'Content-Length': '0'} 2024-04-19T23:35:17.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000157afd7ba543cfe1-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.024 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:17.024 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:17.024 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.024 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.025 DEBUG:boto:Method: GET 2024-04-19T23:35:17.025 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.025 DEBUG:boto:Data: 2024-04-19T23:35:17.025 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.025 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.025 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.025 DEBUG:boto:Params: {} 2024-04-19T23:35:17.025 DEBUG:boto:Token: None 2024-04-19T23:35:17.025 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.026 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+vOgoAN7HsgmO3SenRVGBbKlD/U= 2024-04-19T23:35:17.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+vOgoAN7HsgmO3SenRVGBbKlD/U=', 'Content-Length': '0'} 2024-04-19T23:35:17.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c03c626a77dab12-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.028 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.028 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.028 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:17.028 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:17.028 DEBUG:boto:Method: GET 2024-04-19T23:35:17.028 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:35:17.029 DEBUG:boto:Data: 2024-04-19T23:35:17.029 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.029 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.029 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.029 DEBUG:boto:Params: {} 2024-04-19T23:35:17.029 DEBUG:boto:Token: None 2024-04-19T23:35:17.029 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.029 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6891460c91103ff-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.074 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.075 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.075 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.075 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:17.075 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:17.075 DEBUG:boto:Method: GET 2024-04-19T23:35:17.075 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:35:17.075 DEBUG:boto:Data: 2024-04-19T23:35:17.075 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.075 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.076 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.076 DEBUG:boto:Params: {} 2024-04-19T23:35:17.076 DEBUG:boto:Token: None 2024-04-19T23:35:17.076 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.076 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ff273c9cd573f67-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.122 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.123 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:17.123 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:17.123 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.123 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:35:17.123 DEBUG:boto:Data: 2024-04-19T23:35:17.124 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.124 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.124 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.124 DEBUG:boto:Params: {} 2024-04-19T23:35:17.124 DEBUG:boto:Token: None 2024-04-19T23:35:17.124 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/dummy 2024-04-19T23:35:17.124 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ukIeYDiY2/lR5eI0NKtV37jSgkU= 2024-04-19T23:35:17.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ukIeYDiY2/lR5eI0NKtV37jSgkU=', 'Content-Length': '0'} 2024-04-19T23:35:17.127 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc939c6a2bb8d6f8-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.127 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:17.128 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:17.128 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.128 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:35:17.128 DEBUG:boto:Data: 2024-04-19T23:35:17.128 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.128 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.128 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.128 DEBUG:boto:Params: {} 2024-04-19T23:35:17.128 DEBUG:boto:Token: None 2024-04-19T23:35:17.129 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/dummy 2024-04-19T23:35:17.129 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ukIeYDiY2/lR5eI0NKtV37jSgkU= 2024-04-19T23:35:17.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ukIeYDiY2/lR5eI0NKtV37jSgkU=', 'Content-Length': '0'} 2024-04-19T23:35:17.131 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000864ee6feb305565f-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:17.132 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:17.132 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:17.132 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.132 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.132 DEBUG:boto:Method: GET 2024-04-19T23:35:17.132 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.132 DEBUG:boto:Data: 2024-04-19T23:35:17.133 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.133 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.133 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.133 DEBUG:boto:Params: {} 2024-04-19T23:35:17.133 DEBUG:boto:Token: None 2024-04-19T23:35:17.133 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.133 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+vOgoAN7HsgmO3SenRVGBbKlD/U= 2024-04-19T23:35:17.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+vOgoAN7HsgmO3SenRVGBbKlD/U=', 'Content-Length': '0'} 2024-04-19T23:35:17.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000076d9d9ccd56e9e69-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.135 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:17.136 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:17.136 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.136 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.136 DEBUG:boto:Method: GET 2024-04-19T23:35:17.136 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.136 DEBUG:boto:Data: 2024-04-19T23:35:17.136 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.136 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.136 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.136 DEBUG:boto:Params: {} 2024-04-19T23:35:17.137 DEBUG:boto:Token: None 2024-04-19T23:35:17.137 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:17.137 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+vOgoAN7HsgmO3SenRVGBbKlD/U= 2024-04-19T23:35:17.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+vOgoAN7HsgmO3SenRVGBbKlD/U=', 'Content-Length': '0'} 2024-04-19T23:35:17.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cdd81ec6983bdc82-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.139 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.139 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.139 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:17.139 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:17.140 DEBUG:boto:Method: GET 2024-04-19T23:35:17.140 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:35:17.140 DEBUG:boto:Data: 2024-04-19T23:35:17.140 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.140 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.140 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.140 DEBUG:boto:Params: {} 2024-04-19T23:35:17.140 DEBUG:boto:Token: None 2024-04-19T23:35:17.140 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.141 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068ddef5b61a4b6c2-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.186 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.187 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.187 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.187 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:17.187 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:17.187 DEBUG:boto:Method: GET 2024-04-19T23:35:17.187 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:35:17.187 DEBUG:boto:Data: 2024-04-19T23:35:17.187 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.187 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.188 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.188 DEBUG:boto:Params: {} 2024-04-19T23:35:17.188 DEBUG:boto:Token: None 2024-04-19T23:35:17.188 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.188 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa2cf36a31905662-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.238 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:17.239 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.239 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.239 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.239 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.239 DEBUG:boto:Method: GET 2024-04-19T23:35:17.239 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.239 DEBUG:boto:Data: 2024-04-19T23:35:17.240 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.240 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.240 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.240 DEBUG:boto:Params: {} 2024-04-19T23:35:17.240 DEBUG:boto:Token: None 2024-04-19T23:35:17.240 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.240 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RhUD0e8UVeittm/0aixYIkDfXcs= 2024-04-19T23:35:17.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RhUD0e8UVeittm/0aixYIkDfXcs=', 'Content-Length': '0'} 2024-04-19T23:35:17.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ae9072e660b38f8-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.282 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.282 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.282 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.283 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.283 DEBUG:boto:Method: GET 2024-04-19T23:35:17.283 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.283 DEBUG:boto:Data: 2024-04-19T23:35:17.283 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.283 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.283 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.283 DEBUG:boto:Params: {} 2024-04-19T23:35:17.283 DEBUG:boto:Token: None 2024-04-19T23:35:17.284 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.284 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RhUD0e8UVeittm/0aixYIkDfXcs= 2024-04-19T23:35:17.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RhUD0e8UVeittm/0aixYIkDfXcs=', 'Content-Length': '0'} 2024-04-19T23:35:17.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3e4b31087bff9fb-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.286 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.286 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.287 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.287 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.287 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.287 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.287 DEBUG:boto:Data: 2024-04-19T23:35:17.287 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.287 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.287 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.287 DEBUG:boto:Params: {} 2024-04-19T23:35:17.287 DEBUG:boto:Token: None 2024-04-19T23:35:17.288 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.288 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RRyI71WHA2V1GcfsCCOzXo8ZZ5g= 2024-04-19T23:35:17.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RRyI71WHA2V1GcfsCCOzXo8ZZ5g=', 'Content-Length': '0'} 2024-04-19T23:35:17.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eafe07b3eb631935-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.290 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.290 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.290 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.290 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.290 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.291 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.291 DEBUG:boto:Data: 2024-04-19T23:35:17.291 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.291 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.291 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.291 DEBUG:boto:Params: {} 2024-04-19T23:35:17.291 DEBUG:boto:Token: None 2024-04-19T23:35:17.291 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.291 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RRyI71WHA2V1GcfsCCOzXo8ZZ5g= 2024-04-19T23:35:17.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RRyI71WHA2V1GcfsCCOzXo8ZZ5g=', 'Content-Length': '0'} 2024-04-19T23:35:17.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e85db798200d9a6b-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:17.294 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.294 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.294 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.294 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.294 DEBUG:boto:Method: GET 2024-04-19T23:35:17.294 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.294 DEBUG:boto:Data: 2024-04-19T23:35:17.294 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.295 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.295 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.295 DEBUG:boto:Params: {} 2024-04-19T23:35:17.295 DEBUG:boto:Token: None 2024-04-19T23:35:17.295 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.295 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RhUD0e8UVeittm/0aixYIkDfXcs= 2024-04-19T23:35:17.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RhUD0e8UVeittm/0aixYIkDfXcs=', 'Content-Length': '0'} 2024-04-19T23:35:17.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc9ba20ee8edf81c-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.339 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.339 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.339 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.339 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.339 DEBUG:boto:Method: GET 2024-04-19T23:35:17.339 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.340 DEBUG:boto:Data: 2024-04-19T23:35:17.340 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.340 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.340 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.340 DEBUG:boto:Params: {} 2024-04-19T23:35:17.340 DEBUG:boto:Token: None 2024-04-19T23:35:17.340 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.340 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RhUD0e8UVeittm/0aixYIkDfXcs= 2024-04-19T23:35:17.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RhUD0e8UVeittm/0aixYIkDfXcs=', 'Content-Length': '0'} 2024-04-19T23:35:17.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004c808da3e5a6391-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.343 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.343 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.343 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:17.343 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:17.344 DEBUG:boto:Method: GET 2024-04-19T23:35:17.344 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:17.344 DEBUG:boto:Data: 2024-04-19T23:35:17.344 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.344 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.344 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.344 DEBUG:boto:Params: {} 2024-04-19T23:35:17.344 DEBUG:boto:Token: None 2024-04-19T23:35:17.345 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.345 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cca52cc72dac28ec-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.390 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.391 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.391 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.391 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:17.391 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:17.391 DEBUG:boto:Method: GET 2024-04-19T23:35:17.391 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:17.391 DEBUG:boto:Data: 2024-04-19T23:35:17.391 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.391 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.392 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.392 DEBUG:boto:Params: {} 2024-04-19T23:35:17.392 DEBUG:boto:Token: None 2024-04-19T23:35:17.392 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.392 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3c4f551a07b59d3-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.395 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.395 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.395 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.395 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.395 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:35:17.395 DEBUG:boto:Data: 2024-04-19T23:35:17.395 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.396 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.396 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.396 DEBUG:boto:Params: {} 2024-04-19T23:35:17.396 DEBUG:boto:Token: None 2024-04-19T23:35:17.396 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:35:17.396 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+oFvI/dpHmdf7G8fO69d46uLilo= 2024-04-19T23:35:17.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+oFvI/dpHmdf7G8fO69d46uLilo=', 'Content-Length': '0'} 2024-04-19T23:35:17.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bae9f4a1bfef225-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.399 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.399 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.399 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.399 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:35:17.399 DEBUG:boto:Data: 2024-04-19T23:35:17.399 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.400 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.400 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.400 DEBUG:boto:Params: {} 2024-04-19T23:35:17.400 DEBUG:boto:Token: None 2024-04-19T23:35:17.400 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:35:17.400 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+oFvI/dpHmdf7G8fO69d46uLilo= 2024-04-19T23:35:17.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+oFvI/dpHmdf7G8fO69d46uLilo=', 'Content-Length': '0'} 2024-04-19T23:35:17.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d498593f69b8b569-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:17.403 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.403 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.403 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.403 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.403 DEBUG:boto:Method: GET 2024-04-19T23:35:17.403 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.403 DEBUG:boto:Data: 2024-04-19T23:35:17.404 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.404 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.404 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.404 DEBUG:boto:Params: {} 2024-04-19T23:35:17.405 DEBUG:boto:Token: None 2024-04-19T23:35:17.405 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.405 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RhUD0e8UVeittm/0aixYIkDfXcs= 2024-04-19T23:35:17.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RhUD0e8UVeittm/0aixYIkDfXcs=', 'Content-Length': '0'} 2024-04-19T23:35:17.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007114fef5ee8b5158-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.450 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.451 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:17.451 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.451 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.451 DEBUG:boto:Method: GET 2024-04-19T23:35:17.451 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.451 DEBUG:boto:Data: 2024-04-19T23:35:17.451 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.451 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.451 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.451 DEBUG:boto:Params: {} 2024-04-19T23:35:17.452 DEBUG:boto:Token: None 2024-04-19T23:35:17.452 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:17.452 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RhUD0e8UVeittm/0aixYIkDfXcs= 2024-04-19T23:35:17.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RhUD0e8UVeittm/0aixYIkDfXcs=', 'Content-Length': '0'} 2024-04-19T23:35:17.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061d67d5a8a273842-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.454 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.454 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.454 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:17.455 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:17.455 DEBUG:boto:Method: GET 2024-04-19T23:35:17.455 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:17.455 DEBUG:boto:Data: 2024-04-19T23:35:17.455 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.455 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.455 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.455 DEBUG:boto:Params: {} 2024-04-19T23:35:17.455 DEBUG:boto:Token: None 2024-04-19T23:35:17.455 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.455 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004242d44cd9f5ce2c-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.502 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.503 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.503 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.503 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:17.503 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:17.503 DEBUG:boto:Method: GET 2024-04-19T23:35:17.503 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:17.504 DEBUG:boto:Data: 2024-04-19T23:35:17.504 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.504 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.504 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.504 DEBUG:boto:Params: {} 2024-04-19T23:35:17.504 DEBUG:boto:Token: None 2024-04-19T23:35:17.504 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.504 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009852040affc933f6-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.507 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:17.508 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.508 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.508 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.508 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.508 DEBUG:boto:Method: GET 2024-04-19T23:35:17.508 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.508 DEBUG:boto:Data: 2024-04-19T23:35:17.509 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.509 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.509 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.509 DEBUG:boto:Params: {} 2024-04-19T23:35:17.509 DEBUG:boto:Token: None 2024-04-19T23:35:17.509 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.509 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ESx3vrr+l4p+jdDpHd6ll6XMTrk= 2024-04-19T23:35:17.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ESx3vrr+l4p+jdDpHd6ll6XMTrk=', 'Content-Length': '0'} 2024-04-19T23:35:17.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1da1d5f86171fb9-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.511 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.512 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.512 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.512 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.512 DEBUG:boto:Method: GET 2024-04-19T23:35:17.512 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.512 DEBUG:boto:Data: 2024-04-19T23:35:17.512 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.512 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.512 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.513 DEBUG:boto:Params: {} 2024-04-19T23:35:17.513 DEBUG:boto:Token: None 2024-04-19T23:35:17.513 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.513 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ESx3vrr+l4p+jdDpHd6ll6XMTrk= 2024-04-19T23:35:17.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ESx3vrr+l4p+jdDpHd6ll6XMTrk=', 'Content-Length': '0'} 2024-04-19T23:35:17.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082be29d89a6433bb-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.558 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.559 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.559 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.559 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.559 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.559 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.559 DEBUG:boto:Data: 2024-04-19T23:35:17.559 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.559 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.559 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.559 DEBUG:boto:Params: {} 2024-04-19T23:35:17.559 DEBUG:boto:Token: None 2024-04-19T23:35:17.560 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ShhT7gYCzdsWn6vsC40HBHr7Fz8= 2024-04-19T23:35:17.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ShhT7gYCzdsWn6vsC40HBHr7Fz8=', 'Content-Length': '0'} 2024-04-19T23:35:17.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077f945a2bcb24d6c-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.562 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.562 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.562 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.562 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.563 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.563 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.563 DEBUG:boto:Data: 2024-04-19T23:35:17.563 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.563 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.563 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.563 DEBUG:boto:Params: {} 2024-04-19T23:35:17.563 DEBUG:boto:Token: None 2024-04-19T23:35:17.563 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.563 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ShhT7gYCzdsWn6vsC40HBHr7Fz8= 2024-04-19T23:35:17.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ShhT7gYCzdsWn6vsC40HBHr7Fz8=', 'Content-Length': '0'} 2024-04-19T23:35:17.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062e0985702e23bd0-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:17.566 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.566 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.566 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.566 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.566 DEBUG:boto:Method: GET 2024-04-19T23:35:17.566 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.566 DEBUG:boto:Data: 2024-04-19T23:35:17.566 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.566 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.566 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.566 DEBUG:boto:Params: {} 2024-04-19T23:35:17.567 DEBUG:boto:Token: None 2024-04-19T23:35:17.567 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.567 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ESx3vrr+l4p+jdDpHd6ll6XMTrk= 2024-04-19T23:35:17.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ESx3vrr+l4p+jdDpHd6ll6XMTrk=', 'Content-Length': '0'} 2024-04-19T23:35:17.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ff0cd4694a71e59-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.610 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.611 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.611 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.611 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.611 DEBUG:boto:Method: GET 2024-04-19T23:35:17.611 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.611 DEBUG:boto:Data: 2024-04-19T23:35:17.611 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.611 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.611 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.611 DEBUG:boto:Params: {} 2024-04-19T23:35:17.612 DEBUG:boto:Token: None 2024-04-19T23:35:17.612 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.612 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ESx3vrr+l4p+jdDpHd6ll6XMTrk= 2024-04-19T23:35:17.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ESx3vrr+l4p+jdDpHd6ll6XMTrk=', 'Content-Length': '0'} 2024-04-19T23:35:17.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000255c73691bc8c0f8-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.658 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.659 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.659 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:17.659 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:17.659 DEBUG:boto:Method: GET 2024-04-19T23:35:17.659 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:17.659 DEBUG:boto:Data: 2024-04-19T23:35:17.659 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.659 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.659 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.659 DEBUG:boto:Params: {} 2024-04-19T23:35:17.660 DEBUG:boto:Token: None 2024-04-19T23:35:17.660 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015cb9e847ef9f7ce-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.663 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.663 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.663 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.663 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:17.663 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:17.663 DEBUG:boto:Method: GET 2024-04-19T23:35:17.664 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:17.664 DEBUG:boto:Data: 2024-04-19T23:35:17.664 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.664 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.664 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.664 DEBUG:boto:Params: {} 2024-04-19T23:35:17.664 DEBUG:boto:Token: None 2024-04-19T23:35:17.664 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.664 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f27357511b0f612-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.667 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.667 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.668 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.668 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.668 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:35:17.668 DEBUG:boto:Data: 2024-04-19T23:35:17.668 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.668 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.668 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.668 DEBUG:boto:Params: {} 2024-04-19T23:35:17.668 DEBUG:boto:Token: None 2024-04-19T23:35:17.668 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:35:17.669 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ou1oa2nGEYsOZ9yZk1jiAyDKi+I= 2024-04-19T23:35:17.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ou1oa2nGEYsOZ9yZk1jiAyDKi+I=', 'Content-Length': '0'} 2024-04-19T23:35:17.671 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9bde9f7dfa25566-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.671 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.671 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.671 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.672 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:35:17.672 DEBUG:boto:Data: 2024-04-19T23:35:17.672 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.672 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.672 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.672 DEBUG:boto:Params: {} 2024-04-19T23:35:17.672 DEBUG:boto:Token: None 2024-04-19T23:35:17.672 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:35:17.672 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ou1oa2nGEYsOZ9yZk1jiAyDKi+I= 2024-04-19T23:35:17.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ou1oa2nGEYsOZ9yZk1jiAyDKi+I=', 'Content-Length': '0'} 2024-04-19T23:35:17.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a848492d2259e7cf-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:17.675 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.675 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.675 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.675 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.675 DEBUG:boto:Method: GET 2024-04-19T23:35:17.676 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.676 DEBUG:boto:Data: 2024-04-19T23:35:17.676 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.676 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.676 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.676 DEBUG:boto:Params: {} 2024-04-19T23:35:17.676 DEBUG:boto:Token: None 2024-04-19T23:35:17.676 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ESx3vrr+l4p+jdDpHd6ll6XMTrk= 2024-04-19T23:35:17.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ESx3vrr+l4p+jdDpHd6ll6XMTrk=', 'Content-Length': '0'} 2024-04-19T23:35:17.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed34aa27844f2b5d-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.718 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.718 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:17.719 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.719 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.719 DEBUG:boto:Method: GET 2024-04-19T23:35:17.719 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.719 DEBUG:boto:Data: 2024-04-19T23:35:17.719 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.719 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.719 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.719 DEBUG:boto:Params: {} 2024-04-19T23:35:17.719 DEBUG:boto:Token: None 2024-04-19T23:35:17.720 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:17.720 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ESx3vrr+l4p+jdDpHd6ll6XMTrk= 2024-04-19T23:35:17.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ESx3vrr+l4p+jdDpHd6ll6XMTrk=', 'Content-Length': '0'} 2024-04-19T23:35:17.722 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001426b2442068a9f9-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.762 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.762 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.763 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:17.763 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:17.763 DEBUG:boto:Method: GET 2024-04-19T23:35:17.763 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:17.763 DEBUG:boto:Data: 2024-04-19T23:35:17.763 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.763 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.763 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.763 DEBUG:boto:Params: {} 2024-04-19T23:35:17.763 DEBUG:boto:Token: None 2024-04-19T23:35:17.764 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.764 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005361f34639cec201-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.766 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.767 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.767 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.767 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:17.767 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:17.767 DEBUG:boto:Method: GET 2024-04-19T23:35:17.767 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:17.767 DEBUG:boto:Data: 2024-04-19T23:35:17.767 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.767 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.768 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.768 DEBUG:boto:Params: {} 2024-04-19T23:35:17.768 DEBUG:boto:Token: None 2024-04-19T23:35:17.768 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.768 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc0c109107fad195-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.771 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:17.771 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.771 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.771 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.771 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.772 DEBUG:boto:Method: GET 2024-04-19T23:35:17.772 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.772 DEBUG:boto:Data: 2024-04-19T23:35:17.772 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.772 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.772 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.772 DEBUG:boto:Params: {} 2024-04-19T23:35:17.772 DEBUG:boto:Token: None 2024-04-19T23:35:17.772 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.772 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:F6zVcwQgnNkKDUIlxxwgf8CXhKY= 2024-04-19T23:35:17.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:F6zVcwQgnNkKDUIlxxwgf8CXhKY=', 'Content-Length': '0'} 2024-04-19T23:35:17.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d943f27c92dd4ae-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.775 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.775 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.775 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.775 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.775 DEBUG:boto:Method: GET 2024-04-19T23:35:17.775 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.775 DEBUG:boto:Data: 2024-04-19T23:35:17.775 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.775 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.776 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.776 DEBUG:boto:Params: {} 2024-04-19T23:35:17.776 DEBUG:boto:Token: None 2024-04-19T23:35:17.776 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.776 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:F6zVcwQgnNkKDUIlxxwgf8CXhKY= 2024-04-19T23:35:17.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:F6zVcwQgnNkKDUIlxxwgf8CXhKY=', 'Content-Length': '0'} 2024-04-19T23:35:17.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0e20bd22d6692ea-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.778 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.778 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.778 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.779 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.779 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.779 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.779 DEBUG:boto:Data: 2024-04-19T23:35:17.779 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.779 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.779 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.779 DEBUG:boto:Params: {} 2024-04-19T23:35:17.779 DEBUG:boto:Token: None 2024-04-19T23:35:17.779 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.780 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3bhgVAMRgEG85ccaFa7Aldl7HgY= 2024-04-19T23:35:17.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3bhgVAMRgEG85ccaFa7Aldl7HgY=', 'Content-Length': '0'} 2024-04-19T23:35:17.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2d127d477daa9a8-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.781 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.782 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.782 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.782 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.782 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.782 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.782 DEBUG:boto:Data: 2024-04-19T23:35:17.782 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.782 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.782 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.782 DEBUG:boto:Params: {} 2024-04-19T23:35:17.782 DEBUG:boto:Token: None 2024-04-19T23:35:17.783 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.783 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3bhgVAMRgEG85ccaFa7Aldl7HgY= 2024-04-19T23:35:17.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3bhgVAMRgEG85ccaFa7Aldl7HgY=', 'Content-Length': '0'} 2024-04-19T23:35:17.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c789e4f83e0a74d9-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:17.785 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.785 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.785 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.785 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.786 DEBUG:boto:Method: GET 2024-04-19T23:35:17.786 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.786 DEBUG:boto:Data: 2024-04-19T23:35:17.786 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.786 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.786 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.786 DEBUG:boto:Params: {} 2024-04-19T23:35:17.786 DEBUG:boto:Token: None 2024-04-19T23:35:17.787 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.787 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:F6zVcwQgnNkKDUIlxxwgf8CXhKY= 2024-04-19T23:35:17.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:F6zVcwQgnNkKDUIlxxwgf8CXhKY=', 'Content-Length': '0'} 2024-04-19T23:35:17.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5c9f3bc641cb1a5-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.789 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.789 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.789 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.789 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.789 DEBUG:boto:Method: GET 2024-04-19T23:35:17.789 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.789 DEBUG:boto:Data: 2024-04-19T23:35:17.789 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.789 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.790 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.790 DEBUG:boto:Params: {} 2024-04-19T23:35:17.790 DEBUG:boto:Token: None 2024-04-19T23:35:17.790 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.790 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:F6zVcwQgnNkKDUIlxxwgf8CXhKY= 2024-04-19T23:35:17.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:F6zVcwQgnNkKDUIlxxwgf8CXhKY=', 'Content-Length': '0'} 2024-04-19T23:35:17.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e94ae398c2209503-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.834 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.835 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.835 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:17.835 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:17.835 DEBUG:boto:Method: GET 2024-04-19T23:35:17.835 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:17.835 DEBUG:boto:Data: 2024-04-19T23:35:17.835 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.835 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.835 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.835 DEBUG:boto:Params: {} 2024-04-19T23:35:17.836 DEBUG:boto:Token: None 2024-04-19T23:35:17.836 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.836 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbc38f8f19015fe4-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.839 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.839 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.839 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.839 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:17.839 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:17.839 DEBUG:boto:Method: GET 2024-04-19T23:35:17.839 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:17.840 DEBUG:boto:Data: 2024-04-19T23:35:17.840 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.840 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.840 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.840 DEBUG:boto:Params: {} 2024-04-19T23:35:17.840 DEBUG:boto:Token: None 2024-04-19T23:35:17.840 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.840 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067c2fec9c6c0756b-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.890 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.891 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.891 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.891 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.891 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:35:17.891 DEBUG:boto:Data: 2024-04-19T23:35:17.891 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.892 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.892 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.892 DEBUG:boto:Params: {} 2024-04-19T23:35:17.892 DEBUG:boto:Token: None 2024-04-19T23:35:17.892 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:35:17.892 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WsRN8qCBu9R3dUJxjgYUBr+36G8= 2024-04-19T23:35:17.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WsRN8qCBu9R3dUJxjgYUBr+36G8=', 'Content-Length': '0'} 2024-04-19T23:35:17.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038ac7576a1f696d4-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.896 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.896 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.896 DEBUG:boto:Method: HEAD 2024-04-19T23:35:17.896 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:35:17.896 DEBUG:boto:Data: 2024-04-19T23:35:17.896 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.896 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.896 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.896 DEBUG:boto:Params: {} 2024-04-19T23:35:17.896 DEBUG:boto:Token: None 2024-04-19T23:35:17.897 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:35:17.897 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WsRN8qCBu9R3dUJxjgYUBr+36G8= 2024-04-19T23:35:17.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WsRN8qCBu9R3dUJxjgYUBr+36G8=', 'Content-Length': '0'} 2024-04-19T23:35:17.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e4ae5e6554c2b52-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:17.900 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.900 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.901 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.901 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.901 DEBUG:boto:Method: GET 2024-04-19T23:35:17.901 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.901 DEBUG:boto:Data: 2024-04-19T23:35:17.901 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.901 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.901 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.901 DEBUG:boto:Params: {} 2024-04-19T23:35:17.901 DEBUG:boto:Token: None 2024-04-19T23:35:17.901 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.902 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:F6zVcwQgnNkKDUIlxxwgf8CXhKY= 2024-04-19T23:35:17.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:F6zVcwQgnNkKDUIlxxwgf8CXhKY=', 'Content-Length': '0'} 2024-04-19T23:35:17.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab4bc706cac29845-006622ffb5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.950 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.951 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:17.951 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.951 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.951 DEBUG:boto:Method: GET 2024-04-19T23:35:17.951 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.951 DEBUG:boto:Data: 2024-04-19T23:35:17.951 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.951 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:17.951 DEBUG:boto:Port: 8000 2024-04-19T23:35:17.951 DEBUG:boto:Params: {} 2024-04-19T23:35:17.952 DEBUG:boto:Token: None 2024-04-19T23:35:17.952 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:17.952 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:F6zVcwQgnNkKDUIlxxwgf8CXhKY= 2024-04-19T23:35:17.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:F6zVcwQgnNkKDUIlxxwgf8CXhKY=', 'Content-Length': '0'} 2024-04-19T23:35:17.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a4d17d2ed7c3565-006622ffb5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.994 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.995 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.995 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:17.995 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:17.995 DEBUG:boto:Method: GET 2024-04-19T23:35:17.995 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:17.995 DEBUG:boto:Data: 2024-04-19T23:35:17.995 DEBUG:boto:Headers: {} 2024-04-19T23:35:17.995 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:17.995 DEBUG:boto:Port: 8001 2024-04-19T23:35:17.995 DEBUG:boto:Params: {} 2024-04-19T23:35:17.996 DEBUG:boto:Token: None 2024-04-19T23:35:17.996 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:17 GMT /aktlji-20/ 2024-04-19T23:35:17.996 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM= 2024-04-19T23:35:17.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Q0EWiP5hlvjqLn5w7jajkocPPM=', 'Content-Length': '0'} 2024-04-19T23:35:17.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b14c6a4ada7dddbb-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:17.998 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:17.999 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:17.999 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:17.999 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:17.999 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.000 DEBUG:boto:Method: GET 2024-04-19T23:35:18.000 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.000 DEBUG:boto:Data: 2024-04-19T23:35:18.000 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.000 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:18.000 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.000 DEBUG:boto:Params: {} 2024-04-19T23:35:18.001 DEBUG:boto:Token: None 2024-04-19T23:35:18.001 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.001 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d517f135d95dc6ad-006622ffb6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.004 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.005 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-20 zones={test-zone3, test-zone1} 2024-04-19T23:35:18.005 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-20 zones={test-zone3, test-zone2} 2024-04-19T23:35:18.005 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.005 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.005 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.005 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:35:18.005 DEBUG:boto:Data: 2024-04-19T23:35:18.005 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.005 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.005 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.006 DEBUG:boto:Params: {} 2024-04-19T23:35:18.006 DEBUG:boto:Token: None 2024-04-19T23:35:18.006 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.006 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8IBhfaxR/W3FfpOpkXIQQPLoeMY= 2024-04-19T23:35:18.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8IBhfaxR/W3FfpOpkXIQQPLoeMY=', 'Content-Length': '0'} 2024-04-19T23:35:18.008 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e82e80af694ec186-006622ffb6-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.009 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.009 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.009 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.009 DEBUG:boto:Path: /aktlji-20/ 2024-04-19T23:35:18.009 DEBUG:boto:Data: 2024-04-19T23:35:18.009 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.009 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.009 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.009 DEBUG:boto:Params: {} 2024-04-19T23:35:18.009 DEBUG:boto:Token: None 2024-04-19T23:35:18.010 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.010 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8IBhfaxR/W3FfpOpkXIQQPLoeMY= 2024-04-19T23:35:18.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8IBhfaxR/W3FfpOpkXIQQPLoeMY=', 'Content-Length': '0'} 2024-04-19T23:35:18.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080239836c130323b-006622ffb6-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.012 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:35:18.013 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.013 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.013 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:18.013 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:18.013 DEBUG:boto:Method: GET 2024-04-19T23:35:18.013 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:35:18.013 DEBUG:boto:Data: 2024-04-19T23:35:18.013 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.013 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.013 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.013 DEBUG:boto:Params: {} 2024-04-19T23:35:18.014 DEBUG:boto:Token: None 2024-04-19T23:35:18.014 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/?versions 2024-04-19T23:35:18.014 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o+2BpsDLoqRJneYo28IsOzw1mA0= 2024-04-19T23:35:18.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o+2BpsDLoqRJneYo28IsOzw1mA0=', 'Content-Length': '0'} 2024-04-19T23:35:18.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002adf173d8d61df8c-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.016 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.017 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:35:18.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:35:18.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:35:18.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:35:18.017 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:35:18.017 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.017 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.017 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:18.018 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:18.018 DEBUG:boto:Method: GET 2024-04-19T23:35:18.018 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:35:18.018 DEBUG:boto:Data: 2024-04-19T23:35:18.018 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.018 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.018 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.018 DEBUG:boto:Params: {} 2024-04-19T23:35:18.018 DEBUG:boto:Token: None 2024-04-19T23:35:18.018 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/?versions 2024-04-19T23:35:18.019 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o+2BpsDLoqRJneYo28IsOzw1mA0= 2024-04-19T23:35:18.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o+2BpsDLoqRJneYo28IsOzw1mA0=', 'Content-Length': '0'} 2024-04-19T23:35:18.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df74f55cf160f994-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.021 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.022 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:35:18.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:35:18.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:35:18.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:35:18.022 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.022 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.022 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:18.022 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:18.022 DEBUG:boto:Method: GET 2024-04-19T23:35:18.023 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:35:18.023 DEBUG:boto:Data: 2024-04-19T23:35:18.023 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.023 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.023 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.023 DEBUG:boto:Params: {} 2024-04-19T23:35:18.023 DEBUG:boto:Token: None 2024-04-19T23:35:18.023 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/?versions 2024-04-19T23:35:18.023 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o+2BpsDLoqRJneYo28IsOzw1mA0= 2024-04-19T23:35:18.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o+2BpsDLoqRJneYo28IsOzw1mA0=', 'Content-Length': '0'} 2024-04-19T23:35:18.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cccbba94f36190a0-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.027 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.027 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.027 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.027 DEBUG:boto:path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:18.027 DEBUG:boto:auth_path=/aktlji-20/?versions&max-keys=999 2024-04-19T23:35:18.027 DEBUG:boto:Method: GET 2024-04-19T23:35:18.027 DEBUG:boto:Path: /aktlji-20/?versions&max-keys=999 2024-04-19T23:35:18.028 DEBUG:boto:Data: 2024-04-19T23:35:18.028 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.028 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.028 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.028 DEBUG:boto:Params: {} 2024-04-19T23:35:18.028 DEBUG:boto:Token: None 2024-04-19T23:35:18.028 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/?versions 2024-04-19T23:35:18.028 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o+2BpsDLoqRJneYo28IsOzw1mA0= 2024-04-19T23:35:18.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o+2BpsDLoqRJneYo28IsOzw1mA0=', 'Content-Length': '0'} 2024-04-19T23:35:18.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065e8ab1904d0f100-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.031 DEBUG:boto:b'aktlji-20999falsedummyiRrRmFZs3bOUPbgnHKMwQYejjeBfjb3true2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1cbhikBGTloAAkVRyBUG6AxoyB1Rg35.true2024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2Jdm81gPfYI2WgTFtftcs7OV2uk7V4xztrue2024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3htNuK4kZrW.j1OByzRcdsecrUSmYZCjtrue2024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:18.032 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:18.032 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:18.032 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.032 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.032 DEBUG:boto:Method: GET 2024-04-19T23:35:18.032 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.032 DEBUG:boto:Data: 2024-04-19T23:35:18.032 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.032 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.032 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.032 DEBUG:boto:Params: {} 2024-04-19T23:35:18.033 DEBUG:boto:Token: None 2024-04-19T23:35:18.033 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.033 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N9A5EiIOAZfqo9G3Di4v+irDlok= 2024-04-19T23:35:18.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N9A5EiIOAZfqo9G3Di4v+irDlok=', 'Content-Length': '0'} 2024-04-19T23:35:18.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000090940ff58ef99d4e-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.035 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:18.035 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:18.035 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.036 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.036 DEBUG:boto:Method: GET 2024-04-19T23:35:18.036 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.036 DEBUG:boto:Data: 2024-04-19T23:35:18.036 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.036 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.036 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.036 DEBUG:boto:Params: {} 2024-04-19T23:35:18.036 DEBUG:boto:Token: None 2024-04-19T23:35:18.036 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.037 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N9A5EiIOAZfqo9G3Di4v+irDlok= 2024-04-19T23:35:18.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N9A5EiIOAZfqo9G3Di4v+irDlok=', 'Content-Length': '0'} 2024-04-19T23:35:18.039 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007108fa44c4ab6322-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.039 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:18.039 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:18.039 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.039 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.039 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.039 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.039 DEBUG:boto:Data: 2024-04-19T23:35:18.040 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.040 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.040 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.040 DEBUG:boto:Params: {} 2024-04-19T23:35:18.040 DEBUG:boto:Token: None 2024-04-19T23:35:18.040 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.040 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NiYoiIWmA833kv1P/kQwuygoA90= 2024-04-19T23:35:18.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NiYoiIWmA833kv1P/kQwuygoA90=', 'Content-Length': '0'} 2024-04-19T23:35:18.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd9bcb90a8372ca8-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.042 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:18.042 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:18.043 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.043 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.043 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.043 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.043 DEBUG:boto:Data: 2024-04-19T23:35:18.043 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.043 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.043 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.043 DEBUG:boto:Params: {} 2024-04-19T23:35:18.044 DEBUG:boto:Token: None 2024-04-19T23:35:18.044 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.044 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NiYoiIWmA833kv1P/kQwuygoA90= 2024-04-19T23:35:18.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NiYoiIWmA833kv1P/kQwuygoA90=', 'Content-Length': '0'} 2024-04-19T23:35:18.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000031bf175d2a320e34-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:18.046 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:18.046 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:18.046 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.046 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.046 DEBUG:boto:Method: GET 2024-04-19T23:35:18.047 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.047 DEBUG:boto:Data: 2024-04-19T23:35:18.047 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.047 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.047 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.047 DEBUG:boto:Params: {} 2024-04-19T23:35:18.047 DEBUG:boto:Token: None 2024-04-19T23:35:18.047 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.047 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N9A5EiIOAZfqo9G3Di4v+irDlok= 2024-04-19T23:35:18.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N9A5EiIOAZfqo9G3Di4v+irDlok=', 'Content-Length': '0'} 2024-04-19T23:35:18.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed6d0585eb5af068-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.050 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:18.050 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:18.050 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.050 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.050 DEBUG:boto:Method: GET 2024-04-19T23:35:18.050 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.050 DEBUG:boto:Data: 2024-04-19T23:35:18.050 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.050 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.051 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.051 DEBUG:boto:Params: {} 2024-04-19T23:35:18.051 DEBUG:boto:Token: None 2024-04-19T23:35:18.051 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.051 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N9A5EiIOAZfqo9G3Di4v+irDlok= 2024-04-19T23:35:18.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N9A5EiIOAZfqo9G3Di4v+irDlok=', 'Content-Length': '0'} 2024-04-19T23:35:18.053 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e09eba29a728c508-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.053 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.054 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.054 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:18.054 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:18.054 DEBUG:boto:Method: GET 2024-04-19T23:35:18.054 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:35:18.054 DEBUG:boto:Data: 2024-04-19T23:35:18.055 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.055 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.055 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.055 DEBUG:boto:Params: {} 2024-04-19T23:35:18.055 DEBUG:boto:Token: None 2024-04-19T23:35:18.055 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.056 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee08ae79400b6a3-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.102 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.103 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.103 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.103 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:18.103 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:18.103 DEBUG:boto:Method: GET 2024-04-19T23:35:18.103 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:35:18.103 DEBUG:boto:Data: 2024-04-19T23:35:18.103 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.103 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.104 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.104 DEBUG:boto:Params: {} 2024-04-19T23:35:18.104 DEBUG:boto:Token: None 2024-04-19T23:35:18.104 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1b7b2f79fb1c9a5-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.114 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.115 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:18.115 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:18.115 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.115 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:35:18.115 DEBUG:boto:Data: 2024-04-19T23:35:18.115 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.116 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.116 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.116 DEBUG:boto:Params: {} 2024-04-19T23:35:18.116 DEBUG:boto:Token: None 2024-04-19T23:35:18.116 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/dummy 2024-04-19T23:35:18.116 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7qSjr9vatmaUIccHW8O3HnvRH7E= 2024-04-19T23:35:18.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7qSjr9vatmaUIccHW8O3HnvRH7E=', 'Content-Length': '0'} 2024-04-19T23:35:18.119 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f66817e67e788b3-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.119 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:18.119 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:18.119 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.119 DEBUG:boto:Path: /aktlji-20/dummy 2024-04-19T23:35:18.119 DEBUG:boto:Data: 2024-04-19T23:35:18.120 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.120 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.120 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.120 DEBUG:boto:Params: {} 2024-04-19T23:35:18.120 DEBUG:boto:Token: None 2024-04-19T23:35:18.120 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/dummy 2024-04-19T23:35:18.120 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7qSjr9vatmaUIccHW8O3HnvRH7E= 2024-04-19T23:35:18.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7qSjr9vatmaUIccHW8O3HnvRH7E=', 'Content-Length': '0'} 2024-04-19T23:35:18.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c5cbfce08f83710-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:18.123 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:18.123 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:18.123 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.123 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.124 DEBUG:boto:Method: GET 2024-04-19T23:35:18.124 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.124 DEBUG:boto:Data: 2024-04-19T23:35:18.124 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.124 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.124 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.124 DEBUG:boto:Params: {} 2024-04-19T23:35:18.124 DEBUG:boto:Token: None 2024-04-19T23:35:18.125 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.125 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N9A5EiIOAZfqo9G3Di4v+irDlok= 2024-04-19T23:35:18.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N9A5EiIOAZfqo9G3Di4v+irDlok=', 'Content-Length': '0'} 2024-04-19T23:35:18.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d88c86a38188045f-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.127 DEBUG:boto:path=/aktlji-20/dummy 2024-04-19T23:35:18.127 DEBUG:boto:auth_path=/aktlji-20/dummy 2024-04-19T23:35:18.127 DEBUG:boto:path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.127 DEBUG:boto:auth_path=/aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.127 DEBUG:boto:Method: GET 2024-04-19T23:35:18.127 DEBUG:boto:Path: /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.127 DEBUG:boto:Data: 2024-04-19T23:35:18.128 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.128 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.128 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.128 DEBUG:boto:Params: {} 2024-04-19T23:35:18.128 DEBUG:boto:Token: None 2024-04-19T23:35:18.128 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/dummy?versionId=iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3 2024-04-19T23:35:18.128 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N9A5EiIOAZfqo9G3Di4v+irDlok= 2024-04-19T23:35:18.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N9A5EiIOAZfqo9G3Di4v+irDlok=', 'Content-Length': '0'} 2024-04-19T23:35:18.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:31:59 GMT'), ('x-amz-version-id', 'iRrRmFZs3bOUPbgnHKMwQYejjeBfjb3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fff0f93fcb37a52a-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.130 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.131 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.131 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:18.131 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:18.131 DEBUG:boto:Method: GET 2024-04-19T23:35:18.131 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:35:18.131 DEBUG:boto:Data: 2024-04-19T23:35:18.131 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.131 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.131 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.131 DEBUG:boto:Params: {} 2024-04-19T23:35:18.132 DEBUG:boto:Token: None 2024-04-19T23:35:18.132 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.132 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008694f0f31e5c54d4-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.178 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.179 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.179 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.179 DEBUG:boto:path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:18.179 DEBUG:boto:auth_path=/aktlji-20/?prefix=dummy 2024-04-19T23:35:18.179 DEBUG:boto:Method: GET 2024-04-19T23:35:18.179 DEBUG:boto:Path: /aktlji-20/?prefix=dummy 2024-04-19T23:35:18.179 DEBUG:boto:Data: 2024-04-19T23:35:18.180 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.180 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.180 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.180 DEBUG:boto:Params: {} 2024-04-19T23:35:18.180 DEBUG:boto:Token: None 2024-04-19T23:35:18.180 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a0704727d4e2c88-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.183 DEBUG:boto:b'aktlji-20dummy1000falsedummy2024-04-19T23:31:59.757Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:18.184 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.184 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.184 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.184 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.185 DEBUG:boto:Method: GET 2024-04-19T23:35:18.185 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.185 DEBUG:boto:Data: 2024-04-19T23:35:18.185 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.185 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.185 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.185 DEBUG:boto:Params: {} 2024-04-19T23:35:18.185 DEBUG:boto:Token: None 2024-04-19T23:35:18.185 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.185 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nIrJNbhptXBR/ufyhxEqitc8CX4= 2024-04-19T23:35:18.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nIrJNbhptXBR/ufyhxEqitc8CX4=', 'Content-Length': '0'} 2024-04-19T23:35:18.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000924bd2fa16e34bde-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.230 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.231 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.231 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.231 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.231 DEBUG:boto:Method: GET 2024-04-19T23:35:18.231 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.231 DEBUG:boto:Data: 2024-04-19T23:35:18.231 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.231 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.231 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.231 DEBUG:boto:Params: {} 2024-04-19T23:35:18.232 DEBUG:boto:Token: None 2024-04-19T23:35:18.232 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.232 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nIrJNbhptXBR/ufyhxEqitc8CX4= 2024-04-19T23:35:18.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nIrJNbhptXBR/ufyhxEqitc8CX4=', 'Content-Length': '0'} 2024-04-19T23:35:18.234 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003389c5c6e608b8af-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.234 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.234 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.234 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.234 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.235 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.235 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.235 DEBUG:boto:Data: 2024-04-19T23:35:18.235 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.235 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.235 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.235 DEBUG:boto:Params: {} 2024-04-19T23:35:18.235 DEBUG:boto:Token: None 2024-04-19T23:35:18.235 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.235 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YLB/BEeov8ypQEZG+6RSDlC1pF0= 2024-04-19T23:35:18.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YLB/BEeov8ypQEZG+6RSDlC1pF0=', 'Content-Length': '0'} 2024-04-19T23:35:18.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c625954382420bbf-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.238 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.238 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.238 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.238 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.238 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.238 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.238 DEBUG:boto:Data: 2024-04-19T23:35:18.238 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.238 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.238 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.239 DEBUG:boto:Params: {} 2024-04-19T23:35:18.239 DEBUG:boto:Token: None 2024-04-19T23:35:18.239 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.239 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YLB/BEeov8ypQEZG+6RSDlC1pF0= 2024-04-19T23:35:18.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YLB/BEeov8ypQEZG+6RSDlC1pF0=', 'Content-Length': '0'} 2024-04-19T23:35:18.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036f4cfb7a90feedb-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:18.242 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.242 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.242 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.242 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.242 DEBUG:boto:Method: GET 2024-04-19T23:35:18.242 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.242 DEBUG:boto:Data: 2024-04-19T23:35:18.242 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.242 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.243 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.243 DEBUG:boto:Params: {} 2024-04-19T23:35:18.243 DEBUG:boto:Token: None 2024-04-19T23:35:18.243 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.243 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nIrJNbhptXBR/ufyhxEqitc8CX4= 2024-04-19T23:35:18.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nIrJNbhptXBR/ufyhxEqitc8CX4=', 'Content-Length': '0'} 2024-04-19T23:35:18.245 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a255e9ffe396160e-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.286 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.287 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.287 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.287 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.287 DEBUG:boto:Method: GET 2024-04-19T23:35:18.287 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.287 DEBUG:boto:Data: 2024-04-19T23:35:18.287 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.287 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.287 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.287 DEBUG:boto:Params: {} 2024-04-19T23:35:18.288 DEBUG:boto:Token: None 2024-04-19T23:35:18.288 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.288 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nIrJNbhptXBR/ufyhxEqitc8CX4= 2024-04-19T23:35:18.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nIrJNbhptXBR/ufyhxEqitc8CX4=', 'Content-Length': '0'} 2024-04-19T23:35:18.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052d1da2cd34f738e-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.334 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.335 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.335 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:18.335 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:18.335 DEBUG:boto:Method: GET 2024-04-19T23:35:18.335 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:18.335 DEBUG:boto:Data: 2024-04-19T23:35:18.335 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.335 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.335 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.335 DEBUG:boto:Params: {} 2024-04-19T23:35:18.336 DEBUG:boto:Token: None 2024-04-19T23:35:18.336 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.336 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d00364dfdd9fbf1-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.390 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.391 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.391 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.391 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:18.391 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:18.392 DEBUG:boto:Method: GET 2024-04-19T23:35:18.392 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:18.392 DEBUG:boto:Data: 2024-04-19T23:35:18.392 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.392 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.392 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.392 DEBUG:boto:Params: {} 2024-04-19T23:35:18.392 DEBUG:boto:Token: None 2024-04-19T23:35:18.392 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.393 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044408606bf3d5dfb-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.396 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.396 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.396 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.396 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.397 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:35:18.397 DEBUG:boto:Data: 2024-04-19T23:35:18.397 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.397 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.397 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.397 DEBUG:boto:Params: {} 2024-04-19T23:35:18.397 DEBUG:boto:Token: None 2024-04-19T23:35:18.398 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:35:18.398 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5c30+J3b4lwOiU8Zt1uQJUf48B8= 2024-04-19T23:35:18.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5c30+J3b4lwOiU8Zt1uQJUf48B8=', 'Content-Length': '0'} 2024-04-19T23:35:18.400 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000feaeb10e031d2f6b-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.400 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.400 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.401 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.401 DEBUG:boto:Path: /aktlji-20/obj-test-zone1 2024-04-19T23:35:18.401 DEBUG:boto:Data: 2024-04-19T23:35:18.401 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.401 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.401 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.401 DEBUG:boto:Params: {} 2024-04-19T23:35:18.402 DEBUG:boto:Token: None 2024-04-19T23:35:18.402 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone1 2024-04-19T23:35:18.402 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5c30+J3b4lwOiU8Zt1uQJUf48B8= 2024-04-19T23:35:18.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5c30+J3b4lwOiU8Zt1uQJUf48B8=', 'Content-Length': '0'} 2024-04-19T23:35:18.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000428964608a15b03d-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:18.405 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.405 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.405 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.405 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.405 DEBUG:boto:Method: GET 2024-04-19T23:35:18.406 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.406 DEBUG:boto:Data: 2024-04-19T23:35:18.406 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.406 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.406 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.406 DEBUG:boto:Params: {} 2024-04-19T23:35:18.407 DEBUG:boto:Token: None 2024-04-19T23:35:18.407 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.407 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nIrJNbhptXBR/ufyhxEqitc8CX4= 2024-04-19T23:35:18.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nIrJNbhptXBR/ufyhxEqitc8CX4=', 'Content-Length': '0'} 2024-04-19T23:35:18.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b23648834acf0480-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.450 DEBUG:boto:path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.451 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1 2024-04-19T23:35:18.451 DEBUG:boto:path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.451 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.451 DEBUG:boto:Method: GET 2024-04-19T23:35:18.451 DEBUG:boto:Path: /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.451 DEBUG:boto:Data: 2024-04-19T23:35:18.451 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.451 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.451 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.451 DEBUG:boto:Params: {} 2024-04-19T23:35:18.452 DEBUG:boto:Token: None 2024-04-19T23:35:18.452 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone1?versionId=cbhikBGTloAAkVRyBUG6AxoyB1Rg35. 2024-04-19T23:35:18.452 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nIrJNbhptXBR/ufyhxEqitc8CX4= 2024-04-19T23:35:18.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nIrJNbhptXBR/ufyhxEqitc8CX4=', 'Content-Length': '0'} 2024-04-19T23:35:18.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'cbhikBGTloAAkVRyBUG6AxoyB1Rg35.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028ae6988826c07fb-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.494 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.495 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.495 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:18.495 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:18.495 DEBUG:boto:Method: GET 2024-04-19T23:35:18.495 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:18.495 DEBUG:boto:Data: 2024-04-19T23:35:18.495 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.495 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.496 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.496 DEBUG:boto:Params: {} 2024-04-19T23:35:18.496 DEBUG:boto:Token: None 2024-04-19T23:35:18.496 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.496 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b93c1e6bcef4da69-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.499 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.500 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.500 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.500 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:18.500 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:18.500 DEBUG:boto:Method: GET 2024-04-19T23:35:18.500 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone1 2024-04-19T23:35:18.500 DEBUG:boto:Data: 2024-04-19T23:35:18.500 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.500 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.501 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.501 DEBUG:boto:Params: {} 2024-04-19T23:35:18.501 DEBUG:boto:Token: None 2024-04-19T23:35:18.501 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.501 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5535de8532a8a03-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.546 DEBUG:boto:b'aktlji-20obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:18.547 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.547 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.547 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.547 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.548 DEBUG:boto:Method: GET 2024-04-19T23:35:18.548 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.548 DEBUG:boto:Data: 2024-04-19T23:35:18.548 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.548 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.548 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.548 DEBUG:boto:Params: {} 2024-04-19T23:35:18.548 DEBUG:boto:Token: None 2024-04-19T23:35:18.548 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.549 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8cJdIZAtCwiF4MPG+4d+QH/SLw= 2024-04-19T23:35:18.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8cJdIZAtCwiF4MPG+4d+QH/SLw=', 'Content-Length': '0'} 2024-04-19T23:35:18.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb89ddaea24e23ad-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.598 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.599 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.599 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.599 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.600 DEBUG:boto:Method: GET 2024-04-19T23:35:18.600 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.600 DEBUG:boto:Data: 2024-04-19T23:35:18.600 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.600 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.600 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.601 DEBUG:boto:Params: {} 2024-04-19T23:35:18.601 DEBUG:boto:Token: None 2024-04-19T23:35:18.601 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.601 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8cJdIZAtCwiF4MPG+4d+QH/SLw= 2024-04-19T23:35:18.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8cJdIZAtCwiF4MPG+4d+QH/SLw=', 'Content-Length': '0'} 2024-04-19T23:35:18.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebe052b7f92fd545-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.604 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.605 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.605 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.605 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.605 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.605 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.605 DEBUG:boto:Data: 2024-04-19T23:35:18.606 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.606 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.606 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.606 DEBUG:boto:Params: {} 2024-04-19T23:35:18.606 DEBUG:boto:Token: None 2024-04-19T23:35:18.606 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.606 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:t95NNqnQF1kQNExGcPs/RLVK74U= 2024-04-19T23:35:18.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:t95NNqnQF1kQNExGcPs/RLVK74U=', 'Content-Length': '0'} 2024-04-19T23:35:18.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067e67d1f7aa8fc6c-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.609 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.609 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.609 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.609 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.609 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.609 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.609 DEBUG:boto:Data: 2024-04-19T23:35:18.609 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.610 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.610 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.610 DEBUG:boto:Params: {} 2024-04-19T23:35:18.610 DEBUG:boto:Token: None 2024-04-19T23:35:18.610 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.610 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:t95NNqnQF1kQNExGcPs/RLVK74U= 2024-04-19T23:35:18.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:t95NNqnQF1kQNExGcPs/RLVK74U=', 'Content-Length': '0'} 2024-04-19T23:35:18.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9b69c6c80d77822-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:18.613 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.613 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.613 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.613 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.613 DEBUG:boto:Method: GET 2024-04-19T23:35:18.613 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.614 DEBUG:boto:Data: 2024-04-19T23:35:18.614 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.614 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.614 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.614 DEBUG:boto:Params: {} 2024-04-19T23:35:18.614 DEBUG:boto:Token: None 2024-04-19T23:35:18.614 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.615 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8cJdIZAtCwiF4MPG+4d+QH/SLw= 2024-04-19T23:35:18.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8cJdIZAtCwiF4MPG+4d+QH/SLw=', 'Content-Length': '0'} 2024-04-19T23:35:18.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ff7ffaf65877c24-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.662 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.662 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.663 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.663 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.663 DEBUG:boto:Method: GET 2024-04-19T23:35:18.663 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.663 DEBUG:boto:Data: 2024-04-19T23:35:18.663 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.663 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.663 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.663 DEBUG:boto:Params: {} 2024-04-19T23:35:18.663 DEBUG:boto:Token: None 2024-04-19T23:35:18.664 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.664 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8cJdIZAtCwiF4MPG+4d+QH/SLw= 2024-04-19T23:35:18.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8cJdIZAtCwiF4MPG+4d+QH/SLw=', 'Content-Length': '0'} 2024-04-19T23:35:18.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2dce4397ca6cde6-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.666 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.667 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.667 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:18.667 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:18.667 DEBUG:boto:Method: GET 2024-04-19T23:35:18.667 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:18.667 DEBUG:boto:Data: 2024-04-19T23:35:18.667 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.668 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.668 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.668 DEBUG:boto:Params: {} 2024-04-19T23:35:18.668 DEBUG:boto:Token: None 2024-04-19T23:35:18.668 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.669 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c704672c4814b358-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.672 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.672 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.672 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.672 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:18.672 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:18.673 DEBUG:boto:Method: GET 2024-04-19T23:35:18.673 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:18.673 DEBUG:boto:Data: 2024-04-19T23:35:18.673 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.673 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.673 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.673 DEBUG:boto:Params: {} 2024-04-19T23:35:18.673 DEBUG:boto:Token: None 2024-04-19T23:35:18.673 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.673 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dd01d5d9d2ff767-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.718 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.719 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.719 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.719 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.719 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:35:18.719 DEBUG:boto:Data: 2024-04-19T23:35:18.719 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.719 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.720 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.720 DEBUG:boto:Params: {} 2024-04-19T23:35:18.720 DEBUG:boto:Token: None 2024-04-19T23:35:18.720 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:35:18.720 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kY5fHCYnM8W7DSww8xSfN/c//+I= 2024-04-19T23:35:18.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kY5fHCYnM8W7DSww8xSfN/c//+I=', 'Content-Length': '0'} 2024-04-19T23:35:18.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3b6806fd67e3c7f-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.723 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.723 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.723 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.723 DEBUG:boto:Path: /aktlji-20/obj-test-zone2 2024-04-19T23:35:18.723 DEBUG:boto:Data: 2024-04-19T23:35:18.723 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.724 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.724 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.724 DEBUG:boto:Params: {} 2024-04-19T23:35:18.724 DEBUG:boto:Token: None 2024-04-19T23:35:18.724 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone2 2024-04-19T23:35:18.724 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kY5fHCYnM8W7DSww8xSfN/c//+I= 2024-04-19T23:35:18.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kY5fHCYnM8W7DSww8xSfN/c//+I=', 'Content-Length': '0'} 2024-04-19T23:35:18.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002227e29b487d7822-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:18.727 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.727 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.727 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.727 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.727 DEBUG:boto:Method: GET 2024-04-19T23:35:18.727 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.728 DEBUG:boto:Data: 2024-04-19T23:35:18.728 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.728 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.728 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.728 DEBUG:boto:Params: {} 2024-04-19T23:35:18.728 DEBUG:boto:Token: None 2024-04-19T23:35:18.728 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.729 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8cJdIZAtCwiF4MPG+4d+QH/SLw= 2024-04-19T23:35:18.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8cJdIZAtCwiF4MPG+4d+QH/SLw=', 'Content-Length': '0'} 2024-04-19T23:35:18.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c67cd2113fb3809-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.775 DEBUG:boto:path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.775 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2 2024-04-19T23:35:18.775 DEBUG:boto:path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.775 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.776 DEBUG:boto:Method: GET 2024-04-19T23:35:18.776 DEBUG:boto:Path: /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.776 DEBUG:boto:Data: 2024-04-19T23:35:18.776 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.776 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.776 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.776 DEBUG:boto:Params: {} 2024-04-19T23:35:18.776 DEBUG:boto:Token: None 2024-04-19T23:35:18.776 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone2?versionId=Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz 2024-04-19T23:35:18.777 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8cJdIZAtCwiF4MPG+4d+QH/SLw= 2024-04-19T23:35:18.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8cJdIZAtCwiF4MPG+4d+QH/SLw=', 'Content-Length': '0'} 2024-04-19T23:35:18.779 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'Jdm81gPfYI2WgTFtftcs7OV2uk7V4xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c22a898de4d72ec-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.779 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.780 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.780 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:18.780 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:18.780 DEBUG:boto:Method: GET 2024-04-19T23:35:18.780 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:18.780 DEBUG:boto:Data: 2024-04-19T23:35:18.780 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.780 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.780 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.780 DEBUG:boto:Params: {} 2024-04-19T23:35:18.781 DEBUG:boto:Token: None 2024-04-19T23:35:18.781 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.781 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6441116e46f3254-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.784 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.784 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.784 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.784 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:18.784 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:18.785 DEBUG:boto:Method: GET 2024-04-19T23:35:18.785 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone2 2024-04-19T23:35:18.785 DEBUG:boto:Data: 2024-04-19T23:35:18.785 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.785 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.785 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.785 DEBUG:boto:Params: {} 2024-04-19T23:35:18.785 DEBUG:boto:Token: None 2024-04-19T23:35:18.785 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.786 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057ffd82adb93de4a-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.788 DEBUG:boto:b'aktlji-20obj-test-zone21000falseobj-test-zone22024-04-19T23:32:35.710Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:18.789 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.789 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.789 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.789 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.789 DEBUG:boto:Method: GET 2024-04-19T23:35:18.789 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.789 DEBUG:boto:Data: 2024-04-19T23:35:18.790 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.790 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.790 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.790 DEBUG:boto:Params: {} 2024-04-19T23:35:18.790 DEBUG:boto:Token: None 2024-04-19T23:35:18.791 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.791 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6iz+b/51cd3zdhnBgkWzzW/WZiI= 2024-04-19T23:35:18.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6iz+b/51cd3zdhnBgkWzzW/WZiI=', 'Content-Length': '0'} 2024-04-19T23:35:18.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e1d9826596b15c3-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.794 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.794 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.794 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.794 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.795 DEBUG:boto:Method: GET 2024-04-19T23:35:18.795 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.795 DEBUG:boto:Data: 2024-04-19T23:35:18.795 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.795 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.796 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.796 DEBUG:boto:Params: {} 2024-04-19T23:35:18.796 DEBUG:boto:Token: None 2024-04-19T23:35:18.796 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.796 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6iz+b/51cd3zdhnBgkWzzW/WZiI= 2024-04-19T23:35:18.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6iz+b/51cd3zdhnBgkWzzW/WZiI=', 'Content-Length': '0'} 2024-04-19T23:35:18.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1f0125328c87849-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.799 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.800 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.800 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.800 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.800 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.800 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.800 DEBUG:boto:Data: 2024-04-19T23:35:18.800 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.800 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.800 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.800 DEBUG:boto:Params: {} 2024-04-19T23:35:18.801 DEBUG:boto:Token: None 2024-04-19T23:35:18.801 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.801 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gs1Ffc38P/hWTtSyKcH9wEG/jO0= 2024-04-19T23:35:18.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gs1Ffc38P/hWTtSyKcH9wEG/jO0=', 'Content-Length': '0'} 2024-04-19T23:35:18.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000098f0c07f681ddc4-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.803 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.803 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.803 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.803 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.804 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.804 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.804 DEBUG:boto:Data: 2024-04-19T23:35:18.804 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.804 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.804 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.804 DEBUG:boto:Params: {} 2024-04-19T23:35:18.804 DEBUG:boto:Token: None 2024-04-19T23:35:18.804 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gs1Ffc38P/hWTtSyKcH9wEG/jO0= 2024-04-19T23:35:18.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gs1Ffc38P/hWTtSyKcH9wEG/jO0=', 'Content-Length': '0'} 2024-04-19T23:35:18.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049139a2ea791ae7e-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:18.807 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.807 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.807 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.807 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.807 DEBUG:boto:Method: GET 2024-04-19T23:35:18.807 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.807 DEBUG:boto:Data: 2024-04-19T23:35:18.807 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.807 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.808 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.808 DEBUG:boto:Params: {} 2024-04-19T23:35:18.808 DEBUG:boto:Token: None 2024-04-19T23:35:18.808 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.808 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6iz+b/51cd3zdhnBgkWzzW/WZiI= 2024-04-19T23:35:18.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6iz+b/51cd3zdhnBgkWzzW/WZiI=', 'Content-Length': '0'} 2024-04-19T23:35:18.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f91b0c60d93c372-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.858 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.859 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.859 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.859 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.859 DEBUG:boto:Method: GET 2024-04-19T23:35:18.859 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.859 DEBUG:boto:Data: 2024-04-19T23:35:18.860 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.860 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.860 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.860 DEBUG:boto:Params: {} 2024-04-19T23:35:18.860 DEBUG:boto:Token: None 2024-04-19T23:35:18.861 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.861 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6iz+b/51cd3zdhnBgkWzzW/WZiI= 2024-04-19T23:35:18.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6iz+b/51cd3zdhnBgkWzzW/WZiI=', 'Content-Length': '0'} 2024-04-19T23:35:18.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e544a8c0684e9e1-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.906 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.907 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.907 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.907 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.907 DEBUG:boto:Method: GET 2024-04-19T23:35:18.907 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.907 DEBUG:boto:Data: 2024-04-19T23:35:18.907 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.908 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.908 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.908 DEBUG:boto:Params: {} 2024-04-19T23:35:18.908 DEBUG:boto:Token: None 2024-04-19T23:35:18.908 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.908 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8ff613d089813c3-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.911 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.912 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.912 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.912 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.912 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.913 DEBUG:boto:Method: GET 2024-04-19T23:35:18.913 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.913 DEBUG:boto:Data: 2024-04-19T23:35:18.913 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.913 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.914 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.914 DEBUG:boto:Params: {} 2024-04-19T23:35:18.914 DEBUG:boto:Token: None 2024-04-19T23:35:18.914 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.915 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bc00cd64c16de36-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.918 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.918 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.919 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.919 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.919 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:35:18.919 DEBUG:boto:Data: 2024-04-19T23:35:18.920 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.920 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.920 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.920 DEBUG:boto:Params: {} 2024-04-19T23:35:18.920 DEBUG:boto:Token: None 2024-04-19T23:35:18.921 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:35:18.921 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gLXpM3Ns7tzElXgW7al6YDl0lPM= 2024-04-19T23:35:18.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gLXpM3Ns7tzElXgW7al6YDl0lPM=', 'Content-Length': '0'} 2024-04-19T23:35:18.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049ea9f6ced682eec-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.924 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.924 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.925 DEBUG:boto:Method: HEAD 2024-04-19T23:35:18.925 DEBUG:boto:Path: /aktlji-20/obj-test-zone3 2024-04-19T23:35:18.925 DEBUG:boto:Data: 2024-04-19T23:35:18.925 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.925 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.925 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.925 DEBUG:boto:Params: {} 2024-04-19T23:35:18.926 DEBUG:boto:Token: None 2024-04-19T23:35:18.926 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone3 2024-04-19T23:35:18.926 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gLXpM3Ns7tzElXgW7al6YDl0lPM= 2024-04-19T23:35:18.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gLXpM3Ns7tzElXgW7al6YDl0lPM=', 'Content-Length': '0'} 2024-04-19T23:35:18.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000154d03781109c02f-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:18.929 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.929 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.929 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.929 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.929 DEBUG:boto:Method: GET 2024-04-19T23:35:18.929 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.929 DEBUG:boto:Data: 2024-04-19T23:35:18.929 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.929 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.929 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.929 DEBUG:boto:Params: {} 2024-04-19T23:35:18.930 DEBUG:boto:Token: None 2024-04-19T23:35:18.930 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.930 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6iz+b/51cd3zdhnBgkWzzW/WZiI= 2024-04-19T23:35:18.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6iz+b/51cd3zdhnBgkWzzW/WZiI=', 'Content-Length': '0'} 2024-04-19T23:35:18.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a1ac97a80d8abce-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.932 DEBUG:boto:path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.933 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3 2024-04-19T23:35:18.933 DEBUG:boto:path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.933 DEBUG:boto:auth_path=/aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.933 DEBUG:boto:Method: GET 2024-04-19T23:35:18.933 DEBUG:boto:Path: /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.934 DEBUG:boto:Data: 2024-04-19T23:35:18.934 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.934 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.934 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.934 DEBUG:boto:Params: {} 2024-04-19T23:35:18.934 DEBUG:boto:Token: None 2024-04-19T23:35:18.934 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/obj-test-zone3?versionId=htNuK4kZrW.j1OByzRcdsecrUSmYZCj 2024-04-19T23:35:18.935 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6iz+b/51cd3zdhnBgkWzzW/WZiI= 2024-04-19T23:35:18.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6iz+b/51cd3zdhnBgkWzzW/WZiI=', 'Content-Length': '0'} 2024-04-19T23:35:18.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'htNuK4kZrW.j1OByzRcdsecrUSmYZCj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff249079f684091c-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.978 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.979 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.979 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.979 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.979 DEBUG:boto:Method: GET 2024-04-19T23:35:18.979 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.979 DEBUG:boto:Data: 2024-04-19T23:35:18.979 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.979 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:18.979 DEBUG:boto:Port: 8001 2024-04-19T23:35:18.979 DEBUG:boto:Params: {} 2024-04-19T23:35:18.980 DEBUG:boto:Token: None 2024-04-19T23:35:18.980 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.980 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8e304355cca06ec-006622ffb6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.983 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.983 DEBUG:boto:path=/aktlji-20/ 2024-04-19T23:35:18.983 DEBUG:boto:auth_path=/aktlji-20/ 2024-04-19T23:35:18.983 DEBUG:boto:path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.983 DEBUG:boto:auth_path=/aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.983 DEBUG:boto:Method: GET 2024-04-19T23:35:18.984 DEBUG:boto:Path: /aktlji-20/?prefix=obj-test-zone3 2024-04-19T23:35:18.984 DEBUG:boto:Data: 2024-04-19T23:35:18.984 DEBUG:boto:Headers: {} 2024-04-19T23:35:18.984 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:18.984 DEBUG:boto:Port: 8000 2024-04-19T23:35:18.984 DEBUG:boto:Params: {} 2024-04-19T23:35:18.984 DEBUG:boto:Token: None 2024-04-19T23:35:18.984 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:18 GMT /aktlji-20/ 2024-04-19T23:35:18.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs= 2024-04-19T23:35:18.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0J8cwWBQIyVRItjo9qlKihcSovs=', 'Content-Length': '0'} 2024-04-19T23:35:18.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047494f271eb325fd-006622ffb6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:18.987 DEBUG:boto:b'aktlji-20obj-test-zone31000falseobj-test-zone32024-04-19T23:32:35.820Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:18.987 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-20 zones={test-zone3, test-zone2} 2024-04-19T23:35:18.987 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-21 2024-04-19T23:35:18.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:18.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:18.988 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:19.055 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:19.055 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:19.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:19.103+0000 7fd039fa0780 1 bucket sync caught up with source: 2024-04-19T23:35:19.098 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000011.35.6, , 00000000005.51.10, , 00000000010.23.10, , , , , ] 2024-04-19T23:35:19.099 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000011.35.6, , 00000000005.51.10, , 00000000010.23.10, , , , , ] 2024-04-19T23:35:19.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:19.103+0000 7fd039fa0780 0 bucket checkpoint complete 2024-04-19T23:35:19.113 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-21 2024-04-19T23:35:19.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:19.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:19.114 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:19.181 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:19.181 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:19.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:19.228+0000 7fdb36341780 1 bucket sync caught up with source: 2024-04-19T23:35:19.224 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000011.35.6, , 00000000005.51.10, , 00000000010.23.10, , , , , ] 2024-04-19T23:35:19.224 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000011.35.6, , 00000000005.51.10, , 00000000010.23.10, , , , , ] 2024-04-19T23:35:19.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:19.228+0000 7fdb36341780 0 bucket checkpoint complete 2024-04-19T23:35:19.238 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-21 2024-04-19T23:35:19.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:19.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:19.239 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:19.303 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:19.303 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:19.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:19.348+0000 7fe3762cb780 1 bucket sync caught up with source: 2024-04-19T23:35:19.344 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.85.10, , 00000000011.121.6, , 00000000007.75.10, , , , , ] 2024-04-19T23:35:19.344 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.85.10, , 00000000011.121.6, , 00000000007.75.10, , , , , ] 2024-04-19T23:35:19.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:19.348+0000 7fe3762cb780 0 bucket checkpoint complete 2024-04-19T23:35:19.358 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-21 2024-04-19T23:35:19.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:19.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:19.359 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:19.422 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:19.422 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:19.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:19.466+0000 7f859f531780 1 bucket sync caught up with source: 2024-04-19T23:35:19.462 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.85.10, , 00000000011.121.6, , 00000000007.75.10, , , , , ] 2024-04-19T23:35:19.462 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.85.10, , 00000000011.121.6, , 00000000007.75.10, , , , , ] 2024-04-19T23:35:19.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:19.466+0000 7f859f531780 0 bucket checkpoint complete 2024-04-19T23:35:19.476 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-21 2024-04-19T23:35:19.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:19.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:19.477 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:19.539 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:19.539 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:19.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:19.583+0000 7fa4bb70c780 1 bucket sync caught up with source: 2024-04-19T23:35:19.579 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000003.45.10, , 00000000005.80.10, , 00000000019.63.6, , , , , ] 2024-04-19T23:35:19.579 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000003.45.10, , 00000000005.80.10, , 00000000019.63.6, , , , , ] 2024-04-19T23:35:19.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:19.583+0000 7fa4bb70c780 0 bucket checkpoint complete 2024-04-19T23:35:19.594 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-21 2024-04-19T23:35:19.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:19.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:19.594 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:19.663 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:19.663 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:19.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:19.710+0000 7f95aa62d780 1 bucket sync caught up with source: 2024-04-19T23:35:19.706 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, 00000000003.45.10, , 00000000005.80.10, , 00000000019.63.6, , , , , ] 2024-04-19T23:35:19.706 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, 00000000003.45.10, , 00000000005.80.10, , 00000000019.63.6, , , , , ] 2024-04-19T23:35:19.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:19.710+0000 7f95aa62d780 0 bucket checkpoint complete 2024-04-19T23:35:19.721 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-21 zones={test-zone2, test-zone1} 2024-04-19T23:35:19.721 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:19.721 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:19.721 DEBUG:boto:Method: HEAD 2024-04-19T23:35:19.721 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:35:19.721 DEBUG:boto:Data: 2024-04-19T23:35:19.721 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.722 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.722 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.722 DEBUG:boto:Params: {} 2024-04-19T23:35:19.722 DEBUG:boto:Token: None 2024-04-19T23:35:19.722 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/ 2024-04-19T23:35:19.722 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bbPd8FfPUob7gG0NXojYp5nWXmo= 2024-04-19T23:35:19.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bbPd8FfPUob7gG0NXojYp5nWXmo=', 'Content-Length': '0'} 2024-04-19T23:35:19.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071ee941804c0d40b-006622ffb7-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.725 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:19.725 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:19.725 DEBUG:boto:Method: HEAD 2024-04-19T23:35:19.725 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:35:19.726 DEBUG:boto:Data: 2024-04-19T23:35:19.726 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.726 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:19.726 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.726 DEBUG:boto:Params: {} 2024-04-19T23:35:19.726 DEBUG:boto:Token: None 2024-04-19T23:35:19.726 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/ 2024-04-19T23:35:19.726 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bbPd8FfPUob7gG0NXojYp5nWXmo= 2024-04-19T23:35:19.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bbPd8FfPUob7gG0NXojYp5nWXmo=', 'Content-Length': '0'} 2024-04-19T23:35:19.729 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056cf2e1b9c7cc616-006622ffb7-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.729 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:35:19.729 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:19.730 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:19.730 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:19.730 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:19.730 DEBUG:boto:Method: GET 2024-04-19T23:35:19.730 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:35:19.730 DEBUG:boto:Data: 2024-04-19T23:35:19.730 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.730 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.730 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.730 DEBUG:boto:Params: {} 2024-04-19T23:35:19.731 DEBUG:boto:Token: None 2024-04-19T23:35:19.731 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/?versions 2024-04-19T23:35:19.731 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3T3zEeUiTQ/OXxrSBN8hk4piKJ0= 2024-04-19T23:35:19.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3T3zEeUiTQ/OXxrSBN8hk4piKJ0=', 'Content-Length': '0'} 2024-04-19T23:35:19.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003597caea2ab7618e-006622ffb7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.734 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:19.734 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:35:19.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:35:19.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:35:19.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:35:19.735 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:35:19.735 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:19.735 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:19.735 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:19.735 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:19.736 DEBUG:boto:Method: GET 2024-04-19T23:35:19.736 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:35:19.736 DEBUG:boto:Data: 2024-04-19T23:35:19.736 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.736 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:19.736 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.737 DEBUG:boto:Params: {} 2024-04-19T23:35:19.737 DEBUG:boto:Token: None 2024-04-19T23:35:19.737 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/?versions 2024-04-19T23:35:19.737 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3T3zEeUiTQ/OXxrSBN8hk4piKJ0= 2024-04-19T23:35:19.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3T3zEeUiTQ/OXxrSBN8hk4piKJ0=', 'Content-Length': '0'} 2024-04-19T23:35:19.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb21bb6efaad5bef-006622ffb7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.744 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:19.744 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:35:19.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:35:19.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:35:19.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:35:19.745 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:19.745 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:19.745 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:19.746 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:19.746 DEBUG:boto:Method: GET 2024-04-19T23:35:19.746 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:35:19.746 DEBUG:boto:Data: 2024-04-19T23:35:19.746 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.746 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.746 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.746 DEBUG:boto:Params: {} 2024-04-19T23:35:19.746 DEBUG:boto:Token: None 2024-04-19T23:35:19.747 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/?versions 2024-04-19T23:35:19.747 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3T3zEeUiTQ/OXxrSBN8hk4piKJ0= 2024-04-19T23:35:19.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3T3zEeUiTQ/OXxrSBN8hk4piKJ0=', 'Content-Length': '0'} 2024-04-19T23:35:19.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c7a3cac2862e2c2-006622ffb7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.749 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:19.750 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:19.750 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:19.750 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:19.750 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:19.750 DEBUG:boto:Method: GET 2024-04-19T23:35:19.750 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:35:19.750 DEBUG:boto:Data: 2024-04-19T23:35:19.751 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.751 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:19.751 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.751 DEBUG:boto:Params: {} 2024-04-19T23:35:19.751 DEBUG:boto:Token: None 2024-04-19T23:35:19.751 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/?versions 2024-04-19T23:35:19.751 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3T3zEeUiTQ/OXxrSBN8hk4piKJ0= 2024-04-19T23:35:19.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3T3zEeUiTQ/OXxrSBN8hk4piKJ0=', 'Content-Length': '0'} 2024-04-19T23:35:19.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006327c7cd84bf97da-006622ffb7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.754 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:19.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:19.755 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:19.755 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:19.755 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.755 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.755 DEBUG:boto:Method: GET 2024-04-19T23:35:19.755 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.755 DEBUG:boto:Data: 2024-04-19T23:35:19.756 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.756 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.756 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.756 DEBUG:boto:Params: {} 2024-04-19T23:35:19.756 DEBUG:boto:Token: None 2024-04-19T23:35:19.756 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.757 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:thTVxUKyDLqkah2zWU+YMxkcAC8= 2024-04-19T23:35:19.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:thTVxUKyDLqkah2zWU+YMxkcAC8=', 'Content-Length': '0'} 2024-04-19T23:35:19.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e40ed60f03d74966-006622ffb7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.759 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:19.759 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:19.759 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.759 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.759 DEBUG:boto:Method: GET 2024-04-19T23:35:19.759 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.759 DEBUG:boto:Data: 2024-04-19T23:35:19.759 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.760 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:19.760 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.760 DEBUG:boto:Params: {} 2024-04-19T23:35:19.760 DEBUG:boto:Token: None 2024-04-19T23:35:19.760 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.760 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:thTVxUKyDLqkah2zWU+YMxkcAC8= 2024-04-19T23:35:19.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:thTVxUKyDLqkah2zWU+YMxkcAC8=', 'Content-Length': '0'} 2024-04-19T23:35:19.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd68e53c2da3ac96-006622ffb7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.762 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:19.762 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:19.762 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.763 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.763 DEBUG:boto:Method: HEAD 2024-04-19T23:35:19.763 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.763 DEBUG:boto:Data: 2024-04-19T23:35:19.763 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.763 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.763 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.763 DEBUG:boto:Params: {} 2024-04-19T23:35:19.763 DEBUG:boto:Token: None 2024-04-19T23:35:19.763 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.764 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OOeBJ0CLFfADEUhxkdboj9mvi6s= 2024-04-19T23:35:19.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OOeBJ0CLFfADEUhxkdboj9mvi6s=', 'Content-Length': '0'} 2024-04-19T23:35:19.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ef3ebdfc435ee807-006622ffb7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.765 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:19.766 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:19.766 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.766 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.766 DEBUG:boto:Method: HEAD 2024-04-19T23:35:19.766 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.766 DEBUG:boto:Data: 2024-04-19T23:35:19.766 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.766 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:19.766 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.767 DEBUG:boto:Params: {} 2024-04-19T23:35:19.767 DEBUG:boto:Token: None 2024-04-19T23:35:19.767 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.767 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OOeBJ0CLFfADEUhxkdboj9mvi6s= 2024-04-19T23:35:19.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OOeBJ0CLFfADEUhxkdboj9mvi6s=', 'Content-Length': '0'} 2024-04-19T23:35:19.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c92f1ba742ebb8e-006622ffb7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:19.770 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:19.770 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:19.770 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.770 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.770 DEBUG:boto:Method: GET 2024-04-19T23:35:19.771 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.771 DEBUG:boto:Data: 2024-04-19T23:35:19.771 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.771 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.771 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.771 DEBUG:boto:Params: {} 2024-04-19T23:35:19.771 DEBUG:boto:Token: None 2024-04-19T23:35:19.771 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.771 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:thTVxUKyDLqkah2zWU+YMxkcAC8= 2024-04-19T23:35:19.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:thTVxUKyDLqkah2zWU+YMxkcAC8=', 'Content-Length': '0'} 2024-04-19T23:35:19.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000076fbd20dc149a456-006622ffb7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.774 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:19.774 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:19.774 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.774 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.774 DEBUG:boto:Method: GET 2024-04-19T23:35:19.775 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.775 DEBUG:boto:Data: 2024-04-19T23:35:19.775 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.775 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:19.775 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.775 DEBUG:boto:Params: {} 2024-04-19T23:35:19.775 DEBUG:boto:Token: None 2024-04-19T23:35:19.775 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.775 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:thTVxUKyDLqkah2zWU+YMxkcAC8= 2024-04-19T23:35:19.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:thTVxUKyDLqkah2zWU+YMxkcAC8=', 'Content-Length': '0'} 2024-04-19T23:35:19.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000005993ace8762f295-006622ffb7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.778 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:19.778 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:19.778 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:19.778 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:19.778 DEBUG:boto:Method: GET 2024-04-19T23:35:19.778 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:35:19.778 DEBUG:boto:Data: 2024-04-19T23:35:19.778 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.779 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.779 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.779 DEBUG:boto:Params: {} 2024-04-19T23:35:19.779 DEBUG:boto:Token: None 2024-04-19T23:35:19.779 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/ 2024-04-19T23:35:19.779 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ywzT95FQhojDt0zpmHfP/FPsM08= 2024-04-19T23:35:19.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ywzT95FQhojDt0zpmHfP/FPsM08=', 'Content-Length': '0'} 2024-04-19T23:35:19.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d91db9f0f8c181db-006622ffb7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.822 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:19.823 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:19.823 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:19.823 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:19.823 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:19.823 DEBUG:boto:Method: GET 2024-04-19T23:35:19.823 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:35:19.823 DEBUG:boto:Data: 2024-04-19T23:35:19.824 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.824 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:19.824 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.824 DEBUG:boto:Params: {} 2024-04-19T23:35:19.824 DEBUG:boto:Token: None 2024-04-19T23:35:19.824 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/ 2024-04-19T23:35:19.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ywzT95FQhojDt0zpmHfP/FPsM08= 2024-04-19T23:35:19.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ywzT95FQhojDt0zpmHfP/FPsM08=', 'Content-Length': '0'} 2024-04-19T23:35:19.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000040bce8b361b1afa-006622ffb7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.827 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:19.828 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:19.828 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:19.828 DEBUG:boto:Method: HEAD 2024-04-19T23:35:19.828 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:35:19.828 DEBUG:boto:Data: 2024-04-19T23:35:19.828 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.828 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.828 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.829 DEBUG:boto:Params: {} 2024-04-19T23:35:19.829 DEBUG:boto:Token: None 2024-04-19T23:35:19.829 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/dummy 2024-04-19T23:35:19.829 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EUhYxD/cj+ZhzOc5AwvlGJdvnxQ= 2024-04-19T23:35:19.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EUhYxD/cj+ZhzOc5AwvlGJdvnxQ=', 'Content-Length': '0'} 2024-04-19T23:35:19.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000042f86291f63e7f7e-006622ffb7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.832 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:19.832 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:19.832 DEBUG:boto:Method: HEAD 2024-04-19T23:35:19.832 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:35:19.832 DEBUG:boto:Data: 2024-04-19T23:35:19.833 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.833 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:19.833 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.833 DEBUG:boto:Params: {} 2024-04-19T23:35:19.833 DEBUG:boto:Token: None 2024-04-19T23:35:19.833 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/dummy 2024-04-19T23:35:19.833 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EUhYxD/cj+ZhzOc5AwvlGJdvnxQ= 2024-04-19T23:35:19.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EUhYxD/cj+ZhzOc5AwvlGJdvnxQ=', 'Content-Length': '0'} 2024-04-19T23:35:19.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000adf04203b93d9f1d-006622ffb7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:19.836 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:19.836 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:19.836 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.836 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.836 DEBUG:boto:Method: GET 2024-04-19T23:35:19.836 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.836 DEBUG:boto:Data: 2024-04-19T23:35:19.836 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.837 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.837 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.837 DEBUG:boto:Params: {} 2024-04-19T23:35:19.837 DEBUG:boto:Token: None 2024-04-19T23:35:19.837 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.837 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:thTVxUKyDLqkah2zWU+YMxkcAC8= 2024-04-19T23:35:19.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:thTVxUKyDLqkah2zWU+YMxkcAC8=', 'Content-Length': '0'} 2024-04-19T23:35:19.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fb84d5402d897a46-006622ffb7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.840 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:19.840 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:19.840 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.840 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.841 DEBUG:boto:Method: GET 2024-04-19T23:35:19.841 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.841 DEBUG:boto:Data: 2024-04-19T23:35:19.841 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.841 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:19.841 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.841 DEBUG:boto:Params: {} 2024-04-19T23:35:19.841 DEBUG:boto:Token: None 2024-04-19T23:35:19.841 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:19.841 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:thTVxUKyDLqkah2zWU+YMxkcAC8= 2024-04-19T23:35:19.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:thTVxUKyDLqkah2zWU+YMxkcAC8=', 'Content-Length': '0'} 2024-04-19T23:35:19.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000047d05c0e615757f3-006622ffb7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.844 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:19.844 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:19.844 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:19.844 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:19.844 DEBUG:boto:Method: GET 2024-04-19T23:35:19.844 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:35:19.844 DEBUG:boto:Data: 2024-04-19T23:35:19.844 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.845 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.845 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.845 DEBUG:boto:Params: {} 2024-04-19T23:35:19.845 DEBUG:boto:Token: None 2024-04-19T23:35:19.845 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/ 2024-04-19T23:35:19.845 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ywzT95FQhojDt0zpmHfP/FPsM08= 2024-04-19T23:35:19.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ywzT95FQhojDt0zpmHfP/FPsM08=', 'Content-Length': '0'} 2024-04-19T23:35:19.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb2d0193a2d3fdf7-006622ffb7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.890 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:19.891 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:19.891 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:19.891 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:19.891 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:19.892 DEBUG:boto:Method: GET 2024-04-19T23:35:19.892 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:35:19.892 DEBUG:boto:Data: 2024-04-19T23:35:19.892 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.892 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:19.892 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.892 DEBUG:boto:Params: {} 2024-04-19T23:35:19.892 DEBUG:boto:Token: None 2024-04-19T23:35:19.893 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/ 2024-04-19T23:35:19.893 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ywzT95FQhojDt0zpmHfP/FPsM08= 2024-04-19T23:35:19.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ywzT95FQhojDt0zpmHfP/FPsM08=', 'Content-Length': '0'} 2024-04-19T23:35:19.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae400310a0151431-006622ffb7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.942 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:19.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:19.943 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:19.943 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:19.944 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.944 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.944 DEBUG:boto:Method: GET 2024-04-19T23:35:19.944 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.944 DEBUG:boto:Data: 2024-04-19T23:35:19.944 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.945 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.945 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.945 DEBUG:boto:Params: {} 2024-04-19T23:35:19.945 DEBUG:boto:Token: None 2024-04-19T23:35:19.945 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.946 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:t7FDX4bMOYPjrEp07GP5vU23D1Q= 2024-04-19T23:35:19.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:t7FDX4bMOYPjrEp07GP5vU23D1Q=', 'Content-Length': '0'} 2024-04-19T23:35:19.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffa7cc1cc3f7afc1-006622ffb7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.948 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:19.949 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:19.949 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.949 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.949 DEBUG:boto:Method: GET 2024-04-19T23:35:19.949 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.949 DEBUG:boto:Data: 2024-04-19T23:35:19.949 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.949 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:19.949 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.949 DEBUG:boto:Params: {} 2024-04-19T23:35:19.949 DEBUG:boto:Token: None 2024-04-19T23:35:19.950 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.950 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:t7FDX4bMOYPjrEp07GP5vU23D1Q= 2024-04-19T23:35:19.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:t7FDX4bMOYPjrEp07GP5vU23D1Q=', 'Content-Length': '0'} 2024-04-19T23:35:19.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d0b263f39cbfcb5-006622ffb7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.953 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:19.953 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:19.953 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.953 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.953 DEBUG:boto:Method: HEAD 2024-04-19T23:35:19.953 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.953 DEBUG:boto:Data: 2024-04-19T23:35:19.953 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.953 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.953 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.953 DEBUG:boto:Params: {} 2024-04-19T23:35:19.954 DEBUG:boto:Token: None 2024-04-19T23:35:19.954 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.954 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yXMDzSUcH81MezeSZR7LhmLtWCY= 2024-04-19T23:35:19.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yXMDzSUcH81MezeSZR7LhmLtWCY=', 'Content-Length': '0'} 2024-04-19T23:35:19.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fb131f915cbe340-006622ffb7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.957 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:19.957 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:19.957 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.957 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.957 DEBUG:boto:Method: HEAD 2024-04-19T23:35:19.957 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.957 DEBUG:boto:Data: 2024-04-19T23:35:19.958 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.958 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:19.958 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.958 DEBUG:boto:Params: {} 2024-04-19T23:35:19.958 DEBUG:boto:Token: None 2024-04-19T23:35:19.958 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.958 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yXMDzSUcH81MezeSZR7LhmLtWCY= 2024-04-19T23:35:19.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yXMDzSUcH81MezeSZR7LhmLtWCY=', 'Content-Length': '0'} 2024-04-19T23:35:19.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfe390740ab68342-006622ffb7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:19.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:19.961 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:19.961 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:19.961 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.962 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.962 DEBUG:boto:Method: GET 2024-04-19T23:35:19.962 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.962 DEBUG:boto:Data: 2024-04-19T23:35:19.962 DEBUG:boto:Headers: {} 2024-04-19T23:35:19.962 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:19.963 DEBUG:boto:Port: 8000 2024-04-19T23:35:19.963 DEBUG:boto:Params: {} 2024-04-19T23:35:19.963 DEBUG:boto:Token: None 2024-04-19T23:35:19.963 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:19 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:19.963 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:t7FDX4bMOYPjrEp07GP5vU23D1Q= 2024-04-19T23:35:19.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:t7FDX4bMOYPjrEp07GP5vU23D1Q=', 'Content-Length': '0'} 2024-04-19T23:35:19.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bceb2799713fcdd7-006622ffb7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.006 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.007 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.007 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.007 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.007 DEBUG:boto:Method: GET 2024-04-19T23:35:20.008 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.008 DEBUG:boto:Data: 2024-04-19T23:35:20.008 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.008 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.008 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.008 DEBUG:boto:Params: {} 2024-04-19T23:35:20.009 DEBUG:boto:Token: None 2024-04-19T23:35:20.009 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.009 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24= 2024-04-19T23:35:20.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24=', 'Content-Length': '0'} 2024-04-19T23:35:20.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2dcfba5c8a974a4-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.012 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.012 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.012 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.012 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.012 DEBUG:boto:Method: GET 2024-04-19T23:35:20.012 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.012 DEBUG:boto:Data: 2024-04-19T23:35:20.012 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.012 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.012 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.012 DEBUG:boto:Params: {} 2024-04-19T23:35:20.013 DEBUG:boto:Token: None 2024-04-19T23:35:20.013 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.013 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c71e9dfe39d965a-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.016 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.016 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.016 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.016 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.016 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.016 DEBUG:boto:Method: GET 2024-04-19T23:35:20.017 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.017 DEBUG:boto:Data: 2024-04-19T23:35:20.017 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.017 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.017 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.017 DEBUG:boto:Params: {} 2024-04-19T23:35:20.017 DEBUG:boto:Token: None 2024-04-19T23:35:20.017 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.017 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039337ddf624c7f13-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.020 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.020 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.020 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.020 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.021 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:35:20.021 DEBUG:boto:Data: 2024-04-19T23:35:20.021 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.021 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.021 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.021 DEBUG:boto:Params: {} 2024-04-19T23:35:20.021 DEBUG:boto:Token: None 2024-04-19T23:35:20.021 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:35:20.021 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:42KNdREyguPCRQjWChFOY6Wg8zI= 2024-04-19T23:35:20.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:42KNdREyguPCRQjWChFOY6Wg8zI=', 'Content-Length': '0'} 2024-04-19T23:35:20.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005019e3dee7185438-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.024 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.024 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.024 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.024 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:35:20.025 DEBUG:boto:Data: 2024-04-19T23:35:20.025 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.025 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.025 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.025 DEBUG:boto:Params: {} 2024-04-19T23:35:20.025 DEBUG:boto:Token: None 2024-04-19T23:35:20.025 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:35:20.025 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:42KNdREyguPCRQjWChFOY6Wg8zI= 2024-04-19T23:35:20.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:42KNdREyguPCRQjWChFOY6Wg8zI=', 'Content-Length': '0'} 2024-04-19T23:35:20.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060492e6847e84d11-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:20.029 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.029 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.029 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.029 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.029 DEBUG:boto:Method: GET 2024-04-19T23:35:20.029 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.029 DEBUG:boto:Data: 2024-04-19T23:35:20.029 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.029 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.030 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.030 DEBUG:boto:Params: {} 2024-04-19T23:35:20.030 DEBUG:boto:Token: None 2024-04-19T23:35:20.030 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.030 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24= 2024-04-19T23:35:20.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24=', 'Content-Length': '0'} 2024-04-19T23:35:20.032 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f4a511912fdecce-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.074 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.075 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.075 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.075 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.075 DEBUG:boto:Method: GET 2024-04-19T23:35:20.075 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.075 DEBUG:boto:Data: 2024-04-19T23:35:20.075 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.075 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.075 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.075 DEBUG:boto:Params: {} 2024-04-19T23:35:20.075 DEBUG:boto:Token: None 2024-04-19T23:35:20.076 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.076 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24= 2024-04-19T23:35:20.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24=', 'Content-Length': '0'} 2024-04-19T23:35:20.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023910509fd803aab-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.078 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.079 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.079 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.079 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.079 DEBUG:boto:Method: GET 2024-04-19T23:35:20.079 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.079 DEBUG:boto:Data: 2024-04-19T23:35:20.079 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.079 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.079 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.079 DEBUG:boto:Params: {} 2024-04-19T23:35:20.079 DEBUG:boto:Token: None 2024-04-19T23:35:20.080 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.080 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8d3f31bc2ae12cf-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.082 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.083 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.083 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.083 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.083 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.083 DEBUG:boto:Method: GET 2024-04-19T23:35:20.084 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.084 DEBUG:boto:Data: 2024-04-19T23:35:20.084 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.084 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.084 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.084 DEBUG:boto:Params: {} 2024-04-19T23:35:20.084 DEBUG:boto:Token: None 2024-04-19T23:35:20.084 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbe066572340304a-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.087 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:20.088 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.088 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.088 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.088 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.088 DEBUG:boto:Method: GET 2024-04-19T23:35:20.088 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.088 DEBUG:boto:Data: 2024-04-19T23:35:20.088 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.089 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.089 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.089 DEBUG:boto:Params: {} 2024-04-19T23:35:20.089 DEBUG:boto:Token: None 2024-04-19T23:35:20.089 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.089 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9N7DHGn51KyK/to9bA2hysIh9qY= 2024-04-19T23:35:20.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9N7DHGn51KyK/to9bA2hysIh9qY=', 'Content-Length': '0'} 2024-04-19T23:35:20.091 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000607d15c8e64f90ab-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.091 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.091 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.091 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.092 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.092 DEBUG:boto:Method: GET 2024-04-19T23:35:20.092 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.092 DEBUG:boto:Data: 2024-04-19T23:35:20.092 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.092 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.092 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.093 DEBUG:boto:Params: {} 2024-04-19T23:35:20.093 DEBUG:boto:Token: None 2024-04-19T23:35:20.093 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.093 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9N7DHGn51KyK/to9bA2hysIh9qY= 2024-04-19T23:35:20.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9N7DHGn51KyK/to9bA2hysIh9qY=', 'Content-Length': '0'} 2024-04-19T23:35:20.095 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c0eb853c494b31f-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.138 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.139 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.139 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.139 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.139 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.139 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.139 DEBUG:boto:Data: 2024-04-19T23:35:20.139 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.139 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.139 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.139 DEBUG:boto:Params: {} 2024-04-19T23:35:20.140 DEBUG:boto:Token: None 2024-04-19T23:35:20.140 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.140 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rymk8hKWLOLOQvw24gpowOXeGbs= 2024-04-19T23:35:20.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rymk8hKWLOLOQvw24gpowOXeGbs=', 'Content-Length': '0'} 2024-04-19T23:35:20.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005c9f1d68320f317-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.142 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.142 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.142 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.143 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.143 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.143 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.143 DEBUG:boto:Data: 2024-04-19T23:35:20.143 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.143 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.143 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.143 DEBUG:boto:Params: {} 2024-04-19T23:35:20.143 DEBUG:boto:Token: None 2024-04-19T23:35:20.143 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.144 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rymk8hKWLOLOQvw24gpowOXeGbs= 2024-04-19T23:35:20.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rymk8hKWLOLOQvw24gpowOXeGbs=', 'Content-Length': '0'} 2024-04-19T23:35:20.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea4e87c4ea30d06b-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:20.146 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.146 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.146 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.146 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.146 DEBUG:boto:Method: GET 2024-04-19T23:35:20.146 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.146 DEBUG:boto:Data: 2024-04-19T23:35:20.147 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.147 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.147 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.147 DEBUG:boto:Params: {} 2024-04-19T23:35:20.147 DEBUG:boto:Token: None 2024-04-19T23:35:20.147 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.147 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9N7DHGn51KyK/to9bA2hysIh9qY= 2024-04-19T23:35:20.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9N7DHGn51KyK/to9bA2hysIh9qY=', 'Content-Length': '0'} 2024-04-19T23:35:20.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a30d1c1cd73a710-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.149 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.149 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.150 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.150 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.150 DEBUG:boto:Method: GET 2024-04-19T23:35:20.150 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.150 DEBUG:boto:Data: 2024-04-19T23:35:20.150 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.150 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.150 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.150 DEBUG:boto:Params: {} 2024-04-19T23:35:20.150 DEBUG:boto:Token: None 2024-04-19T23:35:20.151 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.151 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9N7DHGn51KyK/to9bA2hysIh9qY= 2024-04-19T23:35:20.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9N7DHGn51KyK/to9bA2hysIh9qY=', 'Content-Length': '0'} 2024-04-19T23:35:20.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5b0e616aae36145-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.194 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.195 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.195 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:20.195 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:20.195 DEBUG:boto:Method: GET 2024-04-19T23:35:20.195 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:20.195 DEBUG:boto:Data: 2024-04-19T23:35:20.195 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.195 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.195 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.196 DEBUG:boto:Params: {} 2024-04-19T23:35:20.196 DEBUG:boto:Token: None 2024-04-19T23:35:20.196 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.196 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007d0bbd892765fbf-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.198 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.199 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.199 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.199 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:20.199 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:20.199 DEBUG:boto:Method: GET 2024-04-19T23:35:20.199 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:20.200 DEBUG:boto:Data: 2024-04-19T23:35:20.200 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.200 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.200 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.200 DEBUG:boto:Params: {} 2024-04-19T23:35:20.200 DEBUG:boto:Token: None 2024-04-19T23:35:20.200 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.200 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ccd9aa809941259-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.246 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.247 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.247 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.247 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.247 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:35:20.247 DEBUG:boto:Data: 2024-04-19T23:35:20.247 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.247 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.247 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.248 DEBUG:boto:Params: {} 2024-04-19T23:35:20.248 DEBUG:boto:Token: None 2024-04-19T23:35:20.248 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:35:20.248 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g8CHv/rPVa/dXrBwGmRHKPHJVvE= 2024-04-19T23:35:20.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g8CHv/rPVa/dXrBwGmRHKPHJVvE=', 'Content-Length': '0'} 2024-04-19T23:35:20.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001912032576d174b4-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.251 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.251 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.251 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.251 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:35:20.251 DEBUG:boto:Data: 2024-04-19T23:35:20.251 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.251 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.251 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.252 DEBUG:boto:Params: {} 2024-04-19T23:35:20.252 DEBUG:boto:Token: None 2024-04-19T23:35:20.252 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:35:20.252 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g8CHv/rPVa/dXrBwGmRHKPHJVvE= 2024-04-19T23:35:20.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g8CHv/rPVa/dXrBwGmRHKPHJVvE=', 'Content-Length': '0'} 2024-04-19T23:35:20.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc6b1bfc464b1af2-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:20.255 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.255 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.255 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.255 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.255 DEBUG:boto:Method: GET 2024-04-19T23:35:20.255 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.255 DEBUG:boto:Data: 2024-04-19T23:35:20.255 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.255 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.255 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.256 DEBUG:boto:Params: {} 2024-04-19T23:35:20.256 DEBUG:boto:Token: None 2024-04-19T23:35:20.256 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.256 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9N7DHGn51KyK/to9bA2hysIh9qY= 2024-04-19T23:35:20.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9N7DHGn51KyK/to9bA2hysIh9qY=', 'Content-Length': '0'} 2024-04-19T23:35:20.258 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb7cc4c08ac3353d-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.298 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.299 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:20.299 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.299 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.299 DEBUG:boto:Method: GET 2024-04-19T23:35:20.299 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.299 DEBUG:boto:Data: 2024-04-19T23:35:20.299 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.299 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.299 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.299 DEBUG:boto:Params: {} 2024-04-19T23:35:20.299 DEBUG:boto:Token: None 2024-04-19T23:35:20.300 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:20.300 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9N7DHGn51KyK/to9bA2hysIh9qY= 2024-04-19T23:35:20.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9N7DHGn51KyK/to9bA2hysIh9qY=', 'Content-Length': '0'} 2024-04-19T23:35:20.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc7264e982b87d13-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.346 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.347 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.347 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:20.347 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:20.348 DEBUG:boto:Method: GET 2024-04-19T23:35:20.348 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:20.348 DEBUG:boto:Data: 2024-04-19T23:35:20.348 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.348 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.348 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.349 DEBUG:boto:Params: {} 2024-04-19T23:35:20.349 DEBUG:boto:Token: None 2024-04-19T23:35:20.349 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.350 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0589ca2d1c878c9-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.353 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.353 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.353 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.353 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:20.353 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:20.353 DEBUG:boto:Method: GET 2024-04-19T23:35:20.353 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:20.354 DEBUG:boto:Data: 2024-04-19T23:35:20.354 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.354 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.354 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.354 DEBUG:boto:Params: {} 2024-04-19T23:35:20.354 DEBUG:boto:Token: None 2024-04-19T23:35:20.359 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.359 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e22f5a099cc69d63-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.406 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:20.407 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.407 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.408 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.408 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.408 DEBUG:boto:Method: GET 2024-04-19T23:35:20.408 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.408 DEBUG:boto:Data: 2024-04-19T23:35:20.408 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.408 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.409 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.409 DEBUG:boto:Params: {} 2024-04-19T23:35:20.409 DEBUG:boto:Token: None 2024-04-19T23:35:20.409 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.409 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vkMZ27CygQAnon1dTaknd41vIuQ= 2024-04-19T23:35:20.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vkMZ27CygQAnon1dTaknd41vIuQ=', 'Content-Length': '0'} 2024-04-19T23:35:20.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9fc357d52497abd-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.412 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.412 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.413 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.413 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.413 DEBUG:boto:Method: GET 2024-04-19T23:35:20.413 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.413 DEBUG:boto:Data: 2024-04-19T23:35:20.413 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.413 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.414 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.414 DEBUG:boto:Params: {} 2024-04-19T23:35:20.414 DEBUG:boto:Token: None 2024-04-19T23:35:20.414 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.415 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vkMZ27CygQAnon1dTaknd41vIuQ= 2024-04-19T23:35:20.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vkMZ27CygQAnon1dTaknd41vIuQ=', 'Content-Length': '0'} 2024-04-19T23:35:20.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca0e6e85cabda2d5-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.418 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.418 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.418 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.418 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.418 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.418 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.419 DEBUG:boto:Data: 2024-04-19T23:35:20.419 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.419 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.419 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.419 DEBUG:boto:Params: {} 2024-04-19T23:35:20.419 DEBUG:boto:Token: None 2024-04-19T23:35:20.419 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.420 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9eZhFQLviLK4Z6m+YdtZl2O9cIo= 2024-04-19T23:35:20.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9eZhFQLviLK4Z6m+YdtZl2O9cIo=', 'Content-Length': '0'} 2024-04-19T23:35:20.421 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b2c8266c198d4b5-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.422 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.422 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.422 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.422 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.422 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.422 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.422 DEBUG:boto:Data: 2024-04-19T23:35:20.423 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.423 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.423 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.423 DEBUG:boto:Params: {} 2024-04-19T23:35:20.423 DEBUG:boto:Token: None 2024-04-19T23:35:20.423 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.423 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9eZhFQLviLK4Z6m+YdtZl2O9cIo= 2024-04-19T23:35:20.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9eZhFQLviLK4Z6m+YdtZl2O9cIo=', 'Content-Length': '0'} 2024-04-19T23:35:20.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b744fb8046a63dd2-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:20.426 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.426 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.426 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.426 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.426 DEBUG:boto:Method: GET 2024-04-19T23:35:20.426 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.426 DEBUG:boto:Data: 2024-04-19T23:35:20.426 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.426 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.427 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.427 DEBUG:boto:Params: {} 2024-04-19T23:35:20.427 DEBUG:boto:Token: None 2024-04-19T23:35:20.427 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.427 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vkMZ27CygQAnon1dTaknd41vIuQ= 2024-04-19T23:35:20.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vkMZ27CygQAnon1dTaknd41vIuQ=', 'Content-Length': '0'} 2024-04-19T23:35:20.429 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5a1ff9ae6a2fb4d-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.470 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.471 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.471 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.471 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.471 DEBUG:boto:Method: GET 2024-04-19T23:35:20.471 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.471 DEBUG:boto:Data: 2024-04-19T23:35:20.471 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.471 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.471 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.471 DEBUG:boto:Params: {} 2024-04-19T23:35:20.472 DEBUG:boto:Token: None 2024-04-19T23:35:20.472 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.472 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vkMZ27CygQAnon1dTaknd41vIuQ= 2024-04-19T23:35:20.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vkMZ27CygQAnon1dTaknd41vIuQ=', 'Content-Length': '0'} 2024-04-19T23:35:20.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fce89efb249c7cc1-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.518 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.519 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.519 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:20.519 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:20.519 DEBUG:boto:Method: GET 2024-04-19T23:35:20.519 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:20.519 DEBUG:boto:Data: 2024-04-19T23:35:20.519 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.519 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.519 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.519 DEBUG:boto:Params: {} 2024-04-19T23:35:20.520 DEBUG:boto:Token: None 2024-04-19T23:35:20.520 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.520 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034a4ad4dfc49bbd7-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.523 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.524 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.524 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.524 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:20.524 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:20.524 DEBUG:boto:Method: GET 2024-04-19T23:35:20.524 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:20.524 DEBUG:boto:Data: 2024-04-19T23:35:20.525 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.525 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.525 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.525 DEBUG:boto:Params: {} 2024-04-19T23:35:20.525 DEBUG:boto:Token: None 2024-04-19T23:35:20.525 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.525 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000863f6d674353aa04-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.528 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.528 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.528 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.529 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.529 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:35:20.529 DEBUG:boto:Data: 2024-04-19T23:35:20.529 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.529 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.529 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.529 DEBUG:boto:Params: {} 2024-04-19T23:35:20.529 DEBUG:boto:Token: None 2024-04-19T23:35:20.529 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:35:20.530 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:t9Rh0jpoJDmd+glH0J2vW9cwrbc= 2024-04-19T23:35:20.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:t9Rh0jpoJDmd+glH0J2vW9cwrbc=', 'Content-Length': '0'} 2024-04-19T23:35:20.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f4d0088738a6fe6-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.532 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.532 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.532 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.533 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:35:20.533 DEBUG:boto:Data: 2024-04-19T23:35:20.533 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.533 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.533 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.533 DEBUG:boto:Params: {} 2024-04-19T23:35:20.533 DEBUG:boto:Token: None 2024-04-19T23:35:20.533 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:35:20.533 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:t9Rh0jpoJDmd+glH0J2vW9cwrbc= 2024-04-19T23:35:20.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:t9Rh0jpoJDmd+glH0J2vW9cwrbc=', 'Content-Length': '0'} 2024-04-19T23:35:20.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb676799b5eadfff-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:20.537 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.537 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.537 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.537 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.537 DEBUG:boto:Method: GET 2024-04-19T23:35:20.537 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.537 DEBUG:boto:Data: 2024-04-19T23:35:20.537 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.537 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.537 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.538 DEBUG:boto:Params: {} 2024-04-19T23:35:20.538 DEBUG:boto:Token: None 2024-04-19T23:35:20.538 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.538 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vkMZ27CygQAnon1dTaknd41vIuQ= 2024-04-19T23:35:20.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vkMZ27CygQAnon1dTaknd41vIuQ=', 'Content-Length': '0'} 2024-04-19T23:35:20.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088a2f857e4332922-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.586 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.587 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:20.587 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.587 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.588 DEBUG:boto:Method: GET 2024-04-19T23:35:20.588 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.588 DEBUG:boto:Data: 2024-04-19T23:35:20.588 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.588 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.589 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.589 DEBUG:boto:Params: {} 2024-04-19T23:35:20.589 DEBUG:boto:Token: None 2024-04-19T23:35:20.589 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:20.590 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vkMZ27CygQAnon1dTaknd41vIuQ= 2024-04-19T23:35:20.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vkMZ27CygQAnon1dTaknd41vIuQ=', 'Content-Length': '0'} 2024-04-19T23:35:20.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d0c9fa5db597e06-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.638 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.639 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.639 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:20.639 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:20.639 DEBUG:boto:Method: GET 2024-04-19T23:35:20.639 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:20.639 DEBUG:boto:Data: 2024-04-19T23:35:20.640 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.640 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:20.640 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.640 DEBUG:boto:Params: {} 2024-04-19T23:35:20.640 DEBUG:boto:Token: None 2024-04-19T23:35:20.641 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.641 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef69078373383e38-006622ffb8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.644 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.645 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.645 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.645 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:20.645 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:20.646 DEBUG:boto:Method: GET 2024-04-19T23:35:20.646 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:20.646 DEBUG:boto:Data: 2024-04-19T23:35:20.646 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.646 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.647 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.647 DEBUG:boto:Params: {} 2024-04-19T23:35:20.647 DEBUG:boto:Token: None 2024-04-19T23:35:20.647 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.647 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000473bcd9f31831c9f-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.650 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.651 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-21 zones={test-zone2, test-zone1} 2024-04-19T23:35:20.651 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-21 zones={test-zone3, test-zone1} 2024-04-19T23:35:20.651 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.651 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.651 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.652 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:35:20.652 DEBUG:boto:Data: 2024-04-19T23:35:20.652 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.652 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.652 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.652 DEBUG:boto:Params: {} 2024-04-19T23:35:20.652 DEBUG:boto:Token: None 2024-04-19T23:35:20.652 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.652 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6Nsk7XBiIxeevqVZCFMYHmZIKoc= 2024-04-19T23:35:20.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6Nsk7XBiIxeevqVZCFMYHmZIKoc=', 'Content-Length': '0'} 2024-04-19T23:35:20.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000aa544a7a5e7db7c-006622ffb8-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.655 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.655 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.655 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.655 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:35:20.655 DEBUG:boto:Data: 2024-04-19T23:35:20.655 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.655 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.655 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.656 DEBUG:boto:Params: {} 2024-04-19T23:35:20.656 DEBUG:boto:Token: None 2024-04-19T23:35:20.656 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.656 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6Nsk7XBiIxeevqVZCFMYHmZIKoc= 2024-04-19T23:35:20.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6Nsk7XBiIxeevqVZCFMYHmZIKoc=', 'Content-Length': '0'} 2024-04-19T23:35:20.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a5a5dc3dc1cd086-006622ffb8-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.659 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:35:20.659 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.659 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.659 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:20.659 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:20.659 DEBUG:boto:Method: GET 2024-04-19T23:35:20.659 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:35:20.660 DEBUG:boto:Data: 2024-04-19T23:35:20.660 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.660 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.660 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.660 DEBUG:boto:Params: {} 2024-04-19T23:35:20.660 DEBUG:boto:Token: None 2024-04-19T23:35:20.660 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/?versions 2024-04-19T23:35:20.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:znjny1fnxQF8tCTKGlk3DjMfB+g= 2024-04-19T23:35:20.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:znjny1fnxQF8tCTKGlk3DjMfB+g=', 'Content-Length': '0'} 2024-04-19T23:35:20.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d832024995a3b3fe-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.664 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.664 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:35:20.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:35:20.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:35:20.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:35:20.665 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:35:20.665 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.665 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.665 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:20.665 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:20.665 DEBUG:boto:Method: GET 2024-04-19T23:35:20.665 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:35:20.665 DEBUG:boto:Data: 2024-04-19T23:35:20.665 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.666 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.666 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.666 DEBUG:boto:Params: {} 2024-04-19T23:35:20.666 DEBUG:boto:Token: None 2024-04-19T23:35:20.666 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/?versions 2024-04-19T23:35:20.666 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:znjny1fnxQF8tCTKGlk3DjMfB+g= 2024-04-19T23:35:20.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:znjny1fnxQF8tCTKGlk3DjMfB+g=', 'Content-Length': '0'} 2024-04-19T23:35:20.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019056235fcd060ec-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.669 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.669 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:35:20.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:35:20.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:35:20.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:35:20.670 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.670 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.670 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:20.670 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:20.670 DEBUG:boto:Method: GET 2024-04-19T23:35:20.670 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:35:20.670 DEBUG:boto:Data: 2024-04-19T23:35:20.670 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.671 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.671 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.671 DEBUG:boto:Params: {} 2024-04-19T23:35:20.671 DEBUG:boto:Token: None 2024-04-19T23:35:20.671 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/?versions 2024-04-19T23:35:20.671 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:znjny1fnxQF8tCTKGlk3DjMfB+g= 2024-04-19T23:35:20.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:znjny1fnxQF8tCTKGlk3DjMfB+g=', 'Content-Length': '0'} 2024-04-19T23:35:20.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0041dc65c6f9e8e-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.673 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.674 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.674 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.674 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:20.674 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:20.674 DEBUG:boto:Method: GET 2024-04-19T23:35:20.674 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:35:20.674 DEBUG:boto:Data: 2024-04-19T23:35:20.675 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.675 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.675 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.675 DEBUG:boto:Params: {} 2024-04-19T23:35:20.675 DEBUG:boto:Token: None 2024-04-19T23:35:20.675 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/?versions 2024-04-19T23:35:20.675 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:znjny1fnxQF8tCTKGlk3DjMfB+g= 2024-04-19T23:35:20.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:znjny1fnxQF8tCTKGlk3DjMfB+g=', 'Content-Length': '0'} 2024-04-19T23:35:20.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afcb41dc5377a0bf-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.678 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:20.678 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:20.678 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:20.679 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.679 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.679 DEBUG:boto:Method: GET 2024-04-19T23:35:20.679 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.679 DEBUG:boto:Data: 2024-04-19T23:35:20.679 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.679 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.679 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.679 DEBUG:boto:Params: {} 2024-04-19T23:35:20.679 DEBUG:boto:Token: None 2024-04-19T23:35:20.680 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.680 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dv5F1L3/3Gdm8WJoNS3oEbag/Mo= 2024-04-19T23:35:20.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dv5F1L3/3Gdm8WJoNS3oEbag/Mo=', 'Content-Length': '0'} 2024-04-19T23:35:20.681 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e8dedea15c636b0-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.682 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:20.682 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:20.682 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.682 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.682 DEBUG:boto:Method: GET 2024-04-19T23:35:20.682 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.682 DEBUG:boto:Data: 2024-04-19T23:35:20.682 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.682 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.683 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.683 DEBUG:boto:Params: {} 2024-04-19T23:35:20.683 DEBUG:boto:Token: None 2024-04-19T23:35:20.683 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.683 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dv5F1L3/3Gdm8WJoNS3oEbag/Mo= 2024-04-19T23:35:20.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dv5F1L3/3Gdm8WJoNS3oEbag/Mo=', 'Content-Length': '0'} 2024-04-19T23:35:20.685 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ff9808feaeb4942-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.685 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:20.685 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:20.685 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.685 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.685 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.686 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.686 DEBUG:boto:Data: 2024-04-19T23:35:20.686 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.686 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.686 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.686 DEBUG:boto:Params: {} 2024-04-19T23:35:20.686 DEBUG:boto:Token: None 2024-04-19T23:35:20.686 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.686 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dzw33WOmjjIPU8uXtP/Oi9jdfLk= 2024-04-19T23:35:20.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dzw33WOmjjIPU8uXtP/Oi9jdfLk=', 'Content-Length': '0'} 2024-04-19T23:35:20.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c42cae939243093-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.689 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:20.689 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:20.689 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.689 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.689 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.689 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.689 DEBUG:boto:Data: 2024-04-19T23:35:20.689 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.690 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.690 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.690 DEBUG:boto:Params: {} 2024-04-19T23:35:20.690 DEBUG:boto:Token: None 2024-04-19T23:35:20.690 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.690 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dzw33WOmjjIPU8uXtP/Oi9jdfLk= 2024-04-19T23:35:20.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dzw33WOmjjIPU8uXtP/Oi9jdfLk=', 'Content-Length': '0'} 2024-04-19T23:35:20.692 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa5805396821c20b-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:20.692 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:20.692 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:20.693 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.693 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.693 DEBUG:boto:Method: GET 2024-04-19T23:35:20.693 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.693 DEBUG:boto:Data: 2024-04-19T23:35:20.693 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.693 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.693 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.693 DEBUG:boto:Params: {} 2024-04-19T23:35:20.693 DEBUG:boto:Token: None 2024-04-19T23:35:20.693 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.694 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dv5F1L3/3Gdm8WJoNS3oEbag/Mo= 2024-04-19T23:35:20.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dv5F1L3/3Gdm8WJoNS3oEbag/Mo=', 'Content-Length': '0'} 2024-04-19T23:35:20.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000679bcaca2bcfaee8-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.696 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:20.696 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:20.696 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.696 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.696 DEBUG:boto:Method: GET 2024-04-19T23:35:20.696 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.696 DEBUG:boto:Data: 2024-04-19T23:35:20.697 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.697 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.697 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.697 DEBUG:boto:Params: {} 2024-04-19T23:35:20.697 DEBUG:boto:Token: None 2024-04-19T23:35:20.697 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.697 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dv5F1L3/3Gdm8WJoNS3oEbag/Mo= 2024-04-19T23:35:20.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dv5F1L3/3Gdm8WJoNS3oEbag/Mo=', 'Content-Length': '0'} 2024-04-19T23:35:20.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000948d9a8ae7063119-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.700 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.700 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.700 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:20.700 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:20.700 DEBUG:boto:Method: GET 2024-04-19T23:35:20.700 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:35:20.700 DEBUG:boto:Data: 2024-04-19T23:35:20.700 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.700 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.700 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.701 DEBUG:boto:Params: {} 2024-04-19T23:35:20.701 DEBUG:boto:Token: None 2024-04-19T23:35:20.701 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.701 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5d6caf259f3373f-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.750 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.751 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.751 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.751 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:20.752 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:20.752 DEBUG:boto:Method: GET 2024-04-19T23:35:20.752 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:35:20.752 DEBUG:boto:Data: 2024-04-19T23:35:20.753 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.753 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.753 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.753 DEBUG:boto:Params: {} 2024-04-19T23:35:20.754 DEBUG:boto:Token: None 2024-04-19T23:35:20.754 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.754 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e0ac3c3114bbc9b-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.798 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.799 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:20.799 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:20.799 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.799 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:35:20.799 DEBUG:boto:Data: 2024-04-19T23:35:20.799 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.799 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.799 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.800 DEBUG:boto:Params: {} 2024-04-19T23:35:20.800 DEBUG:boto:Token: None 2024-04-19T23:35:20.800 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/dummy 2024-04-19T23:35:20.800 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:73sKvLuis0TDOsomaJrhp2vfmx4= 2024-04-19T23:35:20.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:73sKvLuis0TDOsomaJrhp2vfmx4=', 'Content-Length': '0'} 2024-04-19T23:35:20.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cc7d263ba6c0e1f0-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.803 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:20.803 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:20.803 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.803 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:35:20.803 DEBUG:boto:Data: 2024-04-19T23:35:20.803 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.804 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.804 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.804 DEBUG:boto:Params: {} 2024-04-19T23:35:20.804 DEBUG:boto:Token: None 2024-04-19T23:35:20.804 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/dummy 2024-04-19T23:35:20.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:73sKvLuis0TDOsomaJrhp2vfmx4= 2024-04-19T23:35:20.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:73sKvLuis0TDOsomaJrhp2vfmx4=', 'Content-Length': '0'} 2024-04-19T23:35:20.807 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000015a6e59f29bf5677-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:20.807 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:20.807 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:20.808 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.808 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.808 DEBUG:boto:Method: GET 2024-04-19T23:35:20.808 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.808 DEBUG:boto:Data: 2024-04-19T23:35:20.808 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.808 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.808 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.809 DEBUG:boto:Params: {} 2024-04-19T23:35:20.809 DEBUG:boto:Token: None 2024-04-19T23:35:20.809 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.809 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dv5F1L3/3Gdm8WJoNS3oEbag/Mo= 2024-04-19T23:35:20.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dv5F1L3/3Gdm8WJoNS3oEbag/Mo=', 'Content-Length': '0'} 2024-04-19T23:35:20.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056e6e8d347e4e75f-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.811 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:20.811 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:20.811 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.811 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.811 DEBUG:boto:Method: GET 2024-04-19T23:35:20.811 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.811 DEBUG:boto:Data: 2024-04-19T23:35:20.811 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.811 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.812 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.812 DEBUG:boto:Params: {} 2024-04-19T23:35:20.812 DEBUG:boto:Token: None 2024-04-19T23:35:20.812 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:20.812 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dv5F1L3/3Gdm8WJoNS3oEbag/Mo= 2024-04-19T23:35:20.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dv5F1L3/3Gdm8WJoNS3oEbag/Mo=', 'Content-Length': '0'} 2024-04-19T23:35:20.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000014faf62d7c84c8d2-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.815 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.815 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.815 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:20.815 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:20.815 DEBUG:boto:Method: GET 2024-04-19T23:35:20.816 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:35:20.816 DEBUG:boto:Data: 2024-04-19T23:35:20.816 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.816 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.816 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.816 DEBUG:boto:Params: {} 2024-04-19T23:35:20.817 DEBUG:boto:Token: None 2024-04-19T23:35:20.817 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.817 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ebb77ce5080b4f5-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.819 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.820 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.820 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.820 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:20.820 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:20.820 DEBUG:boto:Method: GET 2024-04-19T23:35:20.820 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:35:20.820 DEBUG:boto:Data: 2024-04-19T23:35:20.821 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.821 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.821 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.821 DEBUG:boto:Params: {} 2024-04-19T23:35:20.821 DEBUG:boto:Token: None 2024-04-19T23:35:20.821 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.821 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000071e0846eceb0279-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.824 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:20.825 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.825 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.825 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.825 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.826 DEBUG:boto:Method: GET 2024-04-19T23:35:20.826 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.826 DEBUG:boto:Data: 2024-04-19T23:35:20.826 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.826 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.826 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.826 DEBUG:boto:Params: {} 2024-04-19T23:35:20.826 DEBUG:boto:Token: None 2024-04-19T23:35:20.826 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.827 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24= 2024-04-19T23:35:20.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24=', 'Content-Length': '0'} 2024-04-19T23:35:20.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a596e1b4131f111-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.870 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.871 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.871 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.871 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.871 DEBUG:boto:Method: GET 2024-04-19T23:35:20.871 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.871 DEBUG:boto:Data: 2024-04-19T23:35:20.871 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.871 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.871 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.871 DEBUG:boto:Params: {} 2024-04-19T23:35:20.872 DEBUG:boto:Token: None 2024-04-19T23:35:20.872 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.872 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24= 2024-04-19T23:35:20.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24=', 'Content-Length': '0'} 2024-04-19T23:35:20.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000535bb4f1061afcc3-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.918 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.919 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.919 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.919 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.919 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.919 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.919 DEBUG:boto:Data: 2024-04-19T23:35:20.919 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.919 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.920 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.920 DEBUG:boto:Params: {} 2024-04-19T23:35:20.920 DEBUG:boto:Token: None 2024-04-19T23:35:20.920 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.920 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaLTggH2NIgmesL2B3gVbj9hmR8= 2024-04-19T23:35:20.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaLTggH2NIgmesL2B3gVbj9hmR8=', 'Content-Length': '0'} 2024-04-19T23:35:20.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e805d3e68d21f442-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.922 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.923 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.923 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.923 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.923 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.923 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.923 DEBUG:boto:Data: 2024-04-19T23:35:20.923 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.923 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.923 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.923 DEBUG:boto:Params: {} 2024-04-19T23:35:20.923 DEBUG:boto:Token: None 2024-04-19T23:35:20.924 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.924 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaLTggH2NIgmesL2B3gVbj9hmR8= 2024-04-19T23:35:20.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaLTggH2NIgmesL2B3gVbj9hmR8=', 'Content-Length': '0'} 2024-04-19T23:35:20.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b8a093ff8a2b9fc-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:20.926 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.926 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.926 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.927 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.927 DEBUG:boto:Method: GET 2024-04-19T23:35:20.927 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.927 DEBUG:boto:Data: 2024-04-19T23:35:20.927 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.927 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.927 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.927 DEBUG:boto:Params: {} 2024-04-19T23:35:20.927 DEBUG:boto:Token: None 2024-04-19T23:35:20.927 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.928 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24= 2024-04-19T23:35:20.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24=', 'Content-Length': '0'} 2024-04-19T23:35:20.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9b2b87ba2149a1b-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.970 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.971 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.971 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.971 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.971 DEBUG:boto:Method: GET 2024-04-19T23:35:20.971 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.971 DEBUG:boto:Data: 2024-04-19T23:35:20.971 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.971 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.971 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.972 DEBUG:boto:Params: {} 2024-04-19T23:35:20.972 DEBUG:boto:Token: None 2024-04-19T23:35:20.972 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:20.972 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24= 2024-04-19T23:35:20.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gio1T24xRwV6Aj9RbJzVd7w3w24=', 'Content-Length': '0'} 2024-04-19T23:35:20.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059a26dae071520d5-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.975 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.975 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.975 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.976 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.976 DEBUG:boto:Method: GET 2024-04-19T23:35:20.976 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.976 DEBUG:boto:Data: 2024-04-19T23:35:20.976 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.976 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.976 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.977 DEBUG:boto:Params: {} 2024-04-19T23:35:20.977 DEBUG:boto:Token: None 2024-04-19T23:35:20.977 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.977 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb54ec851b988307-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.981 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.982 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:20.982 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:20.983 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.983 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.983 DEBUG:boto:Method: GET 2024-04-19T23:35:20.983 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:20.983 DEBUG:boto:Data: 2024-04-19T23:35:20.983 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.983 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.983 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.983 DEBUG:boto:Params: {} 2024-04-19T23:35:20.984 DEBUG:boto:Token: None 2024-04-19T23:35:20.984 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/ 2024-04-19T23:35:20.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k= 2024-04-19T23:35:20.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fp4GgBEqK4PacVEBdULQoNKHF6k=', 'Content-Length': '0'} 2024-04-19T23:35:20.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042e753f836d778e8-006622ffb8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.987 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:20.987 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.987 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.987 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.988 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:35:20.988 DEBUG:boto:Data: 2024-04-19T23:35:20.988 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.988 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:20.988 DEBUG:boto:Port: 8001 2024-04-19T23:35:20.988 DEBUG:boto:Params: {} 2024-04-19T23:35:20.988 DEBUG:boto:Token: None 2024-04-19T23:35:20.989 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:35:20.989 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:42KNdREyguPCRQjWChFOY6Wg8zI= 2024-04-19T23:35:20.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:42KNdREyguPCRQjWChFOY6Wg8zI=', 'Content-Length': '0'} 2024-04-19T23:35:20.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ae99fdb34559325-006622ffb8-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.993 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.993 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.994 DEBUG:boto:Method: HEAD 2024-04-19T23:35:20.994 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:35:20.994 DEBUG:boto:Data: 2024-04-19T23:35:20.994 DEBUG:boto:Headers: {} 2024-04-19T23:35:20.994 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:20.994 DEBUG:boto:Port: 8000 2024-04-19T23:35:20.995 DEBUG:boto:Params: {} 2024-04-19T23:35:20.995 DEBUG:boto:Token: None 2024-04-19T23:35:20.995 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:20 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:35:20.995 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:42KNdREyguPCRQjWChFOY6Wg8zI= 2024-04-19T23:35:20.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:42KNdREyguPCRQjWChFOY6Wg8zI=', 'Content-Length': '0'} 2024-04-19T23:35:20.999 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6b5dd5d00dbf032-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:20.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:20.999 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.999 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:20.999 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.000 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.000 DEBUG:boto:Method: GET 2024-04-19T23:35:21.000 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.000 DEBUG:boto:Data: 2024-04-19T23:35:21.000 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.000 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.000 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.000 DEBUG:boto:Params: {} 2024-04-19T23:35:21.000 DEBUG:boto:Token: None 2024-04-19T23:35:21.000 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.001 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BSB/4NtjpFhFyuu/RH9B+rpxFz4= 2024-04-19T23:35:21.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BSB/4NtjpFhFyuu/RH9B+rpxFz4=', 'Content-Length': '0'} 2024-04-19T23:35:21.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef23004f094a4b43-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.051 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.051 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.051 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.051 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.052 DEBUG:boto:Method: GET 2024-04-19T23:35:21.052 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.052 DEBUG:boto:Data: 2024-04-19T23:35:21.052 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.052 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.053 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.053 DEBUG:boto:Params: {} 2024-04-19T23:35:21.053 DEBUG:boto:Token: None 2024-04-19T23:35:21.053 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.053 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BSB/4NtjpFhFyuu/RH9B+rpxFz4= 2024-04-19T23:35:21.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BSB/4NtjpFhFyuu/RH9B+rpxFz4=', 'Content-Length': '0'} 2024-04-19T23:35:21.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077e028dedbb7b0aa-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.056 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.057 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.057 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:21.057 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:21.057 DEBUG:boto:Method: GET 2024-04-19T23:35:21.057 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:21.057 DEBUG:boto:Data: 2024-04-19T23:35:21.058 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.058 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.058 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.058 DEBUG:boto:Params: {} 2024-04-19T23:35:21.058 DEBUG:boto:Token: None 2024-04-19T23:35:21.059 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.059 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005533c2521d677fb1-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.062 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.062 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.063 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.063 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:21.063 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:21.063 DEBUG:boto:Method: GET 2024-04-19T23:35:21.063 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:21.063 DEBUG:boto:Data: 2024-04-19T23:35:21.063 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.063 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.063 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.063 DEBUG:boto:Params: {} 2024-04-19T23:35:21.063 DEBUG:boto:Token: None 2024-04-19T23:35:21.064 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.064 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002864e39d4da3cf1c-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.067 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:21.067 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.068 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.068 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.068 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.068 DEBUG:boto:Method: GET 2024-04-19T23:35:21.068 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.068 DEBUG:boto:Data: 2024-04-19T23:35:21.068 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.068 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.068 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.068 DEBUG:boto:Params: {} 2024-04-19T23:35:21.068 DEBUG:boto:Token: None 2024-04-19T23:35:21.069 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.069 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:D6lvBXJL3LzlE2+q07cwdah/FCM= 2024-04-19T23:35:21.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:D6lvBXJL3LzlE2+q07cwdah/FCM=', 'Content-Length': '0'} 2024-04-19T23:35:21.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098faa04c787be412-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.071 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.071 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.071 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.071 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.071 DEBUG:boto:Method: GET 2024-04-19T23:35:21.071 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.071 DEBUG:boto:Data: 2024-04-19T23:35:21.071 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.071 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.071 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.071 DEBUG:boto:Params: {} 2024-04-19T23:35:21.072 DEBUG:boto:Token: None 2024-04-19T23:35:21.072 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.072 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:D6lvBXJL3LzlE2+q07cwdah/FCM= 2024-04-19T23:35:21.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:D6lvBXJL3LzlE2+q07cwdah/FCM=', 'Content-Length': '0'} 2024-04-19T23:35:21.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b8dbfac0010d6d1-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.115 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.115 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.115 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.115 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.115 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.115 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.115 DEBUG:boto:Data: 2024-04-19T23:35:21.116 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.116 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.116 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.116 DEBUG:boto:Params: {} 2024-04-19T23:35:21.116 DEBUG:boto:Token: None 2024-04-19T23:35:21.116 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.116 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:v6wr75WsgBdyGqh+uhYiM9bNXfU= 2024-04-19T23:35:21.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:v6wr75WsgBdyGqh+uhYiM9bNXfU=', 'Content-Length': '0'} 2024-04-19T23:35:21.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd49688487b77745-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.119 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.119 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.119 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.119 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.119 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.120 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.120 DEBUG:boto:Data: 2024-04-19T23:35:21.120 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.120 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.120 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.120 DEBUG:boto:Params: {} 2024-04-19T23:35:21.120 DEBUG:boto:Token: None 2024-04-19T23:35:21.121 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.121 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:v6wr75WsgBdyGqh+uhYiM9bNXfU= 2024-04-19T23:35:21.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:v6wr75WsgBdyGqh+uhYiM9bNXfU=', 'Content-Length': '0'} 2024-04-19T23:35:21.123 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000478e580286300837-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:21.124 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.124 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.124 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.124 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.124 DEBUG:boto:Method: GET 2024-04-19T23:35:21.124 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.124 DEBUG:boto:Data: 2024-04-19T23:35:21.124 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.124 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.124 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.125 DEBUG:boto:Params: {} 2024-04-19T23:35:21.125 DEBUG:boto:Token: None 2024-04-19T23:35:21.125 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.125 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:D6lvBXJL3LzlE2+q07cwdah/FCM= 2024-04-19T23:35:21.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:D6lvBXJL3LzlE2+q07cwdah/FCM=', 'Content-Length': '0'} 2024-04-19T23:35:21.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2809311e4708415-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.178 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.179 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.180 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.180 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.180 DEBUG:boto:Method: GET 2024-04-19T23:35:21.180 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.180 DEBUG:boto:Data: 2024-04-19T23:35:21.181 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.181 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.181 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.181 DEBUG:boto:Params: {} 2024-04-19T23:35:21.182 DEBUG:boto:Token: None 2024-04-19T23:35:21.182 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.182 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:D6lvBXJL3LzlE2+q07cwdah/FCM= 2024-04-19T23:35:21.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:D6lvBXJL3LzlE2+q07cwdah/FCM=', 'Content-Length': '0'} 2024-04-19T23:35:21.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0042bcec3a2fcb2-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.226 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.227 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.227 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:21.227 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:21.227 DEBUG:boto:Method: GET 2024-04-19T23:35:21.227 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:21.227 DEBUG:boto:Data: 2024-04-19T23:35:21.227 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.227 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.227 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.227 DEBUG:boto:Params: {} 2024-04-19T23:35:21.228 DEBUG:boto:Token: None 2024-04-19T23:35:21.228 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.228 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004beae14ea331a0a8-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.274 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.275 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.275 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.275 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:21.275 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:21.275 DEBUG:boto:Method: GET 2024-04-19T23:35:21.275 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:21.275 DEBUG:boto:Data: 2024-04-19T23:35:21.275 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.276 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.276 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.276 DEBUG:boto:Params: {} 2024-04-19T23:35:21.276 DEBUG:boto:Token: None 2024-04-19T23:35:21.276 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.276 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007290eeb9c80fed86-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.279 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.279 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.279 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.279 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.279 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:35:21.280 DEBUG:boto:Data: 2024-04-19T23:35:21.280 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.280 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.280 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.280 DEBUG:boto:Params: {} 2024-04-19T23:35:21.280 DEBUG:boto:Token: None 2024-04-19T23:35:21.280 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:35:21.280 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rCp+mIslyxTBTxGAZSnljADL8Kg= 2024-04-19T23:35:21.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rCp+mIslyxTBTxGAZSnljADL8Kg=', 'Content-Length': '0'} 2024-04-19T23:35:21.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009014d17fcd3b6e55-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.283 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.283 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.283 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.283 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:35:21.284 DEBUG:boto:Data: 2024-04-19T23:35:21.284 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.284 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.284 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.284 DEBUG:boto:Params: {} 2024-04-19T23:35:21.284 DEBUG:boto:Token: None 2024-04-19T23:35:21.284 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:35:21.284 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rCp+mIslyxTBTxGAZSnljADL8Kg= 2024-04-19T23:35:21.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rCp+mIslyxTBTxGAZSnljADL8Kg=', 'Content-Length': '0'} 2024-04-19T23:35:21.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c463d361da09589d-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:21.287 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.287 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.287 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.287 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.287 DEBUG:boto:Method: GET 2024-04-19T23:35:21.287 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.287 DEBUG:boto:Data: 2024-04-19T23:35:21.287 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.288 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.288 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.288 DEBUG:boto:Params: {} 2024-04-19T23:35:21.288 DEBUG:boto:Token: None 2024-04-19T23:35:21.288 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.288 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:D6lvBXJL3LzlE2+q07cwdah/FCM= 2024-04-19T23:35:21.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:D6lvBXJL3LzlE2+q07cwdah/FCM=', 'Content-Length': '0'} 2024-04-19T23:35:21.290 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c5f03a152541687-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.334 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.335 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:21.335 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.335 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.335 DEBUG:boto:Method: GET 2024-04-19T23:35:21.335 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.335 DEBUG:boto:Data: 2024-04-19T23:35:21.335 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.335 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.335 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.336 DEBUG:boto:Params: {} 2024-04-19T23:35:21.336 DEBUG:boto:Token: None 2024-04-19T23:35:21.336 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:21.336 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:D6lvBXJL3LzlE2+q07cwdah/FCM= 2024-04-19T23:35:21.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:D6lvBXJL3LzlE2+q07cwdah/FCM=', 'Content-Length': '0'} 2024-04-19T23:35:21.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055067a496474538e-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.382 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.383 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.383 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:21.383 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:21.383 DEBUG:boto:Method: GET 2024-04-19T23:35:21.383 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:21.383 DEBUG:boto:Data: 2024-04-19T23:35:21.383 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.383 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.383 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.383 DEBUG:boto:Params: {} 2024-04-19T23:35:21.384 DEBUG:boto:Token: None 2024-04-19T23:35:21.384 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.384 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000995805855e4751ec-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.430 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.431 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.431 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.431 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:21.431 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:21.432 DEBUG:boto:Method: GET 2024-04-19T23:35:21.432 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:21.432 DEBUG:boto:Data: 2024-04-19T23:35:21.432 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.432 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.432 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.432 DEBUG:boto:Params: {} 2024-04-19T23:35:21.432 DEBUG:boto:Token: None 2024-04-19T23:35:21.432 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.433 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a792decc9879c5b0-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.478 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:21.479 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.479 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.479 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.479 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.479 DEBUG:boto:Method: GET 2024-04-19T23:35:21.480 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.480 DEBUG:boto:Data: 2024-04-19T23:35:21.480 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.480 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.480 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.480 DEBUG:boto:Params: {} 2024-04-19T23:35:21.480 DEBUG:boto:Token: None 2024-04-19T23:35:21.480 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.481 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaAoj5EJx7BFJuf72NdIhA5gsrY= 2024-04-19T23:35:21.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaAoj5EJx7BFJuf72NdIhA5gsrY=', 'Content-Length': '0'} 2024-04-19T23:35:21.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a28f6905e034e005-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.483 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.483 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.483 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.484 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.484 DEBUG:boto:Method: GET 2024-04-19T23:35:21.484 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.484 DEBUG:boto:Data: 2024-04-19T23:35:21.484 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.484 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.484 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.484 DEBUG:boto:Params: {} 2024-04-19T23:35:21.484 DEBUG:boto:Token: None 2024-04-19T23:35:21.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.485 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaAoj5EJx7BFJuf72NdIhA5gsrY= 2024-04-19T23:35:21.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaAoj5EJx7BFJuf72NdIhA5gsrY=', 'Content-Length': '0'} 2024-04-19T23:35:21.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e98aff4fa2d9cda-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.487 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.487 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.487 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.487 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.487 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.487 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.487 DEBUG:boto:Data: 2024-04-19T23:35:21.488 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.488 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.488 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.488 DEBUG:boto:Params: {} 2024-04-19T23:35:21.488 DEBUG:boto:Token: None 2024-04-19T23:35:21.488 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YKjUBpmj3aK9cgasNU6Qt81cj1o= 2024-04-19T23:35:21.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YKjUBpmj3aK9cgasNU6Qt81cj1o=', 'Content-Length': '0'} 2024-04-19T23:35:21.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000168fae300bab42aa-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.490 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.490 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.491 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.491 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.491 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.491 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.491 DEBUG:boto:Data: 2024-04-19T23:35:21.491 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.491 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.491 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.491 DEBUG:boto:Params: {} 2024-04-19T23:35:21.492 DEBUG:boto:Token: None 2024-04-19T23:35:21.492 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.492 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YKjUBpmj3aK9cgasNU6Qt81cj1o= 2024-04-19T23:35:21.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YKjUBpmj3aK9cgasNU6Qt81cj1o=', 'Content-Length': '0'} 2024-04-19T23:35:21.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000637d0cfe3fcd69b6-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:21.494 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.494 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.494 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.495 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.495 DEBUG:boto:Method: GET 2024-04-19T23:35:21.495 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.495 DEBUG:boto:Data: 2024-04-19T23:35:21.495 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.495 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.495 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.495 DEBUG:boto:Params: {} 2024-04-19T23:35:21.495 DEBUG:boto:Token: None 2024-04-19T23:35:21.495 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.496 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaAoj5EJx7BFJuf72NdIhA5gsrY= 2024-04-19T23:35:21.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaAoj5EJx7BFJuf72NdIhA5gsrY=', 'Content-Length': '0'} 2024-04-19T23:35:21.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b3d396694135ee6-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.498 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.498 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.498 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.498 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.498 DEBUG:boto:Method: GET 2024-04-19T23:35:21.498 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.499 DEBUG:boto:Data: 2024-04-19T23:35:21.499 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.499 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.499 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.499 DEBUG:boto:Params: {} 2024-04-19T23:35:21.499 DEBUG:boto:Token: None 2024-04-19T23:35:21.499 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.499 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaAoj5EJx7BFJuf72NdIhA5gsrY= 2024-04-19T23:35:21.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaAoj5EJx7BFJuf72NdIhA5gsrY=', 'Content-Length': '0'} 2024-04-19T23:35:21.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006cc54f416899253f-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.542 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.543 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.543 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:21.543 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:21.543 DEBUG:boto:Method: GET 2024-04-19T23:35:21.543 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:21.543 DEBUG:boto:Data: 2024-04-19T23:35:21.543 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.544 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.544 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.544 DEBUG:boto:Params: {} 2024-04-19T23:35:21.544 DEBUG:boto:Token: None 2024-04-19T23:35:21.544 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.544 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f479a233692d0a4a-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.547 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.547 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.548 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.548 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:21.548 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:21.548 DEBUG:boto:Method: GET 2024-04-19T23:35:21.548 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:21.548 DEBUG:boto:Data: 2024-04-19T23:35:21.548 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.548 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.548 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.548 DEBUG:boto:Params: {} 2024-04-19T23:35:21.549 DEBUG:boto:Token: None 2024-04-19T23:35:21.549 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.549 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d299c726115f6b7e-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.594 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.595 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.595 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.595 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.595 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:35:21.595 DEBUG:boto:Data: 2024-04-19T23:35:21.595 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.596 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.596 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.596 DEBUG:boto:Params: {} 2024-04-19T23:35:21.596 DEBUG:boto:Token: None 2024-04-19T23:35:21.596 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:35:21.596 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HKtTlM5h6hiZFgzCm3H4g8rqr9k= 2024-04-19T23:35:21.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HKtTlM5h6hiZFgzCm3H4g8rqr9k=', 'Content-Length': '0'} 2024-04-19T23:35:21.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f96e9c77b40e74ad-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.600 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.600 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.600 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.600 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:35:21.601 DEBUG:boto:Data: 2024-04-19T23:35:21.601 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.601 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.601 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.601 DEBUG:boto:Params: {} 2024-04-19T23:35:21.601 DEBUG:boto:Token: None 2024-04-19T23:35:21.601 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:35:21.601 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HKtTlM5h6hiZFgzCm3H4g8rqr9k= 2024-04-19T23:35:21.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HKtTlM5h6hiZFgzCm3H4g8rqr9k=', 'Content-Length': '0'} 2024-04-19T23:35:21.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009610ef900b7e5518-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:21.607 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.607 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.607 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.607 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.608 DEBUG:boto:Method: GET 2024-04-19T23:35:21.608 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.608 DEBUG:boto:Data: 2024-04-19T23:35:21.608 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.608 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.609 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.609 DEBUG:boto:Params: {} 2024-04-19T23:35:21.609 DEBUG:boto:Token: None 2024-04-19T23:35:21.609 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.610 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaAoj5EJx7BFJuf72NdIhA5gsrY= 2024-04-19T23:35:21.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaAoj5EJx7BFJuf72NdIhA5gsrY=', 'Content-Length': '0'} 2024-04-19T23:35:21.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000642772b3231a6fb7-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.654 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.654 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:21.655 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.655 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.655 DEBUG:boto:Method: GET 2024-04-19T23:35:21.655 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.655 DEBUG:boto:Data: 2024-04-19T23:35:21.655 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.655 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.655 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.655 DEBUG:boto:Params: {} 2024-04-19T23:35:21.655 DEBUG:boto:Token: None 2024-04-19T23:35:21.656 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:21.656 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaAoj5EJx7BFJuf72NdIhA5gsrY= 2024-04-19T23:35:21.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaAoj5EJx7BFJuf72NdIhA5gsrY=', 'Content-Length': '0'} 2024-04-19T23:35:21.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1f849d39f8fe916-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.698 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.698 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.699 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:21.699 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:21.699 DEBUG:boto:Method: GET 2024-04-19T23:35:21.699 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:21.699 DEBUG:boto:Data: 2024-04-19T23:35:21.699 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.699 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.699 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.699 DEBUG:boto:Params: {} 2024-04-19T23:35:21.699 DEBUG:boto:Token: None 2024-04-19T23:35:21.700 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.700 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000227846f8831653fc-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.702 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.703 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.703 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.703 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:21.703 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:21.703 DEBUG:boto:Method: GET 2024-04-19T23:35:21.703 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:21.703 DEBUG:boto:Data: 2024-04-19T23:35:21.703 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.704 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:21.704 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.704 DEBUG:boto:Params: {} 2024-04-19T23:35:21.704 DEBUG:boto:Token: None 2024-04-19T23:35:21.704 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7d3a774f1e31729-006622ffb9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.707 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.707 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-21 zones={test-zone3, test-zone1} 2024-04-19T23:35:21.707 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-21 zones={test-zone3, test-zone2} 2024-04-19T23:35:21.707 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.707 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.708 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.708 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:35:21.708 DEBUG:boto:Data: 2024-04-19T23:35:21.708 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.708 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.708 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.708 DEBUG:boto:Params: {} 2024-04-19T23:35:21.708 DEBUG:boto:Token: None 2024-04-19T23:35:21.708 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.708 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oIorZnpeFQxKjLuMEKcoJAJ9gJY= 2024-04-19T23:35:21.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oIorZnpeFQxKjLuMEKcoJAJ9gJY=', 'Content-Length': '0'} 2024-04-19T23:35:21.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a028c37727d116de-006622ffb9-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.711 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.711 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.711 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.711 DEBUG:boto:Path: /aktlji-21/ 2024-04-19T23:35:21.711 DEBUG:boto:Data: 2024-04-19T23:35:21.711 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.711 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.712 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.712 DEBUG:boto:Params: {} 2024-04-19T23:35:21.712 DEBUG:boto:Token: None 2024-04-19T23:35:21.712 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.712 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oIorZnpeFQxKjLuMEKcoJAJ9gJY= 2024-04-19T23:35:21.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oIorZnpeFQxKjLuMEKcoJAJ9gJY=', 'Content-Length': '0'} 2024-04-19T23:35:21.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd61b60f3e6b5e6d-006622ffb9-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.715 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:35:21.715 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.715 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.715 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:21.715 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:21.715 DEBUG:boto:Method: GET 2024-04-19T23:35:21.715 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:35:21.716 DEBUG:boto:Data: 2024-04-19T23:35:21.716 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.716 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.716 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.716 DEBUG:boto:Params: {} 2024-04-19T23:35:21.716 DEBUG:boto:Token: None 2024-04-19T23:35:21.716 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/?versions 2024-04-19T23:35:21.716 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KheWDoPF2GOcu4dyFAHw94Cw1pI= 2024-04-19T23:35:21.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KheWDoPF2GOcu4dyFAHw94Cw1pI=', 'Content-Length': '0'} 2024-04-19T23:35:21.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004dbaa48cfa29916-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.719 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.720 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:35:21.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:35:21.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:35:21.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:35:21.720 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:35:21.720 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.720 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.720 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:21.720 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:21.721 DEBUG:boto:Method: GET 2024-04-19T23:35:21.721 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:35:21.721 DEBUG:boto:Data: 2024-04-19T23:35:21.721 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.721 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.721 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.721 DEBUG:boto:Params: {} 2024-04-19T23:35:21.721 DEBUG:boto:Token: None 2024-04-19T23:35:21.721 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/?versions 2024-04-19T23:35:21.722 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KheWDoPF2GOcu4dyFAHw94Cw1pI= 2024-04-19T23:35:21.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KheWDoPF2GOcu4dyFAHw94Cw1pI=', 'Content-Length': '0'} 2024-04-19T23:35:21.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000274f1896567288f9-006622ffb9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.724 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.725 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:35:21.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-04-19T23:35:21.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-04-19T23:35:21.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-04-19T23:35:21.725 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.725 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.726 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:21.726 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:21.726 DEBUG:boto:Method: GET 2024-04-19T23:35:21.726 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:35:21.726 DEBUG:boto:Data: 2024-04-19T23:35:21.726 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.726 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.726 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.726 DEBUG:boto:Params: {} 2024-04-19T23:35:21.726 DEBUG:boto:Token: None 2024-04-19T23:35:21.727 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/?versions 2024-04-19T23:35:21.727 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KheWDoPF2GOcu4dyFAHw94Cw1pI= 2024-04-19T23:35:21.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KheWDoPF2GOcu4dyFAHw94Cw1pI=', 'Content-Length': '0'} 2024-04-19T23:35:21.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faa36ceb13fd9027-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.730 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.730 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.730 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.730 DEBUG:boto:path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:21.731 DEBUG:boto:auth_path=/aktlji-21/?versions&max-keys=999 2024-04-19T23:35:21.731 DEBUG:boto:Method: GET 2024-04-19T23:35:21.731 DEBUG:boto:Path: /aktlji-21/?versions&max-keys=999 2024-04-19T23:35:21.731 DEBUG:boto:Data: 2024-04-19T23:35:21.731 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.731 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.731 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.731 DEBUG:boto:Params: {} 2024-04-19T23:35:21.731 DEBUG:boto:Token: None 2024-04-19T23:35:21.732 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/?versions 2024-04-19T23:35:21.732 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KheWDoPF2GOcu4dyFAHw94Cw1pI= 2024-04-19T23:35:21.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KheWDoPF2GOcu4dyFAHw94Cw1pI=', 'Content-Length': '0'} 2024-04-19T23:35:21.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073bf17d1be6f17ac-006622ffb9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.735 DEBUG:boto:b'aktlji-21999falsedummyvfzdrjIetS96mKEDMi-wIZKX0jVY70utrue2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone1W3hL9VJEMfT0cTRXLb26ah9fA40WjXntrue2024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone2T5myUwdzrVzAwNaenGKiuz89CXqvIBKtrue2024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-test-zone3hFrlIKscPDy4GxR6kNvVJrXhZTnMWEmtrue2024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:21.735 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:21.736 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:21.736 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.736 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.736 DEBUG:boto:Method: GET 2024-04-19T23:35:21.736 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.736 DEBUG:boto:Data: 2024-04-19T23:35:21.736 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.736 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.736 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.736 DEBUG:boto:Params: {} 2024-04-19T23:35:21.737 DEBUG:boto:Token: None 2024-04-19T23:35:21.737 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.737 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4tQyahhL4STF4QiySIQ8Trm+fuc= 2024-04-19T23:35:21.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4tQyahhL4STF4QiySIQ8Trm+fuc=', 'Content-Length': '0'} 2024-04-19T23:35:21.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006464135cbad79d74-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.739 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:21.739 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:21.740 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.740 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.740 DEBUG:boto:Method: GET 2024-04-19T23:35:21.740 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.740 DEBUG:boto:Data: 2024-04-19T23:35:21.740 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.740 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.740 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.740 DEBUG:boto:Params: {} 2024-04-19T23:35:21.741 DEBUG:boto:Token: None 2024-04-19T23:35:21.741 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.741 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4tQyahhL4STF4QiySIQ8Trm+fuc= 2024-04-19T23:35:21.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4tQyahhL4STF4QiySIQ8Trm+fuc=', 'Content-Length': '0'} 2024-04-19T23:35:21.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063bc442bab4fa59e-006622ffb9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.744 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:21.744 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:21.744 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.744 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.744 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.744 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.745 DEBUG:boto:Data: 2024-04-19T23:35:21.745 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.745 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.745 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.745 DEBUG:boto:Params: {} 2024-04-19T23:35:21.745 DEBUG:boto:Token: None 2024-04-19T23:35:21.745 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.746 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iDT9Tw0gxTVZ5HfXPBt5D0GlhPg= 2024-04-19T23:35:21.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iDT9Tw0gxTVZ5HfXPBt5D0GlhPg=', 'Content-Length': '0'} 2024-04-19T23:35:21.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000981c4a447230ecde-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.748 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:21.748 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:21.748 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.748 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.748 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.748 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.748 DEBUG:boto:Data: 2024-04-19T23:35:21.748 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.748 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.748 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.748 DEBUG:boto:Params: {} 2024-04-19T23:35:21.749 DEBUG:boto:Token: None 2024-04-19T23:35:21.749 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.749 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iDT9Tw0gxTVZ5HfXPBt5D0GlhPg= 2024-04-19T23:35:21.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iDT9Tw0gxTVZ5HfXPBt5D0GlhPg=', 'Content-Length': '0'} 2024-04-19T23:35:21.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c085efcf307194cd-006622ffb9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:21.751 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:21.751 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:21.751 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.751 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.751 DEBUG:boto:Method: GET 2024-04-19T23:35:21.752 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.752 DEBUG:boto:Data: 2024-04-19T23:35:21.752 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.752 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.752 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.752 DEBUG:boto:Params: {} 2024-04-19T23:35:21.752 DEBUG:boto:Token: None 2024-04-19T23:35:21.752 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.752 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4tQyahhL4STF4QiySIQ8Trm+fuc= 2024-04-19T23:35:21.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4tQyahhL4STF4QiySIQ8Trm+fuc=', 'Content-Length': '0'} 2024-04-19T23:35:21.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f6ddfc4ef4531e4-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.755 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:21.755 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:21.755 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.755 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.755 DEBUG:boto:Method: GET 2024-04-19T23:35:21.755 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.755 DEBUG:boto:Data: 2024-04-19T23:35:21.756 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.756 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.756 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.756 DEBUG:boto:Params: {} 2024-04-19T23:35:21.756 DEBUG:boto:Token: None 2024-04-19T23:35:21.756 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.756 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4tQyahhL4STF4QiySIQ8Trm+fuc= 2024-04-19T23:35:21.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4tQyahhL4STF4QiySIQ8Trm+fuc=', 'Content-Length': '0'} 2024-04-19T23:35:21.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8204add9fa8bcc3-006622ffb9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.759 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.759 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.759 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:21.759 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:21.759 DEBUG:boto:Method: GET 2024-04-19T23:35:21.760 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:35:21.760 DEBUG:boto:Data: 2024-04-19T23:35:21.760 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.760 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.760 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.761 DEBUG:boto:Params: {} 2024-04-19T23:35:21.761 DEBUG:boto:Token: None 2024-04-19T23:35:21.761 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.761 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067b6203b2d2534ad-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.814 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.815 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.815 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.815 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:21.815 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:21.815 DEBUG:boto:Method: GET 2024-04-19T23:35:21.815 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:35:21.815 DEBUG:boto:Data: 2024-04-19T23:35:21.815 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.816 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.816 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.816 DEBUG:boto:Params: {} 2024-04-19T23:35:21.816 DEBUG:boto:Token: None 2024-04-19T23:35:21.816 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.816 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1cafe79a4e86ec4-006622ffb9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.862 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.863 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:21.863 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:21.863 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.863 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:35:21.863 DEBUG:boto:Data: 2024-04-19T23:35:21.863 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.863 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.864 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.864 DEBUG:boto:Params: {} 2024-04-19T23:35:21.864 DEBUG:boto:Token: None 2024-04-19T23:35:21.864 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/dummy 2024-04-19T23:35:21.864 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jqkpyFJz8ijH64mj/PwH0QecmxY= 2024-04-19T23:35:21.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jqkpyFJz8ijH64mj/PwH0QecmxY=', 'Content-Length': '0'} 2024-04-19T23:35:21.869 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f77968c3a9b864a3-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.869 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:21.870 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:21.870 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.870 DEBUG:boto:Path: /aktlji-21/dummy 2024-04-19T23:35:21.870 DEBUG:boto:Data: 2024-04-19T23:35:21.870 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.870 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.870 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.870 DEBUG:boto:Params: {} 2024-04-19T23:35:21.870 DEBUG:boto:Token: None 2024-04-19T23:35:21.871 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/dummy 2024-04-19T23:35:21.871 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jqkpyFJz8ijH64mj/PwH0QecmxY= 2024-04-19T23:35:21.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jqkpyFJz8ijH64mj/PwH0QecmxY=', 'Content-Length': '0'} 2024-04-19T23:35:21.873 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000692b318a00d7a402-006622ffb9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:35:21.873 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:21.874 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:21.874 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.874 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.874 DEBUG:boto:Method: GET 2024-04-19T23:35:21.874 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.874 DEBUG:boto:Data: 2024-04-19T23:35:21.874 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.874 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.874 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.874 DEBUG:boto:Params: {} 2024-04-19T23:35:21.875 DEBUG:boto:Token: None 2024-04-19T23:35:21.875 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.875 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4tQyahhL4STF4QiySIQ8Trm+fuc= 2024-04-19T23:35:21.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4tQyahhL4STF4QiySIQ8Trm+fuc=', 'Content-Length': '0'} 2024-04-19T23:35:21.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000892656adc3722edf-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.879 DEBUG:boto:path=/aktlji-21/dummy 2024-04-19T23:35:21.879 DEBUG:boto:auth_path=/aktlji-21/dummy 2024-04-19T23:35:21.879 DEBUG:boto:path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.879 DEBUG:boto:auth_path=/aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.879 DEBUG:boto:Method: GET 2024-04-19T23:35:21.879 DEBUG:boto:Path: /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.879 DEBUG:boto:Data: 2024-04-19T23:35:21.879 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.879 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.880 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.880 DEBUG:boto:Params: {} 2024-04-19T23:35:21.880 DEBUG:boto:Token: None 2024-04-19T23:35:21.880 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/dummy?versionId=vfzdrjIetS96mKEDMi-wIZKX0jVY70u 2024-04-19T23:35:21.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4tQyahhL4STF4QiySIQ8Trm+fuc= 2024-04-19T23:35:21.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4tQyahhL4STF4QiySIQ8Trm+fuc=', 'Content-Length': '0'} 2024-04-19T23:35:21.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:20 GMT'), ('x-amz-version-id', 'vfzdrjIetS96mKEDMi-wIZKX0jVY70u'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040c6d0e0f65c8b6e-006622ffb9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.883 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.883 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.883 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:21.883 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:21.883 DEBUG:boto:Method: GET 2024-04-19T23:35:21.883 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:35:21.883 DEBUG:boto:Data: 2024-04-19T23:35:21.883 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.884 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.884 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.884 DEBUG:boto:Params: {} 2024-04-19T23:35:21.884 DEBUG:boto:Token: None 2024-04-19T23:35:21.884 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.884 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000374f7a68289d97bd-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.888 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.889 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:21.889 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:21.889 DEBUG:boto:path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:21.889 DEBUG:boto:auth_path=/aktlji-21/?prefix=dummy 2024-04-19T23:35:21.889 DEBUG:boto:Method: GET 2024-04-19T23:35:21.889 DEBUG:boto:Path: /aktlji-21/?prefix=dummy 2024-04-19T23:35:21.889 DEBUG:boto:Data: 2024-04-19T23:35:21.889 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.890 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.890 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.890 DEBUG:boto:Params: {} 2024-04-19T23:35:21.890 DEBUG:boto:Token: None 2024-04-19T23:35:21.890 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/ 2024-04-19T23:35:21.890 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo= 2024-04-19T23:35:21.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fDFzozoY28Qkffi6R20CxRH9UJo=', 'Content-Length': '0'} 2024-04-19T23:35:21.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6e9c0e11d48408f-006622ffb9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.893 DEBUG:boto:b'aktlji-21dummy1000falsedummy2024-04-19T23:32:20.023Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:21.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:21.893 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.893 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.893 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.893 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.894 DEBUG:boto:Method: GET 2024-04-19T23:35:21.894 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.894 DEBUG:boto:Data: 2024-04-19T23:35:21.894 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.894 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.894 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.894 DEBUG:boto:Params: {} 2024-04-19T23:35:21.894 DEBUG:boto:Token: None 2024-04-19T23:35:21.894 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.895 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BSB/4NtjpFhFyuu/RH9B+rpxFz4= 2024-04-19T23:35:21.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BSB/4NtjpFhFyuu/RH9B+rpxFz4=', 'Content-Length': '0'} 2024-04-19T23:35:21.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a796daf923ab7011-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.896 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.897 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.897 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.897 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.897 DEBUG:boto:Method: GET 2024-04-19T23:35:21.897 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.897 DEBUG:boto:Data: 2024-04-19T23:35:21.897 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.897 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.897 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.897 DEBUG:boto:Params: {} 2024-04-19T23:35:21.898 DEBUG:boto:Token: None 2024-04-19T23:35:21.898 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.898 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BSB/4NtjpFhFyuu/RH9B+rpxFz4= 2024-04-19T23:35:21.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BSB/4NtjpFhFyuu/RH9B+rpxFz4=', 'Content-Length': '0'} 2024-04-19T23:35:21.900 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0c7871c73d40a61-006622ffb9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.900 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.900 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.900 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.900 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.900 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.900 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.901 DEBUG:boto:Data: 2024-04-19T23:35:21.901 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.901 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.901 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.901 DEBUG:boto:Params: {} 2024-04-19T23:35:21.901 DEBUG:boto:Token: None 2024-04-19T23:35:21.901 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.901 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Id7wuNlKyWOfcELeWUxv1q20RsQ= 2024-04-19T23:35:21.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Id7wuNlKyWOfcELeWUxv1q20RsQ=', 'Content-Length': '0'} 2024-04-19T23:35:21.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5e20cf01ffa4b4f-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.903 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.904 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.904 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.904 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.904 DEBUG:boto:Method: HEAD 2024-04-19T23:35:21.904 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.904 DEBUG:boto:Data: 2024-04-19T23:35:21.904 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.905 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.905 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.905 DEBUG:boto:Params: {} 2024-04-19T23:35:21.905 DEBUG:boto:Token: None 2024-04-19T23:35:21.905 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.905 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Id7wuNlKyWOfcELeWUxv1q20RsQ= 2024-04-19T23:35:21.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Id7wuNlKyWOfcELeWUxv1q20RsQ=', 'Content-Length': '0'} 2024-04-19T23:35:21.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3cc6d2a74f702fd-006622ffb9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:21.907 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.908 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.908 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.908 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.908 DEBUG:boto:Method: GET 2024-04-19T23:35:21.908 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.908 DEBUG:boto:Data: 2024-04-19T23:35:21.908 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.908 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:21.908 DEBUG:boto:Port: 8001 2024-04-19T23:35:21.908 DEBUG:boto:Params: {} 2024-04-19T23:35:21.908 DEBUG:boto:Token: None 2024-04-19T23:35:21.909 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.909 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BSB/4NtjpFhFyuu/RH9B+rpxFz4= 2024-04-19T23:35:21.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BSB/4NtjpFhFyuu/RH9B+rpxFz4=', 'Content-Length': '0'} 2024-04-19T23:35:21.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d2c0b02385e3844-006622ffb9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:21.954 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.955 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:21.955 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.955 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.955 DEBUG:boto:Method: GET 2024-04-19T23:35:21.955 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.955 DEBUG:boto:Data: 2024-04-19T23:35:21.955 DEBUG:boto:Headers: {} 2024-04-19T23:35:21.955 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:21.955 DEBUG:boto:Port: 8000 2024-04-19T23:35:21.955 DEBUG:boto:Params: {} 2024-04-19T23:35:21.956 DEBUG:boto:Token: None 2024-04-19T23:35:21.956 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:21 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:21.956 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BSB/4NtjpFhFyuu/RH9B+rpxFz4= 2024-04-19T23:35:21.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BSB/4NtjpFhFyuu/RH9B+rpxFz4=', 'Content-Length': '0'} 2024-04-19T23:35:21.958 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d2348aeda65e54e-006622ffb9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.002 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:22.003 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:22.003 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:22.003 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:22.003 DEBUG:boto:Method: GET 2024-04-19T23:35:22.003 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:22.003 DEBUG:boto:Data: 2024-04-19T23:35:22.003 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.003 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.003 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.003 DEBUG:boto:Params: {} 2024-04-19T23:35:22.004 DEBUG:boto:Token: None 2024-04-19T23:35:22.004 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/ 2024-04-19T23:35:22.004 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8= 2024-04-19T23:35:22.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8=', 'Content-Length': '0'} 2024-04-19T23:35:22.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bbab07b0c4652a5-006622ffba-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.007 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:22.007 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:22.007 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:22.008 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:22.008 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:22.008 DEBUG:boto:Method: GET 2024-04-19T23:35:22.008 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:22.008 DEBUG:boto:Data: 2024-04-19T23:35:22.008 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.008 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.008 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.008 DEBUG:boto:Params: {} 2024-04-19T23:35:22.008 DEBUG:boto:Token: None 2024-04-19T23:35:22.009 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/ 2024-04-19T23:35:22.009 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8= 2024-04-19T23:35:22.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8=', 'Content-Length': '0'} 2024-04-19T23:35:22.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000560bc2ace4062bdd-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.011 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:22.012 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:22.012 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:22.012 DEBUG:boto:Method: HEAD 2024-04-19T23:35:22.012 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:35:22.012 DEBUG:boto:Data: 2024-04-19T23:35:22.013 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.013 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.013 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.013 DEBUG:boto:Params: {} 2024-04-19T23:35:22.013 DEBUG:boto:Token: None 2024-04-19T23:35:22.013 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:35:22.013 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7857pCfr8Zo/QLpqXYmAJAhx+5w= 2024-04-19T23:35:22.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7857pCfr8Zo/QLpqXYmAJAhx+5w=', 'Content-Length': '0'} 2024-04-19T23:35:22.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed615d5cf0ef5982-006622ffba-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.016 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:22.016 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:22.016 DEBUG:boto:Method: HEAD 2024-04-19T23:35:22.016 DEBUG:boto:Path: /aktlji-21/obj-test-zone1 2024-04-19T23:35:22.016 DEBUG:boto:Data: 2024-04-19T23:35:22.017 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.017 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.017 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.017 DEBUG:boto:Params: {} 2024-04-19T23:35:22.017 DEBUG:boto:Token: None 2024-04-19T23:35:22.017 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone1 2024-04-19T23:35:22.017 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7857pCfr8Zo/QLpqXYmAJAhx+5w= 2024-04-19T23:35:22.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7857pCfr8Zo/QLpqXYmAJAhx+5w=', 'Content-Length': '0'} 2024-04-19T23:35:22.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f1f14cb46a283ac-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-04-19T23:35:22.020 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:22.020 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:22.020 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:22.020 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:22.021 DEBUG:boto:Method: GET 2024-04-19T23:35:22.021 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:22.021 DEBUG:boto:Data: 2024-04-19T23:35:22.021 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.021 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.021 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.021 DEBUG:boto:Params: {} 2024-04-19T23:35:22.021 DEBUG:boto:Token: None 2024-04-19T23:35:22.021 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:22.021 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Kur0SRqyOetvlF8MyIcQBDPaPzM= 2024-04-19T23:35:22.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Kur0SRqyOetvlF8MyIcQBDPaPzM=', 'Content-Length': '0'} 2024-04-19T23:35:22.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c3f6d6edf5736c6-006622ffba-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.067 DEBUG:boto:path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:22.067 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1 2024-04-19T23:35:22.068 DEBUG:boto:path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:22.068 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:22.068 DEBUG:boto:Method: GET 2024-04-19T23:35:22.068 DEBUG:boto:Path: /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:22.068 DEBUG:boto:Data: 2024-04-19T23:35:22.068 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.069 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.069 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.069 DEBUG:boto:Params: {} 2024-04-19T23:35:22.069 DEBUG:boto:Token: None 2024-04-19T23:35:22.069 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone1?versionId=W3hL9VJEMfT0cTRXLb26ah9fA40WjXn 2024-04-19T23:35:22.070 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Kur0SRqyOetvlF8MyIcQBDPaPzM= 2024-04-19T23:35:22.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Kur0SRqyOetvlF8MyIcQBDPaPzM=', 'Content-Length': '0'} 2024-04-19T23:35:22.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:35 GMT'), ('x-amz-version-id', 'W3hL9VJEMfT0cTRXLb26ah9fA40WjXn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000118158d8bfdb0d55-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.114 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:22.115 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:22.115 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:22.115 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:22.115 DEBUG:boto:Method: GET 2024-04-19T23:35:22.115 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:22.115 DEBUG:boto:Data: 2024-04-19T23:35:22.115 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.115 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.116 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.116 DEBUG:boto:Params: {} 2024-04-19T23:35:22.116 DEBUG:boto:Token: None 2024-04-19T23:35:22.116 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/ 2024-04-19T23:35:22.116 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8= 2024-04-19T23:35:22.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8=', 'Content-Length': '0'} 2024-04-19T23:35:22.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000486d61e768b9144e-006622ffba-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.166 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:22.167 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:22.167 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:22.167 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:22.167 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:22.167 DEBUG:boto:Method: GET 2024-04-19T23:35:22.167 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone1 2024-04-19T23:35:22.167 DEBUG:boto:Data: 2024-04-19T23:35:22.168 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.168 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.168 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.168 DEBUG:boto:Params: {} 2024-04-19T23:35:22.168 DEBUG:boto:Token: None 2024-04-19T23:35:22.168 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/ 2024-04-19T23:35:22.168 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8= 2024-04-19T23:35:22.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8=', 'Content-Length': '0'} 2024-04-19T23:35:22.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c7cd7b7de7b4b1f-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.171 DEBUG:boto:b'aktlji-21obj-test-zone11000falseobj-test-zone12024-04-19T23:32:35.921Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:22.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:22.171 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.172 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.172 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.172 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.172 DEBUG:boto:Method: GET 2024-04-19T23:35:22.172 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.172 DEBUG:boto:Data: 2024-04-19T23:35:22.172 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.172 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.172 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.172 DEBUG:boto:Params: {} 2024-04-19T23:35:22.173 DEBUG:boto:Token: None 2024-04-19T23:35:22.173 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.173 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0PH/T3ghsI8enJ/kyWLT9fzcyNM= 2024-04-19T23:35:22.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0PH/T3ghsI8enJ/kyWLT9fzcyNM=', 'Content-Length': '0'} 2024-04-19T23:35:22.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000931f837a2765d1b7-006622ffba-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.175 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.175 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.176 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.176 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.176 DEBUG:boto:Method: GET 2024-04-19T23:35:22.176 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.176 DEBUG:boto:Data: 2024-04-19T23:35:22.176 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.176 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.176 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.176 DEBUG:boto:Params: {} 2024-04-19T23:35:22.176 DEBUG:boto:Token: None 2024-04-19T23:35:22.177 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.177 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0PH/T3ghsI8enJ/kyWLT9fzcyNM= 2024-04-19T23:35:22.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0PH/T3ghsI8enJ/kyWLT9fzcyNM=', 'Content-Length': '0'} 2024-04-19T23:35:22.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032399dd7806d9d9b-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.179 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.179 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.179 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.179 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.179 DEBUG:boto:Method: HEAD 2024-04-19T23:35:22.180 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.180 DEBUG:boto:Data: 2024-04-19T23:35:22.180 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.180 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.180 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.180 DEBUG:boto:Params: {} 2024-04-19T23:35:22.180 DEBUG:boto:Token: None 2024-04-19T23:35:22.180 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZAJpf8U5f6HzXb6GwUvb4/1GoDs= 2024-04-19T23:35:22.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZAJpf8U5f6HzXb6GwUvb4/1GoDs=', 'Content-Length': '0'} 2024-04-19T23:35:22.182 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000338a7a3dc295c816-006622ffba-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.182 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.183 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.183 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.183 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.183 DEBUG:boto:Method: HEAD 2024-04-19T23:35:22.183 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.183 DEBUG:boto:Data: 2024-04-19T23:35:22.183 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.183 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.183 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.183 DEBUG:boto:Params: {} 2024-04-19T23:35:22.183 DEBUG:boto:Token: None 2024-04-19T23:35:22.184 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.184 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZAJpf8U5f6HzXb6GwUvb4/1GoDs= 2024-04-19T23:35:22.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZAJpf8U5f6HzXb6GwUvb4/1GoDs=', 'Content-Length': '0'} 2024-04-19T23:35:22.185 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004bb8ea1a83408f6d-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:22.186 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.186 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.186 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.186 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.186 DEBUG:boto:Method: GET 2024-04-19T23:35:22.187 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.187 DEBUG:boto:Data: 2024-04-19T23:35:22.187 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.187 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.187 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.188 DEBUG:boto:Params: {} 2024-04-19T23:35:22.188 DEBUG:boto:Token: None 2024-04-19T23:35:22.188 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.188 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0PH/T3ghsI8enJ/kyWLT9fzcyNM= 2024-04-19T23:35:22.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0PH/T3ghsI8enJ/kyWLT9fzcyNM=', 'Content-Length': '0'} 2024-04-19T23:35:22.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000410d294989efd511-006622ffba-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.234 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.235 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.235 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.235 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.235 DEBUG:boto:Method: GET 2024-04-19T23:35:22.235 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.235 DEBUG:boto:Data: 2024-04-19T23:35:22.235 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.235 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.235 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.235 DEBUG:boto:Params: {} 2024-04-19T23:35:22.236 DEBUG:boto:Token: None 2024-04-19T23:35:22.236 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.236 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0PH/T3ghsI8enJ/kyWLT9fzcyNM= 2024-04-19T23:35:22.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0PH/T3ghsI8enJ/kyWLT9fzcyNM=', 'Content-Length': '0'} 2024-04-19T23:35:22.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002114edd4c5c12227-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.238 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:22.238 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:22.238 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:22.239 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:22.239 DEBUG:boto:Method: GET 2024-04-19T23:35:22.239 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:22.239 DEBUG:boto:Data: 2024-04-19T23:35:22.239 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.239 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.239 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.239 DEBUG:boto:Params: {} 2024-04-19T23:35:22.239 DEBUG:boto:Token: None 2024-04-19T23:35:22.239 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/ 2024-04-19T23:35:22.239 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8= 2024-04-19T23:35:22.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8=', 'Content-Length': '0'} 2024-04-19T23:35:22.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074d6fe744a5ceb44-006622ffba-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.242 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:22.243 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:22.243 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:22.243 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:22.243 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:22.243 DEBUG:boto:Method: GET 2024-04-19T23:35:22.243 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:22.243 DEBUG:boto:Data: 2024-04-19T23:35:22.243 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.243 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.244 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.244 DEBUG:boto:Params: {} 2024-04-19T23:35:22.244 DEBUG:boto:Token: None 2024-04-19T23:35:22.244 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/ 2024-04-19T23:35:22.244 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8= 2024-04-19T23:35:22.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8=', 'Content-Length': '0'} 2024-04-19T23:35:22.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe4b1665ee2e8275-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.247 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:22.248 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.248 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.248 DEBUG:boto:Method: HEAD 2024-04-19T23:35:22.248 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:35:22.248 DEBUG:boto:Data: 2024-04-19T23:35:22.249 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.249 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.249 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.249 DEBUG:boto:Params: {} 2024-04-19T23:35:22.249 DEBUG:boto:Token: None 2024-04-19T23:35:22.249 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:35:22.249 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a84h3CIsjB5T1loh+vu0nJ74t2A= 2024-04-19T23:35:22.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a84h3CIsjB5T1loh+vu0nJ74t2A=', 'Content-Length': '0'} 2024-04-19T23:35:22.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009573eb4204e72beb-006622ffba-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.252 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.252 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.252 DEBUG:boto:Method: HEAD 2024-04-19T23:35:22.253 DEBUG:boto:Path: /aktlji-21/obj-test-zone2 2024-04-19T23:35:22.253 DEBUG:boto:Data: 2024-04-19T23:35:22.253 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.253 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.253 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.253 DEBUG:boto:Params: {} 2024-04-19T23:35:22.253 DEBUG:boto:Token: None 2024-04-19T23:35:22.253 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone2 2024-04-19T23:35:22.253 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a84h3CIsjB5T1loh+vu0nJ74t2A= 2024-04-19T23:35:22.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a84h3CIsjB5T1loh+vu0nJ74t2A=', 'Content-Length': '0'} 2024-04-19T23:35:22.256 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed631f16579345a3-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-04-19T23:35:22.256 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.257 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.257 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.257 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.257 DEBUG:boto:Method: GET 2024-04-19T23:35:22.257 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.257 DEBUG:boto:Data: 2024-04-19T23:35:22.257 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.257 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.257 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.257 DEBUG:boto:Params: {} 2024-04-19T23:35:22.257 DEBUG:boto:Token: None 2024-04-19T23:35:22.258 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.258 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0PH/T3ghsI8enJ/kyWLT9fzcyNM= 2024-04-19T23:35:22.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0PH/T3ghsI8enJ/kyWLT9fzcyNM=', 'Content-Length': '0'} 2024-04-19T23:35:22.260 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd09ec15b8b1560c-006622ffba-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.302 DEBUG:boto:path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.302 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2 2024-04-19T23:35:22.302 DEBUG:boto:path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.303 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.303 DEBUG:boto:Method: GET 2024-04-19T23:35:22.303 DEBUG:boto:Path: /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.303 DEBUG:boto:Data: 2024-04-19T23:35:22.303 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.303 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.303 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.303 DEBUG:boto:Params: {} 2024-04-19T23:35:22.303 DEBUG:boto:Token: None 2024-04-19T23:35:22.304 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone2?versionId=T5myUwdzrVzAwNaenGKiuz89CXqvIBK 2024-04-19T23:35:22.304 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0PH/T3ghsI8enJ/kyWLT9fzcyNM= 2024-04-19T23:35:22.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0PH/T3ghsI8enJ/kyWLT9fzcyNM=', 'Content-Length': '0'} 2024-04-19T23:35:22.306 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'T5myUwdzrVzAwNaenGKiuz89CXqvIBK'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000004b403b979f45a6-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.306 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:22.306 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:22.306 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:22.306 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:22.307 DEBUG:boto:Method: GET 2024-04-19T23:35:22.307 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:22.307 DEBUG:boto:Data: 2024-04-19T23:35:22.307 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.307 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.307 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.307 DEBUG:boto:Params: {} 2024-04-19T23:35:22.307 DEBUG:boto:Token: None 2024-04-19T23:35:22.307 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/ 2024-04-19T23:35:22.308 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8= 2024-04-19T23:35:22.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8=', 'Content-Length': '0'} 2024-04-19T23:35:22.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8400e0b49b76021-006622ffba-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.310 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:22.310 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:22.310 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:22.311 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:22.311 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:22.311 DEBUG:boto:Method: GET 2024-04-19T23:35:22.311 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone2 2024-04-19T23:35:22.311 DEBUG:boto:Data: 2024-04-19T23:35:22.311 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.311 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.311 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.311 DEBUG:boto:Params: {} 2024-04-19T23:35:22.311 DEBUG:boto:Token: None 2024-04-19T23:35:22.312 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/ 2024-04-19T23:35:22.312 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8= 2024-04-19T23:35:22.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8=', 'Content-Length': '0'} 2024-04-19T23:35:22.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acf5c3b6c233b4da-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.314 DEBUG:boto:b'aktlji-21obj-test-zone21000falseobj-test-zone22024-04-19T23:32:36.031Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:22.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:22.315 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.315 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.315 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.315 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.315 DEBUG:boto:Method: GET 2024-04-19T23:35:22.315 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.315 DEBUG:boto:Data: 2024-04-19T23:35:22.316 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.316 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.316 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.316 DEBUG:boto:Params: {} 2024-04-19T23:35:22.316 DEBUG:boto:Token: None 2024-04-19T23:35:22.316 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.316 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FGSH873smsGmy7bVKw2QkEhV568= 2024-04-19T23:35:22.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FGSH873smsGmy7bVKw2QkEhV568=', 'Content-Length': '0'} 2024-04-19T23:35:22.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a62947652795388-006622ffba-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.319 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.319 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.319 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.319 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.319 DEBUG:boto:Method: GET 2024-04-19T23:35:22.319 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.319 DEBUG:boto:Data: 2024-04-19T23:35:22.319 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.319 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.319 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.320 DEBUG:boto:Params: {} 2024-04-19T23:35:22.320 DEBUG:boto:Token: None 2024-04-19T23:35:22.320 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.320 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FGSH873smsGmy7bVKw2QkEhV568= 2024-04-19T23:35:22.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FGSH873smsGmy7bVKw2QkEhV568=', 'Content-Length': '0'} 2024-04-19T23:35:22.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017dd0cb38365e083-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.322 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.322 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.322 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.322 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.323 DEBUG:boto:Method: HEAD 2024-04-19T23:35:22.323 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.323 DEBUG:boto:Data: 2024-04-19T23:35:22.323 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.323 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.323 DEBUG:boto:Params: {} 2024-04-19T23:35:22.323 DEBUG:boto:Token: None 2024-04-19T23:35:22.323 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.323 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dLRoPh002g9jtneM6/A3zwXY+IY= 2024-04-19T23:35:22.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dLRoPh002g9jtneM6/A3zwXY+IY=', 'Content-Length': '0'} 2024-04-19T23:35:22.325 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb145a347e0dbdc2-006622ffba-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.326 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.326 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.326 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.326 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.326 DEBUG:boto:Method: HEAD 2024-04-19T23:35:22.326 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.326 DEBUG:boto:Data: 2024-04-19T23:35:22.326 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.326 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.326 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.327 DEBUG:boto:Params: {} 2024-04-19T23:35:22.327 DEBUG:boto:Token: None 2024-04-19T23:35:22.327 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.327 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dLRoPh002g9jtneM6/A3zwXY+IY= 2024-04-19T23:35:22.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dLRoPh002g9jtneM6/A3zwXY+IY=', 'Content-Length': '0'} 2024-04-19T23:35:22.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c9ab5ffafdc640e-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:22.329 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.329 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.329 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.329 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.329 DEBUG:boto:Method: GET 2024-04-19T23:35:22.330 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.330 DEBUG:boto:Data: 2024-04-19T23:35:22.330 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.330 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.330 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.330 DEBUG:boto:Params: {} 2024-04-19T23:35:22.331 DEBUG:boto:Token: None 2024-04-19T23:35:22.331 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.331 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FGSH873smsGmy7bVKw2QkEhV568= 2024-04-19T23:35:22.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FGSH873smsGmy7bVKw2QkEhV568=', 'Content-Length': '0'} 2024-04-19T23:35:22.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067571d44732a358b-006622ffba-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.333 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.333 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.333 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.334 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.334 DEBUG:boto:Method: GET 2024-04-19T23:35:22.334 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.334 DEBUG:boto:Data: 2024-04-19T23:35:22.334 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.334 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.334 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.334 DEBUG:boto:Params: {} 2024-04-19T23:35:22.334 DEBUG:boto:Token: None 2024-04-19T23:35:22.334 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.335 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FGSH873smsGmy7bVKw2QkEhV568= 2024-04-19T23:35:22.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FGSH873smsGmy7bVKw2QkEhV568=', 'Content-Length': '0'} 2024-04-19T23:35:22.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6de6066479f6bf5-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.378 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:22.379 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:22.379 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:22.379 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:22.379 DEBUG:boto:Method: GET 2024-04-19T23:35:22.379 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:22.379 DEBUG:boto:Data: 2024-04-19T23:35:22.379 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.379 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.379 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.379 DEBUG:boto:Params: {} 2024-04-19T23:35:22.380 DEBUG:boto:Token: None 2024-04-19T23:35:22.380 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/ 2024-04-19T23:35:22.380 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8= 2024-04-19T23:35:22.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8=', 'Content-Length': '0'} 2024-04-19T23:35:22.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055975bb031ad30e9-006622ffba-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.383 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:22.384 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:22.384 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:22.384 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:22.384 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:22.384 DEBUG:boto:Method: GET 2024-04-19T23:35:22.384 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:22.384 DEBUG:boto:Data: 2024-04-19T23:35:22.385 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.385 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.385 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.385 DEBUG:boto:Params: {} 2024-04-19T23:35:22.385 DEBUG:boto:Token: None 2024-04-19T23:35:22.385 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/ 2024-04-19T23:35:22.385 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8= 2024-04-19T23:35:22.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8=', 'Content-Length': '0'} 2024-04-19T23:35:22.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b79cca394ed0aaa6-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.434 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:22.435 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.435 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.435 DEBUG:boto:Method: HEAD 2024-04-19T23:35:22.435 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:35:22.435 DEBUG:boto:Data: 2024-04-19T23:35:22.435 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.436 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.436 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.436 DEBUG:boto:Params: {} 2024-04-19T23:35:22.436 DEBUG:boto:Token: None 2024-04-19T23:35:22.436 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:35:22.436 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rVyuK3+VJAmWtmQm4ABAY2nveKk= 2024-04-19T23:35:22.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rVyuK3+VJAmWtmQm4ABAY2nveKk=', 'Content-Length': '0'} 2024-04-19T23:35:22.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c7c8e7ff8ef1d0e-006622ffba-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.440 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.440 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.440 DEBUG:boto:Method: HEAD 2024-04-19T23:35:22.440 DEBUG:boto:Path: /aktlji-21/obj-test-zone3 2024-04-19T23:35:22.440 DEBUG:boto:Data: 2024-04-19T23:35:22.440 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.441 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.441 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.441 DEBUG:boto:Params: {} 2024-04-19T23:35:22.441 DEBUG:boto:Token: None 2024-04-19T23:35:22.442 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone3 2024-04-19T23:35:22.442 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rVyuK3+VJAmWtmQm4ABAY2nveKk= 2024-04-19T23:35:22.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rVyuK3+VJAmWtmQm4ABAY2nveKk=', 'Content-Length': '0'} 2024-04-19T23:35:22.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbf94c91e841a634-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-04-19T23:35:22.446 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.446 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.446 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.446 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.446 DEBUG:boto:Method: GET 2024-04-19T23:35:22.446 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.446 DEBUG:boto:Data: 2024-04-19T23:35:22.446 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.446 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.447 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.447 DEBUG:boto:Params: {} 2024-04-19T23:35:22.447 DEBUG:boto:Token: None 2024-04-19T23:35:22.447 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.447 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FGSH873smsGmy7bVKw2QkEhV568= 2024-04-19T23:35:22.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FGSH873smsGmy7bVKw2QkEhV568=', 'Content-Length': '0'} 2024-04-19T23:35:22.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084a8c182b3d00ad6-006622ffba-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.490 DEBUG:boto:path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.491 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3 2024-04-19T23:35:22.491 DEBUG:boto:path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.491 DEBUG:boto:auth_path=/aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.491 DEBUG:boto:Method: GET 2024-04-19T23:35:22.491 DEBUG:boto:Path: /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.491 DEBUG:boto:Data: 2024-04-19T23:35:22.491 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.491 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.491 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.491 DEBUG:boto:Params: {} 2024-04-19T23:35:22.492 DEBUG:boto:Token: None 2024-04-19T23:35:22.492 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/obj-test-zone3?versionId=hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm 2024-04-19T23:35:22.492 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FGSH873smsGmy7bVKw2QkEhV568= 2024-04-19T23:35:22.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FGSH873smsGmy7bVKw2QkEhV568=', 'Content-Length': '0'} 2024-04-19T23:35:22.494 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:32:36 GMT'), ('x-amz-version-id', 'hFrlIKscPDy4GxR6kNvVJrXhZTnMWEm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048c20c44545fa8e1-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.534 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:22.535 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:22.535 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:22.535 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:22.535 DEBUG:boto:Method: GET 2024-04-19T23:35:22.535 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:22.535 DEBUG:boto:Data: 2024-04-19T23:35:22.535 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.535 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:35:22.536 DEBUG:boto:Port: 8001 2024-04-19T23:35:22.536 DEBUG:boto:Params: {} 2024-04-19T23:35:22.536 DEBUG:boto:Token: None 2024-04-19T23:35:22.536 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/ 2024-04-19T23:35:22.536 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8= 2024-04-19T23:35:22.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8=', 'Content-Length': '0'} 2024-04-19T23:35:22.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeb20e349cc0bfa6-006622ffba-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.540 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:22.540 DEBUG:boto:path=/aktlji-21/ 2024-04-19T23:35:22.540 DEBUG:boto:auth_path=/aktlji-21/ 2024-04-19T23:35:22.541 DEBUG:boto:path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:22.541 DEBUG:boto:auth_path=/aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:22.541 DEBUG:boto:Method: GET 2024-04-19T23:35:22.541 DEBUG:boto:Path: /aktlji-21/?prefix=obj-test-zone3 2024-04-19T23:35:22.541 DEBUG:boto:Data: 2024-04-19T23:35:22.541 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.541 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:35:22.541 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.542 DEBUG:boto:Params: {} 2024-04-19T23:35:22.542 DEBUG:boto:Token: None 2024-04-19T23:35:22.542 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:35:22 GMT /aktlji-21/ 2024-04-19T23:35:22.542 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8= 2024-04-19T23:35:22.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LrIfBq5V1lfXTqwjXAQWGSKQxs8=', 'Content-Length': '0'} 2024-04-19T23:35:22.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f74dd39c4fcf245-006622ffba-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.545 DEBUG:boto:b'aktlji-21obj-test-zone31000falseobj-test-zone32024-04-19T23:32:36.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:35:22.545 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-21 zones={test-zone3, test-zone2} 2024-04-19T23:35:22.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2024-04-19T23:35:22.549 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:35:22.549 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:35:22.549 DEBUG:boto:Method: PUT 2024-04-19T23:35:22.549 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:35:22.550 DEBUG:boto:Data: 2024-04-19T23:35:22.550 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.550 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:22.550 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.550 DEBUG:boto:Params: {} 2024-04-19T23:35:22.550 DEBUG:boto:Token: None 2024-04-19T23:35:22.551 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:35:22 GMT /aktlji-22/ 2024-04-19T23:35:22.551 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3XfBRuv/BVqPABfACrAbBXB7hJo= 2024-04-19T23:35:22.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3XfBRuv/BVqPABfACrAbBXB7hJo=', 'Content-Length': '0'} 2024-04-19T23:35:22.568 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aede6bdd5d804716-006622ffba-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.569 DEBUG:rgw_multi.tests:created bucket=aktlji-22 2024-04-19T23:35:22.569 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:35:22.569 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:35:22.569 DEBUG:boto:path=/aktlji-22/?versioning 2024-04-19T23:35:22.569 DEBUG:boto:auth_path=/aktlji-22/?versioning 2024-04-19T23:35:22.569 DEBUG:boto:Method: PUT 2024-04-19T23:35:22.570 DEBUG:boto:Path: /aktlji-22/?versioning 2024-04-19T23:35:22.570 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:35:22.570 DEBUG:boto:Headers: {} 2024-04-19T23:35:22.570 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:22.570 DEBUG:boto:Port: 8000 2024-04-19T23:35:22.570 DEBUG:boto:Params: {} 2024-04-19T23:35:22.570 DEBUG:boto:Token: None 2024-04-19T23:35:22.570 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:35:22 GMT /aktlji-22/?versioning 2024-04-19T23:35:22.571 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eWNUbTSmZcAbwg+qFBYn6ULWr3Y= 2024-04-19T23:35:22.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eWNUbTSmZcAbwg+qFBYn6ULWr3Y=', 'Content-Length': '228'} 2024-04-19T23:35:22.577 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ac10b37648cfe69-006622ffba-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:22.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:35:22.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:35:22.578 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:35:22.636 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:22.636 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:22.674 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:35:22.686 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:35:22.687 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:35:22.687 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:35:22.687 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:35:22.687 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:35:22.687 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:35:22.687 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:35:22.687 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:35:22.687 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:35:22.687 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:35:22.688 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713569722.582528_162.1", 2024-04-19T23:35:22.688 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:35:22.582528Z" 2024-04-19T23:35:22.688 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:35:22.688 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:35:22.688 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:35:22.688 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:35:22.688 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:35:22.688 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:35:22.689 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569722.582528_162.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:35:22.689 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:35:22.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:35:22.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:35:22.690 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:35:22.748 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:22.749 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:22.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:22.791+0000 7f7a56667780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:35:22.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:22.793+0000 7f7a56667780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:35:22.804 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:35:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:35:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:35:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:35:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:35:22.804 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:35:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:35:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:35:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:35:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:35:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:35:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:35:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:35:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:35:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:35:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:35:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:35:22.805 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:35:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:35:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:35:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:35:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:35:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:35:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:35:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:35:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:35:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:35:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:35:22.806 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:35:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:35:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:35:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:35:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:35:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:35:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:35:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:35:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:35:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713569722.564155_157.1", 2024-04-19T23:35:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:35:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:35:22.807 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:35:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:35:22.564155Z", 2024-04-19T23:35:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:35:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:35:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:35:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:35:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:35:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:35:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:35:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:35:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:35:22.808 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:35:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:35:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:35:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:35:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:35:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:35:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:35:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:35:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:35:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:35:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:35:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:35:22.809 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:35:22Z" 2024-04-19T23:35:22.810 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:35:22.811 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569722.564155_157.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:35:22.564155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:35:22Z" } 2024-04-19T23:35:22.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569722.564155_157.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:35:22.564155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:35:22.811 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569722.582528_162.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:35:22.811 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569722.564155_157.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:35:22.811 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713569722.582528_162.1 target=1_1713569722.564155_157.1 2024-04-19T23:35:27.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:35:27.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:35:27.813 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:35:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:27.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:27.913+0000 7fa773ba1780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:35:27.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:27.914+0000 7fa773ba1780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:35:27.924 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:35:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:35:27.924 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:35:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:35:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:35:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:35:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:35:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:35:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:35:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:35:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:35:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:35:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:35:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:35:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:35:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:35:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:35:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:35:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:35:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:35:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:35:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:35:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:35:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:35:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:35:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:35:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:35:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:35:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:35:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:35:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:35:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:35:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:35:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:35:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:35:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:35:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:35:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713569722.582528_162.1", 2024-04-19T23:35:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:35:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:35:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:35:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:35:22.582528Z", 2024-04-19T23:35:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:35:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:35:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:35:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:35:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:35:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:35:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:35:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:35:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:35:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:35:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:35:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:35:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:35:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:35:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:35:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:35:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:35:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:35:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:35:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:35:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:35:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:35:27Z" 2024-04-19T23:35:27.930 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:35:27.931 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569722.582528_162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:35:22.582528Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:35:27Z" } 2024-04-19T23:35:27.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569722.582528_162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:35:22.582528Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:35:27.931 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569722.582528_162.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:35:27.931 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569722.582528_162.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:35:27.932 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:35:27.932 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:35:27.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:35:27.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:35:27.932 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:35:27.991 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:27.992 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:28.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:28.030+0000 7f0454447780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:35:28.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:28.031+0000 7f0454447780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:35:28.040 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:35:28.040 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:35:28.041 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:35:28.041 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:35:28.041 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:35:28.041 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:35:28.041 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:35:28.041 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:35:28.041 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:35:28.041 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:35:28.041 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:35:28.041 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:35:28.041 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:35:28.042 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:35:28.042 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:35:28.042 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:35:28.042 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:35:28.042 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:35:28.042 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:35:28.042 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:35:28.042 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:35:28.042 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:35:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:35:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:35:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:35:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:35:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:35:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:35:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:35:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:35:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:35:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:35:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:35:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:35:28.044 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:35:28.044 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:35:28.044 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:35:28.044 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713569722.582528_162.1", 2024-04-19T23:35:28.044 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:35:28.044 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:35:28.044 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:35:28.044 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:35:22.582528Z", 2024-04-19T23:35:28.044 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:35:28.044 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:35:28.044 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:35:28.044 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:35:28.045 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:35:28.045 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:35:28.045 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:35:28.045 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:35:28.045 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:35:28.045 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:35:28.045 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:35:28.045 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:35:28.045 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:35:28.045 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:35:28.045 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:35:28.045 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:35:28.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:35:28.046 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:35:28.046 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:35:28.046 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:35:28.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:35:28.046 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:35:28Z" 2024-04-19T23:35:28.046 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:35:28.047 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569722.582528_162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:35:22.582528Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:35:28Z" } 2024-04-19T23:35:28.047 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569722.582528_162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:35:22.582528Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:35:28.047 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569722.582528_162.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:35:28.047 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569722.582528_162.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:35:28.048 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:35:28.048 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:35:28.048 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:35:28.048 DEBUG:boto:Method: HEAD 2024-04-19T23:35:28.048 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:35:28.048 DEBUG:boto:Data: 2024-04-19T23:35:28.048 DEBUG:boto:Headers: {} 2024-04-19T23:35:28.049 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:28.049 DEBUG:boto:Port: 8000 2024-04-19T23:35:28.049 DEBUG:boto:Params: {} 2024-04-19T23:35:28.049 DEBUG:boto:Token: None 2024-04-19T23:35:28.049 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:35:28 GMT /aktlji-22/ 2024-04-19T23:35:28.049 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LGokEFxyrCkztGJHlOsIWth7ELw= 2024-04-19T23:35:28.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:35:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LGokEFxyrCkztGJHlOsIWth7ELw=', 'Content-Length': '0'} 2024-04-19T23:35:28.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000079ea565e27f4af3-006622ffc0-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:28.052 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:35:28.052 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:35:28.052 DEBUG:boto:Method: PUT 2024-04-19T23:35:28.052 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:35:28.053 DEBUG:boto:Data: 2024-04-19T23:35:28.053 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:35:28.053 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:35:28.053 DEBUG:boto:Port: 8000 2024-04-19T23:35:28.053 DEBUG:boto:Params: {} 2024-04-19T23:35:28.053 DEBUG:boto:Token: None 2024-04-19T23:35:28.053 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:35:28 GMT /aktlji-22/dummy 2024-04-19T23:35:28.053 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jpScaKeGFwBIIKE/WXmmXDis3LM= 2024-04-19T23:35:28.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:35:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jpScaKeGFwBIIKE/WXmmXDis3LM='} 2024-04-19T23:35:28.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-amz-request-id', 'tx0000080556b8bcebbc800-006622ffc0-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:35:28.066 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-22 2024-04-19T23:35:28.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:28.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:28.067 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:28.133 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:28.133 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:28.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:28.180+0000 7ff9c8df2780 1 waiting to reach incremental sync.. 2024-04-19T23:35:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:33.180+0000 7ff9c8df2780 1 waiting to reach incremental sync.. 2024-04-19T23:35:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:38.181+0000 7ff9c8df2780 1 waiting to reach incremental sync.. 2024-04-19T23:35:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:43.182+0000 7ff9c8df2780 1 waiting to reach incremental sync.. 2024-04-19T23:35:48.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:48.183+0000 7ff9c8df2780 1 bucket sync caught up with source: 2024-04-19T23:35:48.179 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , 00000000002.30.11, , , , , ] 2024-04-19T23:35:48.180 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , 00000000002.30.11, , , , , ] 2024-04-19T23:35:48.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:35:48.183+0000 7ff9c8df2780 0 bucket checkpoint complete 2024-04-19T23:35:48.198 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-22 2024-04-19T23:35:48.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:48.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:48.199 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:48.265 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:48.266 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:48.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:48.315+0000 7f156c9ce780 1 bucket sync caught up with source: 2024-04-19T23:35:48.309 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000002.30.11, , , , , ] 2024-04-19T23:35:48.309 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000002.30.11, , , , , ] 2024-04-19T23:35:48.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:48.315+0000 7f156c9ce780 0 bucket checkpoint complete 2024-04-19T23:35:48.323 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-22 2024-04-19T23:35:48.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:48.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:48.324 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:48.388 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:48.388 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:48.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:48.435+0000 7f1d77534780 1 waiting to reach incremental sync.. 2024-04-19T23:35:53.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:53.435+0000 7f1d77534780 1 waiting to reach incremental sync.. 2024-04-19T23:35:58.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:58.437+0000 7f1d77534780 1 bucket sync caught up with source: 2024-04-19T23:35:58.432 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000002.42.11, , , , , ] 2024-04-19T23:35:58.432 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000002.42.11, , , , , ] 2024-04-19T23:35:58.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:58.437+0000 7f1d77534780 0 bucket checkpoint complete 2024-04-19T23:35:58.449 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-22 2024-04-19T23:35:58.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:58.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:35:58.450 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:35:58.516 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:35:58.516 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:35:58.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:35:58.563+0000 7f76f008d780 1 waiting to reach incremental sync.. 2024-04-19T23:36:03.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:36:03.565+0000 7f76f008d780 1 bucket sync caught up with source: 2024-04-19T23:36:03.560 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000003.44.10, , , , , ] 2024-04-19T23:36:03.560 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000003.44.10, , , , , ] 2024-04-19T23:36:03.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:36:03.565+0000 7f76f008d780 0 bucket checkpoint complete 2024-04-19T23:36:03.576 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-22 2024-04-19T23:36:03.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:36:03.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:36:03.577 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:36:03.643 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:36:03.644 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:36:03.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:36:03.690+0000 7fe1d82b2780 1 waiting for incremental sync to catch up: 2024-04-19T23:36:03.685 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000002.56.11, , , , , ] 2024-04-19T23:36:03.685 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.60.10, , , , , ] 2024-04-19T23:36:08.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:36:08.692+0000 7fe1d82b2780 1 waiting for incremental sync to catch up: 2024-04-19T23:36:08.687 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000002.56.11, , , , , ] 2024-04-19T23:36:08.687 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.60.10, , , , , ] 2024-04-19T23:36:13.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:36:13.693+0000 7fe1d82b2780 1 waiting for incremental sync to catch up: 2024-04-19T23:36:13.688 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000002.56.11, , , , , ] 2024-04-19T23:36:13.688 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.60.10, , , , , ] 2024-04-19T23:36:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:36:18.693+0000 7fe1d82b2780 1 bucket sync caught up with source: 2024-04-19T23:36:18.689 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.60.10, , , , , ] 2024-04-19T23:36:18.689 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.60.10, , , , , ] 2024-04-19T23:36:18.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:36:18.693+0000 7fe1d82b2780 0 bucket checkpoint complete 2024-04-19T23:36:18.707 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-22 2024-04-19T23:36:18.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:36:18.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:36:18.707 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:36:18.775 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:36:18.775 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:36:18.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:36:18.824+0000 7f5e21b92780 1 bucket sync caught up with source: 2024-04-19T23:36:18.819 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , 00000000004.60.10, , , , , ] 2024-04-19T23:36:18.819 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , 00000000004.60.10, , , , , ] 2024-04-19T23:36:18.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:36:18.824+0000 7f5e21b92780 0 bucket checkpoint complete 2024-04-19T23:36:18.834 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-22 zones={test-zone2, test-zone1} 2024-04-19T23:36:18.834 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:18.837 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:18.838 DEBUG:boto:Method: HEAD 2024-04-19T23:36:18.838 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:18.838 DEBUG:boto:Data: 2024-04-19T23:36:18.838 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.838 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:18.838 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.838 DEBUG:boto:Params: {} 2024-04-19T23:36:18.839 DEBUG:boto:Token: None 2024-04-19T23:36:18.839 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/ 2024-04-19T23:36:18.839 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bdS3snr0PkupzP2XT7aOxoBODO4= 2024-04-19T23:36:18.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bdS3snr0PkupzP2XT7aOxoBODO4=', 'Content-Length': '0'} 2024-04-19T23:36:18.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f522fdcb44004ce8-006622fff2-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.842 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:18.843 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:18.843 DEBUG:boto:Method: HEAD 2024-04-19T23:36:18.843 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:18.843 DEBUG:boto:Data: 2024-04-19T23:36:18.843 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.843 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:18.843 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.843 DEBUG:boto:Params: {} 2024-04-19T23:36:18.843 DEBUG:boto:Token: None 2024-04-19T23:36:18.843 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/ 2024-04-19T23:36:18.844 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bdS3snr0PkupzP2XT7aOxoBODO4= 2024-04-19T23:36:18.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bdS3snr0PkupzP2XT7aOxoBODO4=', 'Content-Length': '0'} 2024-04-19T23:36:18.847 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa60d1034af3afd1-006622fff2-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.847 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:36:18.848 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:18.848 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:18.848 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:18.848 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:18.848 DEBUG:boto:Method: GET 2024-04-19T23:36:18.848 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:36:18.848 DEBUG:boto:Data: 2024-04-19T23:36:18.848 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.848 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:18.848 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.848 DEBUG:boto:Params: {} 2024-04-19T23:36:18.849 DEBUG:boto:Token: None 2024-04-19T23:36:18.849 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/?versions 2024-04-19T23:36:18.849 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gQPetUwAgrNl1Ncv95I02ar32vk= 2024-04-19T23:36:18.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gQPetUwAgrNl1Ncv95I02ar32vk=', 'Content-Length': '0'} 2024-04-19T23:36:18.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f0775daa9783851-006622fff2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.851 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:18.852 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:36:18.852 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:36:18.853 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:18.853 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:18.853 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:18.853 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:18.853 DEBUG:boto:Method: GET 2024-04-19T23:36:18.853 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:36:18.853 DEBUG:boto:Data: 2024-04-19T23:36:18.853 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.853 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:18.853 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.853 DEBUG:boto:Params: {} 2024-04-19T23:36:18.854 DEBUG:boto:Token: None 2024-04-19T23:36:18.854 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/?versions 2024-04-19T23:36:18.854 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gQPetUwAgrNl1Ncv95I02ar32vk= 2024-04-19T23:36:18.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gQPetUwAgrNl1Ncv95I02ar32vk=', 'Content-Length': '0'} 2024-04-19T23:36:18.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1e8669d39b3ba3d-006622fff2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.858 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:18.858 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:36:18.858 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:18.858 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:18.858 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:18.859 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:18.859 DEBUG:boto:Method: GET 2024-04-19T23:36:18.859 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:36:18.859 DEBUG:boto:Data: 2024-04-19T23:36:18.859 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.859 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:18.859 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.859 DEBUG:boto:Params: {} 2024-04-19T23:36:18.859 DEBUG:boto:Token: None 2024-04-19T23:36:18.859 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/?versions 2024-04-19T23:36:18.860 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gQPetUwAgrNl1Ncv95I02ar32vk= 2024-04-19T23:36:18.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gQPetUwAgrNl1Ncv95I02ar32vk=', 'Content-Length': '0'} 2024-04-19T23:36:18.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bb013ee9b1a74bb-006622fff2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.902 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:18.903 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:18.903 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:18.903 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:18.903 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:18.903 DEBUG:boto:Method: GET 2024-04-19T23:36:18.903 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:36:18.903 DEBUG:boto:Data: 2024-04-19T23:36:18.904 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.904 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:18.904 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.904 DEBUG:boto:Params: {} 2024-04-19T23:36:18.904 DEBUG:boto:Token: None 2024-04-19T23:36:18.904 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/?versions 2024-04-19T23:36:18.904 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gQPetUwAgrNl1Ncv95I02ar32vk= 2024-04-19T23:36:18.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gQPetUwAgrNl1Ncv95I02ar32vk=', 'Content-Length': '0'} 2024-04-19T23:36:18.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001e02db7c232611d-006622fff2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.907 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:18.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:36:18.908 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:18.908 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:18.908 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.908 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.908 DEBUG:boto:Method: GET 2024-04-19T23:36:18.908 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.908 DEBUG:boto:Data: 2024-04-19T23:36:18.908 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.908 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:18.908 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.908 DEBUG:boto:Params: {} 2024-04-19T23:36:18.909 DEBUG:boto:Token: None 2024-04-19T23:36:18.909 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.909 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T/mjfhi2oiFGj6bwIg6Qkkb0I0o= 2024-04-19T23:36:18.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T/mjfhi2oiFGj6bwIg6Qkkb0I0o=', 'Content-Length': '0'} 2024-04-19T23:36:18.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fefc5044520618fc-006622fff2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.912 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:18.912 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:18.912 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.912 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.912 DEBUG:boto:Method: GET 2024-04-19T23:36:18.912 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.913 DEBUG:boto:Data: 2024-04-19T23:36:18.913 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.913 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:18.913 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.913 DEBUG:boto:Params: {} 2024-04-19T23:36:18.913 DEBUG:boto:Token: None 2024-04-19T23:36:18.913 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.913 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T/mjfhi2oiFGj6bwIg6Qkkb0I0o= 2024-04-19T23:36:18.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T/mjfhi2oiFGj6bwIg6Qkkb0I0o=', 'Content-Length': '0'} 2024-04-19T23:36:18.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036767b1fdf41b269-006622fff2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.916 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:18.916 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:18.916 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.916 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.916 DEBUG:boto:Method: HEAD 2024-04-19T23:36:18.916 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.916 DEBUG:boto:Data: 2024-04-19T23:36:18.916 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.916 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:18.917 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.917 DEBUG:boto:Params: {} 2024-04-19T23:36:18.917 DEBUG:boto:Token: None 2024-04-19T23:36:18.917 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.917 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MmgfAjDaw28hkNSq0BJZycJfGlk= 2024-04-19T23:36:18.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MmgfAjDaw28hkNSq0BJZycJfGlk=', 'Content-Length': '0'} 2024-04-19T23:36:18.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f22e5c42baa0beeb-006622fff2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.920 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:18.920 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:18.920 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.920 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.920 DEBUG:boto:Method: HEAD 2024-04-19T23:36:18.920 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.920 DEBUG:boto:Data: 2024-04-19T23:36:18.920 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.920 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:18.920 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.920 DEBUG:boto:Params: {} 2024-04-19T23:36:18.921 DEBUG:boto:Token: None 2024-04-19T23:36:18.921 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.921 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MmgfAjDaw28hkNSq0BJZycJfGlk= 2024-04-19T23:36:18.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MmgfAjDaw28hkNSq0BJZycJfGlk=', 'Content-Length': '0'} 2024-04-19T23:36:18.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2cb608ca13588e1-006622fff2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:36:18.923 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:18.923 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:18.923 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.923 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.923 DEBUG:boto:Method: GET 2024-04-19T23:36:18.923 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.923 DEBUG:boto:Data: 2024-04-19T23:36:18.924 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.924 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:18.924 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.924 DEBUG:boto:Params: {} 2024-04-19T23:36:18.924 DEBUG:boto:Token: None 2024-04-19T23:36:18.924 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.924 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T/mjfhi2oiFGj6bwIg6Qkkb0I0o= 2024-04-19T23:36:18.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T/mjfhi2oiFGj6bwIg6Qkkb0I0o=', 'Content-Length': '0'} 2024-04-19T23:36:18.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d358d0850a63beb8-006622fff2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.926 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:18.927 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:18.927 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.927 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.927 DEBUG:boto:Method: GET 2024-04-19T23:36:18.927 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.927 DEBUG:boto:Data: 2024-04-19T23:36:18.927 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.927 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:18.927 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.927 DEBUG:boto:Params: {} 2024-04-19T23:36:18.927 DEBUG:boto:Token: None 2024-04-19T23:36:18.928 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.928 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T/mjfhi2oiFGj6bwIg6Qkkb0I0o= 2024-04-19T23:36:18.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T/mjfhi2oiFGj6bwIg6Qkkb0I0o=', 'Content-Length': '0'} 2024-04-19T23:36:18.929 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bd87c198c15210d3-006622fff2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.930 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:18.930 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:18.930 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:18.930 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:18.930 DEBUG:boto:Method: GET 2024-04-19T23:36:18.930 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:36:18.930 DEBUG:boto:Data: 2024-04-19T23:36:18.930 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.931 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:18.931 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.931 DEBUG:boto:Params: {} 2024-04-19T23:36:18.931 DEBUG:boto:Token: None 2024-04-19T23:36:18.931 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/ 2024-04-19T23:36:18.931 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2VnKKr6F2HnF8n0PpstJZ9Kiwo= 2024-04-19T23:36:18.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2VnKKr6F2HnF8n0PpstJZ9Kiwo=', 'Content-Length': '0'} 2024-04-19T23:36:18.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f855f19a437db706-006622fff2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.934 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:18.934 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:18.935 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:18.935 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:18.935 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:18.935 DEBUG:boto:Method: GET 2024-04-19T23:36:18.935 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:36:18.935 DEBUG:boto:Data: 2024-04-19T23:36:18.935 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.935 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:18.935 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.935 DEBUG:boto:Params: {} 2024-04-19T23:36:18.935 DEBUG:boto:Token: None 2024-04-19T23:36:18.936 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/ 2024-04-19T23:36:18.936 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2VnKKr6F2HnF8n0PpstJZ9Kiwo= 2024-04-19T23:36:18.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2VnKKr6F2HnF8n0PpstJZ9Kiwo=', 'Content-Length': '0'} 2024-04-19T23:36:18.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007301912b0bbeb5a3-006622fff2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.982 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:18.983 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:18.983 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:18.983 DEBUG:boto:Method: HEAD 2024-04-19T23:36:18.983 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:36:18.983 DEBUG:boto:Data: 2024-04-19T23:36:18.983 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.983 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:18.983 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.984 DEBUG:boto:Params: {} 2024-04-19T23:36:18.984 DEBUG:boto:Token: None 2024-04-19T23:36:18.984 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/dummy 2024-04-19T23:36:18.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:edc5dz+o81wQ4nBpXB6pxGvRbp8= 2024-04-19T23:36:18.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:edc5dz+o81wQ4nBpXB6pxGvRbp8=', 'Content-Length': '0'} 2024-04-19T23:36:18.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f5a4dc4fde686d3a-006622fff2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.987 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:18.987 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:18.987 DEBUG:boto:Method: HEAD 2024-04-19T23:36:18.987 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:36:18.987 DEBUG:boto:Data: 2024-04-19T23:36:18.987 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.987 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:18.987 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.987 DEBUG:boto:Params: {} 2024-04-19T23:36:18.988 DEBUG:boto:Token: None 2024-04-19T23:36:18.988 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/dummy 2024-04-19T23:36:18.988 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:edc5dz+o81wQ4nBpXB6pxGvRbp8= 2024-04-19T23:36:18.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:edc5dz+o81wQ4nBpXB6pxGvRbp8=', 'Content-Length': '0'} 2024-04-19T23:36:18.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e90d71a33f180a2-006622fff2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:36:18.990 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:18.991 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:18.991 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.991 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.991 DEBUG:boto:Method: GET 2024-04-19T23:36:18.991 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.991 DEBUG:boto:Data: 2024-04-19T23:36:18.991 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.991 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:18.991 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.991 DEBUG:boto:Params: {} 2024-04-19T23:36:18.992 DEBUG:boto:Token: None 2024-04-19T23:36:18.992 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.992 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T/mjfhi2oiFGj6bwIg6Qkkb0I0o= 2024-04-19T23:36:18.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T/mjfhi2oiFGj6bwIg6Qkkb0I0o=', 'Content-Length': '0'} 2024-04-19T23:36:18.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c1d63f1d2a3d6cb2-006622fff2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.994 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:18.994 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:18.994 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.994 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.994 DEBUG:boto:Method: GET 2024-04-19T23:36:18.994 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.995 DEBUG:boto:Data: 2024-04-19T23:36:18.995 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.995 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:18.995 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.995 DEBUG:boto:Params: {} 2024-04-19T23:36:18.995 DEBUG:boto:Token: None 2024-04-19T23:36:18.995 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:18.996 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T/mjfhi2oiFGj6bwIg6Qkkb0I0o= 2024-04-19T23:36:18.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T/mjfhi2oiFGj6bwIg6Qkkb0I0o=', 'Content-Length': '0'} 2024-04-19T23:36:18.997 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000836f2aace0faedb8-006622fff3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:18.998 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:18.998 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:18.998 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:18.998 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:18.998 DEBUG:boto:Method: GET 2024-04-19T23:36:18.998 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:36:18.998 DEBUG:boto:Data: 2024-04-19T23:36:18.998 DEBUG:boto:Headers: {} 2024-04-19T23:36:18.998 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:18.999 DEBUG:boto:Port: 8000 2024-04-19T23:36:18.999 DEBUG:boto:Params: {} 2024-04-19T23:36:18.999 DEBUG:boto:Token: None 2024-04-19T23:36:18.999 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:18 GMT /aktlji-22/ 2024-04-19T23:36:18.999 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2VnKKr6F2HnF8n0PpstJZ9Kiwo= 2024-04-19T23:36:18.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2VnKKr6F2HnF8n0PpstJZ9Kiwo=', 'Content-Length': '0'} 2024-04-19T23:36:19.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020de548545852274-006622fff3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.042 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.043 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.043 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.043 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.043 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.044 DEBUG:boto:Method: GET 2024-04-19T23:36:19.044 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:36:19.044 DEBUG:boto:Data: 2024-04-19T23:36:19.044 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.044 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.044 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.045 DEBUG:boto:Params: {} 2024-04-19T23:36:19.045 DEBUG:boto:Token: None 2024-04-19T23:36:19.045 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.045 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4= 2024-04-19T23:36:19.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4=', 'Content-Length': '0'} 2024-04-19T23:36:19.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f2604f575877110-006622fff3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.049 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.049 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-22 zones={test-zone2, test-zone1} 2024-04-19T23:36:19.050 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-22 zones={test-zone3, test-zone1} 2024-04-19T23:36:19.050 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.050 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.051 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.051 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.051 DEBUG:boto:Data: 2024-04-19T23:36:19.051 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.052 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.052 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.052 DEBUG:boto:Params: {} 2024-04-19T23:36:19.052 DEBUG:boto:Token: None 2024-04-19T23:36:19.053 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.053 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.057 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5457f65d64b8125-006622fff3-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.057 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.057 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.057 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.058 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.058 DEBUG:boto:Data: 2024-04-19T23:36:19.058 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.058 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.058 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.058 DEBUG:boto:Params: {} 2024-04-19T23:36:19.059 DEBUG:boto:Token: None 2024-04-19T23:36:19.059 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.059 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2615b8ac2000fe0-006622fff3-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.062 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:36:19.062 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.063 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.063 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.063 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.063 DEBUG:boto:Method: GET 2024-04-19T23:36:19.063 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.064 DEBUG:boto:Data: 2024-04-19T23:36:19.064 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.064 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.064 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.064 DEBUG:boto:Params: {} 2024-04-19T23:36:19.064 DEBUG:boto:Token: None 2024-04-19T23:36:19.065 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/?versions 2024-04-19T23:36:19.065 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ= 2024-04-19T23:36:19.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ=', 'Content-Length': '0'} 2024-04-19T23:36:19.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a60da981fdb33d18-006622fff3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.068 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.069 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:36:19.069 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:36:19.069 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.069 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.070 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.070 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.070 DEBUG:boto:Method: GET 2024-04-19T23:36:19.070 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.070 DEBUG:boto:Data: 2024-04-19T23:36:19.070 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.070 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.070 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.070 DEBUG:boto:Params: {} 2024-04-19T23:36:19.070 DEBUG:boto:Token: None 2024-04-19T23:36:19.071 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/?versions 2024-04-19T23:36:19.071 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ= 2024-04-19T23:36:19.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ=', 'Content-Length': '0'} 2024-04-19T23:36:19.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002493b3a1279df85a-006622fff3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.074 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.074 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:36:19.074 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.074 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.074 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.075 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.075 DEBUG:boto:Method: GET 2024-04-19T23:36:19.075 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.075 DEBUG:boto:Data: 2024-04-19T23:36:19.075 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.075 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.075 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.075 DEBUG:boto:Params: {} 2024-04-19T23:36:19.075 DEBUG:boto:Token: None 2024-04-19T23:36:19.075 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/?versions 2024-04-19T23:36:19.076 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ= 2024-04-19T23:36:19.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ=', 'Content-Length': '0'} 2024-04-19T23:36:19.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000456dcd56c1d1ec50-006622fff3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.078 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.079 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.079 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.079 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.079 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.079 DEBUG:boto:Method: GET 2024-04-19T23:36:19.079 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.080 DEBUG:boto:Data: 2024-04-19T23:36:19.080 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.080 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.080 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.080 DEBUG:boto:Params: {} 2024-04-19T23:36:19.080 DEBUG:boto:Token: None 2024-04-19T23:36:19.080 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/?versions 2024-04-19T23:36:19.080 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ= 2024-04-19T23:36:19.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ=', 'Content-Length': '0'} 2024-04-19T23:36:19.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a8beffd484274a8-006622fff3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.083 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:36:19.083 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.084 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.084 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.084 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.084 DEBUG:boto:Method: GET 2024-04-19T23:36:19.084 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.084 DEBUG:boto:Data: 2024-04-19T23:36:19.084 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.084 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.084 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.084 DEBUG:boto:Params: {} 2024-04-19T23:36:19.084 DEBUG:boto:Token: None 2024-04-19T23:36:19.085 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.085 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4= 2024-04-19T23:36:19.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4=', 'Content-Length': '0'} 2024-04-19T23:36:19.088 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003005b4604cf439f3-006622fff3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.088 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.088 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.088 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.088 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.089 DEBUG:boto:Method: GET 2024-04-19T23:36:19.089 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.089 DEBUG:boto:Data: 2024-04-19T23:36:19.089 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.089 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.089 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.089 DEBUG:boto:Params: {} 2024-04-19T23:36:19.089 DEBUG:boto:Token: None 2024-04-19T23:36:19.089 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.090 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4= 2024-04-19T23:36:19.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4=', 'Content-Length': '0'} 2024-04-19T23:36:19.092 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9cfef1fdd7bef28-006622fff3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.092 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.092 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.092 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.092 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.092 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.092 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.092 DEBUG:boto:Data: 2024-04-19T23:36:19.092 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.093 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.093 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.093 DEBUG:boto:Params: {} 2024-04-19T23:36:19.093 DEBUG:boto:Token: None 2024-04-19T23:36:19.093 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.093 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yer+Y+Q580OkzDPYoxV5Y6CAhn0= 2024-04-19T23:36:19.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yer+Y+Q580OkzDPYoxV5Y6CAhn0=', 'Content-Length': '0'} 2024-04-19T23:36:19.095 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008b52448b5ede109-006622fff3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.095 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.095 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.095 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.096 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.096 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.096 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.096 DEBUG:boto:Data: 2024-04-19T23:36:19.096 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.096 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.096 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.096 DEBUG:boto:Params: {} 2024-04-19T23:36:19.096 DEBUG:boto:Token: None 2024-04-19T23:36:19.096 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.097 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yer+Y+Q580OkzDPYoxV5Y6CAhn0= 2024-04-19T23:36:19.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yer+Y+Q580OkzDPYoxV5Y6CAhn0=', 'Content-Length': '0'} 2024-04-19T23:36:19.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002769fbe52089aaeb-006622fff3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:36:19.099 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.099 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.099 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.099 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.099 DEBUG:boto:Method: GET 2024-04-19T23:36:19.099 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.099 DEBUG:boto:Data: 2024-04-19T23:36:19.100 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.100 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.100 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.100 DEBUG:boto:Params: {} 2024-04-19T23:36:19.100 DEBUG:boto:Token: None 2024-04-19T23:36:19.100 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.100 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4= 2024-04-19T23:36:19.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4=', 'Content-Length': '0'} 2024-04-19T23:36:19.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc1e8c1b82896099-006622fff3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.102 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.102 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.103 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.103 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.103 DEBUG:boto:Method: GET 2024-04-19T23:36:19.103 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.103 DEBUG:boto:Data: 2024-04-19T23:36:19.103 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.103 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.103 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.103 DEBUG:boto:Params: {} 2024-04-19T23:36:19.103 DEBUG:boto:Token: None 2024-04-19T23:36:19.103 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4= 2024-04-19T23:36:19.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4=', 'Content-Length': '0'} 2024-04-19T23:36:19.105 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000631037d56f43d06c-006622fff3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.106 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.106 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.106 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.106 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.106 DEBUG:boto:Method: GET 2024-04-19T23:36:19.106 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:36:19.106 DEBUG:boto:Data: 2024-04-19T23:36:19.106 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.106 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.107 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.107 DEBUG:boto:Params: {} 2024-04-19T23:36:19.107 DEBUG:boto:Token: None 2024-04-19T23:36:19.107 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.107 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4= 2024-04-19T23:36:19.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4=', 'Content-Length': '0'} 2024-04-19T23:36:19.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003830af5927d10b7-006622fff3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.150 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.151 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.151 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.151 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.151 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.151 DEBUG:boto:Method: GET 2024-04-19T23:36:19.151 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:36:19.151 DEBUG:boto:Data: 2024-04-19T23:36:19.151 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.152 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.152 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.152 DEBUG:boto:Params: {} 2024-04-19T23:36:19.152 DEBUG:boto:Token: None 2024-04-19T23:36:19.152 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.152 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4= 2024-04-19T23:36:19.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4=', 'Content-Length': '0'} 2024-04-19T23:36:19.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098f34d1b36cbc4e1-006622fff3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.155 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.155 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.155 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.155 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.156 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:36:19.156 DEBUG:boto:Data: 2024-04-19T23:36:19.156 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.156 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.156 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.156 DEBUG:boto:Params: {} 2024-04-19T23:36:19.156 DEBUG:boto:Token: None 2024-04-19T23:36:19.156 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy 2024-04-19T23:36:19.156 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qIfC5Vek35a7ODecfjobGyv+QUs= 2024-04-19T23:36:19.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qIfC5Vek35a7ODecfjobGyv+QUs=', 'Content-Length': '0'} 2024-04-19T23:36:19.159 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000095b79f29bd88e541-006622fff3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.159 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.159 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.159 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.159 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:36:19.159 DEBUG:boto:Data: 2024-04-19T23:36:19.160 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.160 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.160 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.160 DEBUG:boto:Params: {} 2024-04-19T23:36:19.160 DEBUG:boto:Token: None 2024-04-19T23:36:19.160 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy 2024-04-19T23:36:19.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qIfC5Vek35a7ODecfjobGyv+QUs= 2024-04-19T23:36:19.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qIfC5Vek35a7ODecfjobGyv+QUs=', 'Content-Length': '0'} 2024-04-19T23:36:19.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000083da95faf7f11e58-006622fff3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:36:19.163 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.163 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.163 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.163 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.163 DEBUG:boto:Method: GET 2024-04-19T23:36:19.163 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.163 DEBUG:boto:Data: 2024-04-19T23:36:19.164 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.164 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.164 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.164 DEBUG:boto:Params: {} 2024-04-19T23:36:19.164 DEBUG:boto:Token: None 2024-04-19T23:36:19.164 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.164 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4= 2024-04-19T23:36:19.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4=', 'Content-Length': '0'} 2024-04-19T23:36:19.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ccc3d9a2e8feb4a4-006622fff3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.167 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.167 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.167 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.167 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.167 DEBUG:boto:Method: GET 2024-04-19T23:36:19.167 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.167 DEBUG:boto:Data: 2024-04-19T23:36:19.167 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.167 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.167 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.167 DEBUG:boto:Params: {} 2024-04-19T23:36:19.168 DEBUG:boto:Token: None 2024-04-19T23:36:19.168 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.168 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4= 2024-04-19T23:36:19.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4=', 'Content-Length': '0'} 2024-04-19T23:36:19.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000336d416c461026e9-006622fff3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.170 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.170 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.170 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.170 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.170 DEBUG:boto:Method: GET 2024-04-19T23:36:19.171 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:36:19.171 DEBUG:boto:Data: 2024-04-19T23:36:19.171 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.171 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.171 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.171 DEBUG:boto:Params: {} 2024-04-19T23:36:19.171 DEBUG:boto:Token: None 2024-04-19T23:36:19.171 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.171 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4= 2024-04-19T23:36:19.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4=', 'Content-Length': '0'} 2024-04-19T23:36:19.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018981589eefc32c4-006622fff3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.214 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.215 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.215 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.215 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.215 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.216 DEBUG:boto:Method: GET 2024-04-19T23:36:19.216 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:36:19.216 DEBUG:boto:Data: 2024-04-19T23:36:19.216 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.216 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.216 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.216 DEBUG:boto:Params: {} 2024-04-19T23:36:19.216 DEBUG:boto:Token: None 2024-04-19T23:36:19.216 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.217 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4= 2024-04-19T23:36:19.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4=', 'Content-Length': '0'} 2024-04-19T23:36:19.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025939134dff68083-006622fff3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.219 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.219 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-22 zones={test-zone3, test-zone1} 2024-04-19T23:36:19.220 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-22 zones={test-zone3, test-zone2} 2024-04-19T23:36:19.220 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.220 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.220 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.220 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.220 DEBUG:boto:Data: 2024-04-19T23:36:19.220 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.220 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.220 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.220 DEBUG:boto:Params: {} 2024-04-19T23:36:19.221 DEBUG:boto:Token: None 2024-04-19T23:36:19.221 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.221 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.223 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f41ea8c641465e2-006622fff3-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.223 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.223 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.224 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.224 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.224 DEBUG:boto:Data: 2024-04-19T23:36:19.224 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.224 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.224 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.224 DEBUG:boto:Params: {} 2024-04-19T23:36:19.224 DEBUG:boto:Token: None 2024-04-19T23:36:19.224 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.224 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca8c51175a11d78b-006622fff3-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.227 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:36:19.227 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.227 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.227 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.228 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.228 DEBUG:boto:Method: GET 2024-04-19T23:36:19.228 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.228 DEBUG:boto:Data: 2024-04-19T23:36:19.228 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.228 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.228 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.228 DEBUG:boto:Params: {} 2024-04-19T23:36:19.228 DEBUG:boto:Token: None 2024-04-19T23:36:19.229 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/?versions 2024-04-19T23:36:19.229 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ= 2024-04-19T23:36:19.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ=', 'Content-Length': '0'} 2024-04-19T23:36:19.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c0a4f06016ca0cc-006622fff3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.232 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.232 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:36:19.232 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:36:19.233 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.233 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.233 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.233 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.233 DEBUG:boto:Method: GET 2024-04-19T23:36:19.233 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.233 DEBUG:boto:Data: 2024-04-19T23:36:19.233 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.233 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.234 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.234 DEBUG:boto:Params: {} 2024-04-19T23:36:19.234 DEBUG:boto:Token: None 2024-04-19T23:36:19.235 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/?versions 2024-04-19T23:36:19.235 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ= 2024-04-19T23:36:19.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ=', 'Content-Length': '0'} 2024-04-19T23:36:19.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dfa5d6e082f5050-006622fff3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.238 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.239 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:36:19.239 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.239 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.239 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.239 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.241 DEBUG:boto:Method: GET 2024-04-19T23:36:19.241 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.241 DEBUG:boto:Data: 2024-04-19T23:36:19.241 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.241 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.241 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.242 DEBUG:boto:Params: {} 2024-04-19T23:36:19.242 DEBUG:boto:Token: None 2024-04-19T23:36:19.242 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/?versions 2024-04-19T23:36:19.242 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ= 2024-04-19T23:36:19.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ=', 'Content-Length': '0'} 2024-04-19T23:36:19.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4b3a549c985bdff-006622fff3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.286 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.288 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.289 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.289 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.289 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.289 DEBUG:boto:Method: GET 2024-04-19T23:36:19.290 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:36:19.290 DEBUG:boto:Data: 2024-04-19T23:36:19.290 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.291 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.291 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.291 DEBUG:boto:Params: {} 2024-04-19T23:36:19.292 DEBUG:boto:Token: None 2024-04-19T23:36:19.292 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/?versions 2024-04-19T23:36:19.292 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ= 2024-04-19T23:36:19.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vehGpCOvrx7xR9gHdOhASATO/AQ=', 'Content-Length': '0'} 2024-04-19T23:36:19.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000092de9c525e5ad02-006622fff3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.296 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:36:19.297 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.297 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.297 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.297 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.298 DEBUG:boto:Method: GET 2024-04-19T23:36:19.299 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.299 DEBUG:boto:Data: 2024-04-19T23:36:19.299 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.299 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.299 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.300 DEBUG:boto:Params: {} 2024-04-19T23:36:19.300 DEBUG:boto:Token: None 2024-04-19T23:36:19.300 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.300 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4= 2024-04-19T23:36:19.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4=', 'Content-Length': '0'} 2024-04-19T23:36:19.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c6056a4702f66921-006622fff3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.303 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.303 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.303 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.303 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.304 DEBUG:boto:Method: GET 2024-04-19T23:36:19.304 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.304 DEBUG:boto:Data: 2024-04-19T23:36:19.304 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.304 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.304 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.304 DEBUG:boto:Params: {} 2024-04-19T23:36:19.304 DEBUG:boto:Token: None 2024-04-19T23:36:19.305 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.305 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4= 2024-04-19T23:36:19.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4=', 'Content-Length': '0'} 2024-04-19T23:36:19.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f4c64e52bb6aea34-006622fff3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.307 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.307 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.307 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.308 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.308 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.308 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.308 DEBUG:boto:Data: 2024-04-19T23:36:19.308 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.308 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.308 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.308 DEBUG:boto:Params: {} 2024-04-19T23:36:19.308 DEBUG:boto:Token: None 2024-04-19T23:36:19.308 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.309 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yer+Y+Q580OkzDPYoxV5Y6CAhn0= 2024-04-19T23:36:19.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yer+Y+Q580OkzDPYoxV5Y6CAhn0=', 'Content-Length': '0'} 2024-04-19T23:36:19.311 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006405d5b6e60dd917-006622fff3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.311 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.311 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.311 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.311 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.311 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.311 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.311 DEBUG:boto:Data: 2024-04-19T23:36:19.312 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.312 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.312 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.312 DEBUG:boto:Params: {} 2024-04-19T23:36:19.312 DEBUG:boto:Token: None 2024-04-19T23:36:19.312 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.312 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Yer+Y+Q580OkzDPYoxV5Y6CAhn0= 2024-04-19T23:36:19.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yer+Y+Q580OkzDPYoxV5Y6CAhn0=', 'Content-Length': '0'} 2024-04-19T23:36:19.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d1cd2938a18b56c-006622fff3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:36:19.315 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.315 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.315 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.316 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.316 DEBUG:boto:Method: GET 2024-04-19T23:36:19.316 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.316 DEBUG:boto:Data: 2024-04-19T23:36:19.316 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.316 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.316 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.316 DEBUG:boto:Params: {} 2024-04-19T23:36:19.316 DEBUG:boto:Token: None 2024-04-19T23:36:19.316 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.317 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4= 2024-04-19T23:36:19.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4=', 'Content-Length': '0'} 2024-04-19T23:36:19.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006791e3e217ca37ce-006622fff3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.319 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.319 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.319 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.319 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.319 DEBUG:boto:Method: GET 2024-04-19T23:36:19.319 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.319 DEBUG:boto:Data: 2024-04-19T23:36:19.319 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.319 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.320 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.320 DEBUG:boto:Params: {} 2024-04-19T23:36:19.320 DEBUG:boto:Token: None 2024-04-19T23:36:19.320 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.320 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4= 2024-04-19T23:36:19.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4=', 'Content-Length': '0'} 2024-04-19T23:36:19.322 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000790abe88fc9cf300-006622fff3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.322 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.322 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.322 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.323 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.323 DEBUG:boto:Method: GET 2024-04-19T23:36:19.323 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:36:19.323 DEBUG:boto:Data: 2024-04-19T23:36:19.323 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.323 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.323 DEBUG:boto:Params: {} 2024-04-19T23:36:19.323 DEBUG:boto:Token: None 2024-04-19T23:36:19.324 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4= 2024-04-19T23:36:19.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4=', 'Content-Length': '0'} 2024-04-19T23:36:19.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a208e163e1e9c4a7-006622fff3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.370 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.371 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.371 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.371 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.371 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.372 DEBUG:boto:Method: GET 2024-04-19T23:36:19.372 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:36:19.372 DEBUG:boto:Data: 2024-04-19T23:36:19.372 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.372 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.372 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.372 DEBUG:boto:Params: {} 2024-04-19T23:36:19.372 DEBUG:boto:Token: None 2024-04-19T23:36:19.372 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.373 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4= 2024-04-19T23:36:19.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4=', 'Content-Length': '0'} 2024-04-19T23:36:19.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008be9920ff744c3ae-006622fff3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.375 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.376 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.376 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.376 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.376 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:36:19.376 DEBUG:boto:Data: 2024-04-19T23:36:19.376 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.376 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.376 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.376 DEBUG:boto:Params: {} 2024-04-19T23:36:19.377 DEBUG:boto:Token: None 2024-04-19T23:36:19.377 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy 2024-04-19T23:36:19.377 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qIfC5Vek35a7ODecfjobGyv+QUs= 2024-04-19T23:36:19.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qIfC5Vek35a7ODecfjobGyv+QUs=', 'Content-Length': '0'} 2024-04-19T23:36:19.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f62cdd921e83c67-006622fff3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.379 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.379 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.380 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.380 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:36:19.380 DEBUG:boto:Data: 2024-04-19T23:36:19.380 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.380 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.380 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.380 DEBUG:boto:Params: {} 2024-04-19T23:36:19.380 DEBUG:boto:Token: None 2024-04-19T23:36:19.380 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy 2024-04-19T23:36:19.380 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qIfC5Vek35a7ODecfjobGyv+QUs= 2024-04-19T23:36:19.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qIfC5Vek35a7ODecfjobGyv+QUs=', 'Content-Length': '0'} 2024-04-19T23:36:19.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000503b738a9d509e6d-006622fff3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:36:19.383 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.383 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.383 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.384 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.384 DEBUG:boto:Method: GET 2024-04-19T23:36:19.384 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.384 DEBUG:boto:Data: 2024-04-19T23:36:19.384 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.384 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.384 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.384 DEBUG:boto:Params: {} 2024-04-19T23:36:19.384 DEBUG:boto:Token: None 2024-04-19T23:36:19.384 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.384 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4= 2024-04-19T23:36:19.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4=', 'Content-Length': '0'} 2024-04-19T23:36:19.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007618321804a28450-006622fff3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.387 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:36:19.387 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:36:19.387 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.387 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.387 DEBUG:boto:Method: GET 2024-04-19T23:36:19.388 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.388 DEBUG:boto:Data: 2024-04-19T23:36:19.388 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.388 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.388 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.388 DEBUG:boto:Params: {} 2024-04-19T23:36:19.388 DEBUG:boto:Token: None 2024-04-19T23:36:19.388 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:36:19.388 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4= 2024-04-19T23:36:19.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pOpzblYPNAY1+y8BrSD5EdHQ9F4=', 'Content-Length': '0'} 2024-04-19T23:36:19.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045cf2511add9f9c2-006622fff3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.391 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.391 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.391 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.391 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.391 DEBUG:boto:Method: GET 2024-04-19T23:36:19.391 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:36:19.391 DEBUG:boto:Data: 2024-04-19T23:36:19.391 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.391 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.391 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.391 DEBUG:boto:Params: {} 2024-04-19T23:36:19.392 DEBUG:boto:Token: None 2024-04-19T23:36:19.392 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.392 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4= 2024-04-19T23:36:19.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4=', 'Content-Length': '0'} 2024-04-19T23:36:19.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4192be4d0c059a7-006622fff3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.438 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.439 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.439 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.439 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.439 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:36:19.439 DEBUG:boto:Method: GET 2024-04-19T23:36:19.440 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:36:19.440 DEBUG:boto:Data: 2024-04-19T23:36:19.440 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.440 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.440 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.440 DEBUG:boto:Params: {} 2024-04-19T23:36:19.440 DEBUG:boto:Token: None 2024-04-19T23:36:19.440 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.441 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4= 2024-04-19T23:36:19.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AYPO0is6lbu5LWqyDwfrTc00+s4=', 'Content-Length': '0'} 2024-04-19T23:36:19.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049680299f166e068-006622fff3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.486 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:36:19.487 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-22 zones={test-zone3, test-zone2} 2024-04-19T23:36:19.487 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.487 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.487 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.487 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.488 DEBUG:boto:Data: 2024-04-19T23:36:19.488 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.488 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.488 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.488 DEBUG:boto:Params: {} 2024-04-19T23:36:19.488 DEBUG:boto:Token: None 2024-04-19T23:36:19.488 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002887e272c028b977-006622fff3-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.492 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.492 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.492 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.492 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.492 DEBUG:boto:Data: 2024-04-19T23:36:19.492 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.492 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.492 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.492 DEBUG:boto:Params: {} 2024-04-19T23:36:19.493 DEBUG:boto:Token: None 2024-04-19T23:36:19.493 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.493 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-amz-request-id', 'tx000005594470d1d398f22-006622fff3-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.539 DEBUG:rgw_multi.tests:zone=test-zone1 version=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:36:19.539 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.539 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.539 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.539 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.539 DEBUG:boto:Data: 2024-04-19T23:36:19.539 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.539 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.539 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.540 DEBUG:boto:Params: {} 2024-04-19T23:36:19.540 DEBUG:boto:Token: None 2024-04-19T23:36:19.540 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.540 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000072360512a86e8d-006622fff3-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.543 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.543 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.543 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.543 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.543 DEBUG:boto:Data: 2024-04-19T23:36:19.543 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.543 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.543 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.543 DEBUG:boto:Params: {} 2024-04-19T23:36:19.544 DEBUG:boto:Token: None 2024-04-19T23:36:19.544 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.544 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-amz-request-id', 'tx0000046b921252d7ad2d5-006622fff3-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.557 DEBUG:rgw_multi.tests:zone=test-zone2 version=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:36:19.557 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.557 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.558 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.558 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.558 DEBUG:boto:Data: 2024-04-19T23:36:19.558 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.558 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.558 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.558 DEBUG:boto:Params: {} 2024-04-19T23:36:19.558 DEBUG:boto:Token: None 2024-04-19T23:36:19.558 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.559 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.561 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006667769a8b9b0971-006622fff3-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.562 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.562 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.562 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.562 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.562 DEBUG:boto:Data: 2024-04-19T23:36:19.562 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.562 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.562 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.562 DEBUG:boto:Params: {} 2024-04-19T23:36:19.562 DEBUG:boto:Token: None 2024-04-19T23:36:19.563 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.563 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-amz-request-id', 'tx00000e5ddd20df49f9ff7-006622fff3-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.576 DEBUG:rgw_multi.tests:zone=test-zone3 version=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:36:19.576 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.576 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.576 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.576 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.576 DEBUG:boto:Data: 2024-04-19T23:36:19.576 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.576 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.577 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.577 DEBUG:boto:Params: {} 2024-04-19T23:36:19.577 DEBUG:boto:Token: None 2024-04-19T23:36:19.577 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.577 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000911d6c9401137f2c-006622fff3-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.580 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.580 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.580 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.581 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.581 DEBUG:boto:Data: 2024-04-19T23:36:19.581 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.581 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.581 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.581 DEBUG:boto:Params: {} 2024-04-19T23:36:19.581 DEBUG:boto:Token: None 2024-04-19T23:36:19.581 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.581 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-amz-request-id', 'tx0000097283d4048fc1cbc-006622fff3-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.593 DEBUG:rgw_multi.tests:zone=test-zone1 version=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:36:19.593 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.593 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.593 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.593 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.593 DEBUG:boto:Data: 2024-04-19T23:36:19.593 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.593 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.593 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.593 DEBUG:boto:Params: {} 2024-04-19T23:36:19.594 DEBUG:boto:Token: None 2024-04-19T23:36:19.594 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.594 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086b4e6ee82792228-006622fff3-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.597 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.597 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.597 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.597 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.597 DEBUG:boto:Data: 2024-04-19T23:36:19.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.597 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.597 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.598 DEBUG:boto:Params: {} 2024-04-19T23:36:19.598 DEBUG:boto:Token: None 2024-04-19T23:36:19.598 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.598 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-amz-request-id', 'tx00000b22dc260ebb496fe-006622fff3-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.643 DEBUG:rgw_multi.tests:zone=test-zone2 version=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:36:19.643 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.643 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.643 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.643 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.644 DEBUG:boto:Data: 2024-04-19T23:36:19.644 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.644 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.644 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.644 DEBUG:boto:Params: {} 2024-04-19T23:36:19.644 DEBUG:boto:Token: None 2024-04-19T23:36:19.644 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.644 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008275178e20e8ca67-006622fff3-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.650 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.650 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.651 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.651 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.651 DEBUG:boto:Data: 2024-04-19T23:36:19.651 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.651 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.651 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.651 DEBUG:boto:Params: {} 2024-04-19T23:36:19.651 DEBUG:boto:Token: None 2024-04-19T23:36:19.651 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.652 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-amz-request-id', 'tx00000a4a89aa15fdfcb9f-006622fff3-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.663 DEBUG:rgw_multi.tests:zone=test-zone3 version=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:36:19.663 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.663 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.663 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.663 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.663 DEBUG:boto:Data: 2024-04-19T23:36:19.663 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.663 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.664 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.664 DEBUG:boto:Params: {} 2024-04-19T23:36:19.664 DEBUG:boto:Token: None 2024-04-19T23:36:19.664 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.664 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c972632fc3d724e-006622fff3-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.667 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.667 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.667 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.667 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.668 DEBUG:boto:Data: 2024-04-19T23:36:19.668 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.668 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.668 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.668 DEBUG:boto:Params: {} 2024-04-19T23:36:19.668 DEBUG:boto:Token: None 2024-04-19T23:36:19.668 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.669 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-amz-request-id', 'tx000004a470b28d34045fd-006622fff3-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.715 DEBUG:rgw_multi.tests:zone=test-zone1 version=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:36:19.715 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.715 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.715 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.715 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.715 DEBUG:boto:Data: 2024-04-19T23:36:19.715 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.715 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.715 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.716 DEBUG:boto:Params: {} 2024-04-19T23:36:19.716 DEBUG:boto:Token: None 2024-04-19T23:36:19.716 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.716 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.718 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2e99fa4443d1e5a-006622fff3-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.719 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.719 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.719 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.719 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.719 DEBUG:boto:Data: 2024-04-19T23:36:19.719 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.719 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.719 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.719 DEBUG:boto:Params: {} 2024-04-19T23:36:19.719 DEBUG:boto:Token: None 2024-04-19T23:36:19.720 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.720 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.731 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-amz-request-id', 'tx000004acf4fe0b5ce4bb9-006622fff3-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.731 DEBUG:rgw_multi.tests:zone=test-zone2 version=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:36:19.731 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.731 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.731 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.731 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.732 DEBUG:boto:Data: 2024-04-19T23:36:19.732 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.732 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.732 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.732 DEBUG:boto:Params: {} 2024-04-19T23:36:19.732 DEBUG:boto:Token: None 2024-04-19T23:36:19.732 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.732 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a28dc5ef895b389-006622fff3-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.735 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.735 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.735 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.735 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.736 DEBUG:boto:Data: 2024-04-19T23:36:19.736 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.736 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.736 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.736 DEBUG:boto:Params: {} 2024-04-19T23:36:19.736 DEBUG:boto:Token: None 2024-04-19T23:36:19.736 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.736 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-amz-request-id', 'tx00000a6502c5619df8aa4-006622fff3-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.747 DEBUG:rgw_multi.tests:zone=test-zone3 version=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:36:19.747 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.748 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.748 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.748 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.748 DEBUG:boto:Data: 2024-04-19T23:36:19.748 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.748 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.748 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.748 DEBUG:boto:Params: {} 2024-04-19T23:36:19.748 DEBUG:boto:Token: None 2024-04-19T23:36:19.748 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.749 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e8a5e7d225af059-006622fff3-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.751 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.751 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.752 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.752 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.752 DEBUG:boto:Data: 2024-04-19T23:36:19.752 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.752 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.752 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.752 DEBUG:boto:Params: {} 2024-04-19T23:36:19.752 DEBUG:boto:Token: None 2024-04-19T23:36:19.752 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.752 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-amz-request-id', 'tx000009374a08c4de0b0e8-006622fff3-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.764 DEBUG:rgw_multi.tests:zone=test-zone1 version=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:36:19.764 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.764 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.764 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.764 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.765 DEBUG:boto:Data: 2024-04-19T23:36:19.765 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.765 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.765 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.765 DEBUG:boto:Params: {} 2024-04-19T23:36:19.765 DEBUG:boto:Token: None 2024-04-19T23:36:19.765 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.765 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.768 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000524e9eef860fdac0-006622fff3-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.769 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.769 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.769 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.769 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.769 DEBUG:boto:Data: 2024-04-19T23:36:19.769 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.769 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.769 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.769 DEBUG:boto:Params: {} 2024-04-19T23:36:19.770 DEBUG:boto:Token: None 2024-04-19T23:36:19.770 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.770 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.815 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-amz-request-id', 'tx00000e1242f0c188c768a-006622fff3-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.815 DEBUG:rgw_multi.tests:zone=test-zone2 version=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:36:19.815 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.815 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.816 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.816 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.816 DEBUG:boto:Data: 2024-04-19T23:36:19.816 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.817 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.817 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.817 DEBUG:boto:Params: {} 2024-04-19T23:36:19.817 DEBUG:boto:Token: None 2024-04-19T23:36:19.817 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.818 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0ffdd477833fdda-006622fff3-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.821 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.821 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.822 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.822 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.822 DEBUG:boto:Data: 2024-04-19T23:36:19.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.822 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.822 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.822 DEBUG:boto:Params: {} 2024-04-19T23:36:19.822 DEBUG:boto:Token: None 2024-04-19T23:36:19.822 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.823 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-amz-request-id', 'tx00000375734231ee3b72d-006622fff3-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.835 DEBUG:rgw_multi.tests:zone=test-zone3 version=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:36:19.835 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.836 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.836 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.836 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.836 DEBUG:boto:Data: 2024-04-19T23:36:19.836 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.836 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.836 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.836 DEBUG:boto:Params: {} 2024-04-19T23:36:19.836 DEBUG:boto:Token: None 2024-04-19T23:36:19.836 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.837 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea7b7b6377d957ea-006622fff3-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.840 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.840 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.840 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.840 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.840 DEBUG:boto:Data: 2024-04-19T23:36:19.840 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.841 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.841 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.841 DEBUG:boto:Params: {} 2024-04-19T23:36:19.841 DEBUG:boto:Token: None 2024-04-19T23:36:19.841 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.841 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.891 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-amz-request-id', 'tx000006342a5db4c24b05c-006622fff3-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.891 DEBUG:rgw_multi.tests:zone=test-zone1 version=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:36:19.891 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.891 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.891 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.892 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.892 DEBUG:boto:Data: 2024-04-19T23:36:19.892 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.892 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.892 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.892 DEBUG:boto:Params: {} 2024-04-19T23:36:19.892 DEBUG:boto:Token: None 2024-04-19T23:36:19.892 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.892 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.895 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c607d462bff52796-006622fff3-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.895 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.895 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.895 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.895 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.895 DEBUG:boto:Data: 2024-04-19T23:36:19.895 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.895 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.896 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.896 DEBUG:boto:Params: {} 2024-04-19T23:36:19.896 DEBUG:boto:Token: None 2024-04-19T23:36:19.896 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.896 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.908 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-amz-request-id', 'tx000000e5004259540b3ab-006622fff3-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.908 DEBUG:rgw_multi.tests:zone=test-zone2 version=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:36:19.908 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.908 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.908 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.908 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.908 DEBUG:boto:Data: 2024-04-19T23:36:19.908 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.908 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.908 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.908 DEBUG:boto:Params: {} 2024-04-19T23:36:19.909 DEBUG:boto:Token: None 2024-04-19T23:36:19.909 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.909 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de47055a984891cb-006622fff3-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.912 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.912 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.912 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.912 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.912 DEBUG:boto:Data: 2024-04-19T23:36:19.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.912 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.912 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.913 DEBUG:boto:Params: {} 2024-04-19T23:36:19.913 DEBUG:boto:Token: None 2024-04-19T23:36:19.913 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.913 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-amz-request-id', 'tx0000073691d47265f8a25-006622fff3-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.955 DEBUG:rgw_multi.tests:zone=test-zone3 version=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:36:19.955 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.955 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.955 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.955 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.955 DEBUG:boto:Data: 2024-04-19T23:36:19.955 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.955 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.955 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.955 DEBUG:boto:Params: {} 2024-04-19T23:36:19.956 DEBUG:boto:Token: None 2024-04-19T23:36:19.956 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.956 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050fba2232cb620c7-006622fff3-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.959 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.960 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.960 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.960 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.960 DEBUG:boto:Data: 2024-04-19T23:36:19.960 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.960 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:19.960 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.960 DEBUG:boto:Params: {} 2024-04-19T23:36:19.960 DEBUG:boto:Token: None 2024-04-19T23:36:19.961 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.961 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-amz-request-id', 'tx0000052fad066d5264e7c-006622fff3-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.972 DEBUG:rgw_multi.tests:zone=test-zone1 version=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:36:19.972 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.972 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.972 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.972 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.972 DEBUG:boto:Data: 2024-04-19T23:36:19.972 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.972 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.972 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.972 DEBUG:boto:Params: {} 2024-04-19T23:36:19.973 DEBUG:boto:Token: None 2024-04-19T23:36:19.974 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.974 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.976 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000567bd85ddf86047a-006622fff3-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.977 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.977 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.977 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.977 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.977 DEBUG:boto:Data: 2024-04-19T23:36:19.977 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.977 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:19.977 DEBUG:boto:Port: 8000 2024-04-19T23:36:19.977 DEBUG:boto:Params: {} 2024-04-19T23:36:19.978 DEBUG:boto:Token: None 2024-04-19T23:36:19.978 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.978 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:19.989 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-amz-request-id', 'tx000000d059f67a432596c-006622fff3-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.989 DEBUG:rgw_multi.tests:zone=test-zone2 version=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:36:19.989 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:19.989 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:19.990 DEBUG:boto:Method: HEAD 2024-04-19T23:36:19.990 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:19.990 DEBUG:boto:Data: 2024-04-19T23:36:19.990 DEBUG:boto:Headers: {} 2024-04-19T23:36:19.990 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.990 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.990 DEBUG:boto:Params: {} 2024-04-19T23:36:19.990 DEBUG:boto:Token: None 2024-04-19T23:36:19.990 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/ 2024-04-19T23:36:19.991 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA= 2024-04-19T23:36:19.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Urivz14WOm5T4hLxSNuAWf7KaVA=', 'Content-Length': '0'} 2024-04-19T23:36:19.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029e212024a90e8d5-006622fff3-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:19.994 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:19.994 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:19.994 DEBUG:boto:Method: PUT 2024-04-19T23:36:19.994 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:19.994 DEBUG:boto:Data: 2024-04-19T23:36:19.994 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:19.994 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:19.994 DEBUG:boto:Port: 8001 2024-04-19T23:36:19.994 DEBUG:boto:Params: {} 2024-04-19T23:36:19.994 DEBUG:boto:Token: None 2024-04-19T23:36:19.995 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:19 GMT /aktlji-22/obj 2024-04-19T23:36:19.995 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34= 2024-04-19T23:36:19.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bi+v8pFbtFpiUOnoy3D5A5HsD34='} 2024-04-19T23:36:20.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-amz-request-id', 'tx00000af1b80f930f573c7-006622fff4-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.007 DEBUG:rgw_multi.tests:zone=test-zone3 version=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:36:20.007 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:20.007 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:20.007 DEBUG:boto:Method: HEAD 2024-04-19T23:36:20.007 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:20.007 DEBUG:boto:Data: 2024-04-19T23:36:20.007 DEBUG:boto:Headers: {} 2024-04-19T23:36:20.007 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:20.007 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.008 DEBUG:boto:Params: {} 2024-04-19T23:36:20.008 DEBUG:boto:Token: None 2024-04-19T23:36:20.008 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/ 2024-04-19T23:36:20.008 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY= 2024-04-19T23:36:20.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY=', 'Content-Length': '0'} 2024-04-19T23:36:20.010 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf46d40828fcc769-006622fff4-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.011 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:20.011 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:20.011 DEBUG:boto:Method: PUT 2024-04-19T23:36:20.011 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:20.011 DEBUG:boto:Data: 2024-04-19T23:36:20.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:20.011 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:20.011 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.012 DEBUG:boto:Params: {} 2024-04-19T23:36:20.012 DEBUG:boto:Token: None 2024-04-19T23:36:20.012 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/obj 2024-04-19T23:36:20.012 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U= 2024-04-19T23:36:20.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U='} 2024-04-19T23:36:20.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-amz-request-id', 'tx000005d92421be41cd6ab-006622fff4-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.055 DEBUG:rgw_multi.tests:zone=test-zone1 version=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:36:20.055 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:20.055 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:20.055 DEBUG:boto:Method: HEAD 2024-04-19T23:36:20.055 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:20.055 DEBUG:boto:Data: 2024-04-19T23:36:20.055 DEBUG:boto:Headers: {} 2024-04-19T23:36:20.055 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:20.057 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.057 DEBUG:boto:Params: {} 2024-04-19T23:36:20.057 DEBUG:boto:Token: None 2024-04-19T23:36:20.057 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/ 2024-04-19T23:36:20.057 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY= 2024-04-19T23:36:20.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY=', 'Content-Length': '0'} 2024-04-19T23:36:20.060 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002cddec63081daeef-006622fff4-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.060 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:20.061 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:20.061 DEBUG:boto:Method: PUT 2024-04-19T23:36:20.061 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:20.061 DEBUG:boto:Data: 2024-04-19T23:36:20.061 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:20.061 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:20.061 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.061 DEBUG:boto:Params: {} 2024-04-19T23:36:20.061 DEBUG:boto:Token: None 2024-04-19T23:36:20.062 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/obj 2024-04-19T23:36:20.062 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U= 2024-04-19T23:36:20.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U='} 2024-04-19T23:36:20.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-amz-request-id', 'tx00000f4dea2e4b5ba1281-006622fff4-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.074 DEBUG:rgw_multi.tests:zone=test-zone2 version=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:36:20.074 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:20.074 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:20.074 DEBUG:boto:Method: HEAD 2024-04-19T23:36:20.074 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:20.074 DEBUG:boto:Data: 2024-04-19T23:36:20.075 DEBUG:boto:Headers: {} 2024-04-19T23:36:20.075 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:20.075 DEBUG:boto:Port: 8001 2024-04-19T23:36:20.075 DEBUG:boto:Params: {} 2024-04-19T23:36:20.075 DEBUG:boto:Token: None 2024-04-19T23:36:20.075 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/ 2024-04-19T23:36:20.075 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY= 2024-04-19T23:36:20.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY=', 'Content-Length': '0'} 2024-04-19T23:36:20.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f2507df73be13e3-006622fff4-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.078 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:20.078 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:20.078 DEBUG:boto:Method: PUT 2024-04-19T23:36:20.078 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:20.078 DEBUG:boto:Data: 2024-04-19T23:36:20.078 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:20.079 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:20.079 DEBUG:boto:Port: 8001 2024-04-19T23:36:20.079 DEBUG:boto:Params: {} 2024-04-19T23:36:20.079 DEBUG:boto:Token: None 2024-04-19T23:36:20.079 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/obj 2024-04-19T23:36:20.079 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U= 2024-04-19T23:36:20.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U='} 2024-04-19T23:36:20.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-amz-request-id', 'tx000004357de6f6e3a64c1-006622fff4-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.090 DEBUG:rgw_multi.tests:zone=test-zone3 version=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:36:20.090 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:20.090 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:20.090 DEBUG:boto:Method: HEAD 2024-04-19T23:36:20.090 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:20.091 DEBUG:boto:Data: 2024-04-19T23:36:20.091 DEBUG:boto:Headers: {} 2024-04-19T23:36:20.091 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:20.091 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.091 DEBUG:boto:Params: {} 2024-04-19T23:36:20.091 DEBUG:boto:Token: None 2024-04-19T23:36:20.091 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/ 2024-04-19T23:36:20.091 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY= 2024-04-19T23:36:20.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY=', 'Content-Length': '0'} 2024-04-19T23:36:20.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f0221d48dee2807-006622fff4-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.094 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:20.094 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:20.094 DEBUG:boto:Method: PUT 2024-04-19T23:36:20.094 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:20.094 DEBUG:boto:Data: 2024-04-19T23:36:20.095 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:20.095 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:20.095 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.095 DEBUG:boto:Params: {} 2024-04-19T23:36:20.095 DEBUG:boto:Token: None 2024-04-19T23:36:20.095 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/obj 2024-04-19T23:36:20.095 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U= 2024-04-19T23:36:20.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U='} 2024-04-19T23:36:20.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-amz-request-id', 'tx00000f156183ed14f08e5-006622fff4-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.107 DEBUG:rgw_multi.tests:zone=test-zone1 version=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:36:20.107 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:20.107 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:20.107 DEBUG:boto:Method: HEAD 2024-04-19T23:36:20.107 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:20.107 DEBUG:boto:Data: 2024-04-19T23:36:20.107 DEBUG:boto:Headers: {} 2024-04-19T23:36:20.108 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:20.108 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.108 DEBUG:boto:Params: {} 2024-04-19T23:36:20.108 DEBUG:boto:Token: None 2024-04-19T23:36:20.108 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/ 2024-04-19T23:36:20.108 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY= 2024-04-19T23:36:20.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY=', 'Content-Length': '0'} 2024-04-19T23:36:20.111 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004602f0febe2d3379-006622fff4-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.111 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:20.111 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:20.111 DEBUG:boto:Method: PUT 2024-04-19T23:36:20.111 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:20.112 DEBUG:boto:Data: 2024-04-19T23:36:20.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:20.112 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:20.112 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.112 DEBUG:boto:Params: {} 2024-04-19T23:36:20.112 DEBUG:boto:Token: None 2024-04-19T23:36:20.112 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/obj 2024-04-19T23:36:20.112 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U= 2024-04-19T23:36:20.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U='} 2024-04-19T23:36:20.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-amz-request-id', 'tx00000fa5cb24a4e0b7bfa-006622fff4-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.159 DEBUG:rgw_multi.tests:zone=test-zone2 version=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:36:20.159 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:20.159 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:20.159 DEBUG:boto:Method: HEAD 2024-04-19T23:36:20.159 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:20.159 DEBUG:boto:Data: 2024-04-19T23:36:20.159 DEBUG:boto:Headers: {} 2024-04-19T23:36:20.159 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:20.160 DEBUG:boto:Port: 8001 2024-04-19T23:36:20.160 DEBUG:boto:Params: {} 2024-04-19T23:36:20.160 DEBUG:boto:Token: None 2024-04-19T23:36:20.160 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/ 2024-04-19T23:36:20.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY= 2024-04-19T23:36:20.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY=', 'Content-Length': '0'} 2024-04-19T23:36:20.163 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010e95451c998a8b3-006622fff4-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.163 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:20.163 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:20.163 DEBUG:boto:Method: PUT 2024-04-19T23:36:20.163 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:20.163 DEBUG:boto:Data: 2024-04-19T23:36:20.164 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:20.164 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:20.164 DEBUG:boto:Port: 8001 2024-04-19T23:36:20.164 DEBUG:boto:Params: {} 2024-04-19T23:36:20.164 DEBUG:boto:Token: None 2024-04-19T23:36:20.164 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/obj 2024-04-19T23:36:20.164 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U= 2024-04-19T23:36:20.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U='} 2024-04-19T23:36:20.175 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-amz-request-id', 'tx000002ac7f210aeab1972-006622fff4-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.175 DEBUG:rgw_multi.tests:zone=test-zone3 version=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:36:20.175 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:20.175 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:20.175 DEBUG:boto:Method: HEAD 2024-04-19T23:36:20.175 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:20.175 DEBUG:boto:Data: 2024-04-19T23:36:20.175 DEBUG:boto:Headers: {} 2024-04-19T23:36:20.176 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:20.176 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.176 DEBUG:boto:Params: {} 2024-04-19T23:36:20.176 DEBUG:boto:Token: None 2024-04-19T23:36:20.176 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/ 2024-04-19T23:36:20.176 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY= 2024-04-19T23:36:20.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY=', 'Content-Length': '0'} 2024-04-19T23:36:20.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c6c2d6d9286d7ca-006622fff4-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.179 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:20.179 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:20.179 DEBUG:boto:Method: PUT 2024-04-19T23:36:20.179 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:20.179 DEBUG:boto:Data: 2024-04-19T23:36:20.179 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:20.179 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:20.180 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.180 DEBUG:boto:Params: {} 2024-04-19T23:36:20.180 DEBUG:boto:Token: None 2024-04-19T23:36:20.180 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/obj 2024-04-19T23:36:20.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U= 2024-04-19T23:36:20.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U='} 2024-04-19T23:36:20.191 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-amz-request-id', 'tx000002ca36c15878cefcc-006622fff4-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.191 DEBUG:rgw_multi.tests:zone=test-zone1 version=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:36:20.192 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:20.192 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:20.192 DEBUG:boto:Method: HEAD 2024-04-19T23:36:20.192 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:20.192 DEBUG:boto:Data: 2024-04-19T23:36:20.192 DEBUG:boto:Headers: {} 2024-04-19T23:36:20.192 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:20.192 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.192 DEBUG:boto:Params: {} 2024-04-19T23:36:20.193 DEBUG:boto:Token: None 2024-04-19T23:36:20.193 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/ 2024-04-19T23:36:20.193 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY= 2024-04-19T23:36:20.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY=', 'Content-Length': '0'} 2024-04-19T23:36:20.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a938c069f4e09768-006622fff4-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.196 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:20.196 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:20.196 DEBUG:boto:Method: PUT 2024-04-19T23:36:20.196 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:20.196 DEBUG:boto:Data: 2024-04-19T23:36:20.196 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:20.196 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:20.197 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.197 DEBUG:boto:Params: {} 2024-04-19T23:36:20.197 DEBUG:boto:Token: None 2024-04-19T23:36:20.197 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/obj 2024-04-19T23:36:20.197 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U= 2024-04-19T23:36:20.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U='} 2024-04-19T23:36:20.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-amz-request-id', 'tx0000043080b78fe338754-006622fff4-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.209 DEBUG:rgw_multi.tests:zone=test-zone2 version=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:36:20.209 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:20.209 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:20.209 DEBUG:boto:Method: HEAD 2024-04-19T23:36:20.209 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:20.209 DEBUG:boto:Data: 2024-04-19T23:36:20.210 DEBUG:boto:Headers: {} 2024-04-19T23:36:20.210 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:20.210 DEBUG:boto:Port: 8001 2024-04-19T23:36:20.210 DEBUG:boto:Params: {} 2024-04-19T23:36:20.210 DEBUG:boto:Token: None 2024-04-19T23:36:20.210 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/ 2024-04-19T23:36:20.210 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY= 2024-04-19T23:36:20.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY=', 'Content-Length': '0'} 2024-04-19T23:36:20.213 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000677657eb77349ef1-006622fff4-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.213 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:20.214 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:20.214 DEBUG:boto:Method: PUT 2024-04-19T23:36:20.214 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:20.214 DEBUG:boto:Data: 2024-04-19T23:36:20.214 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:20.214 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:20.214 DEBUG:boto:Port: 8001 2024-04-19T23:36:20.214 DEBUG:boto:Params: {} 2024-04-19T23:36:20.215 DEBUG:boto:Token: None 2024-04-19T23:36:20.215 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/obj 2024-04-19T23:36:20.215 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U= 2024-04-19T23:36:20.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U='} 2024-04-19T23:36:20.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-amz-request-id', 'tx000005e99b3f5ff864ebd-006622fff4-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.259 DEBUG:rgw_multi.tests:zone=test-zone3 version=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:36:20.259 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:20.259 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:20.259 DEBUG:boto:Method: HEAD 2024-04-19T23:36:20.259 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:20.259 DEBUG:boto:Data: 2024-04-19T23:36:20.259 DEBUG:boto:Headers: {} 2024-04-19T23:36:20.260 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:20.260 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.260 DEBUG:boto:Params: {} 2024-04-19T23:36:20.260 DEBUG:boto:Token: None 2024-04-19T23:36:20.260 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/ 2024-04-19T23:36:20.260 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY= 2024-04-19T23:36:20.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY=', 'Content-Length': '0'} 2024-04-19T23:36:20.264 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ecbbb4ee7be558b-006622fff4-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.264 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:20.264 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:20.264 DEBUG:boto:Method: PUT 2024-04-19T23:36:20.265 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:20.265 DEBUG:boto:Data: 2024-04-19T23:36:20.265 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:20.265 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:36:20.265 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.265 DEBUG:boto:Params: {} 2024-04-19T23:36:20.265 DEBUG:boto:Token: None 2024-04-19T23:36:20.265 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/obj 2024-04-19T23:36:20.265 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U= 2024-04-19T23:36:20.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U='} 2024-04-19T23:36:20.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-amz-request-id', 'tx00000016f1b1844fb600e-006622fff4-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.311 DEBUG:rgw_multi.tests:zone=test-zone1 version=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:36:20.311 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:20.311 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:20.311 DEBUG:boto:Method: HEAD 2024-04-19T23:36:20.311 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:20.311 DEBUG:boto:Data: 2024-04-19T23:36:20.311 DEBUG:boto:Headers: {} 2024-04-19T23:36:20.311 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:20.311 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.312 DEBUG:boto:Params: {} 2024-04-19T23:36:20.312 DEBUG:boto:Token: None 2024-04-19T23:36:20.312 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/ 2024-04-19T23:36:20.312 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY= 2024-04-19T23:36:20.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY=', 'Content-Length': '0'} 2024-04-19T23:36:20.315 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000910aac3663380b92-006622fff4-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.315 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:20.315 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:20.315 DEBUG:boto:Method: PUT 2024-04-19T23:36:20.315 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:20.315 DEBUG:boto:Data: 2024-04-19T23:36:20.316 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:20.316 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:36:20.316 DEBUG:boto:Port: 8000 2024-04-19T23:36:20.316 DEBUG:boto:Params: {} 2024-04-19T23:36:20.316 DEBUG:boto:Token: None 2024-04-19T23:36:20.316 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/obj 2024-04-19T23:36:20.316 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U= 2024-04-19T23:36:20.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U='} 2024-04-19T23:36:20.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-amz-request-id', 'tx0000031cd84d0105f8a81-006622fff4-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.363 DEBUG:rgw_multi.tests:zone=test-zone2 version=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:36:20.363 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:36:20.363 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:36:20.363 DEBUG:boto:Method: HEAD 2024-04-19T23:36:20.363 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:36:20.363 DEBUG:boto:Data: 2024-04-19T23:36:20.363 DEBUG:boto:Headers: {} 2024-04-19T23:36:20.364 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:20.364 DEBUG:boto:Port: 8001 2024-04-19T23:36:20.364 DEBUG:boto:Params: {} 2024-04-19T23:36:20.364 DEBUG:boto:Token: None 2024-04-19T23:36:20.364 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/ 2024-04-19T23:36:20.364 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY= 2024-04-19T23:36:20.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zl7KSpNSPdt3nEMxnZJfmfTzBeY=', 'Content-Length': '0'} 2024-04-19T23:36:20.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdf42f6ffcdf6a1e-006622fff4-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.367 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:36:20.367 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:36:20.368 DEBUG:boto:Method: PUT 2024-04-19T23:36:20.368 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:36:20.368 DEBUG:boto:Data: 2024-04-19T23:36:20.368 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-19T23:36:20.368 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:36:20.368 DEBUG:boto:Port: 8001 2024-04-19T23:36:20.368 DEBUG:boto:Params: {} 2024-04-19T23:36:20.368 DEBUG:boto:Token: None 2024-04-19T23:36:20.368 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 19 Apr 2024 23:36:20 GMT /aktlji-22/obj 2024-04-19T23:36:20.369 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U= 2024-04-19T23:36:20.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:36:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rI1XppfYoLDEqXGUx9w0aDN4d5U='} 2024-04-19T23:36:20.380 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-amz-request-id', 'tx00000c167d32709135525-006622fff4-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:36:20.380 DEBUG:rgw_multi.tests:zone=test-zone3 version=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:36:20.380 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-22 2024-04-19T23:36:20.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:36:20.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:36:20.381 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:36:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:36:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:36:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:36:20.496+0000 7fdef4228780 1 waiting for incremental sync to catch up: 2024-04-19T23:36:20.491 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , 00000000004.34.10, , , , , ] 2024-04-19T23:36:20.491 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , 00000000005.36.10, , , 00000000020.76.12, , ] 2024-04-19T23:36:25.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:36:25.497+0000 7fdef4228780 1 waiting for incremental sync to catch up: 2024-04-19T23:36:25.493 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , 00000000004.34.10, , , , , ] 2024-04-19T23:36:25.493 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , 00000000005.36.10, , , 00000000020.76.12, , ] 2024-04-19T23:36:30.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:36:30.498+0000 7fdef4228780 1 waiting for incremental sync to catch up: 2024-04-19T23:36:30.494 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , 00000000005.36.10, , , , , ] 2024-04-19T23:36:30.494 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , 00000000005.36.10, , , 00000000020.76.12, , ] 2024-04-19T23:36:35.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:36:35.500+0000 7fdef4228780 1 waiting for incremental sync to catch up: 2024-04-19T23:36:35.495 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , 00000000005.36.10, , , , , ] 2024-04-19T23:36:35.496 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , 00000000005.36.10, , , 00000000020.76.12, , ] 2024-04-19T23:36:40.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:36:40.501+0000 7fdef4228780 1 bucket sync caught up with source: 2024-04-19T23:36:40.497 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , 00000000005.36.10, , , 00000000030.106.6, , ] 2024-04-19T23:36:40.497 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , 00000000005.36.10, , , 00000000020.76.12, , ] 2024-04-19T23:36:40.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:36:40.501+0000 7fdef4228780 0 bucket checkpoint complete 2024-04-19T23:36:40.514 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-22 2024-04-19T23:36:40.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:36:40.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:36:40.514 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:36:40.581 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:36:40.582 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:36:40.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:36:40.627+0000 7fd365513780 1 waiting for incremental sync to catch up: 2024-04-19T23:36:40.623 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000005.36.10, , , 00000000022.82.6, , ] 2024-04-19T23:36:40.623 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:36:45.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:36:45.629+0000 7fd365513780 1 waiting for incremental sync to catch up: 2024-04-19T23:36:45.625 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000006.108.10, , , 00000000022.82.6, , ] 2024-04-19T23:36:45.625 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:36:50.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:36:50.630+0000 7fd365513780 1 waiting for incremental sync to catch up: 2024-04-19T23:36:50.627 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000006.108.10, , , 00000000022.82.6, , ] 2024-04-19T23:36:50.627 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:36:55.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:36:55.631+0000 7fd365513780 1 waiting for incremental sync to catch up: 2024-04-19T23:36:55.630 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000006.108.10, , , 00000000022.82.6, , ] 2024-04-19T23:36:55.631 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:37:00.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:00.633+0000 7fd365513780 1 waiting for incremental sync to catch up: 2024-04-19T23:37:00.630 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000006.108.10, , , 00000000022.82.6, , ] 2024-04-19T23:37:00.630 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:37:05.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:05.634+0000 7fd365513780 1 waiting for incremental sync to catch up: 2024-04-19T23:37:05.631 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000006.108.10, , , 00000000022.82.6, , ] 2024-04-19T23:37:05.631 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:37:10.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:10.636+0000 7fd365513780 1 waiting for incremental sync to catch up: 2024-04-19T23:37:10.632 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000006.108.10, , , 00000000022.82.6, , ] 2024-04-19T23:37:10.632 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:37:15.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:15.637+0000 7fd365513780 1 waiting for incremental sync to catch up: 2024-04-19T23:37:15.633 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000006.108.10, , , 00000000022.82.6, , ] 2024-04-19T23:37:15.634 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:37:20.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:20.638+0000 7fd365513780 1 waiting for incremental sync to catch up: 2024-04-19T23:37:20.635 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000006.108.10, , , 00000000022.82.6, , ] 2024-04-19T23:37:20.635 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:37:25.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:25.641+0000 7fd365513780 1 waiting for incremental sync to catch up: 2024-04-19T23:37:25.636 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000006.108.10, , , 00000000022.82.6, , ] 2024-04-19T23:37:25.637 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:37:30.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:30.643+0000 7fd365513780 1 waiting for incremental sync to catch up: 2024-04-19T23:37:30.638 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000006.108.10, , , 00000000022.82.6, , ] 2024-04-19T23:37:30.638 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:37:35.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:35.644+0000 7fd365513780 1 waiting for incremental sync to catch up: 2024-04-19T23:37:35.640 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000006.108.10, , , 00000000022.82.6, , ] 2024-04-19T23:37:35.640 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:37:40.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:40.647+0000 7fd365513780 1 bucket sync caught up with source: 2024-04-19T23:37:40.643 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000006.108.10, , , 00000000044.144.10, , ] 2024-04-19T23:37:40.644 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000006.108.10, , , 00000000030.106.6, , ] 2024-04-19T23:37:40.644 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:40.647+0000 7fd365513780 0 bucket checkpoint complete 2024-04-19T23:37:40.689 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-22 2024-04-19T23:37:40.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:37:40.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:37:40.690 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:37:40.751 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:37:40.751 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:37:40.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:40.797+0000 7f9de764b780 1 waiting for incremental sync to catch up: 2024-04-19T23:37:40.792 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000044.152.10, , ] 2024-04-19T23:37:40.793 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:37:45.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:45.798+0000 7f9de764b780 1 waiting for incremental sync to catch up: 2024-04-19T23:37:45.794 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000044.152.10, , ] 2024-04-19T23:37:45.794 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:37:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:50.800+0000 7f9de764b780 1 waiting for incremental sync to catch up: 2024-04-19T23:37:50.796 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000044.152.10, , ] 2024-04-19T23:37:50.796 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:37:55.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:55.801+0000 7f9de764b780 1 bucket sync caught up with source: 2024-04-19T23:37:55.797 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:37:55.797 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:37:55.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:55.801+0000 7f9de764b780 0 bucket checkpoint complete 2024-04-19T23:37:55.814 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-22 2024-04-19T23:37:55.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:37:55.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:37:55.815 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:37:55.881 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:37:55.881 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:37:55.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:37:55.926+0000 7f9551a7d780 1 waiting for incremental sync to catch up: 2024-04-19T23:37:55.922 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000019.85.6, , ] 2024-04-19T23:37:55.922 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:38:00.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:00.927+0000 7f9551a7d780 1 waiting for incremental sync to catch up: 2024-04-19T23:38:00.924 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000036.134.6, , ] 2024-04-19T23:38:00.924 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:38:05.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:05.928+0000 7f9551a7d780 1 waiting for incremental sync to catch up: 2024-04-19T23:38:05.925 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000036.134.6, , ] 2024-04-19T23:38:05.925 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:38:10.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:10.936+0000 7f9551a7d780 1 waiting for incremental sync to catch up: 2024-04-19T23:38:10.934 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000036.134.6, , ] 2024-04-19T23:38:10.934 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:38:15.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:15.938+0000 7f9551a7d780 1 waiting for incremental sync to catch up: 2024-04-19T23:38:15.935 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000036.134.6, , ] 2024-04-19T23:38:15.935 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:38:20.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:20.939+0000 7f9551a7d780 1 waiting for incremental sync to catch up: 2024-04-19T23:38:20.936 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000036.134.6, , ] 2024-04-19T23:38:20.936 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:38:25.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:25.940+0000 7f9551a7d780 1 waiting for incremental sync to catch up: 2024-04-19T23:38:25.938 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000036.134.6, , ] 2024-04-19T23:38:25.938 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:38:30.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:30.944+0000 7f9551a7d780 1 waiting for incremental sync to catch up: 2024-04-19T23:38:30.942 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000036.134.6, , ] 2024-04-19T23:38:30.942 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:38:35.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:35.946+0000 7f9551a7d780 1 waiting for incremental sync to catch up: 2024-04-19T23:38:35.943 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000036.134.6, , ] 2024-04-19T23:38:35.943 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:38:40.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:40.947+0000 7f9551a7d780 1 bucket sync caught up with source: 2024-04-19T23:38:40.945 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:38:40.945 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.46.10, , , 00000000045.154.10, , ] 2024-04-19T23:38:40.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:40.947+0000 7f9551a7d780 0 bucket checkpoint complete 2024-04-19T23:38:40.962 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-22 2024-04-19T23:38:40.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:38:40.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:38:40.963 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:38:41.028 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:38:41.028 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:38:41.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:41.072+0000 7f29c78e1780 1 waiting for incremental sync to catch up: 2024-04-19T23:38:41.069 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.60.10, , , 00000000044.166.10, , ] 2024-04-19T23:38:41.069 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.60.10, , , 00000000045.168.10, , ] 2024-04-19T23:38:46.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:46.073+0000 7f29c78e1780 1 waiting for incremental sync to catch up: 2024-04-19T23:38:46.070 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.60.10, , , 00000000044.166.10, , ] 2024-04-19T23:38:46.071 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.60.10, , , 00000000045.168.10, , ] 2024-04-19T23:38:51.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:51.074+0000 7f29c78e1780 1 waiting for incremental sync to catch up: 2024-04-19T23:38:51.072 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.60.10, , , 00000000044.166.10, , ] 2024-04-19T23:38:51.072 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.60.10, , , 00000000045.168.10, , ] 2024-04-19T23:38:56.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:38:56.076+0000 7f29c78e1780 1 waiting for incremental sync to catch up: 2024-04-19T23:38:56.073 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.60.10, , , 00000000044.166.10, , ] 2024-04-19T23:38:56.074 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.60.10, , , 00000000045.168.10, , ] 2024-04-19T23:39:01.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:01.077+0000 7f29c78e1780 1 waiting for incremental sync to catch up: 2024-04-19T23:39:01.075 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.60.10, , , 00000000044.166.10, , ] 2024-04-19T23:39:01.075 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.60.10, , , 00000000045.168.10, , ] 2024-04-19T23:39:06.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:06.079+0000 7f29c78e1780 1 waiting for incremental sync to catch up: 2024-04-19T23:39:06.076 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.60.10, , , 00000000044.166.10, , ] 2024-04-19T23:39:06.076 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.60.10, , , 00000000045.168.10, , ] 2024-04-19T23:39:11.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:11.081+0000 7f29c78e1780 1 waiting for incremental sync to catch up: 2024-04-19T23:39:11.078 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.60.10, , , 00000000044.166.10, , ] 2024-04-19T23:39:11.078 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.60.10, , , 00000000045.168.10, , ] 2024-04-19T23:39:14.502 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-19T23:39:14.505+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-19T23:39:16.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:16.082+0000 7f29c78e1780 1 bucket sync caught up with source: 2024-04-19T23:39:16.079 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000004.60.10, , , 00000000045.168.10, , ] 2024-04-19T23:39:16.079 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000004.60.10, , , 00000000045.168.10, , ] 2024-04-19T23:39:16.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:16.082+0000 7f29c78e1780 0 bucket checkpoint complete 2024-04-19T23:39:16.096 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-22 2024-04-19T23:39:16.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:39:16.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:39:16.097 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:39:16.165 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:16.165 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:39:16.212+0000 7f3f32e0f780 1 bucket sync caught up with source: 2024-04-19T23:39:16.209 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , 00000000004.60.10, , , 00000000045.168.10, , ] 2024-04-19T23:39:16.209 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , 00000000004.60.10, , , 00000000045.168.10, , ] 2024-04-19T23:39:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:39:16.212+0000 7f3f32e0f780 0 bucket checkpoint complete 2024-04-19T23:39:16.224 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-22 zones={test-zone2, test-zone1} 2024-04-19T23:39:16.225 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.225 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.225 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.225 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:39:16.225 DEBUG:boto:Data: 2024-04-19T23:39:16.225 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.225 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.225 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.225 DEBUG:boto:Params: {} 2024-04-19T23:39:16.226 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:39:16.226 DEBUG:boto:Token: None 2024-04-19T23:39:16.226 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.227 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ECbJQKbZHy/Cr3V/zjOILGgoCLE= 2024-04-19T23:39:16.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ECbJQKbZHy/Cr3V/zjOILGgoCLE=', 'Content-Length': '0'} 2024-04-19T23:39:16.274 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000614f5f783ecdacbe-00662300a4-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.274 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.275 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.275 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.275 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:39:16.275 DEBUG:boto:Data: 2024-04-19T23:39:16.275 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.275 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.275 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.275 DEBUG:boto:Params: {} 2024-04-19T23:39:16.275 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:39:16.276 DEBUG:boto:Token: None 2024-04-19T23:39:16.276 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.276 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ECbJQKbZHy/Cr3V/zjOILGgoCLE= 2024-04-19T23:39:16.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ECbJQKbZHy/Cr3V/zjOILGgoCLE=', 'Content-Length': '0'} 2024-04-19T23:39:16.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002356175cc4d7bdc7-00662300a4-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:39:16.308 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.308 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.308 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:16.308 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:16.309 DEBUG:boto:Method: GET 2024-04-19T23:39:16.309 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:39:16.309 DEBUG:boto:Data: 2024-04-19T23:39:16.309 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.309 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.309 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.309 DEBUG:boto:Params: {} 2024-04-19T23:39:16.309 DEBUG:boto:Token: None 2024-04-19T23:39:16.309 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/?versions 2024-04-19T23:39:16.309 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q5Q8TARWLj/nEM0ljgSzul5tzDc= 2024-04-19T23:39:16.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q5Q8TARWLj/nEM0ljgSzul5tzDc=', 'Content-Length': '0'} 2024-04-19T23:39:16.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e50278e3f6975af2-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.313 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOotrue2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjI8Qos7O61Drj6zKFSuRQg5BJ8qTn1Itfalse2024-04-19T23:36:20.273Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITUfalse2024-04-19T23:36:20.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUjbMVzWXMoq8kQHxc5fXZiC69vNXkcjfalse2024-04-19T23:36:20.222Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97false2024-04-19T23:36:20.188Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2false2024-04-19T23:36:20.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Jy1wSwswautcJ.pAM0qXnm5CTgNceOfalse2024-04-19T23:36:20.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-false2024-04-19T23:36:20.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7false2024-04-19T23:36:20.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQ1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgRfalse2024-04-19T23:36:20.019Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipPC6NaG1ejWpptAGf0zgZ-gqpdm0DMfalse2024-04-19T23:36:20.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoECNSXdMldM0FoTJgSxtKpnhGWUHc5tfalse2024-04-19T23:36:20.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFo79-FV5lzRfFhYZKDH5zbjzjFRvFmofalse2024-04-19T23:36:19.985Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.9ZzH.1hozF9j0blCcYN54OsXVmWMTfalse2024-04-19T23:36:19.968Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnLS7759d.h0bWJMnVj7IAdVTVbZ2LzAfalse2024-04-19T23:36:20.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.SsB-LJAFCELIGc-2lad2-2jTKdmQIufalse2024-04-19T23:36:19.904Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5IMca-NQG3TDmPnhaU0a46jfFv2nxR8false2024-04-19T23:36:19.920Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3PIDP6j1VZfmBljRrKUGkIUmq4DPgifalse2024-04-19T23:36:19.849Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.sqcIpylWkdZwQp4UqpDBbyySodTBXSfalse2024-04-19T23:36:19.778Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCzz.Nb3nU2CWGe2vMCDYlTHkK18iRLfalse2024-04-19T23:36:19.760Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjudeGFtd00C8TRIOlco59Eefd7c7ffA6false2024-04-19T23:36:19.831Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4false2024-04-19T23:36:19.744Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgoQj3EN8LOJNZ9C2JDigvTcIgypIbDxfalse2024-04-19T23:36:19.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzyfalse2024-04-19T23:36:19.727Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.169NmmrUry8iq70c7.ZNdp4i5iqOCFfalse2024-04-19T23:36:19.659Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE5taibPWiUs3cuEimynrjQD8cXTCAELfalse2024-04-19T23:36:19.589Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNjFtS4LHZJDkGsaKejPMQyqPrZjbuN2false2024-04-19T23:36:19.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmTjHqLZC66BQsk7jiah7FJSChQvGdxfalse2024-04-19T23:36:19.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlKfLYYBA1E.lSLR49AMlK-Kgik5ij5Rfalse2024-04-19T23:36:19.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzwsSl6whrs-j2uo2xvsRdnhCfJSQ4Mafalse2024-04-19T23:36:19.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.315 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:39:16.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.318 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:39:16.318 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.318 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.318 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:16.318 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:16.318 DEBUG:boto:Method: GET 2024-04-19T23:39:16.318 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:39:16.319 DEBUG:boto:Data: 2024-04-19T23:39:16.319 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.319 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.319 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.319 DEBUG:boto:Params: {} 2024-04-19T23:39:16.319 DEBUG:boto:Token: None 2024-04-19T23:39:16.319 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/?versions 2024-04-19T23:39:16.319 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q5Q8TARWLj/nEM0ljgSzul5tzDc= 2024-04-19T23:39:16.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q5Q8TARWLj/nEM0ljgSzul5tzDc=', 'Content-Length': '0'} 2024-04-19T23:39:16.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012e3d5538c05b9a9-00662300a4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.323 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOotrue2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjI8Qos7O61Drj6zKFSuRQg5BJ8qTn1Itfalse2024-04-19T23:36:20.273Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITUfalse2024-04-19T23:36:20.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUjbMVzWXMoq8kQHxc5fXZiC69vNXkcjfalse2024-04-19T23:36:20.222Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97false2024-04-19T23:36:20.188Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2false2024-04-19T23:36:20.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Jy1wSwswautcJ.pAM0qXnm5CTgNceOfalse2024-04-19T23:36:20.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-false2024-04-19T23:36:20.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7false2024-04-19T23:36:20.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQ1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgRfalse2024-04-19T23:36:20.019Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipPC6NaG1ejWpptAGf0zgZ-gqpdm0DMfalse2024-04-19T23:36:20.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoECNSXdMldM0FoTJgSxtKpnhGWUHc5tfalse2024-04-19T23:36:20.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFo79-FV5lzRfFhYZKDH5zbjzjFRvFmofalse2024-04-19T23:36:19.985Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.9ZzH.1hozF9j0blCcYN54OsXVmWMTfalse2024-04-19T23:36:19.968Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnLS7759d.h0bWJMnVj7IAdVTVbZ2LzAfalse2024-04-19T23:36:20.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.SsB-LJAFCELIGc-2lad2-2jTKdmQIufalse2024-04-19T23:36:19.904Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5IMca-NQG3TDmPnhaU0a46jfFv2nxR8false2024-04-19T23:36:19.920Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3PIDP6j1VZfmBljRrKUGkIUmq4DPgifalse2024-04-19T23:36:19.849Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.sqcIpylWkdZwQp4UqpDBbyySodTBXSfalse2024-04-19T23:36:19.778Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCzz.Nb3nU2CWGe2vMCDYlTHkK18iRLfalse2024-04-19T23:36:19.760Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjudeGFtd00C8TRIOlco59Eefd7c7ffA6false2024-04-19T23:36:19.831Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4false2024-04-19T23:36:19.744Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgoQj3EN8LOJNZ9C2JDigvTcIgypIbDxfalse2024-04-19T23:36:19.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzyfalse2024-04-19T23:36:19.727Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.169NmmrUry8iq70c7.ZNdp4i5iqOCFfalse2024-04-19T23:36:19.659Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE5taibPWiUs3cuEimynrjQD8cXTCAELfalse2024-04-19T23:36:19.589Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNjFtS4LHZJDkGsaKejPMQyqPrZjbuN2false2024-04-19T23:36:19.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmTjHqLZC66BQsk7jiah7FJSChQvGdxfalse2024-04-19T23:36:19.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlKfLYYBA1E.lSLR49AMlK-Kgik5ij5Rfalse2024-04-19T23:36:19.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzwsSl6whrs-j2uo2xvsRdnhCfJSQ4Mafalse2024-04-19T23:36:19.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.325 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:39:16.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:16.329 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.329 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.329 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:16.329 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:16.330 DEBUG:boto:Method: GET 2024-04-19T23:39:16.330 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:39:16.330 DEBUG:boto:Data: 2024-04-19T23:39:16.330 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.330 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.330 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.330 DEBUG:boto:Params: {} 2024-04-19T23:39:16.330 DEBUG:boto:Token: None 2024-04-19T23:39:16.330 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/?versions 2024-04-19T23:39:16.331 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q5Q8TARWLj/nEM0ljgSzul5tzDc= 2024-04-19T23:39:16.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q5Q8TARWLj/nEM0ljgSzul5tzDc=', 'Content-Length': '0'} 2024-04-19T23:39:16.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adc93c75b61e59e9-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.334 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOotrue2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjI8Qos7O61Drj6zKFSuRQg5BJ8qTn1Itfalse2024-04-19T23:36:20.273Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITUfalse2024-04-19T23:36:20.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUjbMVzWXMoq8kQHxc5fXZiC69vNXkcjfalse2024-04-19T23:36:20.222Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97false2024-04-19T23:36:20.188Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2false2024-04-19T23:36:20.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Jy1wSwswautcJ.pAM0qXnm5CTgNceOfalse2024-04-19T23:36:20.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-false2024-04-19T23:36:20.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7false2024-04-19T23:36:20.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQ1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgRfalse2024-04-19T23:36:20.019Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipPC6NaG1ejWpptAGf0zgZ-gqpdm0DMfalse2024-04-19T23:36:20.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoECNSXdMldM0FoTJgSxtKpnhGWUHc5tfalse2024-04-19T23:36:20.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFo79-FV5lzRfFhYZKDH5zbjzjFRvFmofalse2024-04-19T23:36:19.985Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.9ZzH.1hozF9j0blCcYN54OsXVmWMTfalse2024-04-19T23:36:19.968Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnLS7759d.h0bWJMnVj7IAdVTVbZ2LzAfalse2024-04-19T23:36:20.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.SsB-LJAFCELIGc-2lad2-2jTKdmQIufalse2024-04-19T23:36:19.904Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5IMca-NQG3TDmPnhaU0a46jfFv2nxR8false2024-04-19T23:36:19.920Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3PIDP6j1VZfmBljRrKUGkIUmq4DPgifalse2024-04-19T23:36:19.849Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.sqcIpylWkdZwQp4UqpDBbyySodTBXSfalse2024-04-19T23:36:19.778Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCzz.Nb3nU2CWGe2vMCDYlTHkK18iRLfalse2024-04-19T23:36:19.760Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjudeGFtd00C8TRIOlco59Eefd7c7ffA6false2024-04-19T23:36:19.831Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4false2024-04-19T23:36:19.744Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgoQj3EN8LOJNZ9C2JDigvTcIgypIbDxfalse2024-04-19T23:36:19.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzyfalse2024-04-19T23:36:19.727Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.169NmmrUry8iq70c7.ZNdp4i5iqOCFfalse2024-04-19T23:36:19.659Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE5taibPWiUs3cuEimynrjQD8cXTCAELfalse2024-04-19T23:36:19.589Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNjFtS4LHZJDkGsaKejPMQyqPrZjbuN2false2024-04-19T23:36:19.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmTjHqLZC66BQsk7jiah7FJSChQvGdxfalse2024-04-19T23:36:19.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlKfLYYBA1E.lSLR49AMlK-Kgik5ij5Rfalse2024-04-19T23:36:19.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzwsSl6whrs-j2uo2xvsRdnhCfJSQ4Mafalse2024-04-19T23:36:19.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.336 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.336 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.336 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:16.336 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:16.336 DEBUG:boto:Method: GET 2024-04-19T23:39:16.336 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:39:16.337 DEBUG:boto:Data: 2024-04-19T23:39:16.337 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.337 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.337 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.337 DEBUG:boto:Params: {} 2024-04-19T23:39:16.337 DEBUG:boto:Token: None 2024-04-19T23:39:16.337 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/?versions 2024-04-19T23:39:16.337 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q5Q8TARWLj/nEM0ljgSzul5tzDc= 2024-04-19T23:39:16.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q5Q8TARWLj/nEM0ljgSzul5tzDc=', 'Content-Length': '0'} 2024-04-19T23:39:16.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b91dcc3d42234d59-00662300a4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.341 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOotrue2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjI8Qos7O61Drj6zKFSuRQg5BJ8qTn1Itfalse2024-04-19T23:36:20.273Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITUfalse2024-04-19T23:36:20.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUjbMVzWXMoq8kQHxc5fXZiC69vNXkcjfalse2024-04-19T23:36:20.222Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97false2024-04-19T23:36:20.188Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2false2024-04-19T23:36:20.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Jy1wSwswautcJ.pAM0qXnm5CTgNceOfalse2024-04-19T23:36:20.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-false2024-04-19T23:36:20.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7false2024-04-19T23:36:20.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQ1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgRfalse2024-04-19T23:36:20.019Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipPC6NaG1ejWpptAGf0zgZ-gqpdm0DMfalse2024-04-19T23:36:20.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoECNSXdMldM0FoTJgSxtKpnhGWUHc5tfalse2024-04-19T23:36:20.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFo79-FV5lzRfFhYZKDH5zbjzjFRvFmofalse2024-04-19T23:36:19.985Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.9ZzH.1hozF9j0blCcYN54OsXVmWMTfalse2024-04-19T23:36:19.968Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnLS7759d.h0bWJMnVj7IAdVTVbZ2LzAfalse2024-04-19T23:36:20.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.SsB-LJAFCELIGc-2lad2-2jTKdmQIufalse2024-04-19T23:36:19.904Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5IMca-NQG3TDmPnhaU0a46jfFv2nxR8false2024-04-19T23:36:19.920Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3PIDP6j1VZfmBljRrKUGkIUmq4DPgifalse2024-04-19T23:36:19.849Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.sqcIpylWkdZwQp4UqpDBbyySodTBXSfalse2024-04-19T23:36:19.778Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCzz.Nb3nU2CWGe2vMCDYlTHkK18iRLfalse2024-04-19T23:36:19.760Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjudeGFtd00C8TRIOlco59Eefd7c7ffA6false2024-04-19T23:36:19.831Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4false2024-04-19T23:36:19.744Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgoQj3EN8LOJNZ9C2JDigvTcIgypIbDxfalse2024-04-19T23:36:19.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzyfalse2024-04-19T23:36:19.727Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.169NmmrUry8iq70c7.ZNdp4i5iqOCFfalse2024-04-19T23:36:19.659Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE5taibPWiUs3cuEimynrjQD8cXTCAELfalse2024-04-19T23:36:19.589Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNjFtS4LHZJDkGsaKejPMQyqPrZjbuN2false2024-04-19T23:36:19.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmTjHqLZC66BQsk7jiah7FJSChQvGdxfalse2024-04-19T23:36:19.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlKfLYYBA1E.lSLR49AMlK-Kgik5ij5Rfalse2024-04-19T23:36:19.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzwsSl6whrs-j2uo2xvsRdnhCfJSQ4Mafalse2024-04-19T23:36:19.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:39:16.343 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:16.343 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:16.343 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.343 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.343 DEBUG:boto:Method: GET 2024-04-19T23:39:16.343 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.343 DEBUG:boto:Data: 2024-04-19T23:39:16.343 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.343 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.344 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.344 DEBUG:boto:Params: {} 2024-04-19T23:39:16.344 DEBUG:boto:Token: None 2024-04-19T23:39:16.344 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wf429kHseXTPEa+OudPrbavz3L8= 2024-04-19T23:39:16.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wf429kHseXTPEa+OudPrbavz3L8=', 'Content-Length': '0'} 2024-04-19T23:39:16.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2a41bc0bda6df48-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.347 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:16.347 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:16.347 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.347 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.347 DEBUG:boto:Method: GET 2024-04-19T23:39:16.347 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.347 DEBUG:boto:Data: 2024-04-19T23:39:16.347 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.347 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.348 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.348 DEBUG:boto:Params: {} 2024-04-19T23:39:16.348 DEBUG:boto:Token: None 2024-04-19T23:39:16.348 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.348 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wf429kHseXTPEa+OudPrbavz3L8= 2024-04-19T23:39:16.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wf429kHseXTPEa+OudPrbavz3L8=', 'Content-Length': '0'} 2024-04-19T23:39:16.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d960aa8d4d3b24dd-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.351 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:16.351 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:16.351 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.351 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.351 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.351 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.351 DEBUG:boto:Data: 2024-04-19T23:39:16.351 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.352 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.352 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.352 DEBUG:boto:Params: {} 2024-04-19T23:39:16.352 DEBUG:boto:Token: None 2024-04-19T23:39:16.352 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.352 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:puWRz4xq/G2ScOyr7ZnrAcgUXeY= 2024-04-19T23:39:16.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:puWRz4xq/G2ScOyr7ZnrAcgUXeY=', 'Content-Length': '0'} 2024-04-19T23:39:16.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004350e7625bcb53f8-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.354 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:16.354 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:16.354 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.355 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.355 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.355 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.355 DEBUG:boto:Data: 2024-04-19T23:39:16.355 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.355 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.355 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.355 DEBUG:boto:Params: {} 2024-04-19T23:39:16.355 DEBUG:boto:Token: None 2024-04-19T23:39:16.355 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.356 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:puWRz4xq/G2ScOyr7ZnrAcgUXeY= 2024-04-19T23:39:16.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:puWRz4xq/G2ScOyr7ZnrAcgUXeY=', 'Content-Length': '0'} 2024-04-19T23:39:16.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b32b6eed0458ba2-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:39:16.358 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:16.358 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:16.358 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.358 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.358 DEBUG:boto:Method: GET 2024-04-19T23:39:16.359 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.359 DEBUG:boto:Data: 2024-04-19T23:39:16.359 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.359 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.359 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.359 DEBUG:boto:Params: {} 2024-04-19T23:39:16.359 DEBUG:boto:Token: None 2024-04-19T23:39:16.359 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.359 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wf429kHseXTPEa+OudPrbavz3L8= 2024-04-19T23:39:16.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wf429kHseXTPEa+OudPrbavz3L8=', 'Content-Length': '0'} 2024-04-19T23:39:16.361 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001d942f0560e5c814-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.362 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:16.362 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:16.362 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.362 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.362 DEBUG:boto:Method: GET 2024-04-19T23:39:16.362 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.363 DEBUG:boto:Data: 2024-04-19T23:39:16.363 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.363 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.363 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.363 DEBUG:boto:Params: {} 2024-04-19T23:39:16.363 DEBUG:boto:Token: None 2024-04-19T23:39:16.363 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.363 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wf429kHseXTPEa+OudPrbavz3L8= 2024-04-19T23:39:16.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wf429kHseXTPEa+OudPrbavz3L8=', 'Content-Length': '0'} 2024-04-19T23:39:16.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009441ab2af594df0b-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.365 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.366 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.366 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:16.366 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:16.366 DEBUG:boto:Method: GET 2024-04-19T23:39:16.366 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:39:16.366 DEBUG:boto:Data: 2024-04-19T23:39:16.366 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.366 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.366 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.366 DEBUG:boto:Params: {} 2024-04-19T23:39:16.366 DEBUG:boto:Token: None 2024-04-19T23:39:16.367 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.367 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000299d260971e4893a-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.410 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.411 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.411 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.411 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:16.411 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:16.411 DEBUG:boto:Method: GET 2024-04-19T23:39:16.411 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:39:16.411 DEBUG:boto:Data: 2024-04-19T23:39:16.412 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.412 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.412 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.412 DEBUG:boto:Params: {} 2024-04-19T23:39:16.412 DEBUG:boto:Token: None 2024-04-19T23:39:16.412 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.412 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009352512a35569871-00662300a4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.415 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.416 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:16.416 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:16.416 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.416 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:39:16.416 DEBUG:boto:Data: 2024-04-19T23:39:16.416 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.416 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.417 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.417 DEBUG:boto:Params: {} 2024-04-19T23:39:16.417 DEBUG:boto:Token: None 2024-04-19T23:39:16.417 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/dummy 2024-04-19T23:39:16.417 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0jVS/q9jm75/tGqbnzcN3HaecBY= 2024-04-19T23:39:16.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0jVS/q9jm75/tGqbnzcN3HaecBY=', 'Content-Length': '0'} 2024-04-19T23:39:16.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b35a25607931f48d-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.420 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:16.420 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:16.420 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.420 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:39:16.420 DEBUG:boto:Data: 2024-04-19T23:39:16.420 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.420 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.420 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.421 DEBUG:boto:Params: {} 2024-04-19T23:39:16.421 DEBUG:boto:Token: None 2024-04-19T23:39:16.421 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/dummy 2024-04-19T23:39:16.421 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0jVS/q9jm75/tGqbnzcN3HaecBY= 2024-04-19T23:39:16.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0jVS/q9jm75/tGqbnzcN3HaecBY=', 'Content-Length': '0'} 2024-04-19T23:39:16.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf6f53854dee46a4-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:39:16.424 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:16.424 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:16.424 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.424 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.424 DEBUG:boto:Method: GET 2024-04-19T23:39:16.424 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.424 DEBUG:boto:Data: 2024-04-19T23:39:16.424 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.425 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.425 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.425 DEBUG:boto:Params: {} 2024-04-19T23:39:16.425 DEBUG:boto:Token: None 2024-04-19T23:39:16.425 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.425 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wf429kHseXTPEa+OudPrbavz3L8= 2024-04-19T23:39:16.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wf429kHseXTPEa+OudPrbavz3L8=', 'Content-Length': '0'} 2024-04-19T23:39:16.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b2af5987618b728-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.428 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:16.428 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:16.428 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.428 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.428 DEBUG:boto:Method: GET 2024-04-19T23:39:16.428 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.428 DEBUG:boto:Data: 2024-04-19T23:39:16.428 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.428 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.428 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.429 DEBUG:boto:Params: {} 2024-04-19T23:39:16.429 DEBUG:boto:Token: None 2024-04-19T23:39:16.429 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:16.429 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wf429kHseXTPEa+OudPrbavz3L8= 2024-04-19T23:39:16.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wf429kHseXTPEa+OudPrbavz3L8=', 'Content-Length': '0'} 2024-04-19T23:39:16.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000298532ffa99a2a3d-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.431 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.432 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.432 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:16.432 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:16.432 DEBUG:boto:Method: GET 2024-04-19T23:39:16.432 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:39:16.432 DEBUG:boto:Data: 2024-04-19T23:39:16.433 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.433 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.433 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.433 DEBUG:boto:Params: {} 2024-04-19T23:39:16.433 DEBUG:boto:Token: None 2024-04-19T23:39:16.433 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.433 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001876b0854ea3b1d5-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.478 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.479 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.479 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.479 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:16.480 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:16.480 DEBUG:boto:Method: GET 2024-04-19T23:39:16.480 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:39:16.480 DEBUG:boto:Data: 2024-04-19T23:39:16.480 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.480 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.480 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.480 DEBUG:boto:Params: {} 2024-04-19T23:39:16.480 DEBUG:boto:Token: None 2024-04-19T23:39:16.481 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.481 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ae026add89c6b0-00662300a4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.484 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:16.484 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.484 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.485 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.485 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.485 DEBUG:boto:Method: GET 2024-04-19T23:39:16.485 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.485 DEBUG:boto:Data: 2024-04-19T23:39:16.485 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.485 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.485 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.485 DEBUG:boto:Params: {} 2024-04-19T23:39:16.485 DEBUG:boto:Token: None 2024-04-19T23:39:16.486 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.486 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA= 2024-04-19T23:39:16.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA=', 'Content-Length': '0'} 2024-04-19T23:39:16.488 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e205a3723869639-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.488 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.488 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.488 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.488 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.489 DEBUG:boto:Method: GET 2024-04-19T23:39:16.489 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.489 DEBUG:boto:Data: 2024-04-19T23:39:16.489 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.489 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.489 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.489 DEBUG:boto:Params: {} 2024-04-19T23:39:16.489 DEBUG:boto:Token: None 2024-04-19T23:39:16.489 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.490 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA= 2024-04-19T23:39:16.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA=', 'Content-Length': '0'} 2024-04-19T23:39:16.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002cde130c1bf156c9-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.538 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.538 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.539 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.539 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.539 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.539 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.539 DEBUG:boto:Data: 2024-04-19T23:39:16.539 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.539 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.539 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.539 DEBUG:boto:Params: {} 2024-04-19T23:39:16.540 DEBUG:boto:Token: None 2024-04-19T23:39:16.540 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.540 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wDa1IUaBvQGQYqFK6dRzMiMgRwQ= 2024-04-19T23:39:16.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wDa1IUaBvQGQYqFK6dRzMiMgRwQ=', 'Content-Length': '0'} 2024-04-19T23:39:16.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a460b44c0039b7c4-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.542 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.542 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.542 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.542 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.542 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.543 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.543 DEBUG:boto:Data: 2024-04-19T23:39:16.543 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.543 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.543 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.543 DEBUG:boto:Params: {} 2024-04-19T23:39:16.543 DEBUG:boto:Token: None 2024-04-19T23:39:16.543 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.543 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wDa1IUaBvQGQYqFK6dRzMiMgRwQ= 2024-04-19T23:39:16.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wDa1IUaBvQGQYqFK6dRzMiMgRwQ=', 'Content-Length': '0'} 2024-04-19T23:39:16.545 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065b5ddff772d9aed-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:16.546 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.546 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.546 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.546 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.546 DEBUG:boto:Method: GET 2024-04-19T23:39:16.546 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.546 DEBUG:boto:Data: 2024-04-19T23:39:16.546 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.547 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.547 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.547 DEBUG:boto:Params: {} 2024-04-19T23:39:16.547 DEBUG:boto:Token: None 2024-04-19T23:39:16.547 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.547 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA= 2024-04-19T23:39:16.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA=', 'Content-Length': '0'} 2024-04-19T23:39:16.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006e68e94f99bbecc-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.549 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.549 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.549 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.549 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.550 DEBUG:boto:Method: GET 2024-04-19T23:39:16.550 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.550 DEBUG:boto:Data: 2024-04-19T23:39:16.550 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.550 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.550 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.550 DEBUG:boto:Params: {} 2024-04-19T23:39:16.550 DEBUG:boto:Token: None 2024-04-19T23:39:16.550 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.550 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA= 2024-04-19T23:39:16.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA=', 'Content-Length': '0'} 2024-04-19T23:39:16.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d1bda5eaacd2290-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.594 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.595 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.595 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.595 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.595 DEBUG:boto:Method: GET 2024-04-19T23:39:16.595 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.595 DEBUG:boto:Data: 2024-04-19T23:39:16.595 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.595 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.595 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.595 DEBUG:boto:Params: {} 2024-04-19T23:39:16.596 DEBUG:boto:Token: None 2024-04-19T23:39:16.596 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.596 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b7fffd572ec9f42-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.599 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.600 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.600 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.600 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.600 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.600 DEBUG:boto:Method: GET 2024-04-19T23:39:16.600 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.600 DEBUG:boto:Data: 2024-04-19T23:39:16.600 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.600 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.600 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.600 DEBUG:boto:Params: {} 2024-04-19T23:39:16.601 DEBUG:boto:Token: None 2024-04-19T23:39:16.601 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.601 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034e4eabbee4ac7e5-00662300a4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.604 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.604 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.604 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.604 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.604 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:16.604 DEBUG:boto:Data: 2024-04-19T23:39:16.604 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.604 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.604 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.605 DEBUG:boto:Params: {} 2024-04-19T23:39:16.605 DEBUG:boto:Token: None 2024-04-19T23:39:16.605 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj 2024-04-19T23:39:16.605 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:e2LQ17QK6ynbliC9gSyGpdKCBYs= 2024-04-19T23:39:16.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:e2LQ17QK6ynbliC9gSyGpdKCBYs=', 'Content-Length': '0'} 2024-04-19T23:39:16.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a78c6dd14ad41be-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.609 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.609 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.609 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.609 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:16.609 DEBUG:boto:Data: 2024-04-19T23:39:16.609 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.609 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.609 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.609 DEBUG:boto:Params: {} 2024-04-19T23:39:16.610 DEBUG:boto:Token: None 2024-04-19T23:39:16.610 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj 2024-04-19T23:39:16.610 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:e2LQ17QK6ynbliC9gSyGpdKCBYs= 2024-04-19T23:39:16.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:e2LQ17QK6ynbliC9gSyGpdKCBYs=', 'Content-Length': '0'} 2024-04-19T23:39:16.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010376c238c5bbf0b-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:16.613 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.614 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.614 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.614 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.614 DEBUG:boto:Method: GET 2024-04-19T23:39:16.614 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.614 DEBUG:boto:Data: 2024-04-19T23:39:16.614 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.614 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.614 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.614 DEBUG:boto:Params: {} 2024-04-19T23:39:16.614 DEBUG:boto:Token: None 2024-04-19T23:39:16.615 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.615 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA= 2024-04-19T23:39:16.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA=', 'Content-Length': '0'} 2024-04-19T23:39:16.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054a13192f2f7eae5-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.658 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.658 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.659 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.659 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.659 DEBUG:boto:Method: GET 2024-04-19T23:39:16.659 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.659 DEBUG:boto:Data: 2024-04-19T23:39:16.659 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.659 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.659 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.659 DEBUG:boto:Params: {} 2024-04-19T23:39:16.659 DEBUG:boto:Token: None 2024-04-19T23:39:16.660 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA= 2024-04-19T23:39:16.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA=', 'Content-Length': '0'} 2024-04-19T23:39:16.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073c777b189ea2ccf-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.663 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.663 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.663 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.663 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.664 DEBUG:boto:Method: GET 2024-04-19T23:39:16.664 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.664 DEBUG:boto:Data: 2024-04-19T23:39:16.664 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.664 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.664 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.664 DEBUG:boto:Params: {} 2024-04-19T23:39:16.664 DEBUG:boto:Token: None 2024-04-19T23:39:16.664 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.664 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081d3bec661d6c7f1-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.669 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.670 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.670 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.670 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.670 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.670 DEBUG:boto:Method: GET 2024-04-19T23:39:16.670 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.670 DEBUG:boto:Data: 2024-04-19T23:39:16.671 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.671 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.671 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.671 DEBUG:boto:Params: {} 2024-04-19T23:39:16.671 DEBUG:boto:Token: None 2024-04-19T23:39:16.671 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.671 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a496dd7a46f42947-00662300a4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.676 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:16.676 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.676 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.676 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.677 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.677 DEBUG:boto:Method: GET 2024-04-19T23:39:16.677 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.677 DEBUG:boto:Data: 2024-04-19T23:39:16.677 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.677 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.677 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.677 DEBUG:boto:Params: {} 2024-04-19T23:39:16.678 DEBUG:boto:Token: None 2024-04-19T23:39:16.678 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.678 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z47Wf7cG4qmLc5hWoX2YK2nN+U4= 2024-04-19T23:39:16.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z47Wf7cG4qmLc5hWoX2YK2nN+U4=', 'Content-Length': '0'} 2024-04-19T23:39:16.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095d2bab12474ef62-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.680 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.681 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.681 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.681 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.681 DEBUG:boto:Method: GET 2024-04-19T23:39:16.681 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.681 DEBUG:boto:Data: 2024-04-19T23:39:16.681 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.681 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.682 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.682 DEBUG:boto:Params: {} 2024-04-19T23:39:16.682 DEBUG:boto:Token: None 2024-04-19T23:39:16.682 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.682 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z47Wf7cG4qmLc5hWoX2YK2nN+U4= 2024-04-19T23:39:16.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z47Wf7cG4qmLc5hWoX2YK2nN+U4=', 'Content-Length': '0'} 2024-04-19T23:39:16.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077376e5fd382ca47-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.685 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.685 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.685 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.685 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.685 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.686 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.686 DEBUG:boto:Data: 2024-04-19T23:39:16.686 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.686 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.686 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.686 DEBUG:boto:Params: {} 2024-04-19T23:39:16.686 DEBUG:boto:Token: None 2024-04-19T23:39:16.686 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.686 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gHWpjs6TCovLgPuvyyEBs6Kepog= 2024-04-19T23:39:16.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gHWpjs6TCovLgPuvyyEBs6Kepog=', 'Content-Length': '0'} 2024-04-19T23:39:16.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000053aecd56a39bfaef-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.688 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.689 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.689 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.689 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.689 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.689 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.689 DEBUG:boto:Data: 2024-04-19T23:39:16.689 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.689 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.689 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.689 DEBUG:boto:Params: {} 2024-04-19T23:39:16.690 DEBUG:boto:Token: None 2024-04-19T23:39:16.690 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.690 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gHWpjs6TCovLgPuvyyEBs6Kepog= 2024-04-19T23:39:16.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gHWpjs6TCovLgPuvyyEBs6Kepog=', 'Content-Length': '0'} 2024-04-19T23:39:16.691 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048515a4e88bb96f7-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:16.692 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.692 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.692 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.692 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.692 DEBUG:boto:Method: GET 2024-04-19T23:39:16.692 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.692 DEBUG:boto:Data: 2024-04-19T23:39:16.692 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.693 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.693 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.693 DEBUG:boto:Params: {} 2024-04-19T23:39:16.693 DEBUG:boto:Token: None 2024-04-19T23:39:16.693 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.693 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z47Wf7cG4qmLc5hWoX2YK2nN+U4= 2024-04-19T23:39:16.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z47Wf7cG4qmLc5hWoX2YK2nN+U4=', 'Content-Length': '0'} 2024-04-19T23:39:16.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5fe1e1844de5683-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.697 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.697 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.697 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.697 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.698 DEBUG:boto:Method: GET 2024-04-19T23:39:16.698 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.698 DEBUG:boto:Data: 2024-04-19T23:39:16.698 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.698 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.698 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.698 DEBUG:boto:Params: {} 2024-04-19T23:39:16.698 DEBUG:boto:Token: None 2024-04-19T23:39:16.698 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:16.698 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z47Wf7cG4qmLc5hWoX2YK2nN+U4= 2024-04-19T23:39:16.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z47Wf7cG4qmLc5hWoX2YK2nN+U4=', 'Content-Length': '0'} 2024-04-19T23:39:16.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031554956672f2c67-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.701 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.702 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.702 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.702 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.702 DEBUG:boto:Method: GET 2024-04-19T23:39:16.702 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.702 DEBUG:boto:Data: 2024-04-19T23:39:16.702 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.702 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.702 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.702 DEBUG:boto:Params: {} 2024-04-19T23:39:16.703 DEBUG:boto:Token: None 2024-04-19T23:39:16.703 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.703 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0447bb25f1a8908-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.754 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.755 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.755 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.755 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.755 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.755 DEBUG:boto:Method: GET 2024-04-19T23:39:16.755 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.755 DEBUG:boto:Data: 2024-04-19T23:39:16.755 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.755 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.756 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.756 DEBUG:boto:Params: {} 2024-04-19T23:39:16.756 DEBUG:boto:Token: None 2024-04-19T23:39:16.756 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.756 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d307a6de6a9d80c4-00662300a4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.759 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.760 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.760 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.760 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.760 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:16.760 DEBUG:boto:Data: 2024-04-19T23:39:16.760 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.760 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.760 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.760 DEBUG:boto:Params: {} 2024-04-19T23:39:16.760 DEBUG:boto:Token: None 2024-04-19T23:39:16.760 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj 2024-04-19T23:39:16.761 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:e2LQ17QK6ynbliC9gSyGpdKCBYs= 2024-04-19T23:39:16.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:e2LQ17QK6ynbliC9gSyGpdKCBYs=', 'Content-Length': '0'} 2024-04-19T23:39:16.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000f8da611faf05af-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.764 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.764 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.764 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.764 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:16.764 DEBUG:boto:Data: 2024-04-19T23:39:16.764 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.764 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.764 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.765 DEBUG:boto:Params: {} 2024-04-19T23:39:16.765 DEBUG:boto:Token: None 2024-04-19T23:39:16.765 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj 2024-04-19T23:39:16.765 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:e2LQ17QK6ynbliC9gSyGpdKCBYs= 2024-04-19T23:39:16.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:e2LQ17QK6ynbliC9gSyGpdKCBYs=', 'Content-Length': '0'} 2024-04-19T23:39:16.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a01cfa4d9c78b453-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:16.769 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.769 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.769 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.769 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.769 DEBUG:boto:Method: GET 2024-04-19T23:39:16.769 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.769 DEBUG:boto:Data: 2024-04-19T23:39:16.769 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.769 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.769 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.769 DEBUG:boto:Params: {} 2024-04-19T23:39:16.769 DEBUG:boto:Token: None 2024-04-19T23:39:16.770 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.770 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA= 2024-04-19T23:39:16.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA=', 'Content-Length': '0'} 2024-04-19T23:39:16.772 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c877a2152aa3a29-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.772 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.772 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.772 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.772 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.772 DEBUG:boto:Method: GET 2024-04-19T23:39:16.772 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.773 DEBUG:boto:Data: 2024-04-19T23:39:16.773 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.773 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.773 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.773 DEBUG:boto:Params: {} 2024-04-19T23:39:16.773 DEBUG:boto:Token: None 2024-04-19T23:39:16.773 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.773 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA= 2024-04-19T23:39:16.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA=', 'Content-Length': '0'} 2024-04-19T23:39:16.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8206d15f1352820-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.818 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.818 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.818 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.819 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.819 DEBUG:boto:Method: GET 2024-04-19T23:39:16.819 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.819 DEBUG:boto:Data: 2024-04-19T23:39:16.819 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.819 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.819 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.819 DEBUG:boto:Params: {} 2024-04-19T23:39:16.819 DEBUG:boto:Token: None 2024-04-19T23:39:16.819 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.820 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000602b85d512125975-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.822 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.823 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.823 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.823 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.823 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.823 DEBUG:boto:Method: GET 2024-04-19T23:39:16.823 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.823 DEBUG:boto:Data: 2024-04-19T23:39:16.823 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.823 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.824 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.824 DEBUG:boto:Params: {} 2024-04-19T23:39:16.824 DEBUG:boto:Token: None 2024-04-19T23:39:16.824 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a25ac89d62eda5c-00662300a4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.827 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:16.827 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.828 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.828 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.828 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.828 DEBUG:boto:Method: GET 2024-04-19T23:39:16.828 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.828 DEBUG:boto:Data: 2024-04-19T23:39:16.828 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.828 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.828 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.828 DEBUG:boto:Params: {} 2024-04-19T23:39:16.828 DEBUG:boto:Token: None 2024-04-19T23:39:16.829 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.829 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GBfFrTirDPcE3yT+a5KcWDOlLpI= 2024-04-19T23:39:16.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GBfFrTirDPcE3yT+a5KcWDOlLpI=', 'Content-Length': '0'} 2024-04-19T23:39:16.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c56ba9f1292f5eb-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.874 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.875 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.875 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.875 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.875 DEBUG:boto:Method: GET 2024-04-19T23:39:16.875 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.875 DEBUG:boto:Data: 2024-04-19T23:39:16.875 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.875 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.875 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.875 DEBUG:boto:Params: {} 2024-04-19T23:39:16.876 DEBUG:boto:Token: None 2024-04-19T23:39:16.876 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.876 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GBfFrTirDPcE3yT+a5KcWDOlLpI= 2024-04-19T23:39:16.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GBfFrTirDPcE3yT+a5KcWDOlLpI=', 'Content-Length': '0'} 2024-04-19T23:39:16.878 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e78fe77f1312a75e-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.878 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.878 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.878 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.879 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.879 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.879 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.879 DEBUG:boto:Data: 2024-04-19T23:39:16.879 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.879 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.879 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.879 DEBUG:boto:Params: {} 2024-04-19T23:39:16.879 DEBUG:boto:Token: None 2024-04-19T23:39:16.879 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mU5BEHWQ7PRZl5Zho2NaRISPVkg= 2024-04-19T23:39:16.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mU5BEHWQ7PRZl5Zho2NaRISPVkg=', 'Content-Length': '0'} 2024-04-19T23:39:16.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000683c41a2f7031547-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.882 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.882 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.882 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.882 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.882 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.882 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.882 DEBUG:boto:Data: 2024-04-19T23:39:16.882 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.883 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.883 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.883 DEBUG:boto:Params: {} 2024-04-19T23:39:16.883 DEBUG:boto:Token: None 2024-04-19T23:39:16.883 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.883 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mU5BEHWQ7PRZl5Zho2NaRISPVkg= 2024-04-19T23:39:16.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mU5BEHWQ7PRZl5Zho2NaRISPVkg=', 'Content-Length': '0'} 2024-04-19T23:39:16.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083535a639d5b5d82-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:16.885 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.885 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.885 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.885 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.885 DEBUG:boto:Method: GET 2024-04-19T23:39:16.886 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.886 DEBUG:boto:Data: 2024-04-19T23:39:16.886 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.886 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.886 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.886 DEBUG:boto:Params: {} 2024-04-19T23:39:16.886 DEBUG:boto:Token: None 2024-04-19T23:39:16.887 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.887 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GBfFrTirDPcE3yT+a5KcWDOlLpI= 2024-04-19T23:39:16.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GBfFrTirDPcE3yT+a5KcWDOlLpI=', 'Content-Length': '0'} 2024-04-19T23:39:16.888 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc9cda47c90dc610-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.889 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.889 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.889 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.889 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.889 DEBUG:boto:Method: GET 2024-04-19T23:39:16.889 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.889 DEBUG:boto:Data: 2024-04-19T23:39:16.889 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.890 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.890 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.890 DEBUG:boto:Params: {} 2024-04-19T23:39:16.890 DEBUG:boto:Token: None 2024-04-19T23:39:16.890 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:16.890 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GBfFrTirDPcE3yT+a5KcWDOlLpI= 2024-04-19T23:39:16.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GBfFrTirDPcE3yT+a5KcWDOlLpI=', 'Content-Length': '0'} 2024-04-19T23:39:16.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb3fb1ee1879fc3c-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.892 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.892 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.893 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.893 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.893 DEBUG:boto:Method: GET 2024-04-19T23:39:16.893 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.893 DEBUG:boto:Data: 2024-04-19T23:39:16.893 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.894 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.894 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.894 DEBUG:boto:Params: {} 2024-04-19T23:39:16.894 DEBUG:boto:Token: None 2024-04-19T23:39:16.894 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.894 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009015574e11da85e1-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.897 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.898 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.898 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.898 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.898 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.899 DEBUG:boto:Method: GET 2024-04-19T23:39:16.899 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.899 DEBUG:boto:Data: 2024-04-19T23:39:16.899 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.899 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.899 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.899 DEBUG:boto:Params: {} 2024-04-19T23:39:16.900 DEBUG:boto:Token: None 2024-04-19T23:39:16.900 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.900 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ad46c2554c0cd49-00662300a4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.903 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.904 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.904 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.904 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.904 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:16.904 DEBUG:boto:Data: 2024-04-19T23:39:16.904 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.904 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.904 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.905 DEBUG:boto:Params: {} 2024-04-19T23:39:16.905 DEBUG:boto:Token: None 2024-04-19T23:39:16.905 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj 2024-04-19T23:39:16.905 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:e2LQ17QK6ynbliC9gSyGpdKCBYs= 2024-04-19T23:39:16.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:e2LQ17QK6ynbliC9gSyGpdKCBYs=', 'Content-Length': '0'} 2024-04-19T23:39:16.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b1e405cf1aee168-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.909 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.909 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.909 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.909 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:16.909 DEBUG:boto:Data: 2024-04-19T23:39:16.909 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.909 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.909 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.909 DEBUG:boto:Params: {} 2024-04-19T23:39:16.909 DEBUG:boto:Token: None 2024-04-19T23:39:16.910 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj 2024-04-19T23:39:16.910 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:e2LQ17QK6ynbliC9gSyGpdKCBYs= 2024-04-19T23:39:16.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:e2LQ17QK6ynbliC9gSyGpdKCBYs=', 'Content-Length': '0'} 2024-04-19T23:39:16.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004415046c6f57d7a5-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:16.914 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.914 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.915 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.915 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.915 DEBUG:boto:Method: GET 2024-04-19T23:39:16.915 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.915 DEBUG:boto:Data: 2024-04-19T23:39:16.915 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.915 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.915 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.915 DEBUG:boto:Params: {} 2024-04-19T23:39:16.915 DEBUG:boto:Token: None 2024-04-19T23:39:16.915 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.916 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA= 2024-04-19T23:39:16.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA=', 'Content-Length': '0'} 2024-04-19T23:39:16.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ceeec9620216c113-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.918 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.918 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.918 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.918 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.919 DEBUG:boto:Method: GET 2024-04-19T23:39:16.919 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.919 DEBUG:boto:Data: 2024-04-19T23:39:16.919 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.919 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.919 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.919 DEBUG:boto:Params: {} 2024-04-19T23:39:16.920 DEBUG:boto:Token: None 2024-04-19T23:39:16.920 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:16.920 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA= 2024-04-19T23:39:16.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xFbxnT/PYO0J+lm9mFNOZvBVhRA=', 'Content-Length': '0'} 2024-04-19T23:39:16.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002587890fb45136c5-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.923 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.923 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.923 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.923 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.923 DEBUG:boto:Method: GET 2024-04-19T23:39:16.923 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.924 DEBUG:boto:Data: 2024-04-19T23:39:16.924 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.924 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.924 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.924 DEBUG:boto:Params: {} 2024-04-19T23:39:16.924 DEBUG:boto:Token: None 2024-04-19T23:39:16.925 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.925 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cee915899568faf8-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.928 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.929 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.929 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.929 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.929 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.929 DEBUG:boto:Method: GET 2024-04-19T23:39:16.929 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.930 DEBUG:boto:Data: 2024-04-19T23:39:16.930 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.930 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.930 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.930 DEBUG:boto:Params: {} 2024-04-19T23:39:16.930 DEBUG:boto:Token: None 2024-04-19T23:39:16.931 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.931 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:16.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007819ccba6b3a60d8-00662300a4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.934 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:16.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:16.934 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.934 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.935 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.935 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.935 DEBUG:boto:Method: GET 2024-04-19T23:39:16.935 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.935 DEBUG:boto:Data: 2024-04-19T23:39:16.935 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.935 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.935 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.935 DEBUG:boto:Params: {} 2024-04-19T23:39:16.935 DEBUG:boto:Token: None 2024-04-19T23:39:16.935 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.936 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IAn7X2zyxFt7tAjukFN+x3ohOQM= 2024-04-19T23:39:16.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IAn7X2zyxFt7tAjukFN+x3ohOQM=', 'Content-Length': '0'} 2024-04-19T23:39:16.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2554fb9147870c5-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.978 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.978 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.979 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.979 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.979 DEBUG:boto:Method: GET 2024-04-19T23:39:16.979 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.979 DEBUG:boto:Data: 2024-04-19T23:39:16.979 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.979 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.979 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.979 DEBUG:boto:Params: {} 2024-04-19T23:39:16.979 DEBUG:boto:Token: None 2024-04-19T23:39:16.979 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.980 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IAn7X2zyxFt7tAjukFN+x3ohOQM= 2024-04-19T23:39:16.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IAn7X2zyxFt7tAjukFN+x3ohOQM=', 'Content-Length': '0'} 2024-04-19T23:39:16.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026074bc739e366d3-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.982 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.982 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.983 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.983 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.983 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.983 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.983 DEBUG:boto:Data: 2024-04-19T23:39:16.983 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.983 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.983 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.983 DEBUG:boto:Params: {} 2024-04-19T23:39:16.983 DEBUG:boto:Token: None 2024-04-19T23:39:16.983 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X6grqhf1mWttwpiflRIG4HD5z/c= 2024-04-19T23:39:16.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X6grqhf1mWttwpiflRIG4HD5z/c=', 'Content-Length': '0'} 2024-04-19T23:39:16.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe7659031dc65ca9-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.986 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.986 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.986 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.986 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.986 DEBUG:boto:Method: HEAD 2024-04-19T23:39:16.986 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.986 DEBUG:boto:Data: 2024-04-19T23:39:16.986 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.986 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.987 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.987 DEBUG:boto:Params: {} 2024-04-19T23:39:16.987 DEBUG:boto:Token: None 2024-04-19T23:39:16.987 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.987 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X6grqhf1mWttwpiflRIG4HD5z/c= 2024-04-19T23:39:16.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X6grqhf1mWttwpiflRIG4HD5z/c=', 'Content-Length': '0'} 2024-04-19T23:39:16.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4681f9dee49c3e9-00662300a4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:16.990 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.990 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.990 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.990 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.990 DEBUG:boto:Method: GET 2024-04-19T23:39:16.990 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.990 DEBUG:boto:Data: 2024-04-19T23:39:16.990 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.990 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.990 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.991 DEBUG:boto:Params: {} 2024-04-19T23:39:16.991 DEBUG:boto:Token: None 2024-04-19T23:39:16.991 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.991 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IAn7X2zyxFt7tAjukFN+x3ohOQM= 2024-04-19T23:39:16.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IAn7X2zyxFt7tAjukFN+x3ohOQM=', 'Content-Length': '0'} 2024-04-19T23:39:16.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c29f91fee9d8c72f-00662300a4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.993 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:16.994 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:16.994 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.994 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.994 DEBUG:boto:Method: GET 2024-04-19T23:39:16.994 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.994 DEBUG:boto:Data: 2024-04-19T23:39:16.994 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.994 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:16.994 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.994 DEBUG:boto:Params: {} 2024-04-19T23:39:16.995 DEBUG:boto:Token: None 2024-04-19T23:39:16.995 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:16.995 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IAn7X2zyxFt7tAjukFN+x3ohOQM= 2024-04-19T23:39:16.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IAn7X2zyxFt7tAjukFN+x3ohOQM=', 'Content-Length': '0'} 2024-04-19T23:39:16.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c2a15fe4d6c0604-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:16.998 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:16.998 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:16.998 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.998 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:16.998 DEBUG:boto:Method: GET 2024-04-19T23:39:16.998 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:16.998 DEBUG:boto:Data: 2024-04-19T23:39:16.998 DEBUG:boto:Headers: {} 2024-04-19T23:39:16.998 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:16.998 DEBUG:boto:Port: 8000 2024-04-19T23:39:16.999 DEBUG:boto:Params: {} 2024-04-19T23:39:16.999 DEBUG:boto:Token: None 2024-04-19T23:39:16.999 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:16 GMT /aktlji-22/ 2024-04-19T23:39:16.999 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU= 2024-04-19T23:39:16.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kiCLH+L5fMUPxVektoeNxxCjTfU=', 'Content-Length': '0'} 2024-04-19T23:39:17.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e4462dd9e5d4367-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.002 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.002 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.002 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.003 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.003 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.003 DEBUG:boto:Method: GET 2024-04-19T23:39:17.003 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.003 DEBUG:boto:Data: 2024-04-19T23:39:17.003 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.003 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.003 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.003 DEBUG:boto:Params: {} 2024-04-19T23:39:17.003 DEBUG:boto:Token: None 2024-04-19T23:39:17.003 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.004 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f562ae0c6da3be4-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.006 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.007 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.007 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.007 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.007 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.007 DEBUG:boto:Data: 2024-04-19T23:39:17.007 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.007 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.007 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.007 DEBUG:boto:Params: {} 2024-04-19T23:39:17.008 DEBUG:boto:Token: None 2024-04-19T23:39:17.008 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.008 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f55d00c5044ad5c2-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.012 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.012 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.012 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.012 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.012 DEBUG:boto:Data: 2024-04-19T23:39:17.012 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.012 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.012 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.012 DEBUG:boto:Params: {} 2024-04-19T23:39:17.012 DEBUG:boto:Token: None 2024-04-19T23:39:17.013 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.013 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db13b4f2e94e1872-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.017 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.017 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.017 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.017 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.017 DEBUG:boto:Method: GET 2024-04-19T23:39:17.017 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.017 DEBUG:boto:Data: 2024-04-19T23:39:17.017 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.017 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.017 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.018 DEBUG:boto:Params: {} 2024-04-19T23:39:17.018 DEBUG:boto:Token: None 2024-04-19T23:39:17.018 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.018 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c0266b49341102c-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.020 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.020 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.020 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.020 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.021 DEBUG:boto:Method: GET 2024-04-19T23:39:17.021 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.021 DEBUG:boto:Data: 2024-04-19T23:39:17.021 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.021 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.021 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.021 DEBUG:boto:Params: {} 2024-04-19T23:39:17.021 DEBUG:boto:Token: None 2024-04-19T23:39:17.021 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.022 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.023 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f524fcb14a03ad66-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.066 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.066 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.066 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.067 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.067 DEBUG:boto:Method: GET 2024-04-19T23:39:17.067 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.067 DEBUG:boto:Data: 2024-04-19T23:39:17.067 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.067 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.067 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.067 DEBUG:boto:Params: {} 2024-04-19T23:39:17.067 DEBUG:boto:Token: None 2024-04-19T23:39:17.067 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.068 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bce2c9ed26e5b617-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.071 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.071 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.071 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.071 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.071 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.071 DEBUG:boto:Method: GET 2024-04-19T23:39:17.071 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.072 DEBUG:boto:Data: 2024-04-19T23:39:17.072 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.072 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.072 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.072 DEBUG:boto:Params: {} 2024-04-19T23:39:17.072 DEBUG:boto:Token: None 2024-04-19T23:39:17.072 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.072 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f56a5dcb5ec20d31-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.075 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.076 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.076 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.076 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.076 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.076 DEBUG:boto:Method: GET 2024-04-19T23:39:17.076 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.076 DEBUG:boto:Data: 2024-04-19T23:39:17.076 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.076 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.076 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.076 DEBUG:boto:Params: {} 2024-04-19T23:39:17.077 DEBUG:boto:Token: None 2024-04-19T23:39:17.077 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.077 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hYSmNQLMmMrtzvymqB8cIDHr7bM= 2024-04-19T23:39:17.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hYSmNQLMmMrtzvymqB8cIDHr7bM=', 'Content-Length': '0'} 2024-04-19T23:39:17.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c49edb60a0312590-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.122 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.122 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.123 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.123 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.123 DEBUG:boto:Method: GET 2024-04-19T23:39:17.123 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.123 DEBUG:boto:Data: 2024-04-19T23:39:17.123 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.123 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.123 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.123 DEBUG:boto:Params: {} 2024-04-19T23:39:17.123 DEBUG:boto:Token: None 2024-04-19T23:39:17.124 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.124 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hYSmNQLMmMrtzvymqB8cIDHr7bM= 2024-04-19T23:39:17.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hYSmNQLMmMrtzvymqB8cIDHr7bM=', 'Content-Length': '0'} 2024-04-19T23:39:17.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb5d5adbee7da4f8-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.126 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.126 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.126 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.126 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.126 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.127 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.127 DEBUG:boto:Data: 2024-04-19T23:39:17.127 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.127 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.127 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.127 DEBUG:boto:Params: {} 2024-04-19T23:39:17.127 DEBUG:boto:Token: None 2024-04-19T23:39:17.127 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.127 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:H7yUhqAH7wRjVmD54da4JLDi6Bk= 2024-04-19T23:39:17.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:H7yUhqAH7wRjVmD54da4JLDi6Bk=', 'Content-Length': '0'} 2024-04-19T23:39:17.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1f25ef656aa0869-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.130 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.130 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.130 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.130 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.130 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.130 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.130 DEBUG:boto:Data: 2024-04-19T23:39:17.131 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.131 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.131 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.131 DEBUG:boto:Params: {} 2024-04-19T23:39:17.131 DEBUG:boto:Token: None 2024-04-19T23:39:17.131 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.131 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:H7yUhqAH7wRjVmD54da4JLDi6Bk= 2024-04-19T23:39:17.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:H7yUhqAH7wRjVmD54da4JLDi6Bk=', 'Content-Length': '0'} 2024-04-19T23:39:17.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067f81ae6837d6d04-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.133 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.133 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.134 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.134 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.134 DEBUG:boto:Method: GET 2024-04-19T23:39:17.134 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.134 DEBUG:boto:Data: 2024-04-19T23:39:17.134 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.134 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.134 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.134 DEBUG:boto:Params: {} 2024-04-19T23:39:17.135 DEBUG:boto:Token: None 2024-04-19T23:39:17.135 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.135 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hYSmNQLMmMrtzvymqB8cIDHr7bM= 2024-04-19T23:39:17.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hYSmNQLMmMrtzvymqB8cIDHr7bM=', 'Content-Length': '0'} 2024-04-19T23:39:17.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1df8f2970e4b433-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.137 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.137 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.137 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.138 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.138 DEBUG:boto:Method: GET 2024-04-19T23:39:17.138 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.138 DEBUG:boto:Data: 2024-04-19T23:39:17.138 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.138 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.138 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.139 DEBUG:boto:Params: {} 2024-04-19T23:39:17.139 DEBUG:boto:Token: None 2024-04-19T23:39:17.139 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:17.139 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hYSmNQLMmMrtzvymqB8cIDHr7bM= 2024-04-19T23:39:17.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hYSmNQLMmMrtzvymqB8cIDHr7bM=', 'Content-Length': '0'} 2024-04-19T23:39:17.142 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa4b85f6ff65f145-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.142 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.142 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.142 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.143 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.143 DEBUG:boto:Method: GET 2024-04-19T23:39:17.143 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.143 DEBUG:boto:Data: 2024-04-19T23:39:17.143 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.143 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.143 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.144 DEBUG:boto:Params: {} 2024-04-19T23:39:17.144 DEBUG:boto:Token: None 2024-04-19T23:39:17.144 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.144 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031be5caf4a754f8d-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.147 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.148 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.148 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.148 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.148 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.149 DEBUG:boto:Method: GET 2024-04-19T23:39:17.149 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.149 DEBUG:boto:Data: 2024-04-19T23:39:17.149 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.149 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.149 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.149 DEBUG:boto:Params: {} 2024-04-19T23:39:17.150 DEBUG:boto:Token: None 2024-04-19T23:39:17.150 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.150 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6e5f0b508a802b5-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.153 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.153 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.153 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.153 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.153 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.153 DEBUG:boto:Data: 2024-04-19T23:39:17.153 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.153 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.154 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.154 DEBUG:boto:Params: {} 2024-04-19T23:39:17.154 DEBUG:boto:Token: None 2024-04-19T23:39:17.154 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.154 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002438a7787613bac8-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.158 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.158 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.158 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.158 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.158 DEBUG:boto:Data: 2024-04-19T23:39:17.158 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.158 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.159 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.159 DEBUG:boto:Params: {} 2024-04-19T23:39:17.159 DEBUG:boto:Token: None 2024-04-19T23:39:17.159 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.159 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e455c042622f156c-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.163 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.163 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.163 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.163 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.163 DEBUG:boto:Method: GET 2024-04-19T23:39:17.163 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.163 DEBUG:boto:Data: 2024-04-19T23:39:17.163 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.163 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.164 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.164 DEBUG:boto:Params: {} 2024-04-19T23:39:17.164 DEBUG:boto:Token: None 2024-04-19T23:39:17.164 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.164 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b3b484bb5232e25-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.166 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.167 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.167 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.167 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.167 DEBUG:boto:Method: GET 2024-04-19T23:39:17.167 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.167 DEBUG:boto:Data: 2024-04-19T23:39:17.167 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.167 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.167 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.167 DEBUG:boto:Params: {} 2024-04-19T23:39:17.168 DEBUG:boto:Token: None 2024-04-19T23:39:17.168 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.168 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e553b2b61ea887f9-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.170 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.170 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.170 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.170 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.170 DEBUG:boto:Method: GET 2024-04-19T23:39:17.170 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.170 DEBUG:boto:Data: 2024-04-19T23:39:17.171 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.171 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.171 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.171 DEBUG:boto:Params: {} 2024-04-19T23:39:17.171 DEBUG:boto:Token: None 2024-04-19T23:39:17.171 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.171 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091d5a6f17c323770-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.218 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.219 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.219 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.219 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.219 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.219 DEBUG:boto:Method: GET 2024-04-19T23:39:17.219 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.219 DEBUG:boto:Data: 2024-04-19T23:39:17.219 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.220 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.220 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.220 DEBUG:boto:Params: {} 2024-04-19T23:39:17.220 DEBUG:boto:Token: None 2024-04-19T23:39:17.220 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.220 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adf97d52754230fd-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.223 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.224 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.224 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.224 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.224 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.224 DEBUG:boto:Method: GET 2024-04-19T23:39:17.224 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.224 DEBUG:boto:Data: 2024-04-19T23:39:17.224 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.224 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.224 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.224 DEBUG:boto:Params: {} 2024-04-19T23:39:17.224 DEBUG:boto:Token: None 2024-04-19T23:39:17.225 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.225 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X3MJVS5UfIwhgXNV2UIJWZePW9o= 2024-04-19T23:39:17.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X3MJVS5UfIwhgXNV2UIJWZePW9o=', 'Content-Length': '0'} 2024-04-19T23:39:17.227 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000307a73c32ca2f463-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.227 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.227 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.227 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.227 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.227 DEBUG:boto:Method: GET 2024-04-19T23:39:17.227 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.228 DEBUG:boto:Data: 2024-04-19T23:39:17.228 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.228 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.228 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.228 DEBUG:boto:Params: {} 2024-04-19T23:39:17.228 DEBUG:boto:Token: None 2024-04-19T23:39:17.228 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.228 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X3MJVS5UfIwhgXNV2UIJWZePW9o= 2024-04-19T23:39:17.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X3MJVS5UfIwhgXNV2UIJWZePW9o=', 'Content-Length': '0'} 2024-04-19T23:39:17.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c547bfefbdf86493-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.278 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.279 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.279 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.279 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.279 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.279 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.279 DEBUG:boto:Data: 2024-04-19T23:39:17.279 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.279 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.279 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.279 DEBUG:boto:Params: {} 2024-04-19T23:39:17.280 DEBUG:boto:Token: None 2024-04-19T23:39:17.280 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.280 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:W5SBGQmIRWDtCu4lDya7wojMtNY= 2024-04-19T23:39:17.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W5SBGQmIRWDtCu4lDya7wojMtNY=', 'Content-Length': '0'} 2024-04-19T23:39:17.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002648fac5696c3828-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.282 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.282 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.282 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.282 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.283 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.283 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.283 DEBUG:boto:Data: 2024-04-19T23:39:17.283 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.283 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.283 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.283 DEBUG:boto:Params: {} 2024-04-19T23:39:17.283 DEBUG:boto:Token: None 2024-04-19T23:39:17.283 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.283 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:W5SBGQmIRWDtCu4lDya7wojMtNY= 2024-04-19T23:39:17.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W5SBGQmIRWDtCu4lDya7wojMtNY=', 'Content-Length': '0'} 2024-04-19T23:39:17.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096dc92547aa5f036-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.286 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.286 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.286 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.286 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.286 DEBUG:boto:Method: GET 2024-04-19T23:39:17.286 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.287 DEBUG:boto:Data: 2024-04-19T23:39:17.287 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.287 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.287 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.287 DEBUG:boto:Params: {} 2024-04-19T23:39:17.287 DEBUG:boto:Token: None 2024-04-19T23:39:17.287 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.287 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X3MJVS5UfIwhgXNV2UIJWZePW9o= 2024-04-19T23:39:17.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X3MJVS5UfIwhgXNV2UIJWZePW9o=', 'Content-Length': '0'} 2024-04-19T23:39:17.289 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4c544fb33567068-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.289 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.289 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.290 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.290 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.290 DEBUG:boto:Method: GET 2024-04-19T23:39:17.290 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.290 DEBUG:boto:Data: 2024-04-19T23:39:17.290 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.290 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.290 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.290 DEBUG:boto:Params: {} 2024-04-19T23:39:17.291 DEBUG:boto:Token: None 2024-04-19T23:39:17.291 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:17.291 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X3MJVS5UfIwhgXNV2UIJWZePW9o= 2024-04-19T23:39:17.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X3MJVS5UfIwhgXNV2UIJWZePW9o=', 'Content-Length': '0'} 2024-04-19T23:39:17.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000685e777602719d68-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.293 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.294 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.294 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.294 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.294 DEBUG:boto:Method: GET 2024-04-19T23:39:17.294 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.294 DEBUG:boto:Data: 2024-04-19T23:39:17.294 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.294 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.294 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.294 DEBUG:boto:Params: {} 2024-04-19T23:39:17.295 DEBUG:boto:Token: None 2024-04-19T23:39:17.295 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.295 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050ea151bac78860a-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.298 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.298 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.299 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.299 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.299 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.299 DEBUG:boto:Method: GET 2024-04-19T23:39:17.299 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.299 DEBUG:boto:Data: 2024-04-19T23:39:17.299 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.299 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.299 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.299 DEBUG:boto:Params: {} 2024-04-19T23:39:17.300 DEBUG:boto:Token: None 2024-04-19T23:39:17.300 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.300 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ce3c32b46b7c77c-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.303 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.303 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.303 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.303 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.303 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.303 DEBUG:boto:Data: 2024-04-19T23:39:17.303 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.303 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.304 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.304 DEBUG:boto:Params: {} 2024-04-19T23:39:17.304 DEBUG:boto:Token: None 2024-04-19T23:39:17.304 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.304 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.307 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba36270c09a539a9-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.307 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.308 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.308 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.308 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.308 DEBUG:boto:Data: 2024-04-19T23:39:17.308 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.308 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.308 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.308 DEBUG:boto:Params: {} 2024-04-19T23:39:17.308 DEBUG:boto:Token: None 2024-04-19T23:39:17.308 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.309 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066f86aa5e2633633-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.312 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.312 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.312 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.312 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.313 DEBUG:boto:Method: GET 2024-04-19T23:39:17.313 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.313 DEBUG:boto:Data: 2024-04-19T23:39:17.313 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.313 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.313 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.313 DEBUG:boto:Params: {} 2024-04-19T23:39:17.313 DEBUG:boto:Token: None 2024-04-19T23:39:17.313 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.313 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.315 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000447af2b7b5bf5c61-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.316 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.316 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.316 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.316 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.316 DEBUG:boto:Method: GET 2024-04-19T23:39:17.316 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.316 DEBUG:boto:Data: 2024-04-19T23:39:17.317 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.317 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.317 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.317 DEBUG:boto:Params: {} 2024-04-19T23:39:17.317 DEBUG:boto:Token: None 2024-04-19T23:39:17.317 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.317 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004a50bbdccb7a3b0-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.366 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.367 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.367 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.367 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.367 DEBUG:boto:Method: GET 2024-04-19T23:39:17.367 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.367 DEBUG:boto:Data: 2024-04-19T23:39:17.367 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.367 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.367 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.367 DEBUG:boto:Params: {} 2024-04-19T23:39:17.368 DEBUG:boto:Token: None 2024-04-19T23:39:17.368 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.368 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e33470e18306d53-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.372 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.372 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.372 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.372 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.372 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.372 DEBUG:boto:Method: GET 2024-04-19T23:39:17.373 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.373 DEBUG:boto:Data: 2024-04-19T23:39:17.373 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.373 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.373 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.373 DEBUG:boto:Params: {} 2024-04-19T23:39:17.373 DEBUG:boto:Token: None 2024-04-19T23:39:17.373 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.374 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c6619697486c0f6-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.377 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.378 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.378 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.378 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.378 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.378 DEBUG:boto:Method: GET 2024-04-19T23:39:17.378 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.378 DEBUG:boto:Data: 2024-04-19T23:39:17.378 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.379 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.379 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.379 DEBUG:boto:Params: {} 2024-04-19T23:39:17.379 DEBUG:boto:Token: None 2024-04-19T23:39:17.379 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.379 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UhVvyukMFZwEhP6IuCWgBc51N00= 2024-04-19T23:39:17.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UhVvyukMFZwEhP6IuCWgBc51N00=', 'Content-Length': '0'} 2024-04-19T23:39:17.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000617c0e54a9330e29-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.427 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.427 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.427 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.427 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.427 DEBUG:boto:Method: GET 2024-04-19T23:39:17.427 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.427 DEBUG:boto:Data: 2024-04-19T23:39:17.427 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.428 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.428 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.428 DEBUG:boto:Params: {} 2024-04-19T23:39:17.428 DEBUG:boto:Token: None 2024-04-19T23:39:17.428 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.428 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UhVvyukMFZwEhP6IuCWgBc51N00= 2024-04-19T23:39:17.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UhVvyukMFZwEhP6IuCWgBc51N00=', 'Content-Length': '0'} 2024-04-19T23:39:17.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af8bb6073c8af446-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.431 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.431 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.431 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.432 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.432 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.432 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.432 DEBUG:boto:Data: 2024-04-19T23:39:17.432 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.432 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.432 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.432 DEBUG:boto:Params: {} 2024-04-19T23:39:17.432 DEBUG:boto:Token: None 2024-04-19T23:39:17.433 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.433 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cLj6vzdJR4Uyk0OtXlpz1RUixRk= 2024-04-19T23:39:17.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cLj6vzdJR4Uyk0OtXlpz1RUixRk=', 'Content-Length': '0'} 2024-04-19T23:39:17.435 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011417f70012579b9-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.435 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.435 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.435 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.435 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.436 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.436 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.436 DEBUG:boto:Data: 2024-04-19T23:39:17.436 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.436 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.436 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.436 DEBUG:boto:Params: {} 2024-04-19T23:39:17.436 DEBUG:boto:Token: None 2024-04-19T23:39:17.436 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.436 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cLj6vzdJR4Uyk0OtXlpz1RUixRk= 2024-04-19T23:39:17.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cLj6vzdJR4Uyk0OtXlpz1RUixRk=', 'Content-Length': '0'} 2024-04-19T23:39:17.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8de07eaae3ea7cf-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.439 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.439 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.439 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.439 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.439 DEBUG:boto:Method: GET 2024-04-19T23:39:17.439 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.439 DEBUG:boto:Data: 2024-04-19T23:39:17.439 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.440 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.440 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.440 DEBUG:boto:Params: {} 2024-04-19T23:39:17.440 DEBUG:boto:Token: None 2024-04-19T23:39:17.440 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.440 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UhVvyukMFZwEhP6IuCWgBc51N00= 2024-04-19T23:39:17.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UhVvyukMFZwEhP6IuCWgBc51N00=', 'Content-Length': '0'} 2024-04-19T23:39:17.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e3296617d93ace5-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.443 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.443 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.443 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.443 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.443 DEBUG:boto:Method: GET 2024-04-19T23:39:17.443 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.443 DEBUG:boto:Data: 2024-04-19T23:39:17.443 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.443 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.443 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.444 DEBUG:boto:Params: {} 2024-04-19T23:39:17.444 DEBUG:boto:Token: None 2024-04-19T23:39:17.444 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:17.444 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UhVvyukMFZwEhP6IuCWgBc51N00= 2024-04-19T23:39:17.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UhVvyukMFZwEhP6IuCWgBc51N00=', 'Content-Length': '0'} 2024-04-19T23:39:17.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea64cc22559c2a63-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.446 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.446 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.446 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.446 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.447 DEBUG:boto:Method: GET 2024-04-19T23:39:17.447 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.447 DEBUG:boto:Data: 2024-04-19T23:39:17.447 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.447 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.447 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.447 DEBUG:boto:Params: {} 2024-04-19T23:39:17.447 DEBUG:boto:Token: None 2024-04-19T23:39:17.447 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.447 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f8bf29d7b9c8519-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.450 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.451 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.451 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.451 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.451 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.451 DEBUG:boto:Method: GET 2024-04-19T23:39:17.451 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.451 DEBUG:boto:Data: 2024-04-19T23:39:17.452 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.452 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.452 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.452 DEBUG:boto:Params: {} 2024-04-19T23:39:17.452 DEBUG:boto:Token: None 2024-04-19T23:39:17.452 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.452 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b9b769fa8a59f39-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.456 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.456 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.456 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.456 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.456 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.456 DEBUG:boto:Data: 2024-04-19T23:39:17.456 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.456 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.457 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.457 DEBUG:boto:Params: {} 2024-04-19T23:39:17.457 DEBUG:boto:Token: None 2024-04-19T23:39:17.457 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.457 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000869c577d69f8d865-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.461 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.461 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.461 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.461 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.461 DEBUG:boto:Data: 2024-04-19T23:39:17.461 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.461 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.461 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.461 DEBUG:boto:Params: {} 2024-04-19T23:39:17.461 DEBUG:boto:Token: None 2024-04-19T23:39:17.461 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.462 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dab5abcdaf718460-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.465 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.466 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.466 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.466 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.466 DEBUG:boto:Method: GET 2024-04-19T23:39:17.466 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.466 DEBUG:boto:Data: 2024-04-19T23:39:17.466 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.466 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.466 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.467 DEBUG:boto:Params: {} 2024-04-19T23:39:17.467 DEBUG:boto:Token: None 2024-04-19T23:39:17.467 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.467 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eecf433697521858-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.514 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.514 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.515 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.515 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.515 DEBUG:boto:Method: GET 2024-04-19T23:39:17.515 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.515 DEBUG:boto:Data: 2024-04-19T23:39:17.515 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.515 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.515 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.515 DEBUG:boto:Params: {} 2024-04-19T23:39:17.515 DEBUG:boto:Token: None 2024-04-19T23:39:17.516 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.516 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dcc880946bc8f76-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.518 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.519 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.519 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.519 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.519 DEBUG:boto:Method: GET 2024-04-19T23:39:17.519 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.519 DEBUG:boto:Data: 2024-04-19T23:39:17.519 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.519 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.519 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.519 DEBUG:boto:Params: {} 2024-04-19T23:39:17.519 DEBUG:boto:Token: None 2024-04-19T23:39:17.520 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.520 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db934ea3d9d6191f-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.523 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.523 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.523 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.523 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.524 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.524 DEBUG:boto:Method: GET 2024-04-19T23:39:17.524 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.524 DEBUG:boto:Data: 2024-04-19T23:39:17.524 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.524 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.524 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.524 DEBUG:boto:Params: {} 2024-04-19T23:39:17.525 DEBUG:boto:Token: None 2024-04-19T23:39:17.525 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.525 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000402a6cb0c1e304ee-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.528 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.529 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.529 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.529 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.529 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.529 DEBUG:boto:Method: GET 2024-04-19T23:39:17.529 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.529 DEBUG:boto:Data: 2024-04-19T23:39:17.529 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.529 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.529 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.530 DEBUG:boto:Params: {} 2024-04-19T23:39:17.530 DEBUG:boto:Token: None 2024-04-19T23:39:17.530 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.530 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m/rIvSjOdQXwmsR8iceybIvq6MQ= 2024-04-19T23:39:17.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m/rIvSjOdQXwmsR8iceybIvq6MQ=', 'Content-Length': '0'} 2024-04-19T23:39:17.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008cfa3fe14f0937d5-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.532 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.532 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.533 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.533 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.533 DEBUG:boto:Method: GET 2024-04-19T23:39:17.533 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.533 DEBUG:boto:Data: 2024-04-19T23:39:17.533 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.533 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.533 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.533 DEBUG:boto:Params: {} 2024-04-19T23:39:17.533 DEBUG:boto:Token: None 2024-04-19T23:39:17.534 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.534 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m/rIvSjOdQXwmsR8iceybIvq6MQ= 2024-04-19T23:39:17.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m/rIvSjOdQXwmsR8iceybIvq6MQ=', 'Content-Length': '0'} 2024-04-19T23:39:17.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e68df293c82d57c2-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.536 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.536 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.536 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.536 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.537 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.537 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.537 DEBUG:boto:Data: 2024-04-19T23:39:17.537 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.537 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.537 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.537 DEBUG:boto:Params: {} 2024-04-19T23:39:17.537 DEBUG:boto:Token: None 2024-04-19T23:39:17.537 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.537 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CJhlzr/5nsWPrMWvtR8Fc4JCFZM= 2024-04-19T23:39:17.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CJhlzr/5nsWPrMWvtR8Fc4JCFZM=', 'Content-Length': '0'} 2024-04-19T23:39:17.539 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016d4eec68fbcf1f2-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.539 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.539 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.540 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.540 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.540 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.540 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.540 DEBUG:boto:Data: 2024-04-19T23:39:17.540 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.540 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.540 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.540 DEBUG:boto:Params: {} 2024-04-19T23:39:17.540 DEBUG:boto:Token: None 2024-04-19T23:39:17.540 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.541 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CJhlzr/5nsWPrMWvtR8Fc4JCFZM= 2024-04-19T23:39:17.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CJhlzr/5nsWPrMWvtR8Fc4JCFZM=', 'Content-Length': '0'} 2024-04-19T23:39:17.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6dea49cea9b48dc-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.543 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.543 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.543 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.543 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.543 DEBUG:boto:Method: GET 2024-04-19T23:39:17.543 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.543 DEBUG:boto:Data: 2024-04-19T23:39:17.543 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.543 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.543 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.544 DEBUG:boto:Params: {} 2024-04-19T23:39:17.544 DEBUG:boto:Token: None 2024-04-19T23:39:17.544 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.544 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m/rIvSjOdQXwmsR8iceybIvq6MQ= 2024-04-19T23:39:17.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m/rIvSjOdQXwmsR8iceybIvq6MQ=', 'Content-Length': '0'} 2024-04-19T23:39:17.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da818652f130bd9c-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.586 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.586 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.587 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.587 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.587 DEBUG:boto:Method: GET 2024-04-19T23:39:17.587 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.587 DEBUG:boto:Data: 2024-04-19T23:39:17.587 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.587 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.587 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.587 DEBUG:boto:Params: {} 2024-04-19T23:39:17.587 DEBUG:boto:Token: None 2024-04-19T23:39:17.588 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:17.588 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m/rIvSjOdQXwmsR8iceybIvq6MQ= 2024-04-19T23:39:17.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m/rIvSjOdQXwmsR8iceybIvq6MQ=', 'Content-Length': '0'} 2024-04-19T23:39:17.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085a57d12d0d8f379-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.590 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.590 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.590 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.591 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.591 DEBUG:boto:Method: GET 2024-04-19T23:39:17.591 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.591 DEBUG:boto:Data: 2024-04-19T23:39:17.591 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.591 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.591 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.591 DEBUG:boto:Params: {} 2024-04-19T23:39:17.591 DEBUG:boto:Token: None 2024-04-19T23:39:17.592 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.592 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca5c3d5eb21478df-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.595 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.595 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.595 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.595 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.596 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.596 DEBUG:boto:Method: GET 2024-04-19T23:39:17.596 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.596 DEBUG:boto:Data: 2024-04-19T23:39:17.596 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.596 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.596 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.596 DEBUG:boto:Params: {} 2024-04-19T23:39:17.596 DEBUG:boto:Token: None 2024-04-19T23:39:17.596 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.597 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015cd45e2581972fc-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.599 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.600 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.600 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.600 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.600 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.600 DEBUG:boto:Data: 2024-04-19T23:39:17.600 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.600 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.600 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.601 DEBUG:boto:Params: {} 2024-04-19T23:39:17.601 DEBUG:boto:Token: None 2024-04-19T23:39:17.601 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.601 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000604fc286a79ab969-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.605 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.605 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.605 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.605 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.605 DEBUG:boto:Data: 2024-04-19T23:39:17.605 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.605 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.605 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.605 DEBUG:boto:Params: {} 2024-04-19T23:39:17.606 DEBUG:boto:Token: None 2024-04-19T23:39:17.606 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.606 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa55a4a83007a51f-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.610 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.610 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.610 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.610 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.610 DEBUG:boto:Method: GET 2024-04-19T23:39:17.610 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.610 DEBUG:boto:Data: 2024-04-19T23:39:17.610 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.611 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.611 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.611 DEBUG:boto:Params: {} 2024-04-19T23:39:17.611 DEBUG:boto:Token: None 2024-04-19T23:39:17.611 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.611 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.613 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b05658fba8812d43-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.658 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.658 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.659 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.659 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.659 DEBUG:boto:Method: GET 2024-04-19T23:39:17.659 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.659 DEBUG:boto:Data: 2024-04-19T23:39:17.659 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.659 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.659 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.659 DEBUG:boto:Params: {} 2024-04-19T23:39:17.659 DEBUG:boto:Token: None 2024-04-19T23:39:17.660 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf41a97ae2b7bfec-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.662 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.662 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.662 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.662 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.663 DEBUG:boto:Method: GET 2024-04-19T23:39:17.663 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.663 DEBUG:boto:Data: 2024-04-19T23:39:17.663 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.663 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.663 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.663 DEBUG:boto:Params: {} 2024-04-19T23:39:17.663 DEBUG:boto:Token: None 2024-04-19T23:39:17.663 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.664 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f219277f5a7f73d2-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.667 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.667 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.667 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.667 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.668 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.668 DEBUG:boto:Method: GET 2024-04-19T23:39:17.668 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.668 DEBUG:boto:Data: 2024-04-19T23:39:17.668 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.668 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.668 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.668 DEBUG:boto:Params: {} 2024-04-19T23:39:17.668 DEBUG:boto:Token: None 2024-04-19T23:39:17.669 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.669 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f229d1976674085-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.671 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.672 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.672 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.672 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.672 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.673 DEBUG:boto:Method: GET 2024-04-19T23:39:17.673 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.673 DEBUG:boto:Data: 2024-04-19T23:39:17.673 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.673 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.673 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.673 DEBUG:boto:Params: {} 2024-04-19T23:39:17.673 DEBUG:boto:Token: None 2024-04-19T23:39:17.673 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.674 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gdCB2hVBXTj4SizLKVl3QPw+JKI= 2024-04-19T23:39:17.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gdCB2hVBXTj4SizLKVl3QPw+JKI=', 'Content-Length': '0'} 2024-04-19T23:39:17.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f38fa936eb21dadd-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.676 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.676 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.676 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.676 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.676 DEBUG:boto:Method: GET 2024-04-19T23:39:17.676 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.676 DEBUG:boto:Data: 2024-04-19T23:39:17.677 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.677 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.677 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.677 DEBUG:boto:Params: {} 2024-04-19T23:39:17.677 DEBUG:boto:Token: None 2024-04-19T23:39:17.677 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.677 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gdCB2hVBXTj4SizLKVl3QPw+JKI= 2024-04-19T23:39:17.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gdCB2hVBXTj4SizLKVl3QPw+JKI=', 'Content-Length': '0'} 2024-04-19T23:39:17.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea113407f69ce48a-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.726 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.727 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.727 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.727 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.727 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.727 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.727 DEBUG:boto:Data: 2024-04-19T23:39:17.727 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.727 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.728 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.728 DEBUG:boto:Params: {} 2024-04-19T23:39:17.728 DEBUG:boto:Token: None 2024-04-19T23:39:17.728 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.728 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tHHNMQv07xmhYBzbDDAjSx3SAtg= 2024-04-19T23:39:17.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tHHNMQv07xmhYBzbDDAjSx3SAtg=', 'Content-Length': '0'} 2024-04-19T23:39:17.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ac1df76f166e7ae-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.731 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.731 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.731 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.731 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.731 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.731 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.731 DEBUG:boto:Data: 2024-04-19T23:39:17.732 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.732 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.732 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.732 DEBUG:boto:Params: {} 2024-04-19T23:39:17.732 DEBUG:boto:Token: None 2024-04-19T23:39:17.732 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.732 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tHHNMQv07xmhYBzbDDAjSx3SAtg= 2024-04-19T23:39:17.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tHHNMQv07xmhYBzbDDAjSx3SAtg=', 'Content-Length': '0'} 2024-04-19T23:39:17.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1cce33bfd84cb08-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.735 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.735 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.735 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.735 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.735 DEBUG:boto:Method: GET 2024-04-19T23:39:17.735 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.735 DEBUG:boto:Data: 2024-04-19T23:39:17.735 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.735 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.735 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.735 DEBUG:boto:Params: {} 2024-04-19T23:39:17.736 DEBUG:boto:Token: None 2024-04-19T23:39:17.736 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.736 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gdCB2hVBXTj4SizLKVl3QPw+JKI= 2024-04-19T23:39:17.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gdCB2hVBXTj4SizLKVl3QPw+JKI=', 'Content-Length': '0'} 2024-04-19T23:39:17.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006926b8f035d5f979-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.782 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.783 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.783 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.783 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.783 DEBUG:boto:Method: GET 2024-04-19T23:39:17.783 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.783 DEBUG:boto:Data: 2024-04-19T23:39:17.783 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.783 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.783 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.783 DEBUG:boto:Params: {} 2024-04-19T23:39:17.784 DEBUG:boto:Token: None 2024-04-19T23:39:17.784 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:17.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gdCB2hVBXTj4SizLKVl3QPw+JKI= 2024-04-19T23:39:17.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gdCB2hVBXTj4SizLKVl3QPw+JKI=', 'Content-Length': '0'} 2024-04-19T23:39:17.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d25137b65b240ee-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.786 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.787 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.787 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.787 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.788 DEBUG:boto:Method: GET 2024-04-19T23:39:17.788 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.788 DEBUG:boto:Data: 2024-04-19T23:39:17.788 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.788 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.788 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.788 DEBUG:boto:Params: {} 2024-04-19T23:39:17.788 DEBUG:boto:Token: None 2024-04-19T23:39:17.788 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.788 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f707072000d35a0-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.792 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.792 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.793 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.793 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.793 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.793 DEBUG:boto:Method: GET 2024-04-19T23:39:17.793 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.793 DEBUG:boto:Data: 2024-04-19T23:39:17.793 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.793 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.793 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.793 DEBUG:boto:Params: {} 2024-04-19T23:39:17.794 DEBUG:boto:Token: None 2024-04-19T23:39:17.794 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.794 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f94838cd03f52417-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.797 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.797 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.797 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.797 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.798 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.798 DEBUG:boto:Data: 2024-04-19T23:39:17.798 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.798 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.798 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.798 DEBUG:boto:Params: {} 2024-04-19T23:39:17.798 DEBUG:boto:Token: None 2024-04-19T23:39:17.798 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.798 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065fe82d4b8863fc9-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.802 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.802 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.802 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.802 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.802 DEBUG:boto:Data: 2024-04-19T23:39:17.803 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.803 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.803 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.803 DEBUG:boto:Params: {} 2024-04-19T23:39:17.803 DEBUG:boto:Token: None 2024-04-19T23:39:17.803 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.803 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cebcea286e741c9-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.807 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.807 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.807 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.807 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.807 DEBUG:boto:Method: GET 2024-04-19T23:39:17.807 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.807 DEBUG:boto:Data: 2024-04-19T23:39:17.807 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.807 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.807 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.807 DEBUG:boto:Params: {} 2024-04-19T23:39:17.808 DEBUG:boto:Token: None 2024-04-19T23:39:17.808 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.808 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bd61b62bbe8023b-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.810 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.810 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.810 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.810 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.810 DEBUG:boto:Method: GET 2024-04-19T23:39:17.810 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.810 DEBUG:boto:Data: 2024-04-19T23:39:17.811 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.811 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.811 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.811 DEBUG:boto:Params: {} 2024-04-19T23:39:17.811 DEBUG:boto:Token: None 2024-04-19T23:39:17.811 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.811 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093903d23f6fd7b4f-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.813 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.814 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.814 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.814 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.814 DEBUG:boto:Method: GET 2024-04-19T23:39:17.814 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.814 DEBUG:boto:Data: 2024-04-19T23:39:17.814 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.814 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.814 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.814 DEBUG:boto:Params: {} 2024-04-19T23:39:17.815 DEBUG:boto:Token: None 2024-04-19T23:39:17.815 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.815 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018467aff9550a6e7-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.818 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.818 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.818 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.818 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.819 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.819 DEBUG:boto:Method: GET 2024-04-19T23:39:17.819 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.819 DEBUG:boto:Data: 2024-04-19T23:39:17.819 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.819 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.819 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.819 DEBUG:boto:Params: {} 2024-04-19T23:39:17.819 DEBUG:boto:Token: None 2024-04-19T23:39:17.819 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.820 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70fc484dc225af1-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.822 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.823 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.823 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.823 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.823 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.823 DEBUG:boto:Method: GET 2024-04-19T23:39:17.823 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.823 DEBUG:boto:Data: 2024-04-19T23:39:17.823 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.823 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.824 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.824 DEBUG:boto:Params: {} 2024-04-19T23:39:17.824 DEBUG:boto:Token: None 2024-04-19T23:39:17.824 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vD2fISuQ5zH19zgi6LV9y4Bwpok= 2024-04-19T23:39:17.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vD2fISuQ5zH19zgi6LV9y4Bwpok=', 'Content-Length': '0'} 2024-04-19T23:39:17.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0c430275d5c925d-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.826 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.826 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.826 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.826 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.827 DEBUG:boto:Method: GET 2024-04-19T23:39:17.827 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.827 DEBUG:boto:Data: 2024-04-19T23:39:17.827 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.827 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.827 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.827 DEBUG:boto:Params: {} 2024-04-19T23:39:17.827 DEBUG:boto:Token: None 2024-04-19T23:39:17.827 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.827 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vD2fISuQ5zH19zgi6LV9y4Bwpok= 2024-04-19T23:39:17.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vD2fISuQ5zH19zgi6LV9y4Bwpok=', 'Content-Length': '0'} 2024-04-19T23:39:17.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa3dd3e1794f7680-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.870 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.871 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.871 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.871 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.871 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.871 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.871 DEBUG:boto:Data: 2024-04-19T23:39:17.871 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.871 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.871 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.871 DEBUG:boto:Params: {} 2024-04-19T23:39:17.872 DEBUG:boto:Token: None 2024-04-19T23:39:17.872 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.872 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/4g7Pn18QKuZwFrIiYF3YsGxttA= 2024-04-19T23:39:17.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/4g7Pn18QKuZwFrIiYF3YsGxttA=', 'Content-Length': '0'} 2024-04-19T23:39:17.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f86cbfcef59a2fca-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.874 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.874 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.874 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.874 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.875 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.875 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.875 DEBUG:boto:Data: 2024-04-19T23:39:17.875 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.875 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.875 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.875 DEBUG:boto:Params: {} 2024-04-19T23:39:17.875 DEBUG:boto:Token: None 2024-04-19T23:39:17.875 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.875 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/4g7Pn18QKuZwFrIiYF3YsGxttA= 2024-04-19T23:39:17.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/4g7Pn18QKuZwFrIiYF3YsGxttA=', 'Content-Length': '0'} 2024-04-19T23:39:17.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c7526736c8af932-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.878 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.878 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.878 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.878 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.878 DEBUG:boto:Method: GET 2024-04-19T23:39:17.878 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.878 DEBUG:boto:Data: 2024-04-19T23:39:17.878 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.879 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.879 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.879 DEBUG:boto:Params: {} 2024-04-19T23:39:17.879 DEBUG:boto:Token: None 2024-04-19T23:39:17.879 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.879 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vD2fISuQ5zH19zgi6LV9y4Bwpok= 2024-04-19T23:39:17.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vD2fISuQ5zH19zgi6LV9y4Bwpok=', 'Content-Length': '0'} 2024-04-19T23:39:17.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000029c9ea9e676ddd2-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.881 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.881 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.881 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.882 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.882 DEBUG:boto:Method: GET 2024-04-19T23:39:17.882 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.882 DEBUG:boto:Data: 2024-04-19T23:39:17.882 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.882 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.882 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.882 DEBUG:boto:Params: {} 2024-04-19T23:39:17.882 DEBUG:boto:Token: None 2024-04-19T23:39:17.882 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:17.883 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vD2fISuQ5zH19zgi6LV9y4Bwpok= 2024-04-19T23:39:17.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vD2fISuQ5zH19zgi6LV9y4Bwpok=', 'Content-Length': '0'} 2024-04-19T23:39:17.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060b3f5b20ac0c6e3-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.885 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.885 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.885 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.885 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.885 DEBUG:boto:Method: GET 2024-04-19T23:39:17.885 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.885 DEBUG:boto:Data: 2024-04-19T23:39:17.885 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.885 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.885 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.886 DEBUG:boto:Params: {} 2024-04-19T23:39:17.886 DEBUG:boto:Token: None 2024-04-19T23:39:17.886 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.886 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3d45e5147aa0fc5-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.889 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.890 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.890 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.890 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.890 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.890 DEBUG:boto:Method: GET 2024-04-19T23:39:17.890 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.890 DEBUG:boto:Data: 2024-04-19T23:39:17.890 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.890 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.890 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.891 DEBUG:boto:Params: {} 2024-04-19T23:39:17.891 DEBUG:boto:Token: None 2024-04-19T23:39:17.891 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.891 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043ecd5b5d3d2f33b-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.894 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.894 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.894 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.894 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.894 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.895 DEBUG:boto:Data: 2024-04-19T23:39:17.895 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.895 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.895 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.895 DEBUG:boto:Params: {} 2024-04-19T23:39:17.895 DEBUG:boto:Token: None 2024-04-19T23:39:17.895 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.895 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b9cad92081d365b-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.899 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.899 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.899 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.899 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:17.899 DEBUG:boto:Data: 2024-04-19T23:39:17.899 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.900 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.900 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.900 DEBUG:boto:Params: {} 2024-04-19T23:39:17.900 DEBUG:boto:Token: None 2024-04-19T23:39:17.900 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj 2024-04-19T23:39:17.900 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4= 2024-04-19T23:39:17.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bY3DAP9zmt+FFiCHPG8VGdpmlK4=', 'Content-Length': '0'} 2024-04-19T23:39:17.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008320917a91de0ef4-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.903 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.903 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.904 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.904 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.904 DEBUG:boto:Method: GET 2024-04-19T23:39:17.904 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.904 DEBUG:boto:Data: 2024-04-19T23:39:17.904 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.904 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.904 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.904 DEBUG:boto:Params: {} 2024-04-19T23:39:17.904 DEBUG:boto:Token: None 2024-04-19T23:39:17.904 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.905 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.907 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2c110269715b6e2-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.950 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.951 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.951 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.951 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.951 DEBUG:boto:Method: GET 2024-04-19T23:39:17.951 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.951 DEBUG:boto:Data: 2024-04-19T23:39:17.951 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.951 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.951 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.951 DEBUG:boto:Params: {} 2024-04-19T23:39:17.952 DEBUG:boto:Token: None 2024-04-19T23:39:17.952 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:17.952 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s= 2024-04-19T23:39:17.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:31A1pAmYDerIuKK2fzoYSbAHk/s=', 'Content-Length': '0'} 2024-04-19T23:39:17.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ceacb04a1c2ae2b-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.954 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.954 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.954 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.955 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.955 DEBUG:boto:Method: GET 2024-04-19T23:39:17.955 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.955 DEBUG:boto:Data: 2024-04-19T23:39:17.955 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.955 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.955 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.955 DEBUG:boto:Params: {} 2024-04-19T23:39:17.955 DEBUG:boto:Token: None 2024-04-19T23:39:17.956 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.956 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000749205876b0baf7b-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.958 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.959 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:17.959 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:17.959 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.959 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:17.959 DEBUG:boto:Method: GET 2024-04-19T23:39:17.959 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:17.960 DEBUG:boto:Data: 2024-04-19T23:39:17.960 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.960 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.960 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.960 DEBUG:boto:Params: {} 2024-04-19T23:39:17.960 DEBUG:boto:Token: None 2024-04-19T23:39:17.960 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/ 2024-04-19T23:39:17.960 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE= 2024-04-19T23:39:17.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JOHt0KETgBDN9QbkFIVkKaxfMZE=', 'Content-Length': '0'} 2024-04-19T23:39:17.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dca9877809570384-00662300a5-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.963 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:17.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.964 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.964 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.964 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.964 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.965 DEBUG:boto:Method: GET 2024-04-19T23:39:17.965 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.965 DEBUG:boto:Data: 2024-04-19T23:39:17.965 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.965 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.966 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.966 DEBUG:boto:Params: {} 2024-04-19T23:39:17.966 DEBUG:boto:Token: None 2024-04-19T23:39:17.966 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.967 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iEsUMw7O5+eU80tVtNMckjkjqMM= 2024-04-19T23:39:17.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iEsUMw7O5+eU80tVtNMckjkjqMM=', 'Content-Length': '0'} 2024-04-19T23:39:17.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce39ebed407bdd87-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.969 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.970 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.970 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.970 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.970 DEBUG:boto:Method: GET 2024-04-19T23:39:17.971 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.971 DEBUG:boto:Data: 2024-04-19T23:39:17.971 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.971 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.971 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.972 DEBUG:boto:Params: {} 2024-04-19T23:39:17.972 DEBUG:boto:Token: None 2024-04-19T23:39:17.972 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.972 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iEsUMw7O5+eU80tVtNMckjkjqMM= 2024-04-19T23:39:17.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iEsUMw7O5+eU80tVtNMckjkjqMM=', 'Content-Length': '0'} 2024-04-19T23:39:17.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005547f23c77c31e0-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.975 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.975 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.976 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.976 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.976 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.976 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.976 DEBUG:boto:Data: 2024-04-19T23:39:17.977 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.977 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.977 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.977 DEBUG:boto:Params: {} 2024-04-19T23:39:17.978 DEBUG:boto:Token: None 2024-04-19T23:39:17.978 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.978 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mElodU6QJcxI2g3oGRHB4uhYRRI= 2024-04-19T23:39:17.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mElodU6QJcxI2g3oGRHB4uhYRRI=', 'Content-Length': '0'} 2024-04-19T23:39:17.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3cac1de59d3fd7d-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.981 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.981 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.981 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.981 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.982 DEBUG:boto:Method: HEAD 2024-04-19T23:39:17.982 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.982 DEBUG:boto:Data: 2024-04-19T23:39:17.982 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.983 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.983 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.983 DEBUG:boto:Params: {} 2024-04-19T23:39:17.983 DEBUG:boto:Token: None 2024-04-19T23:39:17.983 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mElodU6QJcxI2g3oGRHB4uhYRRI= 2024-04-19T23:39:17.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mElodU6QJcxI2g3oGRHB4uhYRRI=', 'Content-Length': '0'} 2024-04-19T23:39:17.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007333667a9a4e0d51-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:17.986 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.986 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.987 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.987 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.987 DEBUG:boto:Method: GET 2024-04-19T23:39:17.987 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.987 DEBUG:boto:Data: 2024-04-19T23:39:17.987 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.987 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:17.988 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.988 DEBUG:boto:Params: {} 2024-04-19T23:39:17.988 DEBUG:boto:Token: None 2024-04-19T23:39:17.988 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.988 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iEsUMw7O5+eU80tVtNMckjkjqMM= 2024-04-19T23:39:17.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iEsUMw7O5+eU80tVtNMckjkjqMM=', 'Content-Length': '0'} 2024-04-19T23:39:17.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9750900ff73ef4d-00662300a5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:17.991 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:17.991 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:17.991 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.991 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.991 DEBUG:boto:Method: GET 2024-04-19T23:39:17.991 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.991 DEBUG:boto:Data: 2024-04-19T23:39:17.991 DEBUG:boto:Headers: {} 2024-04-19T23:39:17.992 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:17.992 DEBUG:boto:Port: 8000 2024-04-19T23:39:17.992 DEBUG:boto:Params: {} 2024-04-19T23:39:17.992 DEBUG:boto:Token: None 2024-04-19T23:39:17.992 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:17 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:17.992 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iEsUMw7O5+eU80tVtNMckjkjqMM= 2024-04-19T23:39:17.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:17 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iEsUMw7O5+eU80tVtNMckjkjqMM=', 'Content-Length': '0'} 2024-04-19T23:39:17.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031ce7b7a8ee0174b-00662300a5-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.042 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.043 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.043 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.043 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.043 DEBUG:boto:Method: GET 2024-04-19T23:39:18.043 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.044 DEBUG:boto:Data: 2024-04-19T23:39:18.044 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.044 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.044 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.044 DEBUG:boto:Params: {} 2024-04-19T23:39:18.044 DEBUG:boto:Token: None 2024-04-19T23:39:18.045 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.045 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e5b3c4fb63d2262-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.048 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.049 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.049 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.049 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.049 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.050 DEBUG:boto:Method: GET 2024-04-19T23:39:18.050 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.050 DEBUG:boto:Data: 2024-04-19T23:39:18.050 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.050 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.050 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.051 DEBUG:boto:Params: {} 2024-04-19T23:39:18.051 DEBUG:boto:Token: None 2024-04-19T23:39:18.051 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.051 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1f7376bb2d56f6e-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.054 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.055 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.055 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.055 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.055 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.055 DEBUG:boto:Data: 2024-04-19T23:39:18.055 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.056 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.056 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.056 DEBUG:boto:Params: {} 2024-04-19T23:39:18.056 DEBUG:boto:Token: None 2024-04-19T23:39:18.056 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.056 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0d96ab4e0774081-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.060 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.060 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.060 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.060 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.060 DEBUG:boto:Data: 2024-04-19T23:39:18.060 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.060 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.060 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.060 DEBUG:boto:Params: {} 2024-04-19T23:39:18.060 DEBUG:boto:Token: None 2024-04-19T23:39:18.061 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.061 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a28103287deb5715-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.064 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.065 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.065 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.065 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.065 DEBUG:boto:Method: GET 2024-04-19T23:39:18.066 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.066 DEBUG:boto:Data: 2024-04-19T23:39:18.066 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.066 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.066 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.066 DEBUG:boto:Params: {} 2024-04-19T23:39:18.067 DEBUG:boto:Token: None 2024-04-19T23:39:18.067 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.067 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e1ac2bac5620986-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.110 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.111 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.111 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.111 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.111 DEBUG:boto:Method: GET 2024-04-19T23:39:18.111 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.111 DEBUG:boto:Data: 2024-04-19T23:39:18.111 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.111 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.111 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.111 DEBUG:boto:Params: {} 2024-04-19T23:39:18.112 DEBUG:boto:Token: None 2024-04-19T23:39:18.112 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.112 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb9a07fa0c9e09e2-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.114 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.114 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.114 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.115 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.115 DEBUG:boto:Method: GET 2024-04-19T23:39:18.115 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.115 DEBUG:boto:Data: 2024-04-19T23:39:18.115 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.115 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.115 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.115 DEBUG:boto:Params: {} 2024-04-19T23:39:18.115 DEBUG:boto:Token: None 2024-04-19T23:39:18.115 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.116 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003599edc1c329ba5b-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.119 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.119 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.119 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.120 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.120 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.120 DEBUG:boto:Method: GET 2024-04-19T23:39:18.120 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.120 DEBUG:boto:Data: 2024-04-19T23:39:18.120 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.120 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.120 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.120 DEBUG:boto:Params: {} 2024-04-19T23:39:18.121 DEBUG:boto:Token: None 2024-04-19T23:39:18.121 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.121 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000effb2bb8e5fe16c5-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.124 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.124 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.125 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.125 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.125 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.125 DEBUG:boto:Method: GET 2024-04-19T23:39:18.125 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.125 DEBUG:boto:Data: 2024-04-19T23:39:18.125 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.125 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.125 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.125 DEBUG:boto:Params: {} 2024-04-19T23:39:18.126 DEBUG:boto:Token: None 2024-04-19T23:39:18.126 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.126 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SDjsyISMlEYcs83N393y6HHOZ7o= 2024-04-19T23:39:18.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SDjsyISMlEYcs83N393y6HHOZ7o=', 'Content-Length': '0'} 2024-04-19T23:39:18.128 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc22bfe0ea3f0c54-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.128 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.129 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.129 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.129 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.129 DEBUG:boto:Method: GET 2024-04-19T23:39:18.129 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.129 DEBUG:boto:Data: 2024-04-19T23:39:18.129 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.129 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.129 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.129 DEBUG:boto:Params: {} 2024-04-19T23:39:18.129 DEBUG:boto:Token: None 2024-04-19T23:39:18.130 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.130 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SDjsyISMlEYcs83N393y6HHOZ7o= 2024-04-19T23:39:18.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SDjsyISMlEYcs83N393y6HHOZ7o=', 'Content-Length': '0'} 2024-04-19T23:39:18.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000965858b92af2f7e1-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.178 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.179 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.179 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.179 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.179 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.179 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.179 DEBUG:boto:Data: 2024-04-19T23:39:18.179 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.179 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.179 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.179 DEBUG:boto:Params: {} 2024-04-19T23:39:18.180 DEBUG:boto:Token: None 2024-04-19T23:39:18.180 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wGDDLcdR3CAhKWHOocIejPwksN0= 2024-04-19T23:39:18.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wGDDLcdR3CAhKWHOocIejPwksN0=', 'Content-Length': '0'} 2024-04-19T23:39:18.182 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a09398e3f20f83c-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.182 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.182 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.182 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.183 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.183 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.183 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.183 DEBUG:boto:Data: 2024-04-19T23:39:18.183 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.183 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.183 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.183 DEBUG:boto:Params: {} 2024-04-19T23:39:18.183 DEBUG:boto:Token: None 2024-04-19T23:39:18.184 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.184 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wGDDLcdR3CAhKWHOocIejPwksN0= 2024-04-19T23:39:18.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wGDDLcdR3CAhKWHOocIejPwksN0=', 'Content-Length': '0'} 2024-04-19T23:39:18.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008388060c206c35ed-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.186 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.186 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.186 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.187 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.187 DEBUG:boto:Method: GET 2024-04-19T23:39:18.187 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.187 DEBUG:boto:Data: 2024-04-19T23:39:18.187 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.187 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.187 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.187 DEBUG:boto:Params: {} 2024-04-19T23:39:18.187 DEBUG:boto:Token: None 2024-04-19T23:39:18.187 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.188 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SDjsyISMlEYcs83N393y6HHOZ7o= 2024-04-19T23:39:18.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SDjsyISMlEYcs83N393y6HHOZ7o=', 'Content-Length': '0'} 2024-04-19T23:39:18.189 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c19a0961c97889b-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.190 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.190 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.190 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.190 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.190 DEBUG:boto:Method: GET 2024-04-19T23:39:18.190 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.190 DEBUG:boto:Data: 2024-04-19T23:39:18.190 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.191 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.191 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.191 DEBUG:boto:Params: {} 2024-04-19T23:39:18.191 DEBUG:boto:Token: None 2024-04-19T23:39:18.191 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:18.191 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SDjsyISMlEYcs83N393y6HHOZ7o= 2024-04-19T23:39:18.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SDjsyISMlEYcs83N393y6HHOZ7o=', 'Content-Length': '0'} 2024-04-19T23:39:18.193 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014ed1b15e1666d79-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.193 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.194 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.194 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.194 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.194 DEBUG:boto:Method: GET 2024-04-19T23:39:18.194 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.195 DEBUG:boto:Data: 2024-04-19T23:39:18.195 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.195 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.195 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.195 DEBUG:boto:Params: {} 2024-04-19T23:39:18.196 DEBUG:boto:Token: None 2024-04-19T23:39:18.196 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.196 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009537adb29ffcf1b2-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.200 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.200 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.200 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.201 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.201 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.201 DEBUG:boto:Method: GET 2024-04-19T23:39:18.201 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.201 DEBUG:boto:Data: 2024-04-19T23:39:18.201 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.202 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.202 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.202 DEBUG:boto:Params: {} 2024-04-19T23:39:18.202 DEBUG:boto:Token: None 2024-04-19T23:39:18.202 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.203 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bfc97f338c699c7-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.206 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.206 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.206 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.206 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.206 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.207 DEBUG:boto:Data: 2024-04-19T23:39:18.207 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.207 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.207 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.207 DEBUG:boto:Params: {} 2024-04-19T23:39:18.207 DEBUG:boto:Token: None 2024-04-19T23:39:18.207 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.207 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cb7604ba3491913-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.211 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.211 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.211 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.211 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.211 DEBUG:boto:Data: 2024-04-19T23:39:18.211 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.212 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.212 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.212 DEBUG:boto:Params: {} 2024-04-19T23:39:18.212 DEBUG:boto:Token: None 2024-04-19T23:39:18.212 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.212 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a3060be137defe7-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.216 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.216 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.217 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.217 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.217 DEBUG:boto:Method: GET 2024-04-19T23:39:18.217 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.218 DEBUG:boto:Data: 2024-04-19T23:39:18.218 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.218 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.218 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.219 DEBUG:boto:Params: {} 2024-04-19T23:39:18.219 DEBUG:boto:Token: None 2024-04-19T23:39:18.219 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.219 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000774ccaefc80d3bfd-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.262 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.262 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.263 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.263 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.263 DEBUG:boto:Method: GET 2024-04-19T23:39:18.263 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.263 DEBUG:boto:Data: 2024-04-19T23:39:18.263 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.263 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.263 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.263 DEBUG:boto:Params: {} 2024-04-19T23:39:18.263 DEBUG:boto:Token: None 2024-04-19T23:39:18.264 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.264 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016f441109fe69976-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.266 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.266 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.266 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.266 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.267 DEBUG:boto:Method: GET 2024-04-19T23:39:18.267 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.267 DEBUG:boto:Data: 2024-04-19T23:39:18.267 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.267 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.267 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.267 DEBUG:boto:Params: {} 2024-04-19T23:39:18.267 DEBUG:boto:Token: None 2024-04-19T23:39:18.267 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.267 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c88bca0b80f55a5-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.270 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.271 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.271 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.271 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.271 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.271 DEBUG:boto:Method: GET 2024-04-19T23:39:18.271 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.271 DEBUG:boto:Data: 2024-04-19T23:39:18.271 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.272 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.272 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.272 DEBUG:boto:Params: {} 2024-04-19T23:39:18.272 DEBUG:boto:Token: None 2024-04-19T23:39:18.272 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.272 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000732a707f01d74015-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.275 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.276 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.276 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.276 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.276 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.276 DEBUG:boto:Method: GET 2024-04-19T23:39:18.276 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.276 DEBUG:boto:Data: 2024-04-19T23:39:18.276 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.277 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.277 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.277 DEBUG:boto:Params: {} 2024-04-19T23:39:18.277 DEBUG:boto:Token: None 2024-04-19T23:39:18.277 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.277 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0FfTGcsgMZcjzTMND9yC53P1vhk= 2024-04-19T23:39:18.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0FfTGcsgMZcjzTMND9yC53P1vhk=', 'Content-Length': '0'} 2024-04-19T23:39:18.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a7493328d06235a-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.279 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.279 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.280 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.280 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.280 DEBUG:boto:Method: GET 2024-04-19T23:39:18.280 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.280 DEBUG:boto:Data: 2024-04-19T23:39:18.280 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.280 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.280 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.280 DEBUG:boto:Params: {} 2024-04-19T23:39:18.280 DEBUG:boto:Token: None 2024-04-19T23:39:18.280 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.281 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0FfTGcsgMZcjzTMND9yC53P1vhk= 2024-04-19T23:39:18.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0FfTGcsgMZcjzTMND9yC53P1vhk=', 'Content-Length': '0'} 2024-04-19T23:39:18.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7c88cbe4001fadd-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.283 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.283 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.283 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.283 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.283 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.283 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.283 DEBUG:boto:Data: 2024-04-19T23:39:18.283 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.283 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.283 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.283 DEBUG:boto:Params: {} 2024-04-19T23:39:18.284 DEBUG:boto:Token: None 2024-04-19T23:39:18.284 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.284 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mMsL0h87lPyzVuCgCKbwc6oTbGk= 2024-04-19T23:39:18.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mMsL0h87lPyzVuCgCKbwc6oTbGk=', 'Content-Length': '0'} 2024-04-19T23:39:18.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000883ecb590437aa05-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.286 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.286 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.286 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.286 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.287 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.287 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.287 DEBUG:boto:Data: 2024-04-19T23:39:18.287 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.287 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.287 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.287 DEBUG:boto:Params: {} 2024-04-19T23:39:18.287 DEBUG:boto:Token: None 2024-04-19T23:39:18.287 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.287 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mMsL0h87lPyzVuCgCKbwc6oTbGk= 2024-04-19T23:39:18.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mMsL0h87lPyzVuCgCKbwc6oTbGk=', 'Content-Length': '0'} 2024-04-19T23:39:18.289 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000176dfa3a8cd69e17-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.290 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.290 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.290 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.290 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.290 DEBUG:boto:Method: GET 2024-04-19T23:39:18.290 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.290 DEBUG:boto:Data: 2024-04-19T23:39:18.290 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.290 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.291 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.291 DEBUG:boto:Params: {} 2024-04-19T23:39:18.291 DEBUG:boto:Token: None 2024-04-19T23:39:18.291 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.291 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0FfTGcsgMZcjzTMND9yC53P1vhk= 2024-04-19T23:39:18.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0FfTGcsgMZcjzTMND9yC53P1vhk=', 'Content-Length': '0'} 2024-04-19T23:39:18.293 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008eb7d1f151952c2-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.334 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.335 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.335 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.335 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.335 DEBUG:boto:Method: GET 2024-04-19T23:39:18.335 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.335 DEBUG:boto:Data: 2024-04-19T23:39:18.335 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.335 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.335 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.335 DEBUG:boto:Params: {} 2024-04-19T23:39:18.335 DEBUG:boto:Token: None 2024-04-19T23:39:18.336 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:18.336 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0FfTGcsgMZcjzTMND9yC53P1vhk= 2024-04-19T23:39:18.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0FfTGcsgMZcjzTMND9yC53P1vhk=', 'Content-Length': '0'} 2024-04-19T23:39:18.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032bde750d84407c2-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.338 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.338 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.339 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.339 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.339 DEBUG:boto:Method: GET 2024-04-19T23:39:18.339 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.339 DEBUG:boto:Data: 2024-04-19T23:39:18.339 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.339 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.339 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.339 DEBUG:boto:Params: {} 2024-04-19T23:39:18.339 DEBUG:boto:Token: None 2024-04-19T23:39:18.339 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.340 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e39c9afc6e977c72-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.343 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.343 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.343 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.343 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.343 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.343 DEBUG:boto:Method: GET 2024-04-19T23:39:18.343 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.344 DEBUG:boto:Data: 2024-04-19T23:39:18.344 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.344 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.344 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.344 DEBUG:boto:Params: {} 2024-04-19T23:39:18.344 DEBUG:boto:Token: None 2024-04-19T23:39:18.344 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3fb8ba21f2cf838-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.348 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.349 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.349 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.349 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.349 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.349 DEBUG:boto:Data: 2024-04-19T23:39:18.349 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.349 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.349 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.349 DEBUG:boto:Params: {} 2024-04-19T23:39:18.350 DEBUG:boto:Token: None 2024-04-19T23:39:18.350 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.350 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000448dfec97a6e0fa9-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.354 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.354 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.354 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.354 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.354 DEBUG:boto:Data: 2024-04-19T23:39:18.354 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.354 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.354 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.354 DEBUG:boto:Params: {} 2024-04-19T23:39:18.355 DEBUG:boto:Token: None 2024-04-19T23:39:18.355 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.355 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.358 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd6de0007b66d7bb-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.359 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.359 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.359 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.359 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.359 DEBUG:boto:Method: GET 2024-04-19T23:39:18.359 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.359 DEBUG:boto:Data: 2024-04-19T23:39:18.359 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.359 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.360 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.360 DEBUG:boto:Params: {} 2024-04-19T23:39:18.360 DEBUG:boto:Token: None 2024-04-19T23:39:18.360 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.360 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000338833e597eeedf-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.362 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.362 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.362 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.362 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.362 DEBUG:boto:Method: GET 2024-04-19T23:39:18.363 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.363 DEBUG:boto:Data: 2024-04-19T23:39:18.363 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.363 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.363 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.363 DEBUG:boto:Params: {} 2024-04-19T23:39:18.363 DEBUG:boto:Token: None 2024-04-19T23:39:18.363 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.363 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd09f9bfbfb34730-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.365 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.366 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.366 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.366 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.366 DEBUG:boto:Method: GET 2024-04-19T23:39:18.366 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.366 DEBUG:boto:Data: 2024-04-19T23:39:18.366 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.366 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.366 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.366 DEBUG:boto:Params: {} 2024-04-19T23:39:18.367 DEBUG:boto:Token: None 2024-04-19T23:39:18.367 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.367 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0712b72e9f63102-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.370 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.370 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.370 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.370 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.370 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.370 DEBUG:boto:Method: GET 2024-04-19T23:39:18.371 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.371 DEBUG:boto:Data: 2024-04-19T23:39:18.371 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.371 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.371 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.371 DEBUG:boto:Params: {} 2024-04-19T23:39:18.371 DEBUG:boto:Token: None 2024-04-19T23:39:18.371 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.371 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f01534ebc039d89e-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.374 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.375 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.375 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.375 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.375 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.375 DEBUG:boto:Method: GET 2024-04-19T23:39:18.375 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.375 DEBUG:boto:Data: 2024-04-19T23:39:18.375 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.375 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.375 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.376 DEBUG:boto:Params: {} 2024-04-19T23:39:18.376 DEBUG:boto:Token: None 2024-04-19T23:39:18.376 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.376 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NIe8IQDnCs2J9t3CmoHhbClFoRo= 2024-04-19T23:39:18.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NIe8IQDnCs2J9t3CmoHhbClFoRo=', 'Content-Length': '0'} 2024-04-19T23:39:18.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a13ea843ac3f433-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.378 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.378 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.378 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.378 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.379 DEBUG:boto:Method: GET 2024-04-19T23:39:18.379 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.379 DEBUG:boto:Data: 2024-04-19T23:39:18.379 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.379 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.379 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.379 DEBUG:boto:Params: {} 2024-04-19T23:39:18.379 DEBUG:boto:Token: None 2024-04-19T23:39:18.379 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.379 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NIe8IQDnCs2J9t3CmoHhbClFoRo= 2024-04-19T23:39:18.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NIe8IQDnCs2J9t3CmoHhbClFoRo=', 'Content-Length': '0'} 2024-04-19T23:39:18.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007913761e751384c0-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.426 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.427 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.427 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.427 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.427 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.427 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.427 DEBUG:boto:Data: 2024-04-19T23:39:18.427 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.427 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.427 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.427 DEBUG:boto:Params: {} 2024-04-19T23:39:18.428 DEBUG:boto:Token: None 2024-04-19T23:39:18.428 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.428 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0CYNOZcy8+eJOUV0aU1JhX46SH4= 2024-04-19T23:39:18.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0CYNOZcy8+eJOUV0aU1JhX46SH4=', 'Content-Length': '0'} 2024-04-19T23:39:18.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc6f8c942d31c0f7-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.430 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.430 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.430 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.430 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.431 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.431 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.431 DEBUG:boto:Data: 2024-04-19T23:39:18.431 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.431 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.431 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.431 DEBUG:boto:Params: {} 2024-04-19T23:39:18.431 DEBUG:boto:Token: None 2024-04-19T23:39:18.431 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.431 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0CYNOZcy8+eJOUV0aU1JhX46SH4= 2024-04-19T23:39:18.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0CYNOZcy8+eJOUV0aU1JhX46SH4=', 'Content-Length': '0'} 2024-04-19T23:39:18.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc757c8b38b48363-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.434 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.434 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.434 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.434 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.434 DEBUG:boto:Method: GET 2024-04-19T23:39:18.434 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.434 DEBUG:boto:Data: 2024-04-19T23:39:18.434 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.434 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.435 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.435 DEBUG:boto:Params: {} 2024-04-19T23:39:18.435 DEBUG:boto:Token: None 2024-04-19T23:39:18.435 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.435 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NIe8IQDnCs2J9t3CmoHhbClFoRo= 2024-04-19T23:39:18.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NIe8IQDnCs2J9t3CmoHhbClFoRo=', 'Content-Length': '0'} 2024-04-19T23:39:18.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b086bec27a936dfb-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.437 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.437 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.438 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.438 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.438 DEBUG:boto:Method: GET 2024-04-19T23:39:18.438 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.438 DEBUG:boto:Data: 2024-04-19T23:39:18.438 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.438 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.438 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.438 DEBUG:boto:Params: {} 2024-04-19T23:39:18.438 DEBUG:boto:Token: None 2024-04-19T23:39:18.439 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:18.439 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NIe8IQDnCs2J9t3CmoHhbClFoRo= 2024-04-19T23:39:18.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NIe8IQDnCs2J9t3CmoHhbClFoRo=', 'Content-Length': '0'} 2024-04-19T23:39:18.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077484de57ac4b76f-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.441 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.441 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.441 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.441 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.441 DEBUG:boto:Method: GET 2024-04-19T23:39:18.441 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.442 DEBUG:boto:Data: 2024-04-19T23:39:18.442 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.442 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.442 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.442 DEBUG:boto:Params: {} 2024-04-19T23:39:18.442 DEBUG:boto:Token: None 2024-04-19T23:39:18.442 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.442 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007278aed9585d9508-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.446 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.446 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.446 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.446 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.446 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.447 DEBUG:boto:Method: GET 2024-04-19T23:39:18.447 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.447 DEBUG:boto:Data: 2024-04-19T23:39:18.447 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.447 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.447 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.447 DEBUG:boto:Params: {} 2024-04-19T23:39:18.447 DEBUG:boto:Token: None 2024-04-19T23:39:18.447 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.448 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b49ecbe1783f626-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.451 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.451 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.451 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.451 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.451 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.451 DEBUG:boto:Data: 2024-04-19T23:39:18.451 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.452 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.452 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.452 DEBUG:boto:Params: {} 2024-04-19T23:39:18.452 DEBUG:boto:Token: None 2024-04-19T23:39:18.452 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.452 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c76f285787cb986-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.456 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.456 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.456 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.456 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.456 DEBUG:boto:Data: 2024-04-19T23:39:18.456 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.456 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.456 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.456 DEBUG:boto:Params: {} 2024-04-19T23:39:18.456 DEBUG:boto:Token: None 2024-04-19T23:39:18.457 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.457 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df70a2cc39e12a4b-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.461 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.461 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.461 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.461 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.461 DEBUG:boto:Method: GET 2024-04-19T23:39:18.461 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.461 DEBUG:boto:Data: 2024-04-19T23:39:18.461 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.461 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.461 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.461 DEBUG:boto:Params: {} 2024-04-19T23:39:18.462 DEBUG:boto:Token: None 2024-04-19T23:39:18.462 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.462 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a77930bdecfc09e5-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.510 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.510 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.511 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.511 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.511 DEBUG:boto:Method: GET 2024-04-19T23:39:18.511 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.511 DEBUG:boto:Data: 2024-04-19T23:39:18.511 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.511 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.511 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.511 DEBUG:boto:Params: {} 2024-04-19T23:39:18.511 DEBUG:boto:Token: None 2024-04-19T23:39:18.512 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.512 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea1d2154947da233-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.515 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.515 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.515 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.515 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.515 DEBUG:boto:Method: GET 2024-04-19T23:39:18.515 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.515 DEBUG:boto:Data: 2024-04-19T23:39:18.515 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.515 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.515 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.515 DEBUG:boto:Params: {} 2024-04-19T23:39:18.516 DEBUG:boto:Token: None 2024-04-19T23:39:18.516 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.516 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092700ec92bf782d9-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.519 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.519 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.519 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.519 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.520 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.520 DEBUG:boto:Method: GET 2024-04-19T23:39:18.520 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.520 DEBUG:boto:Data: 2024-04-19T23:39:18.520 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.520 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.520 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.520 DEBUG:boto:Params: {} 2024-04-19T23:39:18.520 DEBUG:boto:Token: None 2024-04-19T23:39:18.520 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.521 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de462d9629a0a57d-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.566 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.567 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.567 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.567 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.567 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.567 DEBUG:boto:Method: GET 2024-04-19T23:39:18.568 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.568 DEBUG:boto:Data: 2024-04-19T23:39:18.568 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.568 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.568 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.568 DEBUG:boto:Params: {} 2024-04-19T23:39:18.568 DEBUG:boto:Token: None 2024-04-19T23:39:18.568 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.568 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZDHPFesmuaye+8tjtAm8wy7gEAg= 2024-04-19T23:39:18.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZDHPFesmuaye+8tjtAm8wy7gEAg=', 'Content-Length': '0'} 2024-04-19T23:39:18.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000389d25dac7ccbfac-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.571 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.571 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.571 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.571 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.572 DEBUG:boto:Method: GET 2024-04-19T23:39:18.572 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.572 DEBUG:boto:Data: 2024-04-19T23:39:18.572 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.572 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.572 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.572 DEBUG:boto:Params: {} 2024-04-19T23:39:18.572 DEBUG:boto:Token: None 2024-04-19T23:39:18.573 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.573 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZDHPFesmuaye+8tjtAm8wy7gEAg= 2024-04-19T23:39:18.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZDHPFesmuaye+8tjtAm8wy7gEAg=', 'Content-Length': '0'} 2024-04-19T23:39:18.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b311ed4eefa0cab4-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.575 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.575 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.576 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.576 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.576 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.576 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.576 DEBUG:boto:Data: 2024-04-19T23:39:18.576 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.576 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.576 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.576 DEBUG:boto:Params: {} 2024-04-19T23:39:18.576 DEBUG:boto:Token: None 2024-04-19T23:39:18.576 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.577 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fnMzEdeoJ6r62px0tXbKFfG2TvM= 2024-04-19T23:39:18.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fnMzEdeoJ6r62px0tXbKFfG2TvM=', 'Content-Length': '0'} 2024-04-19T23:39:18.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059de570ec0d33a1d-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.579 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.579 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.579 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.579 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.579 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.579 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.580 DEBUG:boto:Data: 2024-04-19T23:39:18.580 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.580 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.580 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.580 DEBUG:boto:Params: {} 2024-04-19T23:39:18.580 DEBUG:boto:Token: None 2024-04-19T23:39:18.580 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.580 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fnMzEdeoJ6r62px0tXbKFfG2TvM= 2024-04-19T23:39:18.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fnMzEdeoJ6r62px0tXbKFfG2TvM=', 'Content-Length': '0'} 2024-04-19T23:39:18.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002aaf4a5409b6b14-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.583 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.583 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.583 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.583 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.583 DEBUG:boto:Method: GET 2024-04-19T23:39:18.583 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.584 DEBUG:boto:Data: 2024-04-19T23:39:18.584 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.584 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.584 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.584 DEBUG:boto:Params: {} 2024-04-19T23:39:18.584 DEBUG:boto:Token: None 2024-04-19T23:39:18.584 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.584 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZDHPFesmuaye+8tjtAm8wy7gEAg= 2024-04-19T23:39:18.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZDHPFesmuaye+8tjtAm8wy7gEAg=', 'Content-Length': '0'} 2024-04-19T23:39:18.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054dbfff81b954353-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.630 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.631 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.631 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.631 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.631 DEBUG:boto:Method: GET 2024-04-19T23:39:18.631 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.631 DEBUG:boto:Data: 2024-04-19T23:39:18.631 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.631 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.631 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.631 DEBUG:boto:Params: {} 2024-04-19T23:39:18.632 DEBUG:boto:Token: None 2024-04-19T23:39:18.632 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:18.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZDHPFesmuaye+8tjtAm8wy7gEAg= 2024-04-19T23:39:18.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZDHPFesmuaye+8tjtAm8wy7gEAg=', 'Content-Length': '0'} 2024-04-19T23:39:18.634 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003516cdd403ebb508-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.635 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.635 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.635 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.635 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.635 DEBUG:boto:Method: GET 2024-04-19T23:39:18.635 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.635 DEBUG:boto:Data: 2024-04-19T23:39:18.636 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.636 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.636 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.636 DEBUG:boto:Params: {} 2024-04-19T23:39:18.636 DEBUG:boto:Token: None 2024-04-19T23:39:18.636 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.636 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d40a129772705047-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.640 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.640 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.640 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.640 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.640 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.640 DEBUG:boto:Method: GET 2024-04-19T23:39:18.640 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.641 DEBUG:boto:Data: 2024-04-19T23:39:18.641 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.641 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.641 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.641 DEBUG:boto:Params: {} 2024-04-19T23:39:18.641 DEBUG:boto:Token: None 2024-04-19T23:39:18.641 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.641 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043aedaf48a97b432-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.644 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.645 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.645 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.645 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.645 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.645 DEBUG:boto:Data: 2024-04-19T23:39:18.645 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.645 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.645 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.645 DEBUG:boto:Params: {} 2024-04-19T23:39:18.646 DEBUG:boto:Token: None 2024-04-19T23:39:18.646 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.646 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.649 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f970bf824c9af538-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.649 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.650 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.650 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.650 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.650 DEBUG:boto:Data: 2024-04-19T23:39:18.650 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.650 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.650 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.650 DEBUG:boto:Params: {} 2024-04-19T23:39:18.650 DEBUG:boto:Token: None 2024-04-19T23:39:18.651 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.651 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c428bad60b64dd8e-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.654 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.655 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.655 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.655 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.655 DEBUG:boto:Method: GET 2024-04-19T23:39:18.655 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.655 DEBUG:boto:Data: 2024-04-19T23:39:18.655 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.655 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.655 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.656 DEBUG:boto:Params: {} 2024-04-19T23:39:18.656 DEBUG:boto:Token: None 2024-04-19T23:39:18.656 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.656 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000296799f97bbddc63-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.658 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.658 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.659 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.659 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.659 DEBUG:boto:Method: GET 2024-04-19T23:39:18.659 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.659 DEBUG:boto:Data: 2024-04-19T23:39:18.659 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.659 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.659 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.659 DEBUG:boto:Params: {} 2024-04-19T23:39:18.659 DEBUG:boto:Token: None 2024-04-19T23:39:18.660 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007075b4773d60cafe-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.702 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.703 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.703 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.703 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.703 DEBUG:boto:Method: GET 2024-04-19T23:39:18.703 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.703 DEBUG:boto:Data: 2024-04-19T23:39:18.703 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.703 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.703 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.703 DEBUG:boto:Params: {} 2024-04-19T23:39:18.704 DEBUG:boto:Token: None 2024-04-19T23:39:18.704 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3443c95bcbd0a6e-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.708 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.708 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.708 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.708 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.708 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.709 DEBUG:boto:Method: GET 2024-04-19T23:39:18.709 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.709 DEBUG:boto:Data: 2024-04-19T23:39:18.709 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.709 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.709 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.709 DEBUG:boto:Params: {} 2024-04-19T23:39:18.709 DEBUG:boto:Token: None 2024-04-19T23:39:18.709 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.709 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f076606f9594b784-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.713 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.714 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.714 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.714 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.715 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.715 DEBUG:boto:Method: GET 2024-04-19T23:39:18.715 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.715 DEBUG:boto:Data: 2024-04-19T23:39:18.715 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.715 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.715 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.715 DEBUG:boto:Params: {} 2024-04-19T23:39:18.715 DEBUG:boto:Token: None 2024-04-19T23:39:18.715 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.716 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fJHQRP9673EJtKx57r5bHuEPsEk= 2024-04-19T23:39:18.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fJHQRP9673EJtKx57r5bHuEPsEk=', 'Content-Length': '0'} 2024-04-19T23:39:18.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1fb3ea2e3aff5cf-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.758 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.759 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.759 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.759 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.759 DEBUG:boto:Method: GET 2024-04-19T23:39:18.759 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.759 DEBUG:boto:Data: 2024-04-19T23:39:18.759 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.759 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.759 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.759 DEBUG:boto:Params: {} 2024-04-19T23:39:18.760 DEBUG:boto:Token: None 2024-04-19T23:39:18.760 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.760 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fJHQRP9673EJtKx57r5bHuEPsEk= 2024-04-19T23:39:18.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fJHQRP9673EJtKx57r5bHuEPsEk=', 'Content-Length': '0'} 2024-04-19T23:39:18.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029236dc3056930dc-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.762 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.763 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.763 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.763 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.763 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.763 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.763 DEBUG:boto:Data: 2024-04-19T23:39:18.763 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.763 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.764 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.764 DEBUG:boto:Params: {} 2024-04-19T23:39:18.764 DEBUG:boto:Token: None 2024-04-19T23:39:18.764 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.764 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y2R/4ue9jiMMuoyN7jEKj00UMh8= 2024-04-19T23:39:18.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y2R/4ue9jiMMuoyN7jEKj00UMh8=', 'Content-Length': '0'} 2024-04-19T23:39:18.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000333a29ae1224ce23-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.767 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.767 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.767 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.767 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.767 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.767 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.767 DEBUG:boto:Data: 2024-04-19T23:39:18.767 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.768 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.768 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.768 DEBUG:boto:Params: {} 2024-04-19T23:39:18.768 DEBUG:boto:Token: None 2024-04-19T23:39:18.768 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.768 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y2R/4ue9jiMMuoyN7jEKj00UMh8= 2024-04-19T23:39:18.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y2R/4ue9jiMMuoyN7jEKj00UMh8=', 'Content-Length': '0'} 2024-04-19T23:39:18.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000104f31eb3d70014e-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.770 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.770 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.771 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.771 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.771 DEBUG:boto:Method: GET 2024-04-19T23:39:18.771 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.771 DEBUG:boto:Data: 2024-04-19T23:39:18.771 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.771 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.771 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.771 DEBUG:boto:Params: {} 2024-04-19T23:39:18.771 DEBUG:boto:Token: None 2024-04-19T23:39:18.772 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.772 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fJHQRP9673EJtKx57r5bHuEPsEk= 2024-04-19T23:39:18.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fJHQRP9673EJtKx57r5bHuEPsEk=', 'Content-Length': '0'} 2024-04-19T23:39:18.773 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031aeebf15f4d7e91-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.774 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.774 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.774 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.774 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.774 DEBUG:boto:Method: GET 2024-04-19T23:39:18.774 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.774 DEBUG:boto:Data: 2024-04-19T23:39:18.774 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.774 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.775 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.775 DEBUG:boto:Params: {} 2024-04-19T23:39:18.775 DEBUG:boto:Token: None 2024-04-19T23:39:18.775 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:18.775 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fJHQRP9673EJtKx57r5bHuEPsEk= 2024-04-19T23:39:18.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fJHQRP9673EJtKx57r5bHuEPsEk=', 'Content-Length': '0'} 2024-04-19T23:39:18.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018810eca9fe08277-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.777 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.778 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.778 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.778 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.778 DEBUG:boto:Method: GET 2024-04-19T23:39:18.778 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.778 DEBUG:boto:Data: 2024-04-19T23:39:18.778 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.778 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.778 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.778 DEBUG:boto:Params: {} 2024-04-19T23:39:18.778 DEBUG:boto:Token: None 2024-04-19T23:39:18.779 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.779 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a942ed5f2cd2f544-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.782 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.782 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.783 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.783 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.783 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.783 DEBUG:boto:Method: GET 2024-04-19T23:39:18.783 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.783 DEBUG:boto:Data: 2024-04-19T23:39:18.783 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.784 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.784 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.784 DEBUG:boto:Params: {} 2024-04-19T23:39:18.784 DEBUG:boto:Token: None 2024-04-19T23:39:18.784 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009239da89df902b6c-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.787 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.788 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.788 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.788 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.788 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.788 DEBUG:boto:Data: 2024-04-19T23:39:18.788 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.788 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.788 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.788 DEBUG:boto:Params: {} 2024-04-19T23:39:18.789 DEBUG:boto:Token: None 2024-04-19T23:39:18.789 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.789 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004654b79f2e74bfb2-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.793 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.793 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.793 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.793 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.793 DEBUG:boto:Data: 2024-04-19T23:39:18.793 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.794 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.794 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.794 DEBUG:boto:Params: {} 2024-04-19T23:39:18.794 DEBUG:boto:Token: None 2024-04-19T23:39:18.794 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.794 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c9399efea7206fd-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.798 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.798 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.799 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.799 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.799 DEBUG:boto:Method: GET 2024-04-19T23:39:18.799 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.799 DEBUG:boto:Data: 2024-04-19T23:39:18.799 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.799 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.799 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.799 DEBUG:boto:Params: {} 2024-04-19T23:39:18.799 DEBUG:boto:Token: None 2024-04-19T23:39:18.800 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.800 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ae82b1035275886-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.842 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.842 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.843 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.843 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.843 DEBUG:boto:Method: GET 2024-04-19T23:39:18.843 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.843 DEBUG:boto:Data: 2024-04-19T23:39:18.843 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.843 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.843 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.843 DEBUG:boto:Params: {} 2024-04-19T23:39:18.843 DEBUG:boto:Token: None 2024-04-19T23:39:18.843 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.844 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c09a9e9ecc036d3-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.846 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.846 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.846 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.846 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.846 DEBUG:boto:Method: GET 2024-04-19T23:39:18.846 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.846 DEBUG:boto:Data: 2024-04-19T23:39:18.846 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.847 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.847 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.847 DEBUG:boto:Params: {} 2024-04-19T23:39:18.847 DEBUG:boto:Token: None 2024-04-19T23:39:18.847 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.847 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b12e7f24029ea15-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.850 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.851 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.851 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.851 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.851 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.851 DEBUG:boto:Method: GET 2024-04-19T23:39:18.851 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.851 DEBUG:boto:Data: 2024-04-19T23:39:18.851 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.852 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.852 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.852 DEBUG:boto:Params: {} 2024-04-19T23:39:18.852 DEBUG:boto:Token: None 2024-04-19T23:39:18.852 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.852 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000370312967967606-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.855 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.857 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.857 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.857 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.857 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.857 DEBUG:boto:Method: GET 2024-04-19T23:39:18.857 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.857 DEBUG:boto:Data: 2024-04-19T23:39:18.858 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.858 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.858 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.858 DEBUG:boto:Params: {} 2024-04-19T23:39:18.858 DEBUG:boto:Token: None 2024-04-19T23:39:18.858 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.858 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KP2sKJIHrisFolbE5DX0AsbRjIg= 2024-04-19T23:39:18.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KP2sKJIHrisFolbE5DX0AsbRjIg=', 'Content-Length': '0'} 2024-04-19T23:39:18.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a69615bcf48f7ce-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.861 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.861 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.861 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.861 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.862 DEBUG:boto:Method: GET 2024-04-19T23:39:18.862 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.862 DEBUG:boto:Data: 2024-04-19T23:39:18.862 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.862 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.862 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.862 DEBUG:boto:Params: {} 2024-04-19T23:39:18.862 DEBUG:boto:Token: None 2024-04-19T23:39:18.862 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.862 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KP2sKJIHrisFolbE5DX0AsbRjIg= 2024-04-19T23:39:18.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KP2sKJIHrisFolbE5DX0AsbRjIg=', 'Content-Length': '0'} 2024-04-19T23:39:18.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000760c4c5feaa4f514-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.865 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.865 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.865 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.865 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.865 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.865 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.865 DEBUG:boto:Data: 2024-04-19T23:39:18.866 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.866 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.866 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.866 DEBUG:boto:Params: {} 2024-04-19T23:39:18.866 DEBUG:boto:Token: None 2024-04-19T23:39:18.866 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.866 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:68wEUQjlkSt3iIUsKcQRBWrbZjI= 2024-04-19T23:39:18.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:68wEUQjlkSt3iIUsKcQRBWrbZjI=', 'Content-Length': '0'} 2024-04-19T23:39:18.868 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ef255c313f14d69-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.868 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.868 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.868 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.869 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.869 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.869 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.869 DEBUG:boto:Data: 2024-04-19T23:39:18.869 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.869 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.869 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.869 DEBUG:boto:Params: {} 2024-04-19T23:39:18.869 DEBUG:boto:Token: None 2024-04-19T23:39:18.869 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.870 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:68wEUQjlkSt3iIUsKcQRBWrbZjI= 2024-04-19T23:39:18.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:68wEUQjlkSt3iIUsKcQRBWrbZjI=', 'Content-Length': '0'} 2024-04-19T23:39:18.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000865d0f9a75e4e786-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.872 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.872 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.872 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.872 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.872 DEBUG:boto:Method: GET 2024-04-19T23:39:18.872 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.873 DEBUG:boto:Data: 2024-04-19T23:39:18.873 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.873 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.873 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.873 DEBUG:boto:Params: {} 2024-04-19T23:39:18.873 DEBUG:boto:Token: None 2024-04-19T23:39:18.873 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.873 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KP2sKJIHrisFolbE5DX0AsbRjIg= 2024-04-19T23:39:18.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KP2sKJIHrisFolbE5DX0AsbRjIg=', 'Content-Length': '0'} 2024-04-19T23:39:18.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da776505bf38b650-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.918 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.919 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.919 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.919 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.919 DEBUG:boto:Method: GET 2024-04-19T23:39:18.919 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.919 DEBUG:boto:Data: 2024-04-19T23:39:18.919 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.919 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.919 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.919 DEBUG:boto:Params: {} 2024-04-19T23:39:18.919 DEBUG:boto:Token: None 2024-04-19T23:39:18.920 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:18.920 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KP2sKJIHrisFolbE5DX0AsbRjIg= 2024-04-19T23:39:18.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KP2sKJIHrisFolbE5DX0AsbRjIg=', 'Content-Length': '0'} 2024-04-19T23:39:18.922 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e4ac5fa04ec88fc-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.922 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.922 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.923 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.923 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.923 DEBUG:boto:Method: GET 2024-04-19T23:39:18.923 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.923 DEBUG:boto:Data: 2024-04-19T23:39:18.923 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.923 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.924 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.924 DEBUG:boto:Params: {} 2024-04-19T23:39:18.924 DEBUG:boto:Token: None 2024-04-19T23:39:18.924 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.924 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2d1f84aa2c3b03d-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.927 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.928 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.928 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.928 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.928 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.928 DEBUG:boto:Method: GET 2024-04-19T23:39:18.928 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.928 DEBUG:boto:Data: 2024-04-19T23:39:18.928 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.928 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.928 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.929 DEBUG:boto:Params: {} 2024-04-19T23:39:18.929 DEBUG:boto:Token: None 2024-04-19T23:39:18.929 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.929 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001737fb635bd2eeba-00662300a6-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.932 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.932 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.932 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.933 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.933 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.933 DEBUG:boto:Data: 2024-04-19T23:39:18.933 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.933 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.933 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.933 DEBUG:boto:Params: {} 2024-04-19T23:39:18.933 DEBUG:boto:Token: None 2024-04-19T23:39:18.933 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.933 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017c6b4600dbd9dc9-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.937 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.937 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.937 DEBUG:boto:Method: HEAD 2024-04-19T23:39:18.937 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:18.937 DEBUG:boto:Data: 2024-04-19T23:39:18.937 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.938 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.938 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.938 DEBUG:boto:Params: {} 2024-04-19T23:39:18.938 DEBUG:boto:Token: None 2024-04-19T23:39:18.938 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj 2024-04-19T23:39:18.938 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs= 2024-04-19T23:39:18.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d4qjKMxAx3OWCnQnrN23D14T+Fs=', 'Content-Length': '0'} 2024-04-19T23:39:18.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6cd093f03035607-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:18.942 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.942 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.942 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.942 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.942 DEBUG:boto:Method: GET 2024-04-19T23:39:18.943 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.943 DEBUG:boto:Data: 2024-04-19T23:39:18.943 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.943 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.943 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.943 DEBUG:boto:Params: {} 2024-04-19T23:39:18.943 DEBUG:boto:Token: None 2024-04-19T23:39:18.943 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.943 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ddc7314909d73c3-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.986 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:18.987 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:18.987 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.987 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.987 DEBUG:boto:Method: GET 2024-04-19T23:39:18.987 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.987 DEBUG:boto:Data: 2024-04-19T23:39:18.987 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.987 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.987 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.987 DEBUG:boto:Params: {} 2024-04-19T23:39:18.988 DEBUG:boto:Token: None 2024-04-19T23:39:18.988 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:18.988 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc= 2024-04-19T23:39:18.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a750P5Qvj+VEF/21QkhOMIvqYdc=', 'Content-Length': '0'} 2024-04-19T23:39:18.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007540cb9b8ced2df6-00662300a6-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.990 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.990 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.990 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.990 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.991 DEBUG:boto:Method: GET 2024-04-19T23:39:18.991 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.991 DEBUG:boto:Data: 2024-04-19T23:39:18.991 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.991 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:18.991 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.991 DEBUG:boto:Params: {} 2024-04-19T23:39:18.991 DEBUG:boto:Token: None 2024-04-19T23:39:18.991 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.991 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de861245ee1e8ef9-00662300a6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.995 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:18.995 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:18.995 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:18.995 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.995 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:18.995 DEBUG:boto:Method: GET 2024-04-19T23:39:18.996 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:18.996 DEBUG:boto:Data: 2024-04-19T23:39:18.996 DEBUG:boto:Headers: {} 2024-04-19T23:39:18.996 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:18.996 DEBUG:boto:Port: 8000 2024-04-19T23:39:18.996 DEBUG:boto:Params: {} 2024-04-19T23:39:18.996 DEBUG:boto:Token: None 2024-04-19T23:39:18.996 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:18 GMT /aktlji-22/ 2024-04-19T23:39:18.996 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0= 2024-04-19T23:39:18.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:18 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHCNdeKEG6bh+RkAVrqx2qsw1I0=', 'Content-Length': '0'} 2024-04-19T23:39:18.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1b95e2c5ac913ea-00662300a7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:18.999 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.000 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.000 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.000 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.000 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.000 DEBUG:boto:Method: GET 2024-04-19T23:39:19.000 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.000 DEBUG:boto:Data: 2024-04-19T23:39:19.000 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.001 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.001 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.001 DEBUG:boto:Params: {} 2024-04-19T23:39:19.001 DEBUG:boto:Token: None 2024-04-19T23:39:19.001 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.001 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KpMtr3r8hLEqDVUxRiL32SF2rKo= 2024-04-19T23:39:19.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KpMtr3r8hLEqDVUxRiL32SF2rKo=', 'Content-Length': '0'} 2024-04-19T23:39:19.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009db93ed962b7e4b2-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.003 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.003 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.004 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.004 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.004 DEBUG:boto:Method: GET 2024-04-19T23:39:19.004 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.004 DEBUG:boto:Data: 2024-04-19T23:39:19.004 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.004 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.004 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.004 DEBUG:boto:Params: {} 2024-04-19T23:39:19.004 DEBUG:boto:Token: None 2024-04-19T23:39:19.004 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.005 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KpMtr3r8hLEqDVUxRiL32SF2rKo= 2024-04-19T23:39:19.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KpMtr3r8hLEqDVUxRiL32SF2rKo=', 'Content-Length': '0'} 2024-04-19T23:39:19.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064b4d78aa693dd13-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.050 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.050 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.051 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.051 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.051 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.051 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.051 DEBUG:boto:Data: 2024-04-19T23:39:19.051 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.051 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.051 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.051 DEBUG:boto:Params: {} 2024-04-19T23:39:19.051 DEBUG:boto:Token: None 2024-04-19T23:39:19.052 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.052 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3DkwadmKAd7kHg96051enpUGPT8= 2024-04-19T23:39:19.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3DkwadmKAd7kHg96051enpUGPT8=', 'Content-Length': '0'} 2024-04-19T23:39:19.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000414084b86091ceb6-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.054 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.054 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.054 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.054 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.054 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.054 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.054 DEBUG:boto:Data: 2024-04-19T23:39:19.054 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.054 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.055 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.055 DEBUG:boto:Params: {} 2024-04-19T23:39:19.055 DEBUG:boto:Token: None 2024-04-19T23:39:19.055 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.055 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3DkwadmKAd7kHg96051enpUGPT8= 2024-04-19T23:39:19.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3DkwadmKAd7kHg96051enpUGPT8=', 'Content-Length': '0'} 2024-04-19T23:39:19.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b61801a2ebeb006-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.057 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.057 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.057 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.058 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.058 DEBUG:boto:Method: GET 2024-04-19T23:39:19.058 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.058 DEBUG:boto:Data: 2024-04-19T23:39:19.058 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.058 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.058 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.058 DEBUG:boto:Params: {} 2024-04-19T23:39:19.059 DEBUG:boto:Token: None 2024-04-19T23:39:19.059 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.059 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KpMtr3r8hLEqDVUxRiL32SF2rKo= 2024-04-19T23:39:19.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KpMtr3r8hLEqDVUxRiL32SF2rKo=', 'Content-Length': '0'} 2024-04-19T23:39:19.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff6dde030e61cfa7-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.102 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.102 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.103 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.103 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.103 DEBUG:boto:Method: GET 2024-04-19T23:39:19.103 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.103 DEBUG:boto:Data: 2024-04-19T23:39:19.103 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.103 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.103 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.103 DEBUG:boto:Params: {} 2024-04-19T23:39:19.103 DEBUG:boto:Token: None 2024-04-19T23:39:19.104 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:19.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KpMtr3r8hLEqDVUxRiL32SF2rKo= 2024-04-19T23:39:19.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KpMtr3r8hLEqDVUxRiL32SF2rKo=', 'Content-Length': '0'} 2024-04-19T23:39:19.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e620c36f49a1ae6-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.106 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.106 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.106 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.106 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.107 DEBUG:boto:Method: GET 2024-04-19T23:39:19.107 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.107 DEBUG:boto:Data: 2024-04-19T23:39:19.107 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.107 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.107 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.107 DEBUG:boto:Params: {} 2024-04-19T23:39:19.107 DEBUG:boto:Token: None 2024-04-19T23:39:19.107 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.107 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ee03e9b4cdd656c-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.154 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.155 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.155 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.155 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.155 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.155 DEBUG:boto:Method: GET 2024-04-19T23:39:19.155 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.156 DEBUG:boto:Data: 2024-04-19T23:39:19.156 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.156 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.156 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.156 DEBUG:boto:Params: {} 2024-04-19T23:39:19.156 DEBUG:boto:Token: None 2024-04-19T23:39:19.156 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.156 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d25f096071487fab-00662300a7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.159 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.160 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.160 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.160 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.160 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:19.160 DEBUG:boto:Data: 2024-04-19T23:39:19.160 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.160 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.160 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.161 DEBUG:boto:Params: {} 2024-04-19T23:39:19.161 DEBUG:boto:Token: None 2024-04-19T23:39:19.161 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj 2024-04-19T23:39:19.161 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4= 2024-04-19T23:39:19.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4=', 'Content-Length': '0'} 2024-04-19T23:39:19.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ae5b14b3020fab1-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.165 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.165 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.165 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.165 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:19.166 DEBUG:boto:Data: 2024-04-19T23:39:19.166 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.166 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.166 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.166 DEBUG:boto:Params: {} 2024-04-19T23:39:19.166 DEBUG:boto:Token: None 2024-04-19T23:39:19.166 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj 2024-04-19T23:39:19.166 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4= 2024-04-19T23:39:19.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4=', 'Content-Length': '0'} 2024-04-19T23:39:19.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bcda72365fb3195-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.170 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.170 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.171 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.171 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.171 DEBUG:boto:Method: GET 2024-04-19T23:39:19.171 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.171 DEBUG:boto:Data: 2024-04-19T23:39:19.171 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.172 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.172 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.172 DEBUG:boto:Params: {} 2024-04-19T23:39:19.172 DEBUG:boto:Token: None 2024-04-19T23:39:19.172 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.172 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4= 2024-04-19T23:39:19.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4=', 'Content-Length': '0'} 2024-04-19T23:39:19.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a2f2e639661a5f3-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.174 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.174 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.174 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.175 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.175 DEBUG:boto:Method: GET 2024-04-19T23:39:19.175 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.175 DEBUG:boto:Data: 2024-04-19T23:39:19.175 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.175 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.175 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.175 DEBUG:boto:Params: {} 2024-04-19T23:39:19.175 DEBUG:boto:Token: None 2024-04-19T23:39:19.175 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.176 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4= 2024-04-19T23:39:19.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4=', 'Content-Length': '0'} 2024-04-19T23:39:19.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d76b2b439c217976-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.178 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.178 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.178 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.178 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.178 DEBUG:boto:Method: GET 2024-04-19T23:39:19.178 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.179 DEBUG:boto:Data: 2024-04-19T23:39:19.179 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.179 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.179 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.179 DEBUG:boto:Params: {} 2024-04-19T23:39:19.179 DEBUG:boto:Token: None 2024-04-19T23:39:19.179 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.179 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec55fc4650f6c46f-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.182 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.183 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.183 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.183 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.183 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.183 DEBUG:boto:Method: GET 2024-04-19T23:39:19.183 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.183 DEBUG:boto:Data: 2024-04-19T23:39:19.183 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.183 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.183 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.183 DEBUG:boto:Params: {} 2024-04-19T23:39:19.184 DEBUG:boto:Token: None 2024-04-19T23:39:19.184 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.184 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dffde365aace225-00662300a7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.187 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.187 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.187 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.187 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.188 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.188 DEBUG:boto:Method: GET 2024-04-19T23:39:19.188 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.188 DEBUG:boto:Data: 2024-04-19T23:39:19.188 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.188 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.188 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.188 DEBUG:boto:Params: {} 2024-04-19T23:39:19.188 DEBUG:boto:Token: None 2024-04-19T23:39:19.189 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.189 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rlgl611soGhLKCWdL9Bur1HP5Lc= 2024-04-19T23:39:19.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rlgl611soGhLKCWdL9Bur1HP5Lc=', 'Content-Length': '0'} 2024-04-19T23:39:19.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000727e85b456ffc15c-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.234 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.235 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.235 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.235 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.235 DEBUG:boto:Method: GET 2024-04-19T23:39:19.235 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.235 DEBUG:boto:Data: 2024-04-19T23:39:19.235 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.235 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.235 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.235 DEBUG:boto:Params: {} 2024-04-19T23:39:19.236 DEBUG:boto:Token: None 2024-04-19T23:39:19.236 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.236 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rlgl611soGhLKCWdL9Bur1HP5Lc= 2024-04-19T23:39:19.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rlgl611soGhLKCWdL9Bur1HP5Lc=', 'Content-Length': '0'} 2024-04-19T23:39:19.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bc535ac917a8026-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.238 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.239 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.239 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.239 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.239 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.239 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.239 DEBUG:boto:Data: 2024-04-19T23:39:19.239 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.240 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.240 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.240 DEBUG:boto:Params: {} 2024-04-19T23:39:19.240 DEBUG:boto:Token: None 2024-04-19T23:39:19.240 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.240 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pJFvMSIVeT/r2cKiT+2S1ORs+00= 2024-04-19T23:39:19.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pJFvMSIVeT/r2cKiT+2S1ORs+00=', 'Content-Length': '0'} 2024-04-19T23:39:19.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2058d812d07a443-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.242 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.242 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.243 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.243 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.243 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.243 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.243 DEBUG:boto:Data: 2024-04-19T23:39:19.243 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.243 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.243 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.243 DEBUG:boto:Params: {} 2024-04-19T23:39:19.243 DEBUG:boto:Token: None 2024-04-19T23:39:19.244 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.244 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pJFvMSIVeT/r2cKiT+2S1ORs+00= 2024-04-19T23:39:19.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pJFvMSIVeT/r2cKiT+2S1ORs+00=', 'Content-Length': '0'} 2024-04-19T23:39:19.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001978f8b8443eaa9c-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.247 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.247 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.247 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.247 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.247 DEBUG:boto:Method: GET 2024-04-19T23:39:19.247 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.247 DEBUG:boto:Data: 2024-04-19T23:39:19.247 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.247 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.247 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.247 DEBUG:boto:Params: {} 2024-04-19T23:39:19.248 DEBUG:boto:Token: None 2024-04-19T23:39:19.248 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.248 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rlgl611soGhLKCWdL9Bur1HP5Lc= 2024-04-19T23:39:19.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rlgl611soGhLKCWdL9Bur1HP5Lc=', 'Content-Length': '0'} 2024-04-19T23:39:19.250 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025dbad35db231c17-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.250 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.251 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.251 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.251 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.251 DEBUG:boto:Method: GET 2024-04-19T23:39:19.251 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.251 DEBUG:boto:Data: 2024-04-19T23:39:19.251 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.251 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.251 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.251 DEBUG:boto:Params: {} 2024-04-19T23:39:19.251 DEBUG:boto:Token: None 2024-04-19T23:39:19.252 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:19.252 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rlgl611soGhLKCWdL9Bur1HP5Lc= 2024-04-19T23:39:19.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rlgl611soGhLKCWdL9Bur1HP5Lc=', 'Content-Length': '0'} 2024-04-19T23:39:19.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065f90557e7b902a1-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.254 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.254 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.254 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.254 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.255 DEBUG:boto:Method: GET 2024-04-19T23:39:19.255 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.255 DEBUG:boto:Data: 2024-04-19T23:39:19.255 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.255 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.255 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.255 DEBUG:boto:Params: {} 2024-04-19T23:39:19.255 DEBUG:boto:Token: None 2024-04-19T23:39:19.255 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.255 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ffa31bd65c81461-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.259 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.259 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.259 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.260 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.260 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.260 DEBUG:boto:Method: GET 2024-04-19T23:39:19.260 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.260 DEBUG:boto:Data: 2024-04-19T23:39:19.260 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.261 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.261 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.261 DEBUG:boto:Params: {} 2024-04-19T23:39:19.261 DEBUG:boto:Token: None 2024-04-19T23:39:19.261 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.261 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7904cef471849be-00662300a7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.265 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.265 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.265 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.265 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.265 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:19.265 DEBUG:boto:Data: 2024-04-19T23:39:19.265 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.265 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.266 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.266 DEBUG:boto:Params: {} 2024-04-19T23:39:19.266 DEBUG:boto:Token: None 2024-04-19T23:39:19.266 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj 2024-04-19T23:39:19.266 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4= 2024-04-19T23:39:19.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4=', 'Content-Length': '0'} 2024-04-19T23:39:19.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031f2f7c84507effd-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.270 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.270 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.270 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.271 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:19.271 DEBUG:boto:Data: 2024-04-19T23:39:19.271 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.271 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.271 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.271 DEBUG:boto:Params: {} 2024-04-19T23:39:19.271 DEBUG:boto:Token: None 2024-04-19T23:39:19.271 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj 2024-04-19T23:39:19.271 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4= 2024-04-19T23:39:19.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4=', 'Content-Length': '0'} 2024-04-19T23:39:19.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000880bd865e32b1693-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.275 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.275 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.275 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.276 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.276 DEBUG:boto:Method: GET 2024-04-19T23:39:19.276 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.276 DEBUG:boto:Data: 2024-04-19T23:39:19.276 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.276 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.276 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.276 DEBUG:boto:Params: {} 2024-04-19T23:39:19.276 DEBUG:boto:Token: None 2024-04-19T23:39:19.276 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.277 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4= 2024-04-19T23:39:19.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4=', 'Content-Length': '0'} 2024-04-19T23:39:19.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006addf654f938696e-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.322 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.322 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.323 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.323 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.323 DEBUG:boto:Method: GET 2024-04-19T23:39:19.323 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.323 DEBUG:boto:Data: 2024-04-19T23:39:19.323 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.323 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.323 DEBUG:boto:Params: {} 2024-04-19T23:39:19.323 DEBUG:boto:Token: None 2024-04-19T23:39:19.324 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4= 2024-04-19T23:39:19.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4=', 'Content-Length': '0'} 2024-04-19T23:39:19.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006869ac9441de171-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.326 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.327 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.327 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.327 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.327 DEBUG:boto:Method: GET 2024-04-19T23:39:19.327 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.327 DEBUG:boto:Data: 2024-04-19T23:39:19.327 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.327 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.327 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.327 DEBUG:boto:Params: {} 2024-04-19T23:39:19.327 DEBUG:boto:Token: None 2024-04-19T23:39:19.328 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.328 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000103d844a2c7a6db2-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.330 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.331 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.331 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.331 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.331 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.331 DEBUG:boto:Method: GET 2024-04-19T23:39:19.331 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.331 DEBUG:boto:Data: 2024-04-19T23:39:19.331 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.332 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.332 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.332 DEBUG:boto:Params: {} 2024-04-19T23:39:19.332 DEBUG:boto:Token: None 2024-04-19T23:39:19.332 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000486b91e47ac16a6e-00662300a7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.337 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.337 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.337 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.337 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.338 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.338 DEBUG:boto:Method: GET 2024-04-19T23:39:19.338 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.338 DEBUG:boto:Data: 2024-04-19T23:39:19.338 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.338 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.338 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.338 DEBUG:boto:Params: {} 2024-04-19T23:39:19.338 DEBUG:boto:Token: None 2024-04-19T23:39:19.338 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.339 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HdtYX/lGcn5RnpupHL4PQ3VuMYE= 2024-04-19T23:39:19.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HdtYX/lGcn5RnpupHL4PQ3VuMYE=', 'Content-Length': '0'} 2024-04-19T23:39:19.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec7f7ff91b5b2db2-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.342 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.342 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.342 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.342 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.343 DEBUG:boto:Method: GET 2024-04-19T23:39:19.343 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.343 DEBUG:boto:Data: 2024-04-19T23:39:19.343 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.343 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.343 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.343 DEBUG:boto:Params: {} 2024-04-19T23:39:19.343 DEBUG:boto:Token: None 2024-04-19T23:39:19.343 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HdtYX/lGcn5RnpupHL4PQ3VuMYE= 2024-04-19T23:39:19.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HdtYX/lGcn5RnpupHL4PQ3VuMYE=', 'Content-Length': '0'} 2024-04-19T23:39:19.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000454a9ea1b6bfccda-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.346 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.346 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.346 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.347 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.347 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.347 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.347 DEBUG:boto:Data: 2024-04-19T23:39:19.347 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.347 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.347 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.347 DEBUG:boto:Params: {} 2024-04-19T23:39:19.347 DEBUG:boto:Token: None 2024-04-19T23:39:19.348 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.348 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2B/Qd/j9j7Kkq/YSFwBokeIAk9I= 2024-04-19T23:39:19.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2B/Qd/j9j7Kkq/YSFwBokeIAk9I=', 'Content-Length': '0'} 2024-04-19T23:39:19.349 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a1cd0c20b296393-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.350 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.350 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.350 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.350 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.350 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.350 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.350 DEBUG:boto:Data: 2024-04-19T23:39:19.350 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.351 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.351 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.351 DEBUG:boto:Params: {} 2024-04-19T23:39:19.351 DEBUG:boto:Token: None 2024-04-19T23:39:19.351 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.351 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2B/Qd/j9j7Kkq/YSFwBokeIAk9I= 2024-04-19T23:39:19.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2B/Qd/j9j7Kkq/YSFwBokeIAk9I=', 'Content-Length': '0'} 2024-04-19T23:39:19.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b77f381b24a0288-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.353 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.353 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.353 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.353 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.354 DEBUG:boto:Method: GET 2024-04-19T23:39:19.354 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.354 DEBUG:boto:Data: 2024-04-19T23:39:19.354 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.354 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.354 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.354 DEBUG:boto:Params: {} 2024-04-19T23:39:19.354 DEBUG:boto:Token: None 2024-04-19T23:39:19.354 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.354 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HdtYX/lGcn5RnpupHL4PQ3VuMYE= 2024-04-19T23:39:19.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HdtYX/lGcn5RnpupHL4PQ3VuMYE=', 'Content-Length': '0'} 2024-04-19T23:39:19.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007dadfd895d0a6f67-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.398 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.399 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.399 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.399 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.399 DEBUG:boto:Method: GET 2024-04-19T23:39:19.399 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.399 DEBUG:boto:Data: 2024-04-19T23:39:19.399 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.399 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.399 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.399 DEBUG:boto:Params: {} 2024-04-19T23:39:19.400 DEBUG:boto:Token: None 2024-04-19T23:39:19.400 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:19.400 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HdtYX/lGcn5RnpupHL4PQ3VuMYE= 2024-04-19T23:39:19.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HdtYX/lGcn5RnpupHL4PQ3VuMYE=', 'Content-Length': '0'} 2024-04-19T23:39:19.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ddf1fa90fb3a642-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.402 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.403 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.403 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.403 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.403 DEBUG:boto:Method: GET 2024-04-19T23:39:19.403 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.403 DEBUG:boto:Data: 2024-04-19T23:39:19.403 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.403 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.403 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.403 DEBUG:boto:Params: {} 2024-04-19T23:39:19.404 DEBUG:boto:Token: None 2024-04-19T23:39:19.404 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.404 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000209ca858176e562f-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.450 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.451 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.451 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.451 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.451 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.451 DEBUG:boto:Method: GET 2024-04-19T23:39:19.451 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.451 DEBUG:boto:Data: 2024-04-19T23:39:19.451 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.451 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.452 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.452 DEBUG:boto:Params: {} 2024-04-19T23:39:19.452 DEBUG:boto:Token: None 2024-04-19T23:39:19.452 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.452 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000850fa0b5f5c3eaa3-00662300a7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.455 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.456 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.456 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.456 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.456 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:19.456 DEBUG:boto:Data: 2024-04-19T23:39:19.456 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.456 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.456 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.456 DEBUG:boto:Params: {} 2024-04-19T23:39:19.457 DEBUG:boto:Token: None 2024-04-19T23:39:19.457 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj 2024-04-19T23:39:19.457 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4= 2024-04-19T23:39:19.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4=', 'Content-Length': '0'} 2024-04-19T23:39:19.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009642455258a3a704-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.461 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.461 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.461 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.461 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:19.461 DEBUG:boto:Data: 2024-04-19T23:39:19.461 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.461 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.461 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.461 DEBUG:boto:Params: {} 2024-04-19T23:39:19.462 DEBUG:boto:Token: None 2024-04-19T23:39:19.462 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj 2024-04-19T23:39:19.462 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4= 2024-04-19T23:39:19.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4=', 'Content-Length': '0'} 2024-04-19T23:39:19.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1c7adccb1ab39de-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.466 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.466 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.466 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.466 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.466 DEBUG:boto:Method: GET 2024-04-19T23:39:19.466 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.466 DEBUG:boto:Data: 2024-04-19T23:39:19.466 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.466 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.467 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.467 DEBUG:boto:Params: {} 2024-04-19T23:39:19.467 DEBUG:boto:Token: None 2024-04-19T23:39:19.467 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.467 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4= 2024-04-19T23:39:19.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4=', 'Content-Length': '0'} 2024-04-19T23:39:19.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ffacf255e803e740-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.469 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.469 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.469 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.469 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.469 DEBUG:boto:Method: GET 2024-04-19T23:39:19.470 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.470 DEBUG:boto:Data: 2024-04-19T23:39:19.470 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.470 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.470 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.470 DEBUG:boto:Params: {} 2024-04-19T23:39:19.470 DEBUG:boto:Token: None 2024-04-19T23:39:19.470 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.470 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4= 2024-04-19T23:39:19.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4=', 'Content-Length': '0'} 2024-04-19T23:39:19.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9dfd9de27355633-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.473 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.473 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.473 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.473 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.473 DEBUG:boto:Method: GET 2024-04-19T23:39:19.474 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.474 DEBUG:boto:Data: 2024-04-19T23:39:19.474 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.474 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.474 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.474 DEBUG:boto:Params: {} 2024-04-19T23:39:19.474 DEBUG:boto:Token: None 2024-04-19T23:39:19.474 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.474 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000019808714247dc6-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.477 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.478 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.478 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.478 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.478 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.478 DEBUG:boto:Method: GET 2024-04-19T23:39:19.478 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.478 DEBUG:boto:Data: 2024-04-19T23:39:19.479 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.479 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.479 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.479 DEBUG:boto:Params: {} 2024-04-19T23:39:19.479 DEBUG:boto:Token: None 2024-04-19T23:39:19.479 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.479 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1a7049c51a734bf-00662300a7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.482 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.483 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.483 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.483 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.483 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.483 DEBUG:boto:Method: GET 2024-04-19T23:39:19.483 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.484 DEBUG:boto:Data: 2024-04-19T23:39:19.484 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.484 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.484 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.484 DEBUG:boto:Params: {} 2024-04-19T23:39:19.484 DEBUG:boto:Token: None 2024-04-19T23:39:19.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:smx9MoV2EOb8J5UZd7rdF1I+xwM= 2024-04-19T23:39:19.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:smx9MoV2EOb8J5UZd7rdF1I+xwM=', 'Content-Length': '0'} 2024-04-19T23:39:19.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096fa60a6c3d7cbae-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.486 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.487 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.487 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.487 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.487 DEBUG:boto:Method: GET 2024-04-19T23:39:19.487 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.487 DEBUG:boto:Data: 2024-04-19T23:39:19.487 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.487 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.487 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.487 DEBUG:boto:Params: {} 2024-04-19T23:39:19.488 DEBUG:boto:Token: None 2024-04-19T23:39:19.488 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:smx9MoV2EOb8J5UZd7rdF1I+xwM= 2024-04-19T23:39:19.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:smx9MoV2EOb8J5UZd7rdF1I+xwM=', 'Content-Length': '0'} 2024-04-19T23:39:19.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ec9a426154b950c-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.530 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.531 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.531 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.531 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.531 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.531 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.531 DEBUG:boto:Data: 2024-04-19T23:39:19.531 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.531 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.531 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.531 DEBUG:boto:Params: {} 2024-04-19T23:39:19.531 DEBUG:boto:Token: None 2024-04-19T23:39:19.532 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.532 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MpkyiI9SOyBYvv/AmFh8XQ+5M0A= 2024-04-19T23:39:19.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MpkyiI9SOyBYvv/AmFh8XQ+5M0A=', 'Content-Length': '0'} 2024-04-19T23:39:19.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000888622441a514ce-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.534 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.534 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.534 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.534 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.534 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.534 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.535 DEBUG:boto:Data: 2024-04-19T23:39:19.535 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.535 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.535 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.535 DEBUG:boto:Params: {} 2024-04-19T23:39:19.535 DEBUG:boto:Token: None 2024-04-19T23:39:19.535 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.535 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MpkyiI9SOyBYvv/AmFh8XQ+5M0A= 2024-04-19T23:39:19.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MpkyiI9SOyBYvv/AmFh8XQ+5M0A=', 'Content-Length': '0'} 2024-04-19T23:39:19.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000511b727dfd86ec36-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.537 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.537 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.537 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.538 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.538 DEBUG:boto:Method: GET 2024-04-19T23:39:19.538 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.538 DEBUG:boto:Data: 2024-04-19T23:39:19.538 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.538 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.538 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.538 DEBUG:boto:Params: {} 2024-04-19T23:39:19.538 DEBUG:boto:Token: None 2024-04-19T23:39:19.538 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.539 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:smx9MoV2EOb8J5UZd7rdF1I+xwM= 2024-04-19T23:39:19.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:smx9MoV2EOb8J5UZd7rdF1I+xwM=', 'Content-Length': '0'} 2024-04-19T23:39:19.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f13b3d588b87cea8-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.587 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.587 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.587 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.587 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.588 DEBUG:boto:Method: GET 2024-04-19T23:39:19.588 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.588 DEBUG:boto:Data: 2024-04-19T23:39:19.588 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.588 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.588 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.589 DEBUG:boto:Params: {} 2024-04-19T23:39:19.589 DEBUG:boto:Token: None 2024-04-19T23:39:19.589 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:19.589 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:smx9MoV2EOb8J5UZd7rdF1I+xwM= 2024-04-19T23:39:19.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:smx9MoV2EOb8J5UZd7rdF1I+xwM=', 'Content-Length': '0'} 2024-04-19T23:39:19.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002f955ab72e98920-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.592 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.592 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.592 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.592 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.592 DEBUG:boto:Method: GET 2024-04-19T23:39:19.592 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.593 DEBUG:boto:Data: 2024-04-19T23:39:19.593 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.593 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.593 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.593 DEBUG:boto:Params: {} 2024-04-19T23:39:19.593 DEBUG:boto:Token: None 2024-04-19T23:39:19.593 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.593 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e95789313e918a1-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.642 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.643 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.643 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.643 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.643 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.643 DEBUG:boto:Method: GET 2024-04-19T23:39:19.643 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.643 DEBUG:boto:Data: 2024-04-19T23:39:19.643 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.643 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.643 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.644 DEBUG:boto:Params: {} 2024-04-19T23:39:19.644 DEBUG:boto:Token: None 2024-04-19T23:39:19.644 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.644 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b90b8a21e5081ed3-00662300a7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.647 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.647 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.647 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.648 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.648 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:19.648 DEBUG:boto:Data: 2024-04-19T23:39:19.648 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.648 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.648 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.648 DEBUG:boto:Params: {} 2024-04-19T23:39:19.648 DEBUG:boto:Token: None 2024-04-19T23:39:19.648 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj 2024-04-19T23:39:19.649 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4= 2024-04-19T23:39:19.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4=', 'Content-Length': '0'} 2024-04-19T23:39:19.652 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c93c09c3645d3bea-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.652 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.653 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.653 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.653 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:19.653 DEBUG:boto:Data: 2024-04-19T23:39:19.653 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.653 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.653 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.653 DEBUG:boto:Params: {} 2024-04-19T23:39:19.653 DEBUG:boto:Token: None 2024-04-19T23:39:19.653 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj 2024-04-19T23:39:19.653 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4= 2024-04-19T23:39:19.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4=', 'Content-Length': '0'} 2024-04-19T23:39:19.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bd3eded1e04c45a-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.657 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.657 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.657 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.657 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.658 DEBUG:boto:Method: GET 2024-04-19T23:39:19.658 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.658 DEBUG:boto:Data: 2024-04-19T23:39:19.658 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.658 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.658 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.658 DEBUG:boto:Params: {} 2024-04-19T23:39:19.658 DEBUG:boto:Token: None 2024-04-19T23:39:19.658 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.658 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4= 2024-04-19T23:39:19.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4=', 'Content-Length': '0'} 2024-04-19T23:39:19.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a10dc199980e3542-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.661 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.661 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.661 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.661 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.662 DEBUG:boto:Method: GET 2024-04-19T23:39:19.662 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.662 DEBUG:boto:Data: 2024-04-19T23:39:19.662 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.662 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.662 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.662 DEBUG:boto:Params: {} 2024-04-19T23:39:19.662 DEBUG:boto:Token: None 2024-04-19T23:39:19.662 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.663 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4= 2024-04-19T23:39:19.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4=', 'Content-Length': '0'} 2024-04-19T23:39:19.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000015cc503e5a1f8593-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.706 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.706 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.707 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.707 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.707 DEBUG:boto:Method: GET 2024-04-19T23:39:19.707 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.707 DEBUG:boto:Data: 2024-04-19T23:39:19.707 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.707 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.707 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.707 DEBUG:boto:Params: {} 2024-04-19T23:39:19.708 DEBUG:boto:Token: None 2024-04-19T23:39:19.710 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.710 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1180065b92774cd-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.713 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.714 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.714 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.714 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.714 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.714 DEBUG:boto:Method: GET 2024-04-19T23:39:19.714 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.715 DEBUG:boto:Data: 2024-04-19T23:39:19.715 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.715 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.715 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.715 DEBUG:boto:Params: {} 2024-04-19T23:39:19.715 DEBUG:boto:Token: None 2024-04-19T23:39:19.715 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.715 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000235bf0e4c4d9371d-00662300a7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.718 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.719 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.719 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.719 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.719 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.719 DEBUG:boto:Method: GET 2024-04-19T23:39:19.719 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.720 DEBUG:boto:Data: 2024-04-19T23:39:19.720 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.720 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.720 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.720 DEBUG:boto:Params: {} 2024-04-19T23:39:19.720 DEBUG:boto:Token: None 2024-04-19T23:39:19.720 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.720 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FTZoLFdDZbtevto3dnTIJpfiNKE= 2024-04-19T23:39:19.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FTZoLFdDZbtevto3dnTIJpfiNKE=', 'Content-Length': '0'} 2024-04-19T23:39:19.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5afdd3ec3109ce3-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.723 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.723 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.723 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.723 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.723 DEBUG:boto:Method: GET 2024-04-19T23:39:19.724 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.724 DEBUG:boto:Data: 2024-04-19T23:39:19.724 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.724 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.724 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.724 DEBUG:boto:Params: {} 2024-04-19T23:39:19.724 DEBUG:boto:Token: None 2024-04-19T23:39:19.724 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.725 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FTZoLFdDZbtevto3dnTIJpfiNKE= 2024-04-19T23:39:19.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FTZoLFdDZbtevto3dnTIJpfiNKE=', 'Content-Length': '0'} 2024-04-19T23:39:19.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005626b74936fcf6b6-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.727 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.727 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.727 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.727 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.727 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.727 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.728 DEBUG:boto:Data: 2024-04-19T23:39:19.728 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.728 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.728 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.728 DEBUG:boto:Params: {} 2024-04-19T23:39:19.728 DEBUG:boto:Token: None 2024-04-19T23:39:19.728 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.728 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:75BQcxDvb3ercqkInBckA1AgEQw= 2024-04-19T23:39:19.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:75BQcxDvb3ercqkInBckA1AgEQw=', 'Content-Length': '0'} 2024-04-19T23:39:19.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0fa426eb85988c1-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.731 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.731 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.731 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.731 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.731 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.731 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.732 DEBUG:boto:Data: 2024-04-19T23:39:19.732 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.732 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.732 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.732 DEBUG:boto:Params: {} 2024-04-19T23:39:19.732 DEBUG:boto:Token: None 2024-04-19T23:39:19.732 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.732 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:75BQcxDvb3ercqkInBckA1AgEQw= 2024-04-19T23:39:19.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:75BQcxDvb3ercqkInBckA1AgEQw=', 'Content-Length': '0'} 2024-04-19T23:39:19.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070be3e84f0fddd55-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.735 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.735 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.735 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.735 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.735 DEBUG:boto:Method: GET 2024-04-19T23:39:19.735 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.735 DEBUG:boto:Data: 2024-04-19T23:39:19.735 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.735 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.735 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.736 DEBUG:boto:Params: {} 2024-04-19T23:39:19.736 DEBUG:boto:Token: None 2024-04-19T23:39:19.736 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.736 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FTZoLFdDZbtevto3dnTIJpfiNKE= 2024-04-19T23:39:19.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FTZoLFdDZbtevto3dnTIJpfiNKE=', 'Content-Length': '0'} 2024-04-19T23:39:19.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003208e402833fe6d7-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.778 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.778 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.779 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.779 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.779 DEBUG:boto:Method: GET 2024-04-19T23:39:19.779 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.779 DEBUG:boto:Data: 2024-04-19T23:39:19.779 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.779 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.779 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.779 DEBUG:boto:Params: {} 2024-04-19T23:39:19.780 DEBUG:boto:Token: None 2024-04-19T23:39:19.780 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:19.780 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FTZoLFdDZbtevto3dnTIJpfiNKE= 2024-04-19T23:39:19.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FTZoLFdDZbtevto3dnTIJpfiNKE=', 'Content-Length': '0'} 2024-04-19T23:39:19.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a35066bf59dde38-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.782 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.782 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.783 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.783 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.783 DEBUG:boto:Method: GET 2024-04-19T23:39:19.783 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.783 DEBUG:boto:Data: 2024-04-19T23:39:19.783 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.783 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.783 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.783 DEBUG:boto:Params: {} 2024-04-19T23:39:19.783 DEBUG:boto:Token: None 2024-04-19T23:39:19.784 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000646f1a116275eb2c-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.787 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.787 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.787 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.787 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.787 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.787 DEBUG:boto:Method: GET 2024-04-19T23:39:19.788 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.788 DEBUG:boto:Data: 2024-04-19T23:39:19.788 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.788 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.788 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.788 DEBUG:boto:Params: {} 2024-04-19T23:39:19.788 DEBUG:boto:Token: None 2024-04-19T23:39:19.788 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.788 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061925a661891ffcf-00662300a7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.791 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.792 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.792 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.792 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.792 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:19.792 DEBUG:boto:Data: 2024-04-19T23:39:19.792 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.792 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.792 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.792 DEBUG:boto:Params: {} 2024-04-19T23:39:19.792 DEBUG:boto:Token: None 2024-04-19T23:39:19.792 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj 2024-04-19T23:39:19.793 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4= 2024-04-19T23:39:19.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4=', 'Content-Length': '0'} 2024-04-19T23:39:19.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb8c2e26f227206e-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.796 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.797 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.797 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.797 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:19.797 DEBUG:boto:Data: 2024-04-19T23:39:19.797 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.797 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.797 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.797 DEBUG:boto:Params: {} 2024-04-19T23:39:19.797 DEBUG:boto:Token: None 2024-04-19T23:39:19.798 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj 2024-04-19T23:39:19.798 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4= 2024-04-19T23:39:19.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4=', 'Content-Length': '0'} 2024-04-19T23:39:19.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b84f968c01193b3-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.801 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.802 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.802 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.802 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.802 DEBUG:boto:Method: GET 2024-04-19T23:39:19.802 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.802 DEBUG:boto:Data: 2024-04-19T23:39:19.802 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.802 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.802 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.802 DEBUG:boto:Params: {} 2024-04-19T23:39:19.802 DEBUG:boto:Token: None 2024-04-19T23:39:19.803 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.803 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4= 2024-04-19T23:39:19.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4=', 'Content-Length': '0'} 2024-04-19T23:39:19.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3f003e155604971-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.846 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.847 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.847 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.847 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.847 DEBUG:boto:Method: GET 2024-04-19T23:39:19.848 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.848 DEBUG:boto:Data: 2024-04-19T23:39:19.848 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.848 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.848 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.849 DEBUG:boto:Params: {} 2024-04-19T23:39:19.849 DEBUG:boto:Token: None 2024-04-19T23:39:19.849 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.849 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4= 2024-04-19T23:39:19.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4=', 'Content-Length': '0'} 2024-04-19T23:39:19.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f815f5aef3156231-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.853 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.853 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.853 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.853 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.853 DEBUG:boto:Method: GET 2024-04-19T23:39:19.854 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.854 DEBUG:boto:Data: 2024-04-19T23:39:19.854 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.854 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.854 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.855 DEBUG:boto:Params: {} 2024-04-19T23:39:19.855 DEBUG:boto:Token: None 2024-04-19T23:39:19.855 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.855 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cc0c4a7d45b1f4b-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.858 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.859 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.859 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.859 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.860 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.860 DEBUG:boto:Method: GET 2024-04-19T23:39:19.860 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.860 DEBUG:boto:Data: 2024-04-19T23:39:19.860 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.861 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.861 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.861 DEBUG:boto:Params: {} 2024-04-19T23:39:19.861 DEBUG:boto:Token: None 2024-04-19T23:39:19.862 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.862 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b47ba16f6f6e376-00662300a7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.865 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.866 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.866 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.866 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.866 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.866 DEBUG:boto:Method: GET 2024-04-19T23:39:19.866 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.867 DEBUG:boto:Data: 2024-04-19T23:39:19.867 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.867 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.867 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.867 DEBUG:boto:Params: {} 2024-04-19T23:39:19.867 DEBUG:boto:Token: None 2024-04-19T23:39:19.868 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.868 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Iaz2o2A8n4WnsTOrhLMrou26FjE= 2024-04-19T23:39:19.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Iaz2o2A8n4WnsTOrhLMrou26FjE=', 'Content-Length': '0'} 2024-04-19T23:39:19.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000547d41c404ddabea-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.873 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.873 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.874 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.874 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.874 DEBUG:boto:Method: GET 2024-04-19T23:39:19.874 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.874 DEBUG:boto:Data: 2024-04-19T23:39:19.874 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.875 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.875 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.875 DEBUG:boto:Params: {} 2024-04-19T23:39:19.875 DEBUG:boto:Token: None 2024-04-19T23:39:19.876 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.876 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Iaz2o2A8n4WnsTOrhLMrou26FjE= 2024-04-19T23:39:19.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Iaz2o2A8n4WnsTOrhLMrou26FjE=', 'Content-Length': '0'} 2024-04-19T23:39:19.878 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005accbfb9054c7a35-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.878 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.879 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.879 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.879 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.879 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.879 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.879 DEBUG:boto:Data: 2024-04-19T23:39:19.880 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.880 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.880 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.880 DEBUG:boto:Params: {} 2024-04-19T23:39:19.880 DEBUG:boto:Token: None 2024-04-19T23:39:19.881 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.881 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/QEDvtIvaz9B1xOP3EeNpuGT+l8= 2024-04-19T23:39:19.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/QEDvtIvaz9B1xOP3EeNpuGT+l8=', 'Content-Length': '0'} 2024-04-19T23:39:19.883 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd485a80fd3d9fc1-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.883 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.883 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.883 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.883 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.884 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.884 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.884 DEBUG:boto:Data: 2024-04-19T23:39:19.884 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.884 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.884 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.884 DEBUG:boto:Params: {} 2024-04-19T23:39:19.884 DEBUG:boto:Token: None 2024-04-19T23:39:19.884 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.884 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/QEDvtIvaz9B1xOP3EeNpuGT+l8= 2024-04-19T23:39:19.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/QEDvtIvaz9B1xOP3EeNpuGT+l8=', 'Content-Length': '0'} 2024-04-19T23:39:19.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6ceb11e3c64e915-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.887 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.887 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.887 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.887 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.887 DEBUG:boto:Method: GET 2024-04-19T23:39:19.887 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.887 DEBUG:boto:Data: 2024-04-19T23:39:19.887 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.887 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.887 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.887 DEBUG:boto:Params: {} 2024-04-19T23:39:19.888 DEBUG:boto:Token: None 2024-04-19T23:39:19.888 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.888 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Iaz2o2A8n4WnsTOrhLMrou26FjE= 2024-04-19T23:39:19.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Iaz2o2A8n4WnsTOrhLMrou26FjE=', 'Content-Length': '0'} 2024-04-19T23:39:19.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e02ff43057d4ddf7-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.930 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.930 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.930 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.931 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.931 DEBUG:boto:Method: GET 2024-04-19T23:39:19.931 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.931 DEBUG:boto:Data: 2024-04-19T23:39:19.931 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.931 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.931 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.931 DEBUG:boto:Params: {} 2024-04-19T23:39:19.931 DEBUG:boto:Token: None 2024-04-19T23:39:19.931 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:19.932 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Iaz2o2A8n4WnsTOrhLMrou26FjE= 2024-04-19T23:39:19.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Iaz2o2A8n4WnsTOrhLMrou26FjE=', 'Content-Length': '0'} 2024-04-19T23:39:19.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7214635d68f6b2d-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.934 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.934 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.934 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.934 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.934 DEBUG:boto:Method: GET 2024-04-19T23:39:19.934 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.935 DEBUG:boto:Data: 2024-04-19T23:39:19.935 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.935 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.935 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.935 DEBUG:boto:Params: {} 2024-04-19T23:39:19.935 DEBUG:boto:Token: None 2024-04-19T23:39:19.935 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.935 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004044ae0fb8ed877f-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.938 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.939 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:19.939 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:19.939 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.939 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:19.939 DEBUG:boto:Method: GET 2024-04-19T23:39:19.939 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:19.939 DEBUG:boto:Data: 2024-04-19T23:39:19.939 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.939 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.939 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.940 DEBUG:boto:Params: {} 2024-04-19T23:39:19.940 DEBUG:boto:Token: None 2024-04-19T23:39:19.940 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/ 2024-04-19T23:39:19.940 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw= 2024-04-19T23:39:19.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+7b1qY2jquUpznhnA1lCSNFP0lw=', 'Content-Length': '0'} 2024-04-19T23:39:19.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e40d28762785d02a-00662300a7-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.943 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:19.943 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.943 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.943 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.943 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:19.943 DEBUG:boto:Data: 2024-04-19T23:39:19.944 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.944 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.944 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.944 DEBUG:boto:Params: {} 2024-04-19T23:39:19.944 DEBUG:boto:Token: None 2024-04-19T23:39:19.944 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj 2024-04-19T23:39:19.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4= 2024-04-19T23:39:19.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4=', 'Content-Length': '0'} 2024-04-19T23:39:19.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cf1bcaa116c2820-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.948 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.948 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.948 DEBUG:boto:Method: HEAD 2024-04-19T23:39:19.948 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:19.948 DEBUG:boto:Data: 2024-04-19T23:39:19.948 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.948 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.948 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.948 DEBUG:boto:Params: {} 2024-04-19T23:39:19.949 DEBUG:boto:Token: None 2024-04-19T23:39:19.949 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj 2024-04-19T23:39:19.949 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4= 2024-04-19T23:39:19.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:884SMZKutCvS6eMdMkAP2d8dnH4=', 'Content-Length': '0'} 2024-04-19T23:39:19.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e2f35a9e21be1c0-00662300a7-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:19.952 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.952 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.953 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.953 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.953 DEBUG:boto:Method: GET 2024-04-19T23:39:19.953 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.953 DEBUG:boto:Data: 2024-04-19T23:39:19.953 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.953 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:19.953 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.953 DEBUG:boto:Params: {} 2024-04-19T23:39:19.953 DEBUG:boto:Token: None 2024-04-19T23:39:19.954 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:19 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.954 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4= 2024-04-19T23:39:19.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:19 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gejzgn3ObzXGCK08a9SFNlFRw+4=', 'Content-Length': '0'} 2024-04-19T23:39:19.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c83fbc2d7fbb0b8b-00662300a7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:19.998 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:19.998 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:19.999 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.999 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.999 DEBUG:boto:Method: GET 2024-04-19T23:39:19.999 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:19.999 DEBUG:boto:Data: 2024-04-19T23:39:19.999 DEBUG:boto:Headers: {} 2024-04-19T23:39:19.999 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:19.999 DEBUG:boto:Port: 8000 2024-04-19T23:39:19.999 DEBUG:boto:Params: {} 2024-04-19T23:39:19.999 DEBUG:boto:Token: None 2024-04-19T23:39:20.000 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.000 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40= 2024-04-19T23:39:20.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40=', 'Content-Length': '0'} 2024-04-19T23:39:20.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001549b0de5ccf534d-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.046 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.047 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.047 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.047 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.047 DEBUG:boto:Method: GET 2024-04-19T23:39:20.047 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.047 DEBUG:boto:Data: 2024-04-19T23:39:20.047 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.047 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.047 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.047 DEBUG:boto:Params: {} 2024-04-19T23:39:20.048 DEBUG:boto:Token: None 2024-04-19T23:39:20.048 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.048 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0ce4244e78bdd9d-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.051 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.052 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.052 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.052 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.052 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.052 DEBUG:boto:Method: GET 2024-04-19T23:39:20.052 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.052 DEBUG:boto:Data: 2024-04-19T23:39:20.052 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.052 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.053 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.053 DEBUG:boto:Params: {} 2024-04-19T23:39:20.053 DEBUG:boto:Token: None 2024-04-19T23:39:20.053 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.054 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3f8ea79e5f2fb0f-00662300a8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.098 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:20.099 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.099 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.099 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.099 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.099 DEBUG:boto:Method: GET 2024-04-19T23:39:20.100 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.100 DEBUG:boto:Data: 2024-04-19T23:39:20.100 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.100 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.100 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.100 DEBUG:boto:Params: {} 2024-04-19T23:39:20.100 DEBUG:boto:Token: None 2024-04-19T23:39:20.100 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.100 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxcLuaZBx2pMLr0b6P2vzqjj5UA= 2024-04-19T23:39:20.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxcLuaZBx2pMLr0b6P2vzqjj5UA=', 'Content-Length': '0'} 2024-04-19T23:39:20.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008dd2fbaf75c9ec3f-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.103 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.103 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.103 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.103 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.103 DEBUG:boto:Method: GET 2024-04-19T23:39:20.103 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.104 DEBUG:boto:Data: 2024-04-19T23:39:20.104 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.104 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.104 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.104 DEBUG:boto:Params: {} 2024-04-19T23:39:20.104 DEBUG:boto:Token: None 2024-04-19T23:39:20.104 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxcLuaZBx2pMLr0b6P2vzqjj5UA= 2024-04-19T23:39:20.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxcLuaZBx2pMLr0b6P2vzqjj5UA=', 'Content-Length': '0'} 2024-04-19T23:39:20.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003751abfbc89e66e2-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.107 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.107 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.107 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.107 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.107 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.107 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.107 DEBUG:boto:Data: 2024-04-19T23:39:20.107 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.107 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.107 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.107 DEBUG:boto:Params: {} 2024-04-19T23:39:20.108 DEBUG:boto:Token: None 2024-04-19T23:39:20.108 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.108 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0IZY2QmHr0o66f63Tou81goN2bw= 2024-04-19T23:39:20.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0IZY2QmHr0o66f63Tou81goN2bw=', 'Content-Length': '0'} 2024-04-19T23:39:20.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000919ba343bb8b5e8e-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.110 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.110 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.110 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.111 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.111 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.111 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.111 DEBUG:boto:Data: 2024-04-19T23:39:20.111 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.111 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.111 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.111 DEBUG:boto:Params: {} 2024-04-19T23:39:20.111 DEBUG:boto:Token: None 2024-04-19T23:39:20.112 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.112 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0IZY2QmHr0o66f63Tou81goN2bw= 2024-04-19T23:39:20.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0IZY2QmHr0o66f63Tou81goN2bw=', 'Content-Length': '0'} 2024-04-19T23:39:20.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000712ae2daa87d1efb-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:20.114 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.114 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.114 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.114 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.114 DEBUG:boto:Method: GET 2024-04-19T23:39:20.114 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.115 DEBUG:boto:Data: 2024-04-19T23:39:20.115 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.115 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.115 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.115 DEBUG:boto:Params: {} 2024-04-19T23:39:20.115 DEBUG:boto:Token: None 2024-04-19T23:39:20.115 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.115 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxcLuaZBx2pMLr0b6P2vzqjj5UA= 2024-04-19T23:39:20.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxcLuaZBx2pMLr0b6P2vzqjj5UA=', 'Content-Length': '0'} 2024-04-19T23:39:20.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000078305da37f1c4-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.118 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.118 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.118 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.118 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.118 DEBUG:boto:Method: GET 2024-04-19T23:39:20.118 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.118 DEBUG:boto:Data: 2024-04-19T23:39:20.118 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.118 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.118 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.119 DEBUG:boto:Params: {} 2024-04-19T23:39:20.119 DEBUG:boto:Token: None 2024-04-19T23:39:20.119 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:20.119 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxcLuaZBx2pMLr0b6P2vzqjj5UA= 2024-04-19T23:39:20.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxcLuaZBx2pMLr0b6P2vzqjj5UA=', 'Content-Length': '0'} 2024-04-19T23:39:20.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002bc3a31443010ff7-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.166 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.167 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.167 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.167 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.167 DEBUG:boto:Method: GET 2024-04-19T23:39:20.167 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.167 DEBUG:boto:Data: 2024-04-19T23:39:20.167 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.167 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.167 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.167 DEBUG:boto:Params: {} 2024-04-19T23:39:20.168 DEBUG:boto:Token: None 2024-04-19T23:39:20.168 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.168 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008edb4da80423c8a3-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.171 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.172 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.172 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.172 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.172 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.172 DEBUG:boto:Method: GET 2024-04-19T23:39:20.172 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.172 DEBUG:boto:Data: 2024-04-19T23:39:20.172 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.172 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.172 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.172 DEBUG:boto:Params: {} 2024-04-19T23:39:20.173 DEBUG:boto:Token: None 2024-04-19T23:39:20.173 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.173 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000125d363f01b43fb2-00662300a8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.218 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.219 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.219 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.220 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.220 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:20.220 DEBUG:boto:Data: 2024-04-19T23:39:20.220 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.220 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.220 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.220 DEBUG:boto:Params: {} 2024-04-19T23:39:20.220 DEBUG:boto:Token: None 2024-04-19T23:39:20.220 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj 2024-04-19T23:39:20.220 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw= 2024-04-19T23:39:20.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw=', 'Content-Length': '0'} 2024-04-19T23:39:20.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070a719781257807b-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.224 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.224 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.224 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.224 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:20.224 DEBUG:boto:Data: 2024-04-19T23:39:20.225 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.225 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.225 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.225 DEBUG:boto:Params: {} 2024-04-19T23:39:20.225 DEBUG:boto:Token: None 2024-04-19T23:39:20.225 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj 2024-04-19T23:39:20.225 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw= 2024-04-19T23:39:20.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw=', 'Content-Length': '0'} 2024-04-19T23:39:20.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d10d9ad9e42fb7f0-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:20.229 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.229 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.229 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.229 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.229 DEBUG:boto:Method: GET 2024-04-19T23:39:20.229 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.229 DEBUG:boto:Data: 2024-04-19T23:39:20.229 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.229 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.229 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.230 DEBUG:boto:Params: {} 2024-04-19T23:39:20.230 DEBUG:boto:Token: None 2024-04-19T23:39:20.230 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.230 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40= 2024-04-19T23:39:20.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40=', 'Content-Length': '0'} 2024-04-19T23:39:20.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021b895c5b4000179-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.278 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.279 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.279 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.279 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.279 DEBUG:boto:Method: GET 2024-04-19T23:39:20.279 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.279 DEBUG:boto:Data: 2024-04-19T23:39:20.279 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.279 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.279 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.279 DEBUG:boto:Params: {} 2024-04-19T23:39:20.279 DEBUG:boto:Token: None 2024-04-19T23:39:20.280 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.280 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40= 2024-04-19T23:39:20.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40=', 'Content-Length': '0'} 2024-04-19T23:39:20.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fa578ac32963e02-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.322 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.323 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.323 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.323 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.323 DEBUG:boto:Method: GET 2024-04-19T23:39:20.323 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.323 DEBUG:boto:Data: 2024-04-19T23:39:20.323 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.323 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.323 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.323 DEBUG:boto:Params: {} 2024-04-19T23:39:20.324 DEBUG:boto:Token: None 2024-04-19T23:39:20.324 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007548ed56388d362-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.327 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.327 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.327 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.328 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.328 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.328 DEBUG:boto:Method: GET 2024-04-19T23:39:20.328 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.328 DEBUG:boto:Data: 2024-04-19T23:39:20.328 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.328 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.328 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.328 DEBUG:boto:Params: {} 2024-04-19T23:39:20.328 DEBUG:boto:Token: None 2024-04-19T23:39:20.328 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.328 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d7dc0825fdeda4f-00662300a8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.332 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:20.332 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.332 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.332 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.332 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.332 DEBUG:boto:Method: GET 2024-04-19T23:39:20.332 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.333 DEBUG:boto:Data: 2024-04-19T23:39:20.333 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.333 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.333 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.333 DEBUG:boto:Params: {} 2024-04-19T23:39:20.333 DEBUG:boto:Token: None 2024-04-19T23:39:20.333 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.333 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uQ0zEe28oaf0RsCRUWW+GpOzHJQ= 2024-04-19T23:39:20.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uQ0zEe28oaf0RsCRUWW+GpOzHJQ=', 'Content-Length': '0'} 2024-04-19T23:39:20.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006350b2e980e2e21c-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.336 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.336 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.336 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.336 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.336 DEBUG:boto:Method: GET 2024-04-19T23:39:20.336 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.336 DEBUG:boto:Data: 2024-04-19T23:39:20.336 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.336 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.337 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.337 DEBUG:boto:Params: {} 2024-04-19T23:39:20.337 DEBUG:boto:Token: None 2024-04-19T23:39:20.337 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.337 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uQ0zEe28oaf0RsCRUWW+GpOzHJQ= 2024-04-19T23:39:20.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uQ0zEe28oaf0RsCRUWW+GpOzHJQ=', 'Content-Length': '0'} 2024-04-19T23:39:20.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f0c6fbb36c654bb-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.382 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.382 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.383 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.383 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.383 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.383 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.383 DEBUG:boto:Data: 2024-04-19T23:39:20.383 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.383 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.383 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.383 DEBUG:boto:Params: {} 2024-04-19T23:39:20.384 DEBUG:boto:Token: None 2024-04-19T23:39:20.384 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.384 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rldqvBkE5xdwRcbrxxlv671rX0Y= 2024-04-19T23:39:20.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rldqvBkE5xdwRcbrxxlv671rX0Y=', 'Content-Length': '0'} 2024-04-19T23:39:20.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000998ea029df323331-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.387 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.387 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.387 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.387 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.387 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.387 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.387 DEBUG:boto:Data: 2024-04-19T23:39:20.387 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.387 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.387 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.387 DEBUG:boto:Params: {} 2024-04-19T23:39:20.388 DEBUG:boto:Token: None 2024-04-19T23:39:20.388 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.388 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rldqvBkE5xdwRcbrxxlv671rX0Y= 2024-04-19T23:39:20.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rldqvBkE5xdwRcbrxxlv671rX0Y=', 'Content-Length': '0'} 2024-04-19T23:39:20.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000590c664815edcabf-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:20.391 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.391 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.391 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.391 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.391 DEBUG:boto:Method: GET 2024-04-19T23:39:20.392 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.392 DEBUG:boto:Data: 2024-04-19T23:39:20.392 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.392 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.392 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.392 DEBUG:boto:Params: {} 2024-04-19T23:39:20.392 DEBUG:boto:Token: None 2024-04-19T23:39:20.392 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.392 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uQ0zEe28oaf0RsCRUWW+GpOzHJQ= 2024-04-19T23:39:20.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uQ0zEe28oaf0RsCRUWW+GpOzHJQ=', 'Content-Length': '0'} 2024-04-19T23:39:20.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000868c4f77313b32f3-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.438 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.439 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.439 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.439 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.439 DEBUG:boto:Method: GET 2024-04-19T23:39:20.439 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.439 DEBUG:boto:Data: 2024-04-19T23:39:20.439 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.439 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.439 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.439 DEBUG:boto:Params: {} 2024-04-19T23:39:20.440 DEBUG:boto:Token: None 2024-04-19T23:39:20.440 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:20.440 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uQ0zEe28oaf0RsCRUWW+GpOzHJQ= 2024-04-19T23:39:20.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uQ0zEe28oaf0RsCRUWW+GpOzHJQ=', 'Content-Length': '0'} 2024-04-19T23:39:20.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076fa29a367a0760b-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.482 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.483 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.483 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.483 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.483 DEBUG:boto:Method: GET 2024-04-19T23:39:20.483 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.483 DEBUG:boto:Data: 2024-04-19T23:39:20.483 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.483 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.483 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.483 DEBUG:boto:Params: {} 2024-04-19T23:39:20.484 DEBUG:boto:Token: None 2024-04-19T23:39:20.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006edf3ff64a455ab9-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.487 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.487 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.487 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.488 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.488 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.488 DEBUG:boto:Method: GET 2024-04-19T23:39:20.488 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.488 DEBUG:boto:Data: 2024-04-19T23:39:20.488 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.488 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.488 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.488 DEBUG:boto:Params: {} 2024-04-19T23:39:20.488 DEBUG:boto:Token: None 2024-04-19T23:39:20.489 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.489 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c53841f64c074de4-00662300a8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.491 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.492 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.492 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.492 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.492 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:20.492 DEBUG:boto:Data: 2024-04-19T23:39:20.492 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.493 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.493 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.493 DEBUG:boto:Params: {} 2024-04-19T23:39:20.493 DEBUG:boto:Token: None 2024-04-19T23:39:20.493 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj 2024-04-19T23:39:20.493 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw= 2024-04-19T23:39:20.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw=', 'Content-Length': '0'} 2024-04-19T23:39:20.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f899fedaa9f18d71-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.496 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.496 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.497 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.497 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:20.497 DEBUG:boto:Data: 2024-04-19T23:39:20.497 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.497 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.497 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.497 DEBUG:boto:Params: {} 2024-04-19T23:39:20.497 DEBUG:boto:Token: None 2024-04-19T23:39:20.497 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj 2024-04-19T23:39:20.497 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw= 2024-04-19T23:39:20.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw=', 'Content-Length': '0'} 2024-04-19T23:39:20.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000483598eeccc1911a-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:20.501 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.502 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.502 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.502 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.502 DEBUG:boto:Method: GET 2024-04-19T23:39:20.503 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.503 DEBUG:boto:Data: 2024-04-19T23:39:20.503 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.503 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.503 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.503 DEBUG:boto:Params: {} 2024-04-19T23:39:20.504 DEBUG:boto:Token: None 2024-04-19T23:39:20.504 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.504 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40= 2024-04-19T23:39:20.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40=', 'Content-Length': '0'} 2024-04-19T23:39:20.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f56425e22e59a8d1-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.550 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.550 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.551 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.551 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.551 DEBUG:boto:Method: GET 2024-04-19T23:39:20.551 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.551 DEBUG:boto:Data: 2024-04-19T23:39:20.551 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.551 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.551 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.551 DEBUG:boto:Params: {} 2024-04-19T23:39:20.551 DEBUG:boto:Token: None 2024-04-19T23:39:20.552 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.552 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40= 2024-04-19T23:39:20.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40=', 'Content-Length': '0'} 2024-04-19T23:39:20.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085724a578f169837-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.594 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.595 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.595 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.595 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.595 DEBUG:boto:Method: GET 2024-04-19T23:39:20.595 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.595 DEBUG:boto:Data: 2024-04-19T23:39:20.595 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.595 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.595 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.596 DEBUG:boto:Params: {} 2024-04-19T23:39:20.596 DEBUG:boto:Token: None 2024-04-19T23:39:20.596 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.596 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e2706480f81062a-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.599 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.600 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.600 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.600 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.600 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.600 DEBUG:boto:Method: GET 2024-04-19T23:39:20.600 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.600 DEBUG:boto:Data: 2024-04-19T23:39:20.600 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.600 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.600 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.600 DEBUG:boto:Params: {} 2024-04-19T23:39:20.601 DEBUG:boto:Token: None 2024-04-19T23:39:20.601 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.601 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9be9751b88a8603-00662300a8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.604 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:20.604 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.604 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.604 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.604 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.605 DEBUG:boto:Method: GET 2024-04-19T23:39:20.605 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.605 DEBUG:boto:Data: 2024-04-19T23:39:20.605 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.605 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.605 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.605 DEBUG:boto:Params: {} 2024-04-19T23:39:20.606 DEBUG:boto:Token: None 2024-04-19T23:39:20.606 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.606 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hYcKEEm22/hv/OddEFnu8j+tXU4= 2024-04-19T23:39:20.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hYcKEEm22/hv/OddEFnu8j+tXU4=', 'Content-Length': '0'} 2024-04-19T23:39:20.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d04c1576a4fd3de6-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.609 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.609 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.609 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.609 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.609 DEBUG:boto:Method: GET 2024-04-19T23:39:20.609 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.609 DEBUG:boto:Data: 2024-04-19T23:39:20.609 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.609 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.610 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.610 DEBUG:boto:Params: {} 2024-04-19T23:39:20.610 DEBUG:boto:Token: None 2024-04-19T23:39:20.610 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.610 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hYcKEEm22/hv/OddEFnu8j+tXU4= 2024-04-19T23:39:20.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hYcKEEm22/hv/OddEFnu8j+tXU4=', 'Content-Length': '0'} 2024-04-19T23:39:20.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cb9e6a42ec39aa3-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.613 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.613 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.613 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.613 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.613 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.613 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.613 DEBUG:boto:Data: 2024-04-19T23:39:20.613 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.613 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.614 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.614 DEBUG:boto:Params: {} 2024-04-19T23:39:20.614 DEBUG:boto:Token: None 2024-04-19T23:39:20.614 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.614 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N+zBwwSgXSCBA38d8pGKqvaUEOQ= 2024-04-19T23:39:20.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N+zBwwSgXSCBA38d8pGKqvaUEOQ=', 'Content-Length': '0'} 2024-04-19T23:39:20.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdd27bc894337998-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.616 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.617 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.617 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.617 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.617 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.617 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.617 DEBUG:boto:Data: 2024-04-19T23:39:20.617 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.617 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.617 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.617 DEBUG:boto:Params: {} 2024-04-19T23:39:20.617 DEBUG:boto:Token: None 2024-04-19T23:39:20.618 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.618 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N+zBwwSgXSCBA38d8pGKqvaUEOQ= 2024-04-19T23:39:20.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N+zBwwSgXSCBA38d8pGKqvaUEOQ=', 'Content-Length': '0'} 2024-04-19T23:39:20.620 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca2d6427ab6289c9-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:20.620 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.620 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.620 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.620 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.620 DEBUG:boto:Method: GET 2024-04-19T23:39:20.620 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.621 DEBUG:boto:Data: 2024-04-19T23:39:20.621 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.621 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.621 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.621 DEBUG:boto:Params: {} 2024-04-19T23:39:20.621 DEBUG:boto:Token: None 2024-04-19T23:39:20.621 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.621 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hYcKEEm22/hv/OddEFnu8j+tXU4= 2024-04-19T23:39:20.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hYcKEEm22/hv/OddEFnu8j+tXU4=', 'Content-Length': '0'} 2024-04-19T23:39:20.623 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044dfd02cf03a8140-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.666 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.667 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.667 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.667 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.667 DEBUG:boto:Method: GET 2024-04-19T23:39:20.667 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.667 DEBUG:boto:Data: 2024-04-19T23:39:20.667 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.667 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.667 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.667 DEBUG:boto:Params: {} 2024-04-19T23:39:20.668 DEBUG:boto:Token: None 2024-04-19T23:39:20.668 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:20.668 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hYcKEEm22/hv/OddEFnu8j+tXU4= 2024-04-19T23:39:20.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hYcKEEm22/hv/OddEFnu8j+tXU4=', 'Content-Length': '0'} 2024-04-19T23:39:20.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d666dac49bd144c-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.671 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.671 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.671 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.671 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.672 DEBUG:boto:Method: GET 2024-04-19T23:39:20.672 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.672 DEBUG:boto:Data: 2024-04-19T23:39:20.672 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.672 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.672 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.672 DEBUG:boto:Params: {} 2024-04-19T23:39:20.672 DEBUG:boto:Token: None 2024-04-19T23:39:20.672 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.672 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f345f921a0b9f3da-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.675 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.676 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.676 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.676 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.676 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.676 DEBUG:boto:Method: GET 2024-04-19T23:39:20.676 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.676 DEBUG:boto:Data: 2024-04-19T23:39:20.676 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.676 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.677 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.677 DEBUG:boto:Params: {} 2024-04-19T23:39:20.677 DEBUG:boto:Token: None 2024-04-19T23:39:20.677 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.677 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcf471f974ff11c0-00662300a8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.680 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.680 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.680 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.680 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.680 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:20.680 DEBUG:boto:Data: 2024-04-19T23:39:20.681 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.681 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.681 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.681 DEBUG:boto:Params: {} 2024-04-19T23:39:20.682 DEBUG:boto:Token: None 2024-04-19T23:39:20.682 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj 2024-04-19T23:39:20.682 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw= 2024-04-19T23:39:20.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw=', 'Content-Length': '0'} 2024-04-19T23:39:20.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017c46080dc264251-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.685 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.686 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.686 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.686 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:20.686 DEBUG:boto:Data: 2024-04-19T23:39:20.686 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.686 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.686 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.686 DEBUG:boto:Params: {} 2024-04-19T23:39:20.686 DEBUG:boto:Token: None 2024-04-19T23:39:20.687 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj 2024-04-19T23:39:20.687 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw= 2024-04-19T23:39:20.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw=', 'Content-Length': '0'} 2024-04-19T23:39:20.690 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000483764d1389b6815-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:20.690 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.690 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.690 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.691 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.691 DEBUG:boto:Method: GET 2024-04-19T23:39:20.691 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.691 DEBUG:boto:Data: 2024-04-19T23:39:20.691 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.691 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.691 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.691 DEBUG:boto:Params: {} 2024-04-19T23:39:20.691 DEBUG:boto:Token: None 2024-04-19T23:39:20.691 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.692 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40= 2024-04-19T23:39:20.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40=', 'Content-Length': '0'} 2024-04-19T23:39:20.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac658dfbcd057b38-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.734 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.735 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.735 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.735 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.735 DEBUG:boto:Method: GET 2024-04-19T23:39:20.735 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.736 DEBUG:boto:Data: 2024-04-19T23:39:20.736 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.736 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.736 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.736 DEBUG:boto:Params: {} 2024-04-19T23:39:20.736 DEBUG:boto:Token: None 2024-04-19T23:39:20.736 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.736 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40= 2024-04-19T23:39:20.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40=', 'Content-Length': '0'} 2024-04-19T23:39:20.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058754951966d4763-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.782 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.783 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.783 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.783 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.783 DEBUG:boto:Method: GET 2024-04-19T23:39:20.783 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.783 DEBUG:boto:Data: 2024-04-19T23:39:20.783 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.783 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.783 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.783 DEBUG:boto:Params: {} 2024-04-19T23:39:20.784 DEBUG:boto:Token: None 2024-04-19T23:39:20.784 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000738acebc6debf6bf-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.787 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.787 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.788 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.788 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.788 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.788 DEBUG:boto:Method: GET 2024-04-19T23:39:20.788 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.788 DEBUG:boto:Data: 2024-04-19T23:39:20.788 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.788 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.788 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.788 DEBUG:boto:Params: {} 2024-04-19T23:39:20.789 DEBUG:boto:Token: None 2024-04-19T23:39:20.789 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.789 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af3818d9daf17def-00662300a8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.792 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:20.792 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.792 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.792 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.793 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.793 DEBUG:boto:Method: GET 2024-04-19T23:39:20.793 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.793 DEBUG:boto:Data: 2024-04-19T23:39:20.793 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.793 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.793 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.793 DEBUG:boto:Params: {} 2024-04-19T23:39:20.793 DEBUG:boto:Token: None 2024-04-19T23:39:20.793 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.794 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Nbo8l739snja7w6WV53B1Snoamk= 2024-04-19T23:39:20.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Nbo8l739snja7w6WV53B1Snoamk=', 'Content-Length': '0'} 2024-04-19T23:39:20.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042bb05f6fe1e6971-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.796 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.796 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.796 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.796 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.796 DEBUG:boto:Method: GET 2024-04-19T23:39:20.796 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.796 DEBUG:boto:Data: 2024-04-19T23:39:20.796 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.796 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.796 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.796 DEBUG:boto:Params: {} 2024-04-19T23:39:20.797 DEBUG:boto:Token: None 2024-04-19T23:39:20.797 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.797 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Nbo8l739snja7w6WV53B1Snoamk= 2024-04-19T23:39:20.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Nbo8l739snja7w6WV53B1Snoamk=', 'Content-Length': '0'} 2024-04-19T23:39:20.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c3509f80b709155-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.799 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.799 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.799 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.799 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.799 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.800 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.800 DEBUG:boto:Data: 2024-04-19T23:39:20.800 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.800 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.800 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.800 DEBUG:boto:Params: {} 2024-04-19T23:39:20.800 DEBUG:boto:Token: None 2024-04-19T23:39:20.800 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.800 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bJOfMdyxjSa4FvvPfjWYWtuvicI= 2024-04-19T23:39:20.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bJOfMdyxjSa4FvvPfjWYWtuvicI=', 'Content-Length': '0'} 2024-04-19T23:39:20.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e5ac96b3b52a98c-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.802 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.802 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.803 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.803 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.803 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.803 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.803 DEBUG:boto:Data: 2024-04-19T23:39:20.803 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.803 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.803 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.803 DEBUG:boto:Params: {} 2024-04-19T23:39:20.803 DEBUG:boto:Token: None 2024-04-19T23:39:20.804 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bJOfMdyxjSa4FvvPfjWYWtuvicI= 2024-04-19T23:39:20.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bJOfMdyxjSa4FvvPfjWYWtuvicI=', 'Content-Length': '0'} 2024-04-19T23:39:20.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ea49679149619e4-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:20.806 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.806 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.806 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.806 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.806 DEBUG:boto:Method: GET 2024-04-19T23:39:20.806 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.806 DEBUG:boto:Data: 2024-04-19T23:39:20.806 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.806 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.807 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.807 DEBUG:boto:Params: {} 2024-04-19T23:39:20.807 DEBUG:boto:Token: None 2024-04-19T23:39:20.807 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.807 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Nbo8l739snja7w6WV53B1Snoamk= 2024-04-19T23:39:20.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Nbo8l739snja7w6WV53B1Snoamk=', 'Content-Length': '0'} 2024-04-19T23:39:20.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005343de584c66eb16-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.810 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.810 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.810 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.810 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.810 DEBUG:boto:Method: GET 2024-04-19T23:39:20.811 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.811 DEBUG:boto:Data: 2024-04-19T23:39:20.811 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.811 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.811 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.811 DEBUG:boto:Params: {} 2024-04-19T23:39:20.811 DEBUG:boto:Token: None 2024-04-19T23:39:20.811 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:20.812 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Nbo8l739snja7w6WV53B1Snoamk= 2024-04-19T23:39:20.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Nbo8l739snja7w6WV53B1Snoamk=', 'Content-Length': '0'} 2024-04-19T23:39:20.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000409fb30de6caf348-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.858 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.859 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.859 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.859 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.859 DEBUG:boto:Method: GET 2024-04-19T23:39:20.859 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.859 DEBUG:boto:Data: 2024-04-19T23:39:20.859 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.859 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.859 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.859 DEBUG:boto:Params: {} 2024-04-19T23:39:20.860 DEBUG:boto:Token: None 2024-04-19T23:39:20.860 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.860 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004db000fe056399b4-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.863 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.864 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.864 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.864 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.864 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.864 DEBUG:boto:Method: GET 2024-04-19T23:39:20.864 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.864 DEBUG:boto:Data: 2024-04-19T23:39:20.864 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.864 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.864 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.864 DEBUG:boto:Params: {} 2024-04-19T23:39:20.865 DEBUG:boto:Token: None 2024-04-19T23:39:20.865 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.865 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000683c65e189998ad8-00662300a8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.868 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.868 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.868 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.868 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.868 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:20.869 DEBUG:boto:Data: 2024-04-19T23:39:20.869 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.869 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.869 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.869 DEBUG:boto:Params: {} 2024-04-19T23:39:20.869 DEBUG:boto:Token: None 2024-04-19T23:39:20.869 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj 2024-04-19T23:39:20.869 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw= 2024-04-19T23:39:20.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw=', 'Content-Length': '0'} 2024-04-19T23:39:20.873 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f223f578651bf9e-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.873 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.873 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.873 DEBUG:boto:Method: HEAD 2024-04-19T23:39:20.873 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:20.873 DEBUG:boto:Data: 2024-04-19T23:39:20.873 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.873 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.873 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.874 DEBUG:boto:Params: {} 2024-04-19T23:39:20.874 DEBUG:boto:Token: None 2024-04-19T23:39:20.874 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj 2024-04-19T23:39:20.874 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw= 2024-04-19T23:39:20.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WsWR4mlkKqSvVbzA6ajFQ9HCiUw=', 'Content-Length': '0'} 2024-04-19T23:39:20.877 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000681df2656bb3b4e-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:20.878 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.878 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.878 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.878 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.878 DEBUG:boto:Method: GET 2024-04-19T23:39:20.878 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.878 DEBUG:boto:Data: 2024-04-19T23:39:20.878 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.878 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.879 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.879 DEBUG:boto:Params: {} 2024-04-19T23:39:20.879 DEBUG:boto:Token: None 2024-04-19T23:39:20.879 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.879 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40= 2024-04-19T23:39:20.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40=', 'Content-Length': '0'} 2024-04-19T23:39:20.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c0ee653bc2dfbe6-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.922 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:20.923 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:20.923 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.923 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.923 DEBUG:boto:Method: GET 2024-04-19T23:39:20.923 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.923 DEBUG:boto:Data: 2024-04-19T23:39:20.923 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.923 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.923 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.923 DEBUG:boto:Params: {} 2024-04-19T23:39:20.923 DEBUG:boto:Token: None 2024-04-19T23:39:20.924 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:20.924 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40= 2024-04-19T23:39:20.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ywp57NOumcrr548Sw9uSvri8s40=', 'Content-Length': '0'} 2024-04-19T23:39:20.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b36fb89e09ca10b-00662300a8-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.966 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.967 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.967 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.967 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.967 DEBUG:boto:Method: GET 2024-04-19T23:39:20.967 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.967 DEBUG:boto:Data: 2024-04-19T23:39:20.967 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.967 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:20.967 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.967 DEBUG:boto:Params: {} 2024-04-19T23:39:20.967 DEBUG:boto:Token: None 2024-04-19T23:39:20.968 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.968 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a3ebca3f0ac390a-00662300a8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:20.971 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:20.971 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:20.971 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:20.971 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.972 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:20.972 DEBUG:boto:Method: GET 2024-04-19T23:39:20.972 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:20.972 DEBUG:boto:Data: 2024-04-19T23:39:20.972 DEBUG:boto:Headers: {} 2024-04-19T23:39:20.972 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:20.972 DEBUG:boto:Port: 8000 2024-04-19T23:39:20.972 DEBUG:boto:Params: {} 2024-04-19T23:39:20.972 DEBUG:boto:Token: None 2024-04-19T23:39:20.972 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:20 GMT /aktlji-22/ 2024-04-19T23:39:20.972 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI= 2024-04-19T23:39:20.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zkjz8S9vQ1YUjMXWxEKJxjnLftI=', 'Content-Length': '0'} 2024-04-19T23:39:20.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6e5bbe50d91dbb3-00662300a8-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.022 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:21.023 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.023 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.023 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.023 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.023 DEBUG:boto:Method: GET 2024-04-19T23:39:21.023 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.023 DEBUG:boto:Data: 2024-04-19T23:39:21.024 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.024 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.024 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.024 DEBUG:boto:Params: {} 2024-04-19T23:39:21.024 DEBUG:boto:Token: None 2024-04-19T23:39:21.024 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.024 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u7zrSc/eiHkCba2JnnVz537T5lc= 2024-04-19T23:39:21.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u7zrSc/eiHkCba2JnnVz537T5lc=', 'Content-Length': '0'} 2024-04-19T23:39:21.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000814b8e4d09aad96c-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.027 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.027 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.027 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.027 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.027 DEBUG:boto:Method: GET 2024-04-19T23:39:21.027 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.027 DEBUG:boto:Data: 2024-04-19T23:39:21.027 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.027 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.027 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.027 DEBUG:boto:Params: {} 2024-04-19T23:39:21.028 DEBUG:boto:Token: None 2024-04-19T23:39:21.028 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.028 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u7zrSc/eiHkCba2JnnVz537T5lc= 2024-04-19T23:39:21.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u7zrSc/eiHkCba2JnnVz537T5lc=', 'Content-Length': '0'} 2024-04-19T23:39:21.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000896781f5af6e89f2-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.030 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.030 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.030 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.031 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.031 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.031 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.031 DEBUG:boto:Data: 2024-04-19T23:39:21.031 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.031 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.031 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.031 DEBUG:boto:Params: {} 2024-04-19T23:39:21.031 DEBUG:boto:Token: None 2024-04-19T23:39:21.031 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.032 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JnBOP3yjLOqq7OZtQMrLrdZNFzs= 2024-04-19T23:39:21.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JnBOP3yjLOqq7OZtQMrLrdZNFzs=', 'Content-Length': '0'} 2024-04-19T23:39:21.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b105efecb58022a1-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.034 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.034 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.034 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.034 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.034 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.034 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.034 DEBUG:boto:Data: 2024-04-19T23:39:21.034 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.034 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.035 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.035 DEBUG:boto:Params: {} 2024-04-19T23:39:21.035 DEBUG:boto:Token: None 2024-04-19T23:39:21.035 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.035 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JnBOP3yjLOqq7OZtQMrLrdZNFzs= 2024-04-19T23:39:21.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JnBOP3yjLOqq7OZtQMrLrdZNFzs=', 'Content-Length': '0'} 2024-04-19T23:39:21.037 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b29f99f54df241e-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:21.037 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.037 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.038 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.038 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.038 DEBUG:boto:Method: GET 2024-04-19T23:39:21.038 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.038 DEBUG:boto:Data: 2024-04-19T23:39:21.038 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.038 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.038 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.038 DEBUG:boto:Params: {} 2024-04-19T23:39:21.038 DEBUG:boto:Token: None 2024-04-19T23:39:21.039 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.039 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u7zrSc/eiHkCba2JnnVz537T5lc= 2024-04-19T23:39:21.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u7zrSc/eiHkCba2JnnVz537T5lc=', 'Content-Length': '0'} 2024-04-19T23:39:21.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013c39def505d8ba7-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.082 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.083 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.083 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.083 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.083 DEBUG:boto:Method: GET 2024-04-19T23:39:21.083 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.083 DEBUG:boto:Data: 2024-04-19T23:39:21.083 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.083 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.083 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.083 DEBUG:boto:Params: {} 2024-04-19T23:39:21.084 DEBUG:boto:Token: None 2024-04-19T23:39:21.084 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:21.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u7zrSc/eiHkCba2JnnVz537T5lc= 2024-04-19T23:39:21.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u7zrSc/eiHkCba2JnnVz537T5lc=', 'Content-Length': '0'} 2024-04-19T23:39:21.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a5eb5aff659e129-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.086 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.086 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.086 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.087 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.087 DEBUG:boto:Method: GET 2024-04-19T23:39:21.087 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:21.087 DEBUG:boto:Data: 2024-04-19T23:39:21.087 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.087 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.087 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.087 DEBUG:boto:Params: {} 2024-04-19T23:39:21.087 DEBUG:boto:Token: None 2024-04-19T23:39:21.087 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.088 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000739e9bad158445d8-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.138 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.139 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.139 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.139 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.139 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.139 DEBUG:boto:Method: GET 2024-04-19T23:39:21.139 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:21.140 DEBUG:boto:Data: 2024-04-19T23:39:21.140 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.140 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.140 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.140 DEBUG:boto:Params: {} 2024-04-19T23:39:21.140 DEBUG:boto:Token: None 2024-04-19T23:39:21.140 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.140 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e92efbd97278dab1-00662300a9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.146 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.146 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.146 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.147 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.147 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:21.147 DEBUG:boto:Data: 2024-04-19T23:39:21.147 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.147 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.147 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.147 DEBUG:boto:Params: {} 2024-04-19T23:39:21.147 DEBUG:boto:Token: None 2024-04-19T23:39:21.147 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj 2024-04-19T23:39:21.148 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ibVZTSpek+b9bfL4Up9XIBcrEAs= 2024-04-19T23:39:21.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ibVZTSpek+b9bfL4Up9XIBcrEAs=', 'Content-Length': '0'} 2024-04-19T23:39:21.151 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063b0eaae593c8ecb-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.151 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.151 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.151 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.151 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:21.151 DEBUG:boto:Data: 2024-04-19T23:39:21.151 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.151 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.152 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.152 DEBUG:boto:Params: {} 2024-04-19T23:39:21.152 DEBUG:boto:Token: None 2024-04-19T23:39:21.152 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj 2024-04-19T23:39:21.152 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ibVZTSpek+b9bfL4Up9XIBcrEAs= 2024-04-19T23:39:21.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ibVZTSpek+b9bfL4Up9XIBcrEAs=', 'Content-Length': '0'} 2024-04-19T23:39:21.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb9b4e83b4150bc8-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:21.156 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.156 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.156 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.156 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.156 DEBUG:boto:Method: GET 2024-04-19T23:39:21.157 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.157 DEBUG:boto:Data: 2024-04-19T23:39:21.157 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.157 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.157 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.157 DEBUG:boto:Params: {} 2024-04-19T23:39:21.157 DEBUG:boto:Token: None 2024-04-19T23:39:21.157 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.157 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU= 2024-04-19T23:39:21.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU=', 'Content-Length': '0'} 2024-04-19T23:39:21.159 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c65dd2cb5a002ba-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.160 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.160 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.160 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.160 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.160 DEBUG:boto:Method: GET 2024-04-19T23:39:21.160 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.160 DEBUG:boto:Data: 2024-04-19T23:39:21.160 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.160 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.160 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.161 DEBUG:boto:Params: {} 2024-04-19T23:39:21.161 DEBUG:boto:Token: None 2024-04-19T23:39:21.161 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.161 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU= 2024-04-19T23:39:21.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU=', 'Content-Length': '0'} 2024-04-19T23:39:21.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2da4a4e6723716e-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.206 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.207 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.207 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.207 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.207 DEBUG:boto:Method: GET 2024-04-19T23:39:21.207 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:21.207 DEBUG:boto:Data: 2024-04-19T23:39:21.207 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.207 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.207 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.207 DEBUG:boto:Params: {} 2024-04-19T23:39:21.208 DEBUG:boto:Token: None 2024-04-19T23:39:21.208 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.208 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003285108ab1ee5ea6-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.211 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.211 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.211 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.212 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.212 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.212 DEBUG:boto:Method: GET 2024-04-19T23:39:21.212 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:21.212 DEBUG:boto:Data: 2024-04-19T23:39:21.212 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.212 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.212 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.212 DEBUG:boto:Params: {} 2024-04-19T23:39:21.212 DEBUG:boto:Token: None 2024-04-19T23:39:21.213 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.213 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cf32beeedc6b17d-00662300a9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.216 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:21.216 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.216 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.216 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.216 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.216 DEBUG:boto:Method: GET 2024-04-19T23:39:21.217 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.217 DEBUG:boto:Data: 2024-04-19T23:39:21.217 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.217 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.217 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.217 DEBUG:boto:Params: {} 2024-04-19T23:39:21.217 DEBUG:boto:Token: None 2024-04-19T23:39:21.217 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.217 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Eg7Raub3gfhaphB4V56/rAy1o5k= 2024-04-19T23:39:21.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Eg7Raub3gfhaphB4V56/rAy1o5k=', 'Content-Length': '0'} 2024-04-19T23:39:21.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004775eef4815f8012-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.262 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.262 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.263 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.263 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.263 DEBUG:boto:Method: GET 2024-04-19T23:39:21.263 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.263 DEBUG:boto:Data: 2024-04-19T23:39:21.263 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.263 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.263 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.264 DEBUG:boto:Params: {} 2024-04-19T23:39:21.264 DEBUG:boto:Token: None 2024-04-19T23:39:21.264 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.264 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Eg7Raub3gfhaphB4V56/rAy1o5k= 2024-04-19T23:39:21.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Eg7Raub3gfhaphB4V56/rAy1o5k=', 'Content-Length': '0'} 2024-04-19T23:39:21.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008935ca5b301b6796-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.266 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.267 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.267 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.267 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.267 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.267 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.267 DEBUG:boto:Data: 2024-04-19T23:39:21.267 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.267 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.267 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.267 DEBUG:boto:Params: {} 2024-04-19T23:39:21.268 DEBUG:boto:Token: None 2024-04-19T23:39:21.268 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.268 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3pyWlbXbGYfNrvnGfAfWvFhYeUo= 2024-04-19T23:39:21.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3pyWlbXbGYfNrvnGfAfWvFhYeUo=', 'Content-Length': '0'} 2024-04-19T23:39:21.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b93a344a53980838-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.270 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.270 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.270 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.270 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.270 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.270 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.271 DEBUG:boto:Data: 2024-04-19T23:39:21.271 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.271 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.271 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.271 DEBUG:boto:Params: {} 2024-04-19T23:39:21.271 DEBUG:boto:Token: None 2024-04-19T23:39:21.271 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.271 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3pyWlbXbGYfNrvnGfAfWvFhYeUo= 2024-04-19T23:39:21.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3pyWlbXbGYfNrvnGfAfWvFhYeUo=', 'Content-Length': '0'} 2024-04-19T23:39:21.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d5ad67f58058eb4-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:21.274 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.274 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.274 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.274 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.274 DEBUG:boto:Method: GET 2024-04-19T23:39:21.274 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.274 DEBUG:boto:Data: 2024-04-19T23:39:21.274 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.274 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.274 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.274 DEBUG:boto:Params: {} 2024-04-19T23:39:21.275 DEBUG:boto:Token: None 2024-04-19T23:39:21.275 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.275 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Eg7Raub3gfhaphB4V56/rAy1o5k= 2024-04-19T23:39:21.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Eg7Raub3gfhaphB4V56/rAy1o5k=', 'Content-Length': '0'} 2024-04-19T23:39:21.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae49122b456e0163-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.322 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.322 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.323 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.323 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.323 DEBUG:boto:Method: GET 2024-04-19T23:39:21.323 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.323 DEBUG:boto:Data: 2024-04-19T23:39:21.323 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.323 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.323 DEBUG:boto:Params: {} 2024-04-19T23:39:21.324 DEBUG:boto:Token: None 2024-04-19T23:39:21.324 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:21.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Eg7Raub3gfhaphB4V56/rAy1o5k= 2024-04-19T23:39:21.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Eg7Raub3gfhaphB4V56/rAy1o5k=', 'Content-Length': '0'} 2024-04-19T23:39:21.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4198bc921ed0906-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.366 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.366 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.367 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.367 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.367 DEBUG:boto:Method: GET 2024-04-19T23:39:21.367 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:21.367 DEBUG:boto:Data: 2024-04-19T23:39:21.367 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.367 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.367 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.367 DEBUG:boto:Params: {} 2024-04-19T23:39:21.368 DEBUG:boto:Token: None 2024-04-19T23:39:21.368 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.368 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f668417093a34ae7-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.371 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.371 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.372 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.372 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.372 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.372 DEBUG:boto:Method: GET 2024-04-19T23:39:21.372 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:21.372 DEBUG:boto:Data: 2024-04-19T23:39:21.372 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.372 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.373 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.373 DEBUG:boto:Params: {} 2024-04-19T23:39:21.373 DEBUG:boto:Token: None 2024-04-19T23:39:21.373 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.373 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000539b1dc70b14cce0-00662300a9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.376 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.377 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.377 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.377 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.377 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:21.377 DEBUG:boto:Data: 2024-04-19T23:39:21.377 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.377 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.377 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.378 DEBUG:boto:Params: {} 2024-04-19T23:39:21.378 DEBUG:boto:Token: None 2024-04-19T23:39:21.378 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj 2024-04-19T23:39:21.378 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ibVZTSpek+b9bfL4Up9XIBcrEAs= 2024-04-19T23:39:21.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ibVZTSpek+b9bfL4Up9XIBcrEAs=', 'Content-Length': '0'} 2024-04-19T23:39:21.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f5088a7433598b0-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.382 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.382 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.382 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.382 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:21.382 DEBUG:boto:Data: 2024-04-19T23:39:21.382 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.382 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.382 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.382 DEBUG:boto:Params: {} 2024-04-19T23:39:21.383 DEBUG:boto:Token: None 2024-04-19T23:39:21.383 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj 2024-04-19T23:39:21.383 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ibVZTSpek+b9bfL4Up9XIBcrEAs= 2024-04-19T23:39:21.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ibVZTSpek+b9bfL4Up9XIBcrEAs=', 'Content-Length': '0'} 2024-04-19T23:39:21.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001570b1405d4795c9-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:21.387 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.387 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.387 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.387 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.387 DEBUG:boto:Method: GET 2024-04-19T23:39:21.387 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.387 DEBUG:boto:Data: 2024-04-19T23:39:21.388 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.388 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.388 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.388 DEBUG:boto:Params: {} 2024-04-19T23:39:21.388 DEBUG:boto:Token: None 2024-04-19T23:39:21.388 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.388 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU= 2024-04-19T23:39:21.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU=', 'Content-Length': '0'} 2024-04-19T23:39:21.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b1a9a5d92af2709-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.430 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.431 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.431 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.431 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.431 DEBUG:boto:Method: GET 2024-04-19T23:39:21.431 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.431 DEBUG:boto:Data: 2024-04-19T23:39:21.431 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.431 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.431 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.432 DEBUG:boto:Params: {} 2024-04-19T23:39:21.432 DEBUG:boto:Token: None 2024-04-19T23:39:21.432 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.432 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU= 2024-04-19T23:39:21.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU=', 'Content-Length': '0'} 2024-04-19T23:39:21.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000019e17875dbd2554-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.479 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.479 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.479 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.479 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.479 DEBUG:boto:Method: GET 2024-04-19T23:39:21.479 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:21.479 DEBUG:boto:Data: 2024-04-19T23:39:21.479 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.479 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.479 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.479 DEBUG:boto:Params: {} 2024-04-19T23:39:21.480 DEBUG:boto:Token: None 2024-04-19T23:39:21.480 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.480 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000085950471de5d3bc-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.483 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.483 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.484 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.484 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.484 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.484 DEBUG:boto:Method: GET 2024-04-19T23:39:21.484 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:21.484 DEBUG:boto:Data: 2024-04-19T23:39:21.484 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.485 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.485 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.485 DEBUG:boto:Params: {} 2024-04-19T23:39:21.485 DEBUG:boto:Token: None 2024-04-19T23:39:21.485 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.485 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000253b6343a6f6e57f-00662300a9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.534 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:21.535 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.535 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.535 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.535 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.535 DEBUG:boto:Method: GET 2024-04-19T23:39:21.535 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.535 DEBUG:boto:Data: 2024-04-19T23:39:21.536 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.536 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.536 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.536 DEBUG:boto:Params: {} 2024-04-19T23:39:21.536 DEBUG:boto:Token: None 2024-04-19T23:39:21.536 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.536 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WGnvzr77l2haEyMTRfGtoKTlgq8= 2024-04-19T23:39:21.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WGnvzr77l2haEyMTRfGtoKTlgq8=', 'Content-Length': '0'} 2024-04-19T23:39:21.538 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f7b261f1e1d92bc-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.539 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.539 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.539 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.539 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.539 DEBUG:boto:Method: GET 2024-04-19T23:39:21.539 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.539 DEBUG:boto:Data: 2024-04-19T23:39:21.539 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.539 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.540 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.540 DEBUG:boto:Params: {} 2024-04-19T23:39:21.540 DEBUG:boto:Token: None 2024-04-19T23:39:21.540 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.540 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WGnvzr77l2haEyMTRfGtoKTlgq8= 2024-04-19T23:39:21.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WGnvzr77l2haEyMTRfGtoKTlgq8=', 'Content-Length': '0'} 2024-04-19T23:39:21.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a3840b39d6095a4-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.543 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.543 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.543 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.543 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.543 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.543 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.543 DEBUG:boto:Data: 2024-04-19T23:39:21.543 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.543 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.543 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.543 DEBUG:boto:Params: {} 2024-04-19T23:39:21.544 DEBUG:boto:Token: None 2024-04-19T23:39:21.544 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.544 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DkSq8jVk7YPDgOSzqRxwvk99bqQ= 2024-04-19T23:39:21.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DkSq8jVk7YPDgOSzqRxwvk99bqQ=', 'Content-Length': '0'} 2024-04-19T23:39:21.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbe2f1aeb149db9f-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.546 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.546 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.546 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.546 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.546 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.546 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.547 DEBUG:boto:Data: 2024-04-19T23:39:21.547 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.547 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.547 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.547 DEBUG:boto:Params: {} 2024-04-19T23:39:21.547 DEBUG:boto:Token: None 2024-04-19T23:39:21.547 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.547 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DkSq8jVk7YPDgOSzqRxwvk99bqQ= 2024-04-19T23:39:21.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DkSq8jVk7YPDgOSzqRxwvk99bqQ=', 'Content-Length': '0'} 2024-04-19T23:39:21.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f44579c8296ca75-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:21.549 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.550 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.550 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.550 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.550 DEBUG:boto:Method: GET 2024-04-19T23:39:21.550 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.550 DEBUG:boto:Data: 2024-04-19T23:39:21.550 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.550 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.550 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.550 DEBUG:boto:Params: {} 2024-04-19T23:39:21.550 DEBUG:boto:Token: None 2024-04-19T23:39:21.551 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.551 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WGnvzr77l2haEyMTRfGtoKTlgq8= 2024-04-19T23:39:21.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WGnvzr77l2haEyMTRfGtoKTlgq8=', 'Content-Length': '0'} 2024-04-19T23:39:21.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbb5c51c466b5d78-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.553 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.553 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.553 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.553 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.553 DEBUG:boto:Method: GET 2024-04-19T23:39:21.553 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.553 DEBUG:boto:Data: 2024-04-19T23:39:21.553 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.554 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.554 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.554 DEBUG:boto:Params: {} 2024-04-19T23:39:21.554 DEBUG:boto:Token: None 2024-04-19T23:39:21.554 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:21.554 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WGnvzr77l2haEyMTRfGtoKTlgq8= 2024-04-19T23:39:21.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WGnvzr77l2haEyMTRfGtoKTlgq8=', 'Content-Length': '0'} 2024-04-19T23:39:21.556 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006286371228fbce81-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.598 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.599 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.599 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.599 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.599 DEBUG:boto:Method: GET 2024-04-19T23:39:21.599 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:21.599 DEBUG:boto:Data: 2024-04-19T23:39:21.599 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.599 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.599 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.599 DEBUG:boto:Params: {} 2024-04-19T23:39:21.600 DEBUG:boto:Token: None 2024-04-19T23:39:21.600 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.600 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074c111710bbd9daa-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.603 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.604 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.604 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.604 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.604 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.604 DEBUG:boto:Method: GET 2024-04-19T23:39:21.604 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:21.604 DEBUG:boto:Data: 2024-04-19T23:39:21.604 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.604 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.604 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.605 DEBUG:boto:Params: {} 2024-04-19T23:39:21.605 DEBUG:boto:Token: None 2024-04-19T23:39:21.605 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.605 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecb82177a65d97fd-00662300a9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.651 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.652 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.652 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.652 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.652 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:21.652 DEBUG:boto:Data: 2024-04-19T23:39:21.652 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.652 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.652 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.652 DEBUG:boto:Params: {} 2024-04-19T23:39:21.653 DEBUG:boto:Token: None 2024-04-19T23:39:21.653 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj 2024-04-19T23:39:21.653 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ibVZTSpek+b9bfL4Up9XIBcrEAs= 2024-04-19T23:39:21.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ibVZTSpek+b9bfL4Up9XIBcrEAs=', 'Content-Length': '0'} 2024-04-19T23:39:21.656 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a1eeb0e5f39c0ba-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.657 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.657 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.657 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.658 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:21.658 DEBUG:boto:Data: 2024-04-19T23:39:21.658 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.658 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.658 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.658 DEBUG:boto:Params: {} 2024-04-19T23:39:21.658 DEBUG:boto:Token: None 2024-04-19T23:39:21.658 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj 2024-04-19T23:39:21.659 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ibVZTSpek+b9bfL4Up9XIBcrEAs= 2024-04-19T23:39:21.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ibVZTSpek+b9bfL4Up9XIBcrEAs=', 'Content-Length': '0'} 2024-04-19T23:39:21.663 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ee217637f5404c5-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:21.664 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.664 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.664 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.664 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.664 DEBUG:boto:Method: GET 2024-04-19T23:39:21.665 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.665 DEBUG:boto:Data: 2024-04-19T23:39:21.665 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.665 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.665 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.665 DEBUG:boto:Params: {} 2024-04-19T23:39:21.665 DEBUG:boto:Token: None 2024-04-19T23:39:21.666 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.666 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU= 2024-04-19T23:39:21.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU=', 'Content-Length': '0'} 2024-04-19T23:39:21.668 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ced2826581970679-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.710 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.711 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.711 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.711 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.711 DEBUG:boto:Method: GET 2024-04-19T23:39:21.711 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.711 DEBUG:boto:Data: 2024-04-19T23:39:21.711 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.711 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.711 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.711 DEBUG:boto:Params: {} 2024-04-19T23:39:21.712 DEBUG:boto:Token: None 2024-04-19T23:39:21.712 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.712 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU= 2024-04-19T23:39:21.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU=', 'Content-Length': '0'} 2024-04-19T23:39:21.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000810e561824357adb-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.758 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.759 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.759 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.759 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.759 DEBUG:boto:Method: GET 2024-04-19T23:39:21.759 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:21.759 DEBUG:boto:Data: 2024-04-19T23:39:21.759 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.759 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:21.759 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.759 DEBUG:boto:Params: {} 2024-04-19T23:39:21.759 DEBUG:boto:Token: None 2024-04-19T23:39:21.760 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.760 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a8beda987123570-00662300a9-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.763 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.763 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.764 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.764 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.764 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:21.764 DEBUG:boto:Method: GET 2024-04-19T23:39:21.764 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:21.764 DEBUG:boto:Data: 2024-04-19T23:39:21.764 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.764 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.764 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.764 DEBUG:boto:Params: {} 2024-04-19T23:39:21.764 DEBUG:boto:Token: None 2024-04-19T23:39:21.765 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.766 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093bc886c2e51164c-00662300a9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.773 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.774 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-22 zones={test-zone2, test-zone1} 2024-04-19T23:39:21.774 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-22 zones={test-zone3, test-zone1} 2024-04-19T23:39:21.774 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.774 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.774 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.775 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:39:21.775 DEBUG:boto:Data: 2024-04-19T23:39:21.775 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.775 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:21.775 DEBUG:boto:Port: 8001 2024-04-19T23:39:21.775 DEBUG:boto:Params: {} 2024-04-19T23:39:21.775 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-19T23:39:21.775 DEBUG:boto:Token: None 2024-04-19T23:39:21.775 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.776 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sBCJgRkhQTpoledQ4GrkQdx0SnQ= 2024-04-19T23:39:21.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sBCJgRkhQTpoledQ4GrkQdx0SnQ=', 'Content-Length': '0'} 2024-04-19T23:39:21.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e692c87243040598-00662300a9-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.805 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.806 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.806 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.806 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:39:21.806 DEBUG:boto:Data: 2024-04-19T23:39:21.806 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.806 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.806 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.806 DEBUG:boto:Params: {} 2024-04-19T23:39:21.806 DEBUG:boto:Token: None 2024-04-19T23:39:21.807 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.807 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sBCJgRkhQTpoledQ4GrkQdx0SnQ= 2024-04-19T23:39:21.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sBCJgRkhQTpoledQ4GrkQdx0SnQ=', 'Content-Length': '0'} 2024-04-19T23:39:21.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014ac0e87052b9ee0-00662300a9-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.810 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:39:21.810 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.810 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.810 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:21.810 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:21.810 DEBUG:boto:Method: GET 2024-04-19T23:39:21.811 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:39:21.811 DEBUG:boto:Data: 2024-04-19T23:39:21.811 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.811 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:21.811 DEBUG:boto:Port: 8001 2024-04-19T23:39:21.811 DEBUG:boto:Params: {} 2024-04-19T23:39:21.811 DEBUG:boto:Token: None 2024-04-19T23:39:21.811 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/?versions 2024-04-19T23:39:21.811 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BOXXFRkCOkcJaD5mwpWrVCM/uf4= 2024-04-19T23:39:21.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BOXXFRkCOkcJaD5mwpWrVCM/uf4=', 'Content-Length': '0'} 2024-04-19T23:39:21.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd6f248121d2a807-00662300a9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.815 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOotrue2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjI8Qos7O61Drj6zKFSuRQg5BJ8qTn1Itfalse2024-04-19T23:36:20.273Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITUfalse2024-04-19T23:36:20.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUjbMVzWXMoq8kQHxc5fXZiC69vNXkcjfalse2024-04-19T23:36:20.222Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97false2024-04-19T23:36:20.188Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2false2024-04-19T23:36:20.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Jy1wSwswautcJ.pAM0qXnm5CTgNceOfalse2024-04-19T23:36:20.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-false2024-04-19T23:36:20.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7false2024-04-19T23:36:20.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQ1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgRfalse2024-04-19T23:36:20.019Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipPC6NaG1ejWpptAGf0zgZ-gqpdm0DMfalse2024-04-19T23:36:20.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoECNSXdMldM0FoTJgSxtKpnhGWUHc5tfalse2024-04-19T23:36:20.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFo79-FV5lzRfFhYZKDH5zbjzjFRvFmofalse2024-04-19T23:36:19.985Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.9ZzH.1hozF9j0blCcYN54OsXVmWMTfalse2024-04-19T23:36:19.968Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnLS7759d.h0bWJMnVj7IAdVTVbZ2LzAfalse2024-04-19T23:36:20.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.SsB-LJAFCELIGc-2lad2-2jTKdmQIufalse2024-04-19T23:36:19.904Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5IMca-NQG3TDmPnhaU0a46jfFv2nxR8false2024-04-19T23:36:19.920Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3PIDP6j1VZfmBljRrKUGkIUmq4DPgifalse2024-04-19T23:36:19.849Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.sqcIpylWkdZwQp4UqpDBbyySodTBXSfalse2024-04-19T23:36:19.778Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCzz.Nb3nU2CWGe2vMCDYlTHkK18iRLfalse2024-04-19T23:36:19.760Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjudeGFtd00C8TRIOlco59Eefd7c7ffA6false2024-04-19T23:36:19.831Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4false2024-04-19T23:36:19.744Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgoQj3EN8LOJNZ9C2JDigvTcIgypIbDxfalse2024-04-19T23:36:19.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzyfalse2024-04-19T23:36:19.727Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.169NmmrUry8iq70c7.ZNdp4i5iqOCFfalse2024-04-19T23:36:19.659Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE5taibPWiUs3cuEimynrjQD8cXTCAELfalse2024-04-19T23:36:19.589Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNjFtS4LHZJDkGsaKejPMQyqPrZjbuN2false2024-04-19T23:36:19.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmTjHqLZC66BQsk7jiah7FJSChQvGdxfalse2024-04-19T23:36:19.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlKfLYYBA1E.lSLR49AMlK-Kgik5ij5Rfalse2024-04-19T23:36:19.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzwsSl6whrs-j2uo2xvsRdnhCfJSQ4Mafalse2024-04-19T23:36:19.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.817 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:39:21.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.819 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:39:21.819 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.820 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.820 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:21.820 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:21.820 DEBUG:boto:Method: GET 2024-04-19T23:39:21.820 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:39:21.820 DEBUG:boto:Data: 2024-04-19T23:39:21.820 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.820 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.820 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.820 DEBUG:boto:Params: {} 2024-04-19T23:39:21.820 DEBUG:boto:Token: None 2024-04-19T23:39:21.821 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/?versions 2024-04-19T23:39:21.821 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BOXXFRkCOkcJaD5mwpWrVCM/uf4= 2024-04-19T23:39:21.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BOXXFRkCOkcJaD5mwpWrVCM/uf4=', 'Content-Length': '0'} 2024-04-19T23:39:21.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005746b8626acedfa-00662300a9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.824 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOotrue2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjI8Qos7O61Drj6zKFSuRQg5BJ8qTn1Itfalse2024-04-19T23:36:20.273Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITUfalse2024-04-19T23:36:20.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUjbMVzWXMoq8kQHxc5fXZiC69vNXkcjfalse2024-04-19T23:36:20.222Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97false2024-04-19T23:36:20.188Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2false2024-04-19T23:36:20.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Jy1wSwswautcJ.pAM0qXnm5CTgNceOfalse2024-04-19T23:36:20.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-false2024-04-19T23:36:20.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7false2024-04-19T23:36:20.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQ1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgRfalse2024-04-19T23:36:20.019Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipPC6NaG1ejWpptAGf0zgZ-gqpdm0DMfalse2024-04-19T23:36:20.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoECNSXdMldM0FoTJgSxtKpnhGWUHc5tfalse2024-04-19T23:36:20.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFo79-FV5lzRfFhYZKDH5zbjzjFRvFmofalse2024-04-19T23:36:19.985Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.9ZzH.1hozF9j0blCcYN54OsXVmWMTfalse2024-04-19T23:36:19.968Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnLS7759d.h0bWJMnVj7IAdVTVbZ2LzAfalse2024-04-19T23:36:20.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.SsB-LJAFCELIGc-2lad2-2jTKdmQIufalse2024-04-19T23:36:19.904Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5IMca-NQG3TDmPnhaU0a46jfFv2nxR8false2024-04-19T23:36:19.920Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3PIDP6j1VZfmBljRrKUGkIUmq4DPgifalse2024-04-19T23:36:19.849Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.sqcIpylWkdZwQp4UqpDBbyySodTBXSfalse2024-04-19T23:36:19.778Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCzz.Nb3nU2CWGe2vMCDYlTHkK18iRLfalse2024-04-19T23:36:19.760Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjudeGFtd00C8TRIOlco59Eefd7c7ffA6false2024-04-19T23:36:19.831Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4false2024-04-19T23:36:19.744Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgoQj3EN8LOJNZ9C2JDigvTcIgypIbDxfalse2024-04-19T23:36:19.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzyfalse2024-04-19T23:36:19.727Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.169NmmrUry8iq70c7.ZNdp4i5iqOCFfalse2024-04-19T23:36:19.659Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE5taibPWiUs3cuEimynrjQD8cXTCAELfalse2024-04-19T23:36:19.589Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNjFtS4LHZJDkGsaKejPMQyqPrZjbuN2false2024-04-19T23:36:19.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmTjHqLZC66BQsk7jiah7FJSChQvGdxfalse2024-04-19T23:36:19.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlKfLYYBA1E.lSLR49AMlK-Kgik5ij5Rfalse2024-04-19T23:36:19.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzwsSl6whrs-j2uo2xvsRdnhCfJSQ4Mafalse2024-04-19T23:36:19.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.826 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:39:21.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:21.829 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.829 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.829 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:21.829 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:21.829 DEBUG:boto:Method: GET 2024-04-19T23:39:21.829 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:39:21.829 DEBUG:boto:Data: 2024-04-19T23:39:21.829 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.829 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:21.829 DEBUG:boto:Port: 8001 2024-04-19T23:39:21.829 DEBUG:boto:Params: {} 2024-04-19T23:39:21.830 DEBUG:boto:Token: None 2024-04-19T23:39:21.830 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/?versions 2024-04-19T23:39:21.830 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BOXXFRkCOkcJaD5mwpWrVCM/uf4= 2024-04-19T23:39:21.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BOXXFRkCOkcJaD5mwpWrVCM/uf4=', 'Content-Length': '0'} 2024-04-19T23:39:21.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f875fae3b18102e6-00662300a9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.834 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOotrue2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjI8Qos7O61Drj6zKFSuRQg5BJ8qTn1Itfalse2024-04-19T23:36:20.273Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITUfalse2024-04-19T23:36:20.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUjbMVzWXMoq8kQHxc5fXZiC69vNXkcjfalse2024-04-19T23:36:20.222Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97false2024-04-19T23:36:20.188Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2false2024-04-19T23:36:20.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Jy1wSwswautcJ.pAM0qXnm5CTgNceOfalse2024-04-19T23:36:20.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-false2024-04-19T23:36:20.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7false2024-04-19T23:36:20.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQ1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgRfalse2024-04-19T23:36:20.019Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipPC6NaG1ejWpptAGf0zgZ-gqpdm0DMfalse2024-04-19T23:36:20.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoECNSXdMldM0FoTJgSxtKpnhGWUHc5tfalse2024-04-19T23:36:20.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFo79-FV5lzRfFhYZKDH5zbjzjFRvFmofalse2024-04-19T23:36:19.985Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.9ZzH.1hozF9j0blCcYN54OsXVmWMTfalse2024-04-19T23:36:19.968Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnLS7759d.h0bWJMnVj7IAdVTVbZ2LzAfalse2024-04-19T23:36:20.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.SsB-LJAFCELIGc-2lad2-2jTKdmQIufalse2024-04-19T23:36:19.904Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5IMca-NQG3TDmPnhaU0a46jfFv2nxR8false2024-04-19T23:36:19.920Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3PIDP6j1VZfmBljRrKUGkIUmq4DPgifalse2024-04-19T23:36:19.849Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.sqcIpylWkdZwQp4UqpDBbyySodTBXSfalse2024-04-19T23:36:19.778Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCzz.Nb3nU2CWGe2vMCDYlTHkK18iRLfalse2024-04-19T23:36:19.760Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjudeGFtd00C8TRIOlco59Eefd7c7ffA6false2024-04-19T23:36:19.831Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4false2024-04-19T23:36:19.744Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgoQj3EN8LOJNZ9C2JDigvTcIgypIbDxfalse2024-04-19T23:36:19.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzyfalse2024-04-19T23:36:19.727Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.169NmmrUry8iq70c7.ZNdp4i5iqOCFfalse2024-04-19T23:36:19.659Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE5taibPWiUs3cuEimynrjQD8cXTCAELfalse2024-04-19T23:36:19.589Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNjFtS4LHZJDkGsaKejPMQyqPrZjbuN2false2024-04-19T23:36:19.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmTjHqLZC66BQsk7jiah7FJSChQvGdxfalse2024-04-19T23:36:19.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlKfLYYBA1E.lSLR49AMlK-Kgik5ij5Rfalse2024-04-19T23:36:19.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzwsSl6whrs-j2uo2xvsRdnhCfJSQ4Mafalse2024-04-19T23:36:19.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.835 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.835 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.835 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:21.836 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:21.836 DEBUG:boto:Method: GET 2024-04-19T23:39:21.836 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:39:21.836 DEBUG:boto:Data: 2024-04-19T23:39:21.836 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.836 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.836 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.836 DEBUG:boto:Params: {} 2024-04-19T23:39:21.836 DEBUG:boto:Token: None 2024-04-19T23:39:21.836 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/?versions 2024-04-19T23:39:21.837 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BOXXFRkCOkcJaD5mwpWrVCM/uf4= 2024-04-19T23:39:21.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BOXXFRkCOkcJaD5mwpWrVCM/uf4=', 'Content-Length': '0'} 2024-04-19T23:39:21.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064a27457b8801742-00662300a9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.840 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOotrue2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjI8Qos7O61Drj6zKFSuRQg5BJ8qTn1Itfalse2024-04-19T23:36:20.273Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITUfalse2024-04-19T23:36:20.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUjbMVzWXMoq8kQHxc5fXZiC69vNXkcjfalse2024-04-19T23:36:20.222Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97false2024-04-19T23:36:20.188Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2false2024-04-19T23:36:20.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Jy1wSwswautcJ.pAM0qXnm5CTgNceOfalse2024-04-19T23:36:20.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-false2024-04-19T23:36:20.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7false2024-04-19T23:36:20.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQ1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgRfalse2024-04-19T23:36:20.019Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipPC6NaG1ejWpptAGf0zgZ-gqpdm0DMfalse2024-04-19T23:36:20.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoECNSXdMldM0FoTJgSxtKpnhGWUHc5tfalse2024-04-19T23:36:20.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFo79-FV5lzRfFhYZKDH5zbjzjFRvFmofalse2024-04-19T23:36:19.985Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.9ZzH.1hozF9j0blCcYN54OsXVmWMTfalse2024-04-19T23:36:19.968Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnLS7759d.h0bWJMnVj7IAdVTVbZ2LzAfalse2024-04-19T23:36:20.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.SsB-LJAFCELIGc-2lad2-2jTKdmQIufalse2024-04-19T23:36:19.904Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5IMca-NQG3TDmPnhaU0a46jfFv2nxR8false2024-04-19T23:36:19.920Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3PIDP6j1VZfmBljRrKUGkIUmq4DPgifalse2024-04-19T23:36:19.849Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.sqcIpylWkdZwQp4UqpDBbyySodTBXSfalse2024-04-19T23:36:19.778Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCzz.Nb3nU2CWGe2vMCDYlTHkK18iRLfalse2024-04-19T23:36:19.760Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjudeGFtd00C8TRIOlco59Eefd7c7ffA6false2024-04-19T23:36:19.831Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4false2024-04-19T23:36:19.744Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgoQj3EN8LOJNZ9C2JDigvTcIgypIbDxfalse2024-04-19T23:36:19.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzyfalse2024-04-19T23:36:19.727Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.169NmmrUry8iq70c7.ZNdp4i5iqOCFfalse2024-04-19T23:36:19.659Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE5taibPWiUs3cuEimynrjQD8cXTCAELfalse2024-04-19T23:36:19.589Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNjFtS4LHZJDkGsaKejPMQyqPrZjbuN2false2024-04-19T23:36:19.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmTjHqLZC66BQsk7jiah7FJSChQvGdxfalse2024-04-19T23:36:19.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlKfLYYBA1E.lSLR49AMlK-Kgik5ij5Rfalse2024-04-19T23:36:19.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzwsSl6whrs-j2uo2xvsRdnhCfJSQ4Mafalse2024-04-19T23:36:19.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:39:21.842 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:21.842 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:21.842 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.842 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.842 DEBUG:boto:Method: GET 2024-04-19T23:39:21.842 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.842 DEBUG:boto:Data: 2024-04-19T23:39:21.843 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.843 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:21.843 DEBUG:boto:Port: 8001 2024-04-19T23:39:21.843 DEBUG:boto:Params: {} 2024-04-19T23:39:21.843 DEBUG:boto:Token: None 2024-04-19T23:39:21.844 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.844 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0UwUsV9aOUZ0GXxYttmERFWZQFQ= 2024-04-19T23:39:21.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0UwUsV9aOUZ0GXxYttmERFWZQFQ=', 'Content-Length': '0'} 2024-04-19T23:39:21.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000005cffed91d88fd56-00662300a9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.847 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:21.847 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:21.847 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.847 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.847 DEBUG:boto:Method: GET 2024-04-19T23:39:21.847 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.847 DEBUG:boto:Data: 2024-04-19T23:39:21.847 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.847 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.848 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.848 DEBUG:boto:Params: {} 2024-04-19T23:39:21.848 DEBUG:boto:Token: None 2024-04-19T23:39:21.848 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.848 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0UwUsV9aOUZ0GXxYttmERFWZQFQ= 2024-04-19T23:39:21.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0UwUsV9aOUZ0GXxYttmERFWZQFQ=', 'Content-Length': '0'} 2024-04-19T23:39:21.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e936936ed73beaca-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.850 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:21.850 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:21.850 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.850 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.851 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.851 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.851 DEBUG:boto:Data: 2024-04-19T23:39:21.851 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.851 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:21.851 DEBUG:boto:Port: 8001 2024-04-19T23:39:21.851 DEBUG:boto:Params: {} 2024-04-19T23:39:21.851 DEBUG:boto:Token: None 2024-04-19T23:39:21.851 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.851 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rDwTK/ZDb4BG9y3fCU/NTuYRYIE= 2024-04-19T23:39:21.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rDwTK/ZDb4BG9y3fCU/NTuYRYIE=', 'Content-Length': '0'} 2024-04-19T23:39:21.853 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005b34015cf9bc6007-00662300a9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.853 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:21.854 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:21.854 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.854 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.855 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.855 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.855 DEBUG:boto:Data: 2024-04-19T23:39:21.855 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.855 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.855 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.855 DEBUG:boto:Params: {} 2024-04-19T23:39:21.855 DEBUG:boto:Token: None 2024-04-19T23:39:21.855 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.856 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rDwTK/ZDb4BG9y3fCU/NTuYRYIE= 2024-04-19T23:39:21.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rDwTK/ZDb4BG9y3fCU/NTuYRYIE=', 'Content-Length': '0'} 2024-04-19T23:39:21.858 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c56ecb2043fb7240-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:39:21.858 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:21.858 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:21.858 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.858 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.858 DEBUG:boto:Method: GET 2024-04-19T23:39:21.859 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.859 DEBUG:boto:Data: 2024-04-19T23:39:21.859 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.859 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:21.859 DEBUG:boto:Port: 8001 2024-04-19T23:39:21.859 DEBUG:boto:Params: {} 2024-04-19T23:39:21.859 DEBUG:boto:Token: None 2024-04-19T23:39:21.860 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.860 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0UwUsV9aOUZ0GXxYttmERFWZQFQ= 2024-04-19T23:39:21.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0UwUsV9aOUZ0GXxYttmERFWZQFQ=', 'Content-Length': '0'} 2024-04-19T23:39:21.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f1e0ea778302b748-00662300a9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.862 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:21.862 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:21.862 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.862 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.862 DEBUG:boto:Method: GET 2024-04-19T23:39:21.862 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.862 DEBUG:boto:Data: 2024-04-19T23:39:21.862 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.862 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.862 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.862 DEBUG:boto:Params: {} 2024-04-19T23:39:21.863 DEBUG:boto:Token: None 2024-04-19T23:39:21.863 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.863 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0UwUsV9aOUZ0GXxYttmERFWZQFQ= 2024-04-19T23:39:21.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0UwUsV9aOUZ0GXxYttmERFWZQFQ=', 'Content-Length': '0'} 2024-04-19T23:39:21.865 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001cf68e0609f27f17-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.865 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.866 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.866 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:21.866 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:21.866 DEBUG:boto:Method: GET 2024-04-19T23:39:21.866 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:39:21.866 DEBUG:boto:Data: 2024-04-19T23:39:21.866 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.866 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:21.866 DEBUG:boto:Port: 8001 2024-04-19T23:39:21.866 DEBUG:boto:Params: {} 2024-04-19T23:39:21.867 DEBUG:boto:Token: None 2024-04-19T23:39:21.867 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.867 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cb0a4468f1fde8b-00662300a9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.870 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.870 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.870 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.870 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:21.870 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:21.870 DEBUG:boto:Method: GET 2024-04-19T23:39:21.871 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:39:21.871 DEBUG:boto:Data: 2024-04-19T23:39:21.871 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.871 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.871 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.871 DEBUG:boto:Params: {} 2024-04-19T23:39:21.871 DEBUG:boto:Token: None 2024-04-19T23:39:21.872 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.872 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005157dacc785d3ea5-00662300a9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.918 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.919 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:21.919 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:21.920 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.920 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:39:21.920 DEBUG:boto:Data: 2024-04-19T23:39:21.920 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.921 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:21.921 DEBUG:boto:Port: 8001 2024-04-19T23:39:21.921 DEBUG:boto:Params: {} 2024-04-19T23:39:21.921 DEBUG:boto:Token: None 2024-04-19T23:39:21.922 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/dummy 2024-04-19T23:39:21.922 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qDolQwenmv8m0/4V42hICIOC4uc= 2024-04-19T23:39:21.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qDolQwenmv8m0/4V42hICIOC4uc=', 'Content-Length': '0'} 2024-04-19T23:39:21.925 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e9d5d4b803b4d6f-00662300a9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.925 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:21.926 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:21.926 DEBUG:boto:Method: HEAD 2024-04-19T23:39:21.926 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:39:21.926 DEBUG:boto:Data: 2024-04-19T23:39:21.927 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.927 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.927 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.927 DEBUG:boto:Params: {} 2024-04-19T23:39:21.927 DEBUG:boto:Token: None 2024-04-19T23:39:21.928 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/dummy 2024-04-19T23:39:21.928 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qDolQwenmv8m0/4V42hICIOC4uc= 2024-04-19T23:39:21.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qDolQwenmv8m0/4V42hICIOC4uc=', 'Content-Length': '0'} 2024-04-19T23:39:21.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000071c6d599a2cb150-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:39:21.931 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:21.931 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:21.932 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.932 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.932 DEBUG:boto:Method: GET 2024-04-19T23:39:21.932 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.932 DEBUG:boto:Data: 2024-04-19T23:39:21.932 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.933 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:21.933 DEBUG:boto:Port: 8001 2024-04-19T23:39:21.933 DEBUG:boto:Params: {} 2024-04-19T23:39:21.933 DEBUG:boto:Token: None 2024-04-19T23:39:21.933 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.933 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0UwUsV9aOUZ0GXxYttmERFWZQFQ= 2024-04-19T23:39:21.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0UwUsV9aOUZ0GXxYttmERFWZQFQ=', 'Content-Length': '0'} 2024-04-19T23:39:21.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee843c2c1d07af82-00662300a9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.936 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:21.936 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:21.937 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.937 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.937 DEBUG:boto:Method: GET 2024-04-19T23:39:21.937 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.937 DEBUG:boto:Data: 2024-04-19T23:39:21.938 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.938 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.938 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.938 DEBUG:boto:Params: {} 2024-04-19T23:39:21.938 DEBUG:boto:Token: None 2024-04-19T23:39:21.939 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:21.939 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0UwUsV9aOUZ0GXxYttmERFWZQFQ= 2024-04-19T23:39:21.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0UwUsV9aOUZ0GXxYttmERFWZQFQ=', 'Content-Length': '0'} 2024-04-19T23:39:21.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e950d5becec20605-00662300a9-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.942 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.942 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.942 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:21.942 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:21.942 DEBUG:boto:Method: GET 2024-04-19T23:39:21.943 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:39:21.943 DEBUG:boto:Data: 2024-04-19T23:39:21.943 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.943 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:21.943 DEBUG:boto:Port: 8001 2024-04-19T23:39:21.943 DEBUG:boto:Params: {} 2024-04-19T23:39:21.944 DEBUG:boto:Token: None 2024-04-19T23:39:21.944 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083d11e6d9d0726e7-00662300a9-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.947 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.948 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:21.948 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:21.948 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:21.949 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:21.949 DEBUG:boto:Method: GET 2024-04-19T23:39:21.949 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:39:21.949 DEBUG:boto:Data: 2024-04-19T23:39:21.949 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.949 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:21.949 DEBUG:boto:Port: 8000 2024-04-19T23:39:21.949 DEBUG:boto:Params: {} 2024-04-19T23:39:21.949 DEBUG:boto:Token: None 2024-04-19T23:39:21.949 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/ 2024-04-19T23:39:21.950 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM= 2024-04-19T23:39:21.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UimffNt8Nwv2nEH92l7VufMDWgM=', 'Content-Length': '0'} 2024-04-19T23:39:21.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca945e1d7b0fad4f-00662300a9-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.952 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:21.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:21.953 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.953 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.953 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.953 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.953 DEBUG:boto:Method: GET 2024-04-19T23:39:21.953 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.953 DEBUG:boto:Data: 2024-04-19T23:39:21.954 DEBUG:boto:Headers: {} 2024-04-19T23:39:21.954 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:21.954 DEBUG:boto:Port: 8001 2024-04-19T23:39:21.954 DEBUG:boto:Params: {} 2024-04-19T23:39:21.954 DEBUG:boto:Token: None 2024-04-19T23:39:21.954 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:21 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.954 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU= 2024-04-19T23:39:21.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BFSiV57wczvJ7fJ8HoifhSDdGEU=', 'Content-Length': '0'} 2024-04-19T23:39:21.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092d1ddcfd52df044-00662300a9-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:21.998 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:21.999 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:21.999 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.999 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:21.999 DEBUG:boto:Method: GET 2024-04-19T23:39:22.000 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.000 DEBUG:boto:Data: 2024-04-19T23:39:22.000 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.000 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.000 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.000 DEBUG:boto:Params: {} 2024-04-19T23:39:22.001 DEBUG:boto:Token: None 2024-04-19T23:39:22.001 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.001 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E= 2024-04-19T23:39:22.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E=', 'Content-Length': '0'} 2024-04-19T23:39:22.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b581dd3f6bc43a39-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.004 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.004 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.005 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.005 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.005 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.005 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.005 DEBUG:boto:Data: 2024-04-19T23:39:22.005 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.006 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.006 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.006 DEBUG:boto:Params: {} 2024-04-19T23:39:22.006 DEBUG:boto:Token: None 2024-04-19T23:39:22.006 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.006 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2GpcGlN1VwLdo4Peml87+gpP49M= 2024-04-19T23:39:22.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2GpcGlN1VwLdo4Peml87+gpP49M=', 'Content-Length': '0'} 2024-04-19T23:39:22.009 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd356a0c86b8e2f9-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.009 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.009 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.010 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.010 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.010 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.010 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.011 DEBUG:boto:Data: 2024-04-19T23:39:22.011 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.011 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.011 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.011 DEBUG:boto:Params: {} 2024-04-19T23:39:22.012 DEBUG:boto:Token: None 2024-04-19T23:39:22.012 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.012 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2GpcGlN1VwLdo4Peml87+gpP49M= 2024-04-19T23:39:22.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2GpcGlN1VwLdo4Peml87+gpP49M=', 'Content-Length': '0'} 2024-04-19T23:39:22.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eab886fec66f03f6-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.015 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.015 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.015 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.015 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.015 DEBUG:boto:Method: GET 2024-04-19T23:39:22.015 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.016 DEBUG:boto:Data: 2024-04-19T23:39:22.016 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.016 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.016 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.016 DEBUG:boto:Params: {} 2024-04-19T23:39:22.016 DEBUG:boto:Token: None 2024-04-19T23:39:22.016 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.016 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E= 2024-04-19T23:39:22.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E=', 'Content-Length': '0'} 2024-04-19T23:39:22.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032e767c420fe022a-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.019 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.019 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.019 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.019 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.019 DEBUG:boto:Method: GET 2024-04-19T23:39:22.019 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.019 DEBUG:boto:Data: 2024-04-19T23:39:22.019 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.019 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.019 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.019 DEBUG:boto:Params: {} 2024-04-19T23:39:22.020 DEBUG:boto:Token: None 2024-04-19T23:39:22.020 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.020 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E= 2024-04-19T23:39:22.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E=', 'Content-Length': '0'} 2024-04-19T23:39:22.022 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c259bd9f19fc84cd-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.062 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.063 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.063 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.063 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.063 DEBUG:boto:Method: GET 2024-04-19T23:39:22.063 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.063 DEBUG:boto:Data: 2024-04-19T23:39:22.063 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.063 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.063 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.063 DEBUG:boto:Params: {} 2024-04-19T23:39:22.064 DEBUG:boto:Token: None 2024-04-19T23:39:22.064 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.064 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cfbbe8d53482ed7-00662300aa-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.067 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.067 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.067 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.067 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.067 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.068 DEBUG:boto:Method: GET 2024-04-19T23:39:22.068 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.068 DEBUG:boto:Data: 2024-04-19T23:39:22.068 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.068 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.068 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.068 DEBUG:boto:Params: {} 2024-04-19T23:39:22.068 DEBUG:boto:Token: None 2024-04-19T23:39:22.068 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.068 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb6fdb866baf1c6c-00662300aa-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.072 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.072 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.072 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.072 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.072 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:22.072 DEBUG:boto:Data: 2024-04-19T23:39:22.073 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.073 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.073 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.073 DEBUG:boto:Params: {} 2024-04-19T23:39:22.073 DEBUG:boto:Token: None 2024-04-19T23:39:22.073 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj 2024-04-19T23:39:22.073 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw= 2024-04-19T23:39:22.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw=', 'Content-Length': '0'} 2024-04-19T23:39:22.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030b2c0f68c6aa000-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.077 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.077 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.077 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.077 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:22.077 DEBUG:boto:Data: 2024-04-19T23:39:22.077 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.077 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.077 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.078 DEBUG:boto:Params: {} 2024-04-19T23:39:22.078 DEBUG:boto:Token: None 2024-04-19T23:39:22.078 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj 2024-04-19T23:39:22.078 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw= 2024-04-19T23:39:22.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw=', 'Content-Length': '0'} 2024-04-19T23:39:22.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c596a02f3199330-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.082 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.082 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.082 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.082 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.082 DEBUG:boto:Method: GET 2024-04-19T23:39:22.082 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.082 DEBUG:boto:Data: 2024-04-19T23:39:22.082 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.082 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.083 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.083 DEBUG:boto:Params: {} 2024-04-19T23:39:22.083 DEBUG:boto:Token: None 2024-04-19T23:39:22.083 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.083 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E= 2024-04-19T23:39:22.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E=', 'Content-Length': '0'} 2024-04-19T23:39:22.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086f4250271855031-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.126 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.127 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.127 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.127 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.127 DEBUG:boto:Method: GET 2024-04-19T23:39:22.127 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.127 DEBUG:boto:Data: 2024-04-19T23:39:22.127 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.127 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.127 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.127 DEBUG:boto:Params: {} 2024-04-19T23:39:22.128 DEBUG:boto:Token: None 2024-04-19T23:39:22.128 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.128 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E= 2024-04-19T23:39:22.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E=', 'Content-Length': '0'} 2024-04-19T23:39:22.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d24682135cf4fc50-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.170 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.171 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.171 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.171 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.171 DEBUG:boto:Method: GET 2024-04-19T23:39:22.171 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.171 DEBUG:boto:Data: 2024-04-19T23:39:22.171 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.171 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.171 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.171 DEBUG:boto:Params: {} 2024-04-19T23:39:22.172 DEBUG:boto:Token: None 2024-04-19T23:39:22.172 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.172 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000711989afc2d89921-00662300aa-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.175 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.176 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.176 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.176 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.176 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.176 DEBUG:boto:Method: GET 2024-04-19T23:39:22.176 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.176 DEBUG:boto:Data: 2024-04-19T23:39:22.176 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.176 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.176 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.177 DEBUG:boto:Params: {} 2024-04-19T23:39:22.177 DEBUG:boto:Token: None 2024-04-19T23:39:22.177 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.177 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042f06da7619d7287-00662300aa-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.222 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.223 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.223 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.223 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.223 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.223 DEBUG:boto:Method: GET 2024-04-19T23:39:22.224 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.224 DEBUG:boto:Data: 2024-04-19T23:39:22.224 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.224 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.224 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.224 DEBUG:boto:Params: {} 2024-04-19T23:39:22.224 DEBUG:boto:Token: None 2024-04-19T23:39:22.224 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.224 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gx+1l8s8bl5VU13Fat/MlkJFvtg= 2024-04-19T23:39:22.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gx+1l8s8bl5VU13Fat/MlkJFvtg=', 'Content-Length': '0'} 2024-04-19T23:39:22.227 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000418dc731ffeb744a-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.227 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.227 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.228 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.228 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.228 DEBUG:boto:Method: GET 2024-04-19T23:39:22.228 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.229 DEBUG:boto:Data: 2024-04-19T23:39:22.229 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.229 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.229 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.229 DEBUG:boto:Params: {} 2024-04-19T23:39:22.230 DEBUG:boto:Token: None 2024-04-19T23:39:22.230 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.230 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gx+1l8s8bl5VU13Fat/MlkJFvtg= 2024-04-19T23:39:22.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gx+1l8s8bl5VU13Fat/MlkJFvtg=', 'Content-Length': '0'} 2024-04-19T23:39:22.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080dee1124d9b94ee-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.233 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.233 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.233 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.234 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.234 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.234 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.234 DEBUG:boto:Data: 2024-04-19T23:39:22.234 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.235 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.235 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.235 DEBUG:boto:Params: {} 2024-04-19T23:39:22.235 DEBUG:boto:Token: None 2024-04-19T23:39:22.235 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.236 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lzD8JHN/1NcexxuUwOvzDkSzm80= 2024-04-19T23:39:22.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lzD8JHN/1NcexxuUwOvzDkSzm80=', 'Content-Length': '0'} 2024-04-19T23:39:22.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007bd30a08c92797b-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.238 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.238 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.238 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.238 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.238 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.238 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.239 DEBUG:boto:Data: 2024-04-19T23:39:22.239 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.239 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.239 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.239 DEBUG:boto:Params: {} 2024-04-19T23:39:22.239 DEBUG:boto:Token: None 2024-04-19T23:39:22.239 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.239 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lzD8JHN/1NcexxuUwOvzDkSzm80= 2024-04-19T23:39:22.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lzD8JHN/1NcexxuUwOvzDkSzm80=', 'Content-Length': '0'} 2024-04-19T23:39:22.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf3949e2976155f4-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.242 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.242 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.242 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.242 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.242 DEBUG:boto:Method: GET 2024-04-19T23:39:22.243 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.243 DEBUG:boto:Data: 2024-04-19T23:39:22.243 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.243 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.243 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.243 DEBUG:boto:Params: {} 2024-04-19T23:39:22.243 DEBUG:boto:Token: None 2024-04-19T23:39:22.243 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.243 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gx+1l8s8bl5VU13Fat/MlkJFvtg= 2024-04-19T23:39:22.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gx+1l8s8bl5VU13Fat/MlkJFvtg=', 'Content-Length': '0'} 2024-04-19T23:39:22.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1fd52265446da83-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.246 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.246 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.246 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.246 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.246 DEBUG:boto:Method: GET 2024-04-19T23:39:22.246 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.247 DEBUG:boto:Data: 2024-04-19T23:39:22.247 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.247 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.247 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.247 DEBUG:boto:Params: {} 2024-04-19T23:39:22.247 DEBUG:boto:Token: None 2024-04-19T23:39:22.247 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:22.247 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gx+1l8s8bl5VU13Fat/MlkJFvtg= 2024-04-19T23:39:22.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gx+1l8s8bl5VU13Fat/MlkJFvtg=', 'Content-Length': '0'} 2024-04-19T23:39:22.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060d84cf62de9ccb0-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.249 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.250 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.250 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.250 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.250 DEBUG:boto:Method: GET 2024-04-19T23:39:22.250 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.250 DEBUG:boto:Data: 2024-04-19T23:39:22.250 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.250 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.250 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.250 DEBUG:boto:Params: {} 2024-04-19T23:39:22.250 DEBUG:boto:Token: None 2024-04-19T23:39:22.251 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.251 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ca5ba331632999c-00662300aa-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.254 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.254 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.254 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.254 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.254 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.255 DEBUG:boto:Method: GET 2024-04-19T23:39:22.255 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.255 DEBUG:boto:Data: 2024-04-19T23:39:22.255 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.255 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.255 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.255 DEBUG:boto:Params: {} 2024-04-19T23:39:22.255 DEBUG:boto:Token: None 2024-04-19T23:39:22.255 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.256 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1ed053738316a97-00662300aa-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.259 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.259 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.260 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.260 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.260 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:22.260 DEBUG:boto:Data: 2024-04-19T23:39:22.260 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.260 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.260 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.260 DEBUG:boto:Params: {} 2024-04-19T23:39:22.260 DEBUG:boto:Token: None 2024-04-19T23:39:22.260 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj 2024-04-19T23:39:22.261 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw= 2024-04-19T23:39:22.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw=', 'Content-Length': '0'} 2024-04-19T23:39:22.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022ccc317d7061248-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.264 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.264 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.264 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.264 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:22.265 DEBUG:boto:Data: 2024-04-19T23:39:22.265 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.265 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.265 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.265 DEBUG:boto:Params: {} 2024-04-19T23:39:22.265 DEBUG:boto:Token: None 2024-04-19T23:39:22.265 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj 2024-04-19T23:39:22.265 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw= 2024-04-19T23:39:22.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw=', 'Content-Length': '0'} 2024-04-19T23:39:22.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ad8ff1f0d486236-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.269 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.269 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.269 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.269 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.269 DEBUG:boto:Method: GET 2024-04-19T23:39:22.269 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.269 DEBUG:boto:Data: 2024-04-19T23:39:22.270 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.270 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.270 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.270 DEBUG:boto:Params: {} 2024-04-19T23:39:22.270 DEBUG:boto:Token: None 2024-04-19T23:39:22.270 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.270 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E= 2024-04-19T23:39:22.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E=', 'Content-Length': '0'} 2024-04-19T23:39:22.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cada9bda810b095-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.273 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.273 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.273 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.273 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.273 DEBUG:boto:Method: GET 2024-04-19T23:39:22.273 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.273 DEBUG:boto:Data: 2024-04-19T23:39:22.273 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.273 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.273 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.273 DEBUG:boto:Params: {} 2024-04-19T23:39:22.274 DEBUG:boto:Token: None 2024-04-19T23:39:22.274 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.274 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E= 2024-04-19T23:39:22.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E=', 'Content-Length': '0'} 2024-04-19T23:39:22.276 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000855c024521c44b47-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.318 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.319 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.319 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.319 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.319 DEBUG:boto:Method: GET 2024-04-19T23:39:22.319 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.319 DEBUG:boto:Data: 2024-04-19T23:39:22.319 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.319 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.319 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.319 DEBUG:boto:Params: {} 2024-04-19T23:39:22.320 DEBUG:boto:Token: None 2024-04-19T23:39:22.320 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.320 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f736b259f3176bba-00662300aa-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.324 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.324 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.324 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.324 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.324 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.325 DEBUG:boto:Method: GET 2024-04-19T23:39:22.325 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.325 DEBUG:boto:Data: 2024-04-19T23:39:22.325 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.325 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.325 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.325 DEBUG:boto:Params: {} 2024-04-19T23:39:22.325 DEBUG:boto:Token: None 2024-04-19T23:39:22.325 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.326 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a4e708abe67aaf-00662300aa-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.329 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.329 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.329 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.329 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.329 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.330 DEBUG:boto:Method: GET 2024-04-19T23:39:22.330 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.330 DEBUG:boto:Data: 2024-04-19T23:39:22.330 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.330 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.330 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.330 DEBUG:boto:Params: {} 2024-04-19T23:39:22.330 DEBUG:boto:Token: None 2024-04-19T23:39:22.330 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.330 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+D+uZevvR8+eKp7q0vYst35wNyk= 2024-04-19T23:39:22.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+D+uZevvR8+eKp7q0vYst35wNyk=', 'Content-Length': '0'} 2024-04-19T23:39:22.333 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f934c1dc8877b0fe-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.374 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.374 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.375 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.375 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.375 DEBUG:boto:Method: GET 2024-04-19T23:39:22.375 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.375 DEBUG:boto:Data: 2024-04-19T23:39:22.375 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.375 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.375 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.375 DEBUG:boto:Params: {} 2024-04-19T23:39:22.375 DEBUG:boto:Token: None 2024-04-19T23:39:22.376 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.376 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+D+uZevvR8+eKp7q0vYst35wNyk= 2024-04-19T23:39:22.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+D+uZevvR8+eKp7q0vYst35wNyk=', 'Content-Length': '0'} 2024-04-19T23:39:22.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d5588141f5dee4b-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.418 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.419 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.419 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.419 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.419 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.419 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.419 DEBUG:boto:Data: 2024-04-19T23:39:22.419 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.419 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.419 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.419 DEBUG:boto:Params: {} 2024-04-19T23:39:22.420 DEBUG:boto:Token: None 2024-04-19T23:39:22.420 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.420 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WT12wvAJHqZ1h12BGJOm+X9hVcs= 2024-04-19T23:39:22.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WT12wvAJHqZ1h12BGJOm+X9hVcs=', 'Content-Length': '0'} 2024-04-19T23:39:22.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041376251315327e7-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.423 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.423 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.423 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.423 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.423 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.423 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.423 DEBUG:boto:Data: 2024-04-19T23:39:22.423 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.423 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.423 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.423 DEBUG:boto:Params: {} 2024-04-19T23:39:22.424 DEBUG:boto:Token: None 2024-04-19T23:39:22.424 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.424 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WT12wvAJHqZ1h12BGJOm+X9hVcs= 2024-04-19T23:39:22.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WT12wvAJHqZ1h12BGJOm+X9hVcs=', 'Content-Length': '0'} 2024-04-19T23:39:22.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b2e2d047c62ae54-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.426 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.426 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.427 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.427 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.427 DEBUG:boto:Method: GET 2024-04-19T23:39:22.427 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.427 DEBUG:boto:Data: 2024-04-19T23:39:22.427 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.427 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.427 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.427 DEBUG:boto:Params: {} 2024-04-19T23:39:22.427 DEBUG:boto:Token: None 2024-04-19T23:39:22.427 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.428 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+D+uZevvR8+eKp7q0vYst35wNyk= 2024-04-19T23:39:22.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+D+uZevvR8+eKp7q0vYst35wNyk=', 'Content-Length': '0'} 2024-04-19T23:39:22.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b210e6561edcff8-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.430 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.430 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.430 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.430 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.431 DEBUG:boto:Method: GET 2024-04-19T23:39:22.431 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.431 DEBUG:boto:Data: 2024-04-19T23:39:22.431 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.431 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.431 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.431 DEBUG:boto:Params: {} 2024-04-19T23:39:22.431 DEBUG:boto:Token: None 2024-04-19T23:39:22.431 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:22.431 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+D+uZevvR8+eKp7q0vYst35wNyk= 2024-04-19T23:39:22.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+D+uZevvR8+eKp7q0vYst35wNyk=', 'Content-Length': '0'} 2024-04-19T23:39:22.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082f56e68f273b593-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.478 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.479 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.479 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.479 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.479 DEBUG:boto:Method: GET 2024-04-19T23:39:22.479 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.479 DEBUG:boto:Data: 2024-04-19T23:39:22.479 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.479 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.479 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.479 DEBUG:boto:Params: {} 2024-04-19T23:39:22.480 DEBUG:boto:Token: None 2024-04-19T23:39:22.480 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.480 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa1773774ae159b5-00662300aa-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.483 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.483 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.484 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.484 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.484 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.484 DEBUG:boto:Method: GET 2024-04-19T23:39:22.484 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.484 DEBUG:boto:Data: 2024-04-19T23:39:22.484 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.484 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.484 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.484 DEBUG:boto:Params: {} 2024-04-19T23:39:22.484 DEBUG:boto:Token: None 2024-04-19T23:39:22.485 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.485 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000332d999e20caf976-00662300aa-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.530 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.531 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.531 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.531 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.531 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:22.531 DEBUG:boto:Data: 2024-04-19T23:39:22.531 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.531 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.532 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.532 DEBUG:boto:Params: {} 2024-04-19T23:39:22.532 DEBUG:boto:Token: None 2024-04-19T23:39:22.532 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj 2024-04-19T23:39:22.532 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw= 2024-04-19T23:39:22.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw=', 'Content-Length': '0'} 2024-04-19T23:39:22.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f737c69e80c65508-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.536 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.536 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.536 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.536 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:22.536 DEBUG:boto:Data: 2024-04-19T23:39:22.536 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.536 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.537 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.537 DEBUG:boto:Params: {} 2024-04-19T23:39:22.537 DEBUG:boto:Token: None 2024-04-19T23:39:22.537 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj 2024-04-19T23:39:22.537 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw= 2024-04-19T23:39:22.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw=', 'Content-Length': '0'} 2024-04-19T23:39:22.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c70c3d884d09bdf8-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.541 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.541 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.541 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.542 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.542 DEBUG:boto:Method: GET 2024-04-19T23:39:22.542 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.542 DEBUG:boto:Data: 2024-04-19T23:39:22.542 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.542 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.542 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.542 DEBUG:boto:Params: {} 2024-04-19T23:39:22.542 DEBUG:boto:Token: None 2024-04-19T23:39:22.543 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.543 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E= 2024-04-19T23:39:22.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E=', 'Content-Length': '0'} 2024-04-19T23:39:22.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbcb4b32d042aa27-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.545 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.545 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.545 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.545 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.545 DEBUG:boto:Method: GET 2024-04-19T23:39:22.545 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.545 DEBUG:boto:Data: 2024-04-19T23:39:22.545 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.545 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.546 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.546 DEBUG:boto:Params: {} 2024-04-19T23:39:22.546 DEBUG:boto:Token: None 2024-04-19T23:39:22.546 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.546 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E= 2024-04-19T23:39:22.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E=', 'Content-Length': '0'} 2024-04-19T23:39:22.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d13dedaf2cad88e-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.595 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.595 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.595 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.595 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.595 DEBUG:boto:Method: GET 2024-04-19T23:39:22.596 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.596 DEBUG:boto:Data: 2024-04-19T23:39:22.596 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.596 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.596 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.596 DEBUG:boto:Params: {} 2024-04-19T23:39:22.596 DEBUG:boto:Token: None 2024-04-19T23:39:22.596 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.596 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000741ee815e1b417b6-00662300aa-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.599 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.600 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.600 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.600 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.600 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.600 DEBUG:boto:Method: GET 2024-04-19T23:39:22.600 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.600 DEBUG:boto:Data: 2024-04-19T23:39:22.600 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.600 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.601 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.601 DEBUG:boto:Params: {} 2024-04-19T23:39:22.601 DEBUG:boto:Token: None 2024-04-19T23:39:22.601 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.601 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a8874057ee7480-00662300aa-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.650 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.651 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.651 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.651 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.651 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.651 DEBUG:boto:Method: GET 2024-04-19T23:39:22.651 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.652 DEBUG:boto:Data: 2024-04-19T23:39:22.652 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.652 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.652 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.652 DEBUG:boto:Params: {} 2024-04-19T23:39:22.654 DEBUG:boto:Token: None 2024-04-19T23:39:22.654 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.654 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jmU3gkbfF6pI/RwJDjGoKreaWqo= 2024-04-19T23:39:22.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jmU3gkbfF6pI/RwJDjGoKreaWqo=', 'Content-Length': '0'} 2024-04-19T23:39:22.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e495d0dabd31eb43-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.657 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.657 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.657 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.657 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.657 DEBUG:boto:Method: GET 2024-04-19T23:39:22.658 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.658 DEBUG:boto:Data: 2024-04-19T23:39:22.658 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.658 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.658 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.658 DEBUG:boto:Params: {} 2024-04-19T23:39:22.658 DEBUG:boto:Token: None 2024-04-19T23:39:22.658 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.658 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jmU3gkbfF6pI/RwJDjGoKreaWqo= 2024-04-19T23:39:22.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jmU3gkbfF6pI/RwJDjGoKreaWqo=', 'Content-Length': '0'} 2024-04-19T23:39:22.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a535025a618972a9-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.702 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.703 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.703 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.703 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.703 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.703 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.703 DEBUG:boto:Data: 2024-04-19T23:39:22.703 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.703 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.703 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.703 DEBUG:boto:Params: {} 2024-04-19T23:39:22.704 DEBUG:boto:Token: None 2024-04-19T23:39:22.704 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NsK9QVZ5l1oO9aaZvFaRVs7AXd8= 2024-04-19T23:39:22.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NsK9QVZ5l1oO9aaZvFaRVs7AXd8=', 'Content-Length': '0'} 2024-04-19T23:39:22.706 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfb021af8a299eb0-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.706 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.706 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.706 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.706 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.706 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.707 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.707 DEBUG:boto:Data: 2024-04-19T23:39:22.707 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.707 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.707 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.707 DEBUG:boto:Params: {} 2024-04-19T23:39:22.707 DEBUG:boto:Token: None 2024-04-19T23:39:22.707 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.708 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NsK9QVZ5l1oO9aaZvFaRVs7AXd8= 2024-04-19T23:39:22.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NsK9QVZ5l1oO9aaZvFaRVs7AXd8=', 'Content-Length': '0'} 2024-04-19T23:39:22.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aab8c738675e3c23-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.710 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.710 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.710 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.710 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.710 DEBUG:boto:Method: GET 2024-04-19T23:39:22.710 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.710 DEBUG:boto:Data: 2024-04-19T23:39:22.710 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.710 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.710 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.711 DEBUG:boto:Params: {} 2024-04-19T23:39:22.711 DEBUG:boto:Token: None 2024-04-19T23:39:22.711 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.711 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jmU3gkbfF6pI/RwJDjGoKreaWqo= 2024-04-19T23:39:22.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jmU3gkbfF6pI/RwJDjGoKreaWqo=', 'Content-Length': '0'} 2024-04-19T23:39:22.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000935877365c95bbef-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.713 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.713 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.714 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.714 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.714 DEBUG:boto:Method: GET 2024-04-19T23:39:22.714 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.714 DEBUG:boto:Data: 2024-04-19T23:39:22.714 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.714 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.714 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.714 DEBUG:boto:Params: {} 2024-04-19T23:39:22.714 DEBUG:boto:Token: None 2024-04-19T23:39:22.715 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:22.715 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jmU3gkbfF6pI/RwJDjGoKreaWqo= 2024-04-19T23:39:22.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jmU3gkbfF6pI/RwJDjGoKreaWqo=', 'Content-Length': '0'} 2024-04-19T23:39:22.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061207665aff9d5b5-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.758 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.759 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.759 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.759 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.759 DEBUG:boto:Method: GET 2024-04-19T23:39:22.759 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.759 DEBUG:boto:Data: 2024-04-19T23:39:22.759 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.759 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.759 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.759 DEBUG:boto:Params: {} 2024-04-19T23:39:22.760 DEBUG:boto:Token: None 2024-04-19T23:39:22.760 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.760 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bedb07e9405f96f4-00662300aa-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.763 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.763 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.763 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.763 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.764 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.764 DEBUG:boto:Method: GET 2024-04-19T23:39:22.764 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.764 DEBUG:boto:Data: 2024-04-19T23:39:22.764 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.764 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.764 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.764 DEBUG:boto:Params: {} 2024-04-19T23:39:22.765 DEBUG:boto:Token: None 2024-04-19T23:39:22.765 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.765 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b281f73c1b17efa-00662300aa-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.810 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.811 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.811 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.811 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.811 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:22.811 DEBUG:boto:Data: 2024-04-19T23:39:22.811 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.811 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.812 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.812 DEBUG:boto:Params: {} 2024-04-19T23:39:22.812 DEBUG:boto:Token: None 2024-04-19T23:39:22.812 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj 2024-04-19T23:39:22.812 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw= 2024-04-19T23:39:22.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw=', 'Content-Length': '0'} 2024-04-19T23:39:22.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047434fae8b3ca3a2-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.816 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.816 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.816 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.816 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:22.816 DEBUG:boto:Data: 2024-04-19T23:39:22.816 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.816 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.816 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.816 DEBUG:boto:Params: {} 2024-04-19T23:39:22.816 DEBUG:boto:Token: None 2024-04-19T23:39:22.817 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj 2024-04-19T23:39:22.817 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw= 2024-04-19T23:39:22.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AwS7oT+ilhwb6Ev6688fsy211aw=', 'Content-Length': '0'} 2024-04-19T23:39:22.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000acfb24a7845d0d8a-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.821 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.821 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.821 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.821 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.821 DEBUG:boto:Method: GET 2024-04-19T23:39:22.821 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.821 DEBUG:boto:Data: 2024-04-19T23:39:22.821 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.821 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.821 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.822 DEBUG:boto:Params: {} 2024-04-19T23:39:22.822 DEBUG:boto:Token: None 2024-04-19T23:39:22.822 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.822 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E= 2024-04-19T23:39:22.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E=', 'Content-Length': '0'} 2024-04-19T23:39:22.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2c8c0f748c8e08f-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.824 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.824 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.824 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.824 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.825 DEBUG:boto:Method: GET 2024-04-19T23:39:22.825 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.825 DEBUG:boto:Data: 2024-04-19T23:39:22.825 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.825 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.825 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.825 DEBUG:boto:Params: {} 2024-04-19T23:39:22.825 DEBUG:boto:Token: None 2024-04-19T23:39:22.825 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:22.825 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E= 2024-04-19T23:39:22.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8myNI5W9rUxqPYy8biy0uRrgI4E=', 'Content-Length': '0'} 2024-04-19T23:39:22.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047f62e5f46e554de-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.874 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.875 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.875 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.875 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.875 DEBUG:boto:Method: GET 2024-04-19T23:39:22.875 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.875 DEBUG:boto:Data: 2024-04-19T23:39:22.875 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.875 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.875 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.875 DEBUG:boto:Params: {} 2024-04-19T23:39:22.876 DEBUG:boto:Token: None 2024-04-19T23:39:22.876 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.876 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068d8ee33d09ae92c-00662300aa-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.879 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.879 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.879 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.879 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.879 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.880 DEBUG:boto:Method: GET 2024-04-19T23:39:22.880 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.880 DEBUG:boto:Data: 2024-04-19T23:39:22.880 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.880 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.880 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.880 DEBUG:boto:Params: {} 2024-04-19T23:39:22.880 DEBUG:boto:Token: None 2024-04-19T23:39:22.880 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e43b006e2a4d7b7-00662300aa-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.926 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:22.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.928 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.928 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.928 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.928 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.928 DEBUG:boto:Method: GET 2024-04-19T23:39:22.928 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.928 DEBUG:boto:Data: 2024-04-19T23:39:22.928 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.929 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.929 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.929 DEBUG:boto:Params: {} 2024-04-19T23:39:22.930 DEBUG:boto:Token: None 2024-04-19T23:39:22.930 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.930 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eJjTybQ3DKzdsE0RLtnyxR5BLkM= 2024-04-19T23:39:22.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eJjTybQ3DKzdsE0RLtnyxR5BLkM=', 'Content-Length': '0'} 2024-04-19T23:39:22.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000701595aa71d53dff-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.933 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.933 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.933 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.933 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.933 DEBUG:boto:Method: GET 2024-04-19T23:39:22.933 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.933 DEBUG:boto:Data: 2024-04-19T23:39:22.933 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.933 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.933 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.933 DEBUG:boto:Params: {} 2024-04-19T23:39:22.934 DEBUG:boto:Token: None 2024-04-19T23:39:22.934 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.934 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eJjTybQ3DKzdsE0RLtnyxR5BLkM= 2024-04-19T23:39:22.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eJjTybQ3DKzdsE0RLtnyxR5BLkM=', 'Content-Length': '0'} 2024-04-19T23:39:22.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9d4b7daf194f1b2-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.937 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.937 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.937 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.937 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.937 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.937 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.937 DEBUG:boto:Data: 2024-04-19T23:39:22.938 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.938 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.938 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.938 DEBUG:boto:Params: {} 2024-04-19T23:39:22.938 DEBUG:boto:Token: None 2024-04-19T23:39:22.938 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.938 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q4yOILkp7S+hRPaqO7XJXJhZGyA= 2024-04-19T23:39:22.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q4yOILkp7S+hRPaqO7XJXJhZGyA=', 'Content-Length': '0'} 2024-04-19T23:39:22.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d9f25de80622dd9-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.941 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.941 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.942 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.942 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.942 DEBUG:boto:Method: HEAD 2024-04-19T23:39:22.942 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.942 DEBUG:boto:Data: 2024-04-19T23:39:22.942 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.942 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.942 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.942 DEBUG:boto:Params: {} 2024-04-19T23:39:22.942 DEBUG:boto:Token: None 2024-04-19T23:39:22.943 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.943 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q4yOILkp7S+hRPaqO7XJXJhZGyA= 2024-04-19T23:39:22.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q4yOILkp7S+hRPaqO7XJXJhZGyA=', 'Content-Length': '0'} 2024-04-19T23:39:22.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8fd319f6998b7ba-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:22.948 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.948 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.948 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.949 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.949 DEBUG:boto:Method: GET 2024-04-19T23:39:22.949 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.949 DEBUG:boto:Data: 2024-04-19T23:39:22.949 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.949 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.949 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.949 DEBUG:boto:Params: {} 2024-04-19T23:39:22.949 DEBUG:boto:Token: None 2024-04-19T23:39:22.949 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.950 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eJjTybQ3DKzdsE0RLtnyxR5BLkM= 2024-04-19T23:39:22.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eJjTybQ3DKzdsE0RLtnyxR5BLkM=', 'Content-Length': '0'} 2024-04-19T23:39:22.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008699fd6c3533dad3-00662300aa-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.952 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:22.952 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:22.952 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.952 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.952 DEBUG:boto:Method: GET 2024-04-19T23:39:22.952 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.953 DEBUG:boto:Data: 2024-04-19T23:39:22.953 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.953 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:22.953 DEBUG:boto:Port: 8000 2024-04-19T23:39:22.953 DEBUG:boto:Params: {} 2024-04-19T23:39:22.953 DEBUG:boto:Token: None 2024-04-19T23:39:22.953 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:22.953 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eJjTybQ3DKzdsE0RLtnyxR5BLkM= 2024-04-19T23:39:22.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eJjTybQ3DKzdsE0RLtnyxR5BLkM=', 'Content-Length': '0'} 2024-04-19T23:39:22.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013f6f2a3a38c23e3-00662300aa-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:22.957 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:22.957 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:22.957 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.957 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:22.957 DEBUG:boto:Method: GET 2024-04-19T23:39:22.957 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:22.957 DEBUG:boto:Data: 2024-04-19T23:39:22.957 DEBUG:boto:Headers: {} 2024-04-19T23:39:22.957 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:22.957 DEBUG:boto:Port: 8001 2024-04-19T23:39:22.958 DEBUG:boto:Params: {} 2024-04-19T23:39:22.958 DEBUG:boto:Token: None 2024-04-19T23:39:22.958 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:22 GMT /aktlji-22/ 2024-04-19T23:39:22.958 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU= 2024-04-19T23:39:22.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wTfti5RvG/EN70VMskWMG7zs+qU=', 'Content-Length': '0'} 2024-04-19T23:39:22.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd00a7cd98120cab-00662300aa-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.002 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.003 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.003 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.003 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.003 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.003 DEBUG:boto:Method: GET 2024-04-19T23:39:23.003 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.003 DEBUG:boto:Data: 2024-04-19T23:39:23.003 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.003 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.004 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.004 DEBUG:boto:Params: {} 2024-04-19T23:39:23.004 DEBUG:boto:Token: None 2024-04-19T23:39:23.004 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.004 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000518f5495ebb88f9d-00662300ab-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.013 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.013 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.013 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.013 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.013 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:23.013 DEBUG:boto:Data: 2024-04-19T23:39:23.013 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.014 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.014 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.014 DEBUG:boto:Params: {} 2024-04-19T23:39:23.014 DEBUG:boto:Token: None 2024-04-19T23:39:23.014 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj 2024-04-19T23:39:23.014 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk= 2024-04-19T23:39:23.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk=', 'Content-Length': '0'} 2024-04-19T23:39:23.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000428d2d1f7ae77477-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.030 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.030 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.030 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.030 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:23.030 DEBUG:boto:Data: 2024-04-19T23:39:23.030 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.030 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.030 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.030 DEBUG:boto:Params: {} 2024-04-19T23:39:23.031 DEBUG:boto:Token: None 2024-04-19T23:39:23.031 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj 2024-04-19T23:39:23.031 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk= 2024-04-19T23:39:23.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk=', 'Content-Length': '0'} 2024-04-19T23:39:23.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028953f299a772b6b-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.050 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.050 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.050 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.050 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.050 DEBUG:boto:Method: GET 2024-04-19T23:39:23.051 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.051 DEBUG:boto:Data: 2024-04-19T23:39:23.051 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.051 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.051 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.051 DEBUG:boto:Params: {} 2024-04-19T23:39:23.051 DEBUG:boto:Token: None 2024-04-19T23:39:23.051 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.051 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA= 2024-04-19T23:39:23.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA=', 'Content-Length': '0'} 2024-04-19T23:39:23.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e3d236eaece19b1a-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.054 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.054 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.054 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.054 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.054 DEBUG:boto:Method: GET 2024-04-19T23:39:23.054 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.054 DEBUG:boto:Data: 2024-04-19T23:39:23.054 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.054 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.055 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.055 DEBUG:boto:Params: {} 2024-04-19T23:39:23.055 DEBUG:boto:Token: None 2024-04-19T23:39:23.055 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.055 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA= 2024-04-19T23:39:23.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA=', 'Content-Length': '0'} 2024-04-19T23:39:23.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8427466c4c924f7-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.098 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.099 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.099 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.099 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.099 DEBUG:boto:Method: GET 2024-04-19T23:39:23.099 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.099 DEBUG:boto:Data: 2024-04-19T23:39:23.099 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.099 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.099 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.099 DEBUG:boto:Params: {} 2024-04-19T23:39:23.100 DEBUG:boto:Token: None 2024-04-19T23:39:23.100 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.100 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cba0fc3dbd9fc3a2-00662300ab-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.103 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.104 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.104 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.104 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.104 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.104 DEBUG:boto:Method: GET 2024-04-19T23:39:23.104 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.104 DEBUG:boto:Data: 2024-04-19T23:39:23.104 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.105 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.105 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.105 DEBUG:boto:Params: {} 2024-04-19T23:39:23.105 DEBUG:boto:Token: None 2024-04-19T23:39:23.105 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.105 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e00e0ffcf6428c71-00662300ab-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.108 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.108 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.109 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.109 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.109 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.109 DEBUG:boto:Method: GET 2024-04-19T23:39:23.109 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.109 DEBUG:boto:Data: 2024-04-19T23:39:23.109 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.109 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.109 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.109 DEBUG:boto:Params: {} 2024-04-19T23:39:23.110 DEBUG:boto:Token: None 2024-04-19T23:39:23.110 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.110 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/SA3OlVJYA7D1DSEwMloarl2K30= 2024-04-19T23:39:23.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/SA3OlVJYA7D1DSEwMloarl2K30=', 'Content-Length': '0'} 2024-04-19T23:39:23.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c77339931dc83a90-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.154 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.155 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.155 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.155 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.155 DEBUG:boto:Method: GET 2024-04-19T23:39:23.155 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.155 DEBUG:boto:Data: 2024-04-19T23:39:23.155 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.155 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.155 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.155 DEBUG:boto:Params: {} 2024-04-19T23:39:23.156 DEBUG:boto:Token: None 2024-04-19T23:39:23.156 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.156 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/SA3OlVJYA7D1DSEwMloarl2K30= 2024-04-19T23:39:23.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/SA3OlVJYA7D1DSEwMloarl2K30=', 'Content-Length': '0'} 2024-04-19T23:39:23.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d3cd261933e5c90-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.158 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.159 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.159 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.159 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.159 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.159 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.159 DEBUG:boto:Data: 2024-04-19T23:39:23.159 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.159 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.159 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.159 DEBUG:boto:Params: {} 2024-04-19T23:39:23.160 DEBUG:boto:Token: None 2024-04-19T23:39:23.160 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0mnv5HRFuhA1C3P5pPqIS6GUYiw= 2024-04-19T23:39:23.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0mnv5HRFuhA1C3P5pPqIS6GUYiw=', 'Content-Length': '0'} 2024-04-19T23:39:23.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000db57c9f553bf451-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.162 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.162 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.163 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.163 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.163 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.163 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.163 DEBUG:boto:Data: 2024-04-19T23:39:23.163 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.163 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.163 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.163 DEBUG:boto:Params: {} 2024-04-19T23:39:23.164 DEBUG:boto:Token: None 2024-04-19T23:39:23.164 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.164 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0mnv5HRFuhA1C3P5pPqIS6GUYiw= 2024-04-19T23:39:23.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0mnv5HRFuhA1C3P5pPqIS6GUYiw=', 'Content-Length': '0'} 2024-04-19T23:39:23.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac1a140597cb57cf-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.166 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.167 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.167 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.167 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.167 DEBUG:boto:Method: GET 2024-04-19T23:39:23.167 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.167 DEBUG:boto:Data: 2024-04-19T23:39:23.167 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.167 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.167 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.167 DEBUG:boto:Params: {} 2024-04-19T23:39:23.167 DEBUG:boto:Token: None 2024-04-19T23:39:23.168 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.168 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/SA3OlVJYA7D1DSEwMloarl2K30= 2024-04-19T23:39:23.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/SA3OlVJYA7D1DSEwMloarl2K30=', 'Content-Length': '0'} 2024-04-19T23:39:23.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd7b3a09f4554826-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.170 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.170 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.170 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.170 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.171 DEBUG:boto:Method: GET 2024-04-19T23:39:23.171 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.171 DEBUG:boto:Data: 2024-04-19T23:39:23.171 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.171 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.171 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.171 DEBUG:boto:Params: {} 2024-04-19T23:39:23.171 DEBUG:boto:Token: None 2024-04-19T23:39:23.171 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:23.171 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/SA3OlVJYA7D1DSEwMloarl2K30= 2024-04-19T23:39:23.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/SA3OlVJYA7D1DSEwMloarl2K30=', 'Content-Length': '0'} 2024-04-19T23:39:23.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094e62eab546da459-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.214 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.215 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.215 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.215 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.215 DEBUG:boto:Method: GET 2024-04-19T23:39:23.215 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.215 DEBUG:boto:Data: 2024-04-19T23:39:23.215 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.215 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.215 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.215 DEBUG:boto:Params: {} 2024-04-19T23:39:23.216 DEBUG:boto:Token: None 2024-04-19T23:39:23.216 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.216 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc34cae8eb1ab097-00662300ab-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.219 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.219 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.220 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.220 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.220 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.220 DEBUG:boto:Method: GET 2024-04-19T23:39:23.220 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.220 DEBUG:boto:Data: 2024-04-19T23:39:23.220 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.220 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.220 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.220 DEBUG:boto:Params: {} 2024-04-19T23:39:23.220 DEBUG:boto:Token: None 2024-04-19T23:39:23.221 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.221 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040e20c176d888ccb-00662300ab-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.224 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.224 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.224 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.224 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.224 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:23.224 DEBUG:boto:Data: 2024-04-19T23:39:23.224 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.224 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.225 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.225 DEBUG:boto:Params: {} 2024-04-19T23:39:23.225 DEBUG:boto:Token: None 2024-04-19T23:39:23.225 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj 2024-04-19T23:39:23.225 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk= 2024-04-19T23:39:23.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk=', 'Content-Length': '0'} 2024-04-19T23:39:23.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efafcdd53f29468f-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.228 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.228 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.229 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.229 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:23.229 DEBUG:boto:Data: 2024-04-19T23:39:23.229 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.229 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.229 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.229 DEBUG:boto:Params: {} 2024-04-19T23:39:23.229 DEBUG:boto:Token: None 2024-04-19T23:39:23.229 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj 2024-04-19T23:39:23.230 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk= 2024-04-19T23:39:23.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk=', 'Content-Length': '0'} 2024-04-19T23:39:23.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065264433d1f7c161-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.233 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.234 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.234 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.234 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.234 DEBUG:boto:Method: GET 2024-04-19T23:39:23.234 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.234 DEBUG:boto:Data: 2024-04-19T23:39:23.234 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.234 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.234 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.234 DEBUG:boto:Params: {} 2024-04-19T23:39:23.235 DEBUG:boto:Token: None 2024-04-19T23:39:23.235 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.235 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA= 2024-04-19T23:39:23.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA=', 'Content-Length': '0'} 2024-04-19T23:39:23.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d96ee4fa995ee41-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.278 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.279 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.279 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.279 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.279 DEBUG:boto:Method: GET 2024-04-19T23:39:23.279 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.279 DEBUG:boto:Data: 2024-04-19T23:39:23.279 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.279 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.279 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.279 DEBUG:boto:Params: {} 2024-04-19T23:39:23.279 DEBUG:boto:Token: None 2024-04-19T23:39:23.280 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.280 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA= 2024-04-19T23:39:23.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA=', 'Content-Length': '0'} 2024-04-19T23:39:23.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001038138ee238269f-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.322 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.323 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.323 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.323 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.323 DEBUG:boto:Method: GET 2024-04-19T23:39:23.323 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.323 DEBUG:boto:Data: 2024-04-19T23:39:23.323 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.323 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.323 DEBUG:boto:Params: {} 2024-04-19T23:39:23.324 DEBUG:boto:Token: None 2024-04-19T23:39:23.324 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dc27d73a06affb5-00662300ab-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.327 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.328 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.328 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.328 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.328 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.328 DEBUG:boto:Method: GET 2024-04-19T23:39:23.328 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.328 DEBUG:boto:Data: 2024-04-19T23:39:23.328 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.328 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.328 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.328 DEBUG:boto:Params: {} 2024-04-19T23:39:23.329 DEBUG:boto:Token: None 2024-04-19T23:39:23.329 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.329 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e825bc0b6a25fe15-00662300ab-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.375 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.376 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.376 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.376 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.376 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.376 DEBUG:boto:Method: GET 2024-04-19T23:39:23.376 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.376 DEBUG:boto:Data: 2024-04-19T23:39:23.376 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.376 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.376 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.376 DEBUG:boto:Params: {} 2024-04-19T23:39:23.377 DEBUG:boto:Token: None 2024-04-19T23:39:23.377 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.377 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RbGnX9ac7L79KTF1rmEiBZ3Y1rw= 2024-04-19T23:39:23.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RbGnX9ac7L79KTF1rmEiBZ3Y1rw=', 'Content-Length': '0'} 2024-04-19T23:39:23.379 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f24a8ed830a99d19-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.379 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.379 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.380 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.380 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.380 DEBUG:boto:Method: GET 2024-04-19T23:39:23.380 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.380 DEBUG:boto:Data: 2024-04-19T23:39:23.380 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.380 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.380 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.380 DEBUG:boto:Params: {} 2024-04-19T23:39:23.380 DEBUG:boto:Token: None 2024-04-19T23:39:23.380 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.381 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RbGnX9ac7L79KTF1rmEiBZ3Y1rw= 2024-04-19T23:39:23.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RbGnX9ac7L79KTF1rmEiBZ3Y1rw=', 'Content-Length': '0'} 2024-04-19T23:39:23.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004538e5711fdc7343-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.430 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.431 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.431 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.431 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.431 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.431 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.431 DEBUG:boto:Data: 2024-04-19T23:39:23.431 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.431 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.431 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.432 DEBUG:boto:Params: {} 2024-04-19T23:39:23.432 DEBUG:boto:Token: None 2024-04-19T23:39:23.432 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.432 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0hs3BQdwdnlQFySkG4+FJf0EVg8= 2024-04-19T23:39:23.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0hs3BQdwdnlQFySkG4+FJf0EVg8=', 'Content-Length': '0'} 2024-04-19T23:39:23.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbbc6f24b15be40c-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.434 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.435 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.435 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.435 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.435 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.435 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.435 DEBUG:boto:Data: 2024-04-19T23:39:23.435 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.435 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.435 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.435 DEBUG:boto:Params: {} 2024-04-19T23:39:23.436 DEBUG:boto:Token: None 2024-04-19T23:39:23.436 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.436 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0hs3BQdwdnlQFySkG4+FJf0EVg8= 2024-04-19T23:39:23.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0hs3BQdwdnlQFySkG4+FJf0EVg8=', 'Content-Length': '0'} 2024-04-19T23:39:23.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ddc1151d67a1a6fa-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.438 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.438 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.439 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.439 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.439 DEBUG:boto:Method: GET 2024-04-19T23:39:23.439 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.439 DEBUG:boto:Data: 2024-04-19T23:39:23.439 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.439 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.439 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.439 DEBUG:boto:Params: {} 2024-04-19T23:39:23.439 DEBUG:boto:Token: None 2024-04-19T23:39:23.440 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.440 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RbGnX9ac7L79KTF1rmEiBZ3Y1rw= 2024-04-19T23:39:23.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RbGnX9ac7L79KTF1rmEiBZ3Y1rw=', 'Content-Length': '0'} 2024-04-19T23:39:23.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000516e828996bf725e-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.442 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.442 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.442 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.442 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.442 DEBUG:boto:Method: GET 2024-04-19T23:39:23.443 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.443 DEBUG:boto:Data: 2024-04-19T23:39:23.443 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.443 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.443 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.443 DEBUG:boto:Params: {} 2024-04-19T23:39:23.443 DEBUG:boto:Token: None 2024-04-19T23:39:23.443 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:23.443 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RbGnX9ac7L79KTF1rmEiBZ3Y1rw= 2024-04-19T23:39:23.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RbGnX9ac7L79KTF1rmEiBZ3Y1rw=', 'Content-Length': '0'} 2024-04-19T23:39:23.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097b5f83af8a25936-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.486 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.487 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.487 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.487 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.487 DEBUG:boto:Method: GET 2024-04-19T23:39:23.487 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.487 DEBUG:boto:Data: 2024-04-19T23:39:23.487 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.487 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.487 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.487 DEBUG:boto:Params: {} 2024-04-19T23:39:23.487 DEBUG:boto:Token: None 2024-04-19T23:39:23.488 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fed73443c9f7bf31-00662300ab-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.490 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.491 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.491 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.491 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.491 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.491 DEBUG:boto:Method: GET 2024-04-19T23:39:23.491 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.491 DEBUG:boto:Data: 2024-04-19T23:39:23.491 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.492 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.492 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.492 DEBUG:boto:Params: {} 2024-04-19T23:39:23.492 DEBUG:boto:Token: None 2024-04-19T23:39:23.492 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.492 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ba230779bb0dfb0-00662300ab-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.538 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.539 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.539 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.539 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.539 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:23.539 DEBUG:boto:Data: 2024-04-19T23:39:23.539 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.540 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.540 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.540 DEBUG:boto:Params: {} 2024-04-19T23:39:23.540 DEBUG:boto:Token: None 2024-04-19T23:39:23.540 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj 2024-04-19T23:39:23.540 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk= 2024-04-19T23:39:23.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk=', 'Content-Length': '0'} 2024-04-19T23:39:23.544 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097bc6916c63f8cd5-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.544 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.544 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.544 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.544 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:23.544 DEBUG:boto:Data: 2024-04-19T23:39:23.544 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.544 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.545 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.545 DEBUG:boto:Params: {} 2024-04-19T23:39:23.545 DEBUG:boto:Token: None 2024-04-19T23:39:23.545 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj 2024-04-19T23:39:23.545 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk= 2024-04-19T23:39:23.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk=', 'Content-Length': '0'} 2024-04-19T23:39:23.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009dc523c4fc4c9351-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.549 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.549 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.549 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.549 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.549 DEBUG:boto:Method: GET 2024-04-19T23:39:23.549 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.549 DEBUG:boto:Data: 2024-04-19T23:39:23.550 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.550 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.550 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.550 DEBUG:boto:Params: {} 2024-04-19T23:39:23.550 DEBUG:boto:Token: None 2024-04-19T23:39:23.550 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.550 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA= 2024-04-19T23:39:23.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA=', 'Content-Length': '0'} 2024-04-19T23:39:23.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bf3742fa8d58aca-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.553 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.553 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.553 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.553 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.553 DEBUG:boto:Method: GET 2024-04-19T23:39:23.553 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.553 DEBUG:boto:Data: 2024-04-19T23:39:23.553 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.554 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.554 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.554 DEBUG:boto:Params: {} 2024-04-19T23:39:23.554 DEBUG:boto:Token: None 2024-04-19T23:39:23.554 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.554 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA= 2024-04-19T23:39:23.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA=', 'Content-Length': '0'} 2024-04-19T23:39:23.556 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046c1374dd327f412-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.602 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.603 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.603 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.603 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.603 DEBUG:boto:Method: GET 2024-04-19T23:39:23.603 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.603 DEBUG:boto:Data: 2024-04-19T23:39:23.603 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.603 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.603 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.603 DEBUG:boto:Params: {} 2024-04-19T23:39:23.604 DEBUG:boto:Token: None 2024-04-19T23:39:23.604 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.604 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000054743007de2e6b5-00662300ab-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.607 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.607 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.607 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.607 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.607 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.608 DEBUG:boto:Method: GET 2024-04-19T23:39:23.608 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.608 DEBUG:boto:Data: 2024-04-19T23:39:23.608 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.608 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.608 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.608 DEBUG:boto:Params: {} 2024-04-19T23:39:23.608 DEBUG:boto:Token: None 2024-04-19T23:39:23.608 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.608 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002b301f461beb0f2-00662300ab-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.654 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.655 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.655 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.655 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.655 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.655 DEBUG:boto:Method: GET 2024-04-19T23:39:23.656 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.656 DEBUG:boto:Data: 2024-04-19T23:39:23.656 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.656 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.656 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.656 DEBUG:boto:Params: {} 2024-04-19T23:39:23.656 DEBUG:boto:Token: None 2024-04-19T23:39:23.656 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.656 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KptoJx8D0YB8TQtjOlMtbaRtvpo= 2024-04-19T23:39:23.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KptoJx8D0YB8TQtjOlMtbaRtvpo=', 'Content-Length': '0'} 2024-04-19T23:39:23.659 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5bef8a3e272e515-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.659 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.659 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.659 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.659 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.659 DEBUG:boto:Method: GET 2024-04-19T23:39:23.659 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.659 DEBUG:boto:Data: 2024-04-19T23:39:23.660 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.660 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.660 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.660 DEBUG:boto:Params: {} 2024-04-19T23:39:23.660 DEBUG:boto:Token: None 2024-04-19T23:39:23.660 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KptoJx8D0YB8TQtjOlMtbaRtvpo= 2024-04-19T23:39:23.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KptoJx8D0YB8TQtjOlMtbaRtvpo=', 'Content-Length': '0'} 2024-04-19T23:39:23.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e7c4a6e6a0576f4-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.664 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.664 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.664 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.664 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.665 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.665 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.665 DEBUG:boto:Data: 2024-04-19T23:39:23.665 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.665 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.665 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.665 DEBUG:boto:Params: {} 2024-04-19T23:39:23.666 DEBUG:boto:Token: None 2024-04-19T23:39:23.666 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.666 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oRvDzRiaxSV4oDfml6a4VOMHRmc= 2024-04-19T23:39:23.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oRvDzRiaxSV4oDfml6a4VOMHRmc=', 'Content-Length': '0'} 2024-04-19T23:39:23.668 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d654e069636e54c-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.668 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.669 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.669 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.669 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.669 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.669 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.669 DEBUG:boto:Data: 2024-04-19T23:39:23.669 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.669 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.669 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.669 DEBUG:boto:Params: {} 2024-04-19T23:39:23.670 DEBUG:boto:Token: None 2024-04-19T23:39:23.670 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.670 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oRvDzRiaxSV4oDfml6a4VOMHRmc= 2024-04-19T23:39:23.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oRvDzRiaxSV4oDfml6a4VOMHRmc=', 'Content-Length': '0'} 2024-04-19T23:39:23.672 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000759e9947a849a60e-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.673 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.673 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.673 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.673 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.673 DEBUG:boto:Method: GET 2024-04-19T23:39:23.673 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.673 DEBUG:boto:Data: 2024-04-19T23:39:23.673 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.674 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.674 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.674 DEBUG:boto:Params: {} 2024-04-19T23:39:23.674 DEBUG:boto:Token: None 2024-04-19T23:39:23.674 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.674 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KptoJx8D0YB8TQtjOlMtbaRtvpo= 2024-04-19T23:39:23.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KptoJx8D0YB8TQtjOlMtbaRtvpo=', 'Content-Length': '0'} 2024-04-19T23:39:23.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be9d58a81ddcaaeb-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.677 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.677 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.677 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.677 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.677 DEBUG:boto:Method: GET 2024-04-19T23:39:23.677 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.678 DEBUG:boto:Data: 2024-04-19T23:39:23.678 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.678 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.678 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.678 DEBUG:boto:Params: {} 2024-04-19T23:39:23.678 DEBUG:boto:Token: None 2024-04-19T23:39:23.678 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:23.678 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KptoJx8D0YB8TQtjOlMtbaRtvpo= 2024-04-19T23:39:23.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KptoJx8D0YB8TQtjOlMtbaRtvpo=', 'Content-Length': '0'} 2024-04-19T23:39:23.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000498da073bc0ccded-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.726 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.727 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.727 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.727 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.727 DEBUG:boto:Method: GET 2024-04-19T23:39:23.727 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.728 DEBUG:boto:Data: 2024-04-19T23:39:23.728 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.728 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.728 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.728 DEBUG:boto:Params: {} 2024-04-19T23:39:23.728 DEBUG:boto:Token: None 2024-04-19T23:39:23.728 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.728 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3548993b8d997fb-00662300ab-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.731 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.732 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.732 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.732 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.732 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.732 DEBUG:boto:Method: GET 2024-04-19T23:39:23.732 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.732 DEBUG:boto:Data: 2024-04-19T23:39:23.732 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.732 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.733 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.733 DEBUG:boto:Params: {} 2024-04-19T23:39:23.733 DEBUG:boto:Token: None 2024-04-19T23:39:23.733 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.733 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a12f3553c0180cc-00662300ab-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.778 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.779 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.779 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.779 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.779 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:23.779 DEBUG:boto:Data: 2024-04-19T23:39:23.779 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.779 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.779 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.780 DEBUG:boto:Params: {} 2024-04-19T23:39:23.780 DEBUG:boto:Token: None 2024-04-19T23:39:23.780 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj 2024-04-19T23:39:23.780 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk= 2024-04-19T23:39:23.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk=', 'Content-Length': '0'} 2024-04-19T23:39:23.784 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007868b8b464cb856d-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.784 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.784 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.784 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.785 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:23.785 DEBUG:boto:Data: 2024-04-19T23:39:23.785 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.785 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.785 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.785 DEBUG:boto:Params: {} 2024-04-19T23:39:23.785 DEBUG:boto:Token: None 2024-04-19T23:39:23.785 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj 2024-04-19T23:39:23.785 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk= 2024-04-19T23:39:23.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk=', 'Content-Length': '0'} 2024-04-19T23:39:23.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f170751c5687bd6-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.789 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.789 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.790 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.790 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.790 DEBUG:boto:Method: GET 2024-04-19T23:39:23.790 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.790 DEBUG:boto:Data: 2024-04-19T23:39:23.790 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.790 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.790 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.790 DEBUG:boto:Params: {} 2024-04-19T23:39:23.791 DEBUG:boto:Token: None 2024-04-19T23:39:23.791 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.791 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA= 2024-04-19T23:39:23.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA=', 'Content-Length': '0'} 2024-04-19T23:39:23.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5316e7e0bd827a6-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.834 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.835 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.835 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.835 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.835 DEBUG:boto:Method: GET 2024-04-19T23:39:23.835 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.835 DEBUG:boto:Data: 2024-04-19T23:39:23.835 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.835 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.835 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.835 DEBUG:boto:Params: {} 2024-04-19T23:39:23.836 DEBUG:boto:Token: None 2024-04-19T23:39:23.836 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.836 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA= 2024-04-19T23:39:23.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA=', 'Content-Length': '0'} 2024-04-19T23:39:23.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d355a47169eb0692-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.878 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.879 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.879 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.879 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.879 DEBUG:boto:Method: GET 2024-04-19T23:39:23.879 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.879 DEBUG:boto:Data: 2024-04-19T23:39:23.879 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.879 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.879 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.879 DEBUG:boto:Params: {} 2024-04-19T23:39:23.880 DEBUG:boto:Token: None 2024-04-19T23:39:23.880 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3bb358ce087b95e-00662300ab-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.883 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.883 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.883 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.883 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.884 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.884 DEBUG:boto:Method: GET 2024-04-19T23:39:23.884 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.884 DEBUG:boto:Data: 2024-04-19T23:39:23.884 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.884 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.884 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.884 DEBUG:boto:Params: {} 2024-04-19T23:39:23.884 DEBUG:boto:Token: None 2024-04-19T23:39:23.884 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.885 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e97002634b9d1b32-00662300ab-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.888 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.888 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.888 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.889 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.889 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.889 DEBUG:boto:Method: GET 2024-04-19T23:39:23.889 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.889 DEBUG:boto:Data: 2024-04-19T23:39:23.889 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.889 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.889 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.889 DEBUG:boto:Params: {} 2024-04-19T23:39:23.889 DEBUG:boto:Token: None 2024-04-19T23:39:23.889 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.890 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZrYfeXg9iQ/FrRZK4n12stznrMk= 2024-04-19T23:39:23.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZrYfeXg9iQ/FrRZK4n12stznrMk=', 'Content-Length': '0'} 2024-04-19T23:39:23.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001695230758094f7b-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.892 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.892 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.892 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.892 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.892 DEBUG:boto:Method: GET 2024-04-19T23:39:23.893 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.893 DEBUG:boto:Data: 2024-04-19T23:39:23.893 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.893 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.893 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.893 DEBUG:boto:Params: {} 2024-04-19T23:39:23.893 DEBUG:boto:Token: None 2024-04-19T23:39:23.893 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.893 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZrYfeXg9iQ/FrRZK4n12stznrMk= 2024-04-19T23:39:23.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZrYfeXg9iQ/FrRZK4n12stznrMk=', 'Content-Length': '0'} 2024-04-19T23:39:23.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb2f0c13aecb8d91-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.938 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.939 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.939 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.939 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.939 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.939 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.939 DEBUG:boto:Data: 2024-04-19T23:39:23.939 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.939 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.939 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.939 DEBUG:boto:Params: {} 2024-04-19T23:39:23.940 DEBUG:boto:Token: None 2024-04-19T23:39:23.940 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.940 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fldPLWEk3WyzFId/wUSODuVLCwY= 2024-04-19T23:39:23.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fldPLWEk3WyzFId/wUSODuVLCwY=', 'Content-Length': '0'} 2024-04-19T23:39:23.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f66edbc9f687b54d-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.942 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.943 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.943 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.943 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.943 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.943 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.943 DEBUG:boto:Data: 2024-04-19T23:39:23.943 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.943 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.943 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.943 DEBUG:boto:Params: {} 2024-04-19T23:39:23.944 DEBUG:boto:Token: None 2024-04-19T23:39:23.944 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fldPLWEk3WyzFId/wUSODuVLCwY= 2024-04-19T23:39:23.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fldPLWEk3WyzFId/wUSODuVLCwY=', 'Content-Length': '0'} 2024-04-19T23:39:23.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a74d16e1cf00ba7a-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.946 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.946 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.946 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.946 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.946 DEBUG:boto:Method: GET 2024-04-19T23:39:23.946 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.946 DEBUG:boto:Data: 2024-04-19T23:39:23.947 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.947 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.947 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.947 DEBUG:boto:Params: {} 2024-04-19T23:39:23.947 DEBUG:boto:Token: None 2024-04-19T23:39:23.947 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.947 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZrYfeXg9iQ/FrRZK4n12stznrMk= 2024-04-19T23:39:23.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZrYfeXg9iQ/FrRZK4n12stznrMk=', 'Content-Length': '0'} 2024-04-19T23:39:23.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b55bf1f6f9681d9d-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.950 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.950 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.950 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.950 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.950 DEBUG:boto:Method: GET 2024-04-19T23:39:23.950 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.950 DEBUG:boto:Data: 2024-04-19T23:39:23.950 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.950 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.950 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.951 DEBUG:boto:Params: {} 2024-04-19T23:39:23.951 DEBUG:boto:Token: None 2024-04-19T23:39:23.951 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:23.951 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZrYfeXg9iQ/FrRZK4n12stznrMk= 2024-04-19T23:39:23.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZrYfeXg9iQ/FrRZK4n12stznrMk=', 'Content-Length': '0'} 2024-04-19T23:39:23.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039bc986bb28411bd-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.953 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.954 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.954 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.954 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.954 DEBUG:boto:Method: GET 2024-04-19T23:39:23.954 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.954 DEBUG:boto:Data: 2024-04-19T23:39:23.954 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.954 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.954 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.954 DEBUG:boto:Params: {} 2024-04-19T23:39:23.955 DEBUG:boto:Token: None 2024-04-19T23:39:23.955 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.955 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1fd9adc19911189-00662300ab-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.958 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.958 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:23.958 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:23.958 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.958 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:23.959 DEBUG:boto:Method: GET 2024-04-19T23:39:23.959 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:23.959 DEBUG:boto:Data: 2024-04-19T23:39:23.959 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.959 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.959 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.959 DEBUG:boto:Params: {} 2024-04-19T23:39:23.959 DEBUG:boto:Token: None 2024-04-19T23:39:23.959 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/ 2024-04-19T23:39:23.959 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk= 2024-04-19T23:39:23.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+F0oPD8Rjkh8MP7t8jn/BRO1Xuk=', 'Content-Length': '0'} 2024-04-19T23:39:23.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043aefc8c1bd1f07d-00662300ab-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.962 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:23.963 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.963 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.963 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.963 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:23.963 DEBUG:boto:Data: 2024-04-19T23:39:23.963 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.963 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.963 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.964 DEBUG:boto:Params: {} 2024-04-19T23:39:23.964 DEBUG:boto:Token: None 2024-04-19T23:39:23.964 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj 2024-04-19T23:39:23.964 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk= 2024-04-19T23:39:23.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk=', 'Content-Length': '0'} 2024-04-19T23:39:23.967 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a32b7ec06c71980e-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.967 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.968 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.968 DEBUG:boto:Method: HEAD 2024-04-19T23:39:23.968 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:23.968 DEBUG:boto:Data: 2024-04-19T23:39:23.968 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.968 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.968 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.968 DEBUG:boto:Params: {} 2024-04-19T23:39:23.968 DEBUG:boto:Token: None 2024-04-19T23:39:23.968 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj 2024-04-19T23:39:23.969 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk= 2024-04-19T23:39:23.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6fWGeSI4MH9yJDKRxOyfqN9B2Vk=', 'Content-Length': '0'} 2024-04-19T23:39:23.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b6f8eb524468f2b-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:23.972 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.972 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.972 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.972 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.973 DEBUG:boto:Method: GET 2024-04-19T23:39:23.973 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.973 DEBUG:boto:Data: 2024-04-19T23:39:23.973 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.973 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:23.973 DEBUG:boto:Port: 8001 2024-04-19T23:39:23.973 DEBUG:boto:Params: {} 2024-04-19T23:39:23.973 DEBUG:boto:Token: None 2024-04-19T23:39:23.973 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.973 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA= 2024-04-19T23:39:23.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA=', 'Content-Length': '0'} 2024-04-19T23:39:23.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090eb2e163c933632-00662300ab-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:23.975 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:23.975 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:23.975 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.976 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.976 DEBUG:boto:Method: GET 2024-04-19T23:39:23.976 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.976 DEBUG:boto:Data: 2024-04-19T23:39:23.976 DEBUG:boto:Headers: {} 2024-04-19T23:39:23.976 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:23.976 DEBUG:boto:Port: 8000 2024-04-19T23:39:23.976 DEBUG:boto:Params: {} 2024-04-19T23:39:23.976 DEBUG:boto:Token: None 2024-04-19T23:39:23.976 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:23 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:23.977 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA= 2024-04-19T23:39:23.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EdXDOlAoQEpRQaGIZ8F1paAU9wA=', 'Content-Length': '0'} 2024-04-19T23:39:23.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051cdf6858ab0ed98-00662300ab-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.026 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.027 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.027 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.027 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.027 DEBUG:boto:Method: GET 2024-04-19T23:39:24.027 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.027 DEBUG:boto:Data: 2024-04-19T23:39:24.027 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.027 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.027 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.027 DEBUG:boto:Params: {} 2024-04-19T23:39:24.028 DEBUG:boto:Token: None 2024-04-19T23:39:24.028 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.028 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5e93863ed4840b0-00662300ac-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.031 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.031 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.032 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.032 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.032 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.032 DEBUG:boto:Method: GET 2024-04-19T23:39:24.032 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.032 DEBUG:boto:Data: 2024-04-19T23:39:24.032 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.032 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.032 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.032 DEBUG:boto:Params: {} 2024-04-19T23:39:24.032 DEBUG:boto:Token: None 2024-04-19T23:39:24.033 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.033 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff0d178becd94a38-00662300ac-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.036 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.036 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.036 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.036 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.036 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.036 DEBUG:boto:Method: GET 2024-04-19T23:39:24.037 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.037 DEBUG:boto:Data: 2024-04-19T23:39:24.037 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.037 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.037 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.037 DEBUG:boto:Params: {} 2024-04-19T23:39:24.037 DEBUG:boto:Token: None 2024-04-19T23:39:24.037 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.037 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ncXeFFu5GBc6KMQWoe2iKweujks= 2024-04-19T23:39:24.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ncXeFFu5GBc6KMQWoe2iKweujks=', 'Content-Length': '0'} 2024-04-19T23:39:24.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004064f9a148eed42f-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.086 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.087 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.087 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.087 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.087 DEBUG:boto:Method: GET 2024-04-19T23:39:24.087 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.087 DEBUG:boto:Data: 2024-04-19T23:39:24.087 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.087 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.087 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.087 DEBUG:boto:Params: {} 2024-04-19T23:39:24.088 DEBUG:boto:Token: None 2024-04-19T23:39:24.088 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.088 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ncXeFFu5GBc6KMQWoe2iKweujks= 2024-04-19T23:39:24.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ncXeFFu5GBc6KMQWoe2iKweujks=', 'Content-Length': '0'} 2024-04-19T23:39:24.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b21fa5a9aeee6364-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.090 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.091 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.091 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.091 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.091 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.091 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.091 DEBUG:boto:Data: 2024-04-19T23:39:24.091 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.091 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.091 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.091 DEBUG:boto:Params: {} 2024-04-19T23:39:24.092 DEBUG:boto:Token: None 2024-04-19T23:39:24.092 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.092 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O9pNnPqemGFBk4DPg0WE47xrvi4= 2024-04-19T23:39:24.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O9pNnPqemGFBk4DPg0WE47xrvi4=', 'Content-Length': '0'} 2024-04-19T23:39:24.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056486cf700b3437f-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.094 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.094 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.095 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.095 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.095 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.095 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.095 DEBUG:boto:Data: 2024-04-19T23:39:24.095 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.095 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.095 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.095 DEBUG:boto:Params: {} 2024-04-19T23:39:24.096 DEBUG:boto:Token: None 2024-04-19T23:39:24.096 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.096 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O9pNnPqemGFBk4DPg0WE47xrvi4= 2024-04-19T23:39:24.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O9pNnPqemGFBk4DPg0WE47xrvi4=', 'Content-Length': '0'} 2024-04-19T23:39:24.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f99fa3095896cf6e-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.098 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.098 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.098 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.098 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.099 DEBUG:boto:Method: GET 2024-04-19T23:39:24.099 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.099 DEBUG:boto:Data: 2024-04-19T23:39:24.099 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.099 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.099 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.099 DEBUG:boto:Params: {} 2024-04-19T23:39:24.099 DEBUG:boto:Token: None 2024-04-19T23:39:24.099 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.099 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ncXeFFu5GBc6KMQWoe2iKweujks= 2024-04-19T23:39:24.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ncXeFFu5GBc6KMQWoe2iKweujks=', 'Content-Length': '0'} 2024-04-19T23:39:24.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bf97f01821d3bbf-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.102 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.102 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.102 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.102 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.103 DEBUG:boto:Method: GET 2024-04-19T23:39:24.103 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.103 DEBUG:boto:Data: 2024-04-19T23:39:24.103 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.103 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.103 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.103 DEBUG:boto:Params: {} 2024-04-19T23:39:24.103 DEBUG:boto:Token: None 2024-04-19T23:39:24.103 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:24.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ncXeFFu5GBc6KMQWoe2iKweujks= 2024-04-19T23:39:24.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ncXeFFu5GBc6KMQWoe2iKweujks=', 'Content-Length': '0'} 2024-04-19T23:39:24.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b8ff4d541e6748c-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.106 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.106 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.106 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.106 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.106 DEBUG:boto:Method: GET 2024-04-19T23:39:24.107 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.107 DEBUG:boto:Data: 2024-04-19T23:39:24.107 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.107 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.107 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.107 DEBUG:boto:Params: {} 2024-04-19T23:39:24.107 DEBUG:boto:Token: None 2024-04-19T23:39:24.107 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.107 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0e7d43fc4a3e3be-00662300ac-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.110 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.111 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.111 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.111 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.111 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.111 DEBUG:boto:Method: GET 2024-04-19T23:39:24.111 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.111 DEBUG:boto:Data: 2024-04-19T23:39:24.111 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.112 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.112 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.112 DEBUG:boto:Params: {} 2024-04-19T23:39:24.112 DEBUG:boto:Token: None 2024-04-19T23:39:24.112 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.112 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0cefad80634d394-00662300ac-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.115 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.116 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.116 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.116 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.116 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:24.116 DEBUG:boto:Data: 2024-04-19T23:39:24.116 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.116 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.116 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.116 DEBUG:boto:Params: {} 2024-04-19T23:39:24.117 DEBUG:boto:Token: None 2024-04-19T23:39:24.117 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj 2024-04-19T23:39:24.117 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q= 2024-04-19T23:39:24.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q=', 'Content-Length': '0'} 2024-04-19T23:39:24.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004584d2c71045a78-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.121 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.121 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.121 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.121 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:24.121 DEBUG:boto:Data: 2024-04-19T23:39:24.121 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.121 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.121 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.121 DEBUG:boto:Params: {} 2024-04-19T23:39:24.122 DEBUG:boto:Token: None 2024-04-19T23:39:24.122 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj 2024-04-19T23:39:24.122 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q= 2024-04-19T23:39:24.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q=', 'Content-Length': '0'} 2024-04-19T23:39:24.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f38141c8080a27b8-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.125 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.125 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.126 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.126 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.126 DEBUG:boto:Method: GET 2024-04-19T23:39:24.126 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.126 DEBUG:boto:Data: 2024-04-19T23:39:24.126 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.126 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.126 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.126 DEBUG:boto:Params: {} 2024-04-19T23:39:24.126 DEBUG:boto:Token: None 2024-04-19T23:39:24.127 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.127 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k= 2024-04-19T23:39:24.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k=', 'Content-Length': '0'} 2024-04-19T23:39:24.128 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b84411251875ade4-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.170 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.171 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.171 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.171 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.171 DEBUG:boto:Method: GET 2024-04-19T23:39:24.171 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.171 DEBUG:boto:Data: 2024-04-19T23:39:24.171 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.171 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.171 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.171 DEBUG:boto:Params: {} 2024-04-19T23:39:24.172 DEBUG:boto:Token: None 2024-04-19T23:39:24.172 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.172 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k= 2024-04-19T23:39:24.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k=', 'Content-Length': '0'} 2024-04-19T23:39:24.174 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000727aca05ece0f3c-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.214 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.215 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.215 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.215 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.215 DEBUG:boto:Method: GET 2024-04-19T23:39:24.215 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.215 DEBUG:boto:Data: 2024-04-19T23:39:24.215 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.215 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.216 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.216 DEBUG:boto:Params: {} 2024-04-19T23:39:24.216 DEBUG:boto:Token: None 2024-04-19T23:39:24.216 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.216 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013554c2dbca3e4c7-00662300ac-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.219 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.220 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.220 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.220 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.220 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.220 DEBUG:boto:Method: GET 2024-04-19T23:39:24.220 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.220 DEBUG:boto:Data: 2024-04-19T23:39:24.220 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.220 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.220 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.221 DEBUG:boto:Params: {} 2024-04-19T23:39:24.221 DEBUG:boto:Token: None 2024-04-19T23:39:24.221 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.221 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cb92e975060c678-00662300ac-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.226 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.227 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.227 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.227 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.227 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.227 DEBUG:boto:Method: GET 2024-04-19T23:39:24.227 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.227 DEBUG:boto:Data: 2024-04-19T23:39:24.227 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.227 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.227 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.227 DEBUG:boto:Params: {} 2024-04-19T23:39:24.228 DEBUG:boto:Token: None 2024-04-19T23:39:24.228 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.228 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BdqWab2Z1YXbEvzigzbdcus3/zE= 2024-04-19T23:39:24.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BdqWab2Z1YXbEvzigzbdcus3/zE=', 'Content-Length': '0'} 2024-04-19T23:39:24.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d2128d0c37ada12-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.230 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.230 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.230 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.230 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.230 DEBUG:boto:Method: GET 2024-04-19T23:39:24.230 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.231 DEBUG:boto:Data: 2024-04-19T23:39:24.231 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.231 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.231 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.231 DEBUG:boto:Params: {} 2024-04-19T23:39:24.231 DEBUG:boto:Token: None 2024-04-19T23:39:24.231 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.231 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BdqWab2Z1YXbEvzigzbdcus3/zE= 2024-04-19T23:39:24.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BdqWab2Z1YXbEvzigzbdcus3/zE=', 'Content-Length': '0'} 2024-04-19T23:39:24.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062168fa56608a18c-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.233 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.234 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.234 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.234 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.234 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.234 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.234 DEBUG:boto:Data: 2024-04-19T23:39:24.234 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.234 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.234 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.234 DEBUG:boto:Params: {} 2024-04-19T23:39:24.235 DEBUG:boto:Token: None 2024-04-19T23:39:24.235 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.235 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L5+zE5dUAmaPd+VmcRHzS7DhaPw= 2024-04-19T23:39:24.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L5+zE5dUAmaPd+VmcRHzS7DhaPw=', 'Content-Length': '0'} 2024-04-19T23:39:24.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088daafa1f71902bf-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.237 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.237 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.237 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.237 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.237 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.237 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.238 DEBUG:boto:Data: 2024-04-19T23:39:24.238 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.238 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.238 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.238 DEBUG:boto:Params: {} 2024-04-19T23:39:24.238 DEBUG:boto:Token: None 2024-04-19T23:39:24.238 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.238 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L5+zE5dUAmaPd+VmcRHzS7DhaPw= 2024-04-19T23:39:24.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L5+zE5dUAmaPd+VmcRHzS7DhaPw=', 'Content-Length': '0'} 2024-04-19T23:39:24.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2dbe7ea9b41c41e-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.241 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.241 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.241 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.241 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.241 DEBUG:boto:Method: GET 2024-04-19T23:39:24.241 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.241 DEBUG:boto:Data: 2024-04-19T23:39:24.241 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.241 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.241 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.241 DEBUG:boto:Params: {} 2024-04-19T23:39:24.242 DEBUG:boto:Token: None 2024-04-19T23:39:24.242 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.242 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BdqWab2Z1YXbEvzigzbdcus3/zE= 2024-04-19T23:39:24.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BdqWab2Z1YXbEvzigzbdcus3/zE=', 'Content-Length': '0'} 2024-04-19T23:39:24.244 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6382218e5af6e53-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.244 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.244 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.244 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.244 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.245 DEBUG:boto:Method: GET 2024-04-19T23:39:24.245 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.245 DEBUG:boto:Data: 2024-04-19T23:39:24.245 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.245 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.245 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.245 DEBUG:boto:Params: {} 2024-04-19T23:39:24.245 DEBUG:boto:Token: None 2024-04-19T23:39:24.245 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:24.245 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BdqWab2Z1YXbEvzigzbdcus3/zE= 2024-04-19T23:39:24.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BdqWab2Z1YXbEvzigzbdcus3/zE=', 'Content-Length': '0'} 2024-04-19T23:39:24.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aca7883ba32691fc-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.294 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.295 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.295 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.295 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.295 DEBUG:boto:Method: GET 2024-04-19T23:39:24.295 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.295 DEBUG:boto:Data: 2024-04-19T23:39:24.295 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.295 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.295 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.295 DEBUG:boto:Params: {} 2024-04-19T23:39:24.296 DEBUG:boto:Token: None 2024-04-19T23:39:24.296 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.296 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0b5db2003b66fe1-00662300ac-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.299 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.300 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.300 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.300 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.300 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.300 DEBUG:boto:Method: GET 2024-04-19T23:39:24.300 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.300 DEBUG:boto:Data: 2024-04-19T23:39:24.300 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.300 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.300 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.300 DEBUG:boto:Params: {} 2024-04-19T23:39:24.301 DEBUG:boto:Token: None 2024-04-19T23:39:24.301 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.301 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba43d1d30bfc6c3d-00662300ac-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.350 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.351 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.351 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.351 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.351 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:24.351 DEBUG:boto:Data: 2024-04-19T23:39:24.351 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.351 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.351 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.352 DEBUG:boto:Params: {} 2024-04-19T23:39:24.352 DEBUG:boto:Token: None 2024-04-19T23:39:24.352 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj 2024-04-19T23:39:24.352 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q= 2024-04-19T23:39:24.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q=', 'Content-Length': '0'} 2024-04-19T23:39:24.355 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013043f2eab8f6418-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.356 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.356 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.356 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.356 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:24.356 DEBUG:boto:Data: 2024-04-19T23:39:24.356 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.356 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.356 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.356 DEBUG:boto:Params: {} 2024-04-19T23:39:24.356 DEBUG:boto:Token: None 2024-04-19T23:39:24.357 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj 2024-04-19T23:39:24.357 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q= 2024-04-19T23:39:24.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q=', 'Content-Length': '0'} 2024-04-19T23:39:24.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b49a600bb3c8ade9-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.362 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.362 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.362 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.362 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.362 DEBUG:boto:Method: GET 2024-04-19T23:39:24.362 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.362 DEBUG:boto:Data: 2024-04-19T23:39:24.362 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.362 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.363 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.363 DEBUG:boto:Params: {} 2024-04-19T23:39:24.363 DEBUG:boto:Token: None 2024-04-19T23:39:24.363 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.363 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k= 2024-04-19T23:39:24.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k=', 'Content-Length': '0'} 2024-04-19T23:39:24.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000251a208a9834f0e3-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.365 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.365 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.365 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.365 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.365 DEBUG:boto:Method: GET 2024-04-19T23:39:24.365 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.366 DEBUG:boto:Data: 2024-04-19T23:39:24.366 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.366 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.366 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.366 DEBUG:boto:Params: {} 2024-04-19T23:39:24.366 DEBUG:boto:Token: None 2024-04-19T23:39:24.366 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.366 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k= 2024-04-19T23:39:24.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k=', 'Content-Length': '0'} 2024-04-19T23:39:24.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5ae5f383f253df9-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.410 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.411 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.411 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.411 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.411 DEBUG:boto:Method: GET 2024-04-19T23:39:24.411 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.411 DEBUG:boto:Data: 2024-04-19T23:39:24.411 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.411 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.411 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.411 DEBUG:boto:Params: {} 2024-04-19T23:39:24.412 DEBUG:boto:Token: None 2024-04-19T23:39:24.412 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.412 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c16af85ea6077db3-00662300ac-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.415 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.415 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.415 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.416 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.416 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.416 DEBUG:boto:Method: GET 2024-04-19T23:39:24.416 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.416 DEBUG:boto:Data: 2024-04-19T23:39:24.416 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.416 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.416 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.416 DEBUG:boto:Params: {} 2024-04-19T23:39:24.416 DEBUG:boto:Token: None 2024-04-19T23:39:24.416 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.417 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000605be2a6df8c62b5-00662300ac-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.466 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.467 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.467 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.467 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.467 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.467 DEBUG:boto:Method: GET 2024-04-19T23:39:24.467 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.468 DEBUG:boto:Data: 2024-04-19T23:39:24.468 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.468 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.468 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.468 DEBUG:boto:Params: {} 2024-04-19T23:39:24.468 DEBUG:boto:Token: None 2024-04-19T23:39:24.468 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.468 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gFSg8WpESriqEPr/G7t+KfEavR4= 2024-04-19T23:39:24.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gFSg8WpESriqEPr/G7t+KfEavR4=', 'Content-Length': '0'} 2024-04-19T23:39:24.470 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bce64d00741ef12-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.471 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.471 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.471 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.471 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.471 DEBUG:boto:Method: GET 2024-04-19T23:39:24.471 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.471 DEBUG:boto:Data: 2024-04-19T23:39:24.471 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.472 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.472 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.472 DEBUG:boto:Params: {} 2024-04-19T23:39:24.472 DEBUG:boto:Token: None 2024-04-19T23:39:24.472 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.472 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gFSg8WpESriqEPr/G7t+KfEavR4= 2024-04-19T23:39:24.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gFSg8WpESriqEPr/G7t+KfEavR4=', 'Content-Length': '0'} 2024-04-19T23:39:24.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8f92ab158e1a4ac-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.518 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.519 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.519 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.519 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.519 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.519 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.519 DEBUG:boto:Data: 2024-04-19T23:39:24.520 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.520 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.520 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.520 DEBUG:boto:Params: {} 2024-04-19T23:39:24.520 DEBUG:boto:Token: None 2024-04-19T23:39:24.520 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.520 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YtQN23cmIubtubrDwLSfDvszdAI= 2024-04-19T23:39:24.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YtQN23cmIubtubrDwLSfDvszdAI=', 'Content-Length': '0'} 2024-04-19T23:39:24.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdba9116d02f8573-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.522 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.523 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.523 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.523 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.523 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.523 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.523 DEBUG:boto:Data: 2024-04-19T23:39:24.523 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.523 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.523 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.523 DEBUG:boto:Params: {} 2024-04-19T23:39:24.523 DEBUG:boto:Token: None 2024-04-19T23:39:24.524 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.524 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YtQN23cmIubtubrDwLSfDvszdAI= 2024-04-19T23:39:24.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YtQN23cmIubtubrDwLSfDvszdAI=', 'Content-Length': '0'} 2024-04-19T23:39:24.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edb8de3cd12d3a9c-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.526 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.526 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.526 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.526 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.526 DEBUG:boto:Method: GET 2024-04-19T23:39:24.527 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.527 DEBUG:boto:Data: 2024-04-19T23:39:24.527 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.527 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.527 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.527 DEBUG:boto:Params: {} 2024-04-19T23:39:24.527 DEBUG:boto:Token: None 2024-04-19T23:39:24.527 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.528 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gFSg8WpESriqEPr/G7t+KfEavR4= 2024-04-19T23:39:24.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gFSg8WpESriqEPr/G7t+KfEavR4=', 'Content-Length': '0'} 2024-04-19T23:39:24.529 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000073b1244516c5cb8-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.530 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.530 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.530 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.530 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.530 DEBUG:boto:Method: GET 2024-04-19T23:39:24.530 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.530 DEBUG:boto:Data: 2024-04-19T23:39:24.531 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.531 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.531 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.531 DEBUG:boto:Params: {} 2024-04-19T23:39:24.531 DEBUG:boto:Token: None 2024-04-19T23:39:24.531 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:24.531 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gFSg8WpESriqEPr/G7t+KfEavR4= 2024-04-19T23:39:24.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gFSg8WpESriqEPr/G7t+KfEavR4=', 'Content-Length': '0'} 2024-04-19T23:39:24.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005dda8a7476070a0a-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.574 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.575 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.575 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.575 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.575 DEBUG:boto:Method: GET 2024-04-19T23:39:24.575 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.575 DEBUG:boto:Data: 2024-04-19T23:39:24.575 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.576 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.576 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.576 DEBUG:boto:Params: {} 2024-04-19T23:39:24.576 DEBUG:boto:Token: None 2024-04-19T23:39:24.576 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.576 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c30b2b8a9885ce16-00662300ac-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.579 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.580 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.580 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.580 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.580 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.580 DEBUG:boto:Method: GET 2024-04-19T23:39:24.580 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.580 DEBUG:boto:Data: 2024-04-19T23:39:24.580 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.581 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.581 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.581 DEBUG:boto:Params: {} 2024-04-19T23:39:24.581 DEBUG:boto:Token: None 2024-04-19T23:39:24.581 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.581 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fd92a89958baeae-00662300ac-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.626 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.627 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.627 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.627 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.627 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:24.627 DEBUG:boto:Data: 2024-04-19T23:39:24.627 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.628 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.628 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.628 DEBUG:boto:Params: {} 2024-04-19T23:39:24.628 DEBUG:boto:Token: None 2024-04-19T23:39:24.628 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj 2024-04-19T23:39:24.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q= 2024-04-19T23:39:24.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q=', 'Content-Length': '0'} 2024-04-19T23:39:24.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbd7dcde026ee429-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.632 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.632 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.632 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.633 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:24.633 DEBUG:boto:Data: 2024-04-19T23:39:24.633 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.633 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.633 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.633 DEBUG:boto:Params: {} 2024-04-19T23:39:24.633 DEBUG:boto:Token: None 2024-04-19T23:39:24.633 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj 2024-04-19T23:39:24.633 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q= 2024-04-19T23:39:24.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q=', 'Content-Length': '0'} 2024-04-19T23:39:24.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000424bf23906e87a13-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.637 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.637 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.637 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.637 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.638 DEBUG:boto:Method: GET 2024-04-19T23:39:24.638 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.638 DEBUG:boto:Data: 2024-04-19T23:39:24.638 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.638 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.638 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.638 DEBUG:boto:Params: {} 2024-04-19T23:39:24.638 DEBUG:boto:Token: None 2024-04-19T23:39:24.638 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.639 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k= 2024-04-19T23:39:24.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k=', 'Content-Length': '0'} 2024-04-19T23:39:24.640 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004721296a87ca936-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.682 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.682 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.683 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.683 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.683 DEBUG:boto:Method: GET 2024-04-19T23:39:24.683 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.683 DEBUG:boto:Data: 2024-04-19T23:39:24.683 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.683 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.683 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.683 DEBUG:boto:Params: {} 2024-04-19T23:39:24.683 DEBUG:boto:Token: None 2024-04-19T23:39:24.684 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.684 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k= 2024-04-19T23:39:24.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k=', 'Content-Length': '0'} 2024-04-19T23:39:24.686 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006117e6fccc1da652-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.726 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.727 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.727 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.727 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.727 DEBUG:boto:Method: GET 2024-04-19T23:39:24.727 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.727 DEBUG:boto:Data: 2024-04-19T23:39:24.727 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.727 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.727 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.727 DEBUG:boto:Params: {} 2024-04-19T23:39:24.728 DEBUG:boto:Token: None 2024-04-19T23:39:24.728 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.728 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5638de440666d55-00662300ac-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.731 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.731 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.731 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.731 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.731 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.732 DEBUG:boto:Method: GET 2024-04-19T23:39:24.732 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.732 DEBUG:boto:Data: 2024-04-19T23:39:24.732 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.732 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.732 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.732 DEBUG:boto:Params: {} 2024-04-19T23:39:24.732 DEBUG:boto:Token: None 2024-04-19T23:39:24.732 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.732 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014edb55e30861fef-00662300ac-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.736 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.736 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.736 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.736 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.736 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.737 DEBUG:boto:Method: GET 2024-04-19T23:39:24.737 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.737 DEBUG:boto:Data: 2024-04-19T23:39:24.737 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.737 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.737 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.737 DEBUG:boto:Params: {} 2024-04-19T23:39:24.737 DEBUG:boto:Token: None 2024-04-19T23:39:24.737 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.737 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LsAw/UUkrJow/G5U79+0eR8pOJc= 2024-04-19T23:39:24.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LsAw/UUkrJow/G5U79+0eR8pOJc=', 'Content-Length': '0'} 2024-04-19T23:39:24.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000311394c134c8786c-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.740 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.740 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.740 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.740 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.740 DEBUG:boto:Method: GET 2024-04-19T23:39:24.740 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.740 DEBUG:boto:Data: 2024-04-19T23:39:24.741 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.741 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.741 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.741 DEBUG:boto:Params: {} 2024-04-19T23:39:24.741 DEBUG:boto:Token: None 2024-04-19T23:39:24.741 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.741 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LsAw/UUkrJow/G5U79+0eR8pOJc= 2024-04-19T23:39:24.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LsAw/UUkrJow/G5U79+0eR8pOJc=', 'Content-Length': '0'} 2024-04-19T23:39:24.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091e6dc3fcf72cbb3-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.786 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.787 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.787 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.787 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.787 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.787 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.787 DEBUG:boto:Data: 2024-04-19T23:39:24.787 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.787 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.787 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.787 DEBUG:boto:Params: {} 2024-04-19T23:39:24.787 DEBUG:boto:Token: None 2024-04-19T23:39:24.788 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.788 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:c62OzebyJkSvXD1nAmY3SH6SbR8= 2024-04-19T23:39:24.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c62OzebyJkSvXD1nAmY3SH6SbR8=', 'Content-Length': '0'} 2024-04-19T23:39:24.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6d6b9885b0595c1-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.790 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.790 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.790 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.791 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.791 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.791 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.791 DEBUG:boto:Data: 2024-04-19T23:39:24.791 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.791 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.791 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.791 DEBUG:boto:Params: {} 2024-04-19T23:39:24.791 DEBUG:boto:Token: None 2024-04-19T23:39:24.791 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.792 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:c62OzebyJkSvXD1nAmY3SH6SbR8= 2024-04-19T23:39:24.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c62OzebyJkSvXD1nAmY3SH6SbR8=', 'Content-Length': '0'} 2024-04-19T23:39:24.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093e5d2b15b421f0e-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.794 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.794 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.794 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.794 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.794 DEBUG:boto:Method: GET 2024-04-19T23:39:24.794 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.795 DEBUG:boto:Data: 2024-04-19T23:39:24.795 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.795 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.795 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.795 DEBUG:boto:Params: {} 2024-04-19T23:39:24.795 DEBUG:boto:Token: None 2024-04-19T23:39:24.795 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.795 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LsAw/UUkrJow/G5U79+0eR8pOJc= 2024-04-19T23:39:24.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LsAw/UUkrJow/G5U79+0eR8pOJc=', 'Content-Length': '0'} 2024-04-19T23:39:24.797 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a45e8603e1e93d4c-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.798 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.798 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.798 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.798 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.798 DEBUG:boto:Method: GET 2024-04-19T23:39:24.798 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.798 DEBUG:boto:Data: 2024-04-19T23:39:24.798 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.798 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.798 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.798 DEBUG:boto:Params: {} 2024-04-19T23:39:24.799 DEBUG:boto:Token: None 2024-04-19T23:39:24.799 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:24.799 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LsAw/UUkrJow/G5U79+0eR8pOJc= 2024-04-19T23:39:24.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LsAw/UUkrJow/G5U79+0eR8pOJc=', 'Content-Length': '0'} 2024-04-19T23:39:24.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ffff5f6badae0df-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.846 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.846 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.846 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.846 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.846 DEBUG:boto:Method: GET 2024-04-19T23:39:24.846 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.846 DEBUG:boto:Data: 2024-04-19T23:39:24.847 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.847 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.847 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.847 DEBUG:boto:Params: {} 2024-04-19T23:39:24.847 DEBUG:boto:Token: None 2024-04-19T23:39:24.847 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.847 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5d8da8da888f0d4-00662300ac-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.851 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.851 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.851 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.851 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.851 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.852 DEBUG:boto:Method: GET 2024-04-19T23:39:24.852 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.852 DEBUG:boto:Data: 2024-04-19T23:39:24.852 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.852 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.852 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.852 DEBUG:boto:Params: {} 2024-04-19T23:39:24.852 DEBUG:boto:Token: None 2024-04-19T23:39:24.852 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.853 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044f6a300a4b5acfa-00662300ac-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.855 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.856 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.856 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.856 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.856 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:24.856 DEBUG:boto:Data: 2024-04-19T23:39:24.856 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.856 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.856 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.857 DEBUG:boto:Params: {} 2024-04-19T23:39:24.857 DEBUG:boto:Token: None 2024-04-19T23:39:24.857 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj 2024-04-19T23:39:24.857 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q= 2024-04-19T23:39:24.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q=', 'Content-Length': '0'} 2024-04-19T23:39:24.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8069bc873a46197-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.861 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.861 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.861 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.861 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:24.861 DEBUG:boto:Data: 2024-04-19T23:39:24.861 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.861 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.862 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.862 DEBUG:boto:Params: {} 2024-04-19T23:39:24.862 DEBUG:boto:Token: None 2024-04-19T23:39:24.862 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj 2024-04-19T23:39:24.862 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q= 2024-04-19T23:39:24.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie8PVxuxgtsurF0taPYaeFCCW2Q=', 'Content-Length': '0'} 2024-04-19T23:39:24.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069e0e66130e7efe2-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.866 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.866 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.866 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.866 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.866 DEBUG:boto:Method: GET 2024-04-19T23:39:24.866 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.866 DEBUG:boto:Data: 2024-04-19T23:39:24.866 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.867 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.867 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.867 DEBUG:boto:Params: {} 2024-04-19T23:39:24.867 DEBUG:boto:Token: None 2024-04-19T23:39:24.867 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.867 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k= 2024-04-19T23:39:24.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k=', 'Content-Length': '0'} 2024-04-19T23:39:24.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071d890963a988c06-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.910 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.911 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.911 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.911 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.911 DEBUG:boto:Method: GET 2024-04-19T23:39:24.911 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.911 DEBUG:boto:Data: 2024-04-19T23:39:24.911 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.911 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.911 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.911 DEBUG:boto:Params: {} 2024-04-19T23:39:24.912 DEBUG:boto:Token: None 2024-04-19T23:39:24.912 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:24.912 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k= 2024-04-19T23:39:24.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ukm7L8293xxkPbQ4gy2mnl9Sa3k=', 'Content-Length': '0'} 2024-04-19T23:39:24.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f77b0081fad47b90-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.954 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.955 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.955 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.955 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.955 DEBUG:boto:Method: GET 2024-04-19T23:39:24.955 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.955 DEBUG:boto:Data: 2024-04-19T23:39:24.955 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.955 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.955 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.955 DEBUG:boto:Params: {} 2024-04-19T23:39:24.956 DEBUG:boto:Token: None 2024-04-19T23:39:24.956 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.956 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c4d298e0a940880-00662300ac-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.959 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.959 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:24.959 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:24.959 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.959 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:24.960 DEBUG:boto:Method: GET 2024-04-19T23:39:24.960 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:24.960 DEBUG:boto:Data: 2024-04-19T23:39:24.960 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.960 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.960 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.960 DEBUG:boto:Params: {} 2024-04-19T23:39:24.960 DEBUG:boto:Token: None 2024-04-19T23:39:24.960 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/ 2024-04-19T23:39:24.960 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8= 2024-04-19T23:39:24.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uWOX/RYUDdz0wgmnXE0GuHOvBm8=', 'Content-Length': '0'} 2024-04-19T23:39:24.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3cf701a00577153-00662300ac-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.963 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:24.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.964 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.964 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.964 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.964 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.964 DEBUG:boto:Method: GET 2024-04-19T23:39:24.965 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.965 DEBUG:boto:Data: 2024-04-19T23:39:24.965 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.965 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.965 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.965 DEBUG:boto:Params: {} 2024-04-19T23:39:24.965 DEBUG:boto:Token: None 2024-04-19T23:39:24.965 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.965 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Xt/Rm/y4g5rVbw347kcb9t3Gv3c= 2024-04-19T23:39:24.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Xt/Rm/y4g5rVbw347kcb9t3Gv3c=', 'Content-Length': '0'} 2024-04-19T23:39:24.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008831589b9e9d86b0-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.968 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.968 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.968 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.968 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.968 DEBUG:boto:Method: GET 2024-04-19T23:39:24.969 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.969 DEBUG:boto:Data: 2024-04-19T23:39:24.969 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.969 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.969 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.969 DEBUG:boto:Params: {} 2024-04-19T23:39:24.969 DEBUG:boto:Token: None 2024-04-19T23:39:24.969 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.969 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Xt/Rm/y4g5rVbw347kcb9t3Gv3c= 2024-04-19T23:39:24.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Xt/Rm/y4g5rVbw347kcb9t3Gv3c=', 'Content-Length': '0'} 2024-04-19T23:39:24.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e43a1f5bb6dc9f2f-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.972 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.972 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.972 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.972 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.972 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.972 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.972 DEBUG:boto:Data: 2024-04-19T23:39:24.972 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.973 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.973 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.973 DEBUG:boto:Params: {} 2024-04-19T23:39:24.973 DEBUG:boto:Token: None 2024-04-19T23:39:24.973 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.973 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bwa0x6XfFqNCD9dFlv78BtGa5QM= 2024-04-19T23:39:24.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bwa0x6XfFqNCD9dFlv78BtGa5QM=', 'Content-Length': '0'} 2024-04-19T23:39:24.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044b9c8516f36e926-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.975 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.976 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.976 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.976 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.976 DEBUG:boto:Method: HEAD 2024-04-19T23:39:24.976 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.976 DEBUG:boto:Data: 2024-04-19T23:39:24.976 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.976 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:24.976 DEBUG:boto:Port: 8000 2024-04-19T23:39:24.976 DEBUG:boto:Params: {} 2024-04-19T23:39:24.977 DEBUG:boto:Token: None 2024-04-19T23:39:24.977 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.977 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bwa0x6XfFqNCD9dFlv78BtGa5QM= 2024-04-19T23:39:24.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bwa0x6XfFqNCD9dFlv78BtGa5QM=', 'Content-Length': '0'} 2024-04-19T23:39:24.979 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ae2bd824c86f9a3-00662300ac-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:24.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:24.979 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:24.979 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:24.979 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.980 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.980 DEBUG:boto:Method: GET 2024-04-19T23:39:24.980 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.980 DEBUG:boto:Data: 2024-04-19T23:39:24.980 DEBUG:boto:Headers: {} 2024-04-19T23:39:24.980 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:24.980 DEBUG:boto:Port: 8001 2024-04-19T23:39:24.980 DEBUG:boto:Params: {} 2024-04-19T23:39:24.980 DEBUG:boto:Token: None 2024-04-19T23:39:24.980 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:24 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:24.981 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Xt/Rm/y4g5rVbw347kcb9t3Gv3c= 2024-04-19T23:39:24.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:24 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Xt/Rm/y4g5rVbw347kcb9t3Gv3c=', 'Content-Length': '0'} 2024-04-19T23:39:24.982 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5e566afd7de3495-00662300ac-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.026 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.027 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.027 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:25.027 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:25.027 DEBUG:boto:Method: GET 2024-04-19T23:39:25.027 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:25.027 DEBUG:boto:Data: 2024-04-19T23:39:25.027 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.027 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.027 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.027 DEBUG:boto:Params: {} 2024-04-19T23:39:25.028 DEBUG:boto:Token: None 2024-04-19T23:39:25.028 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:25.028 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:64iKt/Qrkk7zPb2Pfc0xjb+IcCU= 2024-04-19T23:39:25.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:64iKt/Qrkk7zPb2Pfc0xjb+IcCU=', 'Content-Length': '0'} 2024-04-19T23:39:25.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009362d0062b059674-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.030 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.030 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.031 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.031 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.031 DEBUG:boto:Method: GET 2024-04-19T23:39:25.031 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.031 DEBUG:boto:Data: 2024-04-19T23:39:25.031 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.031 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.031 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.031 DEBUG:boto:Params: {} 2024-04-19T23:39:25.031 DEBUG:boto:Token: None 2024-04-19T23:39:25.032 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.032 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000145e9082474b744-00662300ad-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.035 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.035 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.035 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.035 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.035 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.035 DEBUG:boto:Method: GET 2024-04-19T23:39:25.036 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.036 DEBUG:boto:Data: 2024-04-19T23:39:25.036 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.036 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.036 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.036 DEBUG:boto:Params: {} 2024-04-19T23:39:25.037 DEBUG:boto:Token: None 2024-04-19T23:39:25.037 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.037 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a0a7596413311e6-00662300ad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.040 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.040 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.040 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.040 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.041 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:25.041 DEBUG:boto:Data: 2024-04-19T23:39:25.041 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.041 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.041 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.041 DEBUG:boto:Params: {} 2024-04-19T23:39:25.041 DEBUG:boto:Token: None 2024-04-19T23:39:25.041 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj 2024-04-19T23:39:25.041 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg= 2024-04-19T23:39:25.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg=', 'Content-Length': '0'} 2024-04-19T23:39:25.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001168990b8e8d7354-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.045 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.045 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.045 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.045 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:25.045 DEBUG:boto:Data: 2024-04-19T23:39:25.046 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.046 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.046 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.046 DEBUG:boto:Params: {} 2024-04-19T23:39:25.046 DEBUG:boto:Token: None 2024-04-19T23:39:25.046 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj 2024-04-19T23:39:25.046 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg= 2024-04-19T23:39:25.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg=', 'Content-Length': '0'} 2024-04-19T23:39:25.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000401c6d41942161ca-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:25.050 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.050 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.050 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.050 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.050 DEBUG:boto:Method: GET 2024-04-19T23:39:25.050 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.050 DEBUG:boto:Data: 2024-04-19T23:39:25.051 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.051 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.051 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.051 DEBUG:boto:Params: {} 2024-04-19T23:39:25.054 DEBUG:boto:Token: None 2024-04-19T23:39:25.054 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.054 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak= 2024-04-19T23:39:25.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak=', 'Content-Length': '0'} 2024-04-19T23:39:25.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e1697e3624204d5-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.098 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.099 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.099 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.099 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.099 DEBUG:boto:Method: GET 2024-04-19T23:39:25.099 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.099 DEBUG:boto:Data: 2024-04-19T23:39:25.099 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.099 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.099 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.100 DEBUG:boto:Params: {} 2024-04-19T23:39:25.100 DEBUG:boto:Token: None 2024-04-19T23:39:25.100 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.100 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak= 2024-04-19T23:39:25.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak=', 'Content-Length': '0'} 2024-04-19T23:39:25.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a23d397cba200717-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.142 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.143 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.143 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.143 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.143 DEBUG:boto:Method: GET 2024-04-19T23:39:25.143 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.143 DEBUG:boto:Data: 2024-04-19T23:39:25.143 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.143 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.143 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.143 DEBUG:boto:Params: {} 2024-04-19T23:39:25.144 DEBUG:boto:Token: None 2024-04-19T23:39:25.144 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.144 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053f5c00f06528e0d-00662300ad-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.147 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.147 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.147 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.147 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.148 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.148 DEBUG:boto:Method: GET 2024-04-19T23:39:25.148 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.148 DEBUG:boto:Data: 2024-04-19T23:39:25.148 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.148 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.148 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.148 DEBUG:boto:Params: {} 2024-04-19T23:39:25.148 DEBUG:boto:Token: None 2024-04-19T23:39:25.148 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.149 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df4266f183e30d73-00662300ad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.152 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:25.152 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.152 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.152 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.153 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.153 DEBUG:boto:Method: GET 2024-04-19T23:39:25.153 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.153 DEBUG:boto:Data: 2024-04-19T23:39:25.153 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.153 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.153 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.153 DEBUG:boto:Params: {} 2024-04-19T23:39:25.153 DEBUG:boto:Token: None 2024-04-19T23:39:25.154 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.154 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qJxG+34AyX19iRjBa1jn7Ga+0aU= 2024-04-19T23:39:25.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qJxG+34AyX19iRjBa1jn7Ga+0aU=', 'Content-Length': '0'} 2024-04-19T23:39:25.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b1e0930156284ed-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.156 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.156 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.156 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.156 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.156 DEBUG:boto:Method: GET 2024-04-19T23:39:25.156 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.157 DEBUG:boto:Data: 2024-04-19T23:39:25.157 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.157 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.157 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.157 DEBUG:boto:Params: {} 2024-04-19T23:39:25.158 DEBUG:boto:Token: None 2024-04-19T23:39:25.158 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.158 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qJxG+34AyX19iRjBa1jn7Ga+0aU= 2024-04-19T23:39:25.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qJxG+34AyX19iRjBa1jn7Ga+0aU=', 'Content-Length': '0'} 2024-04-19T23:39:25.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c69332eb399e551a-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.202 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.203 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.203 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.203 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.203 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.203 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.203 DEBUG:boto:Data: 2024-04-19T23:39:25.203 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.203 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.203 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.204 DEBUG:boto:Params: {} 2024-04-19T23:39:25.204 DEBUG:boto:Token: None 2024-04-19T23:39:25.204 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.204 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6EeOVaVenwLnxseoUuPQMdhRicU= 2024-04-19T23:39:25.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6EeOVaVenwLnxseoUuPQMdhRicU=', 'Content-Length': '0'} 2024-04-19T23:39:25.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000663fea6f43724235-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.206 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.207 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.207 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.207 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.207 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.207 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.207 DEBUG:boto:Data: 2024-04-19T23:39:25.207 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.207 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.207 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.207 DEBUG:boto:Params: {} 2024-04-19T23:39:25.207 DEBUG:boto:Token: None 2024-04-19T23:39:25.208 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.208 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6EeOVaVenwLnxseoUuPQMdhRicU= 2024-04-19T23:39:25.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6EeOVaVenwLnxseoUuPQMdhRicU=', 'Content-Length': '0'} 2024-04-19T23:39:25.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068d1bc4d1ffd8904-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:25.210 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.211 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.211 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.211 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.211 DEBUG:boto:Method: GET 2024-04-19T23:39:25.211 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.211 DEBUG:boto:Data: 2024-04-19T23:39:25.211 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.211 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.211 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.211 DEBUG:boto:Params: {} 2024-04-19T23:39:25.211 DEBUG:boto:Token: None 2024-04-19T23:39:25.212 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.212 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qJxG+34AyX19iRjBa1jn7Ga+0aU= 2024-04-19T23:39:25.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qJxG+34AyX19iRjBa1jn7Ga+0aU=', 'Content-Length': '0'} 2024-04-19T23:39:25.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb0e5244063ff46d-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.214 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.214 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.214 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.214 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.214 DEBUG:boto:Method: GET 2024-04-19T23:39:25.214 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.214 DEBUG:boto:Data: 2024-04-19T23:39:25.215 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.215 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.215 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.215 DEBUG:boto:Params: {} 2024-04-19T23:39:25.215 DEBUG:boto:Token: None 2024-04-19T23:39:25.215 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:25.215 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qJxG+34AyX19iRjBa1jn7Ga+0aU= 2024-04-19T23:39:25.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qJxG+34AyX19iRjBa1jn7Ga+0aU=', 'Content-Length': '0'} 2024-04-19T23:39:25.217 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbb495c7aadc7822-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.258 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.259 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.259 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.259 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.259 DEBUG:boto:Method: GET 2024-04-19T23:39:25.259 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.260 DEBUG:boto:Data: 2024-04-19T23:39:25.260 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.260 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.260 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.260 DEBUG:boto:Params: {} 2024-04-19T23:39:25.260 DEBUG:boto:Token: None 2024-04-19T23:39:25.260 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.260 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000956080ca8fcf3ecc-00662300ad-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.264 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.264 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.265 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.265 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.265 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.265 DEBUG:boto:Method: GET 2024-04-19T23:39:25.265 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.265 DEBUG:boto:Data: 2024-04-19T23:39:25.265 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.265 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.265 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.265 DEBUG:boto:Params: {} 2024-04-19T23:39:25.266 DEBUG:boto:Token: None 2024-04-19T23:39:25.266 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.266 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b1103cba7dceacb-00662300ad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.269 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.269 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.270 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.270 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.270 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:25.270 DEBUG:boto:Data: 2024-04-19T23:39:25.270 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.270 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.270 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.270 DEBUG:boto:Params: {} 2024-04-19T23:39:25.270 DEBUG:boto:Token: None 2024-04-19T23:39:25.271 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj 2024-04-19T23:39:25.271 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg= 2024-04-19T23:39:25.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg=', 'Content-Length': '0'} 2024-04-19T23:39:25.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfe74cadd51c0654-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.274 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.275 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.275 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.275 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:25.275 DEBUG:boto:Data: 2024-04-19T23:39:25.275 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.275 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.275 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.275 DEBUG:boto:Params: {} 2024-04-19T23:39:25.275 DEBUG:boto:Token: None 2024-04-19T23:39:25.275 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj 2024-04-19T23:39:25.276 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg= 2024-04-19T23:39:25.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg=', 'Content-Length': '0'} 2024-04-19T23:39:25.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f3f05c2282b47f5-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:25.279 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.279 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.279 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.280 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.280 DEBUG:boto:Method: GET 2024-04-19T23:39:25.280 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.280 DEBUG:boto:Data: 2024-04-19T23:39:25.280 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.280 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.280 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.280 DEBUG:boto:Params: {} 2024-04-19T23:39:25.280 DEBUG:boto:Token: None 2024-04-19T23:39:25.280 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.280 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak= 2024-04-19T23:39:25.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak=', 'Content-Length': '0'} 2024-04-19T23:39:25.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc358abf4d7ce819-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.330 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.331 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.331 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.331 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.331 DEBUG:boto:Method: GET 2024-04-19T23:39:25.331 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.331 DEBUG:boto:Data: 2024-04-19T23:39:25.331 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.331 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.331 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.331 DEBUG:boto:Params: {} 2024-04-19T23:39:25.331 DEBUG:boto:Token: None 2024-04-19T23:39:25.332 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak= 2024-04-19T23:39:25.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak=', 'Content-Length': '0'} 2024-04-19T23:39:25.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aad68ee1ad256e1b-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.378 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.379 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.379 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.379 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.379 DEBUG:boto:Method: GET 2024-04-19T23:39:25.379 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.379 DEBUG:boto:Data: 2024-04-19T23:39:25.379 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.379 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.379 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.379 DEBUG:boto:Params: {} 2024-04-19T23:39:25.380 DEBUG:boto:Token: None 2024-04-19T23:39:25.380 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.380 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004877632b0a42fc5b-00662300ad-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.426 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.427 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.427 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.427 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.427 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.427 DEBUG:boto:Method: GET 2024-04-19T23:39:25.427 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.427 DEBUG:boto:Data: 2024-04-19T23:39:25.427 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.428 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.428 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.428 DEBUG:boto:Params: {} 2024-04-19T23:39:25.428 DEBUG:boto:Token: None 2024-04-19T23:39:25.428 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.428 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000328e896d8d9073-00662300ad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.478 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:25.479 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.479 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.479 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.479 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.479 DEBUG:boto:Method: GET 2024-04-19T23:39:25.479 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.479 DEBUG:boto:Data: 2024-04-19T23:39:25.479 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.479 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.479 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.480 DEBUG:boto:Params: {} 2024-04-19T23:39:25.480 DEBUG:boto:Token: None 2024-04-19T23:39:25.480 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.480 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Uo70KD9scgYtXCSOfeeIiCGxCVY= 2024-04-19T23:39:25.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Uo70KD9scgYtXCSOfeeIiCGxCVY=', 'Content-Length': '0'} 2024-04-19T23:39:25.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a8b2561605a0f8f-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.483 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.483 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.483 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.483 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.483 DEBUG:boto:Method: GET 2024-04-19T23:39:25.483 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.483 DEBUG:boto:Data: 2024-04-19T23:39:25.483 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.483 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.483 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.483 DEBUG:boto:Params: {} 2024-04-19T23:39:25.484 DEBUG:boto:Token: None 2024-04-19T23:39:25.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Uo70KD9scgYtXCSOfeeIiCGxCVY= 2024-04-19T23:39:25.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Uo70KD9scgYtXCSOfeeIiCGxCVY=', 'Content-Length': '0'} 2024-04-19T23:39:25.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c209f4faaf40576e-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.486 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.486 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.486 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.486 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.486 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.487 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.487 DEBUG:boto:Data: 2024-04-19T23:39:25.487 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.487 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.487 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.487 DEBUG:boto:Params: {} 2024-04-19T23:39:25.487 DEBUG:boto:Token: None 2024-04-19T23:39:25.487 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.487 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SvXmpRuieIU0V1yI8JIENc9tckA= 2024-04-19T23:39:25.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SvXmpRuieIU0V1yI8JIENc9tckA=', 'Content-Length': '0'} 2024-04-19T23:39:25.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083f374c8b924ae58-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.490 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.490 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.490 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.490 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.490 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.490 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.490 DEBUG:boto:Data: 2024-04-19T23:39:25.490 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.490 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.490 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.491 DEBUG:boto:Params: {} 2024-04-19T23:39:25.491 DEBUG:boto:Token: None 2024-04-19T23:39:25.491 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.491 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SvXmpRuieIU0V1yI8JIENc9tckA= 2024-04-19T23:39:25.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SvXmpRuieIU0V1yI8JIENc9tckA=', 'Content-Length': '0'} 2024-04-19T23:39:25.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c220d8666c3e1718-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:25.493 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.493 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.493 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.493 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.493 DEBUG:boto:Method: GET 2024-04-19T23:39:25.493 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.493 DEBUG:boto:Data: 2024-04-19T23:39:25.494 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.494 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.494 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.494 DEBUG:boto:Params: {} 2024-04-19T23:39:25.494 DEBUG:boto:Token: None 2024-04-19T23:39:25.494 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.494 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Uo70KD9scgYtXCSOfeeIiCGxCVY= 2024-04-19T23:39:25.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Uo70KD9scgYtXCSOfeeIiCGxCVY=', 'Content-Length': '0'} 2024-04-19T23:39:25.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbea6e6188433ebf-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.538 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.538 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.538 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.539 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.539 DEBUG:boto:Method: GET 2024-04-19T23:39:25.539 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.539 DEBUG:boto:Data: 2024-04-19T23:39:25.539 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.539 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.539 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.539 DEBUG:boto:Params: {} 2024-04-19T23:39:25.539 DEBUG:boto:Token: None 2024-04-19T23:39:25.539 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:25.540 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Uo70KD9scgYtXCSOfeeIiCGxCVY= 2024-04-19T23:39:25.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Uo70KD9scgYtXCSOfeeIiCGxCVY=', 'Content-Length': '0'} 2024-04-19T23:39:25.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041ff17e09cf612af-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.582 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.583 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.583 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.583 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.583 DEBUG:boto:Method: GET 2024-04-19T23:39:25.583 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.583 DEBUG:boto:Data: 2024-04-19T23:39:25.583 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.583 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.583 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.583 DEBUG:boto:Params: {} 2024-04-19T23:39:25.584 DEBUG:boto:Token: None 2024-04-19T23:39:25.584 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.584 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6372970462a08a6-00662300ad-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.587 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.587 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.587 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.587 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.587 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.588 DEBUG:boto:Method: GET 2024-04-19T23:39:25.588 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.588 DEBUG:boto:Data: 2024-04-19T23:39:25.588 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.588 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.588 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.588 DEBUG:boto:Params: {} 2024-04-19T23:39:25.588 DEBUG:boto:Token: None 2024-04-19T23:39:25.588 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.588 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000819cb7ae294d72c9-00662300ad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.592 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.592 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.592 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.592 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.592 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:25.592 DEBUG:boto:Data: 2024-04-19T23:39:25.592 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.593 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.593 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.593 DEBUG:boto:Params: {} 2024-04-19T23:39:25.593 DEBUG:boto:Token: None 2024-04-19T23:39:25.593 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj 2024-04-19T23:39:25.593 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg= 2024-04-19T23:39:25.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg=', 'Content-Length': '0'} 2024-04-19T23:39:25.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b42b153720520ce4-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.596 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.597 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.597 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.597 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:25.597 DEBUG:boto:Data: 2024-04-19T23:39:25.597 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.597 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.597 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.597 DEBUG:boto:Params: {} 2024-04-19T23:39:25.597 DEBUG:boto:Token: None 2024-04-19T23:39:25.597 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj 2024-04-19T23:39:25.598 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg= 2024-04-19T23:39:25.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg=', 'Content-Length': '0'} 2024-04-19T23:39:25.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aded9d02de0aab76-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:25.601 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.601 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.602 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.602 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.602 DEBUG:boto:Method: GET 2024-04-19T23:39:25.602 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.602 DEBUG:boto:Data: 2024-04-19T23:39:25.602 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.602 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.602 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.602 DEBUG:boto:Params: {} 2024-04-19T23:39:25.602 DEBUG:boto:Token: None 2024-04-19T23:39:25.603 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.603 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak= 2024-04-19T23:39:25.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak=', 'Content-Length': '0'} 2024-04-19T23:39:25.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4ae09e92964d50d-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.650 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.651 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.651 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.651 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.651 DEBUG:boto:Method: GET 2024-04-19T23:39:25.651 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.651 DEBUG:boto:Data: 2024-04-19T23:39:25.651 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.651 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.651 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.651 DEBUG:boto:Params: {} 2024-04-19T23:39:25.652 DEBUG:boto:Token: None 2024-04-19T23:39:25.652 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.652 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak= 2024-04-19T23:39:25.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak=', 'Content-Length': '0'} 2024-04-19T23:39:25.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032ab8ab2ed906803-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.694 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.695 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.695 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.695 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.695 DEBUG:boto:Method: GET 2024-04-19T23:39:25.695 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.695 DEBUG:boto:Data: 2024-04-19T23:39:25.695 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.695 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.695 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.695 DEBUG:boto:Params: {} 2024-04-19T23:39:25.696 DEBUG:boto:Token: None 2024-04-19T23:39:25.696 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.696 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fab15adab8c62796-00662300ad-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.699 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.700 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.700 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.700 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.700 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.700 DEBUG:boto:Method: GET 2024-04-19T23:39:25.700 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.700 DEBUG:boto:Data: 2024-04-19T23:39:25.700 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.700 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.701 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.701 DEBUG:boto:Params: {} 2024-04-19T23:39:25.701 DEBUG:boto:Token: None 2024-04-19T23:39:25.701 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.701 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064fb9ff07395fcd8-00662300ad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.704 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:25.704 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.704 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.704 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.704 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.705 DEBUG:boto:Method: GET 2024-04-19T23:39:25.705 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.705 DEBUG:boto:Data: 2024-04-19T23:39:25.705 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.705 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.705 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.705 DEBUG:boto:Params: {} 2024-04-19T23:39:25.705 DEBUG:boto:Token: None 2024-04-19T23:39:25.705 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.705 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AMTIKAYhjz0rYN/FovhS3LKhC84= 2024-04-19T23:39:25.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AMTIKAYhjz0rYN/FovhS3LKhC84=', 'Content-Length': '0'} 2024-04-19T23:39:25.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006446a1d0a13a26de-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.708 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.708 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.708 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.708 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.708 DEBUG:boto:Method: GET 2024-04-19T23:39:25.708 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.708 DEBUG:boto:Data: 2024-04-19T23:39:25.708 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.708 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.708 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.708 DEBUG:boto:Params: {} 2024-04-19T23:39:25.709 DEBUG:boto:Token: None 2024-04-19T23:39:25.709 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.709 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AMTIKAYhjz0rYN/FovhS3LKhC84= 2024-04-19T23:39:25.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AMTIKAYhjz0rYN/FovhS3LKhC84=', 'Content-Length': '0'} 2024-04-19T23:39:25.711 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000587e7b6ccdb9d5e0-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.754 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.755 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.755 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.755 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.755 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.755 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.755 DEBUG:boto:Data: 2024-04-19T23:39:25.755 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.755 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.755 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.755 DEBUG:boto:Params: {} 2024-04-19T23:39:25.756 DEBUG:boto:Token: None 2024-04-19T23:39:25.756 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.756 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YNrn+hBwNAuE0TSKRxEv4VMpTWc= 2024-04-19T23:39:25.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YNrn+hBwNAuE0TSKRxEv4VMpTWc=', 'Content-Length': '0'} 2024-04-19T23:39:25.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000183d83ce7c3ba30-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.758 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.758 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.758 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.759 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.759 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.759 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.759 DEBUG:boto:Data: 2024-04-19T23:39:25.759 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.759 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.759 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.759 DEBUG:boto:Params: {} 2024-04-19T23:39:25.759 DEBUG:boto:Token: None 2024-04-19T23:39:25.759 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.760 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YNrn+hBwNAuE0TSKRxEv4VMpTWc= 2024-04-19T23:39:25.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YNrn+hBwNAuE0TSKRxEv4VMpTWc=', 'Content-Length': '0'} 2024-04-19T23:39:25.761 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7ce506cf4ce745b-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:25.762 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.762 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.762 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.762 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.762 DEBUG:boto:Method: GET 2024-04-19T23:39:25.762 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.762 DEBUG:boto:Data: 2024-04-19T23:39:25.762 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.762 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.762 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.763 DEBUG:boto:Params: {} 2024-04-19T23:39:25.763 DEBUG:boto:Token: None 2024-04-19T23:39:25.763 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.763 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AMTIKAYhjz0rYN/FovhS3LKhC84= 2024-04-19T23:39:25.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AMTIKAYhjz0rYN/FovhS3LKhC84=', 'Content-Length': '0'} 2024-04-19T23:39:25.765 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005223cbf40fb82b61-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.765 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.765 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.765 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.765 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.765 DEBUG:boto:Method: GET 2024-04-19T23:39:25.765 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.766 DEBUG:boto:Data: 2024-04-19T23:39:25.766 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.766 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.766 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.766 DEBUG:boto:Params: {} 2024-04-19T23:39:25.766 DEBUG:boto:Token: None 2024-04-19T23:39:25.766 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:25.766 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AMTIKAYhjz0rYN/FovhS3LKhC84= 2024-04-19T23:39:25.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AMTIKAYhjz0rYN/FovhS3LKhC84=', 'Content-Length': '0'} 2024-04-19T23:39:25.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c5859efe7902e19-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.811 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.811 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.811 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.811 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.811 DEBUG:boto:Method: GET 2024-04-19T23:39:25.812 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.812 DEBUG:boto:Data: 2024-04-19T23:39:25.812 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.812 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.812 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.812 DEBUG:boto:Params: {} 2024-04-19T23:39:25.813 DEBUG:boto:Token: None 2024-04-19T23:39:25.813 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.813 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008afd4c934b69b599-00662300ad-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.816 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.817 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.817 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.817 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.817 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.817 DEBUG:boto:Method: GET 2024-04-19T23:39:25.817 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.817 DEBUG:boto:Data: 2024-04-19T23:39:25.817 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.817 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.818 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.818 DEBUG:boto:Params: {} 2024-04-19T23:39:25.818 DEBUG:boto:Token: None 2024-04-19T23:39:25.818 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.818 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad92dace526daa6c-00662300ad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.821 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.821 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.821 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.821 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.822 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:25.822 DEBUG:boto:Data: 2024-04-19T23:39:25.822 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.822 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.822 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.822 DEBUG:boto:Params: {} 2024-04-19T23:39:25.822 DEBUG:boto:Token: None 2024-04-19T23:39:25.822 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj 2024-04-19T23:39:25.822 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg= 2024-04-19T23:39:25.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg=', 'Content-Length': '0'} 2024-04-19T23:39:25.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6b8f691f7a6c4d1-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.826 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.826 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.826 DEBUG:boto:Method: HEAD 2024-04-19T23:39:25.826 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:25.827 DEBUG:boto:Data: 2024-04-19T23:39:25.827 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.827 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.827 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.827 DEBUG:boto:Params: {} 2024-04-19T23:39:25.827 DEBUG:boto:Token: None 2024-04-19T23:39:25.827 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj 2024-04-19T23:39:25.827 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg= 2024-04-19T23:39:25.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3cd3TMgleHxuqFFJNlae3o60GGg=', 'Content-Length': '0'} 2024-04-19T23:39:25.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064ae090428ed56ab-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:25.831 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.831 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.831 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.831 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.832 DEBUG:boto:Method: GET 2024-04-19T23:39:25.832 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.832 DEBUG:boto:Data: 2024-04-19T23:39:25.832 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.832 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.832 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.832 DEBUG:boto:Params: {} 2024-04-19T23:39:25.832 DEBUG:boto:Token: None 2024-04-19T23:39:25.832 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.832 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak= 2024-04-19T23:39:25.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak=', 'Content-Length': '0'} 2024-04-19T23:39:25.834 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032797984c04ce684-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.882 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.883 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.883 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.883 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.883 DEBUG:boto:Method: GET 2024-04-19T23:39:25.884 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.884 DEBUG:boto:Data: 2024-04-19T23:39:25.884 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.884 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.884 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.884 DEBUG:boto:Params: {} 2024-04-19T23:39:25.885 DEBUG:boto:Token: None 2024-04-19T23:39:25.885 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:25.885 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak= 2024-04-19T23:39:25.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a8cOXHiuKXbf7hT8NesXX3k8Wak=', 'Content-Length': '0'} 2024-04-19T23:39:25.888 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000624018a03322ab45-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.934 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.935 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.935 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.935 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.935 DEBUG:boto:Method: GET 2024-04-19T23:39:25.935 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.935 DEBUG:boto:Data: 2024-04-19T23:39:25.935 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.936 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.936 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.936 DEBUG:boto:Params: {} 2024-04-19T23:39:25.936 DEBUG:boto:Token: None 2024-04-19T23:39:25.936 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.936 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7e3c9f5f84ed92-00662300ad-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.982 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.983 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:25.983 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:25.983 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.983 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:25.983 DEBUG:boto:Method: GET 2024-04-19T23:39:25.983 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:25.984 DEBUG:boto:Data: 2024-04-19T23:39:25.984 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.984 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.984 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.984 DEBUG:boto:Params: {} 2024-04-19T23:39:25.984 DEBUG:boto:Token: None 2024-04-19T23:39:25.984 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/ 2024-04-19T23:39:25.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM= 2024-04-19T23:39:25.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jTTEIodfQnpqGIXo1zKqUUrGKM=', 'Content-Length': '0'} 2024-04-19T23:39:25.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f9f4c74482c6b52-00662300ad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.987 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:25.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:25.988 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.988 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.988 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:25.988 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:25.988 DEBUG:boto:Method: GET 2024-04-19T23:39:25.988 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:25.989 DEBUG:boto:Data: 2024-04-19T23:39:25.989 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.989 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:25.989 DEBUG:boto:Port: 8001 2024-04-19T23:39:25.989 DEBUG:boto:Params: {} 2024-04-19T23:39:25.989 DEBUG:boto:Token: None 2024-04-19T23:39:25.989 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:25.989 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QKadQ/Rv2r9iHlpRZAsT+gu6TZE= 2024-04-19T23:39:25.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QKadQ/Rv2r9iHlpRZAsT+gu6TZE=', 'Content-Length': '0'} 2024-04-19T23:39:25.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000547613d8564ae350-00662300ad-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:25.992 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:25.992 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:25.992 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:25.992 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:25.992 DEBUG:boto:Method: GET 2024-04-19T23:39:25.992 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:25.992 DEBUG:boto:Data: 2024-04-19T23:39:25.992 DEBUG:boto:Headers: {} 2024-04-19T23:39:25.992 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:25.992 DEBUG:boto:Port: 8000 2024-04-19T23:39:25.993 DEBUG:boto:Params: {} 2024-04-19T23:39:25.993 DEBUG:boto:Token: None 2024-04-19T23:39:25.993 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:25 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:25.993 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QKadQ/Rv2r9iHlpRZAsT+gu6TZE= 2024-04-19T23:39:25.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:25 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QKadQ/Rv2r9iHlpRZAsT+gu6TZE=', 'Content-Length': '0'} 2024-04-19T23:39:25.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059be9bf57ae9181d-00662300ad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.038 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.039 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.039 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.039 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.039 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.039 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.039 DEBUG:boto:Data: 2024-04-19T23:39:26.039 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.039 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.039 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.039 DEBUG:boto:Params: {} 2024-04-19T23:39:26.040 DEBUG:boto:Token: None 2024-04-19T23:39:26.040 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.040 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aziTO02j8SAMHZ+MLmsjX0QcNpw= 2024-04-19T23:39:26.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aziTO02j8SAMHZ+MLmsjX0QcNpw=', 'Content-Length': '0'} 2024-04-19T23:39:26.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063659acdb9979619-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.042 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.043 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.043 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.043 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.043 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.043 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.043 DEBUG:boto:Data: 2024-04-19T23:39:26.043 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.043 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.043 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.044 DEBUG:boto:Params: {} 2024-04-19T23:39:26.044 DEBUG:boto:Token: None 2024-04-19T23:39:26.044 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.044 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aziTO02j8SAMHZ+MLmsjX0QcNpw= 2024-04-19T23:39:26.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aziTO02j8SAMHZ+MLmsjX0QcNpw=', 'Content-Length': '0'} 2024-04-19T23:39:26.046 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000930cffa228942d34-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:26.047 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.047 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.047 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.047 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.047 DEBUG:boto:Method: GET 2024-04-19T23:39:26.047 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.047 DEBUG:boto:Data: 2024-04-19T23:39:26.047 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.047 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.047 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.047 DEBUG:boto:Params: {} 2024-04-19T23:39:26.048 DEBUG:boto:Token: None 2024-04-19T23:39:26.048 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.048 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4pPR4MBOEgk+tbh/pLorTmODdZg= 2024-04-19T23:39:26.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4pPR4MBOEgk+tbh/pLorTmODdZg=', 'Content-Length': '0'} 2024-04-19T23:39:26.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdb7cb33841f7ae3-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.090 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.091 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.091 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.091 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.091 DEBUG:boto:Method: GET 2024-04-19T23:39:26.091 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.091 DEBUG:boto:Data: 2024-04-19T23:39:26.091 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.091 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.091 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.091 DEBUG:boto:Params: {} 2024-04-19T23:39:26.091 DEBUG:boto:Token: None 2024-04-19T23:39:26.092 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:26.092 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4pPR4MBOEgk+tbh/pLorTmODdZg= 2024-04-19T23:39:26.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4pPR4MBOEgk+tbh/pLorTmODdZg=', 'Content-Length': '0'} 2024-04-19T23:39:26.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb55a24418cd14b6-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.094 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.094 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.094 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.094 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.094 DEBUG:boto:Method: GET 2024-04-19T23:39:26.094 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.095 DEBUG:boto:Data: 2024-04-19T23:39:26.095 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.095 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.095 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.095 DEBUG:boto:Params: {} 2024-04-19T23:39:26.095 DEBUG:boto:Token: None 2024-04-19T23:39:26.095 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.095 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b11b44f6a62ed59b-00662300ae-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.098 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.099 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.099 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.099 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.099 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.099 DEBUG:boto:Method: GET 2024-04-19T23:39:26.099 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.099 DEBUG:boto:Data: 2024-04-19T23:39:26.099 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.099 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.100 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.100 DEBUG:boto:Params: {} 2024-04-19T23:39:26.100 DEBUG:boto:Token: None 2024-04-19T23:39:26.100 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.100 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001456d74ed1480059-00662300ae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.103 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.103 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.103 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.104 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.104 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:26.104 DEBUG:boto:Data: 2024-04-19T23:39:26.104 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.104 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.104 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.104 DEBUG:boto:Params: {} 2024-04-19T23:39:26.104 DEBUG:boto:Token: None 2024-04-19T23:39:26.104 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj 2024-04-19T23:39:26.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4= 2024-04-19T23:39:26.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4=', 'Content-Length': '0'} 2024-04-19T23:39:26.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d652033b6b5fce06-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.108 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.108 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.109 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.109 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:26.109 DEBUG:boto:Data: 2024-04-19T23:39:26.109 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.109 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.109 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.109 DEBUG:boto:Params: {} 2024-04-19T23:39:26.109 DEBUG:boto:Token: None 2024-04-19T23:39:26.109 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj 2024-04-19T23:39:26.109 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4= 2024-04-19T23:39:26.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4=', 'Content-Length': '0'} 2024-04-19T23:39:26.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073bbbc3a8598175d-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:26.113 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.113 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.113 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.113 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.113 DEBUG:boto:Method: GET 2024-04-19T23:39:26.113 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.113 DEBUG:boto:Data: 2024-04-19T23:39:26.113 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.114 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.114 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.114 DEBUG:boto:Params: {} 2024-04-19T23:39:26.114 DEBUG:boto:Token: None 2024-04-19T23:39:26.114 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.114 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E= 2024-04-19T23:39:26.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E=', 'Content-Length': '0'} 2024-04-19T23:39:26.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c175e673c1050ed0-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.158 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.159 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.159 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.159 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.159 DEBUG:boto:Method: GET 2024-04-19T23:39:26.159 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.159 DEBUG:boto:Data: 2024-04-19T23:39:26.159 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.159 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.159 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.159 DEBUG:boto:Params: {} 2024-04-19T23:39:26.160 DEBUG:boto:Token: None 2024-04-19T23:39:26.160 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E= 2024-04-19T23:39:26.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E=', 'Content-Length': '0'} 2024-04-19T23:39:26.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a9b6fc6db21d386-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.202 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.203 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.203 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.203 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.203 DEBUG:boto:Method: GET 2024-04-19T23:39:26.203 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.203 DEBUG:boto:Data: 2024-04-19T23:39:26.203 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.203 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.203 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.203 DEBUG:boto:Params: {} 2024-04-19T23:39:26.204 DEBUG:boto:Token: None 2024-04-19T23:39:26.204 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.204 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f1942cdea9a155-00662300ae-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.207 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.207 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.207 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.208 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.208 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.208 DEBUG:boto:Method: GET 2024-04-19T23:39:26.208 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.208 DEBUG:boto:Data: 2024-04-19T23:39:26.208 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.208 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.208 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.208 DEBUG:boto:Params: {} 2024-04-19T23:39:26.208 DEBUG:boto:Token: None 2024-04-19T23:39:26.208 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.209 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003359182f6f220432-00662300ae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.258 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:26.259 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.259 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.259 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.259 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.260 DEBUG:boto:Method: GET 2024-04-19T23:39:26.260 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.260 DEBUG:boto:Data: 2024-04-19T23:39:26.260 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.260 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.260 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.260 DEBUG:boto:Params: {} 2024-04-19T23:39:26.260 DEBUG:boto:Token: None 2024-04-19T23:39:26.260 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.261 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fqhY/hC2LUyMY1yaYkbWtYJbjoY= 2024-04-19T23:39:26.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fqhY/hC2LUyMY1yaYkbWtYJbjoY=', 'Content-Length': '0'} 2024-04-19T23:39:26.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023f89215989be2e8-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.263 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.263 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.263 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.263 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.263 DEBUG:boto:Method: GET 2024-04-19T23:39:26.264 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.264 DEBUG:boto:Data: 2024-04-19T23:39:26.264 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.264 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.264 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.264 DEBUG:boto:Params: {} 2024-04-19T23:39:26.264 DEBUG:boto:Token: None 2024-04-19T23:39:26.264 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.264 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fqhY/hC2LUyMY1yaYkbWtYJbjoY= 2024-04-19T23:39:26.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fqhY/hC2LUyMY1yaYkbWtYJbjoY=', 'Content-Length': '0'} 2024-04-19T23:39:26.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e965c8f08d42e88f-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.267 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.267 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.267 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.267 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.267 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.267 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.268 DEBUG:boto:Data: 2024-04-19T23:39:26.268 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.268 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.268 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.268 DEBUG:boto:Params: {} 2024-04-19T23:39:26.268 DEBUG:boto:Token: None 2024-04-19T23:39:26.268 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.268 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FQOr/RCQW5ZT4CSPv5vCnlDpvZg= 2024-04-19T23:39:26.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FQOr/RCQW5ZT4CSPv5vCnlDpvZg=', 'Content-Length': '0'} 2024-04-19T23:39:26.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069672ed944ce4a15-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.270 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.271 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.271 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.271 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.271 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.271 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.271 DEBUG:boto:Data: 2024-04-19T23:39:26.271 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.271 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.271 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.271 DEBUG:boto:Params: {} 2024-04-19T23:39:26.272 DEBUG:boto:Token: None 2024-04-19T23:39:26.272 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.272 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FQOr/RCQW5ZT4CSPv5vCnlDpvZg= 2024-04-19T23:39:26.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FQOr/RCQW5ZT4CSPv5vCnlDpvZg=', 'Content-Length': '0'} 2024-04-19T23:39:26.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d044c40f644a362-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:26.274 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.274 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.275 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.275 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.275 DEBUG:boto:Method: GET 2024-04-19T23:39:26.275 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.275 DEBUG:boto:Data: 2024-04-19T23:39:26.275 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.275 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.275 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.275 DEBUG:boto:Params: {} 2024-04-19T23:39:26.275 DEBUG:boto:Token: None 2024-04-19T23:39:26.275 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.276 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fqhY/hC2LUyMY1yaYkbWtYJbjoY= 2024-04-19T23:39:26.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fqhY/hC2LUyMY1yaYkbWtYJbjoY=', 'Content-Length': '0'} 2024-04-19T23:39:26.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8d9fd00db7e249f-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.318 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.319 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.319 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.319 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.319 DEBUG:boto:Method: GET 2024-04-19T23:39:26.319 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.319 DEBUG:boto:Data: 2024-04-19T23:39:26.319 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.319 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.319 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.319 DEBUG:boto:Params: {} 2024-04-19T23:39:26.320 DEBUG:boto:Token: None 2024-04-19T23:39:26.320 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:26.320 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fqhY/hC2LUyMY1yaYkbWtYJbjoY= 2024-04-19T23:39:26.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fqhY/hC2LUyMY1yaYkbWtYJbjoY=', 'Content-Length': '0'} 2024-04-19T23:39:26.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000519edf82644dbd8a-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.362 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.363 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.363 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.363 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.363 DEBUG:boto:Method: GET 2024-04-19T23:39:26.363 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.363 DEBUG:boto:Data: 2024-04-19T23:39:26.363 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.363 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.363 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.363 DEBUG:boto:Params: {} 2024-04-19T23:39:26.364 DEBUG:boto:Token: None 2024-04-19T23:39:26.364 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.364 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba95054e30cd80c4-00662300ae-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.410 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.411 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.411 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.411 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.411 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.411 DEBUG:boto:Method: GET 2024-04-19T23:39:26.411 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.412 DEBUG:boto:Data: 2024-04-19T23:39:26.412 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.412 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.412 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.412 DEBUG:boto:Params: {} 2024-04-19T23:39:26.412 DEBUG:boto:Token: None 2024-04-19T23:39:26.412 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.412 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a3c3da368f6e582-00662300ae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.458 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.459 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.459 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.459 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.459 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:26.459 DEBUG:boto:Data: 2024-04-19T23:39:26.459 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.459 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.459 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.459 DEBUG:boto:Params: {} 2024-04-19T23:39:26.460 DEBUG:boto:Token: None 2024-04-19T23:39:26.460 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj 2024-04-19T23:39:26.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4= 2024-04-19T23:39:26.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4=', 'Content-Length': '0'} 2024-04-19T23:39:26.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abe3bf4bbabbabd3-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.464 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.464 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.464 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.464 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:26.464 DEBUG:boto:Data: 2024-04-19T23:39:26.464 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.465 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.465 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.465 DEBUG:boto:Params: {} 2024-04-19T23:39:26.465 DEBUG:boto:Token: None 2024-04-19T23:39:26.465 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj 2024-04-19T23:39:26.465 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4= 2024-04-19T23:39:26.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4=', 'Content-Length': '0'} 2024-04-19T23:39:26.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db5d6f86b917c386-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:26.469 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.469 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.469 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.470 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.470 DEBUG:boto:Method: GET 2024-04-19T23:39:26.470 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.470 DEBUG:boto:Data: 2024-04-19T23:39:26.470 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.470 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.470 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.470 DEBUG:boto:Params: {} 2024-04-19T23:39:26.470 DEBUG:boto:Token: None 2024-04-19T23:39:26.470 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.471 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E= 2024-04-19T23:39:26.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E=', 'Content-Length': '0'} 2024-04-19T23:39:26.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f81aa18b587c142-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.514 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.514 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.515 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.515 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.515 DEBUG:boto:Method: GET 2024-04-19T23:39:26.515 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.515 DEBUG:boto:Data: 2024-04-19T23:39:26.515 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.515 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.515 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.515 DEBUG:boto:Params: {} 2024-04-19T23:39:26.515 DEBUG:boto:Token: None 2024-04-19T23:39:26.515 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.516 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E= 2024-04-19T23:39:26.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E=', 'Content-Length': '0'} 2024-04-19T23:39:26.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd1adb6ce8100a43-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.558 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.559 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.559 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.559 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.559 DEBUG:boto:Method: GET 2024-04-19T23:39:26.559 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.559 DEBUG:boto:Data: 2024-04-19T23:39:26.559 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.559 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.559 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.559 DEBUG:boto:Params: {} 2024-04-19T23:39:26.560 DEBUG:boto:Token: None 2024-04-19T23:39:26.560 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085d186a16677a257-00662300ae-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.610 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.611 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.612 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.612 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.612 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.612 DEBUG:boto:Method: GET 2024-04-19T23:39:26.612 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.612 DEBUG:boto:Data: 2024-04-19T23:39:26.613 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.613 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.613 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.613 DEBUG:boto:Params: {} 2024-04-19T23:39:26.613 DEBUG:boto:Token: None 2024-04-19T23:39:26.614 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.614 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d6f003c0f35c94d-00662300ae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.617 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:26.618 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.618 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.618 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.618 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.618 DEBUG:boto:Method: GET 2024-04-19T23:39:26.619 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.619 DEBUG:boto:Data: 2024-04-19T23:39:26.619 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.619 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.619 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.619 DEBUG:boto:Params: {} 2024-04-19T23:39:26.619 DEBUG:boto:Token: None 2024-04-19T23:39:26.619 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.619 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YqgjhTYUXGaFYk25XQwk6cxHnCA= 2024-04-19T23:39:26.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YqgjhTYUXGaFYk25XQwk6cxHnCA=', 'Content-Length': '0'} 2024-04-19T23:39:26.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050153136fe50b7a3-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.622 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.622 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.622 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.622 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.622 DEBUG:boto:Method: GET 2024-04-19T23:39:26.622 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.622 DEBUG:boto:Data: 2024-04-19T23:39:26.623 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.623 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.623 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.623 DEBUG:boto:Params: {} 2024-04-19T23:39:26.623 DEBUG:boto:Token: None 2024-04-19T23:39:26.623 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.623 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YqgjhTYUXGaFYk25XQwk6cxHnCA= 2024-04-19T23:39:26.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YqgjhTYUXGaFYk25XQwk6cxHnCA=', 'Content-Length': '0'} 2024-04-19T23:39:26.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7ddeb2e358190bb-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.626 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.626 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.626 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.626 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.626 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.626 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.626 DEBUG:boto:Data: 2024-04-19T23:39:26.627 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.627 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.627 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.627 DEBUG:boto:Params: {} 2024-04-19T23:39:26.627 DEBUG:boto:Token: None 2024-04-19T23:39:26.627 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.627 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k/cTnG0SEiEfTyiDvMs+iwUwsuc= 2024-04-19T23:39:26.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k/cTnG0SEiEfTyiDvMs+iwUwsuc=', 'Content-Length': '0'} 2024-04-19T23:39:26.629 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a66c6f45986558f-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.629 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.629 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.629 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.629 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.630 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.630 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.630 DEBUG:boto:Data: 2024-04-19T23:39:26.630 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.630 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.630 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.630 DEBUG:boto:Params: {} 2024-04-19T23:39:26.630 DEBUG:boto:Token: None 2024-04-19T23:39:26.630 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.631 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:k/cTnG0SEiEfTyiDvMs+iwUwsuc= 2024-04-19T23:39:26.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k/cTnG0SEiEfTyiDvMs+iwUwsuc=', 'Content-Length': '0'} 2024-04-19T23:39:26.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000329cb022cf6d8a54-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:26.633 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.633 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.633 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.633 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.633 DEBUG:boto:Method: GET 2024-04-19T23:39:26.633 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.633 DEBUG:boto:Data: 2024-04-19T23:39:26.633 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.633 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.634 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.634 DEBUG:boto:Params: {} 2024-04-19T23:39:26.634 DEBUG:boto:Token: None 2024-04-19T23:39:26.634 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.634 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YqgjhTYUXGaFYk25XQwk6cxHnCA= 2024-04-19T23:39:26.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YqgjhTYUXGaFYk25XQwk6cxHnCA=', 'Content-Length': '0'} 2024-04-19T23:39:26.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c64a8d47f7dc0ab2-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.678 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.678 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.679 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.679 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.679 DEBUG:boto:Method: GET 2024-04-19T23:39:26.679 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.679 DEBUG:boto:Data: 2024-04-19T23:39:26.679 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.679 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.679 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.679 DEBUG:boto:Params: {} 2024-04-19T23:39:26.679 DEBUG:boto:Token: None 2024-04-19T23:39:26.680 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:26.680 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YqgjhTYUXGaFYk25XQwk6cxHnCA= 2024-04-19T23:39:26.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YqgjhTYUXGaFYk25XQwk6cxHnCA=', 'Content-Length': '0'} 2024-04-19T23:39:26.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef3cdd55ccdec464-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.682 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.682 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.682 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.682 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.683 DEBUG:boto:Method: GET 2024-04-19T23:39:26.683 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.683 DEBUG:boto:Data: 2024-04-19T23:39:26.683 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.683 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.683 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.683 DEBUG:boto:Params: {} 2024-04-19T23:39:26.683 DEBUG:boto:Token: None 2024-04-19T23:39:26.683 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.683 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1e7f0257cf14e9a-00662300ae-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.686 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.687 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.687 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.687 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.687 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.687 DEBUG:boto:Method: GET 2024-04-19T23:39:26.688 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.688 DEBUG:boto:Data: 2024-04-19T23:39:26.688 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.688 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.688 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.688 DEBUG:boto:Params: {} 2024-04-19T23:39:26.688 DEBUG:boto:Token: None 2024-04-19T23:39:26.688 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.688 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ede981ad42dec92d-00662300ae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.691 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.691 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.692 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.692 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.692 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:26.692 DEBUG:boto:Data: 2024-04-19T23:39:26.692 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.692 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.692 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.692 DEBUG:boto:Params: {} 2024-04-19T23:39:26.692 DEBUG:boto:Token: None 2024-04-19T23:39:26.692 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj 2024-04-19T23:39:26.692 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4= 2024-04-19T23:39:26.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4=', 'Content-Length': '0'} 2024-04-19T23:39:26.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099ecf353bf04bbb6-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.696 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.696 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.696 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.696 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:26.696 DEBUG:boto:Data: 2024-04-19T23:39:26.696 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.696 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.696 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.697 DEBUG:boto:Params: {} 2024-04-19T23:39:26.697 DEBUG:boto:Token: None 2024-04-19T23:39:26.697 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj 2024-04-19T23:39:26.697 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4= 2024-04-19T23:39:26.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4=', 'Content-Length': '0'} 2024-04-19T23:39:26.700 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006d1fad06c0b7a6a-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:26.700 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.701 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.701 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.701 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.701 DEBUG:boto:Method: GET 2024-04-19T23:39:26.701 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.701 DEBUG:boto:Data: 2024-04-19T23:39:26.701 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.701 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.701 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.701 DEBUG:boto:Params: {} 2024-04-19T23:39:26.701 DEBUG:boto:Token: None 2024-04-19T23:39:26.702 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.702 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E= 2024-04-19T23:39:26.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E=', 'Content-Length': '0'} 2024-04-19T23:39:26.703 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d3939007ab67dbd-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.746 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.746 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.747 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.747 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.747 DEBUG:boto:Method: GET 2024-04-19T23:39:26.747 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.747 DEBUG:boto:Data: 2024-04-19T23:39:26.747 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.747 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.747 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.747 DEBUG:boto:Params: {} 2024-04-19T23:39:26.747 DEBUG:boto:Token: None 2024-04-19T23:39:26.748 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.748 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E= 2024-04-19T23:39:26.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E=', 'Content-Length': '0'} 2024-04-19T23:39:26.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b7ce5375b670cfb-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.794 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.794 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.795 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.795 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.795 DEBUG:boto:Method: GET 2024-04-19T23:39:26.795 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.795 DEBUG:boto:Data: 2024-04-19T23:39:26.795 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.795 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.795 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.795 DEBUG:boto:Params: {} 2024-04-19T23:39:26.795 DEBUG:boto:Token: None 2024-04-19T23:39:26.796 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.796 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000265edb24a18fb0b4-00662300ae-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.799 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.799 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.799 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.799 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.799 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.799 DEBUG:boto:Method: GET 2024-04-19T23:39:26.800 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.800 DEBUG:boto:Data: 2024-04-19T23:39:26.800 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.800 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.800 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.800 DEBUG:boto:Params: {} 2024-04-19T23:39:26.800 DEBUG:boto:Token: None 2024-04-19T23:39:26.800 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.800 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050f868048509b873-00662300ae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.803 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:26.804 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.804 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.804 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.804 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.804 DEBUG:boto:Method: GET 2024-04-19T23:39:26.804 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.804 DEBUG:boto:Data: 2024-04-19T23:39:26.805 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.805 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.805 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.805 DEBUG:boto:Params: {} 2024-04-19T23:39:26.805 DEBUG:boto:Token: None 2024-04-19T23:39:26.805 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.805 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z9gdMPim3CXEuHbYbD4lX0dR7Yw= 2024-04-19T23:39:26.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z9gdMPim3CXEuHbYbD4lX0dR7Yw=', 'Content-Length': '0'} 2024-04-19T23:39:26.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e082e831e2c8608-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.808 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.808 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.808 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.808 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.808 DEBUG:boto:Method: GET 2024-04-19T23:39:26.808 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.808 DEBUG:boto:Data: 2024-04-19T23:39:26.808 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.809 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.809 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.809 DEBUG:boto:Params: {} 2024-04-19T23:39:26.809 DEBUG:boto:Token: None 2024-04-19T23:39:26.809 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.809 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z9gdMPim3CXEuHbYbD4lX0dR7Yw= 2024-04-19T23:39:26.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z9gdMPim3CXEuHbYbD4lX0dR7Yw=', 'Content-Length': '0'} 2024-04-19T23:39:26.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7180f79fce35462-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.855 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.855 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.855 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.855 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.855 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.855 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.856 DEBUG:boto:Data: 2024-04-19T23:39:26.856 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.856 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.856 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.856 DEBUG:boto:Params: {} 2024-04-19T23:39:26.856 DEBUG:boto:Token: None 2024-04-19T23:39:26.856 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.856 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vifm4qWmPlBTLioLWO2g0ymMzgc= 2024-04-19T23:39:26.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vifm4qWmPlBTLioLWO2g0ymMzgc=', 'Content-Length': '0'} 2024-04-19T23:39:26.859 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001be90bedd2789b13-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.859 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.859 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.859 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.859 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.859 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.860 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.860 DEBUG:boto:Data: 2024-04-19T23:39:26.860 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.860 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.860 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.860 DEBUG:boto:Params: {} 2024-04-19T23:39:26.860 DEBUG:boto:Token: None 2024-04-19T23:39:26.860 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.861 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vifm4qWmPlBTLioLWO2g0ymMzgc= 2024-04-19T23:39:26.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vifm4qWmPlBTLioLWO2g0ymMzgc=', 'Content-Length': '0'} 2024-04-19T23:39:26.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000805640023bb1d5c2-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:26.863 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.863 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.863 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.864 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.864 DEBUG:boto:Method: GET 2024-04-19T23:39:26.864 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.864 DEBUG:boto:Data: 2024-04-19T23:39:26.864 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.864 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.864 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.864 DEBUG:boto:Params: {} 2024-04-19T23:39:26.864 DEBUG:boto:Token: None 2024-04-19T23:39:26.865 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.865 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z9gdMPim3CXEuHbYbD4lX0dR7Yw= 2024-04-19T23:39:26.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z9gdMPim3CXEuHbYbD4lX0dR7Yw=', 'Content-Length': '0'} 2024-04-19T23:39:26.867 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020db84f8547ffcaf-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.867 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.867 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.867 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.867 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.867 DEBUG:boto:Method: GET 2024-04-19T23:39:26.868 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.868 DEBUG:boto:Data: 2024-04-19T23:39:26.868 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.868 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.868 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.868 DEBUG:boto:Params: {} 2024-04-19T23:39:26.868 DEBUG:boto:Token: None 2024-04-19T23:39:26.868 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:26.869 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z9gdMPim3CXEuHbYbD4lX0dR7Yw= 2024-04-19T23:39:26.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z9gdMPim3CXEuHbYbD4lX0dR7Yw=', 'Content-Length': '0'} 2024-04-19T23:39:26.871 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a7d74a8baf74535-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.914 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.915 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.915 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.915 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.915 DEBUG:boto:Method: GET 2024-04-19T23:39:26.915 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.915 DEBUG:boto:Data: 2024-04-19T23:39:26.915 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.915 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.915 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.915 DEBUG:boto:Params: {} 2024-04-19T23:39:26.916 DEBUG:boto:Token: None 2024-04-19T23:39:26.916 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.916 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090ddc4d3dbd5c16b-00662300ae-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.919 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.919 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:26.920 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:26.920 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.920 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:26.920 DEBUG:boto:Method: GET 2024-04-19T23:39:26.920 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:26.920 DEBUG:boto:Data: 2024-04-19T23:39:26.920 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.920 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.920 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.920 DEBUG:boto:Params: {} 2024-04-19T23:39:26.920 DEBUG:boto:Token: None 2024-04-19T23:39:26.921 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/ 2024-04-19T23:39:26.921 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg= 2024-04-19T23:39:26.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:06sXmdU0CB0ilngkQ2k/om9McPg=', 'Content-Length': '0'} 2024-04-19T23:39:26.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a12e582bce867197-00662300ae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.924 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:26.924 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.925 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.925 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.925 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:26.925 DEBUG:boto:Data: 2024-04-19T23:39:26.925 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.925 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.925 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.925 DEBUG:boto:Params: {} 2024-04-19T23:39:26.925 DEBUG:boto:Token: None 2024-04-19T23:39:26.925 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj 2024-04-19T23:39:26.926 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4= 2024-04-19T23:39:26.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4=', 'Content-Length': '0'} 2024-04-19T23:39:26.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bc3c7d2f5da832f-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.929 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.929 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.929 DEBUG:boto:Method: HEAD 2024-04-19T23:39:26.929 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:26.929 DEBUG:boto:Data: 2024-04-19T23:39:26.929 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.930 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.930 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.930 DEBUG:boto:Params: {} 2024-04-19T23:39:26.930 DEBUG:boto:Token: None 2024-04-19T23:39:26.930 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj 2024-04-19T23:39:26.930 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4= 2024-04-19T23:39:26.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4D/v1aQsWV0+1cNYOq6w4C4I1m4=', 'Content-Length': '0'} 2024-04-19T23:39:26.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045c77b2fbfb43866-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:26.934 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.934 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.935 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.935 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.935 DEBUG:boto:Method: GET 2024-04-19T23:39:26.935 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.935 DEBUG:boto:Data: 2024-04-19T23:39:26.935 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.935 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:26.935 DEBUG:boto:Port: 8001 2024-04-19T23:39:26.935 DEBUG:boto:Params: {} 2024-04-19T23:39:26.935 DEBUG:boto:Token: None 2024-04-19T23:39:26.935 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.935 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E= 2024-04-19T23:39:26.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E=', 'Content-Length': '0'} 2024-04-19T23:39:26.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003dccb9119fe5086-00662300ae-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:26.982 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:26.983 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:26.983 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.983 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.983 DEBUG:boto:Method: GET 2024-04-19T23:39:26.983 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.983 DEBUG:boto:Data: 2024-04-19T23:39:26.983 DEBUG:boto:Headers: {} 2024-04-19T23:39:26.983 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:26.983 DEBUG:boto:Port: 8000 2024-04-19T23:39:26.983 DEBUG:boto:Params: {} 2024-04-19T23:39:26.984 DEBUG:boto:Token: None 2024-04-19T23:39:26.984 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:26 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:26.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E= 2024-04-19T23:39:26.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:26 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WTyTnD7OTCirDiTo8C8CxFt3P9E=', 'Content-Length': '0'} 2024-04-19T23:39:26.986 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6d6b1e5771b49b2-00662300ae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.026 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.027 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.027 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.027 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.027 DEBUG:boto:Method: GET 2024-04-19T23:39:27.027 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.027 DEBUG:boto:Data: 2024-04-19T23:39:27.027 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.027 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.027 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.027 DEBUG:boto:Params: {} 2024-04-19T23:39:27.028 DEBUG:boto:Token: None 2024-04-19T23:39:27.028 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.028 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068cee9e7ad82c72a-00662300af-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.031 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.032 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.032 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.032 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.032 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.032 DEBUG:boto:Method: GET 2024-04-19T23:39:27.032 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.032 DEBUG:boto:Data: 2024-04-19T23:39:27.032 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.032 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.032 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.032 DEBUG:boto:Params: {} 2024-04-19T23:39:27.033 DEBUG:boto:Token: None 2024-04-19T23:39:27.033 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.033 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4aa31f3f0c5a6bf-00662300af-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.078 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.079 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.079 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.079 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.079 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.079 DEBUG:boto:Method: GET 2024-04-19T23:39:27.079 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.080 DEBUG:boto:Data: 2024-04-19T23:39:27.080 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.080 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.080 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.080 DEBUG:boto:Params: {} 2024-04-19T23:39:27.080 DEBUG:boto:Token: None 2024-04-19T23:39:27.080 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.080 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tK7aR+jf1A4iCfpx48aRhSeS9dE= 2024-04-19T23:39:27.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tK7aR+jf1A4iCfpx48aRhSeS9dE=', 'Content-Length': '0'} 2024-04-19T23:39:27.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002270e7bd3b395a41-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.083 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.083 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.083 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.083 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.083 DEBUG:boto:Method: GET 2024-04-19T23:39:27.083 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.083 DEBUG:boto:Data: 2024-04-19T23:39:27.083 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.083 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.084 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.084 DEBUG:boto:Params: {} 2024-04-19T23:39:27.084 DEBUG:boto:Token: None 2024-04-19T23:39:27.084 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tK7aR+jf1A4iCfpx48aRhSeS9dE= 2024-04-19T23:39:27.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tK7aR+jf1A4iCfpx48aRhSeS9dE=', 'Content-Length': '0'} 2024-04-19T23:39:27.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cd80b26a428effb-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.086 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.086 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.086 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.087 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.087 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.087 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.087 DEBUG:boto:Data: 2024-04-19T23:39:27.087 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.087 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.087 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.087 DEBUG:boto:Params: {} 2024-04-19T23:39:27.087 DEBUG:boto:Token: None 2024-04-19T23:39:27.087 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.088 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zN1306Bcs904AOmIWKQt3DHB4mY= 2024-04-19T23:39:27.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zN1306Bcs904AOmIWKQt3DHB4mY=', 'Content-Length': '0'} 2024-04-19T23:39:27.089 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000069b10f1427e8116-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.090 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.090 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.090 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.090 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.090 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.090 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.090 DEBUG:boto:Data: 2024-04-19T23:39:27.090 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.091 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.091 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.091 DEBUG:boto:Params: {} 2024-04-19T23:39:27.091 DEBUG:boto:Token: None 2024-04-19T23:39:27.091 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.091 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zN1306Bcs904AOmIWKQt3DHB4mY= 2024-04-19T23:39:27.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zN1306Bcs904AOmIWKQt3DHB4mY=', 'Content-Length': '0'} 2024-04-19T23:39:27.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000866b8a1ae2c3d094-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.093 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.093 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.093 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.094 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.094 DEBUG:boto:Method: GET 2024-04-19T23:39:27.094 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.094 DEBUG:boto:Data: 2024-04-19T23:39:27.094 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.094 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.094 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.094 DEBUG:boto:Params: {} 2024-04-19T23:39:27.094 DEBUG:boto:Token: None 2024-04-19T23:39:27.094 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.095 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tK7aR+jf1A4iCfpx48aRhSeS9dE= 2024-04-19T23:39:27.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tK7aR+jf1A4iCfpx48aRhSeS9dE=', 'Content-Length': '0'} 2024-04-19T23:39:27.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1953c2772ef0162-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.097 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.097 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.097 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.097 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.097 DEBUG:boto:Method: GET 2024-04-19T23:39:27.098 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.098 DEBUG:boto:Data: 2024-04-19T23:39:27.098 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.098 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.098 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.098 DEBUG:boto:Params: {} 2024-04-19T23:39:27.098 DEBUG:boto:Token: None 2024-04-19T23:39:27.098 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:27.098 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tK7aR+jf1A4iCfpx48aRhSeS9dE= 2024-04-19T23:39:27.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tK7aR+jf1A4iCfpx48aRhSeS9dE=', 'Content-Length': '0'} 2024-04-19T23:39:27.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfe56df21f9855bf-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.142 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.143 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.143 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.143 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.143 DEBUG:boto:Method: GET 2024-04-19T23:39:27.143 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.143 DEBUG:boto:Data: 2024-04-19T23:39:27.143 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.143 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.143 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.143 DEBUG:boto:Params: {} 2024-04-19T23:39:27.144 DEBUG:boto:Token: None 2024-04-19T23:39:27.144 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.144 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a82ced619a783c60-00662300af-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.147 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.147 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.147 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.147 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.148 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.148 DEBUG:boto:Method: GET 2024-04-19T23:39:27.148 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.148 DEBUG:boto:Data: 2024-04-19T23:39:27.148 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.148 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.148 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.148 DEBUG:boto:Params: {} 2024-04-19T23:39:27.149 DEBUG:boto:Token: None 2024-04-19T23:39:27.149 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.149 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd992f3dd177c54d-00662300af-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.152 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.152 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.152 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.152 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.153 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:27.153 DEBUG:boto:Data: 2024-04-19T23:39:27.153 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.153 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.153 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.153 DEBUG:boto:Params: {} 2024-04-19T23:39:27.153 DEBUG:boto:Token: None 2024-04-19T23:39:27.153 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj 2024-04-19T23:39:27.153 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE= 2024-04-19T23:39:27.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE=', 'Content-Length': '0'} 2024-04-19T23:39:27.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000689657af2a55991b-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.157 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.157 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.157 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.157 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:27.157 DEBUG:boto:Data: 2024-04-19T23:39:27.157 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.157 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.158 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.158 DEBUG:boto:Params: {} 2024-04-19T23:39:27.158 DEBUG:boto:Token: None 2024-04-19T23:39:27.158 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj 2024-04-19T23:39:27.158 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE= 2024-04-19T23:39:27.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE=', 'Content-Length': '0'} 2024-04-19T23:39:27.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a4ac101ba5fdb23-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.162 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.162 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.162 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.162 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.162 DEBUG:boto:Method: GET 2024-04-19T23:39:27.162 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.162 DEBUG:boto:Data: 2024-04-19T23:39:27.163 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.163 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.163 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.163 DEBUG:boto:Params: {} 2024-04-19T23:39:27.163 DEBUG:boto:Token: None 2024-04-19T23:39:27.163 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.163 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE= 2024-04-19T23:39:27.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE=', 'Content-Length': '0'} 2024-04-19T23:39:27.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4a492d96eba85da-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.206 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.207 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.207 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.207 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.207 DEBUG:boto:Method: GET 2024-04-19T23:39:27.207 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.207 DEBUG:boto:Data: 2024-04-19T23:39:27.207 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.207 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.207 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.207 DEBUG:boto:Params: {} 2024-04-19T23:39:27.207 DEBUG:boto:Token: None 2024-04-19T23:39:27.208 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.208 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE= 2024-04-19T23:39:27.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE=', 'Content-Length': '0'} 2024-04-19T23:39:27.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007735bbd274a6c4de-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.250 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.251 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.251 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.251 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.251 DEBUG:boto:Method: GET 2024-04-19T23:39:27.251 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.251 DEBUG:boto:Data: 2024-04-19T23:39:27.251 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.251 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.251 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.251 DEBUG:boto:Params: {} 2024-04-19T23:39:27.252 DEBUG:boto:Token: None 2024-04-19T23:39:27.252 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.252 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048255afa785bcc82-00662300af-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.255 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.255 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.256 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.256 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.256 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.256 DEBUG:boto:Method: GET 2024-04-19T23:39:27.256 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.256 DEBUG:boto:Data: 2024-04-19T23:39:27.256 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.256 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.256 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.256 DEBUG:boto:Params: {} 2024-04-19T23:39:27.257 DEBUG:boto:Token: None 2024-04-19T23:39:27.257 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.257 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003334f96d68f3d9f9-00662300af-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.261 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.261 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.262 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.262 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.262 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.262 DEBUG:boto:Method: GET 2024-04-19T23:39:27.262 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.262 DEBUG:boto:Data: 2024-04-19T23:39:27.262 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.262 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.262 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.262 DEBUG:boto:Params: {} 2024-04-19T23:39:27.262 DEBUG:boto:Token: None 2024-04-19T23:39:27.263 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.263 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jZq9o/di+B97azvRrRdw9QIpXdE= 2024-04-19T23:39:27.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jZq9o/di+B97azvRrRdw9QIpXdE=', 'Content-Length': '0'} 2024-04-19T23:39:27.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1a13435d6537d00-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.265 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.265 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.265 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.265 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.265 DEBUG:boto:Method: GET 2024-04-19T23:39:27.265 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.265 DEBUG:boto:Data: 2024-04-19T23:39:27.266 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.266 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.266 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.266 DEBUG:boto:Params: {} 2024-04-19T23:39:27.266 DEBUG:boto:Token: None 2024-04-19T23:39:27.266 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.266 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jZq9o/di+B97azvRrRdw9QIpXdE= 2024-04-19T23:39:27.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jZq9o/di+B97azvRrRdw9QIpXdE=', 'Content-Length': '0'} 2024-04-19T23:39:27.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8effe308475ba87-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.269 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.269 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.269 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.269 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.269 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.269 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.269 DEBUG:boto:Data: 2024-04-19T23:39:27.269 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.269 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.270 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.270 DEBUG:boto:Params: {} 2024-04-19T23:39:27.270 DEBUG:boto:Token: None 2024-04-19T23:39:27.270 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.270 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zt7/g/K+IT+X/bgG0zQTdlQTFvA= 2024-04-19T23:39:27.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zt7/g/K+IT+X/bgG0zQTdlQTFvA=', 'Content-Length': '0'} 2024-04-19T23:39:27.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eedad0cd041d0965-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.272 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.272 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.272 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.273 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.273 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.273 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.273 DEBUG:boto:Data: 2024-04-19T23:39:27.273 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.273 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.273 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.273 DEBUG:boto:Params: {} 2024-04-19T23:39:27.273 DEBUG:boto:Token: None 2024-04-19T23:39:27.273 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.274 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Zt7/g/K+IT+X/bgG0zQTdlQTFvA= 2024-04-19T23:39:27.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zt7/g/K+IT+X/bgG0zQTdlQTFvA=', 'Content-Length': '0'} 2024-04-19T23:39:27.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e231156500269858-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.275 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.276 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.276 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.276 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.276 DEBUG:boto:Method: GET 2024-04-19T23:39:27.276 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.276 DEBUG:boto:Data: 2024-04-19T23:39:27.276 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.276 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.276 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.276 DEBUG:boto:Params: {} 2024-04-19T23:39:27.277 DEBUG:boto:Token: None 2024-04-19T23:39:27.277 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.277 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jZq9o/di+B97azvRrRdw9QIpXdE= 2024-04-19T23:39:27.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jZq9o/di+B97azvRrRdw9QIpXdE=', 'Content-Length': '0'} 2024-04-19T23:39:27.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cb9d75dc6ee6ed2-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.323 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.323 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.323 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.323 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.323 DEBUG:boto:Method: GET 2024-04-19T23:39:27.323 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.323 DEBUG:boto:Data: 2024-04-19T23:39:27.323 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.324 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.324 DEBUG:boto:Params: {} 2024-04-19T23:39:27.324 DEBUG:boto:Token: None 2024-04-19T23:39:27.324 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:27.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jZq9o/di+B97azvRrRdw9QIpXdE= 2024-04-19T23:39:27.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jZq9o/di+B97azvRrRdw9QIpXdE=', 'Content-Length': '0'} 2024-04-19T23:39:27.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004850a55b2beeae13-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.326 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.327 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.327 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.327 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.327 DEBUG:boto:Method: GET 2024-04-19T23:39:27.327 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.327 DEBUG:boto:Data: 2024-04-19T23:39:27.327 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.327 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.327 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.327 DEBUG:boto:Params: {} 2024-04-19T23:39:27.327 DEBUG:boto:Token: None 2024-04-19T23:39:27.328 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.328 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5b27fe93d10f218-00662300af-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.331 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.331 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.331 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.331 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.331 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.332 DEBUG:boto:Method: GET 2024-04-19T23:39:27.332 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.332 DEBUG:boto:Data: 2024-04-19T23:39:27.332 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.332 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.332 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.332 DEBUG:boto:Params: {} 2024-04-19T23:39:27.332 DEBUG:boto:Token: None 2024-04-19T23:39:27.332 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f7b580c17b7f62f-00662300af-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.335 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.336 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.336 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.336 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.336 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:27.336 DEBUG:boto:Data: 2024-04-19T23:39:27.336 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.336 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.336 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.336 DEBUG:boto:Params: {} 2024-04-19T23:39:27.337 DEBUG:boto:Token: None 2024-04-19T23:39:27.337 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj 2024-04-19T23:39:27.337 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE= 2024-04-19T23:39:27.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE=', 'Content-Length': '0'} 2024-04-19T23:39:27.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a5ece66c306cbe2-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.341 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.341 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.341 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.341 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:27.341 DEBUG:boto:Data: 2024-04-19T23:39:27.341 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.341 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.341 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.341 DEBUG:boto:Params: {} 2024-04-19T23:39:27.342 DEBUG:boto:Token: None 2024-04-19T23:39:27.342 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj 2024-04-19T23:39:27.342 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE= 2024-04-19T23:39:27.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE=', 'Content-Length': '0'} 2024-04-19T23:39:27.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a2c8bff7c84e4e9-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.345 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.346 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.346 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.346 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.346 DEBUG:boto:Method: GET 2024-04-19T23:39:27.346 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.346 DEBUG:boto:Data: 2024-04-19T23:39:27.346 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.346 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.346 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.346 DEBUG:boto:Params: {} 2024-04-19T23:39:27.347 DEBUG:boto:Token: None 2024-04-19T23:39:27.347 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.347 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE= 2024-04-19T23:39:27.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE=', 'Content-Length': '0'} 2024-04-19T23:39:27.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d33b86d185359637-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.390 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.391 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.391 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.391 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.391 DEBUG:boto:Method: GET 2024-04-19T23:39:27.391 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.391 DEBUG:boto:Data: 2024-04-19T23:39:27.391 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.391 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.391 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.391 DEBUG:boto:Params: {} 2024-04-19T23:39:27.392 DEBUG:boto:Token: None 2024-04-19T23:39:27.392 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.392 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE= 2024-04-19T23:39:27.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE=', 'Content-Length': '0'} 2024-04-19T23:39:27.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4ba0d077275613c-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.434 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.435 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.435 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.435 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.435 DEBUG:boto:Method: GET 2024-04-19T23:39:27.435 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.435 DEBUG:boto:Data: 2024-04-19T23:39:27.435 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.435 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.435 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.435 DEBUG:boto:Params: {} 2024-04-19T23:39:27.436 DEBUG:boto:Token: None 2024-04-19T23:39:27.436 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.436 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092049bf61126586d-00662300af-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.439 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.440 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.440 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.440 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.440 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.440 DEBUG:boto:Method: GET 2024-04-19T23:39:27.440 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.440 DEBUG:boto:Data: 2024-04-19T23:39:27.440 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.440 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.440 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.440 DEBUG:boto:Params: {} 2024-04-19T23:39:27.441 DEBUG:boto:Token: None 2024-04-19T23:39:27.441 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.441 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e16737f0f99bcff-00662300af-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.444 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.444 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.444 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.444 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.445 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.445 DEBUG:boto:Method: GET 2024-04-19T23:39:27.445 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.445 DEBUG:boto:Data: 2024-04-19T23:39:27.445 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.445 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.445 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.445 DEBUG:boto:Params: {} 2024-04-19T23:39:27.445 DEBUG:boto:Token: None 2024-04-19T23:39:27.445 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.446 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ZAMnZRWvbOrEmrcVGWGuOVWO08= 2024-04-19T23:39:27.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ZAMnZRWvbOrEmrcVGWGuOVWO08=', 'Content-Length': '0'} 2024-04-19T23:39:27.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069f87c45496cefe2-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.448 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.448 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.448 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.448 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.449 DEBUG:boto:Method: GET 2024-04-19T23:39:27.449 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.449 DEBUG:boto:Data: 2024-04-19T23:39:27.449 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.449 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.449 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.449 DEBUG:boto:Params: {} 2024-04-19T23:39:27.449 DEBUG:boto:Token: None 2024-04-19T23:39:27.449 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.449 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ZAMnZRWvbOrEmrcVGWGuOVWO08= 2024-04-19T23:39:27.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ZAMnZRWvbOrEmrcVGWGuOVWO08=', 'Content-Length': '0'} 2024-04-19T23:39:27.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004145f26c2946a0c4-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.452 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.452 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.452 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.452 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.452 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.452 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.452 DEBUG:boto:Data: 2024-04-19T23:39:27.452 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.452 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.453 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.453 DEBUG:boto:Params: {} 2024-04-19T23:39:27.453 DEBUG:boto:Token: None 2024-04-19T23:39:27.453 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.453 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SBeZI9cl2HwC5oTolN0qmB3BMtg= 2024-04-19T23:39:27.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SBeZI9cl2HwC5oTolN0qmB3BMtg=', 'Content-Length': '0'} 2024-04-19T23:39:27.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ed9eada8e2d6c9b-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.455 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.455 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.455 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.455 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.456 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.456 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.456 DEBUG:boto:Data: 2024-04-19T23:39:27.456 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.456 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.456 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.456 DEBUG:boto:Params: {} 2024-04-19T23:39:27.456 DEBUG:boto:Token: None 2024-04-19T23:39:27.456 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.456 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SBeZI9cl2HwC5oTolN0qmB3BMtg= 2024-04-19T23:39:27.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SBeZI9cl2HwC5oTolN0qmB3BMtg=', 'Content-Length': '0'} 2024-04-19T23:39:27.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0729c958744757b-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.459 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.459 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.459 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.459 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.459 DEBUG:boto:Method: GET 2024-04-19T23:39:27.459 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.459 DEBUG:boto:Data: 2024-04-19T23:39:27.459 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.459 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.460 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.460 DEBUG:boto:Params: {} 2024-04-19T23:39:27.460 DEBUG:boto:Token: None 2024-04-19T23:39:27.460 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ZAMnZRWvbOrEmrcVGWGuOVWO08= 2024-04-19T23:39:27.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ZAMnZRWvbOrEmrcVGWGuOVWO08=', 'Content-Length': '0'} 2024-04-19T23:39:27.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084c31e7538ff008a-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.502 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.503 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.503 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.503 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.503 DEBUG:boto:Method: GET 2024-04-19T23:39:27.503 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.503 DEBUG:boto:Data: 2024-04-19T23:39:27.503 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.503 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.503 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.503 DEBUG:boto:Params: {} 2024-04-19T23:39:27.504 DEBUG:boto:Token: None 2024-04-19T23:39:27.504 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:27.504 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ZAMnZRWvbOrEmrcVGWGuOVWO08= 2024-04-19T23:39:27.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ZAMnZRWvbOrEmrcVGWGuOVWO08=', 'Content-Length': '0'} 2024-04-19T23:39:27.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b88deccff94b2be-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.546 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.547 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.547 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.547 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.547 DEBUG:boto:Method: GET 2024-04-19T23:39:27.547 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.547 DEBUG:boto:Data: 2024-04-19T23:39:27.547 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.547 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.547 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.547 DEBUG:boto:Params: {} 2024-04-19T23:39:27.548 DEBUG:boto:Token: None 2024-04-19T23:39:27.548 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.548 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b399220546c4310-00662300af-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.598 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.599 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.599 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.599 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.599 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.599 DEBUG:boto:Method: GET 2024-04-19T23:39:27.600 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.600 DEBUG:boto:Data: 2024-04-19T23:39:27.600 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.600 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.600 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.600 DEBUG:boto:Params: {} 2024-04-19T23:39:27.600 DEBUG:boto:Token: None 2024-04-19T23:39:27.600 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.600 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e61c62a88ab47c60-00662300af-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.604 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.604 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.604 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.605 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.605 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:27.605 DEBUG:boto:Data: 2024-04-19T23:39:27.605 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.605 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.605 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.605 DEBUG:boto:Params: {} 2024-04-19T23:39:27.605 DEBUG:boto:Token: None 2024-04-19T23:39:27.605 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj 2024-04-19T23:39:27.605 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE= 2024-04-19T23:39:27.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE=', 'Content-Length': '0'} 2024-04-19T23:39:27.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e29c3dd374e6e1f-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.609 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.609 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.609 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.610 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:27.610 DEBUG:boto:Data: 2024-04-19T23:39:27.610 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.610 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.610 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.610 DEBUG:boto:Params: {} 2024-04-19T23:39:27.610 DEBUG:boto:Token: None 2024-04-19T23:39:27.610 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj 2024-04-19T23:39:27.610 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE= 2024-04-19T23:39:27.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE=', 'Content-Length': '0'} 2024-04-19T23:39:27.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072ba521d01e2514e-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.614 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.614 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.614 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.614 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.615 DEBUG:boto:Method: GET 2024-04-19T23:39:27.615 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.615 DEBUG:boto:Data: 2024-04-19T23:39:27.615 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.615 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.615 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.615 DEBUG:boto:Params: {} 2024-04-19T23:39:27.615 DEBUG:boto:Token: None 2024-04-19T23:39:27.615 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.615 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE= 2024-04-19T23:39:27.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE=', 'Content-Length': '0'} 2024-04-19T23:39:27.618 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cfa9c22fe5860ac-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.658 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.659 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.659 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.659 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.659 DEBUG:boto:Method: GET 2024-04-19T23:39:27.659 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.659 DEBUG:boto:Data: 2024-04-19T23:39:27.659 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.659 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.659 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.659 DEBUG:boto:Params: {} 2024-04-19T23:39:27.660 DEBUG:boto:Token: None 2024-04-19T23:39:27.660 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE= 2024-04-19T23:39:27.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE=', 'Content-Length': '0'} 2024-04-19T23:39:27.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb171ac76765a7db-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.702 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.703 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.703 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.703 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.703 DEBUG:boto:Method: GET 2024-04-19T23:39:27.703 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.703 DEBUG:boto:Data: 2024-04-19T23:39:27.703 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.703 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.703 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.703 DEBUG:boto:Params: {} 2024-04-19T23:39:27.704 DEBUG:boto:Token: None 2024-04-19T23:39:27.704 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e98c3dbf462657e7-00662300af-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.754 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.755 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.755 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.755 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.755 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.755 DEBUG:boto:Method: GET 2024-04-19T23:39:27.755 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.755 DEBUG:boto:Data: 2024-04-19T23:39:27.756 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.756 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.756 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.756 DEBUG:boto:Params: {} 2024-04-19T23:39:27.756 DEBUG:boto:Token: None 2024-04-19T23:39:27.756 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.756 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d8d374e4e219557-00662300af-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.802 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.803 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.803 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.803 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.803 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.803 DEBUG:boto:Method: GET 2024-04-19T23:39:27.803 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.804 DEBUG:boto:Data: 2024-04-19T23:39:27.804 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.804 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.804 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.804 DEBUG:boto:Params: {} 2024-04-19T23:39:27.804 DEBUG:boto:Token: None 2024-04-19T23:39:27.804 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CZFVvavVmlCSUMEHAm+itt09SHY= 2024-04-19T23:39:27.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CZFVvavVmlCSUMEHAm+itt09SHY=', 'Content-Length': '0'} 2024-04-19T23:39:27.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e786a250d215740-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.807 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.807 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.807 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.807 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.807 DEBUG:boto:Method: GET 2024-04-19T23:39:27.807 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.807 DEBUG:boto:Data: 2024-04-19T23:39:27.807 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.807 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.807 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.807 DEBUG:boto:Params: {} 2024-04-19T23:39:27.808 DEBUG:boto:Token: None 2024-04-19T23:39:27.808 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.808 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CZFVvavVmlCSUMEHAm+itt09SHY= 2024-04-19T23:39:27.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CZFVvavVmlCSUMEHAm+itt09SHY=', 'Content-Length': '0'} 2024-04-19T23:39:27.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad8a669bfc863c4a-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.810 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.810 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.810 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.810 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.811 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.811 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.811 DEBUG:boto:Data: 2024-04-19T23:39:27.811 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.811 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.811 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.811 DEBUG:boto:Params: {} 2024-04-19T23:39:27.811 DEBUG:boto:Token: None 2024-04-19T23:39:27.811 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.812 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2/DlLQGctYnkra2f2VBlZ9lCO1E= 2024-04-19T23:39:27.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2/DlLQGctYnkra2f2VBlZ9lCO1E=', 'Content-Length': '0'} 2024-04-19T23:39:27.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe6b1cbe5aafa1ab-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.814 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.814 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.814 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.814 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.814 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.814 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.814 DEBUG:boto:Data: 2024-04-19T23:39:27.815 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.815 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.815 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.815 DEBUG:boto:Params: {} 2024-04-19T23:39:27.815 DEBUG:boto:Token: None 2024-04-19T23:39:27.815 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.815 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2/DlLQGctYnkra2f2VBlZ9lCO1E= 2024-04-19T23:39:27.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2/DlLQGctYnkra2f2VBlZ9lCO1E=', 'Content-Length': '0'} 2024-04-19T23:39:27.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e92574ad8f94fbf-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.817 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.817 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.817 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.818 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.818 DEBUG:boto:Method: GET 2024-04-19T23:39:27.818 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.818 DEBUG:boto:Data: 2024-04-19T23:39:27.818 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.818 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.818 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.818 DEBUG:boto:Params: {} 2024-04-19T23:39:27.818 DEBUG:boto:Token: None 2024-04-19T23:39:27.818 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.819 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CZFVvavVmlCSUMEHAm+itt09SHY= 2024-04-19T23:39:27.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CZFVvavVmlCSUMEHAm+itt09SHY=', 'Content-Length': '0'} 2024-04-19T23:39:27.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000317b79807ce1ec2-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.821 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.821 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.821 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.821 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.821 DEBUG:boto:Method: GET 2024-04-19T23:39:27.822 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.822 DEBUG:boto:Data: 2024-04-19T23:39:27.822 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.822 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.822 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.822 DEBUG:boto:Params: {} 2024-04-19T23:39:27.822 DEBUG:boto:Token: None 2024-04-19T23:39:27.822 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:27.822 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CZFVvavVmlCSUMEHAm+itt09SHY= 2024-04-19T23:39:27.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CZFVvavVmlCSUMEHAm+itt09SHY=', 'Content-Length': '0'} 2024-04-19T23:39:27.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000157b4b1d4d1426b6-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.870 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.871 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.871 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.871 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.871 DEBUG:boto:Method: GET 2024-04-19T23:39:27.871 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.871 DEBUG:boto:Data: 2024-04-19T23:39:27.871 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.871 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.872 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.872 DEBUG:boto:Params: {} 2024-04-19T23:39:27.872 DEBUG:boto:Token: None 2024-04-19T23:39:27.872 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.872 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001224383f4741151d-00662300af-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.875 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.876 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.876 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.876 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.876 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.876 DEBUG:boto:Method: GET 2024-04-19T23:39:27.876 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.876 DEBUG:boto:Data: 2024-04-19T23:39:27.876 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.876 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.876 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.876 DEBUG:boto:Params: {} 2024-04-19T23:39:27.877 DEBUG:boto:Token: None 2024-04-19T23:39:27.877 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.877 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004907b4868d9c53e2-00662300af-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.880 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.880 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.880 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.880 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.880 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:27.881 DEBUG:boto:Data: 2024-04-19T23:39:27.881 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.881 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.881 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.881 DEBUG:boto:Params: {} 2024-04-19T23:39:27.881 DEBUG:boto:Token: None 2024-04-19T23:39:27.881 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj 2024-04-19T23:39:27.881 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE= 2024-04-19T23:39:27.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE=', 'Content-Length': '0'} 2024-04-19T23:39:27.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6d0fb52690739c5-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.885 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.885 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.885 DEBUG:boto:Method: HEAD 2024-04-19T23:39:27.885 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:27.885 DEBUG:boto:Data: 2024-04-19T23:39:27.885 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.885 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.885 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.886 DEBUG:boto:Params: {} 2024-04-19T23:39:27.886 DEBUG:boto:Token: None 2024-04-19T23:39:27.886 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj 2024-04-19T23:39:27.886 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE= 2024-04-19T23:39:27.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g6T6FghbXuM6dvugur5Ce2XyGXE=', 'Content-Length': '0'} 2024-04-19T23:39:27.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a690c4ee3ab5f2b-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.890 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.890 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.890 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.890 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.890 DEBUG:boto:Method: GET 2024-04-19T23:39:27.890 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.890 DEBUG:boto:Data: 2024-04-19T23:39:27.890 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.890 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.890 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.890 DEBUG:boto:Params: {} 2024-04-19T23:39:27.891 DEBUG:boto:Token: None 2024-04-19T23:39:27.891 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.891 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE= 2024-04-19T23:39:27.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE=', 'Content-Length': '0'} 2024-04-19T23:39:27.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b7079a7fcc4805b-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.934 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.935 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.935 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.935 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.935 DEBUG:boto:Method: GET 2024-04-19T23:39:27.935 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.935 DEBUG:boto:Data: 2024-04-19T23:39:27.935 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.935 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.935 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.935 DEBUG:boto:Params: {} 2024-04-19T23:39:27.935 DEBUG:boto:Token: None 2024-04-19T23:39:27.936 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:27.936 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE= 2024-04-19T23:39:27.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsC52ESbRyIjvMlK5+g6tgOOGnE=', 'Content-Length': '0'} 2024-04-19T23:39:27.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c72efc749ec7f3f9-00662300af-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.982 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.983 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.983 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.983 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.983 DEBUG:boto:Method: GET 2024-04-19T23:39:27.983 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.983 DEBUG:boto:Data: 2024-04-19T23:39:27.983 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.983 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.983 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.983 DEBUG:boto:Params: {} 2024-04-19T23:39:27.983 DEBUG:boto:Token: None 2024-04-19T23:39:27.984 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000759c9682a04a0a1a-00662300af-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.987 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.987 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:27.988 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:27.988 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.988 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:27.988 DEBUG:boto:Method: GET 2024-04-19T23:39:27.988 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:27.988 DEBUG:boto:Data: 2024-04-19T23:39:27.988 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.988 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.988 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.988 DEBUG:boto:Params: {} 2024-04-19T23:39:27.988 DEBUG:boto:Token: None 2024-04-19T23:39:27.989 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/ 2024-04-19T23:39:27.989 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k= 2024-04-19T23:39:27.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x4utcs4w2KJ9gFTfp7Ewo1rIe+k=', 'Content-Length': '0'} 2024-04-19T23:39:27.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cb9abccf0974806-00662300af-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.992 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:27.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:27.992 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.992 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.992 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:27.993 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:27.993 DEBUG:boto:Method: GET 2024-04-19T23:39:27.993 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:27.993 DEBUG:boto:Data: 2024-04-19T23:39:27.993 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.993 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:27.993 DEBUG:boto:Port: 8001 2024-04-19T23:39:27.993 DEBUG:boto:Params: {} 2024-04-19T23:39:27.993 DEBUG:boto:Token: None 2024-04-19T23:39:27.993 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:27.994 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ALuECiwy8Xiq5jiC4IIqw8uZJ8= 2024-04-19T23:39:27.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ALuECiwy8Xiq5jiC4IIqw8uZJ8=', 'Content-Length': '0'} 2024-04-19T23:39:27.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed47119af4912d53-00662300af-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.996 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.996 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:27.996 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:27.996 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:27.996 DEBUG:boto:Method: GET 2024-04-19T23:39:27.996 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:27.996 DEBUG:boto:Data: 2024-04-19T23:39:27.996 DEBUG:boto:Headers: {} 2024-04-19T23:39:27.997 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:27.997 DEBUG:boto:Port: 8000 2024-04-19T23:39:27.997 DEBUG:boto:Params: {} 2024-04-19T23:39:27.997 DEBUG:boto:Token: None 2024-04-19T23:39:27.997 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:27 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:27.997 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ALuECiwy8Xiq5jiC4IIqw8uZJ8= 2024-04-19T23:39:27.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ALuECiwy8Xiq5jiC4IIqw8uZJ8=', 'Content-Length': '0'} 2024-04-19T23:39:27.999 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003bcaf519a417e825-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:27.999 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:27.999 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.000 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.000 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.000 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.000 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.000 DEBUG:boto:Data: 2024-04-19T23:39:28.000 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.000 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.000 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.000 DEBUG:boto:Params: {} 2024-04-19T23:39:28.000 DEBUG:boto:Token: None 2024-04-19T23:39:28.000 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.001 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KYlqCK3NIgaJO5ptX1ok+aMTFFI= 2024-04-19T23:39:28.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KYlqCK3NIgaJO5ptX1ok+aMTFFI=', 'Content-Length': '0'} 2024-04-19T23:39:28.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000386966f83e29aa7e-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.003 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.003 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.003 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.003 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.003 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.003 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.003 DEBUG:boto:Data: 2024-04-19T23:39:28.003 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.004 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.004 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.004 DEBUG:boto:Params: {} 2024-04-19T23:39:28.004 DEBUG:boto:Token: None 2024-04-19T23:39:28.004 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.004 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KYlqCK3NIgaJO5ptX1ok+aMTFFI= 2024-04-19T23:39:28.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KYlqCK3NIgaJO5ptX1ok+aMTFFI=', 'Content-Length': '0'} 2024-04-19T23:39:28.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c32af47165aacee-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.006 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.006 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.006 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.006 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.007 DEBUG:boto:Method: GET 2024-04-19T23:39:28.007 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.007 DEBUG:boto:Data: 2024-04-19T23:39:28.007 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.007 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.007 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.007 DEBUG:boto:Params: {} 2024-04-19T23:39:28.007 DEBUG:boto:Token: None 2024-04-19T23:39:28.007 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.007 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fu/BNs0rVT+QASazbyo3Q8HX4r8= 2024-04-19T23:39:28.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fu/BNs0rVT+QASazbyo3Q8HX4r8=', 'Content-Length': '0'} 2024-04-19T23:39:28.010 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097204f92b811a00d-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.054 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.055 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.055 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.055 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.055 DEBUG:boto:Method: GET 2024-04-19T23:39:28.055 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.055 DEBUG:boto:Data: 2024-04-19T23:39:28.055 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.055 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.055 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.055 DEBUG:boto:Params: {} 2024-04-19T23:39:28.056 DEBUG:boto:Token: None 2024-04-19T23:39:28.056 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:28.056 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fu/BNs0rVT+QASazbyo3Q8HX4r8= 2024-04-19T23:39:28.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fu/BNs0rVT+QASazbyo3Q8HX4r8=', 'Content-Length': '0'} 2024-04-19T23:39:28.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a6328179e2e9e20-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.059 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.059 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.059 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.059 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.060 DEBUG:boto:Method: GET 2024-04-19T23:39:28.060 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.060 DEBUG:boto:Data: 2024-04-19T23:39:28.060 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.060 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.060 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.060 DEBUG:boto:Params: {} 2024-04-19T23:39:28.060 DEBUG:boto:Token: None 2024-04-19T23:39:28.060 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.061 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8f68fa482ca7425-00662300b0-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.063 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.064 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.064 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.064 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.064 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.064 DEBUG:boto:Method: GET 2024-04-19T23:39:28.064 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.064 DEBUG:boto:Data: 2024-04-19T23:39:28.064 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.065 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.065 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.065 DEBUG:boto:Params: {} 2024-04-19T23:39:28.065 DEBUG:boto:Token: None 2024-04-19T23:39:28.065 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.065 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011c2055d6ad4b964-00662300b0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.068 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.068 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.069 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.069 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.069 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:28.069 DEBUG:boto:Data: 2024-04-19T23:39:28.069 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.069 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.069 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.069 DEBUG:boto:Params: {} 2024-04-19T23:39:28.069 DEBUG:boto:Token: None 2024-04-19T23:39:28.069 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj 2024-04-19T23:39:28.070 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0= 2024-04-19T23:39:28.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0=', 'Content-Length': '0'} 2024-04-19T23:39:28.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8332b4dcdf6ed36-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.073 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.073 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.073 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.073 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:28.073 DEBUG:boto:Data: 2024-04-19T23:39:28.073 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.074 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.074 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.074 DEBUG:boto:Params: {} 2024-04-19T23:39:28.074 DEBUG:boto:Token: None 2024-04-19T23:39:28.074 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj 2024-04-19T23:39:28.074 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0= 2024-04-19T23:39:28.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0=', 'Content-Length': '0'} 2024-04-19T23:39:28.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ae86620d371dccb-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.078 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.078 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.078 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.078 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.078 DEBUG:boto:Method: GET 2024-04-19T23:39:28.078 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.078 DEBUG:boto:Data: 2024-04-19T23:39:28.078 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.078 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.078 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.079 DEBUG:boto:Params: {} 2024-04-19T23:39:28.079 DEBUG:boto:Token: None 2024-04-19T23:39:28.079 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.079 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA= 2024-04-19T23:39:28.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA=', 'Content-Length': '0'} 2024-04-19T23:39:28.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001378d559761acdfe-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.122 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.123 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.123 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.123 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.123 DEBUG:boto:Method: GET 2024-04-19T23:39:28.123 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.123 DEBUG:boto:Data: 2024-04-19T23:39:28.123 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.123 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.123 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.123 DEBUG:boto:Params: {} 2024-04-19T23:39:28.124 DEBUG:boto:Token: None 2024-04-19T23:39:28.124 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.124 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA= 2024-04-19T23:39:28.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA=', 'Content-Length': '0'} 2024-04-19T23:39:28.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af4bf8500c09781f-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.166 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.167 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.167 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.167 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.167 DEBUG:boto:Method: GET 2024-04-19T23:39:28.167 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.167 DEBUG:boto:Data: 2024-04-19T23:39:28.167 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.167 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.167 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.167 DEBUG:boto:Params: {} 2024-04-19T23:39:28.168 DEBUG:boto:Token: None 2024-04-19T23:39:28.168 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.168 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d422f9cb3a184e-00662300b0-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.214 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.215 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.215 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.215 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.215 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.215 DEBUG:boto:Method: GET 2024-04-19T23:39:28.216 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.216 DEBUG:boto:Data: 2024-04-19T23:39:28.216 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.216 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.217 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.217 DEBUG:boto:Params: {} 2024-04-19T23:39:28.217 DEBUG:boto:Token: None 2024-04-19T23:39:28.217 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.218 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc3e6df9fc561a41-00662300b0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.262 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.263 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.263 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.263 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.263 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.263 DEBUG:boto:Method: GET 2024-04-19T23:39:28.263 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.263 DEBUG:boto:Data: 2024-04-19T23:39:28.264 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.264 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.264 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.264 DEBUG:boto:Params: {} 2024-04-19T23:39:28.264 DEBUG:boto:Token: None 2024-04-19T23:39:28.264 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.264 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1Q4s+ZPrcxRoySgsQbaMuWTnU1U= 2024-04-19T23:39:28.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1Q4s+ZPrcxRoySgsQbaMuWTnU1U=', 'Content-Length': '0'} 2024-04-19T23:39:28.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d591755aab1ce8dc-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.267 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.267 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.267 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.267 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.267 DEBUG:boto:Method: GET 2024-04-19T23:39:28.267 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.267 DEBUG:boto:Data: 2024-04-19T23:39:28.267 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.268 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.268 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.268 DEBUG:boto:Params: {} 2024-04-19T23:39:28.268 DEBUG:boto:Token: None 2024-04-19T23:39:28.268 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.268 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1Q4s+ZPrcxRoySgsQbaMuWTnU1U= 2024-04-19T23:39:28.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1Q4s+ZPrcxRoySgsQbaMuWTnU1U=', 'Content-Length': '0'} 2024-04-19T23:39:28.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a7ba03aeabbe576-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.270 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.270 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.270 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.270 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.271 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.271 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.271 DEBUG:boto:Data: 2024-04-19T23:39:28.271 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.271 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.271 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.271 DEBUG:boto:Params: {} 2024-04-19T23:39:28.271 DEBUG:boto:Token: None 2024-04-19T23:39:28.271 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.271 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+6Rv14dgS+V7ILKilD/C7Z2/Ig4= 2024-04-19T23:39:28.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+6Rv14dgS+V7ILKilD/C7Z2/Ig4=', 'Content-Length': '0'} 2024-04-19T23:39:28.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078f1b2372411aed8-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.274 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.274 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.274 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.274 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.274 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.274 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.274 DEBUG:boto:Data: 2024-04-19T23:39:28.274 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.274 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.274 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.274 DEBUG:boto:Params: {} 2024-04-19T23:39:28.275 DEBUG:boto:Token: None 2024-04-19T23:39:28.275 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.275 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+6Rv14dgS+V7ILKilD/C7Z2/Ig4= 2024-04-19T23:39:28.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+6Rv14dgS+V7ILKilD/C7Z2/Ig4=', 'Content-Length': '0'} 2024-04-19T23:39:28.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7c458b9982b5bce-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.277 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.277 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.277 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.277 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.277 DEBUG:boto:Method: GET 2024-04-19T23:39:28.278 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.278 DEBUG:boto:Data: 2024-04-19T23:39:28.278 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.278 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.278 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.278 DEBUG:boto:Params: {} 2024-04-19T23:39:28.278 DEBUG:boto:Token: None 2024-04-19T23:39:28.278 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.278 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1Q4s+ZPrcxRoySgsQbaMuWTnU1U= 2024-04-19T23:39:28.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1Q4s+ZPrcxRoySgsQbaMuWTnU1U=', 'Content-Length': '0'} 2024-04-19T23:39:28.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074110d0cc69d2c0e-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.322 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.323 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.323 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.323 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.323 DEBUG:boto:Method: GET 2024-04-19T23:39:28.323 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.323 DEBUG:boto:Data: 2024-04-19T23:39:28.323 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.323 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.323 DEBUG:boto:Params: {} 2024-04-19T23:39:28.324 DEBUG:boto:Token: None 2024-04-19T23:39:28.324 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:28.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1Q4s+ZPrcxRoySgsQbaMuWTnU1U= 2024-04-19T23:39:28.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1Q4s+ZPrcxRoySgsQbaMuWTnU1U=', 'Content-Length': '0'} 2024-04-19T23:39:28.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b46648667cc4d31-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.366 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.367 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.367 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.367 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.367 DEBUG:boto:Method: GET 2024-04-19T23:39:28.367 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.367 DEBUG:boto:Data: 2024-04-19T23:39:28.367 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.367 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.368 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.368 DEBUG:boto:Params: {} 2024-04-19T23:39:28.368 DEBUG:boto:Token: None 2024-04-19T23:39:28.368 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.368 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081e2beab931f30d8-00662300b0-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.371 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.372 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.372 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.372 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.372 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.372 DEBUG:boto:Method: GET 2024-04-19T23:39:28.373 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.373 DEBUG:boto:Data: 2024-04-19T23:39:28.373 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.373 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.373 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.373 DEBUG:boto:Params: {} 2024-04-19T23:39:28.373 DEBUG:boto:Token: None 2024-04-19T23:39:28.373 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.373 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001e55eaf6bc12f2d-00662300b0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.378 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.378 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.378 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.379 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.379 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:28.379 DEBUG:boto:Data: 2024-04-19T23:39:28.379 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.379 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.379 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.379 DEBUG:boto:Params: {} 2024-04-19T23:39:28.379 DEBUG:boto:Token: None 2024-04-19T23:39:28.379 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj 2024-04-19T23:39:28.379 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0= 2024-04-19T23:39:28.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0=', 'Content-Length': '0'} 2024-04-19T23:39:28.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000278245bb9557c685-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.383 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.383 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.383 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.383 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:28.383 DEBUG:boto:Data: 2024-04-19T23:39:28.383 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.383 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.384 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.384 DEBUG:boto:Params: {} 2024-04-19T23:39:28.384 DEBUG:boto:Token: None 2024-04-19T23:39:28.384 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj 2024-04-19T23:39:28.384 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0= 2024-04-19T23:39:28.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0=', 'Content-Length': '0'} 2024-04-19T23:39:28.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017473650ab617ff1-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.388 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.388 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.388 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.388 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.388 DEBUG:boto:Method: GET 2024-04-19T23:39:28.388 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.388 DEBUG:boto:Data: 2024-04-19T23:39:28.388 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.388 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.388 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.388 DEBUG:boto:Params: {} 2024-04-19T23:39:28.389 DEBUG:boto:Token: None 2024-04-19T23:39:28.389 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.389 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA= 2024-04-19T23:39:28.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA=', 'Content-Length': '0'} 2024-04-19T23:39:28.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ef5b2cc0d867126-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.438 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.439 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.439 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.439 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.439 DEBUG:boto:Method: GET 2024-04-19T23:39:28.439 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.439 DEBUG:boto:Data: 2024-04-19T23:39:28.439 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.439 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.439 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.439 DEBUG:boto:Params: {} 2024-04-19T23:39:28.440 DEBUG:boto:Token: None 2024-04-19T23:39:28.440 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.440 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA= 2024-04-19T23:39:28.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA=', 'Content-Length': '0'} 2024-04-19T23:39:28.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023366db087ddadef-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.486 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.487 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.487 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.487 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.487 DEBUG:boto:Method: GET 2024-04-19T23:39:28.487 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.487 DEBUG:boto:Data: 2024-04-19T23:39:28.487 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.487 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.487 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.488 DEBUG:boto:Params: {} 2024-04-19T23:39:28.488 DEBUG:boto:Token: None 2024-04-19T23:39:28.488 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a5ac4df50fd0cda-00662300b0-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.492 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.492 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.492 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.493 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.493 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.493 DEBUG:boto:Method: GET 2024-04-19T23:39:28.493 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.493 DEBUG:boto:Data: 2024-04-19T23:39:28.493 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.493 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.493 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.493 DEBUG:boto:Params: {} 2024-04-19T23:39:28.494 DEBUG:boto:Token: None 2024-04-19T23:39:28.494 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.494 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c7d2c41c9d6fd23-00662300b0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.498 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.498 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.499 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.499 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.499 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.499 DEBUG:boto:Method: GET 2024-04-19T23:39:28.499 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.499 DEBUG:boto:Data: 2024-04-19T23:39:28.499 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.499 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.499 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.499 DEBUG:boto:Params: {} 2024-04-19T23:39:28.499 DEBUG:boto:Token: None 2024-04-19T23:39:28.500 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.500 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LVJH4ICHaHQiDK1hvPl+M/azSyA= 2024-04-19T23:39:28.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LVJH4ICHaHQiDK1hvPl+M/azSyA=', 'Content-Length': '0'} 2024-04-19T23:39:28.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008313ba41fc05ef12-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.502 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.502 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.502 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.502 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.502 DEBUG:boto:Method: GET 2024-04-19T23:39:28.503 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.503 DEBUG:boto:Data: 2024-04-19T23:39:28.503 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.503 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.503 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.503 DEBUG:boto:Params: {} 2024-04-19T23:39:28.503 DEBUG:boto:Token: None 2024-04-19T23:39:28.503 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.503 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LVJH4ICHaHQiDK1hvPl+M/azSyA= 2024-04-19T23:39:28.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LVJH4ICHaHQiDK1hvPl+M/azSyA=', 'Content-Length': '0'} 2024-04-19T23:39:28.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c866064ca2ba55a8-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.505 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.506 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.506 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.506 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.506 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.506 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.506 DEBUG:boto:Data: 2024-04-19T23:39:28.506 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.506 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.506 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.506 DEBUG:boto:Params: {} 2024-04-19T23:39:28.507 DEBUG:boto:Token: None 2024-04-19T23:39:28.507 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.507 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vFyXkvdeikFmv5U8kaNeJhYRmak= 2024-04-19T23:39:28.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vFyXkvdeikFmv5U8kaNeJhYRmak=', 'Content-Length': '0'} 2024-04-19T23:39:28.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1338799f74b7168-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.509 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.509 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.509 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.509 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.509 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.509 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.509 DEBUG:boto:Data: 2024-04-19T23:39:28.509 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.510 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.510 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.510 DEBUG:boto:Params: {} 2024-04-19T23:39:28.510 DEBUG:boto:Token: None 2024-04-19T23:39:28.510 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.510 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vFyXkvdeikFmv5U8kaNeJhYRmak= 2024-04-19T23:39:28.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vFyXkvdeikFmv5U8kaNeJhYRmak=', 'Content-Length': '0'} 2024-04-19T23:39:28.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003dbe10649b552ed3-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.512 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.512 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.512 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.512 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.513 DEBUG:boto:Method: GET 2024-04-19T23:39:28.513 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.513 DEBUG:boto:Data: 2024-04-19T23:39:28.513 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.513 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.513 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.513 DEBUG:boto:Params: {} 2024-04-19T23:39:28.513 DEBUG:boto:Token: None 2024-04-19T23:39:28.513 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.513 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LVJH4ICHaHQiDK1hvPl+M/azSyA= 2024-04-19T23:39:28.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LVJH4ICHaHQiDK1hvPl+M/azSyA=', 'Content-Length': '0'} 2024-04-19T23:39:28.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ca7a580f229361b-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.558 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.559 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.559 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.559 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.559 DEBUG:boto:Method: GET 2024-04-19T23:39:28.559 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.559 DEBUG:boto:Data: 2024-04-19T23:39:28.559 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.559 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.559 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.559 DEBUG:boto:Params: {} 2024-04-19T23:39:28.560 DEBUG:boto:Token: None 2024-04-19T23:39:28.560 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:28.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LVJH4ICHaHQiDK1hvPl+M/azSyA= 2024-04-19T23:39:28.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LVJH4ICHaHQiDK1hvPl+M/azSyA=', 'Content-Length': '0'} 2024-04-19T23:39:28.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008f0d6e4539514b9-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.562 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.563 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.563 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.563 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.563 DEBUG:boto:Method: GET 2024-04-19T23:39:28.563 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.563 DEBUG:boto:Data: 2024-04-19T23:39:28.563 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.563 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.563 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.563 DEBUG:boto:Params: {} 2024-04-19T23:39:28.564 DEBUG:boto:Token: None 2024-04-19T23:39:28.564 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.564 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007855ac8ea016277-00662300b0-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.567 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.567 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.567 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.567 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.567 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.567 DEBUG:boto:Method: GET 2024-04-19T23:39:28.567 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.568 DEBUG:boto:Data: 2024-04-19T23:39:28.568 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.568 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.568 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.568 DEBUG:boto:Params: {} 2024-04-19T23:39:28.568 DEBUG:boto:Token: None 2024-04-19T23:39:28.568 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.568 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0c8ede7ebd53f96-00662300b0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.571 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.571 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.572 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.572 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.572 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:28.572 DEBUG:boto:Data: 2024-04-19T23:39:28.572 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.572 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.572 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.572 DEBUG:boto:Params: {} 2024-04-19T23:39:28.572 DEBUG:boto:Token: None 2024-04-19T23:39:28.572 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj 2024-04-19T23:39:28.573 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0= 2024-04-19T23:39:28.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0=', 'Content-Length': '0'} 2024-04-19T23:39:28.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044348556176ad96a-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.576 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.577 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.577 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.577 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:28.577 DEBUG:boto:Data: 2024-04-19T23:39:28.577 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.577 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.577 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.577 DEBUG:boto:Params: {} 2024-04-19T23:39:28.577 DEBUG:boto:Token: None 2024-04-19T23:39:28.577 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj 2024-04-19T23:39:28.578 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0= 2024-04-19T23:39:28.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0=', 'Content-Length': '0'} 2024-04-19T23:39:28.581 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000add075dd6a952a01-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.581 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.581 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.581 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.581 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.582 DEBUG:boto:Method: GET 2024-04-19T23:39:28.582 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.582 DEBUG:boto:Data: 2024-04-19T23:39:28.582 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.582 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.582 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.582 DEBUG:boto:Params: {} 2024-04-19T23:39:28.582 DEBUG:boto:Token: None 2024-04-19T23:39:28.582 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.583 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA= 2024-04-19T23:39:28.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA=', 'Content-Length': '0'} 2024-04-19T23:39:28.584 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070bb45396a902aff-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.630 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.631 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.631 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.631 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.631 DEBUG:boto:Method: GET 2024-04-19T23:39:28.631 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.631 DEBUG:boto:Data: 2024-04-19T23:39:28.631 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.631 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.631 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.631 DEBUG:boto:Params: {} 2024-04-19T23:39:28.632 DEBUG:boto:Token: None 2024-04-19T23:39:28.632 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA= 2024-04-19T23:39:28.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA=', 'Content-Length': '0'} 2024-04-19T23:39:28.634 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000733fc835f3e54011-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.674 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.675 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.675 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.675 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.675 DEBUG:boto:Method: GET 2024-04-19T23:39:28.675 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.675 DEBUG:boto:Data: 2024-04-19T23:39:28.675 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.675 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.675 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.676 DEBUG:boto:Params: {} 2024-04-19T23:39:28.676 DEBUG:boto:Token: None 2024-04-19T23:39:28.676 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000315ea4f533c5c3c1-00662300b0-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.679 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.680 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.680 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.680 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.680 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.680 DEBUG:boto:Method: GET 2024-04-19T23:39:28.680 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.680 DEBUG:boto:Data: 2024-04-19T23:39:28.680 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.680 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.680 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.681 DEBUG:boto:Params: {} 2024-04-19T23:39:28.681 DEBUG:boto:Token: None 2024-04-19T23:39:28.681 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.681 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002615b0391b72c5d6-00662300b0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.684 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.684 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.684 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.684 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.684 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.685 DEBUG:boto:Method: GET 2024-04-19T23:39:28.685 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.685 DEBUG:boto:Data: 2024-04-19T23:39:28.685 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.685 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.685 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.685 DEBUG:boto:Params: {} 2024-04-19T23:39:28.685 DEBUG:boto:Token: None 2024-04-19T23:39:28.685 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.685 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eWjXLPlLmIhs61eyW2NekFM+zT4= 2024-04-19T23:39:28.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eWjXLPlLmIhs61eyW2NekFM+zT4=', 'Content-Length': '0'} 2024-04-19T23:39:28.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017531b704510d998-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.688 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.688 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.688 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.688 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.688 DEBUG:boto:Method: GET 2024-04-19T23:39:28.688 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.688 DEBUG:boto:Data: 2024-04-19T23:39:28.688 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.688 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.689 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.689 DEBUG:boto:Params: {} 2024-04-19T23:39:28.689 DEBUG:boto:Token: None 2024-04-19T23:39:28.689 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.689 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eWjXLPlLmIhs61eyW2NekFM+zT4= 2024-04-19T23:39:28.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eWjXLPlLmIhs61eyW2NekFM+zT4=', 'Content-Length': '0'} 2024-04-19T23:39:28.691 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fd0dc8b733c7b4a-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.691 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.691 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.691 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.691 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.692 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.692 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.692 DEBUG:boto:Data: 2024-04-19T23:39:28.692 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.692 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.692 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.692 DEBUG:boto:Params: {} 2024-04-19T23:39:28.692 DEBUG:boto:Token: None 2024-04-19T23:39:28.692 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.692 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Tx4N8d3/w17/Yw+M0KsL3u7vTJA= 2024-04-19T23:39:28.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Tx4N8d3/w17/Yw+M0KsL3u7vTJA=', 'Content-Length': '0'} 2024-04-19T23:39:28.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f1826f458ebc850-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.694 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.695 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.695 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.695 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.695 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.695 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.695 DEBUG:boto:Data: 2024-04-19T23:39:28.695 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.695 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.695 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.695 DEBUG:boto:Params: {} 2024-04-19T23:39:28.695 DEBUG:boto:Token: None 2024-04-19T23:39:28.696 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.696 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Tx4N8d3/w17/Yw+M0KsL3u7vTJA= 2024-04-19T23:39:28.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Tx4N8d3/w17/Yw+M0KsL3u7vTJA=', 'Content-Length': '0'} 2024-04-19T23:39:28.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc7ee462eb9082bf-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.698 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.698 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.698 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.698 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.698 DEBUG:boto:Method: GET 2024-04-19T23:39:28.698 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.698 DEBUG:boto:Data: 2024-04-19T23:39:28.699 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.699 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.699 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.699 DEBUG:boto:Params: {} 2024-04-19T23:39:28.699 DEBUG:boto:Token: None 2024-04-19T23:39:28.699 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.699 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eWjXLPlLmIhs61eyW2NekFM+zT4= 2024-04-19T23:39:28.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eWjXLPlLmIhs61eyW2NekFM+zT4=', 'Content-Length': '0'} 2024-04-19T23:39:28.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d33ebc38aa76bb00-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.702 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.702 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.702 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.702 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.702 DEBUG:boto:Method: GET 2024-04-19T23:39:28.702 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.702 DEBUG:boto:Data: 2024-04-19T23:39:28.703 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.703 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.703 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.703 DEBUG:boto:Params: {} 2024-04-19T23:39:28.703 DEBUG:boto:Token: None 2024-04-19T23:39:28.703 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:28.703 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eWjXLPlLmIhs61eyW2NekFM+zT4= 2024-04-19T23:39:28.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eWjXLPlLmIhs61eyW2NekFM+zT4=', 'Content-Length': '0'} 2024-04-19T23:39:28.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3a842e108e0441a-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.747 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.747 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.747 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.747 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.748 DEBUG:boto:Method: GET 2024-04-19T23:39:28.748 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.748 DEBUG:boto:Data: 2024-04-19T23:39:28.748 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.748 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.748 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.748 DEBUG:boto:Params: {} 2024-04-19T23:39:28.748 DEBUG:boto:Token: None 2024-04-19T23:39:28.748 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.748 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ebfc322844b7dd9-00662300b0-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.755 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.756 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.756 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.757 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.757 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.757 DEBUG:boto:Method: GET 2024-04-19T23:39:28.757 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.757 DEBUG:boto:Data: 2024-04-19T23:39:28.757 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.757 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.758 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.758 DEBUG:boto:Params: {} 2024-04-19T23:39:28.759 DEBUG:boto:Token: None 2024-04-19T23:39:28.759 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.759 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053396e12f1eee674-00662300b0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.762 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.763 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.763 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.763 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.763 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:28.763 DEBUG:boto:Data: 2024-04-19T23:39:28.763 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.764 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.764 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.764 DEBUG:boto:Params: {} 2024-04-19T23:39:28.764 DEBUG:boto:Token: None 2024-04-19T23:39:28.764 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj 2024-04-19T23:39:28.765 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0= 2024-04-19T23:39:28.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0=', 'Content-Length': '0'} 2024-04-19T23:39:28.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8a9a8359220ba28-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.768 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.769 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.769 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.769 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:28.769 DEBUG:boto:Data: 2024-04-19T23:39:28.769 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.769 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.769 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.769 DEBUG:boto:Params: {} 2024-04-19T23:39:28.769 DEBUG:boto:Token: None 2024-04-19T23:39:28.769 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj 2024-04-19T23:39:28.770 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0= 2024-04-19T23:39:28.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8hX1Wa9JuC8k6s2u67yDfA2oNP0=', 'Content-Length': '0'} 2024-04-19T23:39:28.773 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075df452ad2f0c094-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.773 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.774 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.774 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.774 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.774 DEBUG:boto:Method: GET 2024-04-19T23:39:28.774 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.774 DEBUG:boto:Data: 2024-04-19T23:39:28.774 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.774 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.774 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.774 DEBUG:boto:Params: {} 2024-04-19T23:39:28.775 DEBUG:boto:Token: None 2024-04-19T23:39:28.775 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.775 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA= 2024-04-19T23:39:28.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA=', 'Content-Length': '0'} 2024-04-19T23:39:28.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9ae35e751f4dda9-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.823 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.823 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.823 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.823 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.823 DEBUG:boto:Method: GET 2024-04-19T23:39:28.823 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.823 DEBUG:boto:Data: 2024-04-19T23:39:28.823 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.823 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.824 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.824 DEBUG:boto:Params: {} 2024-04-19T23:39:28.824 DEBUG:boto:Token: None 2024-04-19T23:39:28.824 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:28.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA= 2024-04-19T23:39:28.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZBQ8ISmBGkaZHuqO58MbWimuiQA=', 'Content-Length': '0'} 2024-04-19T23:39:28.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d146af85aa43743d-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.870 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.871 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.871 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.871 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.871 DEBUG:boto:Method: GET 2024-04-19T23:39:28.871 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.871 DEBUG:boto:Data: 2024-04-19T23:39:28.871 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.871 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.871 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.871 DEBUG:boto:Params: {} 2024-04-19T23:39:28.872 DEBUG:boto:Token: None 2024-04-19T23:39:28.872 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.872 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035db9e7f7397c9e5-00662300b0-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.875 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.875 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:28.875 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:28.876 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.876 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:28.876 DEBUG:boto:Method: GET 2024-04-19T23:39:28.876 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:28.876 DEBUG:boto:Data: 2024-04-19T23:39:28.876 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.876 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.876 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.876 DEBUG:boto:Params: {} 2024-04-19T23:39:28.877 DEBUG:boto:Token: None 2024-04-19T23:39:28.877 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/ 2024-04-19T23:39:28.877 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU= 2024-04-19T23:39:28.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R4pgVhjtNPS/QmCO5mp/N5nzXdU=', 'Content-Length': '0'} 2024-04-19T23:39:28.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a77c8919242a2536-00662300b0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.880 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:28.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.880 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.880 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.880 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.881 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.881 DEBUG:boto:Method: GET 2024-04-19T23:39:28.881 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.881 DEBUG:boto:Data: 2024-04-19T23:39:28.881 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.881 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.881 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.881 DEBUG:boto:Params: {} 2024-04-19T23:39:28.881 DEBUG:boto:Token: None 2024-04-19T23:39:28.882 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.882 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LVPpmUXjRiHNjAppzyQM8Eo1Ew4= 2024-04-19T23:39:28.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LVPpmUXjRiHNjAppzyQM8Eo1Ew4=', 'Content-Length': '0'} 2024-04-19T23:39:28.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbc0b0f160418815-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.884 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.884 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.884 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.884 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.884 DEBUG:boto:Method: GET 2024-04-19T23:39:28.884 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.885 DEBUG:boto:Data: 2024-04-19T23:39:28.885 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.885 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.885 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.885 DEBUG:boto:Params: {} 2024-04-19T23:39:28.885 DEBUG:boto:Token: None 2024-04-19T23:39:28.885 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.885 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LVPpmUXjRiHNjAppzyQM8Eo1Ew4= 2024-04-19T23:39:28.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LVPpmUXjRiHNjAppzyQM8Eo1Ew4=', 'Content-Length': '0'} 2024-04-19T23:39:28.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054e71177c8bcd25f-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.934 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.934 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.935 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.935 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.935 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.935 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.935 DEBUG:boto:Data: 2024-04-19T23:39:28.935 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.935 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.935 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.935 DEBUG:boto:Params: {} 2024-04-19T23:39:28.935 DEBUG:boto:Token: None 2024-04-19T23:39:28.936 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.936 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CqJ+MqEMCgv8/w3A1bAo68gbYO0= 2024-04-19T23:39:28.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CqJ+MqEMCgv8/w3A1bAo68gbYO0=', 'Content-Length': '0'} 2024-04-19T23:39:28.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d21315347d6166a5-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.938 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.938 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.938 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.938 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.939 DEBUG:boto:Method: HEAD 2024-04-19T23:39:28.939 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.939 DEBUG:boto:Data: 2024-04-19T23:39:28.939 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.939 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.939 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.939 DEBUG:boto:Params: {} 2024-04-19T23:39:28.939 DEBUG:boto:Token: None 2024-04-19T23:39:28.939 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.939 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CqJ+MqEMCgv8/w3A1bAo68gbYO0= 2024-04-19T23:39:28.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CqJ+MqEMCgv8/w3A1bAo68gbYO0=', 'Content-Length': '0'} 2024-04-19T23:39:28.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ea0e3c9ef12f9ed-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:28.942 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.942 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.942 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.942 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.943 DEBUG:boto:Method: GET 2024-04-19T23:39:28.943 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.943 DEBUG:boto:Data: 2024-04-19T23:39:28.943 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.943 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:28.943 DEBUG:boto:Port: 8001 2024-04-19T23:39:28.943 DEBUG:boto:Params: {} 2024-04-19T23:39:28.943 DEBUG:boto:Token: None 2024-04-19T23:39:28.943 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.943 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LVPpmUXjRiHNjAppzyQM8Eo1Ew4= 2024-04-19T23:39:28.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LVPpmUXjRiHNjAppzyQM8Eo1Ew4=', 'Content-Length': '0'} 2024-04-19T23:39:28.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aecd8503b27bfe00-00662300b0-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:28.990 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:28.991 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:28.991 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.991 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.991 DEBUG:boto:Method: GET 2024-04-19T23:39:28.991 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.991 DEBUG:boto:Data: 2024-04-19T23:39:28.991 DEBUG:boto:Headers: {} 2024-04-19T23:39:28.991 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:28.991 DEBUG:boto:Port: 8000 2024-04-19T23:39:28.991 DEBUG:boto:Params: {} 2024-04-19T23:39:28.992 DEBUG:boto:Token: None 2024-04-19T23:39:28.992 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:28 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:28.992 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LVPpmUXjRiHNjAppzyQM8Eo1Ew4= 2024-04-19T23:39:28.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LVPpmUXjRiHNjAppzyQM8Eo1Ew4=', 'Content-Length': '0'} 2024-04-19T23:39:28.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b03f5aa15dc029e-00662300b0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.034 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.035 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.035 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.035 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.035 DEBUG:boto:Method: GET 2024-04-19T23:39:29.035 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.035 DEBUG:boto:Data: 2024-04-19T23:39:29.035 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.035 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.035 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.035 DEBUG:boto:Params: {} 2024-04-19T23:39:29.035 DEBUG:boto:Token: None 2024-04-19T23:39:29.036 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.036 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e5147ff9b5899d0-00662300b1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.039 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.039 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.039 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.039 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.039 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.040 DEBUG:boto:Method: GET 2024-04-19T23:39:29.040 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.040 DEBUG:boto:Data: 2024-04-19T23:39:29.040 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.040 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:29.040 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.040 DEBUG:boto:Params: {} 2024-04-19T23:39:29.040 DEBUG:boto:Token: None 2024-04-19T23:39:29.040 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.040 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f23ae1bda68d03e0-00662300b1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.043 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.044 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.044 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.044 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.044 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:29.044 DEBUG:boto:Data: 2024-04-19T23:39:29.044 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.044 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.044 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.044 DEBUG:boto:Params: {} 2024-04-19T23:39:29.044 DEBUG:boto:Token: None 2024-04-19T23:39:29.045 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj 2024-04-19T23:39:29.045 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE= 2024-04-19T23:39:29.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE=', 'Content-Length': '0'} 2024-04-19T23:39:29.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f38726440da9a796-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.048 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.049 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.049 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.049 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:29.049 DEBUG:boto:Data: 2024-04-19T23:39:29.049 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.049 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:29.049 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.049 DEBUG:boto:Params: {} 2024-04-19T23:39:29.049 DEBUG:boto:Token: None 2024-04-19T23:39:29.049 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj 2024-04-19T23:39:29.050 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE= 2024-04-19T23:39:29.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE=', 'Content-Length': '0'} 2024-04-19T23:39:29.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d84ddc9e5e6a4fad-00662300b1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:29.053 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.053 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.053 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.053 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.053 DEBUG:boto:Method: GET 2024-04-19T23:39:29.054 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.054 DEBUG:boto:Data: 2024-04-19T23:39:29.054 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.054 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.054 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.054 DEBUG:boto:Params: {} 2024-04-19T23:39:29.054 DEBUG:boto:Token: None 2024-04-19T23:39:29.054 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.054 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk= 2024-04-19T23:39:29.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk=', 'Content-Length': '0'} 2024-04-19T23:39:29.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000feb7e8b03d3f6bee-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.102 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.103 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.103 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.103 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.103 DEBUG:boto:Method: GET 2024-04-19T23:39:29.103 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.103 DEBUG:boto:Data: 2024-04-19T23:39:29.103 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.103 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:29.103 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.104 DEBUG:boto:Params: {} 2024-04-19T23:39:29.104 DEBUG:boto:Token: None 2024-04-19T23:39:29.104 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk= 2024-04-19T23:39:29.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk=', 'Content-Length': '0'} 2024-04-19T23:39:29.106 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005dd8ea5da0a34008-00662300b1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.146 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.146 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.147 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.147 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.147 DEBUG:boto:Method: GET 2024-04-19T23:39:29.147 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.147 DEBUG:boto:Data: 2024-04-19T23:39:29.147 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.147 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.147 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.147 DEBUG:boto:Params: {} 2024-04-19T23:39:29.147 DEBUG:boto:Token: None 2024-04-19T23:39:29.147 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.148 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1a421b2d6b420bd-00662300b1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.151 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.151 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.151 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.151 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.151 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.151 DEBUG:boto:Method: GET 2024-04-19T23:39:29.151 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.151 DEBUG:boto:Data: 2024-04-19T23:39:29.152 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.152 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:29.152 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.152 DEBUG:boto:Params: {} 2024-04-19T23:39:29.152 DEBUG:boto:Token: None 2024-04-19T23:39:29.152 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.152 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b2d117baafb486f-00662300b1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.155 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.155 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-22 zones={test-zone3, test-zone1} 2024-04-19T23:39:29.155 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-22 zones={test-zone3, test-zone2} 2024-04-19T23:39:29.156 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.156 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.156 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.156 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:39:29.156 DEBUG:boto:Data: 2024-04-19T23:39:29.156 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.156 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.156 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.156 DEBUG:boto:Params: {} 2024-04-19T23:39:29.156 DEBUG:boto:Token: None 2024-04-19T23:39:29.156 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.157 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:i/PV5pm/ABfQfmLNRxpUjboB1RI= 2024-04-19T23:39:29.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:i/PV5pm/ABfQfmLNRxpUjboB1RI=', 'Content-Length': '0'} 2024-04-19T23:39:29.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003820cb41817e024d-00662300b1-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.159 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.159 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.159 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.159 DEBUG:boto:Path: /aktlji-22/ 2024-04-19T23:39:29.159 DEBUG:boto:Data: 2024-04-19T23:39:29.159 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.160 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.160 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.160 DEBUG:boto:Params: {} 2024-04-19T23:39:29.160 DEBUG:boto:Token: None 2024-04-19T23:39:29.160 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:i/PV5pm/ABfQfmLNRxpUjboB1RI= 2024-04-19T23:39:29.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:i/PV5pm/ABfQfmLNRxpUjboB1RI=', 'Content-Length': '0'} 2024-04-19T23:39:29.163 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000062324d5c2022e28-00662300b1-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.163 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:39:29.163 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.163 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.163 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:29.163 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:29.163 DEBUG:boto:Method: GET 2024-04-19T23:39:29.163 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:39:29.164 DEBUG:boto:Data: 2024-04-19T23:39:29.164 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.164 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.164 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.164 DEBUG:boto:Params: {} 2024-04-19T23:39:29.164 DEBUG:boto:Token: None 2024-04-19T23:39:29.164 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/?versions 2024-04-19T23:39:29.164 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JxZDkPJsii5O1oeX06krkmFHCMQ= 2024-04-19T23:39:29.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JxZDkPJsii5O1oeX06krkmFHCMQ=', 'Content-Length': '0'} 2024-04-19T23:39:29.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f63329655910b71-00662300b1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.170 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOotrue2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjI8Qos7O61Drj6zKFSuRQg5BJ8qTn1Itfalse2024-04-19T23:36:20.273Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITUfalse2024-04-19T23:36:20.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUjbMVzWXMoq8kQHxc5fXZiC69vNXkcjfalse2024-04-19T23:36:20.222Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97false2024-04-19T23:36:20.188Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2false2024-04-19T23:36:20.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Jy1wSwswautcJ.pAM0qXnm5CTgNceOfalse2024-04-19T23:36:20.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-false2024-04-19T23:36:20.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7false2024-04-19T23:36:20.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQ1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgRfalse2024-04-19T23:36:20.019Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipPC6NaG1ejWpptAGf0zgZ-gqpdm0DMfalse2024-04-19T23:36:20.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoECNSXdMldM0FoTJgSxtKpnhGWUHc5tfalse2024-04-19T23:36:20.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFo79-FV5lzRfFhYZKDH5zbjzjFRvFmofalse2024-04-19T23:36:19.985Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.9ZzH.1hozF9j0blCcYN54OsXVmWMTfalse2024-04-19T23:36:19.968Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnLS7759d.h0bWJMnVj7IAdVTVbZ2LzAfalse2024-04-19T23:36:20.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.SsB-LJAFCELIGc-2lad2-2jTKdmQIufalse2024-04-19T23:36:19.904Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5IMca-NQG3TDmPnhaU0a46jfFv2nxR8false2024-04-19T23:36:19.920Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3PIDP6j1VZfmBljRrKUGkIUmq4DPgifalse2024-04-19T23:36:19.849Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.sqcIpylWkdZwQp4UqpDBbyySodTBXSfalse2024-04-19T23:36:19.778Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCzz.Nb3nU2CWGe2vMCDYlTHkK18iRLfalse2024-04-19T23:36:19.760Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjudeGFtd00C8TRIOlco59Eefd7c7ffA6false2024-04-19T23:36:19.831Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4false2024-04-19T23:36:19.744Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgoQj3EN8LOJNZ9C2JDigvTcIgypIbDxfalse2024-04-19T23:36:19.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzyfalse2024-04-19T23:36:19.727Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.169NmmrUry8iq70c7.ZNdp4i5iqOCFfalse2024-04-19T23:36:19.659Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE5taibPWiUs3cuEimynrjQD8cXTCAELfalse2024-04-19T23:36:19.589Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNjFtS4LHZJDkGsaKejPMQyqPrZjbuN2false2024-04-19T23:36:19.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmTjHqLZC66BQsk7jiah7FJSChQvGdxfalse2024-04-19T23:36:19.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlKfLYYBA1E.lSLR49AMlK-Kgik5ij5Rfalse2024-04-19T23:36:19.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzwsSl6whrs-j2uo2xvsRdnhCfJSQ4Mafalse2024-04-19T23:36:19.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.172 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:39:29.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.174 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:39:29.174 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.175 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.175 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:29.175 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:29.175 DEBUG:boto:Method: GET 2024-04-19T23:39:29.175 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:39:29.175 DEBUG:boto:Data: 2024-04-19T23:39:29.175 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.175 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.175 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.175 DEBUG:boto:Params: {} 2024-04-19T23:39:29.175 DEBUG:boto:Token: None 2024-04-19T23:39:29.176 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/?versions 2024-04-19T23:39:29.176 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JxZDkPJsii5O1oeX06krkmFHCMQ= 2024-04-19T23:39:29.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JxZDkPJsii5O1oeX06krkmFHCMQ=', 'Content-Length': '0'} 2024-04-19T23:39:29.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eebf3021d8e916db-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.179 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOotrue2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjI8Qos7O61Drj6zKFSuRQg5BJ8qTn1Itfalse2024-04-19T23:36:20.273Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITUfalse2024-04-19T23:36:20.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUjbMVzWXMoq8kQHxc5fXZiC69vNXkcjfalse2024-04-19T23:36:20.222Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97false2024-04-19T23:36:20.188Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2false2024-04-19T23:36:20.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Jy1wSwswautcJ.pAM0qXnm5CTgNceOfalse2024-04-19T23:36:20.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-false2024-04-19T23:36:20.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7false2024-04-19T23:36:20.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQ1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgRfalse2024-04-19T23:36:20.019Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipPC6NaG1ejWpptAGf0zgZ-gqpdm0DMfalse2024-04-19T23:36:20.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoECNSXdMldM0FoTJgSxtKpnhGWUHc5tfalse2024-04-19T23:36:20.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFo79-FV5lzRfFhYZKDH5zbjzjFRvFmofalse2024-04-19T23:36:19.985Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.9ZzH.1hozF9j0blCcYN54OsXVmWMTfalse2024-04-19T23:36:19.968Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnLS7759d.h0bWJMnVj7IAdVTVbZ2LzAfalse2024-04-19T23:36:20.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.SsB-LJAFCELIGc-2lad2-2jTKdmQIufalse2024-04-19T23:36:19.904Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5IMca-NQG3TDmPnhaU0a46jfFv2nxR8false2024-04-19T23:36:19.920Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3PIDP6j1VZfmBljRrKUGkIUmq4DPgifalse2024-04-19T23:36:19.849Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.sqcIpylWkdZwQp4UqpDBbyySodTBXSfalse2024-04-19T23:36:19.778Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCzz.Nb3nU2CWGe2vMCDYlTHkK18iRLfalse2024-04-19T23:36:19.760Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjudeGFtd00C8TRIOlco59Eefd7c7ffA6false2024-04-19T23:36:19.831Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4false2024-04-19T23:36:19.744Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgoQj3EN8LOJNZ9C2JDigvTcIgypIbDxfalse2024-04-19T23:36:19.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzyfalse2024-04-19T23:36:19.727Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.169NmmrUry8iq70c7.ZNdp4i5iqOCFfalse2024-04-19T23:36:19.659Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE5taibPWiUs3cuEimynrjQD8cXTCAELfalse2024-04-19T23:36:19.589Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNjFtS4LHZJDkGsaKejPMQyqPrZjbuN2false2024-04-19T23:36:19.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmTjHqLZC66BQsk7jiah7FJSChQvGdxfalse2024-04-19T23:36:19.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlKfLYYBA1E.lSLR49AMlK-Kgik5ij5Rfalse2024-04-19T23:36:19.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzwsSl6whrs-j2uo2xvsRdnhCfJSQ4Mafalse2024-04-19T23:36:19.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.181 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-19T23:39:29.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:39:29.183 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.184 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.184 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:29.184 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:29.184 DEBUG:boto:Method: GET 2024-04-19T23:39:29.184 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:39:29.184 DEBUG:boto:Data: 2024-04-19T23:39:29.184 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.184 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.184 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.184 DEBUG:boto:Params: {} 2024-04-19T23:39:29.185 DEBUG:boto:Token: None 2024-04-19T23:39:29.185 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/?versions 2024-04-19T23:39:29.185 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JxZDkPJsii5O1oeX06krkmFHCMQ= 2024-04-19T23:39:29.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JxZDkPJsii5O1oeX06krkmFHCMQ=', 'Content-Length': '0'} 2024-04-19T23:39:29.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb3ed29be5f2160f-00662300b1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.189 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOotrue2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjI8Qos7O61Drj6zKFSuRQg5BJ8qTn1Itfalse2024-04-19T23:36:20.273Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITUfalse2024-04-19T23:36:20.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUjbMVzWXMoq8kQHxc5fXZiC69vNXkcjfalse2024-04-19T23:36:20.222Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97false2024-04-19T23:36:20.188Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2false2024-04-19T23:36:20.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Jy1wSwswautcJ.pAM0qXnm5CTgNceOfalse2024-04-19T23:36:20.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-false2024-04-19T23:36:20.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7false2024-04-19T23:36:20.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQ1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgRfalse2024-04-19T23:36:20.019Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipPC6NaG1ejWpptAGf0zgZ-gqpdm0DMfalse2024-04-19T23:36:20.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoECNSXdMldM0FoTJgSxtKpnhGWUHc5tfalse2024-04-19T23:36:20.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFo79-FV5lzRfFhYZKDH5zbjzjFRvFmofalse2024-04-19T23:36:19.985Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.9ZzH.1hozF9j0blCcYN54OsXVmWMTfalse2024-04-19T23:36:19.968Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnLS7759d.h0bWJMnVj7IAdVTVbZ2LzAfalse2024-04-19T23:36:20.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.SsB-LJAFCELIGc-2lad2-2jTKdmQIufalse2024-04-19T23:36:19.904Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5IMca-NQG3TDmPnhaU0a46jfFv2nxR8false2024-04-19T23:36:19.920Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3PIDP6j1VZfmBljRrKUGkIUmq4DPgifalse2024-04-19T23:36:19.849Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.sqcIpylWkdZwQp4UqpDBbyySodTBXSfalse2024-04-19T23:36:19.778Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCzz.Nb3nU2CWGe2vMCDYlTHkK18iRLfalse2024-04-19T23:36:19.760Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjudeGFtd00C8TRIOlco59Eefd7c7ffA6false2024-04-19T23:36:19.831Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4false2024-04-19T23:36:19.744Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgoQj3EN8LOJNZ9C2JDigvTcIgypIbDxfalse2024-04-19T23:36:19.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzyfalse2024-04-19T23:36:19.727Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.169NmmrUry8iq70c7.ZNdp4i5iqOCFfalse2024-04-19T23:36:19.659Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE5taibPWiUs3cuEimynrjQD8cXTCAELfalse2024-04-19T23:36:19.589Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNjFtS4LHZJDkGsaKejPMQyqPrZjbuN2false2024-04-19T23:36:19.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmTjHqLZC66BQsk7jiah7FJSChQvGdxfalse2024-04-19T23:36:19.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlKfLYYBA1E.lSLR49AMlK-Kgik5ij5Rfalse2024-04-19T23:36:19.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzwsSl6whrs-j2uo2xvsRdnhCfJSQ4Mafalse2024-04-19T23:36:19.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.190 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.190 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.190 DEBUG:boto:path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:29.190 DEBUG:boto:auth_path=/aktlji-22/?versions&max-keys=999 2024-04-19T23:39:29.191 DEBUG:boto:Method: GET 2024-04-19T23:39:29.191 DEBUG:boto:Path: /aktlji-22/?versions&max-keys=999 2024-04-19T23:39:29.191 DEBUG:boto:Data: 2024-04-19T23:39:29.191 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.191 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.191 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.191 DEBUG:boto:Params: {} 2024-04-19T23:39:29.191 DEBUG:boto:Token: None 2024-04-19T23:39:29.191 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/?versions 2024-04-19T23:39:29.191 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JxZDkPJsii5O1oeX06krkmFHCMQ= 2024-04-19T23:39:29.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JxZDkPJsii5O1oeX06krkmFHCMQ=', 'Content-Length': '0'} 2024-04-19T23:39:29.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027ba8e12f2a1768a-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.195 DEBUG:boto:b'aktlji-22999falsedummyu9OrRVGuuBh3iVUvK8xzR387aNSvARTtrue2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOotrue2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjI8Qos7O61Drj6zKFSuRQg5BJ8qTn1Itfalse2024-04-19T23:36:20.273Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjdkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITUfalse2024-04-19T23:36:20.376Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjUjbMVzWXMoq8kQHxc5fXZiC69vNXkcjfalse2024-04-19T23:36:20.222Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97false2024-04-19T23:36:20.188Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2false2024-04-19T23:36:20.205Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Jy1wSwswautcJ.pAM0qXnm5CTgNceOfalse2024-04-19T23:36:20.172Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-false2024-04-19T23:36:20.120Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7false2024-04-19T23:36:20.103Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQ1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgRfalse2024-04-19T23:36:20.019Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjipPC6NaG1ejWpptAGf0zgZ-gqpdm0DMfalse2024-04-19T23:36:20.069Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoECNSXdMldM0FoTJgSxtKpnhGWUHc5tfalse2024-04-19T23:36:20.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFo79-FV5lzRfFhYZKDH5zbjzjFRvFmofalse2024-04-19T23:36:19.985Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.9ZzH.1hozF9j0blCcYN54OsXVmWMTfalse2024-04-19T23:36:19.968Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnLS7759d.h0bWJMnVj7IAdVTVbZ2LzAfalse2024-04-19T23:36:20.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.SsB-LJAFCELIGc-2lad2-2jTKdmQIufalse2024-04-19T23:36:19.904Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5IMca-NQG3TDmPnhaU0a46jfFv2nxR8false2024-04-19T23:36:19.920Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp3PIDP6j1VZfmBljRrKUGkIUmq4DPgifalse2024-04-19T23:36:19.849Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.sqcIpylWkdZwQp4UqpDBbyySodTBXSfalse2024-04-19T23:36:19.778Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCzz.Nb3nU2CWGe2vMCDYlTHkK18iRLfalse2024-04-19T23:36:19.760Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjudeGFtd00C8TRIOlco59Eefd7c7ffA6false2024-04-19T23:36:19.831Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4false2024-04-19T23:36:19.744Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgoQj3EN8LOJNZ9C2JDigvTcIgypIbDxfalse2024-04-19T23:36:19.676Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzyfalse2024-04-19T23:36:19.727Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.169NmmrUry8iq70c7.ZNdp4i5iqOCFfalse2024-04-19T23:36:19.659Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjE5taibPWiUs3cuEimynrjQD8cXTCAELfalse2024-04-19T23:36:19.589Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNjFtS4LHZJDkGsaKejPMQyqPrZjbuN2false2024-04-19T23:36:19.605Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFmTjHqLZC66BQsk7jiah7FJSChQvGdxfalse2024-04-19T23:36:19.500Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjlKfLYYBA1E.lSLR49AMlK-Kgik5ij5Rfalse2024-04-19T23:36:19.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzwsSl6whrs-j2uo2xvsRdnhCfJSQ4Mafalse2024-04-19T23:36:19.552Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:39:29.196 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:29.196 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:29.196 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.197 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.197 DEBUG:boto:Method: GET 2024-04-19T23:39:29.197 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.197 DEBUG:boto:Data: 2024-04-19T23:39:29.197 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.197 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.197 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.197 DEBUG:boto:Params: {} 2024-04-19T23:39:29.197 DEBUG:boto:Token: None 2024-04-19T23:39:29.197 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.198 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ffl0d4ISCS7UPO+nVm0TuEqzyyM= 2024-04-19T23:39:29.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ffl0d4ISCS7UPO+nVm0TuEqzyyM=', 'Content-Length': '0'} 2024-04-19T23:39:29.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a60321ee17475045-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.200 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:29.200 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:29.200 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.200 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.200 DEBUG:boto:Method: GET 2024-04-19T23:39:29.200 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.200 DEBUG:boto:Data: 2024-04-19T23:39:29.200 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.200 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.200 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.200 DEBUG:boto:Params: {} 2024-04-19T23:39:29.201 DEBUG:boto:Token: None 2024-04-19T23:39:29.201 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.201 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ffl0d4ISCS7UPO+nVm0TuEqzyyM= 2024-04-19T23:39:29.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ffl0d4ISCS7UPO+nVm0TuEqzyyM=', 'Content-Length': '0'} 2024-04-19T23:39:29.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000095d77922934810b6-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.203 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:29.203 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:29.203 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.203 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.203 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.204 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.204 DEBUG:boto:Data: 2024-04-19T23:39:29.204 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.204 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.204 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.204 DEBUG:boto:Params: {} 2024-04-19T23:39:29.204 DEBUG:boto:Token: None 2024-04-19T23:39:29.204 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.204 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eIFT12FBtjO4N+NhmAYx6XDCuNQ= 2024-04-19T23:39:29.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eIFT12FBtjO4N+NhmAYx6XDCuNQ=', 'Content-Length': '0'} 2024-04-19T23:39:29.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee26b1e0a43912cf-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.206 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:29.207 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:29.207 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.207 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.207 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.207 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.207 DEBUG:boto:Data: 2024-04-19T23:39:29.207 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.207 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.207 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.207 DEBUG:boto:Params: {} 2024-04-19T23:39:29.207 DEBUG:boto:Token: None 2024-04-19T23:39:29.208 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.208 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eIFT12FBtjO4N+NhmAYx6XDCuNQ= 2024-04-19T23:39:29.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eIFT12FBtjO4N+NhmAYx6XDCuNQ=', 'Content-Length': '0'} 2024-04-19T23:39:29.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a88643ff6b9e2068-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:39:29.210 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:29.210 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:29.210 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.210 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.210 DEBUG:boto:Method: GET 2024-04-19T23:39:29.211 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.211 DEBUG:boto:Data: 2024-04-19T23:39:29.211 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.211 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.211 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.211 DEBUG:boto:Params: {} 2024-04-19T23:39:29.212 DEBUG:boto:Token: None 2024-04-19T23:39:29.212 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.212 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ffl0d4ISCS7UPO+nVm0TuEqzyyM= 2024-04-19T23:39:29.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ffl0d4ISCS7UPO+nVm0TuEqzyyM=', 'Content-Length': '0'} 2024-04-19T23:39:29.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f46a0664e808b83-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.214 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:29.214 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:29.215 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.215 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.215 DEBUG:boto:Method: GET 2024-04-19T23:39:29.215 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.215 DEBUG:boto:Data: 2024-04-19T23:39:29.215 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.215 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.215 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.215 DEBUG:boto:Params: {} 2024-04-19T23:39:29.215 DEBUG:boto:Token: None 2024-04-19T23:39:29.215 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.216 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ffl0d4ISCS7UPO+nVm0TuEqzyyM= 2024-04-19T23:39:29.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ffl0d4ISCS7UPO+nVm0TuEqzyyM=', 'Content-Length': '0'} 2024-04-19T23:39:29.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af6b084a7a314628-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.218 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.218 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.218 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:29.218 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:29.218 DEBUG:boto:Method: GET 2024-04-19T23:39:29.218 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:39:29.218 DEBUG:boto:Data: 2024-04-19T23:39:29.218 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.218 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.218 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.218 DEBUG:boto:Params: {} 2024-04-19T23:39:29.219 DEBUG:boto:Token: None 2024-04-19T23:39:29.219 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.219 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe8804c82dc7d2ad-00662300b1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.262 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.263 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.263 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.263 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:29.264 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:29.264 DEBUG:boto:Method: GET 2024-04-19T23:39:29.264 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:39:29.264 DEBUG:boto:Data: 2024-04-19T23:39:29.264 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.264 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.264 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.264 DEBUG:boto:Params: {} 2024-04-19T23:39:29.264 DEBUG:boto:Token: None 2024-04-19T23:39:29.265 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.265 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cd8c403c2f92195-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.268 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.268 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:29.268 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:29.268 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.268 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:39:29.268 DEBUG:boto:Data: 2024-04-19T23:39:29.268 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.268 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.269 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.269 DEBUG:boto:Params: {} 2024-04-19T23:39:29.269 DEBUG:boto:Token: None 2024-04-19T23:39:29.269 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/dummy 2024-04-19T23:39:29.269 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tA/ZJTV2M8SvFDRlladZJEm1FnA= 2024-04-19T23:39:29.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tA/ZJTV2M8SvFDRlladZJEm1FnA=', 'Content-Length': '0'} 2024-04-19T23:39:29.271 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006213d4087f3d50cd-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.272 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:29.272 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:29.272 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.272 DEBUG:boto:Path: /aktlji-22/dummy 2024-04-19T23:39:29.272 DEBUG:boto:Data: 2024-04-19T23:39:29.272 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.272 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.272 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.272 DEBUG:boto:Params: {} 2024-04-19T23:39:29.272 DEBUG:boto:Token: None 2024-04-19T23:39:29.273 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/dummy 2024-04-19T23:39:29.273 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tA/ZJTV2M8SvFDRlladZJEm1FnA= 2024-04-19T23:39:29.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tA/ZJTV2M8SvFDRlladZJEm1FnA=', 'Content-Length': '0'} 2024-04-19T23:39:29.275 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dc05a13f7b278d07-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-19T23:39:29.276 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:29.276 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:29.276 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.276 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.276 DEBUG:boto:Method: GET 2024-04-19T23:39:29.276 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.276 DEBUG:boto:Data: 2024-04-19T23:39:29.276 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.276 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.276 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.276 DEBUG:boto:Params: {} 2024-04-19T23:39:29.277 DEBUG:boto:Token: None 2024-04-19T23:39:29.277 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.277 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ffl0d4ISCS7UPO+nVm0TuEqzyyM= 2024-04-19T23:39:29.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ffl0d4ISCS7UPO+nVm0TuEqzyyM=', 'Content-Length': '0'} 2024-04-19T23:39:29.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a89f1c7a29a4d38-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.279 DEBUG:boto:path=/aktlji-22/dummy 2024-04-19T23:39:29.279 DEBUG:boto:auth_path=/aktlji-22/dummy 2024-04-19T23:39:29.279 DEBUG:boto:path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.279 DEBUG:boto:auth_path=/aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.279 DEBUG:boto:Method: GET 2024-04-19T23:39:29.279 DEBUG:boto:Path: /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.279 DEBUG:boto:Data: 2024-04-19T23:39:29.280 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.280 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.280 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.280 DEBUG:boto:Params: {} 2024-04-19T23:39:29.280 DEBUG:boto:Token: None 2024-04-19T23:39:29.280 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/dummy?versionId=u9OrRVGuuBh3iVUvK8xzR387aNSvART 2024-04-19T23:39:29.280 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ffl0d4ISCS7UPO+nVm0TuEqzyyM= 2024-04-19T23:39:29.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ffl0d4ISCS7UPO+nVm0TuEqzyyM=', 'Content-Length': '0'} 2024-04-19T23:39:29.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:35:28 GMT'), ('x-amz-version-id', 'u9OrRVGuuBh3iVUvK8xzR387aNSvART'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000097b9544ab1b725da-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.282 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.283 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.283 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:29.283 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:29.283 DEBUG:boto:Method: GET 2024-04-19T23:39:29.283 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:39:29.283 DEBUG:boto:Data: 2024-04-19T23:39:29.283 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.283 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.283 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.283 DEBUG:boto:Params: {} 2024-04-19T23:39:29.284 DEBUG:boto:Token: None 2024-04-19T23:39:29.284 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.284 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bb15d70a415e1a9-00662300b1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.330 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.331 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.331 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.331 DEBUG:boto:path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:29.331 DEBUG:boto:auth_path=/aktlji-22/?prefix=dummy 2024-04-19T23:39:29.331 DEBUG:boto:Method: GET 2024-04-19T23:39:29.331 DEBUG:boto:Path: /aktlji-22/?prefix=dummy 2024-04-19T23:39:29.331 DEBUG:boto:Data: 2024-04-19T23:39:29.332 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.332 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.332 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.332 DEBUG:boto:Params: {} 2024-04-19T23:39:29.332 DEBUG:boto:Token: None 2024-04-19T23:39:29.332 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000350a09ab77dc135-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.378 DEBUG:boto:b'aktlji-22dummy1000falsedummy2024-04-19T23:35:28.061Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:29.379 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.379 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.379 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.379 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.379 DEBUG:boto:Method: GET 2024-04-19T23:39:29.379 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.379 DEBUG:boto:Data: 2024-04-19T23:39:29.380 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.380 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.380 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.380 DEBUG:boto:Params: {} 2024-04-19T23:39:29.381 DEBUG:boto:Token: None 2024-04-19T23:39:29.381 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.381 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk= 2024-04-19T23:39:29.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk=', 'Content-Length': '0'} 2024-04-19T23:39:29.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029c35f368c331763-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.426 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.426 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.427 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.427 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.427 DEBUG:boto:Method: GET 2024-04-19T23:39:29.427 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.427 DEBUG:boto:Data: 2024-04-19T23:39:29.427 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.427 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.427 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.427 DEBUG:boto:Params: {} 2024-04-19T23:39:29.427 DEBUG:boto:Token: None 2024-04-19T23:39:29.428 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.429 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk= 2024-04-19T23:39:29.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk=', 'Content-Length': '0'} 2024-04-19T23:39:29.431 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da6f80873303ee37-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.431 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.431 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.431 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.431 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.432 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.432 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.432 DEBUG:boto:Data: 2024-04-19T23:39:29.432 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.432 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.432 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.432 DEBUG:boto:Params: {} 2024-04-19T23:39:29.432 DEBUG:boto:Token: None 2024-04-19T23:39:29.432 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.433 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yVmgoojXlmpOMkZV1OWZZjFUWO0= 2024-04-19T23:39:29.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yVmgoojXlmpOMkZV1OWZZjFUWO0=', 'Content-Length': '0'} 2024-04-19T23:39:29.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006400f69766783b77-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.435 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.435 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.435 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.435 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.435 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.435 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.435 DEBUG:boto:Data: 2024-04-19T23:39:29.435 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.435 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.435 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.436 DEBUG:boto:Params: {} 2024-04-19T23:39:29.436 DEBUG:boto:Token: None 2024-04-19T23:39:29.436 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.436 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yVmgoojXlmpOMkZV1OWZZjFUWO0= 2024-04-19T23:39:29.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yVmgoojXlmpOMkZV1OWZZjFUWO0=', 'Content-Length': '0'} 2024-04-19T23:39:29.438 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5957962fa568da8-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:29.438 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.438 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.439 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.439 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.439 DEBUG:boto:Method: GET 2024-04-19T23:39:29.439 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.439 DEBUG:boto:Data: 2024-04-19T23:39:29.439 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.439 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.439 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.439 DEBUG:boto:Params: {} 2024-04-19T23:39:29.439 DEBUG:boto:Token: None 2024-04-19T23:39:29.440 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.440 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk= 2024-04-19T23:39:29.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk=', 'Content-Length': '0'} 2024-04-19T23:39:29.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ee37f6317f487a9-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.482 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.482 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.483 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.483 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.483 DEBUG:boto:Method: GET 2024-04-19T23:39:29.483 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.483 DEBUG:boto:Data: 2024-04-19T23:39:29.483 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.483 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.483 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.483 DEBUG:boto:Params: {} 2024-04-19T23:39:29.483 DEBUG:boto:Token: None 2024-04-19T23:39:29.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk= 2024-04-19T23:39:29.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk=', 'Content-Length': '0'} 2024-04-19T23:39:29.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052e9c81f98fd533f-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.486 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.486 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.486 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.486 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.487 DEBUG:boto:Method: GET 2024-04-19T23:39:29.487 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.487 DEBUG:boto:Data: 2024-04-19T23:39:29.487 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.487 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.487 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.487 DEBUG:boto:Params: {} 2024-04-19T23:39:29.487 DEBUG:boto:Token: None 2024-04-19T23:39:29.487 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.487 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baf0190b663aecc3-00662300b1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.490 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.491 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.491 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.491 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.491 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.491 DEBUG:boto:Method: GET 2024-04-19T23:39:29.491 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.491 DEBUG:boto:Data: 2024-04-19T23:39:29.491 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.492 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.492 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.492 DEBUG:boto:Params: {} 2024-04-19T23:39:29.492 DEBUG:boto:Token: None 2024-04-19T23:39:29.492 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.492 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d0ae9f7fae3f650-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.495 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.495 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.496 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.496 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.496 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:29.496 DEBUG:boto:Data: 2024-04-19T23:39:29.496 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.496 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.496 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.496 DEBUG:boto:Params: {} 2024-04-19T23:39:29.496 DEBUG:boto:Token: None 2024-04-19T23:39:29.497 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj 2024-04-19T23:39:29.497 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE= 2024-04-19T23:39:29.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE=', 'Content-Length': '0'} 2024-04-19T23:39:29.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8b315b887e47b79-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.500 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.500 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.500 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.501 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:29.501 DEBUG:boto:Data: 2024-04-19T23:39:29.501 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.501 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.501 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.501 DEBUG:boto:Params: {} 2024-04-19T23:39:29.501 DEBUG:boto:Token: None 2024-04-19T23:39:29.501 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj 2024-04-19T23:39:29.501 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE= 2024-04-19T23:39:29.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE=', 'Content-Length': '0'} 2024-04-19T23:39:29.505 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db7f49df959c6c63-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:29.505 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.505 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.505 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.505 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.505 DEBUG:boto:Method: GET 2024-04-19T23:39:29.505 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.506 DEBUG:boto:Data: 2024-04-19T23:39:29.506 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.506 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.506 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.506 DEBUG:boto:Params: {} 2024-04-19T23:39:29.506 DEBUG:boto:Token: None 2024-04-19T23:39:29.506 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.506 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk= 2024-04-19T23:39:29.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk=', 'Content-Length': '0'} 2024-04-19T23:39:29.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b18f371f9cc0f2c-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.550 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.551 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.551 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.551 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.551 DEBUG:boto:Method: GET 2024-04-19T23:39:29.551 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.551 DEBUG:boto:Data: 2024-04-19T23:39:29.551 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.551 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.551 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.552 DEBUG:boto:Params: {} 2024-04-19T23:39:29.552 DEBUG:boto:Token: None 2024-04-19T23:39:29.552 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.552 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk= 2024-04-19T23:39:29.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk=', 'Content-Length': '0'} 2024-04-19T23:39:29.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d53a824a9399343d-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.554 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.555 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.555 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.555 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.555 DEBUG:boto:Method: GET 2024-04-19T23:39:29.555 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.555 DEBUG:boto:Data: 2024-04-19T23:39:29.555 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.555 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.555 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.555 DEBUG:boto:Params: {} 2024-04-19T23:39:29.555 DEBUG:boto:Token: None 2024-04-19T23:39:29.556 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.556 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c6603d749746f83-00662300b1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.558 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.559 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.559 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.559 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.559 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.559 DEBUG:boto:Method: GET 2024-04-19T23:39:29.559 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.560 DEBUG:boto:Data: 2024-04-19T23:39:29.560 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.560 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.560 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.560 DEBUG:boto:Params: {} 2024-04-19T23:39:29.560 DEBUG:boto:Token: None 2024-04-19T23:39:29.560 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d67e10c4831e9ee-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.564 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:29.564 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.564 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.565 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.565 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.565 DEBUG:boto:Method: GET 2024-04-19T23:39:29.565 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.565 DEBUG:boto:Data: 2024-04-19T23:39:29.565 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.565 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.565 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.565 DEBUG:boto:Params: {} 2024-04-19T23:39:29.565 DEBUG:boto:Token: None 2024-04-19T23:39:29.566 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.566 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lg42d9mJ7zTfgqNr3hmBBH6sxa4= 2024-04-19T23:39:29.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lg42d9mJ7zTfgqNr3hmBBH6sxa4=', 'Content-Length': '0'} 2024-04-19T23:39:29.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009efdedc2e613f1fe-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.568 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.568 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.568 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.568 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.568 DEBUG:boto:Method: GET 2024-04-19T23:39:29.568 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.569 DEBUG:boto:Data: 2024-04-19T23:39:29.569 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.569 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.569 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.569 DEBUG:boto:Params: {} 2024-04-19T23:39:29.569 DEBUG:boto:Token: None 2024-04-19T23:39:29.569 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.569 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lg42d9mJ7zTfgqNr3hmBBH6sxa4= 2024-04-19T23:39:29.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lg42d9mJ7zTfgqNr3hmBBH6sxa4=', 'Content-Length': '0'} 2024-04-19T23:39:29.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a487e50d1996a07-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.571 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.571 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.572 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.572 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.572 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.572 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.572 DEBUG:boto:Data: 2024-04-19T23:39:29.572 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.572 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.572 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.572 DEBUG:boto:Params: {} 2024-04-19T23:39:29.572 DEBUG:boto:Token: None 2024-04-19T23:39:29.572 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.573 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DtJmS0dRgAEbbrUNLhVcYFAPZak= 2024-04-19T23:39:29.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DtJmS0dRgAEbbrUNLhVcYFAPZak=', 'Content-Length': '0'} 2024-04-19T23:39:29.574 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a7088c7da8690a4-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.575 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.575 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.575 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.575 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.575 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.575 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.575 DEBUG:boto:Data: 2024-04-19T23:39:29.575 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.576 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.576 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.576 DEBUG:boto:Params: {} 2024-04-19T23:39:29.576 DEBUG:boto:Token: None 2024-04-19T23:39:29.576 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.576 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DtJmS0dRgAEbbrUNLhVcYFAPZak= 2024-04-19T23:39:29.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DtJmS0dRgAEbbrUNLhVcYFAPZak=', 'Content-Length': '0'} 2024-04-19T23:39:29.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbdb4a17845145c4-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:29.582 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.582 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.582 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.582 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.582 DEBUG:boto:Method: GET 2024-04-19T23:39:29.582 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.582 DEBUG:boto:Data: 2024-04-19T23:39:29.582 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.582 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.583 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.583 DEBUG:boto:Params: {} 2024-04-19T23:39:29.583 DEBUG:boto:Token: None 2024-04-19T23:39:29.583 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.583 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lg42d9mJ7zTfgqNr3hmBBH6sxa4= 2024-04-19T23:39:29.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lg42d9mJ7zTfgqNr3hmBBH6sxa4=', 'Content-Length': '0'} 2024-04-19T23:39:29.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cdd44a592182fd6-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.626 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.627 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.627 DEBUG:boto:path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.627 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.627 DEBUG:boto:Method: GET 2024-04-19T23:39:29.627 DEBUG:boto:Path: /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.627 DEBUG:boto:Data: 2024-04-19T23:39:29.627 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.627 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.627 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.627 DEBUG:boto:Params: {} 2024-04-19T23:39:29.627 DEBUG:boto:Token: None 2024-04-19T23:39:29.628 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It 2024-04-19T23:39:29.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lg42d9mJ7zTfgqNr3hmBBH6sxa4= 2024-04-19T23:39:29.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lg42d9mJ7zTfgqNr3hmBBH6sxa4=', 'Content-Length': '0'} 2024-04-19T23:39:29.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'I8Qos7O61Drj6zKFSuRQg5BJ8qTn1It'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f580fe16f953059b-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.674 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.675 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.675 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.675 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.675 DEBUG:boto:Method: GET 2024-04-19T23:39:29.675 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.675 DEBUG:boto:Data: 2024-04-19T23:39:29.675 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.675 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.675 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.675 DEBUG:boto:Params: {} 2024-04-19T23:39:29.675 DEBUG:boto:Token: None 2024-04-19T23:39:29.676 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc69e28425891cd9-00662300b1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.679 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.679 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.679 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.679 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.679 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.680 DEBUG:boto:Method: GET 2024-04-19T23:39:29.680 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.680 DEBUG:boto:Data: 2024-04-19T23:39:29.680 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.680 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.680 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.680 DEBUG:boto:Params: {} 2024-04-19T23:39:29.680 DEBUG:boto:Token: None 2024-04-19T23:39:29.680 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.680 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000533d2beddd2ad8c6-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.684 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.684 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.684 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.684 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.684 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:29.684 DEBUG:boto:Data: 2024-04-19T23:39:29.684 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.685 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.685 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.685 DEBUG:boto:Params: {} 2024-04-19T23:39:29.685 DEBUG:boto:Token: None 2024-04-19T23:39:29.685 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj 2024-04-19T23:39:29.685 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE= 2024-04-19T23:39:29.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE=', 'Content-Length': '0'} 2024-04-19T23:39:29.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d34e57beab8f41cc-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.689 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.689 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.689 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.689 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:29.689 DEBUG:boto:Data: 2024-04-19T23:39:29.689 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.689 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.689 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.689 DEBUG:boto:Params: {} 2024-04-19T23:39:29.690 DEBUG:boto:Token: None 2024-04-19T23:39:29.690 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj 2024-04-19T23:39:29.690 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE= 2024-04-19T23:39:29.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE=', 'Content-Length': '0'} 2024-04-19T23:39:29.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008da6c99d4b643cde-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:29.693 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.694 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.694 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.694 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.694 DEBUG:boto:Method: GET 2024-04-19T23:39:29.694 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.694 DEBUG:boto:Data: 2024-04-19T23:39:29.694 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.694 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.694 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.694 DEBUG:boto:Params: {} 2024-04-19T23:39:29.694 DEBUG:boto:Token: None 2024-04-19T23:39:29.695 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.695 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk= 2024-04-19T23:39:29.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk=', 'Content-Length': '0'} 2024-04-19T23:39:29.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093fb83427ae15c0e-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.742 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.742 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.743 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.743 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.743 DEBUG:boto:Method: GET 2024-04-19T23:39:29.743 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.743 DEBUG:boto:Data: 2024-04-19T23:39:29.743 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.743 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.743 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.743 DEBUG:boto:Params: {} 2024-04-19T23:39:29.743 DEBUG:boto:Token: None 2024-04-19T23:39:29.744 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.744 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk= 2024-04-19T23:39:29.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk=', 'Content-Length': '0'} 2024-04-19T23:39:29.746 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000048b8cd9396a0809-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.746 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.746 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.746 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.746 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.746 DEBUG:boto:Method: GET 2024-04-19T23:39:29.747 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.747 DEBUG:boto:Data: 2024-04-19T23:39:29.747 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.747 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.747 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.747 DEBUG:boto:Params: {} 2024-04-19T23:39:29.747 DEBUG:boto:Token: None 2024-04-19T23:39:29.747 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.747 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fc1d4115d8ca3c8-00662300b1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.794 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.795 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.795 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.795 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.795 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.795 DEBUG:boto:Method: GET 2024-04-19T23:39:29.795 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.795 DEBUG:boto:Data: 2024-04-19T23:39:29.796 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.796 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.796 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.796 DEBUG:boto:Params: {} 2024-04-19T23:39:29.796 DEBUG:boto:Token: None 2024-04-19T23:39:29.796 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.796 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c9054457cda4850-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.799 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:29.800 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.800 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.800 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.800 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.800 DEBUG:boto:Method: GET 2024-04-19T23:39:29.801 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.801 DEBUG:boto:Data: 2024-04-19T23:39:29.801 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.801 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.801 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.801 DEBUG:boto:Params: {} 2024-04-19T23:39:29.801 DEBUG:boto:Token: None 2024-04-19T23:39:29.801 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.801 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaT/2PDcNfZO71scPpTY2EvKW7A= 2024-04-19T23:39:29.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaT/2PDcNfZO71scPpTY2EvKW7A=', 'Content-Length': '0'} 2024-04-19T23:39:29.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff24b3190845a9b8-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.804 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.804 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.804 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.804 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.804 DEBUG:boto:Method: GET 2024-04-19T23:39:29.804 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.804 DEBUG:boto:Data: 2024-04-19T23:39:29.804 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.804 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.804 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.804 DEBUG:boto:Params: {} 2024-04-19T23:39:29.805 DEBUG:boto:Token: None 2024-04-19T23:39:29.805 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.805 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaT/2PDcNfZO71scPpTY2EvKW7A= 2024-04-19T23:39:29.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaT/2PDcNfZO71scPpTY2EvKW7A=', 'Content-Length': '0'} 2024-04-19T23:39:29.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3ec044b286e5402-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.850 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.850 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.851 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.851 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.851 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.851 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.851 DEBUG:boto:Data: 2024-04-19T23:39:29.851 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.851 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.851 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.851 DEBUG:boto:Params: {} 2024-04-19T23:39:29.851 DEBUG:boto:Token: None 2024-04-19T23:39:29.852 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.852 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:05JiBrsrnSco/zCexfOzsJPJ5Ig= 2024-04-19T23:39:29.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:05JiBrsrnSco/zCexfOzsJPJ5Ig=', 'Content-Length': '0'} 2024-04-19T23:39:29.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b8958733df81060-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.854 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.854 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.854 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.854 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.854 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.854 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.854 DEBUG:boto:Data: 2024-04-19T23:39:29.855 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.855 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.855 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.855 DEBUG:boto:Params: {} 2024-04-19T23:39:29.855 DEBUG:boto:Token: None 2024-04-19T23:39:29.855 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.855 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:05JiBrsrnSco/zCexfOzsJPJ5Ig= 2024-04-19T23:39:29.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:05JiBrsrnSco/zCexfOzsJPJ5Ig=', 'Content-Length': '0'} 2024-04-19T23:39:29.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043b62189c5d05a4b-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:29.857 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.857 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.858 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.858 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.858 DEBUG:boto:Method: GET 2024-04-19T23:39:29.858 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.858 DEBUG:boto:Data: 2024-04-19T23:39:29.858 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.858 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.858 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.858 DEBUG:boto:Params: {} 2024-04-19T23:39:29.858 DEBUG:boto:Token: None 2024-04-19T23:39:29.859 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.859 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaT/2PDcNfZO71scPpTY2EvKW7A= 2024-04-19T23:39:29.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaT/2PDcNfZO71scPpTY2EvKW7A=', 'Content-Length': '0'} 2024-04-19T23:39:29.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fef83ac8b9d9b0c5-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.861 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.861 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.861 DEBUG:boto:path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.861 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.861 DEBUG:boto:Method: GET 2024-04-19T23:39:29.861 DEBUG:boto:Path: /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.861 DEBUG:boto:Data: 2024-04-19T23:39:29.861 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.861 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.862 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.862 DEBUG:boto:Params: {} 2024-04-19T23:39:29.862 DEBUG:boto:Token: None 2024-04-19T23:39:29.862 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU 2024-04-19T23:39:29.862 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gaT/2PDcNfZO71scPpTY2EvKW7A= 2024-04-19T23:39:29.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gaT/2PDcNfZO71scPpTY2EvKW7A=', 'Content-Length': '0'} 2024-04-19T23:39:29.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'dkcMQ0ll5pNi9w6tvBNOxyf3pJS6ITU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057501d09bc05e30c-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.906 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.907 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.907 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.907 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.907 DEBUG:boto:Method: GET 2024-04-19T23:39:29.907 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.907 DEBUG:boto:Data: 2024-04-19T23:39:29.907 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.907 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.907 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.907 DEBUG:boto:Params: {} 2024-04-19T23:39:29.908 DEBUG:boto:Token: None 2024-04-19T23:39:29.908 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.908 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bb772dc02cf7e53-00662300b1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.911 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.911 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.911 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.911 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.912 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.912 DEBUG:boto:Method: GET 2024-04-19T23:39:29.912 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.912 DEBUG:boto:Data: 2024-04-19T23:39:29.912 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.912 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.912 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.912 DEBUG:boto:Params: {} 2024-04-19T23:39:29.912 DEBUG:boto:Token: None 2024-04-19T23:39:29.912 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.913 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081051219e3bc97c0-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.916 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.916 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.916 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.916 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.916 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:29.916 DEBUG:boto:Data: 2024-04-19T23:39:29.917 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.917 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.917 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.917 DEBUG:boto:Params: {} 2024-04-19T23:39:29.917 DEBUG:boto:Token: None 2024-04-19T23:39:29.917 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj 2024-04-19T23:39:29.917 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE= 2024-04-19T23:39:29.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE=', 'Content-Length': '0'} 2024-04-19T23:39:29.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000337a6aa185f42f48-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.921 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.921 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.921 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.921 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:29.921 DEBUG:boto:Data: 2024-04-19T23:39:29.921 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.921 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.921 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.922 DEBUG:boto:Params: {} 2024-04-19T23:39:29.922 DEBUG:boto:Token: None 2024-04-19T23:39:29.922 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj 2024-04-19T23:39:29.922 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE= 2024-04-19T23:39:29.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GDI0OoK394gz+13F+4gZwU+hhLE=', 'Content-Length': '0'} 2024-04-19T23:39:29.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa9ee31ec02a572b-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:29.925 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.926 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.926 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.926 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.926 DEBUG:boto:Method: GET 2024-04-19T23:39:29.926 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.926 DEBUG:boto:Data: 2024-04-19T23:39:29.926 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.926 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.926 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.926 DEBUG:boto:Params: {} 2024-04-19T23:39:29.927 DEBUG:boto:Token: None 2024-04-19T23:39:29.927 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.927 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk= 2024-04-19T23:39:29.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk=', 'Content-Length': '0'} 2024-04-19T23:39:29.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059f3117f6238b037-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.970 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.971 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.971 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.971 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.971 DEBUG:boto:Method: GET 2024-04-19T23:39:29.971 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.971 DEBUG:boto:Data: 2024-04-19T23:39:29.971 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.971 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.971 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.971 DEBUG:boto:Params: {} 2024-04-19T23:39:29.972 DEBUG:boto:Token: None 2024-04-19T23:39:29.972 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:29.972 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk= 2024-04-19T23:39:29.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GThsJsK4RvxrmaOM5SxMdL6Hduk=', 'Content-Length': '0'} 2024-04-19T23:39:29.974 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf41a70581757d2b-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.974 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.974 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.974 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.974 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.974 DEBUG:boto:Method: GET 2024-04-19T23:39:29.975 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.975 DEBUG:boto:Data: 2024-04-19T23:39:29.975 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.975 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.975 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.975 DEBUG:boto:Params: {} 2024-04-19T23:39:29.975 DEBUG:boto:Token: None 2024-04-19T23:39:29.975 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.975 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd71938f815a7933-00662300b1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.978 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.979 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:29.979 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:29.979 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.979 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:29.979 DEBUG:boto:Method: GET 2024-04-19T23:39:29.979 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:29.979 DEBUG:boto:Data: 2024-04-19T23:39:29.979 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.979 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.980 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.980 DEBUG:boto:Params: {} 2024-04-19T23:39:29.980 DEBUG:boto:Token: None 2024-04-19T23:39:29.980 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/ 2024-04-19T23:39:29.980 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0= 2024-04-19T23:39:29.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O8N1vkWVTRMjBSKCV0gi9RJf9M0=', 'Content-Length': '0'} 2024-04-19T23:39:29.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000631ecfb0eb31086e-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.983 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:29.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:29.984 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.984 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.984 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.984 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.984 DEBUG:boto:Method: GET 2024-04-19T23:39:29.984 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.984 DEBUG:boto:Data: 2024-04-19T23:39:29.984 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.984 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.984 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.984 DEBUG:boto:Params: {} 2024-04-19T23:39:29.985 DEBUG:boto:Token: None 2024-04-19T23:39:29.985 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.985 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:G0EZfiQ9xM8uWQXeZBx9zSAF5tI= 2024-04-19T23:39:29.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:G0EZfiQ9xM8uWQXeZBx9zSAF5tI=', 'Content-Length': '0'} 2024-04-19T23:39:29.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028ef896f68927604-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.987 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.987 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.987 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.987 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.987 DEBUG:boto:Method: GET 2024-04-19T23:39:29.987 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.987 DEBUG:boto:Data: 2024-04-19T23:39:29.988 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.988 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.988 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.988 DEBUG:boto:Params: {} 2024-04-19T23:39:29.988 DEBUG:boto:Token: None 2024-04-19T23:39:29.988 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.988 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:G0EZfiQ9xM8uWQXeZBx9zSAF5tI= 2024-04-19T23:39:29.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:G0EZfiQ9xM8uWQXeZBx9zSAF5tI=', 'Content-Length': '0'} 2024-04-19T23:39:29.990 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d37bbceaf3744658-00662300b1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.991 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.991 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.991 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.991 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.991 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.991 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.991 DEBUG:boto:Data: 2024-04-19T23:39:29.992 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.992 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.992 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.992 DEBUG:boto:Params: {} 2024-04-19T23:39:29.992 DEBUG:boto:Token: None 2024-04-19T23:39:29.992 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.992 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/qwZyb2olScoDrjiKnM/dUMTw6M= 2024-04-19T23:39:29.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/qwZyb2olScoDrjiKnM/dUMTw6M=', 'Content-Length': '0'} 2024-04-19T23:39:29.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000990ae746d182980c-00662300b1-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.994 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.995 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.995 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.995 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.995 DEBUG:boto:Method: HEAD 2024-04-19T23:39:29.995 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.995 DEBUG:boto:Data: 2024-04-19T23:39:29.995 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.995 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:29.995 DEBUG:boto:Port: 8000 2024-04-19T23:39:29.995 DEBUG:boto:Params: {} 2024-04-19T23:39:29.995 DEBUG:boto:Token: None 2024-04-19T23:39:29.996 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.996 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/qwZyb2olScoDrjiKnM/dUMTw6M= 2024-04-19T23:39:29.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/qwZyb2olScoDrjiKnM/dUMTw6M=', 'Content-Length': '0'} 2024-04-19T23:39:29.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fe7bec94f9fabb7-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:29.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:29.998 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:29.998 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:29.998 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.998 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.998 DEBUG:boto:Method: GET 2024-04-19T23:39:29.998 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.998 DEBUG:boto:Data: 2024-04-19T23:39:29.999 DEBUG:boto:Headers: {} 2024-04-19T23:39:29.999 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:29.999 DEBUG:boto:Port: 8001 2024-04-19T23:39:29.999 DEBUG:boto:Params: {} 2024-04-19T23:39:29.999 DEBUG:boto:Token: None 2024-04-19T23:39:29.999 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:29 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:29.999 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:G0EZfiQ9xM8uWQXeZBx9zSAF5tI= 2024-04-19T23:39:29.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:G0EZfiQ9xM8uWQXeZBx9zSAF5tI=', 'Content-Length': '0'} 2024-04-19T23:39:30.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007efeab4987462be6-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.042 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.043 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.043 DEBUG:boto:path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:30.043 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:30.043 DEBUG:boto:Method: GET 2024-04-19T23:39:30.043 DEBUG:boto:Path: /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:30.043 DEBUG:boto:Data: 2024-04-19T23:39:30.043 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.044 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.044 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.044 DEBUG:boto:Params: {} 2024-04-19T23:39:30.044 DEBUG:boto:Token: None 2024-04-19T23:39:30.045 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj 2024-04-19T23:39:30.045 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SsRivoYfX9JtuOMn37PLvrfvZLg= 2024-04-19T23:39:30.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SsRivoYfX9JtuOMn37PLvrfvZLg=', 'Content-Length': '0'} 2024-04-19T23:39:30.047 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'UjbMVzWXMoq8kQHxc5fXZiC69vNXkcj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fec4b1fc4bf37ff3-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.094 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.095 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.095 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.095 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.095 DEBUG:boto:Method: GET 2024-04-19T23:39:30.095 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.095 DEBUG:boto:Data: 2024-04-19T23:39:30.095 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.095 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.095 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.095 DEBUG:boto:Params: {} 2024-04-19T23:39:30.096 DEBUG:boto:Token: None 2024-04-19T23:39:30.096 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.096 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c91416bfe776323e-00662300b2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.099 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.099 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.099 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.100 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.100 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.100 DEBUG:boto:Method: GET 2024-04-19T23:39:30.100 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.100 DEBUG:boto:Data: 2024-04-19T23:39:30.100 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.100 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.101 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.101 DEBUG:boto:Params: {} 2024-04-19T23:39:30.101 DEBUG:boto:Token: None 2024-04-19T23:39:30.101 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.101 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5fd32418bebc97a-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.150 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.151 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.151 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.151 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.151 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:30.151 DEBUG:boto:Data: 2024-04-19T23:39:30.151 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.152 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.152 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.152 DEBUG:boto:Params: {} 2024-04-19T23:39:30.152 DEBUG:boto:Token: None 2024-04-19T23:39:30.152 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj 2024-04-19T23:39:30.152 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y= 2024-04-19T23:39:30.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y=', 'Content-Length': '0'} 2024-04-19T23:39:30.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ee839692482aae2-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.156 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.156 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.156 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.156 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:30.157 DEBUG:boto:Data: 2024-04-19T23:39:30.157 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.157 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.157 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.157 DEBUG:boto:Params: {} 2024-04-19T23:39:30.157 DEBUG:boto:Token: None 2024-04-19T23:39:30.157 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj 2024-04-19T23:39:30.157 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y= 2024-04-19T23:39:30.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y=', 'Content-Length': '0'} 2024-04-19T23:39:30.161 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0cbe457f0cbcbf9-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.161 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.161 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.161 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.161 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.161 DEBUG:boto:Method: GET 2024-04-19T23:39:30.162 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.162 DEBUG:boto:Data: 2024-04-19T23:39:30.162 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.162 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.162 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.162 DEBUG:boto:Params: {} 2024-04-19T23:39:30.162 DEBUG:boto:Token: None 2024-04-19T23:39:30.162 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.162 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I= 2024-04-19T23:39:30.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I=', 'Content-Length': '0'} 2024-04-19T23:39:30.164 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dde3e25a40b731c0-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.206 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.207 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.207 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.207 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.207 DEBUG:boto:Method: GET 2024-04-19T23:39:30.207 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.207 DEBUG:boto:Data: 2024-04-19T23:39:30.207 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.207 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.207 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.208 DEBUG:boto:Params: {} 2024-04-19T23:39:30.208 DEBUG:boto:Token: None 2024-04-19T23:39:30.208 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.208 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I= 2024-04-19T23:39:30.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I=', 'Content-Length': '0'} 2024-04-19T23:39:30.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac37d39159695a66-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.211 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.211 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.211 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.211 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.211 DEBUG:boto:Method: GET 2024-04-19T23:39:30.211 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.211 DEBUG:boto:Data: 2024-04-19T23:39:30.211 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.211 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.211 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.211 DEBUG:boto:Params: {} 2024-04-19T23:39:30.212 DEBUG:boto:Token: None 2024-04-19T23:39:30.212 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.212 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e70e3f6714a5fcf-00662300b2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.215 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.215 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.215 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.215 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.215 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.215 DEBUG:boto:Method: GET 2024-04-19T23:39:30.216 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.216 DEBUG:boto:Data: 2024-04-19T23:39:30.216 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.216 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.216 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.216 DEBUG:boto:Params: {} 2024-04-19T23:39:30.216 DEBUG:boto:Token: None 2024-04-19T23:39:30.216 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.217 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7c7a71e6a3e47ba-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.220 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.220 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.220 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.220 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.220 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.220 DEBUG:boto:Method: GET 2024-04-19T23:39:30.221 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.221 DEBUG:boto:Data: 2024-04-19T23:39:30.221 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.221 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.221 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.221 DEBUG:boto:Params: {} 2024-04-19T23:39:30.221 DEBUG:boto:Token: None 2024-04-19T23:39:30.221 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.222 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DTgCXmSBVri+8gtkayR6aLWhk3I= 2024-04-19T23:39:30.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DTgCXmSBVri+8gtkayR6aLWhk3I=', 'Content-Length': '0'} 2024-04-19T23:39:30.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e52233a8bc62347-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.270 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.271 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.271 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.271 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.271 DEBUG:boto:Method: GET 2024-04-19T23:39:30.271 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.271 DEBUG:boto:Data: 2024-04-19T23:39:30.271 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.271 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.272 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.272 DEBUG:boto:Params: {} 2024-04-19T23:39:30.272 DEBUG:boto:Token: None 2024-04-19T23:39:30.272 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.272 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DTgCXmSBVri+8gtkayR6aLWhk3I= 2024-04-19T23:39:30.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DTgCXmSBVri+8gtkayR6aLWhk3I=', 'Content-Length': '0'} 2024-04-19T23:39:30.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b98a091944a84c1-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.318 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.319 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.319 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.319 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.319 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.319 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.319 DEBUG:boto:Data: 2024-04-19T23:39:30.319 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.319 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.319 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.319 DEBUG:boto:Params: {} 2024-04-19T23:39:30.320 DEBUG:boto:Token: None 2024-04-19T23:39:30.320 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.320 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X7BU5S3qBMUhpoMpUW7DXuyh7yw= 2024-04-19T23:39:30.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X7BU5S3qBMUhpoMpUW7DXuyh7yw=', 'Content-Length': '0'} 2024-04-19T23:39:30.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc04c5f26aafb15b-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.323 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.323 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.323 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.323 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.323 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.323 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.323 DEBUG:boto:Data: 2024-04-19T23:39:30.323 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.323 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.323 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.323 DEBUG:boto:Params: {} 2024-04-19T23:39:30.324 DEBUG:boto:Token: None 2024-04-19T23:39:30.324 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.324 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X7BU5S3qBMUhpoMpUW7DXuyh7yw= 2024-04-19T23:39:30.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X7BU5S3qBMUhpoMpUW7DXuyh7yw=', 'Content-Length': '0'} 2024-04-19T23:39:30.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4a3e3ebb504bec3-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.326 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.326 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.326 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.326 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.326 DEBUG:boto:Method: GET 2024-04-19T23:39:30.326 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.327 DEBUG:boto:Data: 2024-04-19T23:39:30.327 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.327 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.327 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.327 DEBUG:boto:Params: {} 2024-04-19T23:39:30.327 DEBUG:boto:Token: None 2024-04-19T23:39:30.327 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.327 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DTgCXmSBVri+8gtkayR6aLWhk3I= 2024-04-19T23:39:30.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DTgCXmSBVri+8gtkayR6aLWhk3I=', 'Content-Length': '0'} 2024-04-19T23:39:30.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e35bc0a8ac1fa58d-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.370 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.371 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.371 DEBUG:boto:path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.371 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.371 DEBUG:boto:Method: GET 2024-04-19T23:39:30.371 DEBUG:boto:Path: /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.371 DEBUG:boto:Data: 2024-04-19T23:39:30.371 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.371 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.371 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.371 DEBUG:boto:Params: {} 2024-04-19T23:39:30.372 DEBUG:boto:Token: None 2024-04-19T23:39:30.372 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97 2024-04-19T23:39:30.372 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DTgCXmSBVri+8gtkayR6aLWhk3I= 2024-04-19T23:39:30.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DTgCXmSBVri+8gtkayR6aLWhk3I=', 'Content-Length': '0'} 2024-04-19T23:39:30.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'gPE3Rb2ALw0Mj7JwzdDjCVySbtPGF97'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008631cd3a466d9e3-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.418 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.418 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.419 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.419 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.419 DEBUG:boto:Method: GET 2024-04-19T23:39:30.419 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.419 DEBUG:boto:Data: 2024-04-19T23:39:30.419 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.419 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.419 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.419 DEBUG:boto:Params: {} 2024-04-19T23:39:30.419 DEBUG:boto:Token: None 2024-04-19T23:39:30.420 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.420 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f95031be9aa6ae6e-00662300b2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.423 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.423 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.423 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.423 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.423 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.424 DEBUG:boto:Method: GET 2024-04-19T23:39:30.424 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.424 DEBUG:boto:Data: 2024-04-19T23:39:30.424 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.424 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.424 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.424 DEBUG:boto:Params: {} 2024-04-19T23:39:30.424 DEBUG:boto:Token: None 2024-04-19T23:39:30.424 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.424 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c66499dad3d2a4a5-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.427 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.428 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.428 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.428 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.428 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:30.428 DEBUG:boto:Data: 2024-04-19T23:39:30.428 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.428 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.428 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.428 DEBUG:boto:Params: {} 2024-04-19T23:39:30.429 DEBUG:boto:Token: None 2024-04-19T23:39:30.429 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj 2024-04-19T23:39:30.429 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y= 2024-04-19T23:39:30.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y=', 'Content-Length': '0'} 2024-04-19T23:39:30.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3584b43660bbf1a-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.432 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.433 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.433 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.433 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:30.433 DEBUG:boto:Data: 2024-04-19T23:39:30.433 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.433 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.433 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.433 DEBUG:boto:Params: {} 2024-04-19T23:39:30.433 DEBUG:boto:Token: None 2024-04-19T23:39:30.433 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj 2024-04-19T23:39:30.434 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y= 2024-04-19T23:39:30.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y=', 'Content-Length': '0'} 2024-04-19T23:39:30.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d051eb142317c4a-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.437 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.438 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.438 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.438 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.438 DEBUG:boto:Method: GET 2024-04-19T23:39:30.438 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.438 DEBUG:boto:Data: 2024-04-19T23:39:30.438 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.438 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.438 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.438 DEBUG:boto:Params: {} 2024-04-19T23:39:30.439 DEBUG:boto:Token: None 2024-04-19T23:39:30.439 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.439 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I= 2024-04-19T23:39:30.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I=', 'Content-Length': '0'} 2024-04-19T23:39:30.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c934bbcc2560b01-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.482 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.482 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.483 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.483 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.483 DEBUG:boto:Method: GET 2024-04-19T23:39:30.483 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.483 DEBUG:boto:Data: 2024-04-19T23:39:30.483 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.483 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.483 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.483 DEBUG:boto:Params: {} 2024-04-19T23:39:30.483 DEBUG:boto:Token: None 2024-04-19T23:39:30.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I= 2024-04-19T23:39:30.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I=', 'Content-Length': '0'} 2024-04-19T23:39:30.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078cbcae533bddafd-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.486 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.486 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.486 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.487 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.487 DEBUG:boto:Method: GET 2024-04-19T23:39:30.487 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.487 DEBUG:boto:Data: 2024-04-19T23:39:30.487 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.487 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.487 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.487 DEBUG:boto:Params: {} 2024-04-19T23:39:30.487 DEBUG:boto:Token: None 2024-04-19T23:39:30.487 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a42b1fd385c32338-00662300b2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.491 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.492 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.492 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.492 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.492 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.492 DEBUG:boto:Method: GET 2024-04-19T23:39:30.492 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.492 DEBUG:boto:Data: 2024-04-19T23:39:30.492 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.492 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.492 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.492 DEBUG:boto:Params: {} 2024-04-19T23:39:30.493 DEBUG:boto:Token: None 2024-04-19T23:39:30.493 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.493 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e78e0ddaa02fb14-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.496 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.496 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.496 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.497 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.497 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.497 DEBUG:boto:Method: GET 2024-04-19T23:39:30.497 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.497 DEBUG:boto:Data: 2024-04-19T23:39:30.497 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.497 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.497 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.497 DEBUG:boto:Params: {} 2024-04-19T23:39:30.497 DEBUG:boto:Token: None 2024-04-19T23:39:30.497 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.498 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vF8qo/ZOhyNVRnLgv3MIOuV7Duc= 2024-04-19T23:39:30.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vF8qo/ZOhyNVRnLgv3MIOuV7Duc=', 'Content-Length': '0'} 2024-04-19T23:39:30.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aab76329b0c3a2cb-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.500 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.500 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.500 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.500 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.500 DEBUG:boto:Method: GET 2024-04-19T23:39:30.501 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.501 DEBUG:boto:Data: 2024-04-19T23:39:30.501 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.501 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.501 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.501 DEBUG:boto:Params: {} 2024-04-19T23:39:30.501 DEBUG:boto:Token: None 2024-04-19T23:39:30.501 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.501 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vF8qo/ZOhyNVRnLgv3MIOuV7Duc= 2024-04-19T23:39:30.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vF8qo/ZOhyNVRnLgv3MIOuV7Duc=', 'Content-Length': '0'} 2024-04-19T23:39:30.504 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000308833c439dcf7bb-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.504 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.504 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.504 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.504 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.504 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.504 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.505 DEBUG:boto:Data: 2024-04-19T23:39:30.505 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.505 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.505 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.505 DEBUG:boto:Params: {} 2024-04-19T23:39:30.505 DEBUG:boto:Token: None 2024-04-19T23:39:30.505 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.505 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gOp6pTyNIBeMBMox0ADrj2HJ8WE= 2024-04-19T23:39:30.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gOp6pTyNIBeMBMox0ADrj2HJ8WE=', 'Content-Length': '0'} 2024-04-19T23:39:30.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013f1e60d8ca5be82-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.507 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.507 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.507 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.508 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.508 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.508 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.508 DEBUG:boto:Data: 2024-04-19T23:39:30.508 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.508 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.508 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.508 DEBUG:boto:Params: {} 2024-04-19T23:39:30.508 DEBUG:boto:Token: None 2024-04-19T23:39:30.508 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.509 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gOp6pTyNIBeMBMox0ADrj2HJ8WE= 2024-04-19T23:39:30.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gOp6pTyNIBeMBMox0ADrj2HJ8WE=', 'Content-Length': '0'} 2024-04-19T23:39:30.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb217f893bf60dfa-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.511 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.511 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.511 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.511 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.511 DEBUG:boto:Method: GET 2024-04-19T23:39:30.511 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.511 DEBUG:boto:Data: 2024-04-19T23:39:30.511 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.511 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.511 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.511 DEBUG:boto:Params: {} 2024-04-19T23:39:30.512 DEBUG:boto:Token: None 2024-04-19T23:39:30.512 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.512 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vF8qo/ZOhyNVRnLgv3MIOuV7Duc= 2024-04-19T23:39:30.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vF8qo/ZOhyNVRnLgv3MIOuV7Duc=', 'Content-Length': '0'} 2024-04-19T23:39:30.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054e4f03db11e8999-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.554 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.555 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.555 DEBUG:boto:path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.555 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.555 DEBUG:boto:Method: GET 2024-04-19T23:39:30.555 DEBUG:boto:Path: /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.555 DEBUG:boto:Data: 2024-04-19T23:39:30.555 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.555 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.555 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.555 DEBUG:boto:Params: {} 2024-04-19T23:39:30.555 DEBUG:boto:Token: None 2024-04-19T23:39:30.556 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2 2024-04-19T23:39:30.556 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vF8qo/ZOhyNVRnLgv3MIOuV7Duc= 2024-04-19T23:39:30.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vF8qo/ZOhyNVRnLgv3MIOuV7Duc=', 'Content-Length': '0'} 2024-04-19T23:39:30.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'mMPt8ztVFU4Ny3y0kGIENINjbbw1ZF2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035c5fb055aaf1a8b-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.558 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.558 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.558 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.558 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.559 DEBUG:boto:Method: GET 2024-04-19T23:39:30.559 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.559 DEBUG:boto:Data: 2024-04-19T23:39:30.559 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.559 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.559 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.559 DEBUG:boto:Params: {} 2024-04-19T23:39:30.559 DEBUG:boto:Token: None 2024-04-19T23:39:30.559 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.559 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089ffd69fceca01b2-00662300b2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.562 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.563 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.563 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.563 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.563 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.563 DEBUG:boto:Method: GET 2024-04-19T23:39:30.563 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.563 DEBUG:boto:Data: 2024-04-19T23:39:30.563 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.563 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.563 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.563 DEBUG:boto:Params: {} 2024-04-19T23:39:30.564 DEBUG:boto:Token: None 2024-04-19T23:39:30.564 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.564 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad466a639ece6a2f-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.567 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.567 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.568 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.568 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.568 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:30.568 DEBUG:boto:Data: 2024-04-19T23:39:30.568 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.568 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.568 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.568 DEBUG:boto:Params: {} 2024-04-19T23:39:30.568 DEBUG:boto:Token: None 2024-04-19T23:39:30.568 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj 2024-04-19T23:39:30.569 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y= 2024-04-19T23:39:30.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y=', 'Content-Length': '0'} 2024-04-19T23:39:30.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007fc358b364dd49ae-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.572 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.572 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.572 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.572 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:30.572 DEBUG:boto:Data: 2024-04-19T23:39:30.572 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.572 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.573 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.573 DEBUG:boto:Params: {} 2024-04-19T23:39:30.573 DEBUG:boto:Token: None 2024-04-19T23:39:30.573 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj 2024-04-19T23:39:30.573 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y= 2024-04-19T23:39:30.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y=', 'Content-Length': '0'} 2024-04-19T23:39:30.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048da7a1b5ba46e20-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.577 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.577 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.577 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.577 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.577 DEBUG:boto:Method: GET 2024-04-19T23:39:30.577 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.577 DEBUG:boto:Data: 2024-04-19T23:39:30.577 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.577 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.577 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.577 DEBUG:boto:Params: {} 2024-04-19T23:39:30.578 DEBUG:boto:Token: None 2024-04-19T23:39:30.578 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.578 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I= 2024-04-19T23:39:30.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I=', 'Content-Length': '0'} 2024-04-19T23:39:30.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006048a23d80060948-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.622 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.623 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.623 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.623 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.623 DEBUG:boto:Method: GET 2024-04-19T23:39:30.623 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.623 DEBUG:boto:Data: 2024-04-19T23:39:30.623 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.623 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.623 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.623 DEBUG:boto:Params: {} 2024-04-19T23:39:30.624 DEBUG:boto:Token: None 2024-04-19T23:39:30.624 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.624 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I= 2024-04-19T23:39:30.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I=', 'Content-Length': '0'} 2024-04-19T23:39:30.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066d8d2942e704c84-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.626 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.626 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.626 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.627 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.627 DEBUG:boto:Method: GET 2024-04-19T23:39:30.627 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.627 DEBUG:boto:Data: 2024-04-19T23:39:30.627 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.627 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.627 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.628 DEBUG:boto:Params: {} 2024-04-19T23:39:30.628 DEBUG:boto:Token: None 2024-04-19T23:39:30.628 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b3898ce148f9dd4-00662300b2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.631 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.632 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.632 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.632 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.632 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.632 DEBUG:boto:Method: GET 2024-04-19T23:39:30.632 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.632 DEBUG:boto:Data: 2024-04-19T23:39:30.632 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.633 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.633 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.633 DEBUG:boto:Params: {} 2024-04-19T23:39:30.633 DEBUG:boto:Token: None 2024-04-19T23:39:30.633 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.633 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ad97a226c60d95-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.636 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.637 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.637 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.637 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.637 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.637 DEBUG:boto:Method: GET 2024-04-19T23:39:30.637 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.637 DEBUG:boto:Data: 2024-04-19T23:39:30.638 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.638 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.638 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.638 DEBUG:boto:Params: {} 2024-04-19T23:39:30.638 DEBUG:boto:Token: None 2024-04-19T23:39:30.638 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.638 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3D79BIPUmbEWUVvUT24jUUic0Ak= 2024-04-19T23:39:30.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3D79BIPUmbEWUVvUT24jUUic0Ak=', 'Content-Length': '0'} 2024-04-19T23:39:30.640 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfac154c76a71f68-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.641 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.641 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.641 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.641 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.641 DEBUG:boto:Method: GET 2024-04-19T23:39:30.641 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.641 DEBUG:boto:Data: 2024-04-19T23:39:30.641 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.641 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.642 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.642 DEBUG:boto:Params: {} 2024-04-19T23:39:30.642 DEBUG:boto:Token: None 2024-04-19T23:39:30.642 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.642 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3D79BIPUmbEWUVvUT24jUUic0Ak= 2024-04-19T23:39:30.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3D79BIPUmbEWUVvUT24jUUic0Ak=', 'Content-Length': '0'} 2024-04-19T23:39:30.644 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057bb1f3d1d149f37-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.690 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.691 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.691 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.691 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.691 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.691 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.691 DEBUG:boto:Data: 2024-04-19T23:39:30.691 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.691 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.691 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.691 DEBUG:boto:Params: {} 2024-04-19T23:39:30.692 DEBUG:boto:Token: None 2024-04-19T23:39:30.692 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.692 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BnFdSLEQNBAUeHtGUhn0LC3BfAQ= 2024-04-19T23:39:30.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BnFdSLEQNBAUeHtGUhn0LC3BfAQ=', 'Content-Length': '0'} 2024-04-19T23:39:30.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0f60cff0fca0802-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.694 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.694 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.694 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.694 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.695 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.695 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.695 DEBUG:boto:Data: 2024-04-19T23:39:30.695 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.695 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.695 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.695 DEBUG:boto:Params: {} 2024-04-19T23:39:30.695 DEBUG:boto:Token: None 2024-04-19T23:39:30.695 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.695 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BnFdSLEQNBAUeHtGUhn0LC3BfAQ= 2024-04-19T23:39:30.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BnFdSLEQNBAUeHtGUhn0LC3BfAQ=', 'Content-Length': '0'} 2024-04-19T23:39:30.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f83e433e9e6c3c55-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.698 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.698 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.698 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.698 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.699 DEBUG:boto:Method: GET 2024-04-19T23:39:30.699 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.699 DEBUG:boto:Data: 2024-04-19T23:39:30.699 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.699 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.699 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.699 DEBUG:boto:Params: {} 2024-04-19T23:39:30.699 DEBUG:boto:Token: None 2024-04-19T23:39:30.699 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.699 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3D79BIPUmbEWUVvUT24jUUic0Ak= 2024-04-19T23:39:30.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3D79BIPUmbEWUVvUT24jUUic0Ak=', 'Content-Length': '0'} 2024-04-19T23:39:30.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072f3d7d501f020b9-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.702 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.702 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.702 DEBUG:boto:path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.702 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.702 DEBUG:boto:Method: GET 2024-04-19T23:39:30.702 DEBUG:boto:Path: /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.702 DEBUG:boto:Data: 2024-04-19T23:39:30.702 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.703 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.703 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.703 DEBUG:boto:Params: {} 2024-04-19T23:39:30.703 DEBUG:boto:Token: None 2024-04-19T23:39:30.703 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=8Jy1wSwswautcJ.pAM0qXnm5CTgNceO 2024-04-19T23:39:30.703 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3D79BIPUmbEWUVvUT24jUUic0Ak= 2024-04-19T23:39:30.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3D79BIPUmbEWUVvUT24jUUic0Ak=', 'Content-Length': '0'} 2024-04-19T23:39:30.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '8Jy1wSwswautcJ.pAM0qXnm5CTgNceO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf0b3384149cd504-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.750 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.763 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.763 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.763 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.764 DEBUG:boto:Method: GET 2024-04-19T23:39:30.764 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.764 DEBUG:boto:Data: 2024-04-19T23:39:30.764 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.764 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.764 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.764 DEBUG:boto:Params: {} 2024-04-19T23:39:30.764 DEBUG:boto:Token: None 2024-04-19T23:39:30.764 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.765 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001374d7d36a37f33e-00662300b2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.768 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.768 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.768 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.768 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.768 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.769 DEBUG:boto:Method: GET 2024-04-19T23:39:30.769 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.769 DEBUG:boto:Data: 2024-04-19T23:39:30.769 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.769 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.769 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.769 DEBUG:boto:Params: {} 2024-04-19T23:39:30.769 DEBUG:boto:Token: None 2024-04-19T23:39:30.769 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.769 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036c12f2faefdef56-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.772 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.773 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.773 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.773 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.773 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:30.773 DEBUG:boto:Data: 2024-04-19T23:39:30.773 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.773 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.773 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.774 DEBUG:boto:Params: {} 2024-04-19T23:39:30.774 DEBUG:boto:Token: None 2024-04-19T23:39:30.774 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj 2024-04-19T23:39:30.774 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y= 2024-04-19T23:39:30.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y=', 'Content-Length': '0'} 2024-04-19T23:39:30.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048a096fbabafa84a-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.778 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.778 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.778 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.778 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:30.778 DEBUG:boto:Data: 2024-04-19T23:39:30.778 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.778 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.778 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.778 DEBUG:boto:Params: {} 2024-04-19T23:39:30.779 DEBUG:boto:Token: None 2024-04-19T23:39:30.779 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj 2024-04-19T23:39:30.779 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y= 2024-04-19T23:39:30.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y=', 'Content-Length': '0'} 2024-04-19T23:39:30.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d64f447c82fe506-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.782 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.782 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.783 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.783 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.783 DEBUG:boto:Method: GET 2024-04-19T23:39:30.783 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.783 DEBUG:boto:Data: 2024-04-19T23:39:30.783 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.783 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.783 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.783 DEBUG:boto:Params: {} 2024-04-19T23:39:30.783 DEBUG:boto:Token: None 2024-04-19T23:39:30.784 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I= 2024-04-19T23:39:30.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I=', 'Content-Length': '0'} 2024-04-19T23:39:30.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a09ea4179bf9491-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.826 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.826 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.826 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.826 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.827 DEBUG:boto:Method: GET 2024-04-19T23:39:30.827 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.827 DEBUG:boto:Data: 2024-04-19T23:39:30.827 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.827 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.827 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.827 DEBUG:boto:Params: {} 2024-04-19T23:39:30.827 DEBUG:boto:Token: None 2024-04-19T23:39:30.827 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.828 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I= 2024-04-19T23:39:30.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I=', 'Content-Length': '0'} 2024-04-19T23:39:30.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000021b5e641d36b8b7-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.830 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.830 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.830 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.830 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.830 DEBUG:boto:Method: GET 2024-04-19T23:39:30.830 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.830 DEBUG:boto:Data: 2024-04-19T23:39:30.830 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.830 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.830 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.831 DEBUG:boto:Params: {} 2024-04-19T23:39:30.831 DEBUG:boto:Token: None 2024-04-19T23:39:30.831 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.831 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fb4e7bc7ce23d39-00662300b2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.834 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.835 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.835 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.835 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.835 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.835 DEBUG:boto:Method: GET 2024-04-19T23:39:30.835 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.835 DEBUG:boto:Data: 2024-04-19T23:39:30.835 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.835 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.835 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.835 DEBUG:boto:Params: {} 2024-04-19T23:39:30.836 DEBUG:boto:Token: None 2024-04-19T23:39:30.836 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.836 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0a8a577dd7005ad-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.840 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.840 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.840 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.840 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.840 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.840 DEBUG:boto:Method: GET 2024-04-19T23:39:30.840 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.841 DEBUG:boto:Data: 2024-04-19T23:39:30.841 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.841 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.841 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.841 DEBUG:boto:Params: {} 2024-04-19T23:39:30.841 DEBUG:boto:Token: None 2024-04-19T23:39:30.841 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.841 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zVxvfcg83PwwjG8qhPYrwGlC6WU= 2024-04-19T23:39:30.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zVxvfcg83PwwjG8qhPYrwGlC6WU=', 'Content-Length': '0'} 2024-04-19T23:39:30.843 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000282c5bd1cfcf6b8f-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.843 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.844 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.844 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.844 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.844 DEBUG:boto:Method: GET 2024-04-19T23:39:30.844 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.844 DEBUG:boto:Data: 2024-04-19T23:39:30.844 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.844 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.844 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.844 DEBUG:boto:Params: {} 2024-04-19T23:39:30.844 DEBUG:boto:Token: None 2024-04-19T23:39:30.845 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.845 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zVxvfcg83PwwjG8qhPYrwGlC6WU= 2024-04-19T23:39:30.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zVxvfcg83PwwjG8qhPYrwGlC6WU=', 'Content-Length': '0'} 2024-04-19T23:39:30.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049f6ba41a2a2ade5-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.847 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.847 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.847 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.847 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.847 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.847 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.847 DEBUG:boto:Data: 2024-04-19T23:39:30.847 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.847 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.848 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.848 DEBUG:boto:Params: {} 2024-04-19T23:39:30.848 DEBUG:boto:Token: None 2024-04-19T23:39:30.848 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.848 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fAX4K7vA/+IY97Uh85nhgog6Zzg= 2024-04-19T23:39:30.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fAX4K7vA/+IY97Uh85nhgog6Zzg=', 'Content-Length': '0'} 2024-04-19T23:39:30.850 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0630b352b612507-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.850 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.850 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.850 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.851 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.851 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.851 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.851 DEBUG:boto:Data: 2024-04-19T23:39:30.851 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.851 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.851 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.851 DEBUG:boto:Params: {} 2024-04-19T23:39:30.851 DEBUG:boto:Token: None 2024-04-19T23:39:30.851 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.852 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fAX4K7vA/+IY97Uh85nhgog6Zzg= 2024-04-19T23:39:30.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fAX4K7vA/+IY97Uh85nhgog6Zzg=', 'Content-Length': '0'} 2024-04-19T23:39:30.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aed7717dc95d08f4-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.854 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.855 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.855 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.855 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.855 DEBUG:boto:Method: GET 2024-04-19T23:39:30.855 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.855 DEBUG:boto:Data: 2024-04-19T23:39:30.855 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.855 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.856 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.856 DEBUG:boto:Params: {} 2024-04-19T23:39:30.856 DEBUG:boto:Token: None 2024-04-19T23:39:30.856 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.856 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zVxvfcg83PwwjG8qhPYrwGlC6WU= 2024-04-19T23:39:30.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zVxvfcg83PwwjG8qhPYrwGlC6WU=', 'Content-Length': '0'} 2024-04-19T23:39:30.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e1a13374b60be22-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.898 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.898 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.899 DEBUG:boto:path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.899 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.899 DEBUG:boto:Method: GET 2024-04-19T23:39:30.899 DEBUG:boto:Path: /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.899 DEBUG:boto:Data: 2024-04-19T23:39:30.899 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.899 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.899 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.899 DEBUG:boto:Params: {} 2024-04-19T23:39:30.899 DEBUG:boto:Token: None 2024-04-19T23:39:30.900 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys- 2024-04-19T23:39:30.900 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zVxvfcg83PwwjG8qhPYrwGlC6WU= 2024-04-19T23:39:30.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zVxvfcg83PwwjG8qhPYrwGlC6WU=', 'Content-Length': '0'} 2024-04-19T23:39:30.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ZgcrdH3i5S2qdX4yjPO1UZdTR.nfys-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebb62dc25c1785a3-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.902 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.902 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.903 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.903 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.903 DEBUG:boto:Method: GET 2024-04-19T23:39:30.903 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.903 DEBUG:boto:Data: 2024-04-19T23:39:30.903 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.903 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.903 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.903 DEBUG:boto:Params: {} 2024-04-19T23:39:30.903 DEBUG:boto:Token: None 2024-04-19T23:39:30.903 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.904 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c5fc22714096935-00662300b2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.950 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.951 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:30.951 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:30.951 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.951 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:30.951 DEBUG:boto:Method: GET 2024-04-19T23:39:30.951 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:30.951 DEBUG:boto:Data: 2024-04-19T23:39:30.951 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.952 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.952 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.952 DEBUG:boto:Params: {} 2024-04-19T23:39:30.952 DEBUG:boto:Token: None 2024-04-19T23:39:30.952 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/ 2024-04-19T23:39:30.952 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8= 2024-04-19T23:39:30.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KV4bHUnJDWONjJTH3JlGNmAxVA8=', 'Content-Length': '0'} 2024-04-19T23:39:30.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056589d6f4e2eefe5-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.955 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:30.956 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.956 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.956 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.956 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:30.956 DEBUG:boto:Data: 2024-04-19T23:39:30.956 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.956 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.956 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.956 DEBUG:boto:Params: {} 2024-04-19T23:39:30.956 DEBUG:boto:Token: None 2024-04-19T23:39:30.957 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj 2024-04-19T23:39:30.957 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y= 2024-04-19T23:39:30.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y=', 'Content-Length': '0'} 2024-04-19T23:39:30.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063273d1cbd9eedaf-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.960 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.960 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.960 DEBUG:boto:Method: HEAD 2024-04-19T23:39:30.960 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:30.960 DEBUG:boto:Data: 2024-04-19T23:39:30.961 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.961 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:30.961 DEBUG:boto:Port: 8000 2024-04-19T23:39:30.961 DEBUG:boto:Params: {} 2024-04-19T23:39:30.961 DEBUG:boto:Token: None 2024-04-19T23:39:30.961 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj 2024-04-19T23:39:30.961 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y= 2024-04-19T23:39:30.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bOTTqj/nUl4/kZWRKgFvuCLQc6Y=', 'Content-Length': '0'} 2024-04-19T23:39:30.964 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000104ba5ccebb6269f-00662300b2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:30.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:30.965 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:30.965 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:30.965 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.965 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.965 DEBUG:boto:Method: GET 2024-04-19T23:39:30.965 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.965 DEBUG:boto:Data: 2024-04-19T23:39:30.965 DEBUG:boto:Headers: {} 2024-04-19T23:39:30.965 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:30.965 DEBUG:boto:Port: 8001 2024-04-19T23:39:30.965 DEBUG:boto:Params: {} 2024-04-19T23:39:30.966 DEBUG:boto:Token: None 2024-04-19T23:39:30.966 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:30 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:30.966 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I= 2024-04-19T23:39:30.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:30 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zKhAC9un0WbST5aa96DuNOzEm0I=', 'Content-Length': '0'} 2024-04-19T23:39:30.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbcc5bf2dc22bb10-00662300b2-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.010 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.010 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.011 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.011 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.011 DEBUG:boto:Method: GET 2024-04-19T23:39:31.011 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.011 DEBUG:boto:Data: 2024-04-19T23:39:31.011 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.011 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.011 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.011 DEBUG:boto:Params: {} 2024-04-19T23:39:31.011 DEBUG:boto:Token: None 2024-04-19T23:39:31.012 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.012 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc= 2024-04-19T23:39:31.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc=', 'Content-Length': '0'} 2024-04-19T23:39:31.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3dc4f99da6de842-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.014 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.014 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.014 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.014 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.015 DEBUG:boto:Method: GET 2024-04-19T23:39:31.015 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.015 DEBUG:boto:Data: 2024-04-19T23:39:31.015 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.015 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.015 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.015 DEBUG:boto:Params: {} 2024-04-19T23:39:31.015 DEBUG:boto:Token: None 2024-04-19T23:39:31.015 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.015 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d671a88d261e4c35-00662300b3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.062 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.063 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.063 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.063 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.063 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.063 DEBUG:boto:Method: GET 2024-04-19T23:39:31.063 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.063 DEBUG:boto:Data: 2024-04-19T23:39:31.064 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.064 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.064 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.064 DEBUG:boto:Params: {} 2024-04-19T23:39:31.064 DEBUG:boto:Token: None 2024-04-19T23:39:31.064 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.064 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000347a3ddd38fef3a7-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.067 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.068 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.068 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.068 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.068 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.068 DEBUG:boto:Method: GET 2024-04-19T23:39:31.068 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.068 DEBUG:boto:Data: 2024-04-19T23:39:31.068 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.068 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.069 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.069 DEBUG:boto:Params: {} 2024-04-19T23:39:31.069 DEBUG:boto:Token: None 2024-04-19T23:39:31.069 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.069 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ohQDmi81p8NqIRDGwFZ3JK3gkuc= 2024-04-19T23:39:31.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ohQDmi81p8NqIRDGwFZ3JK3gkuc=', 'Content-Length': '0'} 2024-04-19T23:39:31.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073bc53c1534df964-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.072 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.072 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.072 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.072 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.072 DEBUG:boto:Method: GET 2024-04-19T23:39:31.072 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.072 DEBUG:boto:Data: 2024-04-19T23:39:31.072 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.073 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.073 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.073 DEBUG:boto:Params: {} 2024-04-19T23:39:31.073 DEBUG:boto:Token: None 2024-04-19T23:39:31.073 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.073 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ohQDmi81p8NqIRDGwFZ3JK3gkuc= 2024-04-19T23:39:31.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ohQDmi81p8NqIRDGwFZ3JK3gkuc=', 'Content-Length': '0'} 2024-04-19T23:39:31.075 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096f438bd9ba3c5a8-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.075 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.076 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.076 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.076 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.076 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.076 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.076 DEBUG:boto:Data: 2024-04-19T23:39:31.076 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.076 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.076 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.076 DEBUG:boto:Params: {} 2024-04-19T23:39:31.077 DEBUG:boto:Token: None 2024-04-19T23:39:31.077 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.077 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pnIh9GtKXkYcNmV2w5cRe/DMcoE= 2024-04-19T23:39:31.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pnIh9GtKXkYcNmV2w5cRe/DMcoE=', 'Content-Length': '0'} 2024-04-19T23:39:31.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f606b04a0a28a1e2-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.079 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.079 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.079 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.079 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.079 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.079 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.080 DEBUG:boto:Data: 2024-04-19T23:39:31.080 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.080 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.080 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.080 DEBUG:boto:Params: {} 2024-04-19T23:39:31.080 DEBUG:boto:Token: None 2024-04-19T23:39:31.080 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.080 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pnIh9GtKXkYcNmV2w5cRe/DMcoE= 2024-04-19T23:39:31.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pnIh9GtKXkYcNmV2w5cRe/DMcoE=', 'Content-Length': '0'} 2024-04-19T23:39:31.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3168acccd8dcb25-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.082 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.083 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.083 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.083 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.083 DEBUG:boto:Method: GET 2024-04-19T23:39:31.083 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.083 DEBUG:boto:Data: 2024-04-19T23:39:31.083 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.083 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.083 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.083 DEBUG:boto:Params: {} 2024-04-19T23:39:31.084 DEBUG:boto:Token: None 2024-04-19T23:39:31.084 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ohQDmi81p8NqIRDGwFZ3JK3gkuc= 2024-04-19T23:39:31.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ohQDmi81p8NqIRDGwFZ3JK3gkuc=', 'Content-Length': '0'} 2024-04-19T23:39:31.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b96782c80fdaf12d-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.126 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.127 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.127 DEBUG:boto:path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.127 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.127 DEBUG:boto:Method: GET 2024-04-19T23:39:31.127 DEBUG:boto:Path: /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.127 DEBUG:boto:Data: 2024-04-19T23:39:31.127 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.127 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.127 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.127 DEBUG:boto:Params: {} 2024-04-19T23:39:31.128 DEBUG:boto:Token: None 2024-04-19T23:39:31.128 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7 2024-04-19T23:39:31.128 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ohQDmi81p8NqIRDGwFZ3JK3gkuc= 2024-04-19T23:39:31.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ohQDmi81p8NqIRDGwFZ3JK3gkuc=', 'Content-Length': '0'} 2024-04-19T23:39:31.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'iTH-3lp7ITU9E05SQQ.Ld6Ir8uoFit7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026c2dbe1ead8e062-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.170 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.171 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.171 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.171 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.171 DEBUG:boto:Method: GET 2024-04-19T23:39:31.171 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.171 DEBUG:boto:Data: 2024-04-19T23:39:31.171 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.171 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.171 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.171 DEBUG:boto:Params: {} 2024-04-19T23:39:31.171 DEBUG:boto:Token: None 2024-04-19T23:39:31.172 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.172 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044913a07a2ade998-00662300b3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.175 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.175 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.175 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.175 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.175 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.175 DEBUG:boto:Method: GET 2024-04-19T23:39:31.176 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.176 DEBUG:boto:Data: 2024-04-19T23:39:31.176 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.176 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.176 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.176 DEBUG:boto:Params: {} 2024-04-19T23:39:31.176 DEBUG:boto:Token: None 2024-04-19T23:39:31.176 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.176 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c8fc7f4d4e3915c-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.222 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.223 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.224 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.224 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.224 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:31.224 DEBUG:boto:Data: 2024-04-19T23:39:31.224 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.224 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.224 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.224 DEBUG:boto:Params: {} 2024-04-19T23:39:31.224 DEBUG:boto:Token: None 2024-04-19T23:39:31.224 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj 2024-04-19T23:39:31.225 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo= 2024-04-19T23:39:31.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo=', 'Content-Length': '0'} 2024-04-19T23:39:31.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6812ea2bc375a73-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.228 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.228 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.229 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.229 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:31.229 DEBUG:boto:Data: 2024-04-19T23:39:31.229 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.229 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.229 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.229 DEBUG:boto:Params: {} 2024-04-19T23:39:31.229 DEBUG:boto:Token: None 2024-04-19T23:39:31.229 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj 2024-04-19T23:39:31.230 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo= 2024-04-19T23:39:31.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo=', 'Content-Length': '0'} 2024-04-19T23:39:31.233 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f0bb13f55161288-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.233 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.234 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.234 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.234 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.234 DEBUG:boto:Method: GET 2024-04-19T23:39:31.234 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.234 DEBUG:boto:Data: 2024-04-19T23:39:31.234 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.234 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.235 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.235 DEBUG:boto:Params: {} 2024-04-19T23:39:31.235 DEBUG:boto:Token: None 2024-04-19T23:39:31.235 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.235 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc= 2024-04-19T23:39:31.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc=', 'Content-Length': '0'} 2024-04-19T23:39:31.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038a38a688f1a3919-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.282 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.283 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.283 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.283 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.283 DEBUG:boto:Method: GET 2024-04-19T23:39:31.283 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.283 DEBUG:boto:Data: 2024-04-19T23:39:31.283 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.283 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.283 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.283 DEBUG:boto:Params: {} 2024-04-19T23:39:31.283 DEBUG:boto:Token: None 2024-04-19T23:39:31.284 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.284 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc= 2024-04-19T23:39:31.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc=', 'Content-Length': '0'} 2024-04-19T23:39:31.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b3f9c747572ff6f7-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.286 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.286 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.286 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.286 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.287 DEBUG:boto:Method: GET 2024-04-19T23:39:31.287 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.287 DEBUG:boto:Data: 2024-04-19T23:39:31.287 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.287 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.287 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.287 DEBUG:boto:Params: {} 2024-04-19T23:39:31.287 DEBUG:boto:Token: None 2024-04-19T23:39:31.287 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.287 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a20c7ed23c84d21c-00662300b3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.330 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.331 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.331 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.331 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.331 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.331 DEBUG:boto:Method: GET 2024-04-19T23:39:31.331 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.331 DEBUG:boto:Data: 2024-04-19T23:39:31.332 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.332 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.332 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.332 DEBUG:boto:Params: {} 2024-04-19T23:39:31.332 DEBUG:boto:Token: None 2024-04-19T23:39:31.332 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b03938eb38ad76e0-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.335 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.336 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.336 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.336 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.336 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.336 DEBUG:boto:Method: GET 2024-04-19T23:39:31.336 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.336 DEBUG:boto:Data: 2024-04-19T23:39:31.336 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.336 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.337 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.337 DEBUG:boto:Params: {} 2024-04-19T23:39:31.337 DEBUG:boto:Token: None 2024-04-19T23:39:31.337 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.337 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+fZIAcnIsZn6tSl8/9mWFfNqeiQ= 2024-04-19T23:39:31.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+fZIAcnIsZn6tSl8/9mWFfNqeiQ=', 'Content-Length': '0'} 2024-04-19T23:39:31.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca2cfadc7562b91b-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.340 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.340 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.340 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.340 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.340 DEBUG:boto:Method: GET 2024-04-19T23:39:31.340 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.340 DEBUG:boto:Data: 2024-04-19T23:39:31.340 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.341 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.341 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.341 DEBUG:boto:Params: {} 2024-04-19T23:39:31.341 DEBUG:boto:Token: None 2024-04-19T23:39:31.341 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.341 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+fZIAcnIsZn6tSl8/9mWFfNqeiQ= 2024-04-19T23:39:31.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+fZIAcnIsZn6tSl8/9mWFfNqeiQ=', 'Content-Length': '0'} 2024-04-19T23:39:31.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8399925ba0887bc-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.343 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.344 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.344 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.344 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.344 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.344 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.344 DEBUG:boto:Data: 2024-04-19T23:39:31.344 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.344 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.344 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.344 DEBUG:boto:Params: {} 2024-04-19T23:39:31.345 DEBUG:boto:Token: None 2024-04-19T23:39:31.345 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.345 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XpPccO3ItLGsbHWqzRiMBDfJE2k= 2024-04-19T23:39:31.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XpPccO3ItLGsbHWqzRiMBDfJE2k=', 'Content-Length': '0'} 2024-04-19T23:39:31.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a7c45ffffc001e1-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.347 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.347 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.347 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.347 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.347 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.347 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.347 DEBUG:boto:Data: 2024-04-19T23:39:31.347 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.347 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.348 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.348 DEBUG:boto:Params: {} 2024-04-19T23:39:31.348 DEBUG:boto:Token: None 2024-04-19T23:39:31.348 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.348 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XpPccO3ItLGsbHWqzRiMBDfJE2k= 2024-04-19T23:39:31.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XpPccO3ItLGsbHWqzRiMBDfJE2k=', 'Content-Length': '0'} 2024-04-19T23:39:31.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063f2b80d18598985-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.350 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.350 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.351 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.351 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.351 DEBUG:boto:Method: GET 2024-04-19T23:39:31.351 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.351 DEBUG:boto:Data: 2024-04-19T23:39:31.351 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.351 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.351 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.351 DEBUG:boto:Params: {} 2024-04-19T23:39:31.351 DEBUG:boto:Token: None 2024-04-19T23:39:31.351 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.352 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+fZIAcnIsZn6tSl8/9mWFfNqeiQ= 2024-04-19T23:39:31.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+fZIAcnIsZn6tSl8/9mWFfNqeiQ=', 'Content-Length': '0'} 2024-04-19T23:39:31.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c542041a4d31d856-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.394 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.394 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.395 DEBUG:boto:path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.395 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.395 DEBUG:boto:Method: GET 2024-04-19T23:39:31.395 DEBUG:boto:Path: /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.395 DEBUG:boto:Data: 2024-04-19T23:39:31.395 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.395 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.395 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.395 DEBUG:boto:Params: {} 2024-04-19T23:39:31.395 DEBUG:boto:Token: None 2024-04-19T23:39:31.396 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR 2024-04-19T23:39:31.396 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+fZIAcnIsZn6tSl8/9mWFfNqeiQ= 2024-04-19T23:39:31.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+fZIAcnIsZn6tSl8/9mWFfNqeiQ=', 'Content-Length': '0'} 2024-04-19T23:39:31.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'Q1LhHAD3EkxJ2GCgXO1UXq0mp8mHLgR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004eeb29e1266c261-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.438 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.439 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.439 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.439 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.439 DEBUG:boto:Method: GET 2024-04-19T23:39:31.439 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.439 DEBUG:boto:Data: 2024-04-19T23:39:31.439 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.439 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.439 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.439 DEBUG:boto:Params: {} 2024-04-19T23:39:31.440 DEBUG:boto:Token: None 2024-04-19T23:39:31.440 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.440 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfe3d1038bd2492d-00662300b3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.443 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.444 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.444 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.444 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.444 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.444 DEBUG:boto:Method: GET 2024-04-19T23:39:31.444 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.444 DEBUG:boto:Data: 2024-04-19T23:39:31.444 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.444 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.444 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.445 DEBUG:boto:Params: {} 2024-04-19T23:39:31.445 DEBUG:boto:Token: None 2024-04-19T23:39:31.445 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.445 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000721e4eb6ff0c2332-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.448 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.448 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.449 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.449 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.449 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:31.449 DEBUG:boto:Data: 2024-04-19T23:39:31.449 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.449 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.449 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.449 DEBUG:boto:Params: {} 2024-04-19T23:39:31.449 DEBUG:boto:Token: None 2024-04-19T23:39:31.450 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj 2024-04-19T23:39:31.450 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo= 2024-04-19T23:39:31.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo=', 'Content-Length': '0'} 2024-04-19T23:39:31.453 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c416052bf31c6a1-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.454 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.454 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.454 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.454 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:31.454 DEBUG:boto:Data: 2024-04-19T23:39:31.454 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.454 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.454 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.454 DEBUG:boto:Params: {} 2024-04-19T23:39:31.455 DEBUG:boto:Token: None 2024-04-19T23:39:31.455 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj 2024-04-19T23:39:31.455 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo= 2024-04-19T23:39:31.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo=', 'Content-Length': '0'} 2024-04-19T23:39:31.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000771324c9b75dc2c6-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.459 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.459 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.459 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.459 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.459 DEBUG:boto:Method: GET 2024-04-19T23:39:31.459 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.459 DEBUG:boto:Data: 2024-04-19T23:39:31.459 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.459 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.460 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.460 DEBUG:boto:Params: {} 2024-04-19T23:39:31.460 DEBUG:boto:Token: None 2024-04-19T23:39:31.460 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc= 2024-04-19T23:39:31.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc=', 'Content-Length': '0'} 2024-04-19T23:39:31.462 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d8fd2c1046e0b89-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.510 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.510 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.511 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.511 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.511 DEBUG:boto:Method: GET 2024-04-19T23:39:31.511 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.511 DEBUG:boto:Data: 2024-04-19T23:39:31.511 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.511 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.511 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.511 DEBUG:boto:Params: {} 2024-04-19T23:39:31.511 DEBUG:boto:Token: None 2024-04-19T23:39:31.512 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.512 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc= 2024-04-19T23:39:31.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc=', 'Content-Length': '0'} 2024-04-19T23:39:31.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e04dbdea9ed1917-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.514 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.514 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.515 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.515 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.515 DEBUG:boto:Method: GET 2024-04-19T23:39:31.515 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.515 DEBUG:boto:Data: 2024-04-19T23:39:31.515 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.515 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.515 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.515 DEBUG:boto:Params: {} 2024-04-19T23:39:31.516 DEBUG:boto:Token: None 2024-04-19T23:39:31.516 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.516 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d4d1e8aceed4b11-00662300b3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.519 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.519 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.519 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.520 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.520 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.520 DEBUG:boto:Method: GET 2024-04-19T23:39:31.520 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.520 DEBUG:boto:Data: 2024-04-19T23:39:31.520 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.520 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.520 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.520 DEBUG:boto:Params: {} 2024-04-19T23:39:31.520 DEBUG:boto:Token: None 2024-04-19T23:39:31.521 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.521 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d16c6f9c06787e65-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.524 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.525 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.525 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.526 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.526 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.526 DEBUG:boto:Method: GET 2024-04-19T23:39:31.526 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.526 DEBUG:boto:Data: 2024-04-19T23:39:31.526 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.526 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.526 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.527 DEBUG:boto:Params: {} 2024-04-19T23:39:31.527 DEBUG:boto:Token: None 2024-04-19T23:39:31.527 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.527 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aJDT7pZuW8oy0EbeX8+IKrP8FPs= 2024-04-19T23:39:31.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aJDT7pZuW8oy0EbeX8+IKrP8FPs=', 'Content-Length': '0'} 2024-04-19T23:39:31.529 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003134290bea17412-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.574 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.575 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.575 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.575 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.575 DEBUG:boto:Method: GET 2024-04-19T23:39:31.575 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.575 DEBUG:boto:Data: 2024-04-19T23:39:31.575 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.575 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.575 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.575 DEBUG:boto:Params: {} 2024-04-19T23:39:31.576 DEBUG:boto:Token: None 2024-04-19T23:39:31.576 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.576 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aJDT7pZuW8oy0EbeX8+IKrP8FPs= 2024-04-19T23:39:31.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aJDT7pZuW8oy0EbeX8+IKrP8FPs=', 'Content-Length': '0'} 2024-04-19T23:39:31.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e7f3faa8e5e605e-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.618 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.619 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.619 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.619 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.619 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.619 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.619 DEBUG:boto:Data: 2024-04-19T23:39:31.619 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.619 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.619 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.619 DEBUG:boto:Params: {} 2024-04-19T23:39:31.619 DEBUG:boto:Token: None 2024-04-19T23:39:31.620 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.620 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FEXZQ4psqQAxQi0DlrV6PQTrdMU= 2024-04-19T23:39:31.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FEXZQ4psqQAxQi0DlrV6PQTrdMU=', 'Content-Length': '0'} 2024-04-19T23:39:31.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c69246d9408ce1e5-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.622 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.622 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.622 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.622 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.622 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.623 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.623 DEBUG:boto:Data: 2024-04-19T23:39:31.623 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.623 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.623 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.623 DEBUG:boto:Params: {} 2024-04-19T23:39:31.623 DEBUG:boto:Token: None 2024-04-19T23:39:31.623 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.623 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FEXZQ4psqQAxQi0DlrV6PQTrdMU= 2024-04-19T23:39:31.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FEXZQ4psqQAxQi0DlrV6PQTrdMU=', 'Content-Length': '0'} 2024-04-19T23:39:31.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059761db24d584f75-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.626 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.626 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.626 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.626 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.626 DEBUG:boto:Method: GET 2024-04-19T23:39:31.627 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.627 DEBUG:boto:Data: 2024-04-19T23:39:31.627 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.627 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.627 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.627 DEBUG:boto:Params: {} 2024-04-19T23:39:31.627 DEBUG:boto:Token: None 2024-04-19T23:39:31.627 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aJDT7pZuW8oy0EbeX8+IKrP8FPs= 2024-04-19T23:39:31.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aJDT7pZuW8oy0EbeX8+IKrP8FPs=', 'Content-Length': '0'} 2024-04-19T23:39:31.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000561ff39b3ee7a4a-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.670 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.670 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.671 DEBUG:boto:path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.671 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.671 DEBUG:boto:Method: GET 2024-04-19T23:39:31.671 DEBUG:boto:Path: /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.671 DEBUG:boto:Data: 2024-04-19T23:39:31.671 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.671 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.671 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.671 DEBUG:boto:Params: {} 2024-04-19T23:39:31.671 DEBUG:boto:Token: None 2024-04-19T23:39:31.672 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM 2024-04-19T23:39:31.672 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aJDT7pZuW8oy0EbeX8+IKrP8FPs= 2024-04-19T23:39:31.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aJDT7pZuW8oy0EbeX8+IKrP8FPs=', 'Content-Length': '0'} 2024-04-19T23:39:31.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'ipPC6NaG1ejWpptAGf0zgZ-gqpdm0DM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000733dc509ebb0d157-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.674 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.674 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.675 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.675 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.675 DEBUG:boto:Method: GET 2024-04-19T23:39:31.675 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.675 DEBUG:boto:Data: 2024-04-19T23:39:31.675 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.675 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.675 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.675 DEBUG:boto:Params: {} 2024-04-19T23:39:31.675 DEBUG:boto:Token: None 2024-04-19T23:39:31.676 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f27442b713c2b8ba-00662300b3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.679 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.679 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.679 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.679 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.679 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.679 DEBUG:boto:Method: GET 2024-04-19T23:39:31.680 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.680 DEBUG:boto:Data: 2024-04-19T23:39:31.680 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.680 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.680 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.680 DEBUG:boto:Params: {} 2024-04-19T23:39:31.680 DEBUG:boto:Token: None 2024-04-19T23:39:31.680 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.680 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005722a263691d3b3c-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.683 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.684 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.684 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.684 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.684 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:31.684 DEBUG:boto:Data: 2024-04-19T23:39:31.684 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.684 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.684 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.684 DEBUG:boto:Params: {} 2024-04-19T23:39:31.684 DEBUG:boto:Token: None 2024-04-19T23:39:31.685 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj 2024-04-19T23:39:31.685 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo= 2024-04-19T23:39:31.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo=', 'Content-Length': '0'} 2024-04-19T23:39:31.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c63553b89557b0f-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.689 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.689 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.689 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.689 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:31.689 DEBUG:boto:Data: 2024-04-19T23:39:31.689 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.689 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.689 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.690 DEBUG:boto:Params: {} 2024-04-19T23:39:31.690 DEBUG:boto:Token: None 2024-04-19T23:39:31.690 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj 2024-04-19T23:39:31.690 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo= 2024-04-19T23:39:31.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo=', 'Content-Length': '0'} 2024-04-19T23:39:31.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e62c15bd17e1bddd-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.694 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.694 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.694 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.694 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.694 DEBUG:boto:Method: GET 2024-04-19T23:39:31.694 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.695 DEBUG:boto:Data: 2024-04-19T23:39:31.695 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.695 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.695 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.695 DEBUG:boto:Params: {} 2024-04-19T23:39:31.695 DEBUG:boto:Token: None 2024-04-19T23:39:31.695 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.695 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc= 2024-04-19T23:39:31.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc=', 'Content-Length': '0'} 2024-04-19T23:39:31.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000886b6acc66b5dc16-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.742 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.743 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.743 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.743 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.743 DEBUG:boto:Method: GET 2024-04-19T23:39:31.743 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.743 DEBUG:boto:Data: 2024-04-19T23:39:31.743 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.743 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.743 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.743 DEBUG:boto:Params: {} 2024-04-19T23:39:31.743 DEBUG:boto:Token: None 2024-04-19T23:39:31.744 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.744 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc= 2024-04-19T23:39:31.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc=', 'Content-Length': '0'} 2024-04-19T23:39:31.746 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7a135fdf33e5526-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.746 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.747 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.747 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.747 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.747 DEBUG:boto:Method: GET 2024-04-19T23:39:31.747 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.747 DEBUG:boto:Data: 2024-04-19T23:39:31.747 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.747 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.747 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.747 DEBUG:boto:Params: {} 2024-04-19T23:39:31.747 DEBUG:boto:Token: None 2024-04-19T23:39:31.748 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.748 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008de4db54a0660103-00662300b3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.750 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.751 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.751 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.751 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.751 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.751 DEBUG:boto:Method: GET 2024-04-19T23:39:31.751 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.751 DEBUG:boto:Data: 2024-04-19T23:39:31.751 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.751 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.752 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.752 DEBUG:boto:Params: {} 2024-04-19T23:39:31.752 DEBUG:boto:Token: None 2024-04-19T23:39:31.752 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.752 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d2d1c5c648fe497-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.755 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.756 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.756 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.756 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.756 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.756 DEBUG:boto:Method: GET 2024-04-19T23:39:31.756 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.756 DEBUG:boto:Data: 2024-04-19T23:39:31.756 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.756 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.756 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.756 DEBUG:boto:Params: {} 2024-04-19T23:39:31.757 DEBUG:boto:Token: None 2024-04-19T23:39:31.757 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.757 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z0BfoUmQzw8N5lhQ6HSP2djENo0= 2024-04-19T23:39:31.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z0BfoUmQzw8N5lhQ6HSP2djENo0=', 'Content-Length': '0'} 2024-04-19T23:39:31.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000811903a41e1a495e-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.759 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.759 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.759 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.759 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.759 DEBUG:boto:Method: GET 2024-04-19T23:39:31.759 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.760 DEBUG:boto:Data: 2024-04-19T23:39:31.760 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.760 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.760 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.760 DEBUG:boto:Params: {} 2024-04-19T23:39:31.760 DEBUG:boto:Token: None 2024-04-19T23:39:31.760 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.760 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z0BfoUmQzw8N5lhQ6HSP2djENo0= 2024-04-19T23:39:31.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z0BfoUmQzw8N5lhQ6HSP2djENo0=', 'Content-Length': '0'} 2024-04-19T23:39:31.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f09c0a6d6677659-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.802 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.802 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.803 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.803 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.803 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.803 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.803 DEBUG:boto:Data: 2024-04-19T23:39:31.803 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.803 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.803 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.803 DEBUG:boto:Params: {} 2024-04-19T23:39:31.803 DEBUG:boto:Token: None 2024-04-19T23:39:31.804 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ruNSn95lknm4Bo8Qlh+Y4DbKYO8= 2024-04-19T23:39:31.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ruNSn95lknm4Bo8Qlh+Y4DbKYO8=', 'Content-Length': '0'} 2024-04-19T23:39:31.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3b7119400ec85e6-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.806 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.806 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.806 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.806 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.806 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.806 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.807 DEBUG:boto:Data: 2024-04-19T23:39:31.807 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.807 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.807 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.807 DEBUG:boto:Params: {} 2024-04-19T23:39:31.807 DEBUG:boto:Token: None 2024-04-19T23:39:31.807 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.807 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ruNSn95lknm4Bo8Qlh+Y4DbKYO8= 2024-04-19T23:39:31.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ruNSn95lknm4Bo8Qlh+Y4DbKYO8=', 'Content-Length': '0'} 2024-04-19T23:39:31.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecbb090d23b609b7-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.810 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.810 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.810 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.810 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.810 DEBUG:boto:Method: GET 2024-04-19T23:39:31.810 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.810 DEBUG:boto:Data: 2024-04-19T23:39:31.810 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.810 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.811 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.811 DEBUG:boto:Params: {} 2024-04-19T23:39:31.811 DEBUG:boto:Token: None 2024-04-19T23:39:31.811 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.811 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z0BfoUmQzw8N5lhQ6HSP2djENo0= 2024-04-19T23:39:31.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z0BfoUmQzw8N5lhQ6HSP2djENo0=', 'Content-Length': '0'} 2024-04-19T23:39:31.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099f4a2b2affb0a34-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.813 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.813 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.813 DEBUG:boto:path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.813 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.814 DEBUG:boto:Method: GET 2024-04-19T23:39:31.814 DEBUG:boto:Path: /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.814 DEBUG:boto:Data: 2024-04-19T23:39:31.814 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.814 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.814 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.814 DEBUG:boto:Params: {} 2024-04-19T23:39:31.814 DEBUG:boto:Token: None 2024-04-19T23:39:31.814 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=oECNSXdMldM0FoTJgSxtKpnhGWUHc5t 2024-04-19T23:39:31.814 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z0BfoUmQzw8N5lhQ6HSP2djENo0= 2024-04-19T23:39:31.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z0BfoUmQzw8N5lhQ6HSP2djENo0=', 'Content-Length': '0'} 2024-04-19T23:39:31.816 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'oECNSXdMldM0FoTJgSxtKpnhGWUHc5t'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023c14c70bffe51fd-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.862 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.863 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.863 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.863 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.863 DEBUG:boto:Method: GET 2024-04-19T23:39:31.863 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.863 DEBUG:boto:Data: 2024-04-19T23:39:31.863 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.863 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.863 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.863 DEBUG:boto:Params: {} 2024-04-19T23:39:31.864 DEBUG:boto:Token: None 2024-04-19T23:39:31.864 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.864 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000f50dfcacfe831e-00662300b3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.867 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.867 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.867 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.867 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.867 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.867 DEBUG:boto:Method: GET 2024-04-19T23:39:31.867 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.868 DEBUG:boto:Data: 2024-04-19T23:39:31.868 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.868 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.868 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.868 DEBUG:boto:Params: {} 2024-04-19T23:39:31.868 DEBUG:boto:Token: None 2024-04-19T23:39:31.868 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.868 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000781d0e7216cc4666-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.871 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.871 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.872 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.872 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.872 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:31.872 DEBUG:boto:Data: 2024-04-19T23:39:31.872 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.872 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.872 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.872 DEBUG:boto:Params: {} 2024-04-19T23:39:31.872 DEBUG:boto:Token: None 2024-04-19T23:39:31.872 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj 2024-04-19T23:39:31.873 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo= 2024-04-19T23:39:31.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo=', 'Content-Length': '0'} 2024-04-19T23:39:31.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1d627145de93af8-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.876 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.876 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.876 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.876 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:31.877 DEBUG:boto:Data: 2024-04-19T23:39:31.877 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.877 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.877 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.877 DEBUG:boto:Params: {} 2024-04-19T23:39:31.877 DEBUG:boto:Token: None 2024-04-19T23:39:31.877 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj 2024-04-19T23:39:31.877 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo= 2024-04-19T23:39:31.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0eEsjGcaLxl6yRdvTaVmNCOgjzo=', 'Content-Length': '0'} 2024-04-19T23:39:31.880 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b8cdce58bc2af57-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.881 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.881 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.881 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.881 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.881 DEBUG:boto:Method: GET 2024-04-19T23:39:31.881 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.881 DEBUG:boto:Data: 2024-04-19T23:39:31.881 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.881 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.881 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.881 DEBUG:boto:Params: {} 2024-04-19T23:39:31.882 DEBUG:boto:Token: None 2024-04-19T23:39:31.882 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.882 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc= 2024-04-19T23:39:31.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc=', 'Content-Length': '0'} 2024-04-19T23:39:31.884 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051ca361a9e248da6-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.930 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.931 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.931 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.931 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.931 DEBUG:boto:Method: GET 2024-04-19T23:39:31.931 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.931 DEBUG:boto:Data: 2024-04-19T23:39:31.931 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.932 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.932 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.932 DEBUG:boto:Params: {} 2024-04-19T23:39:31.932 DEBUG:boto:Token: None 2024-04-19T23:39:31.932 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:31.932 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc= 2024-04-19T23:39:31.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d+Q9wI51ob49EerhjTpUcbanSOc=', 'Content-Length': '0'} 2024-04-19T23:39:31.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6dc6785c4b2998e-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.935 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.935 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.935 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.935 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.935 DEBUG:boto:Method: GET 2024-04-19T23:39:31.935 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.935 DEBUG:boto:Data: 2024-04-19T23:39:31.935 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.935 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.935 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.935 DEBUG:boto:Params: {} 2024-04-19T23:39:31.936 DEBUG:boto:Token: None 2024-04-19T23:39:31.936 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.936 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eced00afc3e87e6-00662300b3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.939 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.939 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:31.939 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:31.939 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.940 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:31.940 DEBUG:boto:Method: GET 2024-04-19T23:39:31.940 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:31.940 DEBUG:boto:Data: 2024-04-19T23:39:31.940 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.940 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.940 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.940 DEBUG:boto:Params: {} 2024-04-19T23:39:31.940 DEBUG:boto:Token: None 2024-04-19T23:39:31.940 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/ 2024-04-19T23:39:31.941 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE= 2024-04-19T23:39:31.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9TbRGUbJLMauJA27gOkSr/YTtYE=', 'Content-Length': '0'} 2024-04-19T23:39:31.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dc74fa1b95d5b6f-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.944 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:31.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.944 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.944 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.944 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.944 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.944 DEBUG:boto:Method: GET 2024-04-19T23:39:31.944 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.945 DEBUG:boto:Data: 2024-04-19T23:39:31.945 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.945 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.945 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.945 DEBUG:boto:Params: {} 2024-04-19T23:39:31.945 DEBUG:boto:Token: None 2024-04-19T23:39:31.945 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.945 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:P4y+FeSjtvHno2BV2c5sl/h6MdY= 2024-04-19T23:39:31.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:P4y+FeSjtvHno2BV2c5sl/h6MdY=', 'Content-Length': '0'} 2024-04-19T23:39:31.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a9761d766d44e30-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.947 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.948 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.948 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.948 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.948 DEBUG:boto:Method: GET 2024-04-19T23:39:31.948 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.948 DEBUG:boto:Data: 2024-04-19T23:39:31.948 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.948 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.948 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.948 DEBUG:boto:Params: {} 2024-04-19T23:39:31.948 DEBUG:boto:Token: None 2024-04-19T23:39:31.949 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.949 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:P4y+FeSjtvHno2BV2c5sl/h6MdY= 2024-04-19T23:39:31.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:P4y+FeSjtvHno2BV2c5sl/h6MdY=', 'Content-Length': '0'} 2024-04-19T23:39:31.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce5f6ba17bac255b-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.951 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.951 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.951 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.951 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.951 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.951 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.951 DEBUG:boto:Data: 2024-04-19T23:39:31.952 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.952 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.952 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.952 DEBUG:boto:Params: {} 2024-04-19T23:39:31.952 DEBUG:boto:Token: None 2024-04-19T23:39:31.952 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.952 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZLkDZFxD29OyNUsVBhqp/1J4yXY= 2024-04-19T23:39:31.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZLkDZFxD29OyNUsVBhqp/1J4yXY=', 'Content-Length': '0'} 2024-04-19T23:39:31.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069cf1b2090c12dab-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.954 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.954 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.955 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.955 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.955 DEBUG:boto:Method: HEAD 2024-04-19T23:39:31.955 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.955 DEBUG:boto:Data: 2024-04-19T23:39:31.955 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.955 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:31.955 DEBUG:boto:Port: 8000 2024-04-19T23:39:31.955 DEBUG:boto:Params: {} 2024-04-19T23:39:31.955 DEBUG:boto:Token: None 2024-04-19T23:39:31.955 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.956 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZLkDZFxD29OyNUsVBhqp/1J4yXY= 2024-04-19T23:39:31.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZLkDZFxD29OyNUsVBhqp/1J4yXY=', 'Content-Length': '0'} 2024-04-19T23:39:31.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9cc4d9660c5d07c-00662300b3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:31.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:31.958 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:31.958 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:31.958 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.958 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.958 DEBUG:boto:Method: GET 2024-04-19T23:39:31.958 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.958 DEBUG:boto:Data: 2024-04-19T23:39:31.958 DEBUG:boto:Headers: {} 2024-04-19T23:39:31.959 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:31.959 DEBUG:boto:Port: 8001 2024-04-19T23:39:31.959 DEBUG:boto:Params: {} 2024-04-19T23:39:31.959 DEBUG:boto:Token: None 2024-04-19T23:39:31.959 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:31 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:31.959 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:P4y+FeSjtvHno2BV2c5sl/h6MdY= 2024-04-19T23:39:31.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:P4y+FeSjtvHno2BV2c5sl/h6MdY=', 'Content-Length': '0'} 2024-04-19T23:39:31.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd4e8c426a7d6217-00662300b3-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.006 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.007 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.007 DEBUG:boto:path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:32.007 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:32.007 DEBUG:boto:Method: GET 2024-04-19T23:39:32.007 DEBUG:boto:Path: /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:32.007 DEBUG:boto:Data: 2024-04-19T23:39:32.007 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.007 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.007 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.007 DEBUG:boto:Params: {} 2024-04-19T23:39:32.008 DEBUG:boto:Token: None 2024-04-19T23:39:32.008 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo 2024-04-19T23:39:32.008 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TxjxII2KHlG18WB3GpG7BJAm5w0= 2024-04-19T23:39:32.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxjxII2KHlG18WB3GpG7BJAm5w0=', 'Content-Length': '0'} 2024-04-19T23:39:32.010 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Fo79-FV5lzRfFhYZKDH5zbjzjFRvFmo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da7f3b5bc42aad34-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.010 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.010 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.010 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.010 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.011 DEBUG:boto:Method: GET 2024-04-19T23:39:32.011 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.011 DEBUG:boto:Data: 2024-04-19T23:39:32.011 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.011 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.011 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.011 DEBUG:boto:Params: {} 2024-04-19T23:39:32.011 DEBUG:boto:Token: None 2024-04-19T23:39:32.011 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.011 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097c023282f8a4a68-00662300b4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.015 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.015 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.016 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.016 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.016 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.016 DEBUG:boto:Method: GET 2024-04-19T23:39:32.016 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.016 DEBUG:boto:Data: 2024-04-19T23:39:32.016 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.016 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.016 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.017 DEBUG:boto:Params: {} 2024-04-19T23:39:32.017 DEBUG:boto:Token: None 2024-04-19T23:39:32.017 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.017 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098cbd83bfa959d9d-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.020 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.020 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.021 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.021 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.021 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:32.021 DEBUG:boto:Data: 2024-04-19T23:39:32.021 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.021 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.021 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.021 DEBUG:boto:Params: {} 2024-04-19T23:39:32.021 DEBUG:boto:Token: None 2024-04-19T23:39:32.022 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj 2024-04-19T23:39:32.022 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po= 2024-04-19T23:39:32.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po=', 'Content-Length': '0'} 2024-04-19T23:39:32.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000504390ebc6b20125-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.025 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.026 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.026 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.026 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:32.026 DEBUG:boto:Data: 2024-04-19T23:39:32.026 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.026 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.026 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.026 DEBUG:boto:Params: {} 2024-04-19T23:39:32.026 DEBUG:boto:Token: None 2024-04-19T23:39:32.026 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj 2024-04-19T23:39:32.027 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po= 2024-04-19T23:39:32.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po=', 'Content-Length': '0'} 2024-04-19T23:39:32.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e70adb1cd746aadc-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.030 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.030 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.031 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.031 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.031 DEBUG:boto:Method: GET 2024-04-19T23:39:32.031 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.031 DEBUG:boto:Data: 2024-04-19T23:39:32.031 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.031 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.031 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.031 DEBUG:boto:Params: {} 2024-04-19T23:39:32.031 DEBUG:boto:Token: None 2024-04-19T23:39:32.031 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.032 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk= 2024-04-19T23:39:32.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk=', 'Content-Length': '0'} 2024-04-19T23:39:32.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003489f40a47c2a3b9-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.078 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.079 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.079 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.079 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.079 DEBUG:boto:Method: GET 2024-04-19T23:39:32.079 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.079 DEBUG:boto:Data: 2024-04-19T23:39:32.079 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.079 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.079 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.079 DEBUG:boto:Params: {} 2024-04-19T23:39:32.080 DEBUG:boto:Token: None 2024-04-19T23:39:32.080 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.080 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk= 2024-04-19T23:39:32.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk=', 'Content-Length': '0'} 2024-04-19T23:39:32.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b7859c066b8c727-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.082 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.082 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.082 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.083 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.083 DEBUG:boto:Method: GET 2024-04-19T23:39:32.083 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.083 DEBUG:boto:Data: 2024-04-19T23:39:32.083 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.083 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.083 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.083 DEBUG:boto:Params: {} 2024-04-19T23:39:32.083 DEBUG:boto:Token: None 2024-04-19T23:39:32.083 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bab1c17eed8c1be-00662300b4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.134 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.135 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.135 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.135 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.135 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.135 DEBUG:boto:Method: GET 2024-04-19T23:39:32.135 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.135 DEBUG:boto:Data: 2024-04-19T23:39:32.136 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.136 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.136 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.136 DEBUG:boto:Params: {} 2024-04-19T23:39:32.136 DEBUG:boto:Token: None 2024-04-19T23:39:32.136 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.136 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026183ab5690c8356-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.139 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.140 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.140 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.140 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.140 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.140 DEBUG:boto:Method: GET 2024-04-19T23:39:32.140 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.140 DEBUG:boto:Data: 2024-04-19T23:39:32.140 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.140 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.141 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.141 DEBUG:boto:Params: {} 2024-04-19T23:39:32.141 DEBUG:boto:Token: None 2024-04-19T23:39:32.141 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.141 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BHtFHTPQTN4e24u7ge2CAoLzhMw= 2024-04-19T23:39:32.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BHtFHTPQTN4e24u7ge2CAoLzhMw=', 'Content-Length': '0'} 2024-04-19T23:39:32.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eacff525ca92d054-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.144 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.144 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.144 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.144 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.144 DEBUG:boto:Method: GET 2024-04-19T23:39:32.144 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.144 DEBUG:boto:Data: 2024-04-19T23:39:32.144 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.144 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.144 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.144 DEBUG:boto:Params: {} 2024-04-19T23:39:32.145 DEBUG:boto:Token: None 2024-04-19T23:39:32.145 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.145 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BHtFHTPQTN4e24u7ge2CAoLzhMw= 2024-04-19T23:39:32.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BHtFHTPQTN4e24u7ge2CAoLzhMw=', 'Content-Length': '0'} 2024-04-19T23:39:32.147 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a3f5bcc024c56ba-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.147 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.147 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.147 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.147 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.147 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.147 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.148 DEBUG:boto:Data: 2024-04-19T23:39:32.148 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.148 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.148 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.148 DEBUG:boto:Params: {} 2024-04-19T23:39:32.148 DEBUG:boto:Token: None 2024-04-19T23:39:32.148 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.148 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iH52wrCgirlMn/TetQF/jNk8mGw= 2024-04-19T23:39:32.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iH52wrCgirlMn/TetQF/jNk8mGw=', 'Content-Length': '0'} 2024-04-19T23:39:32.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000daa891d4f75d9758-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.150 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.150 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.150 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.151 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.151 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.151 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.151 DEBUG:boto:Data: 2024-04-19T23:39:32.151 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.151 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.151 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.151 DEBUG:boto:Params: {} 2024-04-19T23:39:32.151 DEBUG:boto:Token: None 2024-04-19T23:39:32.151 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.151 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iH52wrCgirlMn/TetQF/jNk8mGw= 2024-04-19T23:39:32.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iH52wrCgirlMn/TetQF/jNk8mGw=', 'Content-Length': '0'} 2024-04-19T23:39:32.153 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bb5e26b55723cb6-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.154 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.154 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.154 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.154 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.154 DEBUG:boto:Method: GET 2024-04-19T23:39:32.154 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.154 DEBUG:boto:Data: 2024-04-19T23:39:32.154 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.155 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.155 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.155 DEBUG:boto:Params: {} 2024-04-19T23:39:32.155 DEBUG:boto:Token: None 2024-04-19T23:39:32.155 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.155 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BHtFHTPQTN4e24u7ge2CAoLzhMw= 2024-04-19T23:39:32.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BHtFHTPQTN4e24u7ge2CAoLzhMw=', 'Content-Length': '0'} 2024-04-19T23:39:32.157 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000508cb4a3c91d7ca0-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.202 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.203 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.203 DEBUG:boto:path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.203 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.203 DEBUG:boto:Method: GET 2024-04-19T23:39:32.203 DEBUG:boto:Path: /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.203 DEBUG:boto:Data: 2024-04-19T23:39:32.203 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.203 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.203 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.203 DEBUG:boto:Params: {} 2024-04-19T23:39:32.204 DEBUG:boto:Token: None 2024-04-19T23:39:32.204 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT 2024-04-19T23:39:32.204 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BHtFHTPQTN4e24u7ge2CAoLzhMw= 2024-04-19T23:39:32.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BHtFHTPQTN4e24u7ge2CAoLzhMw=', 'Content-Length': '0'} 2024-04-19T23:39:32.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'Z.9ZzH.1hozF9j0blCcYN54OsXVmWMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7663f9116190058-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.250 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.251 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.251 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.251 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.251 DEBUG:boto:Method: GET 2024-04-19T23:39:32.251 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.251 DEBUG:boto:Data: 2024-04-19T23:39:32.251 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.251 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.251 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.251 DEBUG:boto:Params: {} 2024-04-19T23:39:32.252 DEBUG:boto:Token: None 2024-04-19T23:39:32.252 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.252 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c51ff210541d834f-00662300b4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.302 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.303 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.303 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.303 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.303 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.303 DEBUG:boto:Method: GET 2024-04-19T23:39:32.303 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.303 DEBUG:boto:Data: 2024-04-19T23:39:32.303 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.304 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.304 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.304 DEBUG:boto:Params: {} 2024-04-19T23:39:32.304 DEBUG:boto:Token: None 2024-04-19T23:39:32.304 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.304 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea6f38dcae4307d7-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.307 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.308 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.308 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.308 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.308 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:32.308 DEBUG:boto:Data: 2024-04-19T23:39:32.308 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.308 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.308 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.308 DEBUG:boto:Params: {} 2024-04-19T23:39:32.309 DEBUG:boto:Token: None 2024-04-19T23:39:32.309 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj 2024-04-19T23:39:32.309 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po= 2024-04-19T23:39:32.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po=', 'Content-Length': '0'} 2024-04-19T23:39:32.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9f2748af1bc30d0-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.313 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.313 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.313 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.313 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:32.313 DEBUG:boto:Data: 2024-04-19T23:39:32.313 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.313 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.313 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.313 DEBUG:boto:Params: {} 2024-04-19T23:39:32.313 DEBUG:boto:Token: None 2024-04-19T23:39:32.314 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj 2024-04-19T23:39:32.314 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po= 2024-04-19T23:39:32.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po=', 'Content-Length': '0'} 2024-04-19T23:39:32.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008974b3fdf3d959f5-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.318 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.318 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.318 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.318 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.318 DEBUG:boto:Method: GET 2024-04-19T23:39:32.318 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.318 DEBUG:boto:Data: 2024-04-19T23:39:32.318 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.318 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.318 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.319 DEBUG:boto:Params: {} 2024-04-19T23:39:32.319 DEBUG:boto:Token: None 2024-04-19T23:39:32.319 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.319 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk= 2024-04-19T23:39:32.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk=', 'Content-Length': '0'} 2024-04-19T23:39:32.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa557c7e9b6ecca0-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.362 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.363 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.363 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.363 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.363 DEBUG:boto:Method: GET 2024-04-19T23:39:32.363 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.363 DEBUG:boto:Data: 2024-04-19T23:39:32.363 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.363 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.363 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.364 DEBUG:boto:Params: {} 2024-04-19T23:39:32.364 DEBUG:boto:Token: None 2024-04-19T23:39:32.364 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.364 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk= 2024-04-19T23:39:32.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk=', 'Content-Length': '0'} 2024-04-19T23:39:32.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce9a67b03bb46d48-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.367 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.367 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.367 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.367 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.367 DEBUG:boto:Method: GET 2024-04-19T23:39:32.367 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.367 DEBUG:boto:Data: 2024-04-19T23:39:32.367 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.368 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.368 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.368 DEBUG:boto:Params: {} 2024-04-19T23:39:32.368 DEBUG:boto:Token: None 2024-04-19T23:39:32.368 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.368 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081b40ff8a23b17ea-00662300b4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.414 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.415 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.415 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.415 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.415 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.415 DEBUG:boto:Method: GET 2024-04-19T23:39:32.415 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.415 DEBUG:boto:Data: 2024-04-19T23:39:32.415 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.415 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.416 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.416 DEBUG:boto:Params: {} 2024-04-19T23:39:32.416 DEBUG:boto:Token: None 2024-04-19T23:39:32.416 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.416 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d22736345015f0ba-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.419 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.420 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.420 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.420 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.420 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.420 DEBUG:boto:Method: GET 2024-04-19T23:39:32.420 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.420 DEBUG:boto:Data: 2024-04-19T23:39:32.420 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.420 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.421 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.421 DEBUG:boto:Params: {} 2024-04-19T23:39:32.421 DEBUG:boto:Token: None 2024-04-19T23:39:32.421 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.422 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uzP8dyj2XJD4b95DBcWq6TQBGQo= 2024-04-19T23:39:32.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uzP8dyj2XJD4b95DBcWq6TQBGQo=', 'Content-Length': '0'} 2024-04-19T23:39:32.424 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007eba35baeb540349-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.424 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.424 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.424 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.424 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.424 DEBUG:boto:Method: GET 2024-04-19T23:39:32.424 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.425 DEBUG:boto:Data: 2024-04-19T23:39:32.425 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.425 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.425 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.425 DEBUG:boto:Params: {} 2024-04-19T23:39:32.425 DEBUG:boto:Token: None 2024-04-19T23:39:32.425 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.425 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uzP8dyj2XJD4b95DBcWq6TQBGQo= 2024-04-19T23:39:32.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uzP8dyj2XJD4b95DBcWq6TQBGQo=', 'Content-Length': '0'} 2024-04-19T23:39:32.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0d84bda4d597f3f-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.470 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.471 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.471 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.471 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.471 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.472 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.472 DEBUG:boto:Data: 2024-04-19T23:39:32.472 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.472 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.472 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.473 DEBUG:boto:Params: {} 2024-04-19T23:39:32.473 DEBUG:boto:Token: None 2024-04-19T23:39:32.473 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.474 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fH01t09yOBno1oGQmsqtQrnfP70= 2024-04-19T23:39:32.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fH01t09yOBno1oGQmsqtQrnfP70=', 'Content-Length': '0'} 2024-04-19T23:39:32.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050aba5c8a774decc-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.476 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.477 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.477 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.477 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.477 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.477 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.478 DEBUG:boto:Data: 2024-04-19T23:39:32.478 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.478 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.478 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.478 DEBUG:boto:Params: {} 2024-04-19T23:39:32.479 DEBUG:boto:Token: None 2024-04-19T23:39:32.479 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.479 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fH01t09yOBno1oGQmsqtQrnfP70= 2024-04-19T23:39:32.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fH01t09yOBno1oGQmsqtQrnfP70=', 'Content-Length': '0'} 2024-04-19T23:39:32.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce433f1308ba588d-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.482 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.482 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.482 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.482 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.482 DEBUG:boto:Method: GET 2024-04-19T23:39:32.483 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.483 DEBUG:boto:Data: 2024-04-19T23:39:32.483 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.483 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.483 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.483 DEBUG:boto:Params: {} 2024-04-19T23:39:32.483 DEBUG:boto:Token: None 2024-04-19T23:39:32.483 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.483 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uzP8dyj2XJD4b95DBcWq6TQBGQo= 2024-04-19T23:39:32.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uzP8dyj2XJD4b95DBcWq6TQBGQo=', 'Content-Length': '0'} 2024-04-19T23:39:32.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1bf1d032d1ab5f8-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.485 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.486 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.486 DEBUG:boto:path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.486 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.486 DEBUG:boto:Method: GET 2024-04-19T23:39:32.486 DEBUG:boto:Path: /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.486 DEBUG:boto:Data: 2024-04-19T23:39:32.486 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.486 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.486 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.487 DEBUG:boto:Params: {} 2024-04-19T23:39:32.487 DEBUG:boto:Token: None 2024-04-19T23:39:32.487 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA 2024-04-19T23:39:32.487 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uzP8dyj2XJD4b95DBcWq6TQBGQo= 2024-04-19T23:39:32.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uzP8dyj2XJD4b95DBcWq6TQBGQo=', 'Content-Length': '0'} 2024-04-19T23:39:32.489 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', 'nLS7759d.h0bWJMnVj7IAdVTVbZ2LzA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a3b8b576d856088-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.530 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.531 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.531 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.531 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.531 DEBUG:boto:Method: GET 2024-04-19T23:39:32.531 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.531 DEBUG:boto:Data: 2024-04-19T23:39:32.531 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.531 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.531 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.532 DEBUG:boto:Params: {} 2024-04-19T23:39:32.532 DEBUG:boto:Token: None 2024-04-19T23:39:32.532 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.532 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6d67437177885a1-00662300b4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.535 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.536 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.536 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.536 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.536 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.537 DEBUG:boto:Method: GET 2024-04-19T23:39:32.537 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.537 DEBUG:boto:Data: 2024-04-19T23:39:32.537 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.537 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.538 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.538 DEBUG:boto:Params: {} 2024-04-19T23:39:32.538 DEBUG:boto:Token: None 2024-04-19T23:39:32.538 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.539 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ee08e2655a8328b-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.543 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.543 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.543 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.544 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.544 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:32.544 DEBUG:boto:Data: 2024-04-19T23:39:32.544 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.544 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.545 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.545 DEBUG:boto:Params: {} 2024-04-19T23:39:32.545 DEBUG:boto:Token: None 2024-04-19T23:39:32.545 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj 2024-04-19T23:39:32.545 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po= 2024-04-19T23:39:32.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po=', 'Content-Length': '0'} 2024-04-19T23:39:32.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009cdb6b659f64a9ec-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.549 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.550 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.550 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.550 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:32.550 DEBUG:boto:Data: 2024-04-19T23:39:32.550 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.550 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.550 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.551 DEBUG:boto:Params: {} 2024-04-19T23:39:32.551 DEBUG:boto:Token: None 2024-04-19T23:39:32.551 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj 2024-04-19T23:39:32.551 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po= 2024-04-19T23:39:32.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po=', 'Content-Length': '0'} 2024-04-19T23:39:32.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c21ca54ec506743-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.555 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.555 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.555 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.555 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.555 DEBUG:boto:Method: GET 2024-04-19T23:39:32.556 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.556 DEBUG:boto:Data: 2024-04-19T23:39:32.556 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.556 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.556 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.556 DEBUG:boto:Params: {} 2024-04-19T23:39:32.556 DEBUG:boto:Token: None 2024-04-19T23:39:32.556 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.556 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk= 2024-04-19T23:39:32.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk=', 'Content-Length': '0'} 2024-04-19T23:39:32.558 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e08d21eb212ac765-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.598 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.599 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.599 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.599 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.599 DEBUG:boto:Method: GET 2024-04-19T23:39:32.599 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.599 DEBUG:boto:Data: 2024-04-19T23:39:32.599 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.599 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.599 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.599 DEBUG:boto:Params: {} 2024-04-19T23:39:32.600 DEBUG:boto:Token: None 2024-04-19T23:39:32.600 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.600 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk= 2024-04-19T23:39:32.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk=', 'Content-Length': '0'} 2024-04-19T23:39:32.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071d807b4746554b1-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.602 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.603 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.603 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.603 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.603 DEBUG:boto:Method: GET 2024-04-19T23:39:32.603 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.603 DEBUG:boto:Data: 2024-04-19T23:39:32.603 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.603 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.603 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.603 DEBUG:boto:Params: {} 2024-04-19T23:39:32.603 DEBUG:boto:Token: None 2024-04-19T23:39:32.604 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.604 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a19b52d5324ba386-00662300b4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.606 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.607 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.607 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.607 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.607 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.607 DEBUG:boto:Method: GET 2024-04-19T23:39:32.607 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.607 DEBUG:boto:Data: 2024-04-19T23:39:32.607 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.608 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.608 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.608 DEBUG:boto:Params: {} 2024-04-19T23:39:32.608 DEBUG:boto:Token: None 2024-04-19T23:39:32.608 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.608 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2ed6494531a2736-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.612 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.612 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.612 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.612 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.612 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.613 DEBUG:boto:Method: GET 2024-04-19T23:39:32.613 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.613 DEBUG:boto:Data: 2024-04-19T23:39:32.613 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.613 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.613 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.613 DEBUG:boto:Params: {} 2024-04-19T23:39:32.613 DEBUG:boto:Token: None 2024-04-19T23:39:32.613 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.613 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jb3/qI+xE1bs2+SPyUaRrlKN7xo= 2024-04-19T23:39:32.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jb3/qI+xE1bs2+SPyUaRrlKN7xo=', 'Content-Length': '0'} 2024-04-19T23:39:32.615 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007abddf95a44e2981-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.616 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.616 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.616 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.616 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.616 DEBUG:boto:Method: GET 2024-04-19T23:39:32.617 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.617 DEBUG:boto:Data: 2024-04-19T23:39:32.617 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.617 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.617 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.617 DEBUG:boto:Params: {} 2024-04-19T23:39:32.617 DEBUG:boto:Token: None 2024-04-19T23:39:32.617 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.617 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jb3/qI+xE1bs2+SPyUaRrlKN7xo= 2024-04-19T23:39:32.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jb3/qI+xE1bs2+SPyUaRrlKN7xo=', 'Content-Length': '0'} 2024-04-19T23:39:32.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003dbfca7afe10486-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.620 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.620 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.620 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.620 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.620 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.620 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.620 DEBUG:boto:Data: 2024-04-19T23:39:32.620 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.620 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.620 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.620 DEBUG:boto:Params: {} 2024-04-19T23:39:32.621 DEBUG:boto:Token: None 2024-04-19T23:39:32.621 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.621 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YCRujJXoKBY1ddav/dK9ABojcF4= 2024-04-19T23:39:32.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YCRujJXoKBY1ddav/dK9ABojcF4=', 'Content-Length': '0'} 2024-04-19T23:39:32.623 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab9aaca2c10fcbae-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.623 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.623 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.623 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.623 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.623 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.623 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.624 DEBUG:boto:Data: 2024-04-19T23:39:32.624 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.624 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.624 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.624 DEBUG:boto:Params: {} 2024-04-19T23:39:32.624 DEBUG:boto:Token: None 2024-04-19T23:39:32.624 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.624 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YCRujJXoKBY1ddav/dK9ABojcF4= 2024-04-19T23:39:32.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YCRujJXoKBY1ddav/dK9ABojcF4=', 'Content-Length': '0'} 2024-04-19T23:39:32.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d4cdb0ccc5165a6-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.626 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.627 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.627 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.627 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.627 DEBUG:boto:Method: GET 2024-04-19T23:39:32.627 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.627 DEBUG:boto:Data: 2024-04-19T23:39:32.627 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.627 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.627 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.627 DEBUG:boto:Params: {} 2024-04-19T23:39:32.627 DEBUG:boto:Token: None 2024-04-19T23:39:32.628 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jb3/qI+xE1bs2+SPyUaRrlKN7xo= 2024-04-19T23:39:32.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jb3/qI+xE1bs2+SPyUaRrlKN7xo=', 'Content-Length': '0'} 2024-04-19T23:39:32.630 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086ae29eb8dcdf643-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.670 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.671 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.671 DEBUG:boto:path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.671 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.671 DEBUG:boto:Method: GET 2024-04-19T23:39:32.671 DEBUG:boto:Path: /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.671 DEBUG:boto:Data: 2024-04-19T23:39:32.671 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.671 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.671 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.672 DEBUG:boto:Params: {} 2024-04-19T23:39:32.672 DEBUG:boto:Token: None 2024-04-19T23:39:32.672 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=.SsB-LJAFCELIGc-2lad2-2jTKdmQIu 2024-04-19T23:39:32.672 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jb3/qI+xE1bs2+SPyUaRrlKN7xo= 2024-04-19T23:39:32.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jb3/qI+xE1bs2+SPyUaRrlKN7xo=', 'Content-Length': '0'} 2024-04-19T23:39:32.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.SsB-LJAFCELIGc-2lad2-2jTKdmQIu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004abef75453c4cd8b-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.675 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.675 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.675 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.675 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.675 DEBUG:boto:Method: GET 2024-04-19T23:39:32.675 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.675 DEBUG:boto:Data: 2024-04-19T23:39:32.675 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.675 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.675 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.675 DEBUG:boto:Params: {} 2024-04-19T23:39:32.676 DEBUG:boto:Token: None 2024-04-19T23:39:32.676 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.676 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a8019edf64d0819-00662300b4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.679 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.679 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.679 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.679 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.679 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.679 DEBUG:boto:Method: GET 2024-04-19T23:39:32.680 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.680 DEBUG:boto:Data: 2024-04-19T23:39:32.680 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.680 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.680 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.680 DEBUG:boto:Params: {} 2024-04-19T23:39:32.680 DEBUG:boto:Token: None 2024-04-19T23:39:32.680 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.680 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8e071bf8b22487-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.726 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.727 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.727 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.727 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.727 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:32.727 DEBUG:boto:Data: 2024-04-19T23:39:32.727 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.727 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.727 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.728 DEBUG:boto:Params: {} 2024-04-19T23:39:32.728 DEBUG:boto:Token: None 2024-04-19T23:39:32.728 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj 2024-04-19T23:39:32.728 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po= 2024-04-19T23:39:32.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po=', 'Content-Length': '0'} 2024-04-19T23:39:32.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011b01541b7c6fd39-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.731 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.732 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.732 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.732 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:32.732 DEBUG:boto:Data: 2024-04-19T23:39:32.732 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.732 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.732 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.732 DEBUG:boto:Params: {} 2024-04-19T23:39:32.732 DEBUG:boto:Token: None 2024-04-19T23:39:32.732 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj 2024-04-19T23:39:32.733 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po= 2024-04-19T23:39:32.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po=', 'Content-Length': '0'} 2024-04-19T23:39:32.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be58e31681e8f870-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.736 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.736 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.736 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.737 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.737 DEBUG:boto:Method: GET 2024-04-19T23:39:32.737 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.737 DEBUG:boto:Data: 2024-04-19T23:39:32.737 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.737 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.737 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.737 DEBUG:boto:Params: {} 2024-04-19T23:39:32.737 DEBUG:boto:Token: None 2024-04-19T23:39:32.737 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.737 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk= 2024-04-19T23:39:32.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk=', 'Content-Length': '0'} 2024-04-19T23:39:32.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a314ab5ab7ac511-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.786 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.786 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.787 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.787 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.787 DEBUG:boto:Method: GET 2024-04-19T23:39:32.787 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.787 DEBUG:boto:Data: 2024-04-19T23:39:32.787 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.787 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.787 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.787 DEBUG:boto:Params: {} 2024-04-19T23:39:32.787 DEBUG:boto:Token: None 2024-04-19T23:39:32.787 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.788 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk= 2024-04-19T23:39:32.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk=', 'Content-Length': '0'} 2024-04-19T23:39:32.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000afaaef57bfe5b72b-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.790 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.790 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.790 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.790 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.790 DEBUG:boto:Method: GET 2024-04-19T23:39:32.791 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.791 DEBUG:boto:Data: 2024-04-19T23:39:32.791 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.791 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.791 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.791 DEBUG:boto:Params: {} 2024-04-19T23:39:32.791 DEBUG:boto:Token: None 2024-04-19T23:39:32.791 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.791 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026bccc6e4f300cd3-00662300b4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.794 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.795 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.795 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.795 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.795 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.795 DEBUG:boto:Method: GET 2024-04-19T23:39:32.795 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.795 DEBUG:boto:Data: 2024-04-19T23:39:32.795 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.795 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.795 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.796 DEBUG:boto:Params: {} 2024-04-19T23:39:32.796 DEBUG:boto:Token: None 2024-04-19T23:39:32.796 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.796 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d40c3b460de15e3a-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.799 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.800 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.800 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.800 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.800 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.800 DEBUG:boto:Method: GET 2024-04-19T23:39:32.800 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.800 DEBUG:boto:Data: 2024-04-19T23:39:32.800 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.800 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.800 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.800 DEBUG:boto:Params: {} 2024-04-19T23:39:32.801 DEBUG:boto:Token: None 2024-04-19T23:39:32.801 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.801 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SH1yxT/Y1+pfVxhp4A+k6c/ldnA= 2024-04-19T23:39:32.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SH1yxT/Y1+pfVxhp4A+k6c/ldnA=', 'Content-Length': '0'} 2024-04-19T23:39:32.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000371913ec427283ad-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.803 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.803 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.803 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.803 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.804 DEBUG:boto:Method: GET 2024-04-19T23:39:32.804 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.804 DEBUG:boto:Data: 2024-04-19T23:39:32.804 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.804 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.804 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.804 DEBUG:boto:Params: {} 2024-04-19T23:39:32.804 DEBUG:boto:Token: None 2024-04-19T23:39:32.804 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SH1yxT/Y1+pfVxhp4A+k6c/ldnA= 2024-04-19T23:39:32.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SH1yxT/Y1+pfVxhp4A+k6c/ldnA=', 'Content-Length': '0'} 2024-04-19T23:39:32.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1c0d6a6ef98f953-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.807 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.807 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.807 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.807 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.807 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.807 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.808 DEBUG:boto:Data: 2024-04-19T23:39:32.808 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.808 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.808 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.808 DEBUG:boto:Params: {} 2024-04-19T23:39:32.808 DEBUG:boto:Token: None 2024-04-19T23:39:32.808 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.808 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SWZSmHCbmGTe8swyAtrPOxWQ2bk= 2024-04-19T23:39:32.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SWZSmHCbmGTe8swyAtrPOxWQ2bk=', 'Content-Length': '0'} 2024-04-19T23:39:32.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046e0091c216d4563-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.811 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.811 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.811 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.811 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.811 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.811 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.811 DEBUG:boto:Data: 2024-04-19T23:39:32.811 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.811 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.811 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.812 DEBUG:boto:Params: {} 2024-04-19T23:39:32.812 DEBUG:boto:Token: None 2024-04-19T23:39:32.812 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.812 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SWZSmHCbmGTe8swyAtrPOxWQ2bk= 2024-04-19T23:39:32.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SWZSmHCbmGTe8swyAtrPOxWQ2bk=', 'Content-Length': '0'} 2024-04-19T23:39:32.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000126cc0eb7a289b1f-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.814 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.814 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.814 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.814 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.815 DEBUG:boto:Method: GET 2024-04-19T23:39:32.815 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.815 DEBUG:boto:Data: 2024-04-19T23:39:32.815 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.815 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.815 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.815 DEBUG:boto:Params: {} 2024-04-19T23:39:32.815 DEBUG:boto:Token: None 2024-04-19T23:39:32.815 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.815 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SH1yxT/Y1+pfVxhp4A+k6c/ldnA= 2024-04-19T23:39:32.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SH1yxT/Y1+pfVxhp4A+k6c/ldnA=', 'Content-Length': '0'} 2024-04-19T23:39:32.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c8b421a8314e10eb-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.858 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.859 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.862 DEBUG:boto:path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.862 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.862 DEBUG:boto:Method: GET 2024-04-19T23:39:32.862 DEBUG:boto:Path: /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.862 DEBUG:boto:Data: 2024-04-19T23:39:32.862 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.862 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.862 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.862 DEBUG:boto:Params: {} 2024-04-19T23:39:32.863 DEBUG:boto:Token: None 2024-04-19T23:39:32.863 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=5IMca-NQG3TDmPnhaU0a46jfFv2nxR8 2024-04-19T23:39:32.863 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SH1yxT/Y1+pfVxhp4A+k6c/ldnA= 2024-04-19T23:39:32.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SH1yxT/Y1+pfVxhp4A+k6c/ldnA=', 'Content-Length': '0'} 2024-04-19T23:39:32.866 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '5IMca-NQG3TDmPnhaU0a46jfFv2nxR8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027c48f809440e8bb-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.906 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.907 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.907 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.907 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.907 DEBUG:boto:Method: GET 2024-04-19T23:39:32.907 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.907 DEBUG:boto:Data: 2024-04-19T23:39:32.907 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.907 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.907 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.907 DEBUG:boto:Params: {} 2024-04-19T23:39:32.908 DEBUG:boto:Token: None 2024-04-19T23:39:32.908 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.908 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001857e09b8156da11-00662300b4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.911 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.911 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.911 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.912 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.912 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.912 DEBUG:boto:Method: GET 2024-04-19T23:39:32.912 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.912 DEBUG:boto:Data: 2024-04-19T23:39:32.912 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.912 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.912 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.912 DEBUG:boto:Params: {} 2024-04-19T23:39:32.912 DEBUG:boto:Token: None 2024-04-19T23:39:32.913 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.913 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027553b967c21e8a5-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.916 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:32.916 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.916 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.916 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.916 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:32.916 DEBUG:boto:Data: 2024-04-19T23:39:32.917 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.917 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.917 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.917 DEBUG:boto:Params: {} 2024-04-19T23:39:32.917 DEBUG:boto:Token: None 2024-04-19T23:39:32.917 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj 2024-04-19T23:39:32.917 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po= 2024-04-19T23:39:32.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po=', 'Content-Length': '0'} 2024-04-19T23:39:32.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073b344002d9f66eb-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.921 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.921 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.921 DEBUG:boto:Method: HEAD 2024-04-19T23:39:32.921 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:32.921 DEBUG:boto:Data: 2024-04-19T23:39:32.921 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.921 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.922 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.922 DEBUG:boto:Params: {} 2024-04-19T23:39:32.922 DEBUG:boto:Token: None 2024-04-19T23:39:32.922 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj 2024-04-19T23:39:32.922 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po= 2024-04-19T23:39:32.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8r6IU0H02cTiFmJKQ8TMfJRw4Po=', 'Content-Length': '0'} 2024-04-19T23:39:32.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002234f017fa2a5552-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:32.926 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.926 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.926 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.926 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.926 DEBUG:boto:Method: GET 2024-04-19T23:39:32.926 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.927 DEBUG:boto:Data: 2024-04-19T23:39:32.927 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.927 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.927 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.927 DEBUG:boto:Params: {} 2024-04-19T23:39:32.927 DEBUG:boto:Token: None 2024-04-19T23:39:32.927 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.928 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk= 2024-04-19T23:39:32.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk=', 'Content-Length': '0'} 2024-04-19T23:39:32.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c30de28e7d0fa7b-00662300b4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.970 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:32.971 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:32.971 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.971 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.971 DEBUG:boto:Method: GET 2024-04-19T23:39:32.971 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.971 DEBUG:boto:Data: 2024-04-19T23:39:32.971 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.971 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:32.971 DEBUG:boto:Port: 8000 2024-04-19T23:39:32.972 DEBUG:boto:Params: {} 2024-04-19T23:39:32.972 DEBUG:boto:Token: None 2024-04-19T23:39:32.972 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:32.972 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk= 2024-04-19T23:39:32.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bhI203zIKpKaTNNrm+x4ZsvvYLk=', 'Content-Length': '0'} 2024-04-19T23:39:32.974 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b1d6d3fa56a146f-00662300b4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:32.974 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:32.975 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:32.975 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.975 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:32.975 DEBUG:boto:Method: GET 2024-04-19T23:39:32.975 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:32.975 DEBUG:boto:Data: 2024-04-19T23:39:32.975 DEBUG:boto:Headers: {} 2024-04-19T23:39:32.975 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:32.975 DEBUG:boto:Port: 8001 2024-04-19T23:39:32.975 DEBUG:boto:Params: {} 2024-04-19T23:39:32.976 DEBUG:boto:Token: None 2024-04-19T23:39:32.976 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:32 GMT /aktlji-22/ 2024-04-19T23:39:32.976 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY= 2024-04-19T23:39:32.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rslEaG+l/KtnfeY3oiPorJGC9FY=', 'Content-Length': '0'} 2024-04-19T23:39:32.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002de7d694e7d2f874-00662300b4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.026 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.027 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.027 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.027 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.027 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.028 DEBUG:boto:Method: GET 2024-04-19T23:39:33.028 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.028 DEBUG:boto:Data: 2024-04-19T23:39:33.028 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.028 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.028 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.028 DEBUG:boto:Params: {} 2024-04-19T23:39:33.028 DEBUG:boto:Token: None 2024-04-19T23:39:33.029 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.029 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6553b3351f13f9a-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.032 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:33.032 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.032 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.032 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.033 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.033 DEBUG:boto:Method: GET 2024-04-19T23:39:33.033 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.033 DEBUG:boto:Data: 2024-04-19T23:39:33.033 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.033 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.033 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.033 DEBUG:boto:Params: {} 2024-04-19T23:39:33.033 DEBUG:boto:Token: None 2024-04-19T23:39:33.033 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.033 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MatrNDDOQO812Luzy1t7h81JTZM= 2024-04-19T23:39:33.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MatrNDDOQO812Luzy1t7h81JTZM=', 'Content-Length': '0'} 2024-04-19T23:39:33.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0a5b9bf1df7d4b7-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.036 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.036 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.036 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.036 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.036 DEBUG:boto:Method: GET 2024-04-19T23:39:33.036 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.036 DEBUG:boto:Data: 2024-04-19T23:39:33.037 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.037 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.037 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.037 DEBUG:boto:Params: {} 2024-04-19T23:39:33.037 DEBUG:boto:Token: None 2024-04-19T23:39:33.037 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.037 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MatrNDDOQO812Luzy1t7h81JTZM= 2024-04-19T23:39:33.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MatrNDDOQO812Luzy1t7h81JTZM=', 'Content-Length': '0'} 2024-04-19T23:39:33.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e23dccb685a24a74-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.082 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.083 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.083 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.083 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.083 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.083 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.083 DEBUG:boto:Data: 2024-04-19T23:39:33.083 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.084 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.084 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.084 DEBUG:boto:Params: {} 2024-04-19T23:39:33.084 DEBUG:boto:Token: None 2024-04-19T23:39:33.084 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.084 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vOZfJha+Z+5bDG1bC7qV2En2cyo= 2024-04-19T23:39:33.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vOZfJha+Z+5bDG1bC7qV2En2cyo=', 'Content-Length': '0'} 2024-04-19T23:39:33.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000394c1eca03c30275-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.087 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.087 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.087 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.087 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.087 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.087 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.087 DEBUG:boto:Data: 2024-04-19T23:39:33.088 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.088 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.088 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.088 DEBUG:boto:Params: {} 2024-04-19T23:39:33.088 DEBUG:boto:Token: None 2024-04-19T23:39:33.088 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.088 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vOZfJha+Z+5bDG1bC7qV2En2cyo= 2024-04-19T23:39:33.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vOZfJha+Z+5bDG1bC7qV2En2cyo=', 'Content-Length': '0'} 2024-04-19T23:39:33.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e91ba12f057b65d4-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:33.091 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.091 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.091 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.091 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.091 DEBUG:boto:Method: GET 2024-04-19T23:39:33.091 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.091 DEBUG:boto:Data: 2024-04-19T23:39:33.091 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.091 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.091 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.091 DEBUG:boto:Params: {} 2024-04-19T23:39:33.092 DEBUG:boto:Token: None 2024-04-19T23:39:33.092 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.092 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MatrNDDOQO812Luzy1t7h81JTZM= 2024-04-19T23:39:33.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MatrNDDOQO812Luzy1t7h81JTZM=', 'Content-Length': '0'} 2024-04-19T23:39:33.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b5185fe9d24a392-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.134 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.135 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.135 DEBUG:boto:path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.135 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.135 DEBUG:boto:Method: GET 2024-04-19T23:39:33.135 DEBUG:boto:Path: /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.135 DEBUG:boto:Data: 2024-04-19T23:39:33.135 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.135 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.135 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.135 DEBUG:boto:Params: {} 2024-04-19T23:39:33.135 DEBUG:boto:Token: None 2024-04-19T23:39:33.136 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi 2024-04-19T23:39:33.136 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MatrNDDOQO812Luzy1t7h81JTZM= 2024-04-19T23:39:33.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MatrNDDOQO812Luzy1t7h81JTZM=', 'Content-Length': '0'} 2024-04-19T23:39:33.138 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'p3PIDP6j1VZfmBljRrKUGkIUmq4DPgi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007379ec5b4532e3a4-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.178 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.178 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.179 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.179 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.179 DEBUG:boto:Method: GET 2024-04-19T23:39:33.179 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.179 DEBUG:boto:Data: 2024-04-19T23:39:33.179 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.179 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.179 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.179 DEBUG:boto:Params: {} 2024-04-19T23:39:33.179 DEBUG:boto:Token: None 2024-04-19T23:39:33.180 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a09a04ab3ceb8c5a-00662300b5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.226 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.227 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.227 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.227 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.227 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.227 DEBUG:boto:Method: GET 2024-04-19T23:39:33.228 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.228 DEBUG:boto:Data: 2024-04-19T23:39:33.228 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.228 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.228 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.228 DEBUG:boto:Params: {} 2024-04-19T23:39:33.228 DEBUG:boto:Token: None 2024-04-19T23:39:33.228 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.229 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8b4aa6257a9fe3-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.232 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.232 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.232 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.232 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.233 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:33.233 DEBUG:boto:Data: 2024-04-19T23:39:33.233 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.233 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.233 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.233 DEBUG:boto:Params: {} 2024-04-19T23:39:33.233 DEBUG:boto:Token: None 2024-04-19T23:39:33.233 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj 2024-04-19T23:39:33.234 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts= 2024-04-19T23:39:33.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts=', 'Content-Length': '0'} 2024-04-19T23:39:33.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085e7c5c5cf35907b-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.237 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.238 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.238 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.238 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:33.238 DEBUG:boto:Data: 2024-04-19T23:39:33.238 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.238 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.238 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.238 DEBUG:boto:Params: {} 2024-04-19T23:39:33.239 DEBUG:boto:Token: None 2024-04-19T23:39:33.239 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj 2024-04-19T23:39:33.239 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts= 2024-04-19T23:39:33.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts=', 'Content-Length': '0'} 2024-04-19T23:39:33.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a4cf758dcd35d89-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:33.243 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.243 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.243 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.243 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.243 DEBUG:boto:Method: GET 2024-04-19T23:39:33.243 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.243 DEBUG:boto:Data: 2024-04-19T23:39:33.244 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.244 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.244 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.244 DEBUG:boto:Params: {} 2024-04-19T23:39:33.244 DEBUG:boto:Token: None 2024-04-19T23:39:33.244 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.244 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k= 2024-04-19T23:39:33.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k=', 'Content-Length': '0'} 2024-04-19T23:39:33.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000805488835a4cce0a-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.294 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.295 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.295 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.295 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.295 DEBUG:boto:Method: GET 2024-04-19T23:39:33.295 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.295 DEBUG:boto:Data: 2024-04-19T23:39:33.295 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.296 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.296 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.296 DEBUG:boto:Params: {} 2024-04-19T23:39:33.296 DEBUG:boto:Token: None 2024-04-19T23:39:33.296 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.296 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k= 2024-04-19T23:39:33.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k=', 'Content-Length': '0'} 2024-04-19T23:39:33.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f00ca7ebdd9951a-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.299 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.299 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.299 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.299 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.300 DEBUG:boto:Method: GET 2024-04-19T23:39:33.300 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.300 DEBUG:boto:Data: 2024-04-19T23:39:33.300 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.300 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.300 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.300 DEBUG:boto:Params: {} 2024-04-19T23:39:33.300 DEBUG:boto:Token: None 2024-04-19T23:39:33.301 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.301 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f3e0b48c7d1702-00662300b5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.304 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.304 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.304 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.305 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.305 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.305 DEBUG:boto:Method: GET 2024-04-19T23:39:33.305 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.305 DEBUG:boto:Data: 2024-04-19T23:39:33.305 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.305 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.305 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.305 DEBUG:boto:Params: {} 2024-04-19T23:39:33.305 DEBUG:boto:Token: None 2024-04-19T23:39:33.306 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.306 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3b6e68b0ada8ab5-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.309 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:33.309 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.309 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.309 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.310 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.310 DEBUG:boto:Method: GET 2024-04-19T23:39:33.311 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.311 DEBUG:boto:Data: 2024-04-19T23:39:33.311 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.311 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.311 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.311 DEBUG:boto:Params: {} 2024-04-19T23:39:33.311 DEBUG:boto:Token: None 2024-04-19T23:39:33.311 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.311 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g44wdqdW18vshQ9RahGCs75abzk= 2024-04-19T23:39:33.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g44wdqdW18vshQ9RahGCs75abzk=', 'Content-Length': '0'} 2024-04-19T23:39:33.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a302d7b99338fc84-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.358 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.358 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.359 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.359 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.359 DEBUG:boto:Method: GET 2024-04-19T23:39:33.359 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.359 DEBUG:boto:Data: 2024-04-19T23:39:33.359 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.359 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.359 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.359 DEBUG:boto:Params: {} 2024-04-19T23:39:33.359 DEBUG:boto:Token: None 2024-04-19T23:39:33.360 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.360 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g44wdqdW18vshQ9RahGCs75abzk= 2024-04-19T23:39:33.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g44wdqdW18vshQ9RahGCs75abzk=', 'Content-Length': '0'} 2024-04-19T23:39:33.362 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063eea1ad7fb83366-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.362 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.362 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.362 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.362 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.363 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.363 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.363 DEBUG:boto:Data: 2024-04-19T23:39:33.363 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.363 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.363 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.363 DEBUG:boto:Params: {} 2024-04-19T23:39:33.363 DEBUG:boto:Token: None 2024-04-19T23:39:33.363 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.363 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2wtAWbP8jYUgTZ/T/cwBKNyGbUk= 2024-04-19T23:39:33.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2wtAWbP8jYUgTZ/T/cwBKNyGbUk=', 'Content-Length': '0'} 2024-04-19T23:39:33.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000034454e7693304c5-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.366 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.366 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.366 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.366 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.366 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.366 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.366 DEBUG:boto:Data: 2024-04-19T23:39:33.367 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.367 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.367 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.367 DEBUG:boto:Params: {} 2024-04-19T23:39:33.367 DEBUG:boto:Token: None 2024-04-19T23:39:33.367 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.367 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2wtAWbP8jYUgTZ/T/cwBKNyGbUk= 2024-04-19T23:39:33.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2wtAWbP8jYUgTZ/T/cwBKNyGbUk=', 'Content-Length': '0'} 2024-04-19T23:39:33.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086b6df3e051b97a5-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:33.370 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.370 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.370 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.370 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.370 DEBUG:boto:Method: GET 2024-04-19T23:39:33.370 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.370 DEBUG:boto:Data: 2024-04-19T23:39:33.370 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.370 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.370 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.370 DEBUG:boto:Params: {} 2024-04-19T23:39:33.371 DEBUG:boto:Token: None 2024-04-19T23:39:33.371 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.371 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g44wdqdW18vshQ9RahGCs75abzk= 2024-04-19T23:39:33.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g44wdqdW18vshQ9RahGCs75abzk=', 'Content-Length': '0'} 2024-04-19T23:39:33.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e84636c712576ae9-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.414 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.414 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.415 DEBUG:boto:path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.415 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.415 DEBUG:boto:Method: GET 2024-04-19T23:39:33.415 DEBUG:boto:Path: /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.415 DEBUG:boto:Data: 2024-04-19T23:39:33.415 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.415 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.415 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.415 DEBUG:boto:Params: {} 2024-04-19T23:39:33.415 DEBUG:boto:Token: None 2024-04-19T23:39:33.416 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=.sqcIpylWkdZwQp4UqpDBbyySodTBXS 2024-04-19T23:39:33.416 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g44wdqdW18vshQ9RahGCs75abzk= 2024-04-19T23:39:33.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g44wdqdW18vshQ9RahGCs75abzk=', 'Content-Length': '0'} 2024-04-19T23:39:33.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.sqcIpylWkdZwQp4UqpDBbyySodTBXS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072d01bc62e60173e-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.418 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.418 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.418 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.418 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.418 DEBUG:boto:Method: GET 2024-04-19T23:39:33.418 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.418 DEBUG:boto:Data: 2024-04-19T23:39:33.418 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.419 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.419 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.419 DEBUG:boto:Params: {} 2024-04-19T23:39:33.419 DEBUG:boto:Token: None 2024-04-19T23:39:33.419 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.419 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec4c4cfc59773b01-00662300b5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.422 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.423 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.423 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.423 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.423 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.423 DEBUG:boto:Method: GET 2024-04-19T23:39:33.423 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.423 DEBUG:boto:Data: 2024-04-19T23:39:33.423 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.423 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.423 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.423 DEBUG:boto:Params: {} 2024-04-19T23:39:33.424 DEBUG:boto:Token: None 2024-04-19T23:39:33.424 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.424 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6d87bcb21ef726b-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.427 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.427 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.427 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.427 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.428 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:33.428 DEBUG:boto:Data: 2024-04-19T23:39:33.428 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.428 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.428 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.428 DEBUG:boto:Params: {} 2024-04-19T23:39:33.428 DEBUG:boto:Token: None 2024-04-19T23:39:33.428 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj 2024-04-19T23:39:33.428 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts= 2024-04-19T23:39:33.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts=', 'Content-Length': '0'} 2024-04-19T23:39:33.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3d0795c2860f4d1-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.432 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.432 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.432 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.432 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:33.432 DEBUG:boto:Data: 2024-04-19T23:39:33.432 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.433 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.433 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.433 DEBUG:boto:Params: {} 2024-04-19T23:39:33.433 DEBUG:boto:Token: None 2024-04-19T23:39:33.433 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj 2024-04-19T23:39:33.433 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts= 2024-04-19T23:39:33.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts=', 'Content-Length': '0'} 2024-04-19T23:39:33.437 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000980d95f067a9d087-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:33.437 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.437 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.437 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.437 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.437 DEBUG:boto:Method: GET 2024-04-19T23:39:33.437 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.438 DEBUG:boto:Data: 2024-04-19T23:39:33.438 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.438 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.438 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.438 DEBUG:boto:Params: {} 2024-04-19T23:39:33.438 DEBUG:boto:Token: None 2024-04-19T23:39:33.438 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.438 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k= 2024-04-19T23:39:33.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k=', 'Content-Length': '0'} 2024-04-19T23:39:33.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f472059d01fc686-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.482 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.483 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.483 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.483 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.483 DEBUG:boto:Method: GET 2024-04-19T23:39:33.483 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.483 DEBUG:boto:Data: 2024-04-19T23:39:33.483 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.483 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.483 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.483 DEBUG:boto:Params: {} 2024-04-19T23:39:33.484 DEBUG:boto:Token: None 2024-04-19T23:39:33.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k= 2024-04-19T23:39:33.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k=', 'Content-Length': '0'} 2024-04-19T23:39:33.486 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007479e6ac0c48716f-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.486 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.486 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.486 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.487 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.487 DEBUG:boto:Method: GET 2024-04-19T23:39:33.487 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.487 DEBUG:boto:Data: 2024-04-19T23:39:33.487 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.487 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.487 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.487 DEBUG:boto:Params: {} 2024-04-19T23:39:33.487 DEBUG:boto:Token: None 2024-04-19T23:39:33.487 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fa9b6e3bfa75519-00662300b5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.490 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.491 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.491 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.491 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.491 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.491 DEBUG:boto:Method: GET 2024-04-19T23:39:33.491 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.491 DEBUG:boto:Data: 2024-04-19T23:39:33.491 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.492 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.492 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.492 DEBUG:boto:Params: {} 2024-04-19T23:39:33.492 DEBUG:boto:Token: None 2024-04-19T23:39:33.492 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.492 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e93e4f6c4eaafa99-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.495 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:33.495 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.496 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.497 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.497 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.497 DEBUG:boto:Method: GET 2024-04-19T23:39:33.497 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.497 DEBUG:boto:Data: 2024-04-19T23:39:33.497 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.497 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.497 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.497 DEBUG:boto:Params: {} 2024-04-19T23:39:33.498 DEBUG:boto:Token: None 2024-04-19T23:39:33.498 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.498 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:knzH81/eQnlKiYjjSuiXOWDs9vo= 2024-04-19T23:39:33.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:knzH81/eQnlKiYjjSuiXOWDs9vo=', 'Content-Length': '0'} 2024-04-19T23:39:33.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f86604bf01eeed23-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.500 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.500 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.501 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.501 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.501 DEBUG:boto:Method: GET 2024-04-19T23:39:33.501 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.501 DEBUG:boto:Data: 2024-04-19T23:39:33.501 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.501 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.501 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.501 DEBUG:boto:Params: {} 2024-04-19T23:39:33.501 DEBUG:boto:Token: None 2024-04-19T23:39:33.501 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.502 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:knzH81/eQnlKiYjjSuiXOWDs9vo= 2024-04-19T23:39:33.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:knzH81/eQnlKiYjjSuiXOWDs9vo=', 'Content-Length': '0'} 2024-04-19T23:39:33.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c305450e11df7893-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.550 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.551 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.551 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.551 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.551 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.551 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.551 DEBUG:boto:Data: 2024-04-19T23:39:33.551 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.551 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.551 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.551 DEBUG:boto:Params: {} 2024-04-19T23:39:33.552 DEBUG:boto:Token: None 2024-04-19T23:39:33.552 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.552 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X3t+VHvObLgT/5LOSDFprsfd0rI= 2024-04-19T23:39:33.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X3t+VHvObLgT/5LOSDFprsfd0rI=', 'Content-Length': '0'} 2024-04-19T23:39:33.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092138ca9baa40885-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.554 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.554 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.554 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.554 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.554 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.555 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.555 DEBUG:boto:Data: 2024-04-19T23:39:33.555 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.555 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.555 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.555 DEBUG:boto:Params: {} 2024-04-19T23:39:33.555 DEBUG:boto:Token: None 2024-04-19T23:39:33.555 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.555 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X3t+VHvObLgT/5LOSDFprsfd0rI= 2024-04-19T23:39:33.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X3t+VHvObLgT/5LOSDFprsfd0rI=', 'Content-Length': '0'} 2024-04-19T23:39:33.557 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af246cd8443d00ec-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:33.558 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.558 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.558 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.558 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.558 DEBUG:boto:Method: GET 2024-04-19T23:39:33.558 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.558 DEBUG:boto:Data: 2024-04-19T23:39:33.558 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.559 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.559 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.559 DEBUG:boto:Params: {} 2024-04-19T23:39:33.559 DEBUG:boto:Token: None 2024-04-19T23:39:33.559 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.559 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:knzH81/eQnlKiYjjSuiXOWDs9vo= 2024-04-19T23:39:33.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:knzH81/eQnlKiYjjSuiXOWDs9vo=', 'Content-Length': '0'} 2024-04-19T23:39:33.561 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049da1224f053800b-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.603 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.603 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.603 DEBUG:boto:path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.603 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.603 DEBUG:boto:Method: GET 2024-04-19T23:39:33.603 DEBUG:boto:Path: /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.603 DEBUG:boto:Data: 2024-04-19T23:39:33.603 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.603 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.604 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.604 DEBUG:boto:Params: {} 2024-04-19T23:39:33.604 DEBUG:boto:Token: None 2024-04-19T23:39:33.604 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL 2024-04-19T23:39:33.604 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:knzH81/eQnlKiYjjSuiXOWDs9vo= 2024-04-19T23:39:33.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:knzH81/eQnlKiYjjSuiXOWDs9vo=', 'Content-Length': '0'} 2024-04-19T23:39:33.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'BCzz.Nb3nU2CWGe2vMCDYlTHkK18iRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dbfb99bb566db42-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.646 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.647 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.647 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.647 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.647 DEBUG:boto:Method: GET 2024-04-19T23:39:33.647 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.647 DEBUG:boto:Data: 2024-04-19T23:39:33.647 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.647 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.647 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.647 DEBUG:boto:Params: {} 2024-04-19T23:39:33.648 DEBUG:boto:Token: None 2024-04-19T23:39:33.648 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.648 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb4b3f95854da533-00662300b5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.694 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.695 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.695 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.695 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.695 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.695 DEBUG:boto:Method: GET 2024-04-19T23:39:33.695 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.695 DEBUG:boto:Data: 2024-04-19T23:39:33.695 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.695 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.696 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.696 DEBUG:boto:Params: {} 2024-04-19T23:39:33.696 DEBUG:boto:Token: None 2024-04-19T23:39:33.696 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.696 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d755e9c935a1d70-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.699 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.700 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.700 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.700 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.700 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:33.700 DEBUG:boto:Data: 2024-04-19T23:39:33.700 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.700 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.700 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.700 DEBUG:boto:Params: {} 2024-04-19T23:39:33.700 DEBUG:boto:Token: None 2024-04-19T23:39:33.701 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj 2024-04-19T23:39:33.701 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts= 2024-04-19T23:39:33.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts=', 'Content-Length': '0'} 2024-04-19T23:39:33.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000df7d1ae59ef37eb-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.704 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.704 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.705 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.705 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:33.705 DEBUG:boto:Data: 2024-04-19T23:39:33.705 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.705 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.705 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.705 DEBUG:boto:Params: {} 2024-04-19T23:39:33.705 DEBUG:boto:Token: None 2024-04-19T23:39:33.705 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj 2024-04-19T23:39:33.705 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts= 2024-04-19T23:39:33.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts=', 'Content-Length': '0'} 2024-04-19T23:39:33.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055f45d73e476be4b-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:33.709 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.709 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.709 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.710 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.710 DEBUG:boto:Method: GET 2024-04-19T23:39:33.710 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.710 DEBUG:boto:Data: 2024-04-19T23:39:33.710 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.710 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.710 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.710 DEBUG:boto:Params: {} 2024-04-19T23:39:33.710 DEBUG:boto:Token: None 2024-04-19T23:39:33.710 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.711 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k= 2024-04-19T23:39:33.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k=', 'Content-Length': '0'} 2024-04-19T23:39:33.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bbbcced1e293070-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.754 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.754 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.755 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.755 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.755 DEBUG:boto:Method: GET 2024-04-19T23:39:33.755 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.755 DEBUG:boto:Data: 2024-04-19T23:39:33.755 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.755 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.755 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.755 DEBUG:boto:Params: {} 2024-04-19T23:39:33.755 DEBUG:boto:Token: None 2024-04-19T23:39:33.756 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.756 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k= 2024-04-19T23:39:33.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k=', 'Content-Length': '0'} 2024-04-19T23:39:33.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b44829fc8a5b1e99-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.758 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.758 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.758 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.758 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.758 DEBUG:boto:Method: GET 2024-04-19T23:39:33.759 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.759 DEBUG:boto:Data: 2024-04-19T23:39:33.759 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.759 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.759 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.759 DEBUG:boto:Params: {} 2024-04-19T23:39:33.759 DEBUG:boto:Token: None 2024-04-19T23:39:33.759 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.759 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de3dccfbddc5a94b-00662300b5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.802 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.803 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.803 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.803 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.803 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.803 DEBUG:boto:Method: GET 2024-04-19T23:39:33.803 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.804 DEBUG:boto:Data: 2024-04-19T23:39:33.804 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.804 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.804 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.804 DEBUG:boto:Params: {} 2024-04-19T23:39:33.804 DEBUG:boto:Token: None 2024-04-19T23:39:33.804 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1bc89ec3b16554e-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.808 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:33.808 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.808 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.808 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.809 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.809 DEBUG:boto:Method: GET 2024-04-19T23:39:33.809 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.809 DEBUG:boto:Data: 2024-04-19T23:39:33.809 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.809 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.809 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.809 DEBUG:boto:Params: {} 2024-04-19T23:39:33.809 DEBUG:boto:Token: None 2024-04-19T23:39:33.809 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.810 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bv/e7zcp11LVdiU4RN8r50H0yos= 2024-04-19T23:39:33.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bv/e7zcp11LVdiU4RN8r50H0yos=', 'Content-Length': '0'} 2024-04-19T23:39:33.812 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c08a9d5a1699e000-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.812 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.812 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.812 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.813 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.813 DEBUG:boto:Method: GET 2024-04-19T23:39:33.813 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.813 DEBUG:boto:Data: 2024-04-19T23:39:33.813 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.813 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.813 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.813 DEBUG:boto:Params: {} 2024-04-19T23:39:33.813 DEBUG:boto:Token: None 2024-04-19T23:39:33.813 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.813 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bv/e7zcp11LVdiU4RN8r50H0yos= 2024-04-19T23:39:33.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bv/e7zcp11LVdiU4RN8r50H0yos=', 'Content-Length': '0'} 2024-04-19T23:39:33.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7bf733d955291f1-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.816 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.816 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.816 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.816 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.816 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.816 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.816 DEBUG:boto:Data: 2024-04-19T23:39:33.816 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.817 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.817 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.817 DEBUG:boto:Params: {} 2024-04-19T23:39:33.817 DEBUG:boto:Token: None 2024-04-19T23:39:33.817 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.817 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pXOfRPXK7W+pTJSh9OYr57nb/aQ= 2024-04-19T23:39:33.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pXOfRPXK7W+pTJSh9OYr57nb/aQ=', 'Content-Length': '0'} 2024-04-19T23:39:33.819 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c353054df1bd627-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.819 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.819 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.820 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.820 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.820 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.820 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.820 DEBUG:boto:Data: 2024-04-19T23:39:33.820 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.820 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.820 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.820 DEBUG:boto:Params: {} 2024-04-19T23:39:33.820 DEBUG:boto:Token: None 2024-04-19T23:39:33.820 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.821 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pXOfRPXK7W+pTJSh9OYr57nb/aQ= 2024-04-19T23:39:33.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pXOfRPXK7W+pTJSh9OYr57nb/aQ=', 'Content-Length': '0'} 2024-04-19T23:39:33.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e6b0616cf0f5190-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:33.823 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.823 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.823 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.823 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.823 DEBUG:boto:Method: GET 2024-04-19T23:39:33.823 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.823 DEBUG:boto:Data: 2024-04-19T23:39:33.823 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.824 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.824 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.824 DEBUG:boto:Params: {} 2024-04-19T23:39:33.824 DEBUG:boto:Token: None 2024-04-19T23:39:33.824 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bv/e7zcp11LVdiU4RN8r50H0yos= 2024-04-19T23:39:33.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bv/e7zcp11LVdiU4RN8r50H0yos=', 'Content-Length': '0'} 2024-04-19T23:39:33.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002dda93149a72b3c1-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.870 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.870 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.871 DEBUG:boto:path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.871 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.871 DEBUG:boto:Method: GET 2024-04-19T23:39:33.871 DEBUG:boto:Path: /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.871 DEBUG:boto:Data: 2024-04-19T23:39:33.871 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.871 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.871 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.871 DEBUG:boto:Params: {} 2024-04-19T23:39:33.871 DEBUG:boto:Token: None 2024-04-19T23:39:33.872 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=udeGFtd00C8TRIOlco59Eefd7c7ffA6 2024-04-19T23:39:33.872 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bv/e7zcp11LVdiU4RN8r50H0yos= 2024-04-19T23:39:33.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bv/e7zcp11LVdiU4RN8r50H0yos=', 'Content-Length': '0'} 2024-04-19T23:39:33.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'udeGFtd00C8TRIOlco59Eefd7c7ffA6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064167a746f2a706f-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.914 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.915 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.915 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.915 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.915 DEBUG:boto:Method: GET 2024-04-19T23:39:33.915 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.915 DEBUG:boto:Data: 2024-04-19T23:39:33.915 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.915 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.915 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.915 DEBUG:boto:Params: {} 2024-04-19T23:39:33.916 DEBUG:boto:Token: None 2024-04-19T23:39:33.916 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.916 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050b467f4255941e8-00662300b5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.919 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.920 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:33.920 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:33.920 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.920 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:33.920 DEBUG:boto:Method: GET 2024-04-19T23:39:33.920 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:33.920 DEBUG:boto:Data: 2024-04-19T23:39:33.920 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.920 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.920 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.920 DEBUG:boto:Params: {} 2024-04-19T23:39:33.921 DEBUG:boto:Token: None 2024-04-19T23:39:33.921 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/ 2024-04-19T23:39:33.921 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI= 2024-04-19T23:39:33.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hf2t+rEQQFvKPcUMOD5GyBVfDUI=', 'Content-Length': '0'} 2024-04-19T23:39:33.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b49a5e2cb2d30673-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.970 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:33.971 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.971 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.971 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.971 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:33.971 DEBUG:boto:Data: 2024-04-19T23:39:33.971 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.971 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.971 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.972 DEBUG:boto:Params: {} 2024-04-19T23:39:33.972 DEBUG:boto:Token: None 2024-04-19T23:39:33.972 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj 2024-04-19T23:39:33.972 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts= 2024-04-19T23:39:33.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts=', 'Content-Length': '0'} 2024-04-19T23:39:33.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004807f82be64d9133-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.976 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.976 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.976 DEBUG:boto:Method: HEAD 2024-04-19T23:39:33.976 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:33.976 DEBUG:boto:Data: 2024-04-19T23:39:33.976 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.976 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:33.976 DEBUG:boto:Port: 8000 2024-04-19T23:39:33.976 DEBUG:boto:Params: {} 2024-04-19T23:39:33.976 DEBUG:boto:Token: None 2024-04-19T23:39:33.977 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj 2024-04-19T23:39:33.977 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts= 2024-04-19T23:39:33.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RRymaZZzhT99jgvR2guGDLbo3ts=', 'Content-Length': '0'} 2024-04-19T23:39:33.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e0e6b4551115c22-00662300b5-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:33.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:33.981 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:33.981 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:33.981 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.981 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.981 DEBUG:boto:Method: GET 2024-04-19T23:39:33.981 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.981 DEBUG:boto:Data: 2024-04-19T23:39:33.981 DEBUG:boto:Headers: {} 2024-04-19T23:39:33.981 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:33.982 DEBUG:boto:Port: 8001 2024-04-19T23:39:33.982 DEBUG:boto:Params: {} 2024-04-19T23:39:33.982 DEBUG:boto:Token: None 2024-04-19T23:39:33.982 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:33 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:33.982 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k= 2024-04-19T23:39:33.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z6xt/GtVfq7SPxx6CJHahfli1+k=', 'Content-Length': '0'} 2024-04-19T23:39:33.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ec7c0b45d5ad55e-00662300b5-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.026 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.027 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.027 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.027 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.027 DEBUG:boto:Method: GET 2024-04-19T23:39:34.027 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.027 DEBUG:boto:Data: 2024-04-19T23:39:34.027 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.027 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.027 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.028 DEBUG:boto:Params: {} 2024-04-19T23:39:34.028 DEBUG:boto:Token: None 2024-04-19T23:39:34.028 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.028 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs= 2024-04-19T23:39:34.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs=', 'Content-Length': '0'} 2024-04-19T23:39:34.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5014227373ac0e5-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.030 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.031 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.031 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.031 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.031 DEBUG:boto:Method: GET 2024-04-19T23:39:34.031 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.031 DEBUG:boto:Data: 2024-04-19T23:39:34.031 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.031 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.031 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.031 DEBUG:boto:Params: {} 2024-04-19T23:39:34.031 DEBUG:boto:Token: None 2024-04-19T23:39:34.032 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.032 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000894f650ed1abfa13-00662300b6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.035 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.035 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.036 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.036 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.036 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.036 DEBUG:boto:Method: GET 2024-04-19T23:39:34.036 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.036 DEBUG:boto:Data: 2024-04-19T23:39:34.036 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.036 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.036 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.036 DEBUG:boto:Params: {} 2024-04-19T23:39:34.036 DEBUG:boto:Token: None 2024-04-19T23:39:34.037 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.037 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058baa27127146aeb-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.040 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.040 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.040 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.040 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.040 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.040 DEBUG:boto:Method: GET 2024-04-19T23:39:34.041 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.041 DEBUG:boto:Data: 2024-04-19T23:39:34.041 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.041 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.041 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.041 DEBUG:boto:Params: {} 2024-04-19T23:39:34.041 DEBUG:boto:Token: None 2024-04-19T23:39:34.041 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.041 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nKhh3V/BtOP7plWGgbLEAM+WIKw= 2024-04-19T23:39:34.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nKhh3V/BtOP7plWGgbLEAM+WIKw=', 'Content-Length': '0'} 2024-04-19T23:39:34.043 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d01c14478681d91f-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.090 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.091 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.091 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.091 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.091 DEBUG:boto:Method: GET 2024-04-19T23:39:34.091 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.091 DEBUG:boto:Data: 2024-04-19T23:39:34.091 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.091 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.091 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.091 DEBUG:boto:Params: {} 2024-04-19T23:39:34.092 DEBUG:boto:Token: None 2024-04-19T23:39:34.092 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.092 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nKhh3V/BtOP7plWGgbLEAM+WIKw= 2024-04-19T23:39:34.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nKhh3V/BtOP7plWGgbLEAM+WIKw=', 'Content-Length': '0'} 2024-04-19T23:39:34.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b3f082e21698d22-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.094 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.094 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.095 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.095 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.095 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.095 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.095 DEBUG:boto:Data: 2024-04-19T23:39:34.095 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.095 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.095 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.095 DEBUG:boto:Params: {} 2024-04-19T23:39:34.095 DEBUG:boto:Token: None 2024-04-19T23:39:34.096 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.096 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gK59b2FHfDWb92BhGHETVQsziLg= 2024-04-19T23:39:34.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gK59b2FHfDWb92BhGHETVQsziLg=', 'Content-Length': '0'} 2024-04-19T23:39:34.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e27e67dda7b418fc-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.098 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.098 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.098 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.098 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.098 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.098 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.098 DEBUG:boto:Data: 2024-04-19T23:39:34.098 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.099 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.099 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.099 DEBUG:boto:Params: {} 2024-04-19T23:39:34.099 DEBUG:boto:Token: None 2024-04-19T23:39:34.099 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.099 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gK59b2FHfDWb92BhGHETVQsziLg= 2024-04-19T23:39:34.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gK59b2FHfDWb92BhGHETVQsziLg=', 'Content-Length': '0'} 2024-04-19T23:39:34.101 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a19e491fb72ef492-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.102 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.102 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.102 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.102 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.102 DEBUG:boto:Method: GET 2024-04-19T23:39:34.102 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.102 DEBUG:boto:Data: 2024-04-19T23:39:34.102 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.102 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.103 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.103 DEBUG:boto:Params: {} 2024-04-19T23:39:34.103 DEBUG:boto:Token: None 2024-04-19T23:39:34.103 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.103 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nKhh3V/BtOP7plWGgbLEAM+WIKw= 2024-04-19T23:39:34.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nKhh3V/BtOP7plWGgbLEAM+WIKw=', 'Content-Length': '0'} 2024-04-19T23:39:34.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c7ad63cd723e5f7-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.105 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.105 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.105 DEBUG:boto:path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.106 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.106 DEBUG:boto:Method: GET 2024-04-19T23:39:34.106 DEBUG:boto:Path: /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.106 DEBUG:boto:Data: 2024-04-19T23:39:34.106 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.106 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.106 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.106 DEBUG:boto:Params: {} 2024-04-19T23:39:34.106 DEBUG:boto:Token: None 2024-04-19T23:39:34.106 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4 2024-04-19T23:39:34.107 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nKhh3V/BtOP7plWGgbLEAM+WIKw= 2024-04-19T23:39:34.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nKhh3V/BtOP7plWGgbLEAM+WIKw=', 'Content-Length': '0'} 2024-04-19T23:39:34.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'RkDAmqWVGXN5Q6u97p7.-Ps8vpmzxj4'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b771e3474e3c1286-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.150 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.151 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.151 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.151 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.151 DEBUG:boto:Method: GET 2024-04-19T23:39:34.151 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.151 DEBUG:boto:Data: 2024-04-19T23:39:34.151 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.151 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.151 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.151 DEBUG:boto:Params: {} 2024-04-19T23:39:34.152 DEBUG:boto:Token: None 2024-04-19T23:39:34.152 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.152 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004702751fc8a64909-00662300b6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.155 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.156 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.156 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.156 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.156 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.156 DEBUG:boto:Method: GET 2024-04-19T23:39:34.156 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.156 DEBUG:boto:Data: 2024-04-19T23:39:34.156 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.156 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.156 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.156 DEBUG:boto:Params: {} 2024-04-19T23:39:34.157 DEBUG:boto:Token: None 2024-04-19T23:39:34.157 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.157 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041b044632ff3b797-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.202 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.203 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.203 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.203 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.203 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:34.203 DEBUG:boto:Data: 2024-04-19T23:39:34.203 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.203 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.203 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.204 DEBUG:boto:Params: {} 2024-04-19T23:39:34.204 DEBUG:boto:Token: None 2024-04-19T23:39:34.204 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj 2024-04-19T23:39:34.204 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw= 2024-04-19T23:39:34.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw=', 'Content-Length': '0'} 2024-04-19T23:39:34.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e063bddfe701f967-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.208 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.208 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.208 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.208 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:34.209 DEBUG:boto:Data: 2024-04-19T23:39:34.209 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.209 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.209 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.209 DEBUG:boto:Params: {} 2024-04-19T23:39:34.209 DEBUG:boto:Token: None 2024-04-19T23:39:34.209 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj 2024-04-19T23:39:34.209 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw= 2024-04-19T23:39:34.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw=', 'Content-Length': '0'} 2024-04-19T23:39:34.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1363088b483480a-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.213 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.213 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.213 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.213 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.213 DEBUG:boto:Method: GET 2024-04-19T23:39:34.213 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.213 DEBUG:boto:Data: 2024-04-19T23:39:34.213 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.213 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.214 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.214 DEBUG:boto:Params: {} 2024-04-19T23:39:34.214 DEBUG:boto:Token: None 2024-04-19T23:39:34.214 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.214 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs= 2024-04-19T23:39:34.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs=', 'Content-Length': '0'} 2024-04-19T23:39:34.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c364b08701975bb-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.258 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.259 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.259 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.259 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.259 DEBUG:boto:Method: GET 2024-04-19T23:39:34.259 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.259 DEBUG:boto:Data: 2024-04-19T23:39:34.259 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.259 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.259 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.259 DEBUG:boto:Params: {} 2024-04-19T23:39:34.260 DEBUG:boto:Token: None 2024-04-19T23:39:34.260 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.260 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs= 2024-04-19T23:39:34.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs=', 'Content-Length': '0'} 2024-04-19T23:39:34.262 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0f8f0096accafe7-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.262 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.262 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.262 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.262 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.263 DEBUG:boto:Method: GET 2024-04-19T23:39:34.263 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.263 DEBUG:boto:Data: 2024-04-19T23:39:34.263 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.263 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.263 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.263 DEBUG:boto:Params: {} 2024-04-19T23:39:34.263 DEBUG:boto:Token: None 2024-04-19T23:39:34.263 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.264 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c4be70b052bc8da-00662300b6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.314 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.315 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.315 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.315 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.315 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.315 DEBUG:boto:Method: GET 2024-04-19T23:39:34.315 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.316 DEBUG:boto:Data: 2024-04-19T23:39:34.316 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.316 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.316 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.316 DEBUG:boto:Params: {} 2024-04-19T23:39:34.316 DEBUG:boto:Token: None 2024-04-19T23:39:34.316 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.316 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b7116a0cd70a1e-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.320 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.320 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.320 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.321 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.321 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.321 DEBUG:boto:Method: GET 2024-04-19T23:39:34.321 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.321 DEBUG:boto:Data: 2024-04-19T23:39:34.321 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.321 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.321 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.321 DEBUG:boto:Params: {} 2024-04-19T23:39:34.321 DEBUG:boto:Token: None 2024-04-19T23:39:34.322 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.322 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wDl2dTNtrLO7UUa27tpIfXFQQwc= 2024-04-19T23:39:34.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wDl2dTNtrLO7UUa27tpIfXFQQwc=', 'Content-Length': '0'} 2024-04-19T23:39:34.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000a5337539160121-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.324 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.324 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.324 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.324 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.324 DEBUG:boto:Method: GET 2024-04-19T23:39:34.324 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.325 DEBUG:boto:Data: 2024-04-19T23:39:34.325 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.325 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.325 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.325 DEBUG:boto:Params: {} 2024-04-19T23:39:34.325 DEBUG:boto:Token: None 2024-04-19T23:39:34.325 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.325 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wDl2dTNtrLO7UUa27tpIfXFQQwc= 2024-04-19T23:39:34.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wDl2dTNtrLO7UUa27tpIfXFQQwc=', 'Content-Length': '0'} 2024-04-19T23:39:34.327 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004dda03db10ff84c1-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.327 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.328 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.328 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.328 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.328 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.328 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.328 DEBUG:boto:Data: 2024-04-19T23:39:34.328 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.328 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.328 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.328 DEBUG:boto:Params: {} 2024-04-19T23:39:34.328 DEBUG:boto:Token: None 2024-04-19T23:39:34.329 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.329 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2DGxWPwW0hVuMHuzXSm75L6dSnc= 2024-04-19T23:39:34.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2DGxWPwW0hVuMHuzXSm75L6dSnc=', 'Content-Length': '0'} 2024-04-19T23:39:34.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc6a08a73917055f-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.331 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.331 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.331 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.331 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.331 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.331 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.332 DEBUG:boto:Data: 2024-04-19T23:39:34.332 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.332 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.332 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.332 DEBUG:boto:Params: {} 2024-04-19T23:39:34.332 DEBUG:boto:Token: None 2024-04-19T23:39:34.332 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2DGxWPwW0hVuMHuzXSm75L6dSnc= 2024-04-19T23:39:34.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2DGxWPwW0hVuMHuzXSm75L6dSnc=', 'Content-Length': '0'} 2024-04-19T23:39:34.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000710079cc687896c4-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.334 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.335 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.335 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.335 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.335 DEBUG:boto:Method: GET 2024-04-19T23:39:34.335 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.335 DEBUG:boto:Data: 2024-04-19T23:39:34.335 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.335 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.335 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.335 DEBUG:boto:Params: {} 2024-04-19T23:39:34.335 DEBUG:boto:Token: None 2024-04-19T23:39:34.336 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.336 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wDl2dTNtrLO7UUa27tpIfXFQQwc= 2024-04-19T23:39:34.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wDl2dTNtrLO7UUa27tpIfXFQQwc=', 'Content-Length': '0'} 2024-04-19T23:39:34.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccd7db052b7c410c-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.382 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.382 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.383 DEBUG:boto:path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.383 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.383 DEBUG:boto:Method: GET 2024-04-19T23:39:34.383 DEBUG:boto:Path: /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.383 DEBUG:boto:Data: 2024-04-19T23:39:34.383 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.383 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.383 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.383 DEBUG:boto:Params: {} 2024-04-19T23:39:34.383 DEBUG:boto:Token: None 2024-04-19T23:39:34.384 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=goQj3EN8LOJNZ9C2JDigvTcIgypIbDx 2024-04-19T23:39:34.384 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wDl2dTNtrLO7UUa27tpIfXFQQwc= 2024-04-19T23:39:34.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wDl2dTNtrLO7UUa27tpIfXFQQwc=', 'Content-Length': '0'} 2024-04-19T23:39:34.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'goQj3EN8LOJNZ9C2JDigvTcIgypIbDx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d416efb70b4f59ef-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.430 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.431 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.431 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.431 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.431 DEBUG:boto:Method: GET 2024-04-19T23:39:34.431 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.431 DEBUG:boto:Data: 2024-04-19T23:39:34.431 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.431 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.431 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.431 DEBUG:boto:Params: {} 2024-04-19T23:39:34.431 DEBUG:boto:Token: None 2024-04-19T23:39:34.432 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.432 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000964f67ac9503a315-00662300b6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.482 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.483 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.483 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.483 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.483 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.483 DEBUG:boto:Method: GET 2024-04-19T23:39:34.483 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.483 DEBUG:boto:Data: 2024-04-19T23:39:34.483 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.484 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.484 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.484 DEBUG:boto:Params: {} 2024-04-19T23:39:34.484 DEBUG:boto:Token: None 2024-04-19T23:39:34.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000073ad9acb4667a6d-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.487 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.488 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.488 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.488 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.488 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:34.488 DEBUG:boto:Data: 2024-04-19T23:39:34.488 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.488 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.488 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.488 DEBUG:boto:Params: {} 2024-04-19T23:39:34.489 DEBUG:boto:Token: None 2024-04-19T23:39:34.489 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj 2024-04-19T23:39:34.489 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw= 2024-04-19T23:39:34.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw=', 'Content-Length': '0'} 2024-04-19T23:39:34.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030164450cb65e709-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.492 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.493 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.493 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.493 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:34.493 DEBUG:boto:Data: 2024-04-19T23:39:34.493 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.493 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.493 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.493 DEBUG:boto:Params: {} 2024-04-19T23:39:34.493 DEBUG:boto:Token: None 2024-04-19T23:39:34.493 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj 2024-04-19T23:39:34.494 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw= 2024-04-19T23:39:34.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw=', 'Content-Length': '0'} 2024-04-19T23:39:34.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035f856942e17bf15-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.497 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.497 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.498 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.498 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.498 DEBUG:boto:Method: GET 2024-04-19T23:39:34.498 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.498 DEBUG:boto:Data: 2024-04-19T23:39:34.498 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.498 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.498 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.498 DEBUG:boto:Params: {} 2024-04-19T23:39:34.498 DEBUG:boto:Token: None 2024-04-19T23:39:34.499 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.499 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs= 2024-04-19T23:39:34.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs=', 'Content-Length': '0'} 2024-04-19T23:39:34.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d93a256b8bfe28c-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.546 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.547 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.547 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.547 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.547 DEBUG:boto:Method: GET 2024-04-19T23:39:34.547 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.547 DEBUG:boto:Data: 2024-04-19T23:39:34.547 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.547 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.547 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.547 DEBUG:boto:Params: {} 2024-04-19T23:39:34.547 DEBUG:boto:Token: None 2024-04-19T23:39:34.548 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.548 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs= 2024-04-19T23:39:34.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs=', 'Content-Length': '0'} 2024-04-19T23:39:34.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ada55b83cfa3bf76-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.550 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.550 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.550 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.550 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.550 DEBUG:boto:Method: GET 2024-04-19T23:39:34.550 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.551 DEBUG:boto:Data: 2024-04-19T23:39:34.551 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.551 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.551 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.551 DEBUG:boto:Params: {} 2024-04-19T23:39:34.551 DEBUG:boto:Token: None 2024-04-19T23:39:34.551 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.551 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fd3de101de10491-00662300b6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.594 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.595 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.595 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.595 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.595 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.595 DEBUG:boto:Method: GET 2024-04-19T23:39:34.595 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.596 DEBUG:boto:Data: 2024-04-19T23:39:34.596 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.596 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.596 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.596 DEBUG:boto:Params: {} 2024-04-19T23:39:34.596 DEBUG:boto:Token: None 2024-04-19T23:39:34.596 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.596 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5dd386525d1ba1c-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.599 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.600 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.600 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.600 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.600 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.600 DEBUG:boto:Method: GET 2024-04-19T23:39:34.600 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.601 DEBUG:boto:Data: 2024-04-19T23:39:34.601 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.601 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.601 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.601 DEBUG:boto:Params: {} 2024-04-19T23:39:34.601 DEBUG:boto:Token: None 2024-04-19T23:39:34.601 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.601 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ejIN+3SRcXI0BcWeKJ13Yg0aTvc= 2024-04-19T23:39:34.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ejIN+3SRcXI0BcWeKJ13Yg0aTvc=', 'Content-Length': '0'} 2024-04-19T23:39:34.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000312ae3e0a902761f-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.604 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.604 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.604 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.604 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.604 DEBUG:boto:Method: GET 2024-04-19T23:39:34.604 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.604 DEBUG:boto:Data: 2024-04-19T23:39:34.605 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.605 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.605 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.605 DEBUG:boto:Params: {} 2024-04-19T23:39:34.605 DEBUG:boto:Token: None 2024-04-19T23:39:34.605 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.605 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ejIN+3SRcXI0BcWeKJ13Yg0aTvc= 2024-04-19T23:39:34.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ejIN+3SRcXI0BcWeKJ13Yg0aTvc=', 'Content-Length': '0'} 2024-04-19T23:39:34.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c05975cd612a5dd-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.650 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.651 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.651 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.651 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.651 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.651 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.651 DEBUG:boto:Data: 2024-04-19T23:39:34.651 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.651 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.651 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.651 DEBUG:boto:Params: {} 2024-04-19T23:39:34.652 DEBUG:boto:Token: None 2024-04-19T23:39:34.652 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.652 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:50bJJLrayhYpR6gyb+6kbg+/34s= 2024-04-19T23:39:34.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:50bJJLrayhYpR6gyb+6kbg+/34s=', 'Content-Length': '0'} 2024-04-19T23:39:34.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065a29e1a97eaf3a7-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.655 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.655 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.655 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.655 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.655 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.655 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.655 DEBUG:boto:Data: 2024-04-19T23:39:34.655 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.656 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.656 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.656 DEBUG:boto:Params: {} 2024-04-19T23:39:34.656 DEBUG:boto:Token: None 2024-04-19T23:39:34.656 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.656 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:50bJJLrayhYpR6gyb+6kbg+/34s= 2024-04-19T23:39:34.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:50bJJLrayhYpR6gyb+6kbg+/34s=', 'Content-Length': '0'} 2024-04-19T23:39:34.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c21229f364bad0cb-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.659 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.659 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.659 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.659 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.659 DEBUG:boto:Method: GET 2024-04-19T23:39:34.659 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.659 DEBUG:boto:Data: 2024-04-19T23:39:34.659 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.659 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.659 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.660 DEBUG:boto:Params: {} 2024-04-19T23:39:34.660 DEBUG:boto:Token: None 2024-04-19T23:39:34.660 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ejIN+3SRcXI0BcWeKJ13Yg0aTvc= 2024-04-19T23:39:34.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ejIN+3SRcXI0BcWeKJ13Yg0aTvc=', 'Content-Length': '0'} 2024-04-19T23:39:34.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2e930320b144928-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.710 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.710 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.710 DEBUG:boto:path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.710 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.710 DEBUG:boto:Method: GET 2024-04-19T23:39:34.710 DEBUG:boto:Path: /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.710 DEBUG:boto:Data: 2024-04-19T23:39:34.710 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.711 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.711 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.711 DEBUG:boto:Params: {} 2024-04-19T23:39:34.711 DEBUG:boto:Token: None 2024-04-19T23:39:34.711 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=m7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy 2024-04-19T23:39:34.711 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ejIN+3SRcXI0BcWeKJ13Yg0aTvc= 2024-04-19T23:39:34.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ejIN+3SRcXI0BcWeKJ13Yg0aTvc=', 'Content-Length': '0'} 2024-04-19T23:39:34.713 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'm7ZJ2gSVWi6V2uwFErRFtCny3.V9mzy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7572b28fa4b2db0-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.713 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.714 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.714 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.714 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.714 DEBUG:boto:Method: GET 2024-04-19T23:39:34.714 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.714 DEBUG:boto:Data: 2024-04-19T23:39:34.714 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.714 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.714 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.714 DEBUG:boto:Params: {} 2024-04-19T23:39:34.714 DEBUG:boto:Token: None 2024-04-19T23:39:34.715 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.715 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004263bc0bba0f8884-00662300b6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.718 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.718 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.718 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.718 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.718 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.718 DEBUG:boto:Method: GET 2024-04-19T23:39:34.719 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.719 DEBUG:boto:Data: 2024-04-19T23:39:34.719 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.719 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.719 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.719 DEBUG:boto:Params: {} 2024-04-19T23:39:34.719 DEBUG:boto:Token: None 2024-04-19T23:39:34.719 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.719 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d54906010531aa8d-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.722 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.722 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.723 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.723 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.723 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:34.723 DEBUG:boto:Data: 2024-04-19T23:39:34.723 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.723 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.723 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.723 DEBUG:boto:Params: {} 2024-04-19T23:39:34.723 DEBUG:boto:Token: None 2024-04-19T23:39:34.723 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj 2024-04-19T23:39:34.724 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw= 2024-04-19T23:39:34.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw=', 'Content-Length': '0'} 2024-04-19T23:39:34.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c41e3a5e28a1e9e-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.727 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.727 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.728 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.728 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:34.728 DEBUG:boto:Data: 2024-04-19T23:39:34.728 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.728 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.728 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.728 DEBUG:boto:Params: {} 2024-04-19T23:39:34.728 DEBUG:boto:Token: None 2024-04-19T23:39:34.728 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj 2024-04-19T23:39:34.729 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw= 2024-04-19T23:39:34.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw=', 'Content-Length': '0'} 2024-04-19T23:39:34.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000077f6b9edee780a7-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.732 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.732 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.733 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.733 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.733 DEBUG:boto:Method: GET 2024-04-19T23:39:34.733 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.733 DEBUG:boto:Data: 2024-04-19T23:39:34.733 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.733 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.733 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.733 DEBUG:boto:Params: {} 2024-04-19T23:39:34.733 DEBUG:boto:Token: None 2024-04-19T23:39:34.734 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.734 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs= 2024-04-19T23:39:34.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs=', 'Content-Length': '0'} 2024-04-19T23:39:34.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8024ea35eec364a-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.782 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.783 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.783 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.783 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.783 DEBUG:boto:Method: GET 2024-04-19T23:39:34.783 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.783 DEBUG:boto:Data: 2024-04-19T23:39:34.783 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.783 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.783 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.783 DEBUG:boto:Params: {} 2024-04-19T23:39:34.784 DEBUG:boto:Token: None 2024-04-19T23:39:34.784 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs= 2024-04-19T23:39:34.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs=', 'Content-Length': '0'} 2024-04-19T23:39:34.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c5eefe4a83393e3-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.787 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.787 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.787 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.787 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.787 DEBUG:boto:Method: GET 2024-04-19T23:39:34.787 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.787 DEBUG:boto:Data: 2024-04-19T23:39:34.787 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.787 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.787 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.787 DEBUG:boto:Params: {} 2024-04-19T23:39:34.788 DEBUG:boto:Token: None 2024-04-19T23:39:34.788 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.788 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c4355063ff9566e-00662300b6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.791 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.791 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.791 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.791 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.792 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.792 DEBUG:boto:Method: GET 2024-04-19T23:39:34.792 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.792 DEBUG:boto:Data: 2024-04-19T23:39:34.792 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.792 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.792 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.792 DEBUG:boto:Params: {} 2024-04-19T23:39:34.792 DEBUG:boto:Token: None 2024-04-19T23:39:34.792 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.792 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091782d1e0bd5c56d-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.795 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.796 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.796 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.796 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.796 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.796 DEBUG:boto:Method: GET 2024-04-19T23:39:34.796 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.796 DEBUG:boto:Data: 2024-04-19T23:39:34.797 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.797 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.797 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.797 DEBUG:boto:Params: {} 2024-04-19T23:39:34.797 DEBUG:boto:Token: None 2024-04-19T23:39:34.797 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.797 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YAZ1F8kg9Vx1tRSnBundf4aH5TU= 2024-04-19T23:39:34.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YAZ1F8kg9Vx1tRSnBundf4aH5TU=', 'Content-Length': '0'} 2024-04-19T23:39:34.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e86d87d83c308cfb-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.799 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.799 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.800 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.800 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.800 DEBUG:boto:Method: GET 2024-04-19T23:39:34.800 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.800 DEBUG:boto:Data: 2024-04-19T23:39:34.800 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.800 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.800 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.800 DEBUG:boto:Params: {} 2024-04-19T23:39:34.800 DEBUG:boto:Token: None 2024-04-19T23:39:34.801 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.801 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YAZ1F8kg9Vx1tRSnBundf4aH5TU= 2024-04-19T23:39:34.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YAZ1F8kg9Vx1tRSnBundf4aH5TU=', 'Content-Length': '0'} 2024-04-19T23:39:34.802 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000196fc95d0e5b279f-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.850 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.851 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.851 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.851 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.851 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.851 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.851 DEBUG:boto:Data: 2024-04-19T23:39:34.851 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.851 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.851 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.852 DEBUG:boto:Params: {} 2024-04-19T23:39:34.852 DEBUG:boto:Token: None 2024-04-19T23:39:34.852 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.852 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:I3J0xwYDIF23VMbz7h7WSn569cM= 2024-04-19T23:39:34.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:I3J0xwYDIF23VMbz7h7WSn569cM=', 'Content-Length': '0'} 2024-04-19T23:39:34.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eddc6137877746a9-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.854 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.854 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.854 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.854 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.855 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.855 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.855 DEBUG:boto:Data: 2024-04-19T23:39:34.855 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.855 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.855 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.855 DEBUG:boto:Params: {} 2024-04-19T23:39:34.855 DEBUG:boto:Token: None 2024-04-19T23:39:34.855 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.855 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:I3J0xwYDIF23VMbz7h7WSn569cM= 2024-04-19T23:39:34.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:I3J0xwYDIF23VMbz7h7WSn569cM=', 'Content-Length': '0'} 2024-04-19T23:39:34.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000997ffc325d0f2717-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.858 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.858 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.858 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.858 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.858 DEBUG:boto:Method: GET 2024-04-19T23:39:34.858 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.858 DEBUG:boto:Data: 2024-04-19T23:39:34.859 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.859 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.859 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.859 DEBUG:boto:Params: {} 2024-04-19T23:39:34.859 DEBUG:boto:Token: None 2024-04-19T23:39:34.859 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.859 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YAZ1F8kg9Vx1tRSnBundf4aH5TU= 2024-04-19T23:39:34.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YAZ1F8kg9Vx1tRSnBundf4aH5TU=', 'Content-Length': '0'} 2024-04-19T23:39:34.861 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018a79fc2673510a9-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.861 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.862 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.862 DEBUG:boto:path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.862 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.862 DEBUG:boto:Method: GET 2024-04-19T23:39:34.862 DEBUG:boto:Path: /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.862 DEBUG:boto:Data: 2024-04-19T23:39:34.862 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.862 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.862 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.862 DEBUG:boto:Params: {} 2024-04-19T23:39:34.863 DEBUG:boto:Token: None 2024-04-19T23:39:34.863 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=.169NmmrUry8iq70c7.ZNdp4i5iqOCF 2024-04-19T23:39:34.863 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YAZ1F8kg9Vx1tRSnBundf4aH5TU= 2024-04-19T23:39:34.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YAZ1F8kg9Vx1tRSnBundf4aH5TU=', 'Content-Length': '0'} 2024-04-19T23:39:34.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', '.169NmmrUry8iq70c7.ZNdp4i5iqOCF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ad156b93fe50fd0-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.910 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.911 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.911 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.911 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.911 DEBUG:boto:Method: GET 2024-04-19T23:39:34.911 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.911 DEBUG:boto:Data: 2024-04-19T23:39:34.911 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.911 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.911 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.912 DEBUG:boto:Params: {} 2024-04-19T23:39:34.912 DEBUG:boto:Token: None 2024-04-19T23:39:34.912 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.912 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e4b1912cf8bb1de-00662300b6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.915 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.915 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.916 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.916 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.916 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.916 DEBUG:boto:Method: GET 2024-04-19T23:39:34.916 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.916 DEBUG:boto:Data: 2024-04-19T23:39:34.916 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.916 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.916 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.916 DEBUG:boto:Params: {} 2024-04-19T23:39:34.917 DEBUG:boto:Token: None 2024-04-19T23:39:34.917 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.917 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070f4ec8ffbdbcdbb-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.920 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.920 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.920 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.921 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.921 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:34.921 DEBUG:boto:Data: 2024-04-19T23:39:34.921 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.921 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.921 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.921 DEBUG:boto:Params: {} 2024-04-19T23:39:34.921 DEBUG:boto:Token: None 2024-04-19T23:39:34.921 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj 2024-04-19T23:39:34.921 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw= 2024-04-19T23:39:34.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw=', 'Content-Length': '0'} 2024-04-19T23:39:34.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cc5287814b3696f-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.925 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.925 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.925 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.925 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:34.925 DEBUG:boto:Data: 2024-04-19T23:39:34.925 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.926 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.926 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.926 DEBUG:boto:Params: {} 2024-04-19T23:39:34.926 DEBUG:boto:Token: None 2024-04-19T23:39:34.926 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj 2024-04-19T23:39:34.926 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw= 2024-04-19T23:39:34.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vctfOzljZG+iQS67JCUZhPzCwLw=', 'Content-Length': '0'} 2024-04-19T23:39:34.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac4bea51a6dc2dfa-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.930 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.930 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.930 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.930 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.930 DEBUG:boto:Method: GET 2024-04-19T23:39:34.930 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.930 DEBUG:boto:Data: 2024-04-19T23:39:34.931 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.931 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.931 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.931 DEBUG:boto:Params: {} 2024-04-19T23:39:34.931 DEBUG:boto:Token: None 2024-04-19T23:39:34.931 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.931 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs= 2024-04-19T23:39:34.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs=', 'Content-Length': '0'} 2024-04-19T23:39:34.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f0e7c6ce3dc2003-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.974 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.975 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.975 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.975 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.975 DEBUG:boto:Method: GET 2024-04-19T23:39:34.975 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.975 DEBUG:boto:Data: 2024-04-19T23:39:34.975 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.975 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.975 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.975 DEBUG:boto:Params: {} 2024-04-19T23:39:34.976 DEBUG:boto:Token: None 2024-04-19T23:39:34.976 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:34.976 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs= 2024-04-19T23:39:34.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7VgPIIC0tza3BsSIubvYkcEhgqs=', 'Content-Length': '0'} 2024-04-19T23:39:34.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ceac3257db9ff3e-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.978 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.978 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.979 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.979 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.979 DEBUG:boto:Method: GET 2024-04-19T23:39:34.979 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.979 DEBUG:boto:Data: 2024-04-19T23:39:34.979 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.979 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.979 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.979 DEBUG:boto:Params: {} 2024-04-19T23:39:34.979 DEBUG:boto:Token: None 2024-04-19T23:39:34.980 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.980 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf73d82aff4efe6-00662300b6-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.982 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.983 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:34.983 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:34.983 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.983 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:34.983 DEBUG:boto:Method: GET 2024-04-19T23:39:34.983 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:34.984 DEBUG:boto:Data: 2024-04-19T23:39:34.984 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.984 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.984 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.984 DEBUG:boto:Params: {} 2024-04-19T23:39:34.984 DEBUG:boto:Token: None 2024-04-19T23:39:34.984 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/ 2024-04-19T23:39:34.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y= 2024-04-19T23:39:34.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MMsiaxw51pzWv6+kjXKYITyxS/Y=', 'Content-Length': '0'} 2024-04-19T23:39:34.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f5ed8665a27c56d-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.987 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:34.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:34.988 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.988 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.988 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.988 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.988 DEBUG:boto:Method: GET 2024-04-19T23:39:34.988 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.988 DEBUG:boto:Data: 2024-04-19T23:39:34.989 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.989 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.989 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.989 DEBUG:boto:Params: {} 2024-04-19T23:39:34.989 DEBUG:boto:Token: None 2024-04-19T23:39:34.989 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.989 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PVviJFtOiwIaUMop/40E+qLJNME= 2024-04-19T23:39:34.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PVviJFtOiwIaUMop/40E+qLJNME=', 'Content-Length': '0'} 2024-04-19T23:39:34.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009075e5d096b07f7d-00662300b6-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.992 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.992 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.992 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.992 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.992 DEBUG:boto:Method: GET 2024-04-19T23:39:34.992 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.993 DEBUG:boto:Data: 2024-04-19T23:39:34.993 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.993 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:34.993 DEBUG:boto:Port: 8000 2024-04-19T23:39:34.993 DEBUG:boto:Params: {} 2024-04-19T23:39:34.993 DEBUG:boto:Token: None 2024-04-19T23:39:34.993 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.993 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PVviJFtOiwIaUMop/40E+qLJNME= 2024-04-19T23:39:34.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PVviJFtOiwIaUMop/40E+qLJNME=', 'Content-Length': '0'} 2024-04-19T23:39:34.995 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086f11baee2dea1e5-00662300b6-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.995 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.996 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.996 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.996 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.996 DEBUG:boto:Method: HEAD 2024-04-19T23:39:34.996 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.996 DEBUG:boto:Data: 2024-04-19T23:39:34.996 DEBUG:boto:Headers: {} 2024-04-19T23:39:34.996 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:34.996 DEBUG:boto:Port: 8001 2024-04-19T23:39:34.997 DEBUG:boto:Params: {} 2024-04-19T23:39:34.997 DEBUG:boto:Token: None 2024-04-19T23:39:34.997 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:34 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.997 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:10bHGpO+UcoICtf+Uot2QEb6uew= 2024-04-19T23:39:34.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:10bHGpO+UcoICtf+Uot2QEb6uew=', 'Content-Length': '0'} 2024-04-19T23:39:34.999 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054494622dc138c2b-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:34.999 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:34.999 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:34.999 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.999 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:34.999 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.000 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:35.000 DEBUG:boto:Data: 2024-04-19T23:39:35.000 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.000 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.000 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.000 DEBUG:boto:Params: {} 2024-04-19T23:39:35.000 DEBUG:boto:Token: None 2024-04-19T23:39:35.000 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:35.000 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ic6DMfkDyP/kFTfXOGRoQ9NMBcw= 2024-04-19T23:39:35.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ic6DMfkDyP/kFTfXOGRoQ9NMBcw=', 'Content-Length': '0'} 2024-04-19T23:39:35.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a087ca1ee3b0ba2a-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.003 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.003 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.003 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:35.003 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:35.003 DEBUG:boto:Method: GET 2024-04-19T23:39:35.003 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:35.003 DEBUG:boto:Data: 2024-04-19T23:39:35.003 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.004 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.004 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.004 DEBUG:boto:Params: {} 2024-04-19T23:39:35.004 DEBUG:boto:Token: None 2024-04-19T23:39:35.004 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:35.004 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IuUkjPc6nZmjSMgX0vvyb/Te7PE= 2024-04-19T23:39:35.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IuUkjPc6nZmjSMgX0vvyb/Te7PE=', 'Content-Length': '0'} 2024-04-19T23:39:35.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc99513af8e6814a-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.046 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.047 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.047 DEBUG:boto:path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:35.047 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:35.047 DEBUG:boto:Method: GET 2024-04-19T23:39:35.047 DEBUG:boto:Path: /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:35.047 DEBUG:boto:Data: 2024-04-19T23:39:35.047 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.047 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.047 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.047 DEBUG:boto:Params: {} 2024-04-19T23:39:35.048 DEBUG:boto:Token: None 2024-04-19T23:39:35.048 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=E5taibPWiUs3cuEimynrjQD8cXTCAEL 2024-04-19T23:39:35.048 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IuUkjPc6nZmjSMgX0vvyb/Te7PE= 2024-04-19T23:39:35.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IuUkjPc6nZmjSMgX0vvyb/Te7PE=', 'Content-Length': '0'} 2024-04-19T23:39:35.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'E5taibPWiUs3cuEimynrjQD8cXTCAEL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b989c482a164a25f-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.094 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.095 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.095 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.095 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.095 DEBUG:boto:Method: GET 2024-04-19T23:39:35.095 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.095 DEBUG:boto:Data: 2024-04-19T23:39:35.095 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.095 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.096 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.096 DEBUG:boto:Params: {} 2024-04-19T23:39:35.096 DEBUG:boto:Token: None 2024-04-19T23:39:35.096 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.096 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017cbb19e9cc5cd5f-00662300b7-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.099 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.100 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.100 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.100 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.100 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.100 DEBUG:boto:Method: GET 2024-04-19T23:39:35.100 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.100 DEBUG:boto:Data: 2024-04-19T23:39:35.100 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.100 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.100 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.101 DEBUG:boto:Params: {} 2024-04-19T23:39:35.101 DEBUG:boto:Token: None 2024-04-19T23:39:35.101 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.101 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d559a3d58ac3078b-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.146 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.147 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.147 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.147 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.147 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:35.147 DEBUG:boto:Data: 2024-04-19T23:39:35.147 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.147 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.148 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.148 DEBUG:boto:Params: {} 2024-04-19T23:39:35.148 DEBUG:boto:Token: None 2024-04-19T23:39:35.148 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj 2024-04-19T23:39:35.148 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE= 2024-04-19T23:39:35.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE=', 'Content-Length': '0'} 2024-04-19T23:39:35.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089a27fcb70606c0b-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.152 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.152 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.152 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.152 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:35.152 DEBUG:boto:Data: 2024-04-19T23:39:35.152 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.152 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.152 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.153 DEBUG:boto:Params: {} 2024-04-19T23:39:35.153 DEBUG:boto:Token: None 2024-04-19T23:39:35.153 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj 2024-04-19T23:39:35.153 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE= 2024-04-19T23:39:35.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE=', 'Content-Length': '0'} 2024-04-19T23:39:35.156 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f7a580f946db481-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.157 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.157 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.157 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.157 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.157 DEBUG:boto:Method: GET 2024-04-19T23:39:35.157 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.157 DEBUG:boto:Data: 2024-04-19T23:39:35.157 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.157 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.157 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.157 DEBUG:boto:Params: {} 2024-04-19T23:39:35.158 DEBUG:boto:Token: None 2024-04-19T23:39:35.158 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.158 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU= 2024-04-19T23:39:35.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU=', 'Content-Length': '0'} 2024-04-19T23:39:35.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000339046233b9928f3-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.206 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.207 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.207 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.207 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.207 DEBUG:boto:Method: GET 2024-04-19T23:39:35.207 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.207 DEBUG:boto:Data: 2024-04-19T23:39:35.207 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.207 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.207 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.207 DEBUG:boto:Params: {} 2024-04-19T23:39:35.207 DEBUG:boto:Token: None 2024-04-19T23:39:35.208 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.208 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU= 2024-04-19T23:39:35.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU=', 'Content-Length': '0'} 2024-04-19T23:39:35.210 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000043640eafae74ae6-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.210 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.210 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.210 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.210 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.210 DEBUG:boto:Method: GET 2024-04-19T23:39:35.210 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.211 DEBUG:boto:Data: 2024-04-19T23:39:35.211 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.211 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.211 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.211 DEBUG:boto:Params: {} 2024-04-19T23:39:35.211 DEBUG:boto:Token: None 2024-04-19T23:39:35.211 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.211 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e095f072a565d119-00662300b7-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.254 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.255 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.255 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.255 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.255 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.255 DEBUG:boto:Method: GET 2024-04-19T23:39:35.255 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.255 DEBUG:boto:Data: 2024-04-19T23:39:35.256 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.256 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.256 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.256 DEBUG:boto:Params: {} 2024-04-19T23:39:35.256 DEBUG:boto:Token: None 2024-04-19T23:39:35.256 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.256 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbfa0f600c2a03e0-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.259 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.260 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.260 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.260 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.260 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.260 DEBUG:boto:Method: GET 2024-04-19T23:39:35.260 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.260 DEBUG:boto:Data: 2024-04-19T23:39:35.261 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.261 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.261 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.261 DEBUG:boto:Params: {} 2024-04-19T23:39:35.261 DEBUG:boto:Token: None 2024-04-19T23:39:35.261 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.261 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T+VZMvbPQ9/AOjE+t6hsewTlGLg= 2024-04-19T23:39:35.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T+VZMvbPQ9/AOjE+t6hsewTlGLg=', 'Content-Length': '0'} 2024-04-19T23:39:35.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d9f35cfe8905457-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.264 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.264 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.264 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.264 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.264 DEBUG:boto:Method: GET 2024-04-19T23:39:35.264 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.264 DEBUG:boto:Data: 2024-04-19T23:39:35.264 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.264 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.265 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.265 DEBUG:boto:Params: {} 2024-04-19T23:39:35.265 DEBUG:boto:Token: None 2024-04-19T23:39:35.265 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.265 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T+VZMvbPQ9/AOjE+t6hsewTlGLg= 2024-04-19T23:39:35.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T+VZMvbPQ9/AOjE+t6hsewTlGLg=', 'Content-Length': '0'} 2024-04-19T23:39:35.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc87b52bb0f3556a-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.267 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.267 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.267 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.267 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.267 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.267 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.268 DEBUG:boto:Data: 2024-04-19T23:39:35.268 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.268 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.268 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.268 DEBUG:boto:Params: {} 2024-04-19T23:39:35.268 DEBUG:boto:Token: None 2024-04-19T23:39:35.268 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.268 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ThRXf5XuojQZgacN7uUmMPTXRiY= 2024-04-19T23:39:35.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ThRXf5XuojQZgacN7uUmMPTXRiY=', 'Content-Length': '0'} 2024-04-19T23:39:35.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2345eaa0f97c33e-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.270 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.271 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.271 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.271 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.271 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.271 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.271 DEBUG:boto:Data: 2024-04-19T23:39:35.271 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.271 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.271 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.271 DEBUG:boto:Params: {} 2024-04-19T23:39:35.271 DEBUG:boto:Token: None 2024-04-19T23:39:35.272 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.272 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ThRXf5XuojQZgacN7uUmMPTXRiY= 2024-04-19T23:39:35.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ThRXf5XuojQZgacN7uUmMPTXRiY=', 'Content-Length': '0'} 2024-04-19T23:39:35.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009863b6581324254a-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.274 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.274 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.274 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.274 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.274 DEBUG:boto:Method: GET 2024-04-19T23:39:35.274 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.274 DEBUG:boto:Data: 2024-04-19T23:39:35.274 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.275 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.275 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.275 DEBUG:boto:Params: {} 2024-04-19T23:39:35.275 DEBUG:boto:Token: None 2024-04-19T23:39:35.275 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.275 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T+VZMvbPQ9/AOjE+t6hsewTlGLg= 2024-04-19T23:39:35.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T+VZMvbPQ9/AOjE+t6hsewTlGLg=', 'Content-Length': '0'} 2024-04-19T23:39:35.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004062a3891633a93d-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.322 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.322 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.322 DEBUG:boto:path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.322 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.322 DEBUG:boto:Method: GET 2024-04-19T23:39:35.323 DEBUG:boto:Path: /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.323 DEBUG:boto:Data: 2024-04-19T23:39:35.323 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.323 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.323 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.323 DEBUG:boto:Params: {} 2024-04-19T23:39:35.323 DEBUG:boto:Token: None 2024-04-19T23:39:35.323 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2 2024-04-19T23:39:35.323 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T+VZMvbPQ9/AOjE+t6hsewTlGLg= 2024-04-19T23:39:35.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T+VZMvbPQ9/AOjE+t6hsewTlGLg=', 'Content-Length': '0'} 2024-04-19T23:39:35.325 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'NjFtS4LHZJDkGsaKejPMQyqPrZjbuN2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059bafd8d1fce160a-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.326 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.326 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.326 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.326 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.326 DEBUG:boto:Method: GET 2024-04-19T23:39:35.326 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.327 DEBUG:boto:Data: 2024-04-19T23:39:35.327 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.327 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.327 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.327 DEBUG:boto:Params: {} 2024-04-19T23:39:35.327 DEBUG:boto:Token: None 2024-04-19T23:39:35.327 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.327 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec9a845f766ae6ef-00662300b7-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.330 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.330 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.331 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.331 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.331 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.331 DEBUG:boto:Method: GET 2024-04-19T23:39:35.331 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.331 DEBUG:boto:Data: 2024-04-19T23:39:35.331 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.331 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.331 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.331 DEBUG:boto:Params: {} 2024-04-19T23:39:35.331 DEBUG:boto:Token: None 2024-04-19T23:39:35.332 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d630ad9b52ec2da4-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.334 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.335 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.335 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.335 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.335 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:35.335 DEBUG:boto:Data: 2024-04-19T23:39:35.335 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.335 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.335 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.335 DEBUG:boto:Params: {} 2024-04-19T23:39:35.336 DEBUG:boto:Token: None 2024-04-19T23:39:35.336 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj 2024-04-19T23:39:35.336 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE= 2024-04-19T23:39:35.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE=', 'Content-Length': '0'} 2024-04-19T23:39:35.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f88b1b158715a5ca-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.339 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.339 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.340 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.340 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:35.340 DEBUG:boto:Data: 2024-04-19T23:39:35.340 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.340 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.340 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.340 DEBUG:boto:Params: {} 2024-04-19T23:39:35.340 DEBUG:boto:Token: None 2024-04-19T23:39:35.340 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj 2024-04-19T23:39:35.341 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE= 2024-04-19T23:39:35.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE=', 'Content-Length': '0'} 2024-04-19T23:39:35.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000265c800c6c4413c2-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.344 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.344 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.344 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.344 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.344 DEBUG:boto:Method: GET 2024-04-19T23:39:35.345 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.345 DEBUG:boto:Data: 2024-04-19T23:39:35.345 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.345 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.345 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.345 DEBUG:boto:Params: {} 2024-04-19T23:39:35.345 DEBUG:boto:Token: None 2024-04-19T23:39:35.345 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.345 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU= 2024-04-19T23:39:35.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU=', 'Content-Length': '0'} 2024-04-19T23:39:35.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008036f2672def15d4-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.390 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.390 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.391 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.391 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.391 DEBUG:boto:Method: GET 2024-04-19T23:39:35.391 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.391 DEBUG:boto:Data: 2024-04-19T23:39:35.391 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.391 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.391 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.391 DEBUG:boto:Params: {} 2024-04-19T23:39:35.391 DEBUG:boto:Token: None 2024-04-19T23:39:35.391 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.392 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU= 2024-04-19T23:39:35.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU=', 'Content-Length': '0'} 2024-04-19T23:39:35.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfa32065debe613e-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.394 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.394 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.394 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.394 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.394 DEBUG:boto:Method: GET 2024-04-19T23:39:35.394 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.395 DEBUG:boto:Data: 2024-04-19T23:39:35.395 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.395 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.395 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.395 DEBUG:boto:Params: {} 2024-04-19T23:39:35.395 DEBUG:boto:Token: None 2024-04-19T23:39:35.395 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.395 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fecbf776069b033f-00662300b7-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.438 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.439 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.439 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.439 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.440 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.440 DEBUG:boto:Method: GET 2024-04-19T23:39:35.440 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.440 DEBUG:boto:Data: 2024-04-19T23:39:35.440 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.440 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.440 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.440 DEBUG:boto:Params: {} 2024-04-19T23:39:35.440 DEBUG:boto:Token: None 2024-04-19T23:39:35.440 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.441 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000377fbcf9e8a14a7d-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.444 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.444 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.444 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.444 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.445 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.445 DEBUG:boto:Method: GET 2024-04-19T23:39:35.445 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.445 DEBUG:boto:Data: 2024-04-19T23:39:35.445 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.445 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.445 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.445 DEBUG:boto:Params: {} 2024-04-19T23:39:35.445 DEBUG:boto:Token: None 2024-04-19T23:39:35.445 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.446 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rvY4rqKWszrGADd09q4xIRBt02k= 2024-04-19T23:39:35.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rvY4rqKWszrGADd09q4xIRBt02k=', 'Content-Length': '0'} 2024-04-19T23:39:35.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000633a3dbefc70443c-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.448 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.448 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.448 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.448 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.449 DEBUG:boto:Method: GET 2024-04-19T23:39:35.449 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.449 DEBUG:boto:Data: 2024-04-19T23:39:35.449 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.449 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.449 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.449 DEBUG:boto:Params: {} 2024-04-19T23:39:35.450 DEBUG:boto:Token: None 2024-04-19T23:39:35.450 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.450 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rvY4rqKWszrGADd09q4xIRBt02k= 2024-04-19T23:39:35.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rvY4rqKWszrGADd09q4xIRBt02k=', 'Content-Length': '0'} 2024-04-19T23:39:35.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d9cff7b66207969-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.452 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.452 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.452 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.452 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.452 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.452 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.453 DEBUG:boto:Data: 2024-04-19T23:39:35.453 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.453 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.453 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.453 DEBUG:boto:Params: {} 2024-04-19T23:39:35.453 DEBUG:boto:Token: None 2024-04-19T23:39:35.453 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.453 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MBuLEC99rO5LmOOP8mrwxnvNpJ8= 2024-04-19T23:39:35.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MBuLEC99rO5LmOOP8mrwxnvNpJ8=', 'Content-Length': '0'} 2024-04-19T23:39:35.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002db11d47572ebf55-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.455 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.455 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.456 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.456 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.456 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.456 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.456 DEBUG:boto:Data: 2024-04-19T23:39:35.456 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.456 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.456 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.456 DEBUG:boto:Params: {} 2024-04-19T23:39:35.456 DEBUG:boto:Token: None 2024-04-19T23:39:35.456 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.457 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MBuLEC99rO5LmOOP8mrwxnvNpJ8= 2024-04-19T23:39:35.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MBuLEC99rO5LmOOP8mrwxnvNpJ8=', 'Content-Length': '0'} 2024-04-19T23:39:35.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bbdd4ba08baf975-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.459 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.459 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.459 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.460 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.460 DEBUG:boto:Method: GET 2024-04-19T23:39:35.460 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.460 DEBUG:boto:Data: 2024-04-19T23:39:35.460 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.460 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.460 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.460 DEBUG:boto:Params: {} 2024-04-19T23:39:35.460 DEBUG:boto:Token: None 2024-04-19T23:39:35.460 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.461 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rvY4rqKWszrGADd09q4xIRBt02k= 2024-04-19T23:39:35.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rvY4rqKWszrGADd09q4xIRBt02k=', 'Content-Length': '0'} 2024-04-19T23:39:35.463 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031bcfb2d952999f7-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.510 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.510 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.511 DEBUG:boto:path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.511 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.511 DEBUG:boto:Method: GET 2024-04-19T23:39:35.511 DEBUG:boto:Path: /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.511 DEBUG:boto:Data: 2024-04-19T23:39:35.511 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.511 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.511 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.511 DEBUG:boto:Params: {} 2024-04-19T23:39:35.511 DEBUG:boto:Token: None 2024-04-19T23:39:35.511 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=FmTjHqLZC66BQsk7jiah7FJSChQvGdx 2024-04-19T23:39:35.512 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rvY4rqKWszrGADd09q4xIRBt02k= 2024-04-19T23:39:35.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rvY4rqKWszrGADd09q4xIRBt02k=', 'Content-Length': '0'} 2024-04-19T23:39:35.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'FmTjHqLZC66BQsk7jiah7FJSChQvGdx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001fe11b7ce8234b7b-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.558 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.559 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.559 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.559 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.559 DEBUG:boto:Method: GET 2024-04-19T23:39:35.559 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.559 DEBUG:boto:Data: 2024-04-19T23:39:35.559 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.560 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.560 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.560 DEBUG:boto:Params: {} 2024-04-19T23:39:35.560 DEBUG:boto:Token: None 2024-04-19T23:39:35.560 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bed15b594a46054a-00662300b7-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.606 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.607 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.607 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.607 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.607 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.607 DEBUG:boto:Method: GET 2024-04-19T23:39:35.607 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.607 DEBUG:boto:Data: 2024-04-19T23:39:35.607 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.607 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.608 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.608 DEBUG:boto:Params: {} 2024-04-19T23:39:35.608 DEBUG:boto:Token: None 2024-04-19T23:39:35.608 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.608 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026675e9b7f4b9623-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.658 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.659 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.659 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.659 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.659 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:35.659 DEBUG:boto:Data: 2024-04-19T23:39:35.659 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.659 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.659 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.660 DEBUG:boto:Params: {} 2024-04-19T23:39:35.660 DEBUG:boto:Token: None 2024-04-19T23:39:35.660 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj 2024-04-19T23:39:35.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE= 2024-04-19T23:39:35.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE=', 'Content-Length': '0'} 2024-04-19T23:39:35.664 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b208db6dfede8f6-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.664 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.664 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.664 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.664 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:35.665 DEBUG:boto:Data: 2024-04-19T23:39:35.665 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.665 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.665 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.665 DEBUG:boto:Params: {} 2024-04-19T23:39:35.665 DEBUG:boto:Token: None 2024-04-19T23:39:35.665 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj 2024-04-19T23:39:35.666 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE= 2024-04-19T23:39:35.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE=', 'Content-Length': '0'} 2024-04-19T23:39:35.669 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043b2fa5da7db8390-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.670 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.670 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.670 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.670 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.671 DEBUG:boto:Method: GET 2024-04-19T23:39:35.671 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.671 DEBUG:boto:Data: 2024-04-19T23:39:35.671 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.671 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.671 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.671 DEBUG:boto:Params: {} 2024-04-19T23:39:35.672 DEBUG:boto:Token: None 2024-04-19T23:39:35.672 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.672 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU= 2024-04-19T23:39:35.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU=', 'Content-Length': '0'} 2024-04-19T23:39:35.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000460dd58354430f51-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.718 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.719 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.719 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.719 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.719 DEBUG:boto:Method: GET 2024-04-19T23:39:35.719 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.719 DEBUG:boto:Data: 2024-04-19T23:39:35.719 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.719 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.719 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.719 DEBUG:boto:Params: {} 2024-04-19T23:39:35.719 DEBUG:boto:Token: None 2024-04-19T23:39:35.720 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.720 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU= 2024-04-19T23:39:35.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU=', 'Content-Length': '0'} 2024-04-19T23:39:35.722 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0b4e9f008042168-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.722 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.723 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.723 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.723 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.723 DEBUG:boto:Method: GET 2024-04-19T23:39:35.723 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.723 DEBUG:boto:Data: 2024-04-19T23:39:35.723 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.723 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.723 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.723 DEBUG:boto:Params: {} 2024-04-19T23:39:35.723 DEBUG:boto:Token: None 2024-04-19T23:39:35.724 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.724 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f4f0b30df94a71c-00662300b7-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.770 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.771 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.771 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.771 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.771 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.771 DEBUG:boto:Method: GET 2024-04-19T23:39:35.772 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.772 DEBUG:boto:Data: 2024-04-19T23:39:35.772 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.772 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.772 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.772 DEBUG:boto:Params: {} 2024-04-19T23:39:35.772 DEBUG:boto:Token: None 2024-04-19T23:39:35.773 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.773 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8890dac19a055e7-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.776 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.776 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.776 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.777 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.777 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.777 DEBUG:boto:Method: GET 2024-04-19T23:39:35.777 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.777 DEBUG:boto:Data: 2024-04-19T23:39:35.777 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.777 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.777 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.777 DEBUG:boto:Params: {} 2024-04-19T23:39:35.777 DEBUG:boto:Token: None 2024-04-19T23:39:35.777 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.778 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FSvDpsS7pYzsUFtNNo876679ENQ= 2024-04-19T23:39:35.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FSvDpsS7pYzsUFtNNo876679ENQ=', 'Content-Length': '0'} 2024-04-19T23:39:35.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e1388f430bb14ad-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.780 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.780 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.780 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.780 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.780 DEBUG:boto:Method: GET 2024-04-19T23:39:35.780 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.781 DEBUG:boto:Data: 2024-04-19T23:39:35.781 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.781 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.781 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.781 DEBUG:boto:Params: {} 2024-04-19T23:39:35.781 DEBUG:boto:Token: None 2024-04-19T23:39:35.781 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.781 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FSvDpsS7pYzsUFtNNo876679ENQ= 2024-04-19T23:39:35.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FSvDpsS7pYzsUFtNNo876679ENQ=', 'Content-Length': '0'} 2024-04-19T23:39:35.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a24e2b10eaa497f0-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.784 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.784 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.784 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.784 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.784 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.784 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.784 DEBUG:boto:Data: 2024-04-19T23:39:35.784 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.784 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.785 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.785 DEBUG:boto:Params: {} 2024-04-19T23:39:35.785 DEBUG:boto:Token: None 2024-04-19T23:39:35.785 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.785 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:F9NeGMaycdJWzM+5I1Wgo3XUkvQ= 2024-04-19T23:39:35.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:F9NeGMaycdJWzM+5I1Wgo3XUkvQ=', 'Content-Length': '0'} 2024-04-19T23:39:35.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040360857b5e47840-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.787 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.787 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.787 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.787 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.788 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.788 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.788 DEBUG:boto:Data: 2024-04-19T23:39:35.788 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.788 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.788 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.788 DEBUG:boto:Params: {} 2024-04-19T23:39:35.788 DEBUG:boto:Token: None 2024-04-19T23:39:35.788 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.788 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:F9NeGMaycdJWzM+5I1Wgo3XUkvQ= 2024-04-19T23:39:35.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:F9NeGMaycdJWzM+5I1Wgo3XUkvQ=', 'Content-Length': '0'} 2024-04-19T23:39:35.790 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096f3cbe463d143ee-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.791 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.791 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.791 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.791 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.791 DEBUG:boto:Method: GET 2024-04-19T23:39:35.791 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.791 DEBUG:boto:Data: 2024-04-19T23:39:35.791 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.791 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.791 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.792 DEBUG:boto:Params: {} 2024-04-19T23:39:35.792 DEBUG:boto:Token: None 2024-04-19T23:39:35.792 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.792 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FSvDpsS7pYzsUFtNNo876679ENQ= 2024-04-19T23:39:35.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FSvDpsS7pYzsUFtNNo876679ENQ=', 'Content-Length': '0'} 2024-04-19T23:39:35.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090959c325a2f5df2-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.794 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.794 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.794 DEBUG:boto:path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.794 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.795 DEBUG:boto:Method: GET 2024-04-19T23:39:35.795 DEBUG:boto:Path: /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.795 DEBUG:boto:Data: 2024-04-19T23:39:35.795 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.795 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.795 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.795 DEBUG:boto:Params: {} 2024-04-19T23:39:35.795 DEBUG:boto:Token: None 2024-04-19T23:39:35.795 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R 2024-04-19T23:39:35.795 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FSvDpsS7pYzsUFtNNo876679ENQ= 2024-04-19T23:39:35.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FSvDpsS7pYzsUFtNNo876679ENQ=', 'Content-Length': '0'} 2024-04-19T23:39:35.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'lKfLYYBA1E.lSLR49AMlK-Kgik5ij5R'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe7360f117f97172-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.838 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.839 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.839 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.839 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.839 DEBUG:boto:Method: GET 2024-04-19T23:39:35.839 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.839 DEBUG:boto:Data: 2024-04-19T23:39:35.839 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.840 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.840 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.840 DEBUG:boto:Params: {} 2024-04-19T23:39:35.840 DEBUG:boto:Token: None 2024-04-19T23:39:35.840 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.841 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003d5acc1732b3a04-00662300b7-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.844 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.845 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.845 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.845 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.845 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.845 DEBUG:boto:Method: GET 2024-04-19T23:39:35.845 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.845 DEBUG:boto:Data: 2024-04-19T23:39:35.846 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.846 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.846 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.846 DEBUG:boto:Params: {} 2024-04-19T23:39:35.846 DEBUG:boto:Token: None 2024-04-19T23:39:35.847 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.847 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd87b94bbd104391-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.850 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.850 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.851 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.851 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.851 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:35.851 DEBUG:boto:Data: 2024-04-19T23:39:35.851 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.851 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.851 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.851 DEBUG:boto:Params: {} 2024-04-19T23:39:35.851 DEBUG:boto:Token: None 2024-04-19T23:39:35.852 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj 2024-04-19T23:39:35.852 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE= 2024-04-19T23:39:35.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE=', 'Content-Length': '0'} 2024-04-19T23:39:35.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9fe3ad17b9994bd-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.855 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.855 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.856 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.856 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:35.856 DEBUG:boto:Data: 2024-04-19T23:39:35.856 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.856 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.856 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.857 DEBUG:boto:Params: {} 2024-04-19T23:39:35.857 DEBUG:boto:Token: None 2024-04-19T23:39:35.857 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj 2024-04-19T23:39:35.857 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE= 2024-04-19T23:39:35.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+yWqlEMrkxEif9BQmnkWFDMCbuE=', 'Content-Length': '0'} 2024-04-19T23:39:35.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ff846a989ef36a6-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.861 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.861 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.861 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.861 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.861 DEBUG:boto:Method: GET 2024-04-19T23:39:35.861 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.861 DEBUG:boto:Data: 2024-04-19T23:39:35.861 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.861 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.861 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.861 DEBUG:boto:Params: {} 2024-04-19T23:39:35.862 DEBUG:boto:Token: None 2024-04-19T23:39:35.862 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.862 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU= 2024-04-19T23:39:35.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU=', 'Content-Length': '0'} 2024-04-19T23:39:35.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b24233cd67ac41f-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.906 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.907 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.907 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.907 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.907 DEBUG:boto:Method: GET 2024-04-19T23:39:35.907 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.907 DEBUG:boto:Data: 2024-04-19T23:39:35.907 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.907 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.907 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.907 DEBUG:boto:Params: {} 2024-04-19T23:39:35.908 DEBUG:boto:Token: None 2024-04-19T23:39:35.908 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:35.908 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU= 2024-04-19T23:39:35.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3RYNy9Cn8FmaHnMIyRpHkZzV2WU=', 'Content-Length': '0'} 2024-04-19T23:39:35.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000775409937f3301c7-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.910 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.910 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.911 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.911 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.911 DEBUG:boto:Method: GET 2024-04-19T23:39:35.911 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.911 DEBUG:boto:Data: 2024-04-19T23:39:35.911 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.911 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.911 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.912 DEBUG:boto:Params: {} 2024-04-19T23:39:35.912 DEBUG:boto:Token: None 2024-04-19T23:39:35.912 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.912 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012fb0663f0d55a06-00662300b7-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.915 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.915 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.916 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.916 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.916 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.916 DEBUG:boto:Method: GET 2024-04-19T23:39:35.916 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.916 DEBUG:boto:Data: 2024-04-19T23:39:35.916 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.916 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.916 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.916 DEBUG:boto:Params: {} 2024-04-19T23:39:35.916 DEBUG:boto:Token: None 2024-04-19T23:39:35.917 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.917 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000719cb1b4063c36cf-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.920 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.920 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.920 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.921 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.921 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.921 DEBUG:boto:Method: GET 2024-04-19T23:39:35.921 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.921 DEBUG:boto:Data: 2024-04-19T23:39:35.921 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.921 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.921 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.921 DEBUG:boto:Params: {} 2024-04-19T23:39:35.921 DEBUG:boto:Token: None 2024-04-19T23:39:35.922 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.922 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZJcdrEQAMhNEoU/eAjagaTFiJIs= 2024-04-19T23:39:35.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZJcdrEQAMhNEoU/eAjagaTFiJIs=', 'Content-Length': '0'} 2024-04-19T23:39:35.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056a1979bbb4c068d-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.924 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.925 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.925 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.925 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.925 DEBUG:boto:Method: GET 2024-04-19T23:39:35.925 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.925 DEBUG:boto:Data: 2024-04-19T23:39:35.925 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.925 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.925 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.925 DEBUG:boto:Params: {} 2024-04-19T23:39:35.925 DEBUG:boto:Token: None 2024-04-19T23:39:35.926 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.926 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZJcdrEQAMhNEoU/eAjagaTFiJIs= 2024-04-19T23:39:35.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZJcdrEQAMhNEoU/eAjagaTFiJIs=', 'Content-Length': '0'} 2024-04-19T23:39:35.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d0464e5c2e74c4f-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.928 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.928 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.928 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.929 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.929 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.929 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.929 DEBUG:boto:Data: 2024-04-19T23:39:35.929 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.929 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.929 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.929 DEBUG:boto:Params: {} 2024-04-19T23:39:35.929 DEBUG:boto:Token: None 2024-04-19T23:39:35.929 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.930 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9UuKIWtsv7LVMvYwj6zz1z3a82I= 2024-04-19T23:39:35.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9UuKIWtsv7LVMvYwj6zz1z3a82I=', 'Content-Length': '0'} 2024-04-19T23:39:35.934 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000102684ec6127c414-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.935 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.935 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.935 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.935 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.935 DEBUG:boto:Method: HEAD 2024-04-19T23:39:35.935 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.935 DEBUG:boto:Data: 2024-04-19T23:39:35.935 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.935 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.935 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.936 DEBUG:boto:Params: {} 2024-04-19T23:39:35.936 DEBUG:boto:Token: None 2024-04-19T23:39:35.936 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.936 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9UuKIWtsv7LVMvYwj6zz1z3a82I= 2024-04-19T23:39:35.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9UuKIWtsv7LVMvYwj6zz1z3a82I=', 'Content-Length': '0'} 2024-04-19T23:39:35.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002582561f09bf6bbb-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:35.938 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.938 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.938 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.938 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.939 DEBUG:boto:Method: GET 2024-04-19T23:39:35.939 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.939 DEBUG:boto:Data: 2024-04-19T23:39:35.939 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.939 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.939 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.939 DEBUG:boto:Params: {} 2024-04-19T23:39:35.939 DEBUG:boto:Token: None 2024-04-19T23:39:35.939 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.939 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZJcdrEQAMhNEoU/eAjagaTFiJIs= 2024-04-19T23:39:35.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZJcdrEQAMhNEoU/eAjagaTFiJIs=', 'Content-Length': '0'} 2024-04-19T23:39:35.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7d1985f6392ba70-00662300b7-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.982 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:35.983 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:35.983 DEBUG:boto:path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.983 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.983 DEBUG:boto:Method: GET 2024-04-19T23:39:35.983 DEBUG:boto:Path: /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.983 DEBUG:boto:Data: 2024-04-19T23:39:35.984 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.984 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.984 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.984 DEBUG:boto:Params: {} 2024-04-19T23:39:35.984 DEBUG:boto:Token: None 2024-04-19T23:39:35.984 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/obj?versionId=zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma 2024-04-19T23:39:35.985 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZJcdrEQAMhNEoU/eAjagaTFiJIs= 2024-04-19T23:39:35.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZJcdrEQAMhNEoU/eAjagaTFiJIs=', 'Content-Length': '0'} 2024-04-19T23:39:35.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:19 GMT'), ('x-amz-version-id', 'zwsSl6whrs-j2uo2xvsRdnhCfJSQ4Ma'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da3bac861e8c22b8-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.987 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.987 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.987 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.987 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.987 DEBUG:boto:Method: GET 2024-04-19T23:39:35.987 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.987 DEBUG:boto:Data: 2024-04-19T23:39:35.988 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.988 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:35.988 DEBUG:boto:Port: 8001 2024-04-19T23:39:35.988 DEBUG:boto:Params: {} 2024-04-19T23:39:35.988 DEBUG:boto:Token: None 2024-04-19T23:39:35.988 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.988 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c5cb9dfc5ecd07f-00662300b7-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:35.991 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:35.991 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:35.992 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:35.992 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.992 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:35.992 DEBUG:boto:Method: GET 2024-04-19T23:39:35.992 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:35.992 DEBUG:boto:Data: 2024-04-19T23:39:35.992 DEBUG:boto:Headers: {} 2024-04-19T23:39:35.992 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:35.992 DEBUG:boto:Port: 8000 2024-04-19T23:39:35.993 DEBUG:boto:Params: {} 2024-04-19T23:39:35.993 DEBUG:boto:Token: None 2024-04-19T23:39:35.993 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:35 GMT /aktlji-22/ 2024-04-19T23:39:35.993 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4= 2024-04-19T23:39:35.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:35 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:brGIxSWtNi4lNPsYaRcP9/HmnV4=', 'Content-Length': '0'} 2024-04-19T23:39:35.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037345b3c3fd310e9-00662300b7-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:36.038 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:36.039 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:36.039 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:36.039 DEBUG:boto:Method: HEAD 2024-04-19T23:39:36.039 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:36.039 DEBUG:boto:Data: 2024-04-19T23:39:36.040 DEBUG:boto:Headers: {} 2024-04-19T23:39:36.040 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:36.040 DEBUG:boto:Port: 8001 2024-04-19T23:39:36.040 DEBUG:boto:Params: {} 2024-04-19T23:39:36.040 DEBUG:boto:Token: None 2024-04-19T23:39:36.040 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:36 GMT /aktlji-22/obj 2024-04-19T23:39:36.040 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T0I+Tm9jBNk1xORpMM80jfA480w= 2024-04-19T23:39:36.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T0I+Tm9jBNk1xORpMM80jfA480w=', 'Content-Length': '0'} 2024-04-19T23:39:36.044 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9eb708a38a8336e-00662300b8-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:36.045 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:36.045 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:36.045 DEBUG:boto:Method: HEAD 2024-04-19T23:39:36.045 DEBUG:boto:Path: /aktlji-22/obj 2024-04-19T23:39:36.045 DEBUG:boto:Data: 2024-04-19T23:39:36.045 DEBUG:boto:Headers: {} 2024-04-19T23:39:36.045 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:36.046 DEBUG:boto:Port: 8000 2024-04-19T23:39:36.046 DEBUG:boto:Params: {} 2024-04-19T23:39:36.046 DEBUG:boto:Token: None 2024-04-19T23:39:36.046 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:36 GMT /aktlji-22/obj 2024-04-19T23:39:36.046 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T0I+Tm9jBNk1xORpMM80jfA480w= 2024-04-19T23:39:36.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T0I+Tm9jBNk1xORpMM80jfA480w=', 'Content-Length': '0'} 2024-04-19T23:39:36.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa3bad8cbf11cbea-00662300b8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:36.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:39:36.050 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:36.050 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:36.051 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:36.051 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:36.051 DEBUG:boto:Method: GET 2024-04-19T23:39:36.051 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:36.051 DEBUG:boto:Data: 2024-04-19T23:39:36.051 DEBUG:boto:Headers: {} 2024-04-19T23:39:36.051 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:36.051 DEBUG:boto:Port: 8001 2024-04-19T23:39:36.051 DEBUG:boto:Params: {} 2024-04-19T23:39:36.052 DEBUG:boto:Token: None 2024-04-19T23:39:36.052 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:36 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:36.052 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uq0nT64zu3K8+Mf/o6uuNK0/Nz8= 2024-04-19T23:39:36.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uq0nT64zu3K8+Mf/o6uuNK0/Nz8=', 'Content-Length': '0'} 2024-04-19T23:39:36.054 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ca64b51c09dcf46-00662300b8-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:36.094 DEBUG:boto:path=/aktlji-22/obj 2024-04-19T23:39:36.095 DEBUG:boto:auth_path=/aktlji-22/obj 2024-04-19T23:39:36.095 DEBUG:boto:path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:36.095 DEBUG:boto:auth_path=/aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:36.095 DEBUG:boto:Method: GET 2024-04-19T23:39:36.095 DEBUG:boto:Path: /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:36.095 DEBUG:boto:Data: 2024-04-19T23:39:36.095 DEBUG:boto:Headers: {} 2024-04-19T23:39:36.095 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:36.095 DEBUG:boto:Port: 8000 2024-04-19T23:39:36.095 DEBUG:boto:Params: {} 2024-04-19T23:39:36.096 DEBUG:boto:Token: None 2024-04-19T23:39:36.096 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:36 GMT /aktlji-22/obj?versionId=-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo 2024-04-19T23:39:36.096 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uq0nT64zu3K8+Mf/o6uuNK0/Nz8= 2024-04-19T23:39:36.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uq0nT64zu3K8+Mf/o6uuNK0/Nz8=', 'Content-Length': '0'} 2024-04-19T23:39:36.098 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:36:20 GMT'), ('x-amz-version-id', '-GSvr6Zc5ZpFy2joTIpRCMXX60OdhOo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001188a65e7ea2c673-00662300b8-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:39:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:36.098 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:36.099 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:36.099 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:36.099 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:36.099 DEBUG:boto:Method: GET 2024-04-19T23:39:36.099 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:36.099 DEBUG:boto:Data: 2024-04-19T23:39:36.099 DEBUG:boto:Headers: {} 2024-04-19T23:39:36.099 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:39:36.099 DEBUG:boto:Port: 8001 2024-04-19T23:39:36.099 DEBUG:boto:Params: {} 2024-04-19T23:39:36.100 DEBUG:boto:Token: None 2024-04-19T23:39:36.100 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:36 GMT /aktlji-22/ 2024-04-19T23:39:36.100 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MZIjd9/xbeTQZxzHmahQQZA3c9g= 2024-04-19T23:39:36.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MZIjd9/xbeTQZxzHmahQQZA3c9g=', 'Content-Length': '0'} 2024-04-19T23:39:36.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000541122fc91dcaf50-00662300b8-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:36.103 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:36.103 DEBUG:boto:path=/aktlji-22/ 2024-04-19T23:39:36.103 DEBUG:boto:auth_path=/aktlji-22/ 2024-04-19T23:39:36.103 DEBUG:boto:path=/aktlji-22/?prefix=obj 2024-04-19T23:39:36.104 DEBUG:boto:auth_path=/aktlji-22/?prefix=obj 2024-04-19T23:39:36.104 DEBUG:boto:Method: GET 2024-04-19T23:39:36.104 DEBUG:boto:Path: /aktlji-22/?prefix=obj 2024-04-19T23:39:36.104 DEBUG:boto:Data: 2024-04-19T23:39:36.104 DEBUG:boto:Headers: {} 2024-04-19T23:39:36.104 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:39:36.104 DEBUG:boto:Port: 8000 2024-04-19T23:39:36.104 DEBUG:boto:Params: {} 2024-04-19T23:39:36.104 DEBUG:boto:Token: None 2024-04-19T23:39:36.104 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:39:36 GMT /aktlji-22/ 2024-04-19T23:39:36.105 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MZIjd9/xbeTQZxzHmahQQZA3c9g= 2024-04-19T23:39:36.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:36 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MZIjd9/xbeTQZxzHmahQQZA3c9g=', 'Content-Length': '0'} 2024-04-19T23:39:36.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a50a269fe6ce5ef-00662300b8-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:39:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:36.108 DEBUG:boto:b'aktlji-22obj1000falseobj2024-04-19T23:36:20.324Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:39:36.108 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-22 zones={test-zone3, test-zone2} 2024-04-19T23:39:36.108 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-04-19T23:39:36.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.109 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:36.168 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:36.168 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:36.226 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:39:36.227 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.227 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-19T23:39:36.227 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:33:20.691358Z" 2024-04-19T23:39:36.227 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.227 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.227 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-19T23:39:36.228 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:38:28.712334Z" 2024-04-19T23:39:36.228 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.228 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.228 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-19T23:39:36.228 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:36:38.704801Z" 2024-04-19T23:39:36.229 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.229 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.229 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-19T23:39:36.229 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:26:44.680751Z" 2024-04-19T23:39:36.229 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:36.230 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:39:36.231 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-19T23:39:36.231 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-19T23:39:36.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.232 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:36.293 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:36.294 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:36.347 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:39:36.347 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:39:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:39:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:39:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:39:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-19T23:39:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:36.348 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:39:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:39:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:39:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-19T23:39:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-19T23:39:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:39:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:39:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:39:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-19T23:39:36.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:39:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:39:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-19T23:39:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-19T23:39:36.350 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:39:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:39:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:39:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:38:28.712334Z" 2024-04-19T23:39:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:39:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:39:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-19T23:39:36.351 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-19T23:39:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:39:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:39:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:39:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-19T23:39:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:39:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:39:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-19T23:39:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-19T23:39:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:39:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:39:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:39:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-19T23:39:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:36.353 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:39:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:39:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-19T23:39:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-19T23:39:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:36.354 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:39:36Z" 2024-04-19T23:39:36.354 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:39:36.355 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:38:28.712334Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-19T23:39:36Z" } 2024-04-19T23:39:36.355 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:38:28.712334Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-19T23:39:36.356 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-19T23:39:36.356 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-19T23:39:36.356 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-19T23:39:36.356 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone3 2024-04-19T23:39:36.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.356 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:36.418 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:36.419 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:36.476 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:39:36.476 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.476 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-19T23:39:36.476 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:33:20.691358Z" 2024-04-19T23:39:36.476 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.477 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-19T23:39:36.477 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:38:28.712334Z" 2024-04-19T23:39:36.477 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.477 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-19T23:39:36.477 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:36:38.704801Z" 2024-04-19T23:39:36.477 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.477 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-19T23:39:36.478 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:26:44.680751Z" 2024-04-19T23:39:36.478 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:36.478 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:39:36.479 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-19T23:39:36.479 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-04-19T23:39:36.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.479 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:36.537 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:36.537 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:36.633 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:39:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:39:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:39:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:39:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:39:36.633 INFO:teuthology.orchestra.run.smithi019.stdout: "instance_id": 13223717182661897465 2024-04-19T23:39:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:39:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:39:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-19T23:39:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:36.634 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-19T23:39:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:39:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-19T23:39:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:38:28.712334Z" 2024-04-19T23:39:36.635 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:39:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-19T23:39:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-19T23:39:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:36.636 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:39:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-19T23:39:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:36.637 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-19T23:39:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:39:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:39:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 0, 2024-04-19T23:39:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 0 2024-04-19T23:39:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.638 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:39:36Z" 2024-04-19T23:39:36.638 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:39:36.639 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 13223717182661897465 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:38:28.712334Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-19T23:39:36Z" } 2024-04-19T23:39:36.639 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:38:28.712334Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-19T23:39:36.640 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-19T23:39:36.640 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000004319', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-19T23:39:36.640 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-04-19T23:39:36.640 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone1 2024-04-19T23:39:36.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.640 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:36.760 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-04-19T23:39:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-19T23:39:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-19T23:34:01.816741Z" 2024-04-19T23:39:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-04-19T23:39:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-19T23:37:44.628385Z" 2024-04-19T23:39:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007421", 2024-04-19T23:39:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-19T23:36:16.622856Z" 2024-04-19T23:39:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003055", 2024-04-19T23:39:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-19T23:26:44.608102Z" 2024-04-19T23:39:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:36.761 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-04-19T23:39:36.762 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000004037', 2: '00000000000000000000:00000000000000007421', 3: '00000000000000000000:00000000000000003055'} 2024-04-19T23:39:36.762 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-19T23:39:36.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.762 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:36.824 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:36.825 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:36.877 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:39:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:39:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:39:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:39:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:39:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "instance_id": 11239922052735367740 2024-04-19T23:39:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.877 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:39:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:39:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-19T23:39:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:34:01.816741Z" 2024-04-19T23:39:36.878 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:39:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-04-19T23:39:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:36.879 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:37:44.628385Z" 2024-04-19T23:39:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:39:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000007421", 2024-04-19T23:39:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:36:16.622856Z" 2024-04-19T23:39:36.880 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:39:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000003055", 2024-04-19T23:39:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:36.881 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:26:44.608102Z" 2024-04-19T23:39:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:39:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:39:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 0, 2024-04-19T23:39:36.882 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 0 2024-04-19T23:39:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:36.883 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:39:36Z" 2024-04-19T23:39:36.883 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:39:36.884 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11239922052735367740 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:34:01.816741Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004037", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:37:44.628385Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007421", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:16.622856Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003055", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.608102Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-19T23:39:36Z" } 2024-04-19T23:39:36.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:34:01.816741Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004037', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:37:44.628385Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007421', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:16.622856Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003055', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.608102Z'}}] 2024-04-19T23:39:36.884 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000004037', 2: '00000000000000000000:00000000000000007421', 3: '00000000000000000000:00000000000000003055'} 2024-04-19T23:39:36.884 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000004037', 2: '00000000000000000000:00000000000000007421', 3: '00000000000000000000:00000000000000003055'} 2024-04-19T23:39:36.884 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-19T23:39:36.884 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone2 target=test-zone3 2024-04-19T23:39:36.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:36.885 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:36.943 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:36.943 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:37.000 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-04-19T23:39:37.001 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:37.001 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-19T23:39:37.001 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-19T23:34:01.816741Z" 2024-04-19T23:39:37.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.001 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:37.002 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-04-19T23:39:37.002 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-19T23:37:44.628385Z" 2024-04-19T23:39:37.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.002 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:37.002 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007421", 2024-04-19T23:39:37.003 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-19T23:36:16.622856Z" 2024-04-19T23:39:37.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:37.003 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003055", 2024-04-19T23:39:37.003 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-19T23:26:44.608102Z" 2024-04-19T23:39:37.004 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:37.004 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-04-19T23:39:37.005 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000004037', 2: '00000000000000000000:00000000000000007421', 3: '00000000000000000000:00000000000000003055'} 2024-04-19T23:39:37.005 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-04-19T23:39:37.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.007 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:37.069 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:37.069 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:37.120 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:39:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:39:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:39:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:39:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:39:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "instance_id": 10713417137535636812 2024-04-19T23:39:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:39:37.120 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:39:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-04-19T23:39:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:34:01.816741Z" 2024-04-19T23:39:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.121 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:39:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-04-19T23:39:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:37:44.628385Z" 2024-04-19T23:39:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.122 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:39:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000007421", 2024-04-19T23:39:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:36:16.622856Z" 2024-04-19T23:39:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:39:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000003055", 2024-04-19T23:39:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:26:44.608102Z" 2024-04-19T23:39:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:39:37.124 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:39:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 0, 2024-04-19T23:39:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 0 2024-04-19T23:39:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.125 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:39:37Z" 2024-04-19T23:39:37.125 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:39:37.126 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10713417137535636812 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004319", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:34:01.816741Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004037", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:37:44.628385Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007421", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:16.622856Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003055", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.608102Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-19T23:39:37Z" } 2024-04-19T23:39:37.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004319', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:34:01.816741Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004037', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:37:44.628385Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007421', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:16.622856Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003055', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.608102Z'}}] 2024-04-19T23:39:37.126 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000004037', 2: '00000000000000000000:00000000000000007421', 3: '00000000000000000000:00000000000000003055'} 2024-04-19T23:39:37.126 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004319', 1: '00000000000000000000:00000000000000004037', 2: '00000000000000000000:00000000000000007421', 3: '00000000000000000000:00000000000000003055'} 2024-04-19T23:39:37.126 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-04-19T23:39:37.126 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone1 2024-04-19T23:39:37.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.127 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:37.181 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:37.181 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:37.236 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:39:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-04-19T23:39:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:33:39.315927Z" 2024-04-19T23:39:37.236 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004742", 2024-04-19T23:39:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:38:56.756407Z" 2024-04-19T23:39:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000007703", 2024-04-19T23:39:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:36:22.746727Z" 2024-04-19T23:39:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000003056", 2024-04-19T23:39:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:26:50.725314Z" 2024-04-19T23:39:37.237 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.238 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:39:37.239 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000004460', 1: '00000000000000000000:00000000000000004742', 2: '00000000000000000000:00000000000000007703', 3: '00000000000000000000:00000000000000003056'} 2024-04-19T23:39:37.239 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-04-19T23:39:37.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.240 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:37.296 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:37.297 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:37.346 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:39:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:39:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:39:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:39:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:39:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "instance_id": 4378739362102240710 2024-04-19T23:39:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:39:37.347 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:39:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-04-19T23:39:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:33:39.315927Z" 2024-04-19T23:39:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.348 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:39:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004742", 2024-04-19T23:39:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:38:56.756407Z" 2024-04-19T23:39:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.349 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:39:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000007703", 2024-04-19T23:39:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:36:22.746727Z" 2024-04-19T23:39:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:39:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-19T23:39:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000003056", 2024-04-19T23:39:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:26:50.725314Z" 2024-04-19T23:39:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:39:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:39:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 0, 2024-04-19T23:39:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 0 2024-04-19T23:39:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.352 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:39:37Z" 2024-04-19T23:39:37.352 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:39:37.353 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 4378739362102240710 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004460", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:39.315927Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004742", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:38:56.756407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007703", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:22.746727Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003056", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:50.725314Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-19T23:39:37Z" } 2024-04-19T23:39:37.354 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004460', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:39.315927Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004742', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:38:56.756407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007703', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:22.746727Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003056', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:50.725314Z'}}] 2024-04-19T23:39:37.354 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004460', 1: '00000000000000000000:00000000000000004742', 2: '00000000000000000000:00000000000000007703', 3: '00000000000000000000:00000000000000003056'} 2024-04-19T23:39:37.354 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004460', 1: '00000000000000000000:00000000000000004742', 2: '00000000000000000000:00000000000000007703', 3: '00000000000000000000:00000000000000003056'} 2024-04-19T23:39:37.354 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-04-19T23:39:37.354 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone3 target=test-zone2 2024-04-19T23:39:37.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.355 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:37.410 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:37.410 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:37.468 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:39:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-04-19T23:39:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:33:39.315927Z" 2024-04-19T23:39:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004742", 2024-04-19T23:39:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:38:56.756407Z" 2024-04-19T23:39:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.468 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000007703", 2024-04-19T23:39:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:36:22.746727Z" 2024-04-19T23:39:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000003056", 2024-04-19T23:39:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:26:50.725314Z" 2024-04-19T23:39:37.469 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.469 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:39:37.470 DEBUG:rgw_multi.tests:data markers for zone=test-zone3 markers={0: '00000000000000000000:00000000000000004460', 1: '00000000000000000000:00000000000000004742', 2: '00000000000000000000:00000000000000007703', 3: '00000000000000000000:00000000000000003056'} 2024-04-19T23:39:37.470 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-04-19T23:39:37.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.471 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:37.533 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:37.533 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:37.587 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:39:37.587 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:39:37.587 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:39:37.587 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:39:37.588 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:39:37.588 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 5719348839851422477 2024-04-19T23:39:37.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.588 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:39:37.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:37.588 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:39:37.588 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:39:37.588 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-19T23:39:37.588 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-04-19T23:39:37.588 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:39:37.588 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:39:37.589 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:39:37.589 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:39.315927Z" 2024-04-19T23:39:37.589 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:37.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:37.589 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:39:37.589 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:39:37.589 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-19T23:39:37.589 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000004742", 2024-04-19T23:39:37.589 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:39:37.590 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:39:37.590 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:39:37.590 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:38:56.756407Z" 2024-04-19T23:39:37.590 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:37.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:37.590 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:39:37.590 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:39:37.590 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-19T23:39:37.590 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007703", 2024-04-19T23:39:37.590 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:39:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:39:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:39:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:22.746727Z" 2024-04-19T23:39:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:39:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:39:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-19T23:39:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003056", 2024-04-19T23:39:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:39:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:39:37.592 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:39:37.592 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:50.725314Z" 2024-04-19T23:39:37.592 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:37.592 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:37.592 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:39:37.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.592 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:39:37.592 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-19T23:39:37.592 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-19T23:39:37.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.592 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:39:37Z" 2024-04-19T23:39:37.593 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:39:37.593 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5719348839851422477 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004460", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:39.315927Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004742", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:38:56.756407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007703", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:22.746727Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003056", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:50.725314Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-19T23:39:37Z" } 2024-04-19T23:39:37.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004460', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:39.315927Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004742', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:38:56.756407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007703', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:22.746727Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003056', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:50.725314Z'}}] 2024-04-19T23:39:37.594 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004460', 1: '00000000000000000000:00000000000000004742', 2: '00000000000000000000:00000000000000007703', 3: '00000000000000000000:00000000000000003056'} 2024-04-19T23:39:37.594 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004460', 1: '00000000000000000000:00000000000000004742', 2: '00000000000000000000:00000000000000007703', 3: '00000000000000000000:00000000000000003056'} 2024-04-19T23:39:37.594 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-04-19T23:39:37.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2024-04-19T23:39:37.597 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:39:37.597 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:39:37.597 DEBUG:boto:Method: PUT 2024-04-19T23:39:37.597 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:39:37.598 DEBUG:boto:Data: 2024-04-19T23:39:37.598 DEBUG:boto:Headers: {} 2024-04-19T23:39:37.598 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:37.598 DEBUG:boto:Port: 8000 2024-04-19T23:39:37.598 DEBUG:boto:Params: {} 2024-04-19T23:39:37.598 DEBUG:boto:Token: None 2024-04-19T23:39:37.598 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:39:37 GMT /aktlji-23/ 2024-04-19T23:39:37.598 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5yjxOBH8cjAYcxZ71Pj5UCHIpcw= 2024-04-19T23:39:37.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5yjxOBH8cjAYcxZ71Pj5UCHIpcw=', 'Content-Length': '0'} 2024-04-19T23:39:37.615 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ac82928a2ebdb807-00662300b9-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:37.615 DEBUG:rgw_multi.tests:created bucket=aktlji-23 2024-04-19T23:39:37.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.616 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:37.673 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:37.673 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:37.710 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:39:37.728 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:39:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:39:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:39:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:39:37.728 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:39:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713569977.617055_202.1", 2024-04-19T23:39:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:39:37.617055Z" 2024-04-19T23:39:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:39:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:39:37.729 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.729 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:39:37.730 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569977.617055_202.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:39:37.730 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:39:37.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.731 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:37.792 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:37.793 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:37.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:39:37.830+0000 7f140add0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:39:37.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:39:37.831+0000 7f140add0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:39:37.843 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:39:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:39:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:39:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:39:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:39:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:39:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:39:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:39:37.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:39:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:39:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:39:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:39:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:39:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:39:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:39:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:39:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:39:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:39:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:39:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:39:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:39:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:39:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:39:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:39:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:39:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:39:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:39:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:39:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:39:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713569977.617055_202.1", 2024-04-19T23:39:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:39:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:39:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:39:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:39:37.617055Z", 2024-04-19T23:39:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:39:37.846 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:39:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:39:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:39:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:39:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:39:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:39:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:39:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:39:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:39:37.847 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:39:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:39:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:39:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:39:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:39:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:39:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:39:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:39:37Z" 2024-04-19T23:39:37.848 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:39:37.849 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569977.617055_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:39:37.617055Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:39:37Z" } 2024-04-19T23:39:37.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569977.617055_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:39:37.617055Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:39:37.850 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569977.617055_202.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:39:37.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569977.617055_202.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:39:37.850 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:39:37.850 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:39:37.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:39:37.850 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:39:37.910 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:37.910 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:37.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:37.947+0000 7ff3f5a13780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:39:37.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:37.948+0000 7ff3f5a13780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:39:37.959 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:39:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:39:37.959 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:39:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:39:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:39:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:39:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:39:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:39:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:39:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:39:37.960 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:39:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:39:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:39:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:39:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:39:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:39:37.961 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:39:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:39:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:39:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:39:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:37.962 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:39:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713569977.617055_202.1", 2024-04-19T23:39:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:39:37.617055Z", 2024-04-19T23:39:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:39:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:39:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:39:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:39:37.963 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:39:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:39:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:39:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:39:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:39:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:39:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:39:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:39:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:39:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.964 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:39:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:39:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:39:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:39:37.965 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:39:37Z" 2024-04-19T23:39:37.965 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:39:37.966 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713569977.617055_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:39:37.617055Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:39:37Z" } 2024-04-19T23:39:37.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569977.617055_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:39:37.617055Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:39:37.966 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569977.617055_202.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:39:37.967 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569977.617055_202.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:39:37.967 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:39:37.967 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:39:37.967 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:39:37.967 DEBUG:boto:Method: HEAD 2024-04-19T23:39:37.967 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:39:37.967 DEBUG:boto:Data: 2024-04-19T23:39:37.967 DEBUG:boto:Headers: {} 2024-04-19T23:39:37.967 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:37.967 DEBUG:boto:Port: 8000 2024-04-19T23:39:37.967 DEBUG:boto:Params: {} 2024-04-19T23:39:37.968 DEBUG:boto:Token: None 2024-04-19T23:39:37.968 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:39:37 GMT /aktlji-23/ 2024-04-19T23:39:37.968 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jD690KVpg1anbS8CtWjuPEP3VOY= 2024-04-19T23:39:37.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:39:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jD690KVpg1anbS8CtWjuPEP3VOY=', 'Content-Length': '0'} 2024-04-19T23:39:37.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccb7d22e3db3adc5-00662300b9-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:37.971 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:39:37.971 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:39:37.971 DEBUG:boto:Method: PUT 2024-04-19T23:39:37.971 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:39:37.971 DEBUG:boto:Data: 2024-04-19T23:39:37.971 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:39:37.971 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:39:37.971 DEBUG:boto:Port: 8000 2024-04-19T23:39:37.971 DEBUG:boto:Params: {} 2024-04-19T23:39:37.972 DEBUG:boto:Token: None 2024-04-19T23:39:37.972 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:39:37 GMT /aktlji-23/obj 2024-04-19T23:39:37.972 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:l0KLQoBfJSS7Rq1xfzBqhjSi39U= 2024-04-19T23:39:37.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:39:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l0KLQoBfJSS7Rq1xfzBqhjSi39U='} 2024-04-19T23:39:37.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033496cfae5fea975-00662300b9-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:39:37.979 DEBUG:rgw_multi.tests:created initial version id=None 2024-04-19T23:39:37.979 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-23 2024-04-19T23:39:37.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:39:37.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:39:37.980 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:39:38.047 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:38.047 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:38.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:39:38.091+0000 7efd1a129780 1 waiting to reach incremental sync.. 2024-04-19T23:39:43.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:39:43.092+0000 7efd1a129780 1 waiting to reach incremental sync.. 2024-04-19T23:39:48.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:39:48.094+0000 7efd1a129780 1 bucket sync caught up with source: 2024-04-19T23:39:48.091 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000001.60.6, , ] 2024-04-19T23:39:48.092 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000001.60.6, , ] 2024-04-19T23:39:48.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:39:48.094+0000 7efd1a129780 0 bucket checkpoint complete 2024-04-19T23:39:48.109 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-23 2024-04-19T23:39:48.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:39:48.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:39:48.109 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:39:48.177 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:48.177 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:48.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:48.222+0000 7ffb0dbde780 1 bucket sync caught up with source: 2024-04-19T23:39:48.220 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.60.6, , ] 2024-04-19T23:39:48.220 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.60.6, , ] 2024-04-19T23:39:48.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:48.222+0000 7ffb0dbde780 0 bucket checkpoint complete 2024-04-19T23:39:48.234 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-23 2024-04-19T23:39:48.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:39:48.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:39:48.235 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:39:48.296 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:48.296 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:48.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:48.339+0000 7fc020e60780 1 waiting to reach incremental sync.. 2024-04-19T23:39:53.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:53.339+0000 7fc020e60780 1 waiting to reach incremental sync.. 2024-04-19T23:39:58.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:58.341+0000 7fc020e60780 1 bucket sync caught up with source: 2024-04-19T23:39:58.339 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.79.6, , ] 2024-04-19T23:39:58.339 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.79.6, , ] 2024-04-19T23:39:58.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:58.341+0000 7fc020e60780 0 bucket checkpoint complete 2024-04-19T23:39:58.359 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-23 2024-04-19T23:39:58.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:39:58.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:39:58.360 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:39:58.431 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:39:58.431 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:39:58.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:39:58.473+0000 7f6546a7d780 1 waiting to reach incremental sync.. 2024-04-19T23:40:03.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:03.475+0000 7f6546a7d780 1 bucket sync caught up with source: 2024-04-19T23:40:03.473 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.79.6, , ] 2024-04-19T23:40:03.474 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.79.6, , ] 2024-04-19T23:40:03.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:03.475+0000 7f6546a7d780 0 bucket checkpoint complete 2024-04-19T23:40:03.490 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-23 2024-04-19T23:40:03.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:03.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:03.490 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:40:03.555 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:40:03.555 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:40:03.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:03.599+0000 7faa72cab780 1 bucket sync caught up with source: 2024-04-19T23:40:03.597 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.72.6, , ] 2024-04-19T23:40:03.597 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.72.6, , ] 2024-04-19T23:40:03.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:03.599+0000 7faa72cab780 0 bucket checkpoint complete 2024-04-19T23:40:03.611 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-23 2024-04-19T23:40:03.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:03.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:03.612 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:40:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:40:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:40:03.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:40:03.722+0000 7f7188cc9780 1 bucket sync caught up with source: 2024-04-19T23:40:03.720 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000001.72.6, , ] 2024-04-19T23:40:03.720 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000001.72.6, , ] 2024-04-19T23:40:03.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:40:03.722+0000 7f7188cc9780 0 bucket checkpoint complete 2024-04-19T23:40:03.735 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-23 zones={test-zone2, test-zone1} 2024-04-19T23:40:03.735 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.735 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.735 DEBUG:boto:Method: HEAD 2024-04-19T23:40:03.735 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:40:03.735 DEBUG:boto:Data: 2024-04-19T23:40:03.735 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.735 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:03.736 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.736 DEBUG:boto:Params: {} 2024-04-19T23:40:03.736 DEBUG:boto:Token: None 2024-04-19T23:40:03.736 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/ 2024-04-19T23:40:03.736 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vyyhEOUpoOuLt7eHquYOosuTZ+E= 2024-04-19T23:40:03.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vyyhEOUpoOuLt7eHquYOosuTZ+E=', 'Content-Length': '0'} 2024-04-19T23:40:03.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6323f60e0f8c3b5-00662300d3-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.739 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.739 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.739 DEBUG:boto:Method: HEAD 2024-04-19T23:40:03.739 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:40:03.739 DEBUG:boto:Data: 2024-04-19T23:40:03.739 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.739 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:03.739 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.740 DEBUG:boto:Params: {} 2024-04-19T23:40:03.740 DEBUG:boto:Token: None 2024-04-19T23:40:03.740 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/ 2024-04-19T23:40:03.740 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vyyhEOUpoOuLt7eHquYOosuTZ+E= 2024-04-19T23:40:03.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vyyhEOUpoOuLt7eHquYOosuTZ+E=', 'Content-Length': '0'} 2024-04-19T23:40:03.742 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d9eba1ea921c51f-00662300d3-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.743 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:40:03.743 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.743 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.743 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.743 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.743 DEBUG:boto:Method: GET 2024-04-19T23:40:03.743 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.743 DEBUG:boto:Data: 2024-04-19T23:40:03.743 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.744 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:03.744 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.744 DEBUG:boto:Params: {} 2024-04-19T23:40:03.744 DEBUG:boto:Token: None 2024-04-19T23:40:03.744 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/?versions 2024-04-19T23:40:03.744 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ= 2024-04-19T23:40:03.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ=', 'Content-Length': '0'} 2024-04-19T23:40:03.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000241c511bedb12cc8-00662300d3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.794 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:03.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:40:03.795 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:40:03.795 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.795 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.795 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.795 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.795 DEBUG:boto:Method: GET 2024-04-19T23:40:03.796 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.796 DEBUG:boto:Data: 2024-04-19T23:40:03.796 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.796 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:03.796 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.796 DEBUG:boto:Params: {} 2024-04-19T23:40:03.796 DEBUG:boto:Token: None 2024-04-19T23:40:03.796 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/?versions 2024-04-19T23:40:03.797 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ= 2024-04-19T23:40:03.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ=', 'Content-Length': '0'} 2024-04-19T23:40:03.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c2c53e513bf71dc-00662300d3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.800 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:03.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:40:03.800 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.800 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.800 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.801 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.801 DEBUG:boto:Method: GET 2024-04-19T23:40:03.801 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.801 DEBUG:boto:Data: 2024-04-19T23:40:03.801 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.801 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:03.801 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.801 DEBUG:boto:Params: {} 2024-04-19T23:40:03.801 DEBUG:boto:Token: None 2024-04-19T23:40:03.801 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/?versions 2024-04-19T23:40:03.801 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ= 2024-04-19T23:40:03.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ=', 'Content-Length': '0'} 2024-04-19T23:40:03.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b1612731fd2cd52-00662300d3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.804 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:03.805 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.805 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.805 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.805 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.805 DEBUG:boto:Method: GET 2024-04-19T23:40:03.805 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.805 DEBUG:boto:Data: 2024-04-19T23:40:03.805 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.805 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:03.805 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.805 DEBUG:boto:Params: {} 2024-04-19T23:40:03.806 DEBUG:boto:Token: None 2024-04-19T23:40:03.806 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/?versions 2024-04-19T23:40:03.806 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ= 2024-04-19T23:40:03.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ=', 'Content-Length': '0'} 2024-04-19T23:40:03.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7eb353466650d0-00662300d3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.850 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:03.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:40:03.851 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:03.851 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:03.851 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:03.851 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:03.851 DEBUG:boto:Method: GET 2024-04-19T23:40:03.851 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:03.851 DEBUG:boto:Data: 2024-04-19T23:40:03.851 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.851 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:03.852 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.852 DEBUG:boto:Params: {} 2024-04-19T23:40:03.852 DEBUG:boto:Token: None 2024-04-19T23:40:03.852 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:03.852 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QxQAQBN3NQrCsGOikFUW3sp2+us= 2024-04-19T23:40:03.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QxQAQBN3NQrCsGOikFUW3sp2+us=', 'Content-Length': '0'} 2024-04-19T23:40:03.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000baa9f96cefc7ad2a-00662300d3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.854 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:03.854 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:03.855 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:03.855 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:03.855 DEBUG:boto:Method: GET 2024-04-19T23:40:03.855 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:03.855 DEBUG:boto:Data: 2024-04-19T23:40:03.855 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.855 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:03.855 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.855 DEBUG:boto:Params: {} 2024-04-19T23:40:03.855 DEBUG:boto:Token: None 2024-04-19T23:40:03.856 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:03.856 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QxQAQBN3NQrCsGOikFUW3sp2+us= 2024-04-19T23:40:03.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QxQAQBN3NQrCsGOikFUW3sp2+us=', 'Content-Length': '0'} 2024-04-19T23:40:03.858 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de9a601457cdb83f-00662300d3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.858 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:03.858 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:03.858 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:03.858 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:03.858 DEBUG:boto:Method: HEAD 2024-04-19T23:40:03.858 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:03.858 DEBUG:boto:Data: 2024-04-19T23:40:03.858 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.859 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:03.859 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.859 DEBUG:boto:Params: {} 2024-04-19T23:40:03.859 DEBUG:boto:Token: None 2024-04-19T23:40:03.859 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:03.859 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Tn9+XOM1LWq4TEFG1aH9Jn8pcwY= 2024-04-19T23:40:03.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Tn9+XOM1LWq4TEFG1aH9Jn8pcwY=', 'Content-Length': '0'} 2024-04-19T23:40:03.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000037a18d1fbfee7eb-00662300d3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.861 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:03.861 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:03.861 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:03.861 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:03.862 DEBUG:boto:Method: HEAD 2024-04-19T23:40:03.862 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:03.862 DEBUG:boto:Data: 2024-04-19T23:40:03.862 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.862 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:03.862 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.862 DEBUG:boto:Params: {} 2024-04-19T23:40:03.862 DEBUG:boto:Token: None 2024-04-19T23:40:03.862 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:03.862 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Tn9+XOM1LWq4TEFG1aH9Jn8pcwY= 2024-04-19T23:40:03.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Tn9+XOM1LWq4TEFG1aH9Jn8pcwY=', 'Content-Length': '0'} 2024-04-19T23:40:03.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000095f50d423ea14215-00662300d3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:40:03.865 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:03.865 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:03.865 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:03.865 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:03.865 DEBUG:boto:Method: GET 2024-04-19T23:40:03.865 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:03.865 DEBUG:boto:Data: 2024-04-19T23:40:03.865 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.865 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:03.865 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.865 DEBUG:boto:Params: {} 2024-04-19T23:40:03.865 DEBUG:boto:Token: None 2024-04-19T23:40:03.866 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:03.866 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QxQAQBN3NQrCsGOikFUW3sp2+us= 2024-04-19T23:40:03.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QxQAQBN3NQrCsGOikFUW3sp2+us=', 'Content-Length': '0'} 2024-04-19T23:40:03.868 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000084573ce065608c09-00662300d3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.868 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:03.868 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:03.868 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:03.868 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:03.868 DEBUG:boto:Method: GET 2024-04-19T23:40:03.868 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:03.869 DEBUG:boto:Data: 2024-04-19T23:40:03.869 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.869 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:03.869 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.869 DEBUG:boto:Params: {} 2024-04-19T23:40:03.869 DEBUG:boto:Token: None 2024-04-19T23:40:03.869 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:03.869 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QxQAQBN3NQrCsGOikFUW3sp2+us= 2024-04-19T23:40:03.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QxQAQBN3NQrCsGOikFUW3sp2+us=', 'Content-Length': '0'} 2024-04-19T23:40:03.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f23ad4bc1ff6427-00662300d3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.871 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.871 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.871 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:40:03.871 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:40:03.872 DEBUG:boto:Method: GET 2024-04-19T23:40:03.872 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:40:03.872 DEBUG:boto:Data: 2024-04-19T23:40:03.872 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.872 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:03.872 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.872 DEBUG:boto:Params: {} 2024-04-19T23:40:03.872 DEBUG:boto:Token: None 2024-04-19T23:40:03.872 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/ 2024-04-19T23:40:03.872 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2MRZHMANYHYtE9EMyMEn8GmMld4= 2024-04-19T23:40:03.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2MRZHMANYHYtE9EMyMEn8GmMld4=', 'Content-Length': '0'} 2024-04-19T23:40:03.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fbadc1faaa917fb-00662300d3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.875 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:03.875 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.876 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.876 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:40:03.876 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:40:03.876 DEBUG:boto:Method: GET 2024-04-19T23:40:03.876 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:40:03.876 DEBUG:boto:Data: 2024-04-19T23:40:03.876 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.876 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:03.876 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.876 DEBUG:boto:Params: {} 2024-04-19T23:40:03.876 DEBUG:boto:Token: None 2024-04-19T23:40:03.877 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/ 2024-04-19T23:40:03.877 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2MRZHMANYHYtE9EMyMEn8GmMld4= 2024-04-19T23:40:03.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2MRZHMANYHYtE9EMyMEn8GmMld4=', 'Content-Length': '0'} 2024-04-19T23:40:03.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe270f42741d478d-00662300d3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.879 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:03.880 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:03.880 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:03.880 DEBUG:boto:Method: HEAD 2024-04-19T23:40:03.880 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:03.880 DEBUG:boto:Data: 2024-04-19T23:40:03.880 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.880 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:03.880 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.880 DEBUG:boto:Params: {} 2024-04-19T23:40:03.880 DEBUG:boto:Token: None 2024-04-19T23:40:03.881 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/obj 2024-04-19T23:40:03.881 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tTndWYZLVHig6skJervxkpu8Ol4= 2024-04-19T23:40:03.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tTndWYZLVHig6skJervxkpu8Ol4=', 'Content-Length': '0'} 2024-04-19T23:40:03.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c18b479e37f49cb-00662300d3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.883 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:03.883 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:03.883 DEBUG:boto:Method: HEAD 2024-04-19T23:40:03.883 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:03.883 DEBUG:boto:Data: 2024-04-19T23:40:03.883 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.883 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:03.883 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.883 DEBUG:boto:Params: {} 2024-04-19T23:40:03.883 DEBUG:boto:Token: None 2024-04-19T23:40:03.884 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/obj 2024-04-19T23:40:03.884 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tTndWYZLVHig6skJervxkpu8Ol4= 2024-04-19T23:40:03.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tTndWYZLVHig6skJervxkpu8Ol4=', 'Content-Length': '0'} 2024-04-19T23:40:03.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000062dd4c5e520ddad2-00662300d3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:40:03.886 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:03.886 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:03.886 DEBUG:boto:Method: GET 2024-04-19T23:40:03.886 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:03.886 DEBUG:boto:Data: 2024-04-19T23:40:03.886 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.887 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:03.887 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.887 DEBUG:boto:Params: {} 2024-04-19T23:40:03.887 DEBUG:boto:Token: None 2024-04-19T23:40:03.887 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/obj 2024-04-19T23:40:03.887 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Fi+mQ+Js6PPA6a2lsdAlXY48xCA= 2024-04-19T23:40:03.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Fi+mQ+Js6PPA6a2lsdAlXY48xCA=', 'Content-Length': '0'} 2024-04-19T23:40:03.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008565a4ad92a8b706-00662300d3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.889 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:03.889 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:03.889 DEBUG:boto:Method: GET 2024-04-19T23:40:03.889 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:03.889 DEBUG:boto:Data: 2024-04-19T23:40:03.889 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.890 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:03.890 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.890 DEBUG:boto:Params: {} 2024-04-19T23:40:03.890 DEBUG:boto:Token: None 2024-04-19T23:40:03.890 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/obj 2024-04-19T23:40:03.890 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Fi+mQ+Js6PPA6a2lsdAlXY48xCA= 2024-04-19T23:40:03.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Fi+mQ+Js6PPA6a2lsdAlXY48xCA=', 'Content-Length': '0'} 2024-04-19T23:40:03.892 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a788780542e4972-00662300d3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.892 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.892 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.892 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:40:03.892 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:40:03.893 DEBUG:boto:Method: GET 2024-04-19T23:40:03.893 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:40:03.893 DEBUG:boto:Data: 2024-04-19T23:40:03.893 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.893 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:03.893 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.893 DEBUG:boto:Params: {} 2024-04-19T23:40:03.893 DEBUG:boto:Token: None 2024-04-19T23:40:03.893 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/ 2024-04-19T23:40:03.893 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2MRZHMANYHYtE9EMyMEn8GmMld4= 2024-04-19T23:40:03.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2MRZHMANYHYtE9EMyMEn8GmMld4=', 'Content-Length': '0'} 2024-04-19T23:40:03.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005317b0302402d9b3-00662300d3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.942 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:03.943 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.943 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.943 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:40:03.943 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:40:03.943 DEBUG:boto:Method: GET 2024-04-19T23:40:03.943 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:40:03.943 DEBUG:boto:Data: 2024-04-19T23:40:03.943 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.943 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:03.943 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.943 DEBUG:boto:Params: {} 2024-04-19T23:40:03.944 DEBUG:boto:Token: None 2024-04-19T23:40:03.944 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/ 2024-04-19T23:40:03.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2MRZHMANYHYtE9EMyMEn8GmMld4= 2024-04-19T23:40:03.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2MRZHMANYHYtE9EMyMEn8GmMld4=', 'Content-Length': '0'} 2024-04-19T23:40:03.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a1ab73f377fd9e0-00662300d3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.947 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:03.947 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-23 zones={test-zone2, test-zone1} 2024-04-19T23:40:03.947 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-23 zones={test-zone3, test-zone1} 2024-04-19T23:40:03.947 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.947 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.947 DEBUG:boto:Method: HEAD 2024-04-19T23:40:03.947 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:40:03.947 DEBUG:boto:Data: 2024-04-19T23:40:03.948 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.948 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:03.948 DEBUG:boto:Port: 8001 2024-04-19T23:40:03.948 DEBUG:boto:Params: {} 2024-04-19T23:40:03.948 DEBUG:boto:Token: None 2024-04-19T23:40:03.948 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/ 2024-04-19T23:40:03.948 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vyyhEOUpoOuLt7eHquYOosuTZ+E= 2024-04-19T23:40:03.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vyyhEOUpoOuLt7eHquYOosuTZ+E=', 'Content-Length': '0'} 2024-04-19T23:40:03.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e36481cea97dec92-00662300d3-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.951 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.951 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.951 DEBUG:boto:Method: HEAD 2024-04-19T23:40:03.951 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:40:03.951 DEBUG:boto:Data: 2024-04-19T23:40:03.951 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.951 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:03.951 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.951 DEBUG:boto:Params: {} 2024-04-19T23:40:03.952 DEBUG:boto:Token: None 2024-04-19T23:40:03.952 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/ 2024-04-19T23:40:03.952 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vyyhEOUpoOuLt7eHquYOosuTZ+E= 2024-04-19T23:40:03.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vyyhEOUpoOuLt7eHquYOosuTZ+E=', 'Content-Length': '0'} 2024-04-19T23:40:03.954 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1d01fbac669552b-00662300d3-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.955 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:40:03.955 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.955 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.955 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.955 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.955 DEBUG:boto:Method: GET 2024-04-19T23:40:03.955 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.955 DEBUG:boto:Data: 2024-04-19T23:40:03.955 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.955 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:03.956 DEBUG:boto:Port: 8001 2024-04-19T23:40:03.956 DEBUG:boto:Params: {} 2024-04-19T23:40:03.956 DEBUG:boto:Token: None 2024-04-19T23:40:03.956 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/?versions 2024-04-19T23:40:03.956 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ= 2024-04-19T23:40:03.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ=', 'Content-Length': '0'} 2024-04-19T23:40:03.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027f736bc8e78f447-00662300d3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.958 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:03.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:40:03.959 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:40:03.959 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.959 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.959 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.959 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.959 DEBUG:boto:Method: GET 2024-04-19T23:40:03.959 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.959 DEBUG:boto:Data: 2024-04-19T23:40:03.959 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.960 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:03.960 DEBUG:boto:Port: 8000 2024-04-19T23:40:03.960 DEBUG:boto:Params: {} 2024-04-19T23:40:03.960 DEBUG:boto:Token: None 2024-04-19T23:40:03.960 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/?versions 2024-04-19T23:40:03.960 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ= 2024-04-19T23:40:03.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ=', 'Content-Length': '0'} 2024-04-19T23:40:03.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000304ebe95424a4c90-00662300d3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:03.963 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:03.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:40:03.963 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:03.963 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:03.963 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.963 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.963 DEBUG:boto:Method: GET 2024-04-19T23:40:03.964 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:40:03.964 DEBUG:boto:Data: 2024-04-19T23:40:03.964 DEBUG:boto:Headers: {} 2024-04-19T23:40:03.964 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:03.964 DEBUG:boto:Port: 8001 2024-04-19T23:40:03.964 DEBUG:boto:Params: {} 2024-04-19T23:40:03.964 DEBUG:boto:Token: None 2024-04-19T23:40:03.964 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:03 GMT /aktlji-23/?versions 2024-04-19T23:40:03.964 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ= 2024-04-19T23:40:03.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:udJWSfAqO1/HQv5JRCAClExfBsQ=', 'Content-Length': '0'} 2024-04-19T23:40:03.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c73525a1c7a1e201-00662300d3-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.010 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.011 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.011 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.011 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.011 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.011 DEBUG:boto:Method: GET 2024-04-19T23:40:04.011 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.011 DEBUG:boto:Data: 2024-04-19T23:40:04.011 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.012 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:04.012 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.012 DEBUG:boto:Params: {} 2024-04-19T23:40:04.012 DEBUG:boto:Token: None 2024-04-19T23:40:04.012 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/?versions 2024-04-19T23:40:04.012 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OJ6UipOpPnj1r6lqzl/6pUn2BuY= 2024-04-19T23:40:04.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OJ6UipOpPnj1r6lqzl/6pUn2BuY=', 'Content-Length': '0'} 2024-04-19T23:40:04.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b07d31559471c56f-00662300d4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.058 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:40:04.059 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.059 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.059 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.059 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.059 DEBUG:boto:Method: GET 2024-04-19T23:40:04.059 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.060 DEBUG:boto:Data: 2024-04-19T23:40:04.060 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.060 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.060 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.060 DEBUG:boto:Params: {} 2024-04-19T23:40:04.060 DEBUG:boto:Token: None 2024-04-19T23:40:04.060 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.060 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8= 2024-04-19T23:40:04.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8=', 'Content-Length': '0'} 2024-04-19T23:40:04.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000022bb425f5d187361-00662300d4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.063 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.063 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.063 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.063 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.063 DEBUG:boto:Method: GET 2024-04-19T23:40:04.063 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.063 DEBUG:boto:Data: 2024-04-19T23:40:04.063 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.063 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:04.064 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.064 DEBUG:boto:Params: {} 2024-04-19T23:40:04.064 DEBUG:boto:Token: None 2024-04-19T23:40:04.064 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.064 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8= 2024-04-19T23:40:04.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8=', 'Content-Length': '0'} 2024-04-19T23:40:04.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f6143f7b7e6a40f-00662300d4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.066 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.066 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.066 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.066 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.067 DEBUG:boto:Method: HEAD 2024-04-19T23:40:04.067 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.067 DEBUG:boto:Data: 2024-04-19T23:40:04.067 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.067 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.067 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.067 DEBUG:boto:Params: {} 2024-04-19T23:40:04.067 DEBUG:boto:Token: None 2024-04-19T23:40:04.067 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.067 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5q4BY8DnyW4hxdHuUvWxqlWLXZ8= 2024-04-19T23:40:04.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5q4BY8DnyW4hxdHuUvWxqlWLXZ8=', 'Content-Length': '0'} 2024-04-19T23:40:04.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c2e14e87a06cc53-00662300d4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.070 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.070 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.070 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.070 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.070 DEBUG:boto:Method: HEAD 2024-04-19T23:40:04.070 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.070 DEBUG:boto:Data: 2024-04-19T23:40:04.070 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.070 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:04.070 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.071 DEBUG:boto:Params: {} 2024-04-19T23:40:04.071 DEBUG:boto:Token: None 2024-04-19T23:40:04.071 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.071 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5q4BY8DnyW4hxdHuUvWxqlWLXZ8= 2024-04-19T23:40:04.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5q4BY8DnyW4hxdHuUvWxqlWLXZ8=', 'Content-Length': '0'} 2024-04-19T23:40:04.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f8e0dda3026e4b89-00662300d4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:40:04.073 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.073 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.073 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.073 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.074 DEBUG:boto:Method: GET 2024-04-19T23:40:04.074 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.074 DEBUG:boto:Data: 2024-04-19T23:40:04.074 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.074 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.074 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.074 DEBUG:boto:Params: {} 2024-04-19T23:40:04.074 DEBUG:boto:Token: None 2024-04-19T23:40:04.074 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.074 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8= 2024-04-19T23:40:04.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8=', 'Content-Length': '0'} 2024-04-19T23:40:04.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b84be2929ae34013-00662300d4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.076 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.077 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.077 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.077 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.077 DEBUG:boto:Method: GET 2024-04-19T23:40:04.077 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.077 DEBUG:boto:Data: 2024-04-19T23:40:04.077 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.077 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:04.077 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.077 DEBUG:boto:Params: {} 2024-04-19T23:40:04.078 DEBUG:boto:Token: None 2024-04-19T23:40:04.078 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.078 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8= 2024-04-19T23:40:04.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8=', 'Content-Length': '0'} 2024-04-19T23:40:04.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000475e18aac7d3b095-00662300d4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.080 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.080 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.080 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.080 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.080 DEBUG:boto:Method: GET 2024-04-19T23:40:04.081 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:40:04.081 DEBUG:boto:Data: 2024-04-19T23:40:04.081 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.081 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.081 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.081 DEBUG:boto:Params: {} 2024-04-19T23:40:04.081 DEBUG:boto:Token: None 2024-04-19T23:40:04.081 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/ 2024-04-19T23:40:04.081 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0= 2024-04-19T23:40:04.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0=', 'Content-Length': '0'} 2024-04-19T23:40:04.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc30a246e91c9f75-00662300d4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.084 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.084 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.085 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.085 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.085 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.085 DEBUG:boto:Method: GET 2024-04-19T23:40:04.085 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:40:04.085 DEBUG:boto:Data: 2024-04-19T23:40:04.085 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.085 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:04.085 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.085 DEBUG:boto:Params: {} 2024-04-19T23:40:04.085 DEBUG:boto:Token: None 2024-04-19T23:40:04.086 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/ 2024-04-19T23:40:04.086 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0= 2024-04-19T23:40:04.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0=', 'Content-Length': '0'} 2024-04-19T23:40:04.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a51a28941c562d5-00662300d4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.088 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.088 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.089 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.089 DEBUG:boto:Method: HEAD 2024-04-19T23:40:04.089 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:04.089 DEBUG:boto:Data: 2024-04-19T23:40:04.089 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.089 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.089 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.089 DEBUG:boto:Params: {} 2024-04-19T23:40:04.089 DEBUG:boto:Token: None 2024-04-19T23:40:04.089 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj 2024-04-19T23:40:04.090 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RXJxDD1af8dLBDiIqUiX9aa0sFI= 2024-04-19T23:40:04.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RXJxDD1af8dLBDiIqUiX9aa0sFI=', 'Content-Length': '0'} 2024-04-19T23:40:04.092 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d84d620e1fb253e-00662300d4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.092 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.092 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.092 DEBUG:boto:Method: HEAD 2024-04-19T23:40:04.092 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:04.092 DEBUG:boto:Data: 2024-04-19T23:40:04.092 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.092 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:04.093 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.093 DEBUG:boto:Params: {} 2024-04-19T23:40:04.093 DEBUG:boto:Token: None 2024-04-19T23:40:04.093 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj 2024-04-19T23:40:04.093 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RXJxDD1af8dLBDiIqUiX9aa0sFI= 2024-04-19T23:40:04.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RXJxDD1af8dLBDiIqUiX9aa0sFI=', 'Content-Length': '0'} 2024-04-19T23:40:04.095 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050d11d675e7fcfe9-00662300d4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:40:04.095 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.095 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.095 DEBUG:boto:Method: GET 2024-04-19T23:40:04.095 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:04.096 DEBUG:boto:Data: 2024-04-19T23:40:04.096 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.096 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.096 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.096 DEBUG:boto:Params: {} 2024-04-19T23:40:04.096 DEBUG:boto:Token: None 2024-04-19T23:40:04.096 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj 2024-04-19T23:40:04.096 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iD8rjSMPKA4mdAUrXfOmw8l7bSk= 2024-04-19T23:40:04.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iD8rjSMPKA4mdAUrXfOmw8l7bSk=', 'Content-Length': '0'} 2024-04-19T23:40:04.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057c1e1c4949430e9-00662300d4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.098 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.098 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.099 DEBUG:boto:Method: GET 2024-04-19T23:40:04.099 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:04.099 DEBUG:boto:Data: 2024-04-19T23:40:04.099 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.099 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:04.099 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.099 DEBUG:boto:Params: {} 2024-04-19T23:40:04.099 DEBUG:boto:Token: None 2024-04-19T23:40:04.099 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj 2024-04-19T23:40:04.099 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iD8rjSMPKA4mdAUrXfOmw8l7bSk= 2024-04-19T23:40:04.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iD8rjSMPKA4mdAUrXfOmw8l7bSk=', 'Content-Length': '0'} 2024-04-19T23:40:04.101 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000030f1eb50f1592dea-00662300d4-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.102 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.102 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.102 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.102 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.102 DEBUG:boto:Method: GET 2024-04-19T23:40:04.102 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:40:04.102 DEBUG:boto:Data: 2024-04-19T23:40:04.102 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.102 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.102 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.103 DEBUG:boto:Params: {} 2024-04-19T23:40:04.103 DEBUG:boto:Token: None 2024-04-19T23:40:04.103 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/ 2024-04-19T23:40:04.103 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0= 2024-04-19T23:40:04.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0=', 'Content-Length': '0'} 2024-04-19T23:40:04.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007acd7fe627dbb256-00662300d4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.146 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.147 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.147 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.147 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.147 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.147 DEBUG:boto:Method: GET 2024-04-19T23:40:04.147 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:40:04.147 DEBUG:boto:Data: 2024-04-19T23:40:04.147 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.148 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:04.148 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.148 DEBUG:boto:Params: {} 2024-04-19T23:40:04.148 DEBUG:boto:Token: None 2024-04-19T23:40:04.148 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/ 2024-04-19T23:40:04.148 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0= 2024-04-19T23:40:04.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0=', 'Content-Length': '0'} 2024-04-19T23:40:04.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000329f0e73865fff1b-00662300d4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.151 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.151 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-23 zones={test-zone3, test-zone1} 2024-04-19T23:40:04.151 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-23 zones={test-zone3, test-zone2} 2024-04-19T23:40:04.151 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.151 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.152 DEBUG:boto:Method: HEAD 2024-04-19T23:40:04.152 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:40:04.152 DEBUG:boto:Data: 2024-04-19T23:40:04.152 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.152 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.152 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.152 DEBUG:boto:Params: {} 2024-04-19T23:40:04.152 DEBUG:boto:Token: None 2024-04-19T23:40:04.152 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/ 2024-04-19T23:40:04.152 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5xFxxf18VTuwBkNCaPgKPT9xU6Y= 2024-04-19T23:40:04.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5xFxxf18VTuwBkNCaPgKPT9xU6Y=', 'Content-Length': '0'} 2024-04-19T23:40:04.155 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002807e7aa9dd5e324-00662300d4-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.155 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.155 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.155 DEBUG:boto:Method: HEAD 2024-04-19T23:40:04.155 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:40:04.155 DEBUG:boto:Data: 2024-04-19T23:40:04.155 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.155 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:04.156 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.156 DEBUG:boto:Params: {} 2024-04-19T23:40:04.156 DEBUG:boto:Token: None 2024-04-19T23:40:04.156 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/ 2024-04-19T23:40:04.156 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5xFxxf18VTuwBkNCaPgKPT9xU6Y= 2024-04-19T23:40:04.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5xFxxf18VTuwBkNCaPgKPT9xU6Y=', 'Content-Length': '0'} 2024-04-19T23:40:04.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d19c8539af84a60-00662300d4-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.159 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:40:04.159 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.159 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.159 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.159 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.159 DEBUG:boto:Method: GET 2024-04-19T23:40:04.159 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.159 DEBUG:boto:Data: 2024-04-19T23:40:04.160 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.160 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.160 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.160 DEBUG:boto:Params: {} 2024-04-19T23:40:04.160 DEBUG:boto:Token: None 2024-04-19T23:40:04.160 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/?versions 2024-04-19T23:40:04.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OJ6UipOpPnj1r6lqzl/6pUn2BuY= 2024-04-19T23:40:04.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OJ6UipOpPnj1r6lqzl/6pUn2BuY=', 'Content-Length': '0'} 2024-04-19T23:40:04.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000331b1ceda5a49cde-00662300d4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.163 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:40:04.163 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:40:04.163 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.164 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.164 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.164 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.164 DEBUG:boto:Method: GET 2024-04-19T23:40:04.164 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.164 DEBUG:boto:Data: 2024-04-19T23:40:04.164 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.164 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:04.164 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.164 DEBUG:boto:Params: {} 2024-04-19T23:40:04.164 DEBUG:boto:Token: None 2024-04-19T23:40:04.165 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/?versions 2024-04-19T23:40:04.165 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OJ6UipOpPnj1r6lqzl/6pUn2BuY= 2024-04-19T23:40:04.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OJ6UipOpPnj1r6lqzl/6pUn2BuY=', 'Content-Length': '0'} 2024-04-19T23:40:04.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009471f5623984b9c8-00662300d4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.167 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:40:04.168 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.168 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.168 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.168 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.168 DEBUG:boto:Method: GET 2024-04-19T23:40:04.168 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.168 DEBUG:boto:Data: 2024-04-19T23:40:04.168 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.168 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.169 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.169 DEBUG:boto:Params: {} 2024-04-19T23:40:04.169 DEBUG:boto:Token: None 2024-04-19T23:40:04.169 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/?versions 2024-04-19T23:40:04.169 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OJ6UipOpPnj1r6lqzl/6pUn2BuY= 2024-04-19T23:40:04.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OJ6UipOpPnj1r6lqzl/6pUn2BuY=', 'Content-Length': '0'} 2024-04-19T23:40:04.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de93d841fe69bddf-00662300d4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.214 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.215 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.215 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.215 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.215 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.215 DEBUG:boto:Method: GET 2024-04-19T23:40:04.216 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:40:04.216 DEBUG:boto:Data: 2024-04-19T23:40:04.216 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.216 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:04.216 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.216 DEBUG:boto:Params: {} 2024-04-19T23:40:04.216 DEBUG:boto:Token: None 2024-04-19T23:40:04.216 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/?versions 2024-04-19T23:40:04.216 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OJ6UipOpPnj1r6lqzl/6pUn2BuY= 2024-04-19T23:40:04.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OJ6UipOpPnj1r6lqzl/6pUn2BuY=', 'Content-Length': '0'} 2024-04-19T23:40:04.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a8e0e92de85cc61-00662300d4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.219 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:40:04.219 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.220 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.220 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.220 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.220 DEBUG:boto:Method: GET 2024-04-19T23:40:04.220 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.220 DEBUG:boto:Data: 2024-04-19T23:40:04.220 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.220 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.220 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.220 DEBUG:boto:Params: {} 2024-04-19T23:40:04.221 DEBUG:boto:Token: None 2024-04-19T23:40:04.221 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.221 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8= 2024-04-19T23:40:04.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8=', 'Content-Length': '0'} 2024-04-19T23:40:04.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007a301944eed76285-00662300d4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.226 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.226 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.226 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.226 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.226 DEBUG:boto:Method: GET 2024-04-19T23:40:04.227 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.227 DEBUG:boto:Data: 2024-04-19T23:40:04.227 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.227 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:04.227 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.227 DEBUG:boto:Params: {} 2024-04-19T23:40:04.227 DEBUG:boto:Token: None 2024-04-19T23:40:04.227 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.228 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8= 2024-04-19T23:40:04.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8=', 'Content-Length': '0'} 2024-04-19T23:40:04.229 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b6a0ab30820e1609-00662300d4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.230 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.230 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.230 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.230 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.230 DEBUG:boto:Method: HEAD 2024-04-19T23:40:04.230 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.230 DEBUG:boto:Data: 2024-04-19T23:40:04.230 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.230 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.231 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.231 DEBUG:boto:Params: {} 2024-04-19T23:40:04.231 DEBUG:boto:Token: None 2024-04-19T23:40:04.231 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.231 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5q4BY8DnyW4hxdHuUvWxqlWLXZ8= 2024-04-19T23:40:04.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5q4BY8DnyW4hxdHuUvWxqlWLXZ8=', 'Content-Length': '0'} 2024-04-19T23:40:04.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008cbbb7e0bd5d644a-00662300d4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.233 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.233 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.233 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.233 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.233 DEBUG:boto:Method: HEAD 2024-04-19T23:40:04.234 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.234 DEBUG:boto:Data: 2024-04-19T23:40:04.234 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.234 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:04.234 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.234 DEBUG:boto:Params: {} 2024-04-19T23:40:04.234 DEBUG:boto:Token: None 2024-04-19T23:40:04.234 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.234 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5q4BY8DnyW4hxdHuUvWxqlWLXZ8= 2024-04-19T23:40:04.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5q4BY8DnyW4hxdHuUvWxqlWLXZ8=', 'Content-Length': '0'} 2024-04-19T23:40:04.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d983907f32fc72f5-00662300d4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:40:04.237 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.237 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.237 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.237 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.237 DEBUG:boto:Method: GET 2024-04-19T23:40:04.237 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.237 DEBUG:boto:Data: 2024-04-19T23:40:04.238 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.238 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.238 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.238 DEBUG:boto:Params: {} 2024-04-19T23:40:04.238 DEBUG:boto:Token: None 2024-04-19T23:40:04.238 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.238 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8= 2024-04-19T23:40:04.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8=', 'Content-Length': '0'} 2024-04-19T23:40:04.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000620fb20460a32dfb-00662300d4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.241 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.241 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.241 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.241 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:40:04.241 DEBUG:boto:Method: GET 2024-04-19T23:40:04.241 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.241 DEBUG:boto:Data: 2024-04-19T23:40:04.241 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.241 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:04.241 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.241 DEBUG:boto:Params: {} 2024-04-19T23:40:04.242 DEBUG:boto:Token: None 2024-04-19T23:40:04.242 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:40:04.242 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8= 2024-04-19T23:40:04.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m1D/3PfC+6yLSio73aBD3+U/Cl8=', 'Content-Length': '0'} 2024-04-19T23:40:04.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009cf55da26dd9a503-00662300d4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.244 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.244 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.244 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.244 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.244 DEBUG:boto:Method: GET 2024-04-19T23:40:04.244 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:40:04.244 DEBUG:boto:Data: 2024-04-19T23:40:04.245 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.245 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.245 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.245 DEBUG:boto:Params: {} 2024-04-19T23:40:04.245 DEBUG:boto:Token: None 2024-04-19T23:40:04.245 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/ 2024-04-19T23:40:04.245 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0= 2024-04-19T23:40:04.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0=', 'Content-Length': '0'} 2024-04-19T23:40:04.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000927a705dfee6f6f6-00662300d4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.248 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.248 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.248 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.248 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.249 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.249 DEBUG:boto:Method: GET 2024-04-19T23:40:04.249 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:40:04.249 DEBUG:boto:Data: 2024-04-19T23:40:04.249 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.249 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:04.249 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.249 DEBUG:boto:Params: {} 2024-04-19T23:40:04.249 DEBUG:boto:Token: None 2024-04-19T23:40:04.249 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/ 2024-04-19T23:40:04.250 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0= 2024-04-19T23:40:04.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0=', 'Content-Length': '0'} 2024-04-19T23:40:04.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d492a7e57e52635f-00662300d4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.253 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.253 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.253 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.253 DEBUG:boto:Method: HEAD 2024-04-19T23:40:04.253 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:04.253 DEBUG:boto:Data: 2024-04-19T23:40:04.253 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.253 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.254 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.254 DEBUG:boto:Params: {} 2024-04-19T23:40:04.254 DEBUG:boto:Token: None 2024-04-19T23:40:04.254 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj 2024-04-19T23:40:04.254 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RXJxDD1af8dLBDiIqUiX9aa0sFI= 2024-04-19T23:40:04.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RXJxDD1af8dLBDiIqUiX9aa0sFI=', 'Content-Length': '0'} 2024-04-19T23:40:04.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c11a43f3c2609572-00662300d4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.256 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.256 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.256 DEBUG:boto:Method: HEAD 2024-04-19T23:40:04.256 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:04.256 DEBUG:boto:Data: 2024-04-19T23:40:04.257 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.257 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:04.257 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.257 DEBUG:boto:Params: {} 2024-04-19T23:40:04.257 DEBUG:boto:Token: None 2024-04-19T23:40:04.257 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj 2024-04-19T23:40:04.257 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RXJxDD1af8dLBDiIqUiX9aa0sFI= 2024-04-19T23:40:04.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RXJxDD1af8dLBDiIqUiX9aa0sFI=', 'Content-Length': '0'} 2024-04-19T23:40:04.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cb110a23eb20bb30-00662300d4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:40:04.259 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.259 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.260 DEBUG:boto:Method: GET 2024-04-19T23:40:04.260 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:04.260 DEBUG:boto:Data: 2024-04-19T23:40:04.260 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.260 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.260 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.260 DEBUG:boto:Params: {} 2024-04-19T23:40:04.260 DEBUG:boto:Token: None 2024-04-19T23:40:04.260 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj 2024-04-19T23:40:04.260 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iD8rjSMPKA4mdAUrXfOmw8l7bSk= 2024-04-19T23:40:04.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iD8rjSMPKA4mdAUrXfOmw8l7bSk=', 'Content-Length': '0'} 2024-04-19T23:40:04.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ab5223b68e2c1ee-00662300d4-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.263 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:04.263 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:04.263 DEBUG:boto:Method: GET 2024-04-19T23:40:04.263 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:04.263 DEBUG:boto:Data: 2024-04-19T23:40:04.263 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.263 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:04.263 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.263 DEBUG:boto:Params: {} 2024-04-19T23:40:04.263 DEBUG:boto:Token: None 2024-04-19T23:40:04.264 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/obj 2024-04-19T23:40:04.264 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iD8rjSMPKA4mdAUrXfOmw8l7bSk= 2024-04-19T23:40:04.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iD8rjSMPKA4mdAUrXfOmw8l7bSk=', 'Content-Length': '0'} 2024-04-19T23:40:04.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000baa6d91e55582af6-00662300d4-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.266 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.266 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.267 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.267 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.267 DEBUG:boto:Method: GET 2024-04-19T23:40:04.267 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:40:04.267 DEBUG:boto:Data: 2024-04-19T23:40:04.267 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.267 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:40:04.267 DEBUG:boto:Port: 8001 2024-04-19T23:40:04.267 DEBUG:boto:Params: {} 2024-04-19T23:40:04.268 DEBUG:boto:Token: None 2024-04-19T23:40:04.269 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/ 2024-04-19T23:40:04.269 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0= 2024-04-19T23:40:04.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0=', 'Content-Length': '0'} 2024-04-19T23:40:04.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009482ec0c6ca504fd-00662300d4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.314 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.315 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.315 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.315 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.315 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:40:04.315 DEBUG:boto:Method: GET 2024-04-19T23:40:04.315 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:40:04.315 DEBUG:boto:Data: 2024-04-19T23:40:04.315 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.316 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:40:04.316 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.316 DEBUG:boto:Params: {} 2024-04-19T23:40:04.316 DEBUG:boto:Token: None 2024-04-19T23:40:04.316 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/ 2024-04-19T23:40:04.316 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0= 2024-04-19T23:40:04.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpbjecmSzf7yhxTkgxeFwb5oZW0=', 'Content-Length': '0'} 2024-04-19T23:40:04.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060005e9e7087de62-00662300d4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.319 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:40:04.319 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-23 zones={test-zone3, test-zone2} 2024-04-19T23:40:04.319 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:04.319 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:04.320 DEBUG:boto:path=/aktlji-23/?versioning 2024-04-19T23:40:04.320 DEBUG:boto:auth_path=/aktlji-23/?versioning 2024-04-19T23:40:04.320 DEBUG:boto:Method: PUT 2024-04-19T23:40:04.320 DEBUG:boto:Path: /aktlji-23/?versioning 2024-04-19T23:40:04.320 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:40:04.320 DEBUG:boto:Headers: {} 2024-04-19T23:40:04.320 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:04.320 DEBUG:boto:Port: 8000 2024-04-19T23:40:04.320 DEBUG:boto:Params: {} 2024-04-19T23:40:04.320 DEBUG:boto:Token: None 2024-04-19T23:40:04.321 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:40:04 GMT /aktlji-23/?versioning 2024-04-19T23:40:04.321 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mYotGMdLU8FTXrYgqw9ACcq6xfg= 2024-04-19T23:40:04.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mYotGMdLU8FTXrYgqw9ACcq6xfg=', 'Content-Length': '228'} 2024-04-19T23:40:04.327 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000df8e50527240e525-00662300d4-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:40:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:04.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:40:04.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:40:04.328 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:40:04.390 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:40:04.390 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:40:04.425 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:40:04.440 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:40:04.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:40:04.440 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:40:04.440 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:40:04.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:04.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:40:04.440 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:40:04.440 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:40:04.441 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:04.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:40:04.441 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713570004.330166_206.1", 2024-04-19T23:40:04.441 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:40:04.330166Z" 2024-04-19T23:40:04.441 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:04.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:40:04.441 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:40:04.441 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:40:04.441 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:40:04.441 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:40:04.442 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570004.330166_206.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:40:04.443 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:40:04.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:40:04.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:40:04.443 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:40:04.502 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:40:04.502 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:40:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:40:04.541+0000 7fd6a505b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:40:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:40:04.542+0000 7fd6a505b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:40:04.555 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:40:04.555 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:40:04.555 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:40:04.555 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:40:04.556 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:40:04.556 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:40:04.556 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:40:04.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:40:04.556 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:40:04.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:40:04.556 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:40:04.556 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:40:04.556 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:40:04.556 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:40:04.556 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:40:04.557 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:40:04.557 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:40:04.557 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:40:04.557 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:40:04.557 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:40:04.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:40:04.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:40:04.557 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:40:04.557 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:40:04.557 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:40:04.558 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:40:04.558 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:40:04.558 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:40:04.558 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:40:04.558 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:40:04.558 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:40:04.558 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:40:04.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:40:04.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:40:04.558 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:40:04.558 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:40:04.558 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:40:04.559 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570004.330166_206.1", 2024-04-19T23:40:04.559 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:40:04.559 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:40:04.559 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:40:04.559 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:40:04.330166Z", 2024-04-19T23:40:04.559 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:40:04.559 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:40:04.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:40:04.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:40:04.559 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:40:04.559 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:40:04.560 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:40:04.560 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:40:04.560 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:40:04.560 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:40:04.560 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:40:04.560 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:40:04.560 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:40:04.560 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:40:04.560 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:40:04.560 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:40:04.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:40:04.561 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:40:04.561 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:40:04.561 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:40:04.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:40:04.561 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:40:04Z" 2024-04-19T23:40:04.561 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:40:04.562 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570004.330166_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:40:04.330166Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:40:04Z" } 2024-04-19T23:40:04.562 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570004.330166_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:40:04.330166Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:40:04.562 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570004.330166_206.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:40:04.562 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570004.330166_206.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:40:04.562 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:40:04.563 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:40:04.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:40:04.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:40:04.563 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:40:04.622 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:40:04.622 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:40:04.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:04.658+0000 7f1ed9910780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:40:04.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:04.660+0000 7f1ed9910780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:40:04.671 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:40:04.671 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:40:04.671 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:40:04.671 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:40:04.671 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:40:04.671 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:40:04.672 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:40:04.672 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:04.672 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:40:04.672 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:40:04.672 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:40:04.672 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:40:04.672 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:40:04.672 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:40:04.672 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:40:04.672 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:40:04.672 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:40:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:40:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:40:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:40:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:40:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:40:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:40:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:40:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:40:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:40:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:40:04.673 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:40:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:40:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:40:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:40:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:40:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:40:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:40:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:40:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713570004.326371_205.1", 2024-04-19T23:40:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:40:04.674 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:40:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:40:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:40:04.326371Z", 2024-04-19T23:40:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:40:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:40:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:40:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:40:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:40:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:40:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:40:04.675 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:40:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:40:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:40:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:40:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:40:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:40:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:40:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:40:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:40:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:40:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:40:04.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:04.677 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:40:04Z" 2024-04-19T23:40:04.677 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:40:04.677 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570004.326371_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:40:04.326371Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:40:04Z" } 2024-04-19T23:40:04.678 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570004.326371_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:40:04.326371Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:40:04.678 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570004.330166_206.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:40:04.678 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570004.326371_205.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:40:04.678 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1713570004.330166_206.1 target=1_1713570004.326371_205.1 2024-04-19T23:40:09.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:40:09.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:40:09.680 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:40:09.739 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:40:09.740 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:40:09.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:09.778+0000 7ffa15e21780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:40:09.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:09.779+0000 7ffa15e21780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:40:09.791 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:40:09.791 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:40:09.791 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:40:09.791 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:40:09.791 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:40:09.791 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:40:09.791 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:40:09.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:09.791 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:40:09.792 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:40:09.792 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:40:09.792 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:40:09.792 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:40:09.792 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:40:09.792 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:40:09.792 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:40:09.792 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:40:09.792 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:40:09.792 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:40:09.793 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:40:09.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:09.793 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:40:09.793 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:40:09.793 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:40:09.793 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:40:09.793 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:40:09.793 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:40:09.793 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:40:09.793 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:40:09.794 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:40:09.794 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:40:09.794 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:40:09.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:09.794 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:40:09.794 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:40:09.794 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:40:09.794 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:40:09.794 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713570004.330166_206.1", 2024-04-19T23:40:09.794 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:40:09.794 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:40:09.794 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:40:09.795 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:40:04.330166Z", 2024-04-19T23:40:09.795 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:40:09.795 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:40:09.795 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:09.795 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:40:09.795 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:40:09.795 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:40:09.795 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:40:09.795 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:40:09.795 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:40:09.796 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:40:09.796 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:40:09.796 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:40:09.796 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:40:09.796 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:40:09.796 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:40:09.796 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:40:09.796 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:09.796 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:40:09.796 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:40:09.796 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:40:09.796 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:40:09.797 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:40:09Z" 2024-04-19T23:40:09.797 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:40:09.798 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570004.330166_206.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:40:04.330166Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:40:09Z" } 2024-04-19T23:40:09.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570004.330166_206.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:40:04.330166Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:40:09.798 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570004.330166_206.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:40:09.798 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570004.330166_206.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:40:09.798 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:40:09.798 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:40:09.799 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:40:09.799 DEBUG:boto:Method: HEAD 2024-04-19T23:40:09.799 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:40:09.799 DEBUG:boto:Data: 2024-04-19T23:40:09.799 DEBUG:boto:Headers: {} 2024-04-19T23:40:09.799 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:09.799 DEBUG:boto:Port: 8000 2024-04-19T23:40:09.799 DEBUG:boto:Params: {} 2024-04-19T23:40:09.799 DEBUG:boto:Token: None 2024-04-19T23:40:09.800 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:40:09 GMT /aktlji-23/ 2024-04-19T23:40:09.800 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:s9+/K/sZkgQqTUqrzPc0+ECj0pk= 2024-04-19T23:40:09.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:40:09 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:s9+/K/sZkgQqTUqrzPc0+ECj0pk=', 'Content-Length': '0'} 2024-04-19T23:40:09.802 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bebbef4d4dd422b-00662300d9-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:09.803 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:40:09.803 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:40:09.803 DEBUG:boto:Method: PUT 2024-04-19T23:40:09.803 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:40:09.803 DEBUG:boto:Data: 2024-04-19T23:40:09.803 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:40:09.803 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:40:09.803 DEBUG:boto:Port: 8000 2024-04-19T23:40:09.803 DEBUG:boto:Params: {} 2024-04-19T23:40:09.803 DEBUG:boto:Token: None 2024-04-19T23:40:09.804 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:40:09 GMT /aktlji-23/obj 2024-04-19T23:40:09.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gnFn6mWgfjy5i16Pftc3kzeE4sw= 2024-04-19T23:40:09.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:40:09 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gnFn6mWgfjy5i16Pftc3kzeE4sw='} 2024-04-19T23:40:09.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-amz-request-id', 'tx000000b519363040090e0-00662300d9-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:40:09.816 DEBUG:rgw_multi.tests:created new version id=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:40:09.816 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-23 2024-04-19T23:40:09.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:09.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:09.816 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:40:09.882 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:40:09.883 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:40:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:40:09.926+0000 7f78eae11780 1 waiting for incremental sync to catch up: 2024-04-19T23:40:09.925 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000001.60.6, , ] 2024-04-19T23:40:09.926 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000003.63.13, , ] 2024-04-19T23:40:14.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:40:14.929+0000 7f78eae11780 1 waiting for incremental sync to catch up: 2024-04-19T23:40:14.927 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000001.60.6, , ] 2024-04-19T23:40:14.927 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000003.63.13, , ] 2024-04-19T23:40:19.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:40:19.930+0000 7f78eae11780 1 waiting for incremental sync to catch up: 2024-04-19T23:40:19.928 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000001.60.6, , ] 2024-04-19T23:40:19.928 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000003.63.13, , ] 2024-04-19T23:40:24.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:40:24.931+0000 7f78eae11780 1 bucket sync caught up with source: 2024-04-19T23:40:24.929 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000003.63.13, , ] 2024-04-19T23:40:24.930 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000003.63.13, , ] 2024-04-19T23:40:24.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:40:24.931+0000 7f78eae11780 0 bucket checkpoint complete 2024-04-19T23:40:24.947 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-23 2024-04-19T23:40:24.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:24.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:24.947 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:40:25.014 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:40:25.015 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:40:25.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:25.059+0000 7efc9a4db780 1 bucket sync caught up with source: 2024-04-19T23:40:25.057 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000003.63.13, , ] 2024-04-19T23:40:25.057 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.63.13, , ] 2024-04-19T23:40:25.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:25.059+0000 7efc9a4db780 0 bucket checkpoint complete 2024-04-19T23:40:25.072 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-23 2024-04-19T23:40:25.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:25.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:25.072 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:40:25.137 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:40:25.137 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:40:25.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:25.179+0000 7fdbcd92a780 1 waiting for incremental sync to catch up: 2024-04-19T23:40:25.177 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.79.6, , ] 2024-04-19T23:40:25.177 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.83.13, , ] 2024-04-19T23:40:30.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:30.181+0000 7fdbcd92a780 1 waiting for incremental sync to catch up: 2024-04-19T23:40:30.179 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.79.6, , ] 2024-04-19T23:40:30.179 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.83.13, , ] 2024-04-19T23:40:35.180 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:35.182+0000 7fdbcd92a780 1 waiting for incremental sync to catch up: 2024-04-19T23:40:35.180 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.79.6, , ] 2024-04-19T23:40:35.180 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.83.13, , ] 2024-04-19T23:40:40.181 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:40.183+0000 7fdbcd92a780 1 bucket sync caught up with source: 2024-04-19T23:40:40.181 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000004.85.10, , ] 2024-04-19T23:40:40.181 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.83.13, , ] 2024-04-19T23:40:40.181 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:40.183+0000 7fdbcd92a780 0 bucket checkpoint complete 2024-04-19T23:40:40.198 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-23 2024-04-19T23:40:40.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:40.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:40.199 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:40:40.264 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:40:40.264 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:40:40.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:40.308+0000 7fcf1ca9d780 1 waiting for incremental sync to catch up: 2024-04-19T23:40:40.306 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.79.6, , ] 2024-04-19T23:40:40.306 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000004.85.10, , ] 2024-04-19T23:40:45.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:45.310+0000 7fcf1ca9d780 1 bucket sync caught up with source: 2024-04-19T23:40:45.308 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000004.85.10, , ] 2024-04-19T23:40:45.308 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000004.85.10, , ] 2024-04-19T23:40:45.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:45.310+0000 7fcf1ca9d780 0 bucket checkpoint complete 2024-04-19T23:40:45.325 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-23 2024-04-19T23:40:45.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:45.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:45.326 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:40:45.390 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:40:45.391 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:40:45.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:45.434+0000 7f218f9dc780 1 waiting for incremental sync to catch up: 2024-04-19T23:40:45.433 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000003.76.13, , ] 2024-04-19T23:40:45.433 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000004.78.10, , ] 2024-04-19T23:40:50.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:50.437+0000 7f218f9dc780 1 waiting for incremental sync to catch up: 2024-04-19T23:40:50.434 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000003.76.13, , ] 2024-04-19T23:40:50.434 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000004.78.10, , ] 2024-04-19T23:40:55.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:55.439+0000 7f218f9dc780 1 bucket sync caught up with source: 2024-04-19T23:40:55.437 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000004.78.10, , ] 2024-04-19T23:40:55.437 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000004.78.10, , ] 2024-04-19T23:40:55.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:40:55.439+0000 7f218f9dc780 0 bucket checkpoint complete 2024-04-19T23:40:55.458 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-23 2024-04-19T23:40:55.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:55.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:40:55.459 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:40:55.523 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:40:55.524 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:40:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:40:55.568+0000 7f30290a6780 1 waiting for incremental sync to catch up: 2024-04-19T23:40:55.566 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000003.76.13, , ] 2024-04-19T23:40:55.566 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000004.78.10, , ] 2024-04-19T23:41:00.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:00.570+0000 7f30290a6780 1 bucket sync caught up with source: 2024-04-19T23:41:00.569 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000004.78.10, , ] 2024-04-19T23:41:00.570 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000004.78.10, , ] 2024-04-19T23:41:00.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:00.570+0000 7f30290a6780 0 bucket checkpoint complete 2024-04-19T23:41:00.587 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-23 zones={test-zone2, test-zone1} 2024-04-19T23:41:00.587 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.587 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.588 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.588 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:00.588 DEBUG:boto:Data: 2024-04-19T23:41:00.588 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.588 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.588 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.588 DEBUG:boto:Params: {} 2024-04-19T23:41:00.588 DEBUG:boto:Token: None 2024-04-19T23:41:00.589 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/ 2024-04-19T23:41:00.589 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YVp63SxIKW88j1btJnppv/Uq6vM= 2024-04-19T23:41:00.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YVp63SxIKW88j1btJnppv/Uq6vM=', 'Content-Length': '0'} 2024-04-19T23:41:00.591 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008be010b697549345-006623010c-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.592 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.592 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.592 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.592 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:00.592 DEBUG:boto:Data: 2024-04-19T23:41:00.592 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.592 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.592 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.592 DEBUG:boto:Params: {} 2024-04-19T23:41:00.592 DEBUG:boto:Token: None 2024-04-19T23:41:00.593 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/ 2024-04-19T23:41:00.593 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YVp63SxIKW88j1btJnppv/Uq6vM= 2024-04-19T23:41:00.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YVp63SxIKW88j1btJnppv/Uq6vM=', 'Content-Length': '0'} 2024-04-19T23:41:00.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fae8afb8993249fb-006623010c-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.596 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:41:00.596 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.596 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.596 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.596 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.596 DEBUG:boto:Method: GET 2024-04-19T23:41:00.596 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.596 DEBUG:boto:Data: 2024-04-19T23:41:00.596 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.597 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.597 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.597 DEBUG:boto:Params: {} 2024-04-19T23:41:00.597 DEBUG:boto:Token: None 2024-04-19T23:41:00.597 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/?versions 2024-04-19T23:41:00.597 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM= 2024-04-19T23:41:00.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM=', 'Content-Length': '0'} 2024-04-19T23:41:00.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6e9b1b9d5830b1c-006623010c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.600 DEBUG:boto:b'aktlji-23999falseobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUtrue2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:00.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:00.601 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:41:00.601 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.601 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.601 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.601 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.601 DEBUG:boto:Method: GET 2024-04-19T23:41:00.601 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.601 DEBUG:boto:Data: 2024-04-19T23:41:00.602 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.602 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.602 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.602 DEBUG:boto:Params: {} 2024-04-19T23:41:00.602 DEBUG:boto:Token: None 2024-04-19T23:41:00.602 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/?versions 2024-04-19T23:41:00.602 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM= 2024-04-19T23:41:00.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM=', 'Content-Length': '0'} 2024-04-19T23:41:00.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bfea6a7c6e53776-006623010c-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.650 DEBUG:boto:b'aktlji-23999falseobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUtrue2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:00.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:00.651 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.651 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.651 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.652 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.652 DEBUG:boto:Method: GET 2024-04-19T23:41:00.652 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.652 DEBUG:boto:Data: 2024-04-19T23:41:00.652 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.652 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.652 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.652 DEBUG:boto:Params: {} 2024-04-19T23:41:00.652 DEBUG:boto:Token: None 2024-04-19T23:41:00.652 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/?versions 2024-04-19T23:41:00.653 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM= 2024-04-19T23:41:00.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM=', 'Content-Length': '0'} 2024-04-19T23:41:00.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000880ecde4e7a60a4c-006623010c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.655 DEBUG:boto:b'aktlji-23999falseobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUtrue2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.656 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.656 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.656 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.656 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.656 DEBUG:boto:Method: GET 2024-04-19T23:41:00.656 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.656 DEBUG:boto:Data: 2024-04-19T23:41:00.656 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.656 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.657 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.657 DEBUG:boto:Params: {} 2024-04-19T23:41:00.657 DEBUG:boto:Token: None 2024-04-19T23:41:00.657 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/?versions 2024-04-19T23:41:00.657 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM= 2024-04-19T23:41:00.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM=', 'Content-Length': '0'} 2024-04-19T23:41:00.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000894db6fa751c72e-006623010c-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.660 DEBUG:boto:b'aktlji-23999falseobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUtrue2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:00.661 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.661 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.661 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.661 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.661 DEBUG:boto:Method: GET 2024-04-19T23:41:00.661 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.661 DEBUG:boto:Data: 2024-04-19T23:41:00.661 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.661 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.662 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.662 DEBUG:boto:Params: {} 2024-04-19T23:41:00.662 DEBUG:boto:Token: None 2024-04-19T23:41:00.662 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.662 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI= 2024-04-19T23:41:00.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI=', 'Content-Length': '0'} 2024-04-19T23:41:00.664 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000488ca8def3143e79-006623010c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.664 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.664 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.664 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.665 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.665 DEBUG:boto:Method: GET 2024-04-19T23:41:00.665 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.665 DEBUG:boto:Data: 2024-04-19T23:41:00.665 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.665 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.665 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.665 DEBUG:boto:Params: {} 2024-04-19T23:41:00.665 DEBUG:boto:Token: None 2024-04-19T23:41:00.665 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.665 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI= 2024-04-19T23:41:00.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI=', 'Content-Length': '0'} 2024-04-19T23:41:00.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012856e97a229d2f0-006623010c-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.668 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.668 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.668 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.668 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.668 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.668 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.668 DEBUG:boto:Data: 2024-04-19T23:41:00.669 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.669 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.669 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.669 DEBUG:boto:Params: {} 2024-04-19T23:41:00.669 DEBUG:boto:Token: None 2024-04-19T23:41:00.669 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.669 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Xs8fC+oduBtwMBnuLcP5HSoGGmA= 2024-04-19T23:41:00.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Xs8fC+oduBtwMBnuLcP5HSoGGmA=', 'Content-Length': '0'} 2024-04-19T23:41:00.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f0d7a259766c838c-006623010c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.673 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.673 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.674 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.674 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.674 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.674 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.674 DEBUG:boto:Data: 2024-04-19T23:41:00.674 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.674 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.674 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.674 DEBUG:boto:Params: {} 2024-04-19T23:41:00.674 DEBUG:boto:Token: None 2024-04-19T23:41:00.675 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.675 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Xs8fC+oduBtwMBnuLcP5HSoGGmA= 2024-04-19T23:41:00.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Xs8fC+oduBtwMBnuLcP5HSoGGmA=', 'Content-Length': '0'} 2024-04-19T23:41:00.676 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fcf24c931b3d72bb-006623010c-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:00.677 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.677 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.677 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.677 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.677 DEBUG:boto:Method: GET 2024-04-19T23:41:00.678 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.678 DEBUG:boto:Data: 2024-04-19T23:41:00.678 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.678 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.678 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.678 DEBUG:boto:Params: {} 2024-04-19T23:41:00.678 DEBUG:boto:Token: None 2024-04-19T23:41:00.678 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.678 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI= 2024-04-19T23:41:00.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI=', 'Content-Length': '0'} 2024-04-19T23:41:00.680 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa51507177af2834-006623010c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.680 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.681 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.681 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.681 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.681 DEBUG:boto:Method: GET 2024-04-19T23:41:00.681 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.681 DEBUG:boto:Data: 2024-04-19T23:41:00.681 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.681 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.681 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.681 DEBUG:boto:Params: {} 2024-04-19T23:41:00.681 DEBUG:boto:Token: None 2024-04-19T23:41:00.682 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.682 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI= 2024-04-19T23:41:00.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI=', 'Content-Length': '0'} 2024-04-19T23:41:00.684 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000030f15cbb2839348d-006623010c-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.684 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.684 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.684 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.684 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.684 DEBUG:boto:Method: GET 2024-04-19T23:41:00.684 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:00.685 DEBUG:boto:Data: 2024-04-19T23:41:00.685 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.685 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.685 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.685 DEBUG:boto:Params: {} 2024-04-19T23:41:00.685 DEBUG:boto:Token: None 2024-04-19T23:41:00.685 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/ 2024-04-19T23:41:00.685 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg= 2024-04-19T23:41:00.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg=', 'Content-Length': '0'} 2024-04-19T23:41:00.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000294199247e549faa-006623010c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.734 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.735 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.735 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.735 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.735 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.735 DEBUG:boto:Method: GET 2024-04-19T23:41:00.735 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:00.736 DEBUG:boto:Data: 2024-04-19T23:41:00.736 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.736 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.736 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.736 DEBUG:boto:Params: {} 2024-04-19T23:41:00.736 DEBUG:boto:Token: None 2024-04-19T23:41:00.736 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/ 2024-04-19T23:41:00.736 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg= 2024-04-19T23:41:00.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg=', 'Content-Length': '0'} 2024-04-19T23:41:00.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b91740de66ed66dc-006623010c-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.740 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.740 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.740 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.740 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.740 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:00.740 DEBUG:boto:Data: 2024-04-19T23:41:00.740 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.741 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.741 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.741 DEBUG:boto:Params: {} 2024-04-19T23:41:00.741 DEBUG:boto:Token: None 2024-04-19T23:41:00.741 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj 2024-04-19T23:41:00.741 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZWzuwpYl8aG0ZVdTUCqS94ohIUA= 2024-04-19T23:41:00.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZWzuwpYl8aG0ZVdTUCqS94ohIUA=', 'Content-Length': '0'} 2024-04-19T23:41:00.744 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000603b5b0230bdcb1-006623010c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.744 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.744 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.744 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.744 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:00.744 DEBUG:boto:Data: 2024-04-19T23:41:00.744 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.744 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.745 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.745 DEBUG:boto:Params: {} 2024-04-19T23:41:00.745 DEBUG:boto:Token: None 2024-04-19T23:41:00.745 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj 2024-04-19T23:41:00.745 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZWzuwpYl8aG0ZVdTUCqS94ohIUA= 2024-04-19T23:41:00.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZWzuwpYl8aG0ZVdTUCqS94ohIUA=', 'Content-Length': '0'} 2024-04-19T23:41:00.748 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005831a499659581ee-006623010c-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:00.748 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.748 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.748 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.748 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.748 DEBUG:boto:Method: GET 2024-04-19T23:41:00.748 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.749 DEBUG:boto:Data: 2024-04-19T23:41:00.749 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.749 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.749 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.749 DEBUG:boto:Params: {} 2024-04-19T23:41:00.749 DEBUG:boto:Token: None 2024-04-19T23:41:00.749 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.750 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI= 2024-04-19T23:41:00.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI=', 'Content-Length': '0'} 2024-04-19T23:41:00.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003b6958c214dfeff6-006623010c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.752 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.752 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.752 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.752 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.752 DEBUG:boto:Method: GET 2024-04-19T23:41:00.752 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.752 DEBUG:boto:Data: 2024-04-19T23:41:00.752 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.753 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.753 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.753 DEBUG:boto:Params: {} 2024-04-19T23:41:00.753 DEBUG:boto:Token: None 2024-04-19T23:41:00.753 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.753 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI= 2024-04-19T23:41:00.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI=', 'Content-Length': '0'} 2024-04-19T23:41:00.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000659baf1c099da28f-006623010c-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.756 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.756 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.756 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.756 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.756 DEBUG:boto:Method: GET 2024-04-19T23:41:00.756 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:00.756 DEBUG:boto:Data: 2024-04-19T23:41:00.756 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.756 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.756 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.756 DEBUG:boto:Params: {} 2024-04-19T23:41:00.757 DEBUG:boto:Token: None 2024-04-19T23:41:00.757 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/ 2024-04-19T23:41:00.757 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg= 2024-04-19T23:41:00.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg=', 'Content-Length': '0'} 2024-04-19T23:41:00.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035687b6f675e76cf-006623010c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.759 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.760 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.760 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.760 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.760 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.760 DEBUG:boto:Method: GET 2024-04-19T23:41:00.760 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:00.760 DEBUG:boto:Data: 2024-04-19T23:41:00.760 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.760 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.761 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.761 DEBUG:boto:Params: {} 2024-04-19T23:41:00.761 DEBUG:boto:Token: None 2024-04-19T23:41:00.761 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/ 2024-04-19T23:41:00.761 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg= 2024-04-19T23:41:00.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg=', 'Content-Length': '0'} 2024-04-19T23:41:00.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af1756168dcf26ca-006623010c-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.806 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:00.807 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.807 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.807 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:00.808 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:00.808 DEBUG:boto:Method: GET 2024-04-19T23:41:00.808 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:00.808 DEBUG:boto:Data: 2024-04-19T23:41:00.808 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.808 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.808 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.808 DEBUG:boto:Params: {} 2024-04-19T23:41:00.808 DEBUG:boto:Token: None 2024-04-19T23:41:00.808 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:00.809 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nYZ1mK77ZY5bUdcuHzIQNqlgOQI= 2024-04-19T23:41:00.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nYZ1mK77ZY5bUdcuHzIQNqlgOQI=', 'Content-Length': '0'} 2024-04-19T23:41:00.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000676d5339d7322d73-006623010c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.811 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.812 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.812 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:00.812 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:00.812 DEBUG:boto:Method: GET 2024-04-19T23:41:00.812 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:00.812 DEBUG:boto:Data: 2024-04-19T23:41:00.812 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.812 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.812 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.812 DEBUG:boto:Params: {} 2024-04-19T23:41:00.813 DEBUG:boto:Token: None 2024-04-19T23:41:00.813 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:00.813 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nYZ1mK77ZY5bUdcuHzIQNqlgOQI= 2024-04-19T23:41:00.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nYZ1mK77ZY5bUdcuHzIQNqlgOQI=', 'Content-Length': '0'} 2024-04-19T23:41:00.815 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e5476c9e533dfcfc-006623010c-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.815 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.815 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.816 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:00.816 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:00.816 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.816 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:00.816 DEBUG:boto:Data: 2024-04-19T23:41:00.816 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.816 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.816 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.816 DEBUG:boto:Params: {} 2024-04-19T23:41:00.816 DEBUG:boto:Token: None 2024-04-19T23:41:00.817 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:00.817 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:znpRr1nu6WjI1rVVBR4AjO8tO5E= 2024-04-19T23:41:00.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:znpRr1nu6WjI1rVVBR4AjO8tO5E=', 'Content-Length': '0'} 2024-04-19T23:41:00.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000accb08aa58227f37-006623010c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.819 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.819 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.819 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:00.819 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:00.819 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.819 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:00.819 DEBUG:boto:Data: 2024-04-19T23:41:00.819 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.820 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.820 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.820 DEBUG:boto:Params: {} 2024-04-19T23:41:00.820 DEBUG:boto:Token: None 2024-04-19T23:41:00.820 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:00.820 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:znpRr1nu6WjI1rVVBR4AjO8tO5E= 2024-04-19T23:41:00.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:znpRr1nu6WjI1rVVBR4AjO8tO5E=', 'Content-Length': '0'} 2024-04-19T23:41:00.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab65466d5e532126-006623010c-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:00.822 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.822 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.822 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:00.822 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:00.823 DEBUG:boto:Method: GET 2024-04-19T23:41:00.823 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:00.823 DEBUG:boto:Data: 2024-04-19T23:41:00.823 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.823 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.823 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.823 DEBUG:boto:Params: {} 2024-04-19T23:41:00.823 DEBUG:boto:Token: None 2024-04-19T23:41:00.823 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:00.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nYZ1mK77ZY5bUdcuHzIQNqlgOQI= 2024-04-19T23:41:00.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nYZ1mK77ZY5bUdcuHzIQNqlgOQI=', 'Content-Length': '0'} 2024-04-19T23:41:00.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed18eee5cb78442c-006623010c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.826 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.826 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.826 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:00.826 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:00.826 DEBUG:boto:Method: GET 2024-04-19T23:41:00.826 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:00.826 DEBUG:boto:Data: 2024-04-19T23:41:00.826 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.826 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.826 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.826 DEBUG:boto:Params: {} 2024-04-19T23:41:00.827 DEBUG:boto:Token: None 2024-04-19T23:41:00.827 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:00.827 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nYZ1mK77ZY5bUdcuHzIQNqlgOQI= 2024-04-19T23:41:00.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nYZ1mK77ZY5bUdcuHzIQNqlgOQI=', 'Content-Length': '0'} 2024-04-19T23:41:00.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000041c5ec30cb146194-006623010c-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.829 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.829 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.829 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.829 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.829 DEBUG:boto:Method: GET 2024-04-19T23:41:00.829 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:00.829 DEBUG:boto:Data: 2024-04-19T23:41:00.830 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.830 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.830 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.830 DEBUG:boto:Params: {} 2024-04-19T23:41:00.830 DEBUG:boto:Token: None 2024-04-19T23:41:00.830 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/ 2024-04-19T23:41:00.830 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg= 2024-04-19T23:41:00.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg=', 'Content-Length': '0'} 2024-04-19T23:41:00.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba3dab62e489ffd7-006623010c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.874 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.875 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.875 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.875 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.875 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.875 DEBUG:boto:Method: GET 2024-04-19T23:41:00.875 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:00.876 DEBUG:boto:Data: 2024-04-19T23:41:00.876 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.876 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.876 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.876 DEBUG:boto:Params: {} 2024-04-19T23:41:00.876 DEBUG:boto:Token: None 2024-04-19T23:41:00.876 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/ 2024-04-19T23:41:00.877 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg= 2024-04-19T23:41:00.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg=', 'Content-Length': '0'} 2024-04-19T23:41:00.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d74da01b7561c7c8-006623010c-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.879 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.880 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.880 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.880 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.880 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:00.880 DEBUG:boto:Data: 2024-04-19T23:41:00.880 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.880 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.880 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.880 DEBUG:boto:Params: {} 2024-04-19T23:41:00.881 DEBUG:boto:Token: None 2024-04-19T23:41:00.881 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj 2024-04-19T23:41:00.881 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZWzuwpYl8aG0ZVdTUCqS94ohIUA= 2024-04-19T23:41:00.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZWzuwpYl8aG0ZVdTUCqS94ohIUA=', 'Content-Length': '0'} 2024-04-19T23:41:00.883 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d4cec5e5532151d-006623010c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.884 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.884 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.884 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.884 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:00.884 DEBUG:boto:Data: 2024-04-19T23:41:00.884 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.885 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.885 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.885 DEBUG:boto:Params: {} 2024-04-19T23:41:00.885 DEBUG:boto:Token: None 2024-04-19T23:41:00.885 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj 2024-04-19T23:41:00.885 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ZWzuwpYl8aG0ZVdTUCqS94ohIUA= 2024-04-19T23:41:00.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZWzuwpYl8aG0ZVdTUCqS94ohIUA=', 'Content-Length': '0'} 2024-04-19T23:41:00.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036bdc28eed6a25c8-006623010c-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:00.888 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.888 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.889 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.889 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.889 DEBUG:boto:Method: GET 2024-04-19T23:41:00.889 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.889 DEBUG:boto:Data: 2024-04-19T23:41:00.889 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.889 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.889 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.889 DEBUG:boto:Params: {} 2024-04-19T23:41:00.889 DEBUG:boto:Token: None 2024-04-19T23:41:00.890 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.890 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI= 2024-04-19T23:41:00.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI=', 'Content-Length': '0'} 2024-04-19T23:41:00.892 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d54678c4b6385e8-006623010c-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.893 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.893 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.893 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.893 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.893 DEBUG:boto:Method: GET 2024-04-19T23:41:00.893 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.893 DEBUG:boto:Data: 2024-04-19T23:41:00.893 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.893 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.894 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.894 DEBUG:boto:Params: {} 2024-04-19T23:41:00.894 DEBUG:boto:Token: None 2024-04-19T23:41:00.894 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.894 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI= 2024-04-19T23:41:00.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI=', 'Content-Length': '0'} 2024-04-19T23:41:00.897 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056ef46fb127e5552-006623010c-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.897 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.897 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.898 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.898 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.898 DEBUG:boto:Method: GET 2024-04-19T23:41:00.898 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:00.898 DEBUG:boto:Data: 2024-04-19T23:41:00.899 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.899 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:00.900 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.900 DEBUG:boto:Params: {} 2024-04-19T23:41:00.900 DEBUG:boto:Token: None 2024-04-19T23:41:00.901 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/ 2024-04-19T23:41:00.901 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg= 2024-04-19T23:41:00.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg=', 'Content-Length': '0'} 2024-04-19T23:41:00.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd70bf1f14677f2d-006623010c-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.904 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.905 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.905 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.905 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.905 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:00.905 DEBUG:boto:Method: GET 2024-04-19T23:41:00.906 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:00.906 DEBUG:boto:Data: 2024-04-19T23:41:00.906 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.906 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.906 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.907 DEBUG:boto:Params: {} 2024-04-19T23:41:00.907 DEBUG:boto:Token: None 2024-04-19T23:41:00.907 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/ 2024-04-19T23:41:00.907 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg= 2024-04-19T23:41:00.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5yvUYTIibl/WlCBRz/yHrq5Twlg=', 'Content-Length': '0'} 2024-04-19T23:41:00.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed76f5a48419f4c3-006623010c-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.950 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.951 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-23 zones={test-zone2, test-zone1} 2024-04-19T23:41:00.951 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-23 zones={test-zone3, test-zone1} 2024-04-19T23:41:00.952 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.952 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.952 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.952 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:00.952 DEBUG:boto:Data: 2024-04-19T23:41:00.953 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.953 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:00.953 DEBUG:boto:Port: 8001 2024-04-19T23:41:00.953 DEBUG:boto:Params: {} 2024-04-19T23:41:00.953 DEBUG:boto:Token: None 2024-04-19T23:41:00.954 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/ 2024-04-19T23:41:00.954 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YVp63SxIKW88j1btJnppv/Uq6vM= 2024-04-19T23:41:00.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YVp63SxIKW88j1btJnppv/Uq6vM=', 'Content-Length': '0'} 2024-04-19T23:41:00.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027616731b9f6d47f-006623010c-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.958 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.958 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.958 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.958 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:00.959 DEBUG:boto:Data: 2024-04-19T23:41:00.959 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.959 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.959 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.959 DEBUG:boto:Params: {} 2024-04-19T23:41:00.960 DEBUG:boto:Token: None 2024-04-19T23:41:00.960 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/ 2024-04-19T23:41:00.960 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YVp63SxIKW88j1btJnppv/Uq6vM= 2024-04-19T23:41:00.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YVp63SxIKW88j1btJnppv/Uq6vM=', 'Content-Length': '0'} 2024-04-19T23:41:00.963 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000817293bb0cba7a5-006623010c-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.963 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:41:00.964 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.964 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.964 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.964 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.964 DEBUG:boto:Method: GET 2024-04-19T23:41:00.965 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.965 DEBUG:boto:Data: 2024-04-19T23:41:00.965 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.965 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:00.965 DEBUG:boto:Port: 8001 2024-04-19T23:41:00.965 DEBUG:boto:Params: {} 2024-04-19T23:41:00.965 DEBUG:boto:Token: None 2024-04-19T23:41:00.966 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/?versions 2024-04-19T23:41:00.966 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM= 2024-04-19T23:41:00.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM=', 'Content-Length': '0'} 2024-04-19T23:41:00.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aec12c855da1ef88-006623010c-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.969 DEBUG:boto:b'aktlji-23999falseobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUtrue2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:00.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:00.970 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:41:00.970 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.970 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.970 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.970 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.970 DEBUG:boto:Method: GET 2024-04-19T23:41:00.970 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.970 DEBUG:boto:Data: 2024-04-19T23:41:00.970 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.971 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.971 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.971 DEBUG:boto:Params: {} 2024-04-19T23:41:00.971 DEBUG:boto:Token: None 2024-04-19T23:41:00.971 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/?versions 2024-04-19T23:41:00.971 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM= 2024-04-19T23:41:00.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM=', 'Content-Length': '0'} 2024-04-19T23:41:00.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000343ea62b2926cf8-006623010c-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.974 DEBUG:boto:b'aktlji-23999falseobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUtrue2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:00.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:00.974 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.975 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.975 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.975 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.975 DEBUG:boto:Method: GET 2024-04-19T23:41:00.975 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.975 DEBUG:boto:Data: 2024-04-19T23:41:00.975 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.975 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:00.975 DEBUG:boto:Port: 8001 2024-04-19T23:41:00.976 DEBUG:boto:Params: {} 2024-04-19T23:41:00.976 DEBUG:boto:Token: None 2024-04-19T23:41:00.976 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/?versions 2024-04-19T23:41:00.976 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM= 2024-04-19T23:41:00.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM=', 'Content-Length': '0'} 2024-04-19T23:41:00.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005238de35de27388b-006623010c-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.978 DEBUG:boto:b'aktlji-23999falseobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUtrue2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.981 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:00.981 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:00.981 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.982 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.982 DEBUG:boto:Method: GET 2024-04-19T23:41:00.982 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:00.982 DEBUG:boto:Data: 2024-04-19T23:41:00.982 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.982 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.982 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.982 DEBUG:boto:Params: {} 2024-04-19T23:41:00.982 DEBUG:boto:Token: None 2024-04-19T23:41:00.982 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/?versions 2024-04-19T23:41:00.983 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM= 2024-04-19T23:41:00.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:38kpTnYwF4HE+bX+CupnXftO6PM=', 'Content-Length': '0'} 2024-04-19T23:41:00.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077b2e6f325e71fe9-006623010c-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.985 DEBUG:boto:b'aktlji-23999falseobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUtrue2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:00.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:00.986 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.986 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.986 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.986 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.986 DEBUG:boto:Method: GET 2024-04-19T23:41:00.986 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.986 DEBUG:boto:Data: 2024-04-19T23:41:00.987 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.987 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:00.987 DEBUG:boto:Port: 8001 2024-04-19T23:41:00.987 DEBUG:boto:Params: {} 2024-04-19T23:41:00.987 DEBUG:boto:Token: None 2024-04-19T23:41:00.987 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.987 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI= 2024-04-19T23:41:00.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI=', 'Content-Length': '0'} 2024-04-19T23:41:00.989 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000adb7d9d34b9ef104-006623010c-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.990 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.990 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.990 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.990 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.990 DEBUG:boto:Method: GET 2024-04-19T23:41:00.990 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.990 DEBUG:boto:Data: 2024-04-19T23:41:00.990 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.990 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.990 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.991 DEBUG:boto:Params: {} 2024-04-19T23:41:00.991 DEBUG:boto:Token: None 2024-04-19T23:41:00.991 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.991 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI= 2024-04-19T23:41:00.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1mCBkPZj5PfaINzEx8WYZ9499TI=', 'Content-Length': '0'} 2024-04-19T23:41:00.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000444e17e9b4d0b2a7-006623010c-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.993 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.993 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.993 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.993 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.993 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.993 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.994 DEBUG:boto:Data: 2024-04-19T23:41:00.994 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.994 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:00.994 DEBUG:boto:Port: 8001 2024-04-19T23:41:00.994 DEBUG:boto:Params: {} 2024-04-19T23:41:00.994 DEBUG:boto:Token: None 2024-04-19T23:41:00.994 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.994 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Xs8fC+oduBtwMBnuLcP5HSoGGmA= 2024-04-19T23:41:00.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Xs8fC+oduBtwMBnuLcP5HSoGGmA=', 'Content-Length': '0'} 2024-04-19T23:41:00.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003af8f43770efbe54-006623010c-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:00.996 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:00.997 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:00.997 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.997 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.997 DEBUG:boto:Method: HEAD 2024-04-19T23:41:00.997 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.997 DEBUG:boto:Data: 2024-04-19T23:41:00.997 DEBUG:boto:Headers: {} 2024-04-19T23:41:00.997 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:00.997 DEBUG:boto:Port: 8000 2024-04-19T23:41:00.997 DEBUG:boto:Params: {} 2024-04-19T23:41:00.997 DEBUG:boto:Token: None 2024-04-19T23:41:00.998 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:00 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:00.998 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Xs8fC+oduBtwMBnuLcP5HSoGGmA= 2024-04-19T23:41:00.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Xs8fC+oduBtwMBnuLcP5HSoGGmA=', 'Content-Length': '0'} 2024-04-19T23:41:00.999 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2fa6701bf1b45e3-006623010d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:01.000 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.000 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.000 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.000 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.000 DEBUG:boto:Method: GET 2024-04-19T23:41:01.000 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.000 DEBUG:boto:Data: 2024-04-19T23:41:01.000 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.000 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.000 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.001 DEBUG:boto:Params: {} 2024-04-19T23:41:01.001 DEBUG:boto:Token: None 2024-04-19T23:41:01.001 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.001 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.003 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000011811f7a2271ad-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.003 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.003 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.003 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.004 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.004 DEBUG:boto:Method: GET 2024-04-19T23:41:01.004 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.004 DEBUG:boto:Data: 2024-04-19T23:41:01.004 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.004 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.004 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.004 DEBUG:boto:Params: {} 2024-04-19T23:41:01.004 DEBUG:boto:Token: None 2024-04-19T23:41:01.005 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.005 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.007 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed2732571c1ad78a-006623010d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.007 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.007 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.007 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.007 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.007 DEBUG:boto:Method: GET 2024-04-19T23:41:01.007 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.008 DEBUG:boto:Data: 2024-04-19T23:41:01.008 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.008 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.008 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.008 DEBUG:boto:Params: {} 2024-04-19T23:41:01.008 DEBUG:boto:Token: None 2024-04-19T23:41:01.008 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.008 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3dff46438cf2a4f-006623010d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.011 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.012 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.012 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.012 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.012 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.012 DEBUG:boto:Method: GET 2024-04-19T23:41:01.012 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.012 DEBUG:boto:Data: 2024-04-19T23:41:01.012 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.012 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.013 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.013 DEBUG:boto:Params: {} 2024-04-19T23:41:01.013 DEBUG:boto:Token: None 2024-04-19T23:41:01.013 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.013 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d34248f1fc9662e-006623010d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.062 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.063 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.063 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.063 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.063 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:01.063 DEBUG:boto:Data: 2024-04-19T23:41:01.063 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.063 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.064 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.064 DEBUG:boto:Params: {} 2024-04-19T23:41:01.064 DEBUG:boto:Token: None 2024-04-19T23:41:01.064 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj 2024-04-19T23:41:01.064 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI= 2024-04-19T23:41:01.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI=', 'Content-Length': '0'} 2024-04-19T23:41:01.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a590244f1e41a2d9-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.067 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.067 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.067 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.067 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:01.067 DEBUG:boto:Data: 2024-04-19T23:41:01.067 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.067 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.067 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.067 DEBUG:boto:Params: {} 2024-04-19T23:41:01.068 DEBUG:boto:Token: None 2024-04-19T23:41:01.068 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj 2024-04-19T23:41:01.068 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI= 2024-04-19T23:41:01.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI=', 'Content-Length': '0'} 2024-04-19T23:41:01.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d79083d34349ca1-006623010d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:01.070 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.071 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.071 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.071 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.071 DEBUG:boto:Method: GET 2024-04-19T23:41:01.071 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.071 DEBUG:boto:Data: 2024-04-19T23:41:01.071 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.071 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.071 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.071 DEBUG:boto:Params: {} 2024-04-19T23:41:01.071 DEBUG:boto:Token: None 2024-04-19T23:41:01.072 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.072 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003e0475c7b59026b9-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.074 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.074 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.074 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.074 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.074 DEBUG:boto:Method: GET 2024-04-19T23:41:01.074 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.074 DEBUG:boto:Data: 2024-04-19T23:41:01.074 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.075 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.075 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.075 DEBUG:boto:Params: {} 2024-04-19T23:41:01.075 DEBUG:boto:Token: None 2024-04-19T23:41:01.075 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.075 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.077 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003f372469fd9388c7-006623010d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.077 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.077 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.078 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.078 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.078 DEBUG:boto:Method: GET 2024-04-19T23:41:01.078 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.078 DEBUG:boto:Data: 2024-04-19T23:41:01.078 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.078 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.078 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.078 DEBUG:boto:Params: {} 2024-04-19T23:41:01.079 DEBUG:boto:Token: None 2024-04-19T23:41:01.079 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.079 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acb85ec7c5ce3a85-006623010d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.126 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.127 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.127 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.127 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.127 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.127 DEBUG:boto:Method: GET 2024-04-19T23:41:01.127 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.127 DEBUG:boto:Data: 2024-04-19T23:41:01.128 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.128 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.128 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.128 DEBUG:boto:Params: {} 2024-04-19T23:41:01.128 DEBUG:boto:Token: None 2024-04-19T23:41:01.128 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.128 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b285271bbd836666-006623010d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.131 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:01.132 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.132 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.132 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.132 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.132 DEBUG:boto:Method: GET 2024-04-19T23:41:01.132 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.132 DEBUG:boto:Data: 2024-04-19T23:41:01.132 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.132 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.132 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.132 DEBUG:boto:Params: {} 2024-04-19T23:41:01.133 DEBUG:boto:Token: None 2024-04-19T23:41:01.133 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.133 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY= 2024-04-19T23:41:01.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY=', 'Content-Length': '0'} 2024-04-19T23:41:01.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001659946accd80081-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.135 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.135 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.136 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.136 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.136 DEBUG:boto:Method: GET 2024-04-19T23:41:01.136 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.136 DEBUG:boto:Data: 2024-04-19T23:41:01.136 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.136 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.136 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.136 DEBUG:boto:Params: {} 2024-04-19T23:41:01.136 DEBUG:boto:Token: None 2024-04-19T23:41:01.137 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.137 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY= 2024-04-19T23:41:01.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY=', 'Content-Length': '0'} 2024-04-19T23:41:01.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008071807d35916a6c-006623010d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.139 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.139 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.139 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.139 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.139 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.139 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.139 DEBUG:boto:Data: 2024-04-19T23:41:01.139 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.139 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.139 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.139 DEBUG:boto:Params: {} 2024-04-19T23:41:01.140 DEBUG:boto:Token: None 2024-04-19T23:41:01.140 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.140 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SXMFIGltbqIKSdTA2965KVDnYmw= 2024-04-19T23:41:01.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SXMFIGltbqIKSdTA2965KVDnYmw=', 'Content-Length': '0'} 2024-04-19T23:41:01.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009cb1f906dde2fd4b-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.142 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.142 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.142 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.142 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.142 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.143 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.143 DEBUG:boto:Data: 2024-04-19T23:41:01.143 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.143 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.143 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.143 DEBUG:boto:Params: {} 2024-04-19T23:41:01.143 DEBUG:boto:Token: None 2024-04-19T23:41:01.143 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.143 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SXMFIGltbqIKSdTA2965KVDnYmw= 2024-04-19T23:41:01.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SXMFIGltbqIKSdTA2965KVDnYmw=', 'Content-Length': '0'} 2024-04-19T23:41:01.145 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000388acd47df2bec1b-006623010d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:01.145 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.146 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.146 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.146 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.146 DEBUG:boto:Method: GET 2024-04-19T23:41:01.146 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.146 DEBUG:boto:Data: 2024-04-19T23:41:01.146 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.146 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.146 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.146 DEBUG:boto:Params: {} 2024-04-19T23:41:01.147 DEBUG:boto:Token: None 2024-04-19T23:41:01.147 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.147 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY= 2024-04-19T23:41:01.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY=', 'Content-Length': '0'} 2024-04-19T23:41:01.149 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007340727ccb38cd98-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.149 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.149 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.149 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.149 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.149 DEBUG:boto:Method: GET 2024-04-19T23:41:01.149 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.149 DEBUG:boto:Data: 2024-04-19T23:41:01.150 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.150 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.150 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.150 DEBUG:boto:Params: {} 2024-04-19T23:41:01.150 DEBUG:boto:Token: None 2024-04-19T23:41:01.150 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.150 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY= 2024-04-19T23:41:01.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY=', 'Content-Length': '0'} 2024-04-19T23:41:01.152 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ea84c11a3933e234-006623010d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.152 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.152 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.153 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.153 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.153 DEBUG:boto:Method: GET 2024-04-19T23:41:01.153 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.153 DEBUG:boto:Data: 2024-04-19T23:41:01.153 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.153 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.153 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.153 DEBUG:boto:Params: {} 2024-04-19T23:41:01.153 DEBUG:boto:Token: None 2024-04-19T23:41:01.154 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.154 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097e52fffafd24759-006623010d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.156 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.157 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.157 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.157 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.157 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.157 DEBUG:boto:Method: GET 2024-04-19T23:41:01.157 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.157 DEBUG:boto:Data: 2024-04-19T23:41:01.157 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.157 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.158 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.158 DEBUG:boto:Params: {} 2024-04-19T23:41:01.158 DEBUG:boto:Token: None 2024-04-19T23:41:01.158 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.158 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c29d3f8cb9071fc-006623010d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.202 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.203 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.203 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.203 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.203 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:01.203 DEBUG:boto:Data: 2024-04-19T23:41:01.203 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.203 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.204 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.204 DEBUG:boto:Params: {} 2024-04-19T23:41:01.204 DEBUG:boto:Token: None 2024-04-19T23:41:01.204 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj 2024-04-19T23:41:01.204 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI= 2024-04-19T23:41:01.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI=', 'Content-Length': '0'} 2024-04-19T23:41:01.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e1db3dc1f9fdd9e2-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.207 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.207 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.207 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.207 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:01.207 DEBUG:boto:Data: 2024-04-19T23:41:01.207 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.207 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.208 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.208 DEBUG:boto:Params: {} 2024-04-19T23:41:01.208 DEBUG:boto:Token: None 2024-04-19T23:41:01.208 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj 2024-04-19T23:41:01.208 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI= 2024-04-19T23:41:01.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI=', 'Content-Length': '0'} 2024-04-19T23:41:01.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a978029e660a9856-006623010d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:01.211 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.211 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.211 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.211 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.211 DEBUG:boto:Method: GET 2024-04-19T23:41:01.211 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.211 DEBUG:boto:Data: 2024-04-19T23:41:01.211 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.211 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.211 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.212 DEBUG:boto:Params: {} 2024-04-19T23:41:01.212 DEBUG:boto:Token: None 2024-04-19T23:41:01.212 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.212 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000088f9e4d078552dd0-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.214 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.215 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.215 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.215 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.215 DEBUG:boto:Method: GET 2024-04-19T23:41:01.215 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.215 DEBUG:boto:Data: 2024-04-19T23:41:01.215 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.215 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.215 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.215 DEBUG:boto:Params: {} 2024-04-19T23:41:01.215 DEBUG:boto:Token: None 2024-04-19T23:41:01.216 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.216 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000528a6c9ab197cf6b-006623010d-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.218 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.218 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.218 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.218 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.218 DEBUG:boto:Method: GET 2024-04-19T23:41:01.218 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.218 DEBUG:boto:Data: 2024-04-19T23:41:01.218 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.219 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.219 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.219 DEBUG:boto:Params: {} 2024-04-19T23:41:01.219 DEBUG:boto:Token: None 2024-04-19T23:41:01.219 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.219 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003733acd4e661b07d-006623010d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.262 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.263 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.263 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.263 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.263 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.263 DEBUG:boto:Method: GET 2024-04-19T23:41:01.263 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.264 DEBUG:boto:Data: 2024-04-19T23:41:01.264 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.264 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.264 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.264 DEBUG:boto:Params: {} 2024-04-19T23:41:01.264 DEBUG:boto:Token: None 2024-04-19T23:41:01.264 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.264 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000261e3655fa8a6141-006623010d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.267 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.268 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-23 zones={test-zone3, test-zone1} 2024-04-19T23:41:01.268 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-23 zones={test-zone3, test-zone2} 2024-04-19T23:41:01.268 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.268 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.268 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.268 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:01.268 DEBUG:boto:Data: 2024-04-19T23:41:01.268 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.268 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.269 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.269 DEBUG:boto:Params: {} 2024-04-19T23:41:01.269 DEBUG:boto:Token: None 2024-04-19T23:41:01.269 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.269 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IbY8fPhbXzjJ1rIhlbQ5kyUFQvk= 2024-04-19T23:41:01.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IbY8fPhbXzjJ1rIhlbQ5kyUFQvk=', 'Content-Length': '0'} 2024-04-19T23:41:01.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a58681f1ad7fd58-006623010d-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.272 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.272 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.272 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.272 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:01.272 DEBUG:boto:Data: 2024-04-19T23:41:01.272 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.273 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.273 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.273 DEBUG:boto:Params: {} 2024-04-19T23:41:01.273 DEBUG:boto:Token: None 2024-04-19T23:41:01.273 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.273 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IbY8fPhbXzjJ1rIhlbQ5kyUFQvk= 2024-04-19T23:41:01.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IbY8fPhbXzjJ1rIhlbQ5kyUFQvk=', 'Content-Length': '0'} 2024-04-19T23:41:01.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099a7457382e69709-006623010d-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.276 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:41:01.276 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.276 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.276 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:01.276 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:01.276 DEBUG:boto:Method: GET 2024-04-19T23:41:01.277 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:01.277 DEBUG:boto:Data: 2024-04-19T23:41:01.277 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.277 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.277 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.277 DEBUG:boto:Params: {} 2024-04-19T23:41:01.277 DEBUG:boto:Token: None 2024-04-19T23:41:01.277 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/?versions 2024-04-19T23:41:01.277 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:60p6VDGP5tQ2D2HX8BHby4wCCAY= 2024-04-19T23:41:01.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:60p6VDGP5tQ2D2HX8BHby4wCCAY=', 'Content-Length': '0'} 2024-04-19T23:41:01.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a416e43b256adf8e-006623010d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.280 DEBUG:boto:b'aktlji-23999falseobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUtrue2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.281 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:01.281 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:01.281 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:41:01.281 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.281 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.281 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:01.281 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:01.281 DEBUG:boto:Method: GET 2024-04-19T23:41:01.281 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:01.282 DEBUG:boto:Data: 2024-04-19T23:41:01.282 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.282 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.282 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.282 DEBUG:boto:Params: {} 2024-04-19T23:41:01.282 DEBUG:boto:Token: None 2024-04-19T23:41:01.282 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/?versions 2024-04-19T23:41:01.283 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:60p6VDGP5tQ2D2HX8BHby4wCCAY= 2024-04-19T23:41:01.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:60p6VDGP5tQ2D2HX8BHby4wCCAY=', 'Content-Length': '0'} 2024-04-19T23:41:01.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cf713630e594db0-006623010d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.326 DEBUG:boto:b'aktlji-23999falseobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUtrue2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:01.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:01.327 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.327 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.327 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:01.327 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:01.327 DEBUG:boto:Method: GET 2024-04-19T23:41:01.328 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:01.328 DEBUG:boto:Data: 2024-04-19T23:41:01.328 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.328 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.328 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.328 DEBUG:boto:Params: {} 2024-04-19T23:41:01.328 DEBUG:boto:Token: None 2024-04-19T23:41:01.328 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/?versions 2024-04-19T23:41:01.328 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:60p6VDGP5tQ2D2HX8BHby4wCCAY= 2024-04-19T23:41:01.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:60p6VDGP5tQ2D2HX8BHby4wCCAY=', 'Content-Length': '0'} 2024-04-19T23:41:01.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000039d2de279ea7054-006623010d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.378 DEBUG:boto:b'aktlji-23999falseobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUtrue2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.379 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.379 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.379 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:01.379 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:01.380 DEBUG:boto:Method: GET 2024-04-19T23:41:01.380 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:01.380 DEBUG:boto:Data: 2024-04-19T23:41:01.380 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.380 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.380 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.380 DEBUG:boto:Params: {} 2024-04-19T23:41:01.380 DEBUG:boto:Token: None 2024-04-19T23:41:01.380 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/?versions 2024-04-19T23:41:01.381 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:60p6VDGP5tQ2D2HX8BHby4wCCAY= 2024-04-19T23:41:01.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:60p6VDGP5tQ2D2HX8BHby4wCCAY=', 'Content-Length': '0'} 2024-04-19T23:41:01.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009274186f3e12807f-006623010d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.383 DEBUG:boto:b'aktlji-23999falseobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUtrue2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-19T23:39:37.978Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:01.384 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.384 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.384 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.384 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.384 DEBUG:boto:Method: GET 2024-04-19T23:41:01.384 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.384 DEBUG:boto:Data: 2024-04-19T23:41:01.385 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.385 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.385 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.385 DEBUG:boto:Params: {} 2024-04-19T23:41:01.385 DEBUG:boto:Token: None 2024-04-19T23:41:01.385 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.385 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.388 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aca7f467daa09c5a-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.388 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.388 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.388 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.389 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.389 DEBUG:boto:Method: GET 2024-04-19T23:41:01.389 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.389 DEBUG:boto:Data: 2024-04-19T23:41:01.389 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.389 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.389 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.389 DEBUG:boto:Params: {} 2024-04-19T23:41:01.390 DEBUG:boto:Token: None 2024-04-19T23:41:01.390 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.390 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.393 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c0d080706e1ae82-006623010d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.394 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.394 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.394 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.394 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.394 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.395 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.395 DEBUG:boto:Data: 2024-04-19T23:41:01.395 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.395 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.395 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.395 DEBUG:boto:Params: {} 2024-04-19T23:41:01.395 DEBUG:boto:Token: None 2024-04-19T23:41:01.395 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.395 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g3/Q4VCGVK6RUXtLAiDo2GisASA= 2024-04-19T23:41:01.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g3/Q4VCGVK6RUXtLAiDo2GisASA=', 'Content-Length': '0'} 2024-04-19T23:41:01.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f855b1b222b785f6-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.398 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.399 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.399 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.399 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.399 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.399 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.400 DEBUG:boto:Data: 2024-04-19T23:41:01.400 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.400 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.400 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.400 DEBUG:boto:Params: {} 2024-04-19T23:41:01.400 DEBUG:boto:Token: None 2024-04-19T23:41:01.400 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.400 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g3/Q4VCGVK6RUXtLAiDo2GisASA= 2024-04-19T23:41:01.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g3/Q4VCGVK6RUXtLAiDo2GisASA=', 'Content-Length': '0'} 2024-04-19T23:41:01.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee1e92eb7a08c0f5-006623010d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:01.403 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.403 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.403 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.403 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.403 DEBUG:boto:Method: GET 2024-04-19T23:41:01.403 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.403 DEBUG:boto:Data: 2024-04-19T23:41:01.404 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.404 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.404 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.404 DEBUG:boto:Params: {} 2024-04-19T23:41:01.404 DEBUG:boto:Token: None 2024-04-19T23:41:01.404 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.404 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002bb5062a447bf644-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.406 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.406 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.407 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.407 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.407 DEBUG:boto:Method: GET 2024-04-19T23:41:01.407 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.407 DEBUG:boto:Data: 2024-04-19T23:41:01.407 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.407 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.407 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.407 DEBUG:boto:Params: {} 2024-04-19T23:41:01.407 DEBUG:boto:Token: None 2024-04-19T23:41:01.407 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.408 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052e88daef7badd39-006623010d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.410 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.410 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.410 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.410 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.410 DEBUG:boto:Method: GET 2024-04-19T23:41:01.411 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.411 DEBUG:boto:Data: 2024-04-19T23:41:01.411 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.411 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.411 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.411 DEBUG:boto:Params: {} 2024-04-19T23:41:01.411 DEBUG:boto:Token: None 2024-04-19T23:41:01.411 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.411 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a41adadf2dc616fa-006623010d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.414 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.415 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.415 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.415 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.415 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.415 DEBUG:boto:Method: GET 2024-04-19T23:41:01.415 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.415 DEBUG:boto:Data: 2024-04-19T23:41:01.415 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.415 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.415 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.415 DEBUG:boto:Params: {} 2024-04-19T23:41:01.416 DEBUG:boto:Token: None 2024-04-19T23:41:01.416 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.416 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006070f5a12e5a0eb4-006623010d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.418 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.419 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.419 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.419 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.419 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:01.419 DEBUG:boto:Data: 2024-04-19T23:41:01.419 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.419 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.419 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.419 DEBUG:boto:Params: {} 2024-04-19T23:41:01.420 DEBUG:boto:Token: None 2024-04-19T23:41:01.420 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj 2024-04-19T23:41:01.420 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI= 2024-04-19T23:41:01.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI=', 'Content-Length': '0'} 2024-04-19T23:41:01.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c943c495db3cb2e-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.422 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.422 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.422 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.423 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:01.423 DEBUG:boto:Data: 2024-04-19T23:41:01.423 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.423 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.423 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.423 DEBUG:boto:Params: {} 2024-04-19T23:41:01.423 DEBUG:boto:Token: None 2024-04-19T23:41:01.423 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj 2024-04-19T23:41:01.423 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI= 2024-04-19T23:41:01.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI=', 'Content-Length': '0'} 2024-04-19T23:41:01.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a40427d40997b4b5-006623010d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:01.426 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.426 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.427 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.427 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.427 DEBUG:boto:Method: GET 2024-04-19T23:41:01.427 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.427 DEBUG:boto:Data: 2024-04-19T23:41:01.427 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.427 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.427 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.427 DEBUG:boto:Params: {} 2024-04-19T23:41:01.427 DEBUG:boto:Token: None 2024-04-19T23:41:01.428 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.428 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ebc5898dbc80ef60-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.430 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.431 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.431 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.432 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.432 DEBUG:boto:Method: GET 2024-04-19T23:41:01.432 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.432 DEBUG:boto:Data: 2024-04-19T23:41:01.432 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.433 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.433 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.433 DEBUG:boto:Params: {} 2024-04-19T23:41:01.433 DEBUG:boto:Token: None 2024-04-19T23:41:01.433 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.434 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3326ced1a548f19-006623010d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.436 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.436 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.437 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.437 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.437 DEBUG:boto:Method: GET 2024-04-19T23:41:01.437 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.437 DEBUG:boto:Data: 2024-04-19T23:41:01.437 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.438 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.438 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.438 DEBUG:boto:Params: {} 2024-04-19T23:41:01.438 DEBUG:boto:Token: None 2024-04-19T23:41:01.438 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.439 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000329d633aab46559f-006623010d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.482 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.483 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.483 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.483 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.483 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.483 DEBUG:boto:Method: GET 2024-04-19T23:41:01.483 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.483 DEBUG:boto:Data: 2024-04-19T23:41:01.484 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.484 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.484 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.484 DEBUG:boto:Params: {} 2024-04-19T23:41:01.484 DEBUG:boto:Token: None 2024-04-19T23:41:01.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7abafcef29a886c-006623010d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.487 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:01.487 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.487 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.487 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.487 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.488 DEBUG:boto:Method: GET 2024-04-19T23:41:01.488 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.488 DEBUG:boto:Data: 2024-04-19T23:41:01.488 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.488 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.488 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.488 DEBUG:boto:Params: {} 2024-04-19T23:41:01.488 DEBUG:boto:Token: None 2024-04-19T23:41:01.488 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY= 2024-04-19T23:41:01.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY=', 'Content-Length': '0'} 2024-04-19T23:41:01.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cac6f3296bf3e6fb-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.491 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.491 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.491 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.491 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.491 DEBUG:boto:Method: GET 2024-04-19T23:41:01.491 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.491 DEBUG:boto:Data: 2024-04-19T23:41:01.491 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.491 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.491 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.491 DEBUG:boto:Params: {} 2024-04-19T23:41:01.491 DEBUG:boto:Token: None 2024-04-19T23:41:01.492 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.492 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY= 2024-04-19T23:41:01.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY=', 'Content-Length': '0'} 2024-04-19T23:41:01.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000044b1340b08dbf93c-006623010d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.494 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.494 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.494 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.494 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.494 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.494 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.494 DEBUG:boto:Data: 2024-04-19T23:41:01.495 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.495 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.495 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.495 DEBUG:boto:Params: {} 2024-04-19T23:41:01.495 DEBUG:boto:Token: None 2024-04-19T23:41:01.495 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.495 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SXMFIGltbqIKSdTA2965KVDnYmw= 2024-04-19T23:41:01.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SXMFIGltbqIKSdTA2965KVDnYmw=', 'Content-Length': '0'} 2024-04-19T23:41:01.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000317c4d5c6985684b-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.497 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.497 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.498 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.498 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.498 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.498 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.498 DEBUG:boto:Data: 2024-04-19T23:41:01.498 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.498 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.498 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.498 DEBUG:boto:Params: {} 2024-04-19T23:41:01.498 DEBUG:boto:Token: None 2024-04-19T23:41:01.499 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.499 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SXMFIGltbqIKSdTA2965KVDnYmw= 2024-04-19T23:41:01.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SXMFIGltbqIKSdTA2965KVDnYmw=', 'Content-Length': '0'} 2024-04-19T23:41:01.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000085860614addc93ed-006623010d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:01.501 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.501 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.501 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.501 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.501 DEBUG:boto:Method: GET 2024-04-19T23:41:01.501 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.501 DEBUG:boto:Data: 2024-04-19T23:41:01.501 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.501 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.502 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.502 DEBUG:boto:Params: {} 2024-04-19T23:41:01.502 DEBUG:boto:Token: None 2024-04-19T23:41:01.502 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.502 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY= 2024-04-19T23:41:01.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY=', 'Content-Length': '0'} 2024-04-19T23:41:01.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019f9e98b64c0411e-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.504 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.505 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.505 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.505 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:01.505 DEBUG:boto:Method: GET 2024-04-19T23:41:01.505 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.505 DEBUG:boto:Data: 2024-04-19T23:41:01.505 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.505 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.506 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.506 DEBUG:boto:Params: {} 2024-04-19T23:41:01.506 DEBUG:boto:Token: None 2024-04-19T23:41:01.506 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:01.506 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY= 2024-04-19T23:41:01.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xy8gNNvP9rpCAJDZ23CdAeBXfKY=', 'Content-Length': '0'} 2024-04-19T23:41:01.508 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:39:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000524a1296b0edbe01-006623010d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.508 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.508 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.508 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.508 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.509 DEBUG:boto:Method: GET 2024-04-19T23:41:01.509 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.509 DEBUG:boto:Data: 2024-04-19T23:41:01.509 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.509 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.509 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.509 DEBUG:boto:Params: {} 2024-04-19T23:41:01.509 DEBUG:boto:Token: None 2024-04-19T23:41:01.509 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.509 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2c2f3b1cf12008a-006623010d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.512 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.512 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.512 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.512 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.513 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.513 DEBUG:boto:Method: GET 2024-04-19T23:41:01.513 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.513 DEBUG:boto:Data: 2024-04-19T23:41:01.513 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.513 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.513 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.513 DEBUG:boto:Params: {} 2024-04-19T23:41:01.513 DEBUG:boto:Token: None 2024-04-19T23:41:01.513 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.514 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8a9d1a4c79d16a-006623010d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.516 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.516 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.516 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.517 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.517 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:01.517 DEBUG:boto:Data: 2024-04-19T23:41:01.517 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.517 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.517 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.517 DEBUG:boto:Params: {} 2024-04-19T23:41:01.517 DEBUG:boto:Token: None 2024-04-19T23:41:01.517 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj 2024-04-19T23:41:01.517 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI= 2024-04-19T23:41:01.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI=', 'Content-Length': '0'} 2024-04-19T23:41:01.520 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000feac4186c356dd63-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.520 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.520 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.520 DEBUG:boto:Method: HEAD 2024-04-19T23:41:01.520 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:01.520 DEBUG:boto:Data: 2024-04-19T23:41:01.521 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.521 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.521 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.521 DEBUG:boto:Params: {} 2024-04-19T23:41:01.521 DEBUG:boto:Token: None 2024-04-19T23:41:01.521 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj 2024-04-19T23:41:01.521 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI= 2024-04-19T23:41:01.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bj6IHrBBcenHoJwOd3aC56tfaHI=', 'Content-Length': '0'} 2024-04-19T23:41:01.523 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed0d6a17632e7695-006623010d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:01.524 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.524 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.524 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.524 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.524 DEBUG:boto:Method: GET 2024-04-19T23:41:01.524 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.524 DEBUG:boto:Data: 2024-04-19T23:41:01.524 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.524 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.525 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.525 DEBUG:boto:Params: {} 2024-04-19T23:41:01.525 DEBUG:boto:Token: None 2024-04-19T23:41:01.525 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.525 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000057e4b70e71cab671-006623010d-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.527 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:01.527 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:01.527 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.527 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.528 DEBUG:boto:Method: GET 2024-04-19T23:41:01.528 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.528 DEBUG:boto:Data: 2024-04-19T23:41:01.528 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.528 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.528 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.528 DEBUG:boto:Params: {} 2024-04-19T23:41:01.528 DEBUG:boto:Token: None 2024-04-19T23:41:01.528 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:01.528 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo= 2024-04-19T23:41:01.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mBbyCFp0RQzzxHJrPtmUDI33cZo=', 'Content-Length': '0'} 2024-04-19T23:41:01.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003bdeeb6060e32c1a-006623010d-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.530 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.530 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.531 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.531 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.531 DEBUG:boto:Method: GET 2024-04-19T23:41:01.531 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.531 DEBUG:boto:Data: 2024-04-19T23:41:01.531 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.531 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:01.531 DEBUG:boto:Port: 8001 2024-04-19T23:41:01.531 DEBUG:boto:Params: {} 2024-04-19T23:41:01.531 DEBUG:boto:Token: None 2024-04-19T23:41:01.531 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.532 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014278d9990e4d131-006623010d-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.574 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.575 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.575 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.575 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.575 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:01.575 DEBUG:boto:Method: GET 2024-04-19T23:41:01.575 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:01.575 DEBUG:boto:Data: 2024-04-19T23:41:01.575 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.576 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:01.576 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.576 DEBUG:boto:Params: {} 2024-04-19T23:41:01.576 DEBUG:boto:Token: None 2024-04-19T23:41:01.576 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/ 2024-04-19T23:41:01.576 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c= 2024-04-19T23:41:01.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NLQK0puDymVYebuPQURUTVupY+c=', 'Content-Length': '0'} 2024-04-19T23:41:01.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077561dfa903d9c8f-006623010d-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.579 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:01.579 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-23 zones={test-zone3, test-zone2} 2024-04-19T23:41:01.580 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:01.580 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:01.580 DEBUG:boto:path=/aktlji-23/?versioning 2024-04-19T23:41:01.580 DEBUG:boto:auth_path=/aktlji-23/?versioning 2024-04-19T23:41:01.580 DEBUG:boto:Method: PUT 2024-04-19T23:41:01.580 DEBUG:boto:Path: /aktlji-23/?versioning 2024-04-19T23:41:01.580 DEBUG:boto:Data: Suspended Disabled 2024-04-19T23:41:01.580 DEBUG:boto:Headers: {} 2024-04-19T23:41:01.580 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:01.580 DEBUG:boto:Port: 8000 2024-04-19T23:41:01.580 DEBUG:boto:Params: {} 2024-04-19T23:41:01.581 DEBUG:boto:Token: None 2024-04-19T23:41:01.581 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:41:01 GMT /aktlji-23/?versioning 2024-04-19T23:41:01.581 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jChU8NccYxLSdngqbXwjkxIoDrw= 2024-04-19T23:41:01.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jChU8NccYxLSdngqbXwjkxIoDrw=', 'Content-Length': '230'} 2024-04-19T23:41:01.588 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a7ce30bb048c805e-006623010d-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:01.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:41:01.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:41:01.588 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:41:01.646 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:41:01.646 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:41:01.682 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:41:01.697 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:41:01.697 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:41:01.697 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:41:01.697 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:41:01.697 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:41:01.697 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:41:01.697 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:41:01.698 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:41:01.698 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:41:01.698 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:41:01.698 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713570061.590151_222.1", 2024-04-19T23:41:01.698 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:41:01.590151Z" 2024-04-19T23:41:01.698 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:41:01.698 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:41:01.698 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:41:01.698 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:41:01.698 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:41:01.698 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:41:01.700 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570061.590151_222.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:41:01.700 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:41:01.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:41:01.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:41:01.700 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:41:01.761 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:41:01.761 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:41:01.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:01.798+0000 7f4042e38780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:41:01.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:01.800+0000 7f4042e38780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:41:01.815 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:41:01.815 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:41:01.815 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:41:01.815 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:41:01.815 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:41:01.815 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:41:01.815 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:01.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:01.816 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:41:01.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:41:01.816 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:41:01.816 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:41:01.816 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:41:01.816 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:41:01.816 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:41:01.816 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:41:01.816 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:41:01.816 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:41:01.816 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:01.817 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:01.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:01.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:41:01.817 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:41:01.817 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:41:01.817 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:41:01.817 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:41:01.817 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:41:01.817 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:41:01.817 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:41:01.817 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:41:01.817 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:01.818 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:01.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:01.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:41:01.818 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:41:01.818 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:41:01.818 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:41:01.818 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570061.586416_221.1", 2024-04-19T23:41:01.818 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:41:01.818 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:41:01.818 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:41:01.819 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:41:01.586416Z", 2024-04-19T23:41:01.819 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:01.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:01.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:01.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:41:01.819 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:41:01.819 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:41:01.819 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:41:01.819 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:41:01.819 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:41:01.819 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:41:01.820 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:41:01.820 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:41:01.820 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:01.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:01.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:01.820 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:41:01.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:01.820 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:41:01.821 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:41:01.821 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:41:01.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:01.821 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:41:01Z" 2024-04-19T23:41:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:41:01.822 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570061.586416_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:41:01.586416Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:41:01Z" } 2024-04-19T23:41:01.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570061.586416_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:41:01.586416Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:41:01.822 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570061.590151_222.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:41:01.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570061.586416_221.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:41:01.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570061.590151_222.1 target=1_1713570061.586416_221.1 2024-04-19T23:41:06.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:41:06.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:41:06.824 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:41:06.885 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:41:06.886 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:41:06.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:06.922+0000 7f587637e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:41:06.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:06.923+0000 7f587637e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:41:06.936 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:41:06.936 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:41:06.936 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:41:06.936 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:41:06.936 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:41:06.936 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:41:06.937 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:06.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:06.937 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:41:06.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:41:06.937 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:41:06.937 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:41:06.937 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:41:06.937 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:41:06.937 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:41:06.937 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:41:06.937 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:41:06.938 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:41:06.938 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:06.938 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:06.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:06.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:41:06.938 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:41:06.938 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:41:06.938 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:41:06.938 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:41:06.939 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:41:06.939 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:41:06.939 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:41:06.939 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:41:06.939 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:06.939 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:06.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:06.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:41:06.939 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:41:06.939 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:41:06.939 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:41:06.939 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570061.590151_222.1", 2024-04-19T23:41:06.940 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:41:06.940 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:41:06.940 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:41:06.940 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:41:01.590151Z", 2024-04-19T23:41:06.940 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:06.941 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:06.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:06.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:41:06.941 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:41:06.941 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:41:06.941 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:41:06.942 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:41:06.942 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:41:06.942 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:41:06.942 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:41:06.942 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:41:06.942 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:06.942 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:06.942 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:06.942 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:41:06.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:06.943 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:41:06.943 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:41:06.943 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:41:06.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:06.943 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:41:06Z" 2024-04-19T23:41:06.943 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:41:06.944 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570061.590151_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:41:01.590151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:41:06Z" } 2024-04-19T23:41:06.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570061.590151_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:41:01.590151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:41:06.944 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570061.590151_222.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:41:06.944 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570061.590151_222.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:41:06.944 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:41:06.944 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:41:06.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:41:06.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:41:06.945 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:41:07.004 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:41:07.005 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:41:07.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:07.042+0000 7fbe7b71d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:41:07.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:07.043+0000 7fbe7b71d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:41:07.054 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:41:07.054 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:41:07.054 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:41:07.054 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:41:07.055 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:41:07.055 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:41:07.055 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:41:07.055 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:41:07.055 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:41:07.055 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:41:07.055 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:41:07.055 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:41:07.055 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:41:07.055 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:41:07.055 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:41:07.056 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:41:07.057 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:41:07.057 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:41:07.057 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:41:07.057 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:41:07.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:41:07.057 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:41:07.057 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:41:07.057 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:41:07.057 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:41:07.057 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713570061.590151_222.1", 2024-04-19T23:41:07.057 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:41:07.058 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:41:07.058 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:41:07.058 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:41:01.590151Z", 2024-04-19T23:41:07.058 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:41:07.058 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:41:07.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:41:07.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:41:07.058 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:41:07.058 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:41:07.058 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:41:07.058 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:41:07.059 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:41:07.059 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:41:07.059 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:41:07.059 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:41:07.059 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:41:07.059 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:41:07.059 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:41:07.059 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:41:07.059 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:41:07.059 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:41:07.059 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:41:07.059 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:41:07.060 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:41:07.060 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:41:07Z" 2024-04-19T23:41:07.060 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:41:07.061 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570061.590151_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:41:01.590151Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:41:07Z" } 2024-04-19T23:41:07.061 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570061.590151_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:41:01.590151Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:41:07.061 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570061.590151_222.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:41:07.061 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570061.590151_222.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:41:07.061 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:41:07.061 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:07.061 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:07.062 DEBUG:boto:Method: HEAD 2024-04-19T23:41:07.062 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:07.062 DEBUG:boto:Data: 2024-04-19T23:41:07.062 DEBUG:boto:Headers: {} 2024-04-19T23:41:07.062 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:07.062 DEBUG:boto:Port: 8000 2024-04-19T23:41:07.062 DEBUG:boto:Params: {} 2024-04-19T23:41:07.063 DEBUG:boto:Token: None 2024-04-19T23:41:07.063 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:07 GMT /aktlji-23/ 2024-04-19T23:41:07.063 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hijXbXdpgEDcC4vK8UEvoGYV2Qc= 2024-04-19T23:41:07.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hijXbXdpgEDcC4vK8UEvoGYV2Qc=', 'Content-Length': '0'} 2024-04-19T23:41:07.065 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c71e6ab09f668540-0066230113-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:07.066 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:07.066 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:07.066 DEBUG:boto:Method: PUT 2024-04-19T23:41:07.066 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:07.066 DEBUG:boto:Data: 2024-04-19T23:41:07.066 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:41:07.066 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:07.066 DEBUG:boto:Port: 8000 2024-04-19T23:41:07.066 DEBUG:boto:Params: {} 2024-04-19T23:41:07.067 DEBUG:boto:Token: None 2024-04-19T23:41:07.067 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:41:07 GMT /aktlji-23/obj 2024-04-19T23:41:07.067 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7jQi6ED1wrKRruRLozi+0u7kwpQ= 2024-04-19T23:41:07.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:41:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7jQi6ED1wrKRruRLozi+0u7kwpQ='} 2024-04-19T23:41:07.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2e3a8e3f53ee6b6-0066230113-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:07.079 DEBUG:rgw_multi.tests:created null version id=None 2024-04-19T23:41:07.080 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-23 2024-04-19T23:41:07.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:41:07.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:41:07.080 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:41:07.146 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:41:07.147 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:41:07.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:07.190+0000 7f26cd9b1780 1 waiting for incremental sync to catch up: 2024-04-19T23:41:07.189 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000003.63.13, , ] 2024-04-19T23:41:07.189 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000005.67.18, , ] 2024-04-19T23:41:12.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:12.193+0000 7f26cd9b1780 1 waiting for incremental sync to catch up: 2024-04-19T23:41:12.191 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000003.63.13, , ] 2024-04-19T23:41:12.191 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000005.67.18, , ] 2024-04-19T23:41:17.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:17.194+0000 7f26cd9b1780 1 waiting for incremental sync to catch up: 2024-04-19T23:41:17.192 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000003.63.13, , ] 2024-04-19T23:41:17.192 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000005.67.18, , ] 2024-04-19T23:41:22.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:22.195+0000 7f26cd9b1780 1 waiting for incremental sync to catch up: 2024-04-19T23:41:22.194 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000003.63.13, , ] 2024-04-19T23:41:22.194 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000005.67.18, , ] 2024-04-19T23:41:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:27.197+0000 7f26cd9b1780 1 bucket sync caught up with source: 2024-04-19T23:41:27.195 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000005.67.18, , ] 2024-04-19T23:41:27.195 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000005.67.18, , ] 2024-04-19T23:41:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:27.197+0000 7f26cd9b1780 0 bucket checkpoint complete 2024-04-19T23:41:27.212 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-23 2024-04-19T23:41:27.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:41:27.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:41:27.213 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:41:27.279 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:41:27.279 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:41:28.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:28.325+0000 7ff300da8780 1 bucket sync caught up with source: 2024-04-19T23:41:28.323 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000005.67.18, , ] 2024-04-19T23:41:28.323 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000005.67.18, , ] 2024-04-19T23:41:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:28.325+0000 7ff300da8780 0 bucket checkpoint complete 2024-04-19T23:41:28.340 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-23 2024-04-19T23:41:28.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:41:28.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:41:28.341 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:41:28.411 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:41:28.411 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:41:28.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:28.454+0000 7f20f25e6780 1 waiting for incremental sync to catch up: 2024-04-19T23:41:28.453 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000004.85.10, , ] 2024-04-19T23:41:28.453 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000006.90.18, , ] 2024-04-19T23:41:33.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:33.456+0000 7f20f25e6780 1 waiting for incremental sync to catch up: 2024-04-19T23:41:33.455 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000004.85.10, , ] 2024-04-19T23:41:33.455 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000006.90.18, , ] 2024-04-19T23:41:38.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:38.457+0000 7f20f25e6780 1 bucket sync caught up with source: 2024-04-19T23:41:38.456 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000006.90.18, , ] 2024-04-19T23:41:38.456 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000006.90.18, , ] 2024-04-19T23:41:38.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:38.457+0000 7f20f25e6780 0 bucket checkpoint complete 2024-04-19T23:41:38.473 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-23 2024-04-19T23:41:38.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:41:38.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:41:38.474 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:41:38.540 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:41:38.540 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:41:38.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:38.585+0000 7f15cb1bb780 1 waiting for incremental sync to catch up: 2024-04-19T23:41:38.584 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000004.85.10, , ] 2024-04-19T23:41:38.584 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000007.92.15, , ] 2024-04-19T23:41:43.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:43.586+0000 7f15cb1bb780 1 bucket sync caught up with source: 2024-04-19T23:41:43.586 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000007.92.15, , ] 2024-04-19T23:41:43.586 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000007.92.15, , ] 2024-04-19T23:41:43.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:43.586+0000 7f15cb1bb780 0 bucket checkpoint complete 2024-04-19T23:41:43.602 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-23 2024-04-19T23:41:43.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:41:43.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:41:43.603 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:41:43.670 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:41:43.670 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:41:43.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:43.714+0000 7fd6e5578780 1 waiting for incremental sync to catch up: 2024-04-19T23:41:43.713 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000006.84.18, , ] 2024-04-19T23:41:43.713 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000007.86.15, , ] 2024-04-19T23:41:48.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:48.716+0000 7fd6e5578780 1 waiting for incremental sync to catch up: 2024-04-19T23:41:48.715 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000006.84.18, , ] 2024-04-19T23:41:48.715 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000007.86.15, , ] 2024-04-19T23:41:53.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:53.718+0000 7fd6e5578780 1 waiting for incremental sync to catch up: 2024-04-19T23:41:53.717 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000006.84.18, , ] 2024-04-19T23:41:53.717 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000007.86.15, , ] 2024-04-19T23:41:58.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:58.719+0000 7fd6e5578780 1 bucket sync caught up with source: 2024-04-19T23:41:58.718 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000007.86.15, , ] 2024-04-19T23:41:58.718 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000007.86.15, , ] 2024-04-19T23:41:58.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:41:58.719+0000 7fd6e5578780 0 bucket checkpoint complete 2024-04-19T23:41:58.735 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-23 2024-04-19T23:41:58.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:41:58.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:41:58.736 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:41:58.802 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:41:58.803 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:41:58.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:58.847+0000 7fd448b8c780 1 bucket sync caught up with source: 2024-04-19T23:41:58.845 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000007.86.15, , ] 2024-04-19T23:41:58.845 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000007.86.15, , ] 2024-04-19T23:41:58.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:58.847+0000 7fd448b8c780 0 bucket checkpoint complete 2024-04-19T23:41:58.860 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-23 zones={test-zone2, test-zone1} 2024-04-19T23:41:58.860 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:58.861 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:58.861 DEBUG:boto:Method: HEAD 2024-04-19T23:41:58.861 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:58.861 DEBUG:boto:Data: 2024-04-19T23:41:58.861 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.861 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.861 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.861 DEBUG:boto:Params: {} 2024-04-19T23:41:58.862 DEBUG:boto:Token: None 2024-04-19T23:41:58.862 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/ 2024-04-19T23:41:58.862 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gHQpm9DK+YAU2fYbNv9NSlouAog= 2024-04-19T23:41:58.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gHQpm9DK+YAU2fYbNv9NSlouAog=', 'Content-Length': '0'} 2024-04-19T23:41:58.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccd95960ef93dba6-0066230146-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.865 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:58.865 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:58.865 DEBUG:boto:Method: HEAD 2024-04-19T23:41:58.865 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:58.865 DEBUG:boto:Data: 2024-04-19T23:41:58.865 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.865 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.865 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.866 DEBUG:boto:Params: {} 2024-04-19T23:41:58.866 DEBUG:boto:Token: None 2024-04-19T23:41:58.866 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/ 2024-04-19T23:41:58.866 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gHQpm9DK+YAU2fYbNv9NSlouAog= 2024-04-19T23:41:58.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gHQpm9DK+YAU2fYbNv9NSlouAog=', 'Content-Length': '0'} 2024-04-19T23:41:58.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff2867fc6ee81c59-0066230146-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.869 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:41:58.869 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:58.869 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:58.869 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:58.869 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:58.869 DEBUG:boto:Method: GET 2024-04-19T23:41:58.869 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:58.869 DEBUG:boto:Data: 2024-04-19T23:41:58.869 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.869 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.870 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.870 DEBUG:boto:Params: {} 2024-04-19T23:41:58.870 DEBUG:boto:Token: None 2024-04-19T23:41:58.870 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/?versions 2024-04-19T23:41:58.870 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gS/etIrepKV/8Ak8f3Y8RxnZ/xc= 2024-04-19T23:41:58.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gS/etIrepKV/8Ak8f3Y8RxnZ/xc=', 'Content-Length': '0'} 2024-04-19T23:41:58.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d27da10521de703-0066230146-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.872 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUfalse2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:58.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:58.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:58.873 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:41:58.874 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:58.874 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:58.874 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:58.874 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:58.874 DEBUG:boto:Method: GET 2024-04-19T23:41:58.874 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:58.874 DEBUG:boto:Data: 2024-04-19T23:41:58.874 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.874 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.874 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.875 DEBUG:boto:Params: {} 2024-04-19T23:41:58.875 DEBUG:boto:Token: None 2024-04-19T23:41:58.875 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/?versions 2024-04-19T23:41:58.875 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gS/etIrepKV/8Ak8f3Y8RxnZ/xc= 2024-04-19T23:41:58.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gS/etIrepKV/8Ak8f3Y8RxnZ/xc=', 'Content-Length': '0'} 2024-04-19T23:41:58.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a2c3fcd1572b187-0066230146-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.878 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUfalse2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:58.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:58.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:58.878 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:58.879 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:58.879 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:58.879 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:58.879 DEBUG:boto:Method: GET 2024-04-19T23:41:58.879 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:58.879 DEBUG:boto:Data: 2024-04-19T23:41:58.879 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.879 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.879 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.879 DEBUG:boto:Params: {} 2024-04-19T23:41:58.880 DEBUG:boto:Token: None 2024-04-19T23:41:58.880 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/?versions 2024-04-19T23:41:58.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gS/etIrepKV/8Ak8f3Y8RxnZ/xc= 2024-04-19T23:41:58.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gS/etIrepKV/8Ak8f3Y8RxnZ/xc=', 'Content-Length': '0'} 2024-04-19T23:41:58.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3b75dfc481c2605-0066230146-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.882 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUfalse2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:58.883 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:58.883 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:58.883 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:58.883 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:58.883 DEBUG:boto:Method: GET 2024-04-19T23:41:58.883 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:58.883 DEBUG:boto:Data: 2024-04-19T23:41:58.883 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.883 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.883 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.883 DEBUG:boto:Params: {} 2024-04-19T23:41:58.884 DEBUG:boto:Token: None 2024-04-19T23:41:58.884 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/?versions 2024-04-19T23:41:58.884 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gS/etIrepKV/8Ak8f3Y8RxnZ/xc= 2024-04-19T23:41:58.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gS/etIrepKV/8Ak8f3Y8RxnZ/xc=', 'Content-Length': '0'} 2024-04-19T23:41:58.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea4b9c08eab77810-0066230146-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.886 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUfalse2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:58.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:58.887 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.887 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.887 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:58.887 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:58.887 DEBUG:boto:Method: GET 2024-04-19T23:41:58.887 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:58.887 DEBUG:boto:Data: 2024-04-19T23:41:58.888 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.888 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.888 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.888 DEBUG:boto:Params: {} 2024-04-19T23:41:58.888 DEBUG:boto:Token: None 2024-04-19T23:41:58.888 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:58.888 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nU560pJlKrUJiBImU4/irrQDKjw= 2024-04-19T23:41:58.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nU560pJlKrUJiBImU4/irrQDKjw=', 'Content-Length': '0'} 2024-04-19T23:41:58.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000157693b661dd6deb-0066230146-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.890 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.890 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.891 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:58.891 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:58.891 DEBUG:boto:Method: GET 2024-04-19T23:41:58.891 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:58.891 DEBUG:boto:Data: 2024-04-19T23:41:58.891 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.891 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.891 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.891 DEBUG:boto:Params: {} 2024-04-19T23:41:58.891 DEBUG:boto:Token: None 2024-04-19T23:41:58.892 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:58.892 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nU560pJlKrUJiBImU4/irrQDKjw= 2024-04-19T23:41:58.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nU560pJlKrUJiBImU4/irrQDKjw=', 'Content-Length': '0'} 2024-04-19T23:41:58.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f25dfe25a83c7214-0066230146-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.894 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.894 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.894 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:58.894 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:58.894 DEBUG:boto:Method: HEAD 2024-04-19T23:41:58.895 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:58.895 DEBUG:boto:Data: 2024-04-19T23:41:58.895 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.895 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.895 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.895 DEBUG:boto:Params: {} 2024-04-19T23:41:58.895 DEBUG:boto:Token: None 2024-04-19T23:41:58.895 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:58.895 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GNfqoTXknJnx3iczctuxaZzHHJs= 2024-04-19T23:41:58.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GNfqoTXknJnx3iczctuxaZzHHJs=', 'Content-Length': '0'} 2024-04-19T23:41:58.897 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c301468e857ca934-0066230146-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.897 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.898 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.898 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:58.898 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:58.898 DEBUG:boto:Method: HEAD 2024-04-19T23:41:58.898 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:58.898 DEBUG:boto:Data: 2024-04-19T23:41:58.898 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.898 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.898 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.898 DEBUG:boto:Params: {} 2024-04-19T23:41:58.899 DEBUG:boto:Token: None 2024-04-19T23:41:58.899 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:58.899 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GNfqoTXknJnx3iczctuxaZzHHJs= 2024-04-19T23:41:58.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GNfqoTXknJnx3iczctuxaZzHHJs=', 'Content-Length': '0'} 2024-04-19T23:41:58.901 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000212bf7cb1685343-0066230146-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:58.901 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.901 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.901 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:58.902 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:58.902 DEBUG:boto:Method: GET 2024-04-19T23:41:58.902 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:58.902 DEBUG:boto:Data: 2024-04-19T23:41:58.902 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.902 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.902 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.902 DEBUG:boto:Params: {} 2024-04-19T23:41:58.902 DEBUG:boto:Token: None 2024-04-19T23:41:58.902 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:58.903 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nU560pJlKrUJiBImU4/irrQDKjw= 2024-04-19T23:41:58.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nU560pJlKrUJiBImU4/irrQDKjw=', 'Content-Length': '0'} 2024-04-19T23:41:58.904 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e07f20f3f2b4e07-0066230146-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.905 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.905 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.905 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:58.905 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:58.905 DEBUG:boto:Method: GET 2024-04-19T23:41:58.905 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:58.905 DEBUG:boto:Data: 2024-04-19T23:41:58.905 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.905 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.905 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.905 DEBUG:boto:Params: {} 2024-04-19T23:41:58.906 DEBUG:boto:Token: None 2024-04-19T23:41:58.906 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:58.906 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nU560pJlKrUJiBImU4/irrQDKjw= 2024-04-19T23:41:58.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nU560pJlKrUJiBImU4/irrQDKjw=', 'Content-Length': '0'} 2024-04-19T23:41:58.908 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000005162bfe50ebf849-0066230146-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.908 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:58.908 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:58.908 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:58.908 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:58.908 DEBUG:boto:Method: GET 2024-04-19T23:41:58.909 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:58.909 DEBUG:boto:Data: 2024-04-19T23:41:58.909 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.909 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.909 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.909 DEBUG:boto:Params: {} 2024-04-19T23:41:58.909 DEBUG:boto:Token: None 2024-04-19T23:41:58.909 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/ 2024-04-19T23:41:58.909 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6FUDtZT0mrTeQi7FFnJfRTNd8OY= 2024-04-19T23:41:58.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6FUDtZT0mrTeQi7FFnJfRTNd8OY=', 'Content-Length': '0'} 2024-04-19T23:41:58.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7f178477c5cc928-0066230146-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.912 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:58.912 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:58.912 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:58.912 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:58.912 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:58.913 DEBUG:boto:Method: GET 2024-04-19T23:41:58.913 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:58.913 DEBUG:boto:Data: 2024-04-19T23:41:58.913 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.913 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.913 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.913 DEBUG:boto:Params: {} 2024-04-19T23:41:58.913 DEBUG:boto:Token: None 2024-04-19T23:41:58.913 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/ 2024-04-19T23:41:58.913 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6FUDtZT0mrTeQi7FFnJfRTNd8OY= 2024-04-19T23:41:58.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6FUDtZT0mrTeQi7FFnJfRTNd8OY=', 'Content-Length': '0'} 2024-04-19T23:41:58.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a09b7ce8fde00848-0066230146-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.916 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:58.916 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.917 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.917 DEBUG:boto:Method: HEAD 2024-04-19T23:41:58.917 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:58.917 DEBUG:boto:Data: 2024-04-19T23:41:58.917 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.917 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.917 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.917 DEBUG:boto:Params: {} 2024-04-19T23:41:58.917 DEBUG:boto:Token: None 2024-04-19T23:41:58.917 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj 2024-04-19T23:41:58.918 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zalCuf0IrD34twVSuCNS5S+kkL8= 2024-04-19T23:41:58.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zalCuf0IrD34twVSuCNS5S+kkL8=', 'Content-Length': '0'} 2024-04-19T23:41:58.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b4b5c12a3e655c56-0066230146-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.920 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.920 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.920 DEBUG:boto:Method: HEAD 2024-04-19T23:41:58.920 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:58.920 DEBUG:boto:Data: 2024-04-19T23:41:58.920 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.920 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.920 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.920 DEBUG:boto:Params: {} 2024-04-19T23:41:58.920 DEBUG:boto:Token: None 2024-04-19T23:41:58.921 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj 2024-04-19T23:41:58.921 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zalCuf0IrD34twVSuCNS5S+kkL8= 2024-04-19T23:41:58.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zalCuf0IrD34twVSuCNS5S+kkL8=', 'Content-Length': '0'} 2024-04-19T23:41:58.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a3993c48b42a0d9-0066230146-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:58.923 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.923 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.923 DEBUG:boto:Method: GET 2024-04-19T23:41:58.923 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:58.923 DEBUG:boto:Data: 2024-04-19T23:41:58.923 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.923 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.923 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.923 DEBUG:boto:Params: {} 2024-04-19T23:41:58.924 DEBUG:boto:Token: None 2024-04-19T23:41:58.924 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj 2024-04-19T23:41:58.924 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+lv/6hUZy8mY0TspxRKdytHR/rw= 2024-04-19T23:41:58.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+lv/6hUZy8mY0TspxRKdytHR/rw=', 'Content-Length': '0'} 2024-04-19T23:41:58.925 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c76518dfdef6ec1e-0066230146-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.926 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.926 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.926 DEBUG:boto:Method: GET 2024-04-19T23:41:58.926 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:58.926 DEBUG:boto:Data: 2024-04-19T23:41:58.926 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.926 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.926 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.926 DEBUG:boto:Params: {} 2024-04-19T23:41:58.926 DEBUG:boto:Token: None 2024-04-19T23:41:58.927 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj 2024-04-19T23:41:58.927 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+lv/6hUZy8mY0TspxRKdytHR/rw= 2024-04-19T23:41:58.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+lv/6hUZy8mY0TspxRKdytHR/rw=', 'Content-Length': '0'} 2024-04-19T23:41:58.929 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056f880594d84462d-0066230146-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.929 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:58.929 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:58.929 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:58.929 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:58.929 DEBUG:boto:Method: GET 2024-04-19T23:41:58.929 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:58.929 DEBUG:boto:Data: 2024-04-19T23:41:58.929 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.930 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.930 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.930 DEBUG:boto:Params: {} 2024-04-19T23:41:58.930 DEBUG:boto:Token: None 2024-04-19T23:41:58.930 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/ 2024-04-19T23:41:58.930 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6FUDtZT0mrTeQi7FFnJfRTNd8OY= 2024-04-19T23:41:58.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6FUDtZT0mrTeQi7FFnJfRTNd8OY=', 'Content-Length': '0'} 2024-04-19T23:41:58.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009868dccdb74fa56c-0066230146-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.974 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:58.975 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:58.975 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:58.975 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:58.975 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:58.975 DEBUG:boto:Method: GET 2024-04-19T23:41:58.975 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:58.975 DEBUG:boto:Data: 2024-04-19T23:41:58.975 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.975 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.975 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.976 DEBUG:boto:Params: {} 2024-04-19T23:41:58.976 DEBUG:boto:Token: None 2024-04-19T23:41:58.976 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/ 2024-04-19T23:41:58.976 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6FUDtZT0mrTeQi7FFnJfRTNd8OY= 2024-04-19T23:41:58.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6FUDtZT0mrTeQi7FFnJfRTNd8OY=', 'Content-Length': '0'} 2024-04-19T23:41:58.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cdf21de46d66e25-0066230146-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.978 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:58.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:58.979 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.979 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.979 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.979 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.979 DEBUG:boto:Method: GET 2024-04-19T23:41:58.979 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.979 DEBUG:boto:Data: 2024-04-19T23:41:58.979 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.980 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.980 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.980 DEBUG:boto:Params: {} 2024-04-19T23:41:58.980 DEBUG:boto:Token: None 2024-04-19T23:41:58.980 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.980 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5kmfG4jJL8lBqlAobjDAP53Ec98= 2024-04-19T23:41:58.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5kmfG4jJL8lBqlAobjDAP53Ec98=', 'Content-Length': '0'} 2024-04-19T23:41:58.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000026dc7eeb10af9c16-0066230146-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.982 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.983 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.983 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.983 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.983 DEBUG:boto:Method: GET 2024-04-19T23:41:58.983 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.983 DEBUG:boto:Data: 2024-04-19T23:41:58.983 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.983 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.983 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.983 DEBUG:boto:Params: {} 2024-04-19T23:41:58.984 DEBUG:boto:Token: None 2024-04-19T23:41:58.984 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.984 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5kmfG4jJL8lBqlAobjDAP53Ec98= 2024-04-19T23:41:58.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5kmfG4jJL8lBqlAobjDAP53Ec98=', 'Content-Length': '0'} 2024-04-19T23:41:58.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c7a764585ebf66b7-0066230146-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.987 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.987 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.987 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.987 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.987 DEBUG:boto:Method: HEAD 2024-04-19T23:41:58.987 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.987 DEBUG:boto:Data: 2024-04-19T23:41:58.987 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.987 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.987 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.988 DEBUG:boto:Params: {} 2024-04-19T23:41:58.988 DEBUG:boto:Token: None 2024-04-19T23:41:58.988 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.988 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:obti1194Dp0/C21k1ytKrNlNYtA= 2024-04-19T23:41:58.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:obti1194Dp0/C21k1ytKrNlNYtA=', 'Content-Length': '0'} 2024-04-19T23:41:58.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002446522c78612ede-0066230146-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.990 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.990 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.990 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.990 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.990 DEBUG:boto:Method: HEAD 2024-04-19T23:41:58.990 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.991 DEBUG:boto:Data: 2024-04-19T23:41:58.991 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.991 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.991 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.991 DEBUG:boto:Params: {} 2024-04-19T23:41:58.991 DEBUG:boto:Token: None 2024-04-19T23:41:58.991 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.991 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:obti1194Dp0/C21k1ytKrNlNYtA= 2024-04-19T23:41:58.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:obti1194Dp0/C21k1ytKrNlNYtA=', 'Content-Length': '0'} 2024-04-19T23:41:58.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019e25289cba8d617-0066230146-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:58.993 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.994 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.994 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.994 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.994 DEBUG:boto:Method: GET 2024-04-19T23:41:58.994 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.994 DEBUG:boto:Data: 2024-04-19T23:41:58.994 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.994 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:58.994 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.994 DEBUG:boto:Params: {} 2024-04-19T23:41:58.995 DEBUG:boto:Token: None 2024-04-19T23:41:58.995 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.995 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5kmfG4jJL8lBqlAobjDAP53Ec98= 2024-04-19T23:41:58.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5kmfG4jJL8lBqlAobjDAP53Ec98=', 'Content-Length': '0'} 2024-04-19T23:41:58.997 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054d9d9bb0164feaf-0066230146-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:58.997 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:58.997 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:58.997 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.997 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.997 DEBUG:boto:Method: GET 2024-04-19T23:41:58.997 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.997 DEBUG:boto:Data: 2024-04-19T23:41:58.998 DEBUG:boto:Headers: {} 2024-04-19T23:41:58.998 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:58.998 DEBUG:boto:Port: 8000 2024-04-19T23:41:58.998 DEBUG:boto:Params: {} 2024-04-19T23:41:58.998 DEBUG:boto:Token: None 2024-04-19T23:41:58.998 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:58 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:58.998 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5kmfG4jJL8lBqlAobjDAP53Ec98= 2024-04-19T23:41:58.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:58 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5kmfG4jJL8lBqlAobjDAP53Ec98=', 'Content-Length': '0'} 2024-04-19T23:41:59.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000056fd2bb2b28c9a91-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.000 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.001 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.001 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.001 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.001 DEBUG:boto:Method: GET 2024-04-19T23:41:59.001 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.001 DEBUG:boto:Data: 2024-04-19T23:41:59.001 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.001 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.001 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.001 DEBUG:boto:Params: {} 2024-04-19T23:41:59.001 DEBUG:boto:Token: None 2024-04-19T23:41:59.002 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.002 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004997b58a67c1ab9e-0066230147-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.004 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.005 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.005 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.005 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.005 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.005 DEBUG:boto:Method: GET 2024-04-19T23:41:59.005 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.005 DEBUG:boto:Data: 2024-04-19T23:41:59.005 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.005 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.006 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.006 DEBUG:boto:Params: {} 2024-04-19T23:41:59.006 DEBUG:boto:Token: None 2024-04-19T23:41:59.006 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.006 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057f18853049541cb-0066230147-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.009 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.009 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.009 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.009 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.009 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.009 DEBUG:boto:Data: 2024-04-19T23:41:59.009 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.010 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.010 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.010 DEBUG:boto:Params: {} 2024-04-19T23:41:59.010 DEBUG:boto:Token: None 2024-04-19T23:41:59.010 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.010 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8= 2024-04-19T23:41:59.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8=', 'Content-Length': '0'} 2024-04-19T23:41:59.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bea243cda39cd447-0066230147-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.012 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.012 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.012 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.012 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.013 DEBUG:boto:Data: 2024-04-19T23:41:59.013 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.013 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.013 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.013 DEBUG:boto:Params: {} 2024-04-19T23:41:59.013 DEBUG:boto:Token: None 2024-04-19T23:41:59.013 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.013 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8= 2024-04-19T23:41:59.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8=', 'Content-Length': '0'} 2024-04-19T23:41:59.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf84b0d1c2d96457-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.015 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.015 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.015 DEBUG:boto:Method: GET 2024-04-19T23:41:59.015 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.016 DEBUG:boto:Data: 2024-04-19T23:41:59.016 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.016 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.016 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.016 DEBUG:boto:Params: {} 2024-04-19T23:41:59.016 DEBUG:boto:Token: None 2024-04-19T23:41:59.016 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.016 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM= 2024-04-19T23:41:59.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM=', 'Content-Length': '0'} 2024-04-19T23:41:59.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008be43f3e3c5cb26-0066230147-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.018 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.019 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.019 DEBUG:boto:Method: GET 2024-04-19T23:41:59.019 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.019 DEBUG:boto:Data: 2024-04-19T23:41:59.019 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.019 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.019 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.019 DEBUG:boto:Params: {} 2024-04-19T23:41:59.019 DEBUG:boto:Token: None 2024-04-19T23:41:59.019 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.020 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM= 2024-04-19T23:41:59.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM=', 'Content-Length': '0'} 2024-04-19T23:41:59.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bd0c215b5a96460d-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.021 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.022 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.022 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.022 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.022 DEBUG:boto:Method: GET 2024-04-19T23:41:59.022 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.022 DEBUG:boto:Data: 2024-04-19T23:41:59.022 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.022 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.022 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.022 DEBUG:boto:Params: {} 2024-04-19T23:41:59.023 DEBUG:boto:Token: None 2024-04-19T23:41:59.023 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.023 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000728f375ae6bb4f0e-0066230147-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.025 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.025 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.026 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.026 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.026 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.026 DEBUG:boto:Method: GET 2024-04-19T23:41:59.026 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.026 DEBUG:boto:Data: 2024-04-19T23:41:59.026 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.026 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.026 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.027 DEBUG:boto:Params: {} 2024-04-19T23:41:59.027 DEBUG:boto:Token: None 2024-04-19T23:41:59.027 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.027 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6ae9fda2a4ad3d4-0066230147-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.030 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.030 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-23 zones={test-zone2, test-zone1} 2024-04-19T23:41:59.030 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-23 zones={test-zone3, test-zone1} 2024-04-19T23:41:59.031 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.031 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.031 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.031 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:59.031 DEBUG:boto:Data: 2024-04-19T23:41:59.031 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.031 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.031 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.031 DEBUG:boto:Params: {} 2024-04-19T23:41:59.031 DEBUG:boto:Token: None 2024-04-19T23:41:59.032 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.032 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XgxOaKg4lk5ssmPyZI9ej3n2l2Y= 2024-04-19T23:41:59.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XgxOaKg4lk5ssmPyZI9ej3n2l2Y=', 'Content-Length': '0'} 2024-04-19T23:41:59.034 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1f67b063236f92d-0066230147-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.034 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.034 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.035 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.035 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:59.035 DEBUG:boto:Data: 2024-04-19T23:41:59.035 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.035 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.035 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.035 DEBUG:boto:Params: {} 2024-04-19T23:41:59.035 DEBUG:boto:Token: None 2024-04-19T23:41:59.035 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.036 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XgxOaKg4lk5ssmPyZI9ej3n2l2Y= 2024-04-19T23:41:59.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XgxOaKg4lk5ssmPyZI9ej3n2l2Y=', 'Content-Length': '0'} 2024-04-19T23:41:59.038 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d178ad29cd81f37e-0066230147-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.038 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:41:59.038 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.038 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.038 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.038 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.038 DEBUG:boto:Method: GET 2024-04-19T23:41:59.038 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.039 DEBUG:boto:Data: 2024-04-19T23:41:59.039 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.039 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.039 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.039 DEBUG:boto:Params: {} 2024-04-19T23:41:59.039 DEBUG:boto:Token: None 2024-04-19T23:41:59.039 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/?versions 2024-04-19T23:41:59.039 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0= 2024-04-19T23:41:59.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0=', 'Content-Length': '0'} 2024-04-19T23:41:59.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfc949d470f9e6d6-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.042 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUfalse2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:59.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:59.043 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:41:59.043 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.043 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.043 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.043 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.043 DEBUG:boto:Method: GET 2024-04-19T23:41:59.043 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.043 DEBUG:boto:Data: 2024-04-19T23:41:59.043 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.043 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.043 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.044 DEBUG:boto:Params: {} 2024-04-19T23:41:59.044 DEBUG:boto:Token: None 2024-04-19T23:41:59.044 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/?versions 2024-04-19T23:41:59.044 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0= 2024-04-19T23:41:59.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0=', 'Content-Length': '0'} 2024-04-19T23:41:59.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014ae604fb5b11dfd-0066230147-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.047 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUfalse2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:59.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:59.047 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.047 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.048 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.048 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.048 DEBUG:boto:Method: GET 2024-04-19T23:41:59.048 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.048 DEBUG:boto:Data: 2024-04-19T23:41:59.048 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.048 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.048 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.048 DEBUG:boto:Params: {} 2024-04-19T23:41:59.048 DEBUG:boto:Token: None 2024-04-19T23:41:59.049 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/?versions 2024-04-19T23:41:59.049 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0= 2024-04-19T23:41:59.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0=', 'Content-Length': '0'} 2024-04-19T23:41:59.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a4d639c1457eb4f-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.052 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUfalse2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.052 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.052 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.052 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.052 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.053 DEBUG:boto:Method: GET 2024-04-19T23:41:59.053 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.053 DEBUG:boto:Data: 2024-04-19T23:41:59.053 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.053 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.053 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.053 DEBUG:boto:Params: {} 2024-04-19T23:41:59.053 DEBUG:boto:Token: None 2024-04-19T23:41:59.053 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/?versions 2024-04-19T23:41:59.053 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0= 2024-04-19T23:41:59.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0=', 'Content-Length': '0'} 2024-04-19T23:41:59.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006880fbd33627776d-0066230147-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.098 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUfalse2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.099 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.099 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.099 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.099 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.099 DEBUG:boto:Method: GET 2024-04-19T23:41:59.099 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.099 DEBUG:boto:Data: 2024-04-19T23:41:59.099 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.100 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.100 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.100 DEBUG:boto:Params: {} 2024-04-19T23:41:59.100 DEBUG:boto:Token: None 2024-04-19T23:41:59.100 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.100 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84= 2024-04-19T23:41:59.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84=', 'Content-Length': '0'} 2024-04-19T23:41:59.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000667a4b7e33fcbc2-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.103 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.103 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.103 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.103 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.103 DEBUG:boto:Method: GET 2024-04-19T23:41:59.103 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.103 DEBUG:boto:Data: 2024-04-19T23:41:59.103 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.103 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.103 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.103 DEBUG:boto:Params: {} 2024-04-19T23:41:59.104 DEBUG:boto:Token: None 2024-04-19T23:41:59.104 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84= 2024-04-19T23:41:59.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84=', 'Content-Length': '0'} 2024-04-19T23:41:59.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000afadb5f9ac580802-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.106 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.106 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.106 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.106 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.107 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.107 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.107 DEBUG:boto:Data: 2024-04-19T23:41:59.107 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.107 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.107 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.107 DEBUG:boto:Params: {} 2024-04-19T23:41:59.107 DEBUG:boto:Token: None 2024-04-19T23:41:59.107 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.107 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:I92g6szo0PLSH9fjJNdTzCdb5V4= 2024-04-19T23:41:59.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:I92g6szo0PLSH9fjJNdTzCdb5V4=', 'Content-Length': '0'} 2024-04-19T23:41:59.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000018c0b9c4c6c5dcfc-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.110 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.110 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.110 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.110 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.110 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.110 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.110 DEBUG:boto:Data: 2024-04-19T23:41:59.111 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.111 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.111 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.111 DEBUG:boto:Params: {} 2024-04-19T23:41:59.111 DEBUG:boto:Token: None 2024-04-19T23:41:59.111 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.111 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:I92g6szo0PLSH9fjJNdTzCdb5V4= 2024-04-19T23:41:59.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:I92g6szo0PLSH9fjJNdTzCdb5V4=', 'Content-Length': '0'} 2024-04-19T23:41:59.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e62f780b542a126c-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.113 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.113 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.113 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.114 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.114 DEBUG:boto:Method: GET 2024-04-19T23:41:59.114 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.114 DEBUG:boto:Data: 2024-04-19T23:41:59.114 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.114 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.114 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.114 DEBUG:boto:Params: {} 2024-04-19T23:41:59.114 DEBUG:boto:Token: None 2024-04-19T23:41:59.114 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.115 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84= 2024-04-19T23:41:59.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84=', 'Content-Length': '0'} 2024-04-19T23:41:59.116 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f2df81ef941ce8fd-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.116 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.117 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.117 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.117 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.117 DEBUG:boto:Method: GET 2024-04-19T23:41:59.117 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.117 DEBUG:boto:Data: 2024-04-19T23:41:59.117 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.117 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.117 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.117 DEBUG:boto:Params: {} 2024-04-19T23:41:59.117 DEBUG:boto:Token: None 2024-04-19T23:41:59.118 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.118 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84= 2024-04-19T23:41:59.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84=', 'Content-Length': '0'} 2024-04-19T23:41:59.120 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab1cc748304e2f84-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.120 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.120 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.120 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.120 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.120 DEBUG:boto:Method: GET 2024-04-19T23:41:59.120 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.120 DEBUG:boto:Data: 2024-04-19T23:41:59.120 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.121 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.121 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.121 DEBUG:boto:Params: {} 2024-04-19T23:41:59.121 DEBUG:boto:Token: None 2024-04-19T23:41:59.121 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.121 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000838f9da285300af6-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.124 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.124 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.124 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.124 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.125 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.125 DEBUG:boto:Method: GET 2024-04-19T23:41:59.125 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.125 DEBUG:boto:Data: 2024-04-19T23:41:59.125 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.125 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.125 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.125 DEBUG:boto:Params: {} 2024-04-19T23:41:59.125 DEBUG:boto:Token: None 2024-04-19T23:41:59.125 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.126 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aabb24311da2d5f-0066230147-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.128 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.128 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.128 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.129 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.129 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.129 DEBUG:boto:Data: 2024-04-19T23:41:59.129 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.129 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.129 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.129 DEBUG:boto:Params: {} 2024-04-19T23:41:59.129 DEBUG:boto:Token: None 2024-04-19T23:41:59.129 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.129 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8= 2024-04-19T23:41:59.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8=', 'Content-Length': '0'} 2024-04-19T23:41:59.131 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000039dd314dea2be932-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.131 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.131 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.131 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.132 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.132 DEBUG:boto:Data: 2024-04-19T23:41:59.132 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.132 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.132 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.132 DEBUG:boto:Params: {} 2024-04-19T23:41:59.132 DEBUG:boto:Token: None 2024-04-19T23:41:59.132 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.132 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8= 2024-04-19T23:41:59.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8=', 'Content-Length': '0'} 2024-04-19T23:41:59.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc65d72e128fcd61-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.135 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.135 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.135 DEBUG:boto:Method: GET 2024-04-19T23:41:59.135 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.135 DEBUG:boto:Data: 2024-04-19T23:41:59.136 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.136 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.136 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.136 DEBUG:boto:Params: {} 2024-04-19T23:41:59.136 DEBUG:boto:Token: None 2024-04-19T23:41:59.137 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.137 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM= 2024-04-19T23:41:59.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM=', 'Content-Length': '0'} 2024-04-19T23:41:59.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019ba00229dd64f28-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.139 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.139 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.140 DEBUG:boto:Method: GET 2024-04-19T23:41:59.140 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.140 DEBUG:boto:Data: 2024-04-19T23:41:59.140 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.140 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.140 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.140 DEBUG:boto:Params: {} 2024-04-19T23:41:59.140 DEBUG:boto:Token: None 2024-04-19T23:41:59.140 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.140 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM= 2024-04-19T23:41:59.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM=', 'Content-Length': '0'} 2024-04-19T23:41:59.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000953c66848bedd46b-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.143 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.143 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.143 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.143 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.143 DEBUG:boto:Method: GET 2024-04-19T23:41:59.143 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.143 DEBUG:boto:Data: 2024-04-19T23:41:59.143 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.143 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.143 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.143 DEBUG:boto:Params: {} 2024-04-19T23:41:59.144 DEBUG:boto:Token: None 2024-04-19T23:41:59.144 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.144 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf598993414b62aa-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.190 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.191 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.191 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.191 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.191 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.191 DEBUG:boto:Method: GET 2024-04-19T23:41:59.191 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.191 DEBUG:boto:Data: 2024-04-19T23:41:59.191 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.192 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.192 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.192 DEBUG:boto:Params: {} 2024-04-19T23:41:59.192 DEBUG:boto:Token: None 2024-04-19T23:41:59.192 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.192 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002caa75eb239e080a-0066230147-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.195 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.195 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.195 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.195 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.196 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.196 DEBUG:boto:Method: GET 2024-04-19T23:41:59.196 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.196 DEBUG:boto:Data: 2024-04-19T23:41:59.196 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.196 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.196 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.196 DEBUG:boto:Params: {} 2024-04-19T23:41:59.196 DEBUG:boto:Token: None 2024-04-19T23:41:59.196 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.197 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY= 2024-04-19T23:41:59.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY=', 'Content-Length': '0'} 2024-04-19T23:41:59.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000058ad3d1548570a45-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.199 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.199 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.199 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.199 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.199 DEBUG:boto:Method: GET 2024-04-19T23:41:59.200 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.200 DEBUG:boto:Data: 2024-04-19T23:41:59.200 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.200 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.200 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.200 DEBUG:boto:Params: {} 2024-04-19T23:41:59.200 DEBUG:boto:Token: None 2024-04-19T23:41:59.200 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.200 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY= 2024-04-19T23:41:59.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY=', 'Content-Length': '0'} 2024-04-19T23:41:59.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054640c58173df698-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.202 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.203 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.203 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.203 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.203 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.203 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.203 DEBUG:boto:Data: 2024-04-19T23:41:59.203 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.203 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.203 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.203 DEBUG:boto:Params: {} 2024-04-19T23:41:59.203 DEBUG:boto:Token: None 2024-04-19T23:41:59.204 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.204 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7atElDinbqTclQBobU1Tuk8+1pQ= 2024-04-19T23:41:59.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7atElDinbqTclQBobU1Tuk8+1pQ=', 'Content-Length': '0'} 2024-04-19T23:41:59.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ae940401b4d8ad31-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.206 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.206 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.206 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.206 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.206 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.206 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.206 DEBUG:boto:Data: 2024-04-19T23:41:59.206 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.206 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.206 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.207 DEBUG:boto:Params: {} 2024-04-19T23:41:59.207 DEBUG:boto:Token: None 2024-04-19T23:41:59.207 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.207 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7atElDinbqTclQBobU1Tuk8+1pQ= 2024-04-19T23:41:59.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7atElDinbqTclQBobU1Tuk8+1pQ=', 'Content-Length': '0'} 2024-04-19T23:41:59.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001ed07db39897d34a-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.209 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.210 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.210 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.210 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.210 DEBUG:boto:Method: GET 2024-04-19T23:41:59.210 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.210 DEBUG:boto:Data: 2024-04-19T23:41:59.210 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.210 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.210 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.210 DEBUG:boto:Params: {} 2024-04-19T23:41:59.211 DEBUG:boto:Token: None 2024-04-19T23:41:59.211 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.211 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY= 2024-04-19T23:41:59.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY=', 'Content-Length': '0'} 2024-04-19T23:41:59.213 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b00b1bb99268b1e2-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.214 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.214 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.214 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.214 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.214 DEBUG:boto:Method: GET 2024-04-19T23:41:59.214 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.214 DEBUG:boto:Data: 2024-04-19T23:41:59.214 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.214 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.214 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.214 DEBUG:boto:Params: {} 2024-04-19T23:41:59.215 DEBUG:boto:Token: None 2024-04-19T23:41:59.215 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.215 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY= 2024-04-19T23:41:59.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY=', 'Content-Length': '0'} 2024-04-19T23:41:59.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee695d8055518f54-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.217 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.217 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.217 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.217 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.218 DEBUG:boto:Method: GET 2024-04-19T23:41:59.218 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.218 DEBUG:boto:Data: 2024-04-19T23:41:59.218 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.218 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.218 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.218 DEBUG:boto:Params: {} 2024-04-19T23:41:59.218 DEBUG:boto:Token: None 2024-04-19T23:41:59.218 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.218 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000091eaef7d667850b-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.221 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.222 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.222 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.222 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.222 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.222 DEBUG:boto:Method: GET 2024-04-19T23:41:59.222 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.222 DEBUG:boto:Data: 2024-04-19T23:41:59.222 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.222 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.222 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.222 DEBUG:boto:Params: {} 2024-04-19T23:41:59.223 DEBUG:boto:Token: None 2024-04-19T23:41:59.223 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.223 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060cf0431eafe3659-0066230147-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.266 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.267 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.267 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.267 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.267 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.267 DEBUG:boto:Data: 2024-04-19T23:41:59.267 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.267 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.267 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.267 DEBUG:boto:Params: {} 2024-04-19T23:41:59.268 DEBUG:boto:Token: None 2024-04-19T23:41:59.268 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.268 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8= 2024-04-19T23:41:59.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8=', 'Content-Length': '0'} 2024-04-19T23:41:59.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005b234118e282004e-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.270 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.270 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.270 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.270 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.271 DEBUG:boto:Data: 2024-04-19T23:41:59.271 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.271 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.271 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.271 DEBUG:boto:Params: {} 2024-04-19T23:41:59.271 DEBUG:boto:Token: None 2024-04-19T23:41:59.271 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.271 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8= 2024-04-19T23:41:59.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8=', 'Content-Length': '0'} 2024-04-19T23:41:59.273 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000269a218807b6f434-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.274 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.274 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.274 DEBUG:boto:Method: GET 2024-04-19T23:41:59.274 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.274 DEBUG:boto:Data: 2024-04-19T23:41:59.274 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.274 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.274 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.274 DEBUG:boto:Params: {} 2024-04-19T23:41:59.275 DEBUG:boto:Token: None 2024-04-19T23:41:59.275 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.275 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM= 2024-04-19T23:41:59.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM=', 'Content-Length': '0'} 2024-04-19T23:41:59.276 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077d216932449c449-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.277 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.277 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.277 DEBUG:boto:Method: GET 2024-04-19T23:41:59.277 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.277 DEBUG:boto:Data: 2024-04-19T23:41:59.277 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.277 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.277 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.277 DEBUG:boto:Params: {} 2024-04-19T23:41:59.278 DEBUG:boto:Token: None 2024-04-19T23:41:59.278 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.278 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM= 2024-04-19T23:41:59.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM=', 'Content-Length': '0'} 2024-04-19T23:41:59.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000099879ede478230d7-0066230147-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.280 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.280 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.280 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.280 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.280 DEBUG:boto:Method: GET 2024-04-19T23:41:59.280 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.280 DEBUG:boto:Data: 2024-04-19T23:41:59.280 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.280 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.281 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.281 DEBUG:boto:Params: {} 2024-04-19T23:41:59.281 DEBUG:boto:Token: None 2024-04-19T23:41:59.281 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.281 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6f916e33ab97f13-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.284 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.284 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.284 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.284 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.284 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.284 DEBUG:boto:Method: GET 2024-04-19T23:41:59.284 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.284 DEBUG:boto:Data: 2024-04-19T23:41:59.285 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.285 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.285 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.285 DEBUG:boto:Params: {} 2024-04-19T23:41:59.285 DEBUG:boto:Token: None 2024-04-19T23:41:59.285 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.285 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e316aa493db44f2b-0066230147-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.288 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.288 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-23 zones={test-zone3, test-zone1} 2024-04-19T23:41:59.289 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-23 zones={test-zone3, test-zone2} 2024-04-19T23:41:59.289 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.289 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.289 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.289 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:59.289 DEBUG:boto:Data: 2024-04-19T23:41:59.289 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.289 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.289 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.289 DEBUG:boto:Params: {} 2024-04-19T23:41:59.290 DEBUG:boto:Token: None 2024-04-19T23:41:59.290 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.290 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XgxOaKg4lk5ssmPyZI9ej3n2l2Y= 2024-04-19T23:41:59.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XgxOaKg4lk5ssmPyZI9ej3n2l2Y=', 'Content-Length': '0'} 2024-04-19T23:41:59.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d83d4491dbe70ad-0066230147-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.292 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.292 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.293 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.293 DEBUG:boto:Path: /aktlji-23/ 2024-04-19T23:41:59.293 DEBUG:boto:Data: 2024-04-19T23:41:59.293 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.293 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.293 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.293 DEBUG:boto:Params: {} 2024-04-19T23:41:59.293 DEBUG:boto:Token: None 2024-04-19T23:41:59.293 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.293 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XgxOaKg4lk5ssmPyZI9ej3n2l2Y= 2024-04-19T23:41:59.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XgxOaKg4lk5ssmPyZI9ej3n2l2Y=', 'Content-Length': '0'} 2024-04-19T23:41:59.296 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002099429959622ff5-0066230147-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.296 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:41:59.296 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.296 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.296 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.297 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.297 DEBUG:boto:Method: GET 2024-04-19T23:41:59.297 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.297 DEBUG:boto:Data: 2024-04-19T23:41:59.297 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.297 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.297 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.297 DEBUG:boto:Params: {} 2024-04-19T23:41:59.297 DEBUG:boto:Token: None 2024-04-19T23:41:59.297 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/?versions 2024-04-19T23:41:59.297 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0= 2024-04-19T23:41:59.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0=', 'Content-Length': '0'} 2024-04-19T23:41:59.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000154a8e05ce3a7739-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.342 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUfalse2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:59.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:59.343 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:41:59.343 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.343 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.343 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.343 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.344 DEBUG:boto:Method: GET 2024-04-19T23:41:59.344 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.344 DEBUG:boto:Data: 2024-04-19T23:41:59.344 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.344 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.344 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.344 DEBUG:boto:Params: {} 2024-04-19T23:41:59.344 DEBUG:boto:Token: None 2024-04-19T23:41:59.344 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/?versions 2024-04-19T23:41:59.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0= 2024-04-19T23:41:59.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0=', 'Content-Length': '0'} 2024-04-19T23:41:59.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5915f3adfc37f7c-0066230147-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.347 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUfalse2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:59.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:41:59.348 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.348 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.348 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.348 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.348 DEBUG:boto:Method: GET 2024-04-19T23:41:59.348 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.348 DEBUG:boto:Data: 2024-04-19T23:41:59.349 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.349 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.349 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.349 DEBUG:boto:Params: {} 2024-04-19T23:41:59.349 DEBUG:boto:Token: None 2024-04-19T23:41:59.349 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/?versions 2024-04-19T23:41:59.349 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0= 2024-04-19T23:41:59.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0=', 'Content-Length': '0'} 2024-04-19T23:41:59.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dff569f5522cb7ad-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.352 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUfalse2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.352 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.352 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.352 DEBUG:boto:path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.353 DEBUG:boto:auth_path=/aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.353 DEBUG:boto:Method: GET 2024-04-19T23:41:59.353 DEBUG:boto:Path: /aktlji-23/?versions&max-keys=999 2024-04-19T23:41:59.353 DEBUG:boto:Data: 2024-04-19T23:41:59.353 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.353 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.353 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.353 DEBUG:boto:Params: {} 2024-04-19T23:41:59.353 DEBUG:boto:Token: None 2024-04-19T23:41:59.353 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/?versions 2024-04-19T23:41:59.353 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0= 2024-04-19T23:41:59.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4Qj3RRgsA6BTyXfECdvy3Vy4WT0=', 'Content-Length': '0'} 2024-04-19T23:41:59.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012f965ae1b1fc368-0066230147-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.398 DEBUG:boto:b'aktlji-23999falseobjnulltrue2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjD8UzudkCnyaXCs61Tf1oa4dsJgWInRUfalse2024-04-19T23:40:09.809Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.399 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.399 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.399 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.399 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.399 DEBUG:boto:Method: GET 2024-04-19T23:41:59.399 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.399 DEBUG:boto:Data: 2024-04-19T23:41:59.399 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.400 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.400 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.400 DEBUG:boto:Params: {} 2024-04-19T23:41:59.400 DEBUG:boto:Token: None 2024-04-19T23:41:59.400 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.400 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84= 2024-04-19T23:41:59.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84=', 'Content-Length': '0'} 2024-04-19T23:41:59.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c1c6e377c9e73e4b-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.402 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.402 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.403 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.403 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.403 DEBUG:boto:Method: GET 2024-04-19T23:41:59.403 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.403 DEBUG:boto:Data: 2024-04-19T23:41:59.403 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.403 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.403 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.403 DEBUG:boto:Params: {} 2024-04-19T23:41:59.403 DEBUG:boto:Token: None 2024-04-19T23:41:59.403 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.404 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84= 2024-04-19T23:41:59.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84=', 'Content-Length': '0'} 2024-04-19T23:41:59.405 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000442accc01b26bc04-0066230147-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.406 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.406 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.406 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.406 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.406 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.406 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.406 DEBUG:boto:Data: 2024-04-19T23:41:59.406 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.406 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.406 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.407 DEBUG:boto:Params: {} 2024-04-19T23:41:59.407 DEBUG:boto:Token: None 2024-04-19T23:41:59.407 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.407 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:I92g6szo0PLSH9fjJNdTzCdb5V4= 2024-04-19T23:41:59.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:I92g6szo0PLSH9fjJNdTzCdb5V4=', 'Content-Length': '0'} 2024-04-19T23:41:59.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009a2d43493a86796f-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.409 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.409 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.410 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.410 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.410 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.410 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.410 DEBUG:boto:Data: 2024-04-19T23:41:59.410 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.410 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.410 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.410 DEBUG:boto:Params: {} 2024-04-19T23:41:59.410 DEBUG:boto:Token: None 2024-04-19T23:41:59.411 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.411 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:I92g6szo0PLSH9fjJNdTzCdb5V4= 2024-04-19T23:41:59.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:I92g6szo0PLSH9fjJNdTzCdb5V4=', 'Content-Length': '0'} 2024-04-19T23:41:59.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b047f68847f7def7-0066230147-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.413 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.413 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.413 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.413 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.413 DEBUG:boto:Method: GET 2024-04-19T23:41:59.413 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.413 DEBUG:boto:Data: 2024-04-19T23:41:59.414 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.414 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.414 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.414 DEBUG:boto:Params: {} 2024-04-19T23:41:59.414 DEBUG:boto:Token: None 2024-04-19T23:41:59.414 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.414 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84= 2024-04-19T23:41:59.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84=', 'Content-Length': '0'} 2024-04-19T23:41:59.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000056cc18a3879d71a-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.417 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.417 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.417 DEBUG:boto:path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.417 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=null 2024-04-19T23:41:59.417 DEBUG:boto:Method: GET 2024-04-19T23:41:59.417 DEBUG:boto:Path: /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.417 DEBUG:boto:Data: 2024-04-19T23:41:59.417 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.417 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.417 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.417 DEBUG:boto:Params: {} 2024-04-19T23:41:59.417 DEBUG:boto:Token: None 2024-04-19T23:41:59.418 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=null 2024-04-19T23:41:59.418 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84= 2024-04-19T23:41:59.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uXM6xtzjuwrRj6aa1ovZVoxw/84=', 'Content-Length': '0'} 2024-04-19T23:41:59.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f3553de6f55cd89-0066230147-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.420 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.420 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.420 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.420 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.420 DEBUG:boto:Method: GET 2024-04-19T23:41:59.420 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.420 DEBUG:boto:Data: 2024-04-19T23:41:59.420 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.420 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.421 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.421 DEBUG:boto:Params: {} 2024-04-19T23:41:59.421 DEBUG:boto:Token: None 2024-04-19T23:41:59.421 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.421 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e9a97d1546274e0-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.424 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.424 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.424 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.424 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.424 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.424 DEBUG:boto:Method: GET 2024-04-19T23:41:59.424 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.424 DEBUG:boto:Data: 2024-04-19T23:41:59.424 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.425 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.425 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.425 DEBUG:boto:Params: {} 2024-04-19T23:41:59.425 DEBUG:boto:Token: None 2024-04-19T23:41:59.425 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.425 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7d88efcd5c84a6f-0066230147-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.428 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.428 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.428 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.428 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.428 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.428 DEBUG:boto:Data: 2024-04-19T23:41:59.429 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.429 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.429 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.429 DEBUG:boto:Params: {} 2024-04-19T23:41:59.429 DEBUG:boto:Token: None 2024-04-19T23:41:59.429 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.429 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8= 2024-04-19T23:41:59.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8=', 'Content-Length': '0'} 2024-04-19T23:41:59.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004b3dc9c15e1baa35-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.431 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.431 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.431 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.432 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.432 DEBUG:boto:Data: 2024-04-19T23:41:59.432 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.432 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.432 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.432 DEBUG:boto:Params: {} 2024-04-19T23:41:59.432 DEBUG:boto:Token: None 2024-04-19T23:41:59.432 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.432 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8= 2024-04-19T23:41:59.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8=', 'Content-Length': '0'} 2024-04-19T23:41:59.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000006f6e9635ccd6868-0066230147-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.435 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.435 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.435 DEBUG:boto:Method: GET 2024-04-19T23:41:59.435 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.435 DEBUG:boto:Data: 2024-04-19T23:41:59.435 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.435 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.435 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.435 DEBUG:boto:Params: {} 2024-04-19T23:41:59.435 DEBUG:boto:Token: None 2024-04-19T23:41:59.436 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.436 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM= 2024-04-19T23:41:59.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM=', 'Content-Length': '0'} 2024-04-19T23:41:59.437 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000076e5a2e060bbb898-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.438 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.438 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.438 DEBUG:boto:Method: GET 2024-04-19T23:41:59.438 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.438 DEBUG:boto:Data: 2024-04-19T23:41:59.438 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.438 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.438 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.438 DEBUG:boto:Params: {} 2024-04-19T23:41:59.438 DEBUG:boto:Token: None 2024-04-19T23:41:59.439 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.439 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM= 2024-04-19T23:41:59.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM=', 'Content-Length': '0'} 2024-04-19T23:41:59.440 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005daed3ed0afd31d8-0066230147-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.441 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.441 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.441 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.441 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.441 DEBUG:boto:Method: GET 2024-04-19T23:41:59.441 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.441 DEBUG:boto:Data: 2024-04-19T23:41:59.441 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.441 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.441 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.441 DEBUG:boto:Params: {} 2024-04-19T23:41:59.442 DEBUG:boto:Token: None 2024-04-19T23:41:59.442 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.442 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f208b0b0e8f17090-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.490 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.491 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.491 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.491 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.491 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.491 DEBUG:boto:Method: GET 2024-04-19T23:41:59.491 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.491 DEBUG:boto:Data: 2024-04-19T23:41:59.491 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.492 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.492 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.492 DEBUG:boto:Params: {} 2024-04-19T23:41:59.492 DEBUG:boto:Token: None 2024-04-19T23:41:59.492 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.492 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014491f94c84cb687-0066230147-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.495 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.496 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.496 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.496 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.496 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.496 DEBUG:boto:Method: GET 2024-04-19T23:41:59.496 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.496 DEBUG:boto:Data: 2024-04-19T23:41:59.496 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.496 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.496 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.496 DEBUG:boto:Params: {} 2024-04-19T23:41:59.497 DEBUG:boto:Token: None 2024-04-19T23:41:59.497 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.497 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY= 2024-04-19T23:41:59.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY=', 'Content-Length': '0'} 2024-04-19T23:41:59.499 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a46eb67f2f667390-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.499 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.499 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.499 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.499 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.499 DEBUG:boto:Method: GET 2024-04-19T23:41:59.499 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.500 DEBUG:boto:Data: 2024-04-19T23:41:59.500 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.500 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.500 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.500 DEBUG:boto:Params: {} 2024-04-19T23:41:59.500 DEBUG:boto:Token: None 2024-04-19T23:41:59.500 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.500 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY= 2024-04-19T23:41:59.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY=', 'Content-Length': '0'} 2024-04-19T23:41:59.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003176db251e7ffdba-0066230147-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.502 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.502 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.503 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.503 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.503 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.503 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.503 DEBUG:boto:Data: 2024-04-19T23:41:59.503 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.503 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.503 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.503 DEBUG:boto:Params: {} 2024-04-19T23:41:59.503 DEBUG:boto:Token: None 2024-04-19T23:41:59.503 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.504 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7atElDinbqTclQBobU1Tuk8+1pQ= 2024-04-19T23:41:59.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7atElDinbqTclQBobU1Tuk8+1pQ=', 'Content-Length': '0'} 2024-04-19T23:41:59.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ef87d1c3bda3191-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.505 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.506 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.506 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.506 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.506 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.506 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.506 DEBUG:boto:Data: 2024-04-19T23:41:59.506 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.506 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.506 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.506 DEBUG:boto:Params: {} 2024-04-19T23:41:59.507 DEBUG:boto:Token: None 2024-04-19T23:41:59.507 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.507 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7atElDinbqTclQBobU1Tuk8+1pQ= 2024-04-19T23:41:59.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7atElDinbqTclQBobU1Tuk8+1pQ=', 'Content-Length': '0'} 2024-04-19T23:41:59.509 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c166cfdf841c978-0066230147-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.509 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.509 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.509 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.509 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.510 DEBUG:boto:Method: GET 2024-04-19T23:41:59.510 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.510 DEBUG:boto:Data: 2024-04-19T23:41:59.510 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.510 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.510 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.510 DEBUG:boto:Params: {} 2024-04-19T23:41:59.510 DEBUG:boto:Token: None 2024-04-19T23:41:59.510 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.510 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY= 2024-04-19T23:41:59.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY=', 'Content-Length': '0'} 2024-04-19T23:41:59.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fe30c5216efcf70d-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.512 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.513 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.513 DEBUG:boto:path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.513 DEBUG:boto:auth_path=/aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.513 DEBUG:boto:Method: GET 2024-04-19T23:41:59.513 DEBUG:boto:Path: /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.513 DEBUG:boto:Data: 2024-04-19T23:41:59.513 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.513 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.513 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.513 DEBUG:boto:Params: {} 2024-04-19T23:41:59.513 DEBUG:boto:Token: None 2024-04-19T23:41:59.514 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj?versionId=D8UzudkCnyaXCs61Tf1oa4dsJgWInRU 2024-04-19T23:41:59.514 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY= 2024-04-19T23:41:59.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rQTM+cS+DKxiEDVjbC+aWw/KIqY=', 'Content-Length': '0'} 2024-04-19T23:41:59.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:40:09 GMT'), ('x-amz-version-id', 'D8UzudkCnyaXCs61Tf1oa4dsJgWInRU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c5cd327d2e3325d-0066230147-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.516 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.516 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.516 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.516 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.516 DEBUG:boto:Method: GET 2024-04-19T23:41:59.516 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.516 DEBUG:boto:Data: 2024-04-19T23:41:59.516 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.516 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.516 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.516 DEBUG:boto:Params: {} 2024-04-19T23:41:59.517 DEBUG:boto:Token: None 2024-04-19T23:41:59.517 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.517 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf6fb70720f6e249-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.520 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.520 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.520 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.520 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.520 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.520 DEBUG:boto:Method: GET 2024-04-19T23:41:59.520 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.520 DEBUG:boto:Data: 2024-04-19T23:41:59.521 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.521 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.521 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.521 DEBUG:boto:Params: {} 2024-04-19T23:41:59.521 DEBUG:boto:Token: None 2024-04-19T23:41:59.521 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.521 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa4c340b38ab680c-0066230147-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.524 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.524 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.524 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.524 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.524 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.525 DEBUG:boto:Data: 2024-04-19T23:41:59.525 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.525 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.525 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.525 DEBUG:boto:Params: {} 2024-04-19T23:41:59.525 DEBUG:boto:Token: None 2024-04-19T23:41:59.525 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.525 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8= 2024-04-19T23:41:59.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8=', 'Content-Length': '0'} 2024-04-19T23:41:59.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000095893027ed4ec87d-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.527 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.527 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.527 DEBUG:boto:Method: HEAD 2024-04-19T23:41:59.528 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.528 DEBUG:boto:Data: 2024-04-19T23:41:59.528 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.528 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.528 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.528 DEBUG:boto:Params: {} 2024-04-19T23:41:59.528 DEBUG:boto:Token: None 2024-04-19T23:41:59.528 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.528 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8= 2024-04-19T23:41:59.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0JLEyKKsDtX/SGrfW7wOk/tMBx8=', 'Content-Length': '0'} 2024-04-19T23:41:59.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f6f7a7f69e8d8036-0066230147-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:41:59.531 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.531 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.531 DEBUG:boto:Method: GET 2024-04-19T23:41:59.531 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.531 DEBUG:boto:Data: 2024-04-19T23:41:59.531 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.531 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.531 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.531 DEBUG:boto:Params: {} 2024-04-19T23:41:59.531 DEBUG:boto:Token: None 2024-04-19T23:41:59.532 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.532 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM= 2024-04-19T23:41:59.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM=', 'Content-Length': '0'} 2024-04-19T23:41:59.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c5cde19188159be-0066230147-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.534 DEBUG:boto:path=/aktlji-23/obj 2024-04-19T23:41:59.534 DEBUG:boto:auth_path=/aktlji-23/obj 2024-04-19T23:41:59.534 DEBUG:boto:Method: GET 2024-04-19T23:41:59.534 DEBUG:boto:Path: /aktlji-23/obj 2024-04-19T23:41:59.534 DEBUG:boto:Data: 2024-04-19T23:41:59.534 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.535 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.535 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.535 DEBUG:boto:Params: {} 2024-04-19T23:41:59.535 DEBUG:boto:Token: None 2024-04-19T23:41:59.535 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/obj 2024-04-19T23:41:59.535 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM= 2024-04-19T23:41:59.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/R0hQyaLRv9s4ojthhpokk2i1dM=', 'Content-Length': '0'} 2024-04-19T23:41:59.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:41:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009dc40ca96d28d2dd-0066230147-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.537 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.538 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.538 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.538 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.538 DEBUG:boto:Method: GET 2024-04-19T23:41:59.538 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.538 DEBUG:boto:Data: 2024-04-19T23:41:59.538 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.538 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.538 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.538 DEBUG:boto:Params: {} 2024-04-19T23:41:59.538 DEBUG:boto:Token: None 2024-04-19T23:41:59.539 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.539 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000487665c1084c7db6-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.586 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.587 DEBUG:boto:path=/aktlji-23/ 2024-04-19T23:41:59.587 DEBUG:boto:auth_path=/aktlji-23/ 2024-04-19T23:41:59.587 DEBUG:boto:path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.587 DEBUG:boto:auth_path=/aktlji-23/?prefix=obj 2024-04-19T23:41:59.587 DEBUG:boto:Method: GET 2024-04-19T23:41:59.587 DEBUG:boto:Path: /aktlji-23/?prefix=obj 2024-04-19T23:41:59.588 DEBUG:boto:Data: 2024-04-19T23:41:59.588 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.588 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.588 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.588 DEBUG:boto:Params: {} 2024-04-19T23:41:59.588 DEBUG:boto:Token: None 2024-04-19T23:41:59.588 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:41:59 GMT /aktlji-23/ 2024-04-19T23:41:59.588 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw= 2024-04-19T23:41:59.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YBIb8AxAaviBXi2/ffdhfNsKnFw=', 'Content-Length': '0'} 2024-04-19T23:41:59.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f84c4606cdf7d8e-0066230147-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.591 DEBUG:boto:b'aktlji-23obj1000falseobj2024-04-19T23:41:07.071Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:41:59.591 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-23 zones={test-zone3, test-zone2} 2024-04-19T23:41:59.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2024-04-19T23:41:59.593 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-24 2024-04-19T23:41:59.594 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:41:59.594 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:41:59.594 DEBUG:boto:Method: PUT 2024-04-19T23:41:59.594 DEBUG:boto:Path: /aktlji-24/ 2024-04-19T23:41:59.594 DEBUG:boto:Data: 2024-04-19T23:41:59.594 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.594 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.594 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.594 DEBUG:boto:Params: {} 2024-04-19T23:41:59.595 DEBUG:boto:Token: None 2024-04-19T23:41:59.595 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:41:59 GMT /aktlji-24/ 2024-04-19T23:41:59.595 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Wn+vRZX+HaCeDuj5b2JBD3cxU60= 2024-04-19T23:41:59.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Wn+vRZX+HaCeDuj5b2JBD3cxU60=', 'Content-Length': '0'} 2024-04-19T23:41:59.612 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002b01f9b17158b138-0066230147-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.612 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-25 2024-04-19T23:41:59.612 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:41:59.612 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:41:59.612 DEBUG:boto:Method: PUT 2024-04-19T23:41:59.612 DEBUG:boto:Path: /aktlji-25/ 2024-04-19T23:41:59.612 DEBUG:boto:Data: 2024-04-19T23:41:59.613 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.613 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.613 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.613 DEBUG:boto:Params: {} 2024-04-19T23:41:59.613 DEBUG:boto:Token: None 2024-04-19T23:41:59.613 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:41:59 GMT /aktlji-25/ 2024-04-19T23:41:59.614 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rNVKuZaSjRPWHO9YVVru6J1Nv4E= 2024-04-19T23:41:59.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rNVKuZaSjRPWHO9YVVru6J1Nv4E=', 'Content-Length': '0'} 2024-04-19T23:41:59.643 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000214701ee7483fe39-0066230147-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.643 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-26 2024-04-19T23:41:59.643 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:41:59.643 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:41:59.644 DEBUG:boto:Method: PUT 2024-04-19T23:41:59.644 DEBUG:boto:Path: /aktlji-26/ 2024-04-19T23:41:59.644 DEBUG:boto:Data: 2024-04-19T23:41:59.644 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.644 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.644 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.644 DEBUG:boto:Params: {} 2024-04-19T23:41:59.644 DEBUG:boto:Token: None 2024-04-19T23:41:59.644 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:41:59 GMT /aktlji-26/ 2024-04-19T23:41:59.645 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1KUz8MlvIUHgD37Ic2ujnSmvqmA= 2024-04-19T23:41:59.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1KUz8MlvIUHgD37Ic2ujnSmvqmA=', 'Content-Length': '0'} 2024-04-19T23:41:59.673 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000048d25e47330ee6d0-0066230147-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.673 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:41:59.673 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:41:59.673 DEBUG:boto:path=/aktlji-24/?versioning 2024-04-19T23:41:59.673 DEBUG:boto:auth_path=/aktlji-24/?versioning 2024-04-19T23:41:59.673 DEBUG:boto:Method: PUT 2024-04-19T23:41:59.673 DEBUG:boto:Path: /aktlji-24/?versioning 2024-04-19T23:41:59.673 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:41:59.674 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.674 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:41:59.674 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.674 DEBUG:boto:Params: {} 2024-04-19T23:41:59.674 DEBUG:boto:Token: None 2024-04-19T23:41:59.674 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:41:59 GMT /aktlji-24/?versioning 2024-04-19T23:41:59.674 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2Ru/SBsbuoL4QRY+f/7MgK/8Ybw= 2024-04-19T23:41:59.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Ru/SBsbuoL4QRY+f/7MgK/8Ybw=', 'Content-Length': '228'} 2024-04-19T23:41:59.680 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007090c863a06692f7-0066230147-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.680 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:41:59.680 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:41:59.680 DEBUG:boto:path=/aktlji-25/?versioning 2024-04-19T23:41:59.681 DEBUG:boto:auth_path=/aktlji-25/?versioning 2024-04-19T23:41:59.681 DEBUG:boto:Method: PUT 2024-04-19T23:41:59.681 DEBUG:boto:Path: /aktlji-25/?versioning 2024-04-19T23:41:59.681 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:41:59.681 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.681 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:41:59.681 DEBUG:boto:Port: 8000 2024-04-19T23:41:59.681 DEBUG:boto:Params: {} 2024-04-19T23:41:59.681 DEBUG:boto:Token: None 2024-04-19T23:41:59.681 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:41:59 GMT /aktlji-25/?versioning 2024-04-19T23:41:59.682 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YECWBb2jX3TaJ554m3PFKyNOMoA= 2024-04-19T23:41:59.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YECWBb2jX3TaJ554m3PFKyNOMoA=', 'Content-Length': '228'} 2024-04-19T23:41:59.691 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b7b222a4413afaaa-0066230147-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.691 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:41:59.691 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:41:59.691 DEBUG:boto:path=/aktlji-26/?versioning 2024-04-19T23:41:59.691 DEBUG:boto:auth_path=/aktlji-26/?versioning 2024-04-19T23:41:59.691 DEBUG:boto:Method: PUT 2024-04-19T23:41:59.691 DEBUG:boto:Path: /aktlji-26/?versioning 2024-04-19T23:41:59.691 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:41:59.691 DEBUG:boto:Headers: {} 2024-04-19T23:41:59.691 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:41:59.691 DEBUG:boto:Port: 8001 2024-04-19T23:41:59.692 DEBUG:boto:Params: {} 2024-04-19T23:41:59.692 DEBUG:boto:Token: None 2024-04-19T23:41:59.692 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:41:59 GMT /aktlji-26/?versioning 2024-04-19T23:41:59.692 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sxLNYG+VAj3sn7yvaNeegHqupRM= 2024-04-19T23:41:59.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:41:59 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sxLNYG+VAj3sn7yvaNeegHqupRM=', 'Content-Length': '228'} 2024-04-19T23:41:59.700 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f6f2c9ae1f6625bc-0066230147-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:41:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:41:59.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:41:59.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:41:59.701 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:41:59.756 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:41:59.756 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:41:59.792 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:41:59.821 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:41:59.821 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:41:59.822 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:41:59.822 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:41:59.822 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:41:59.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:41:59.822 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:41:59.822 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:41:59.822 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:41:59.822 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:41:59.822 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713570119.700556_248.1", 2024-04-19T23:41:59.823 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:41:59.700556Z" 2024-04-19T23:41:59.823 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:41:59.823 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:41:59.823 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:41:59.823 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:41:59.823 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:41:59.823 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:41:59.824 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570119.700556_248.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:41:59.824 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:41:59.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:41:59.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:41:59.825 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:41:59.884 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:41:59.884 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:41:59.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:59.920+0000 7fc2a91eb780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:41:59.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:41:59.922+0000 7fc2a91eb780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:41:59.935 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:41:59.935 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:41:59.935 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:41:59.935 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:41:59.935 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:41:59.935 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:41:59.935 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:59.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:59.935 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:41:59.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:41:59.935 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:41:59.936 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:41:59.936 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:41:59.936 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:41:59.936 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:41:59.936 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:41:59.936 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:41:59.936 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:41:59.936 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:59.936 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:59.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:59.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:41:59.936 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:41:59.937 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:41:59.937 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:41:59.937 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:41:59.937 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:41:59.937 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:41:59.937 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:41:59.937 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:41:59.937 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:59.937 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:59.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:59.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:41:59.937 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570119.608273_232.1", 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:41:59.608273Z", 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:41:59.938 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:41:59.939 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:41:59.939 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:41:59.939 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:41:59.939 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:41:59.939 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:41:59.939 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:41:59.939 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:41:59.939 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:59.939 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:41:59.939 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:41:59.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:59.940 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:41:59.940 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:41:59.940 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:41:59.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:41:59.940 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:41:59Z" 2024-04-19T23:41:59.940 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:41:59.941 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570119.608273_232.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:41:59.608273Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:41:59Z" } 2024-04-19T23:41:59.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570119.608273_232.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:41:59.608273Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:41:59.941 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570119.700556_248.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:41:59.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570119.608273_232.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:41:59.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570119.700556_248.1 target=1_1713570119.608273_232.1 2024-04-19T23:42:04.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:04.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:04.943 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:42:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:05.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:05.040+0000 7fc2a7390780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:42:05.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:05.041+0000 7fc2a7390780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:42:05.053 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:42:05.053 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:42:05.053 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:42:05.053 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:42:05.053 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:42:05.054 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:42:05.054 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:05.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:05.054 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:42:05.054 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:05.054 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:42:05.054 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:05.054 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:05.054 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:42:05.054 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:05.055 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:42:05.055 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:05.055 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:42:05.055 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:05.055 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:05.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:05.055 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:05.055 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:42:05.055 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:05.055 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:42:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:42:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:42:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:42:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:05.056 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570119.612829_234.1", 2024-04-19T23:42:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:42:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:41:59.612829Z", 2024-04-19T23:42:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:42:05.057 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:42:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:42:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:42:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:42:05.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:42:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:42:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:42:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:05.059 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:42:05Z" 2024-04-19T23:42:05.059 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:42:05.060 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570119.612829_234.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:41:59.612829Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:42:05Z" } 2024-04-19T23:42:05.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570119.612829_234.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:41:59.612829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:42:05.060 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570119.700556_248.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:05.060 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570119.612829_234.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:05.060 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570119.700556_248.1 target=1_1713570119.612829_234.1 2024-04-19T23:42:10.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:10.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:10.062 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:42:10.123 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:10.123 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:10.159+0000 7f367a7ae780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:42:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:10.161+0000 7f367a7ae780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:42:10.174 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:42:10.174 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:42:10.174 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:42:10.174 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:42:10.174 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:42:10.175 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:42:10.175 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:10.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:10.175 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:42:10.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:10.175 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:42:10.175 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:10.175 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:10.175 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:42:10.175 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:42:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:42:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:10.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:42:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:42:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:42:10.177 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:42:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:10.178 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:42:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570119.700556_248.1", 2024-04-19T23:42:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:42:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:41:59.700556Z", 2024-04-19T23:42:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:10.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:42:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:42:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:42:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:42:10.180 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:10.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:10.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:10.181 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:42:10.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:10.181 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:42:10.181 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:42:10.181 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:42:10.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:10.181 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:42:10Z" 2024-04-19T23:42:10.181 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:42:10.183 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570119.700556_248.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:41:59.700556Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:42:10Z" } 2024-04-19T23:42:10.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570119.700556_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:41:59.700556Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:42:10.183 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570119.700556_248.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:10.183 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570119.700556_248.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:10.183 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:42:10.183 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:42:10.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:10.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:10.184 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:42:10.243 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:10.243 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:10.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:10.278+0000 7f782ca9b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:42:10.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:10.279+0000 7f782ca9b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:42:10.293 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:42:10.293 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:42:10.293 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:42:10.293 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:42:10.293 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:42:10.293 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:42:10.293 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:42:10.294 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:10.294 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:42:10.294 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:42:10.294 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:42:10.294 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:42:10.294 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:42:10.294 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:42:10.294 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:42:10.294 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:42:10.294 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:42:10.295 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:42:10.295 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:42:10.295 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:42:10.295 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:10.295 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:42:10.295 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:42:10.295 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:42:10.295 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:42:10.295 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:42:10.295 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:42:10.295 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:42:10.296 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:42:10.296 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:42:10.296 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:42:10.296 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:42:10.296 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:10.296 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:42:10.296 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:42:10.296 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:42:10.296 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:42:10.296 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713570119.700556_248.1", 2024-04-19T23:42:10.296 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:42:10.297 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:42:10.297 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:42:10.297 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:41:59.700556Z", 2024-04-19T23:42:10.297 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:42:10.297 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:42:10.297 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:10.297 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:42:10.297 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:42:10.297 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:42:10.297 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:42:10.297 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:42:10.298 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:42:10.298 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:42:10.298 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:42:10.298 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:42:10.298 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:42:10.298 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:42:10.298 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:42:10.298 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:42:10.298 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:10.298 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:42:10.298 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:42:10.299 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:42:10.299 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:10.299 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:42:10Z" 2024-04-19T23:42:10.299 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:42:10.300 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570119.700556_248.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:41:59.700556Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:42:10Z" } 2024-04-19T23:42:10.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570119.700556_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:41:59.700556Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:42:10.300 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570119.700556_248.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:10.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570119.700556_248.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:10.300 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:42:10.300 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:10.301 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:10.301 DEBUG:boto:Method: HEAD 2024-04-19T23:42:10.301 DEBUG:boto:Path: /aktlji-24/ 2024-04-19T23:42:10.301 DEBUG:boto:Data: 2024-04-19T23:42:10.301 DEBUG:boto:Headers: {} 2024-04-19T23:42:10.301 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:10.301 DEBUG:boto:Port: 8000 2024-04-19T23:42:10.301 DEBUG:boto:Params: {} 2024-04-19T23:42:10.302 DEBUG:boto:Token: None 2024-04-19T23:42:10.302 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:10 GMT /aktlji-24/ 2024-04-19T23:42:10.302 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ic4Wo70Qo2i2MEpBrHVjwWz1q9c= 2024-04-19T23:42:10.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:10 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ic4Wo70Qo2i2MEpBrHVjwWz1q9c=', 'Content-Length': '0'} 2024-04-19T23:42:10.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d7e6062d9daa7f98-0066230152-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:10.306 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:10.306 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:10.306 DEBUG:boto:Method: PUT 2024-04-19T23:42:10.306 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:10.306 DEBUG:boto:Data: 2024-04-19T23:42:10.306 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:42:10.307 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:10.307 DEBUG:boto:Port: 8000 2024-04-19T23:42:10.307 DEBUG:boto:Params: {} 2024-04-19T23:42:10.308 DEBUG:boto:Token: None 2024-04-19T23:42:10.309 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:42:10 GMT /aktlji-24/obj 2024-04-19T23:42:10.309 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:65MffU0wPlyVArdgV+/2oA6YKgA= 2024-04-19T23:42:10.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:42:10 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:65MffU0wPlyVArdgV+/2oA6YKgA='} 2024-04-19T23:42:10.323 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-amz-request-id', 'tx00000a39c62a9d1b3f681-0066230152-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:10.323 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:10.323 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:10.323 DEBUG:boto:Method: HEAD 2024-04-19T23:42:10.324 DEBUG:boto:Path: /aktlji-24/ 2024-04-19T23:42:10.324 DEBUG:boto:Data: 2024-04-19T23:42:10.324 DEBUG:boto:Headers: {} 2024-04-19T23:42:10.324 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:10.324 DEBUG:boto:Port: 8000 2024-04-19T23:42:10.324 DEBUG:boto:Params: {} 2024-04-19T23:42:10.325 DEBUG:boto:Token: None 2024-04-19T23:42:10.325 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:10 GMT /aktlji-24/ 2024-04-19T23:42:10.325 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ic4Wo70Qo2i2MEpBrHVjwWz1q9c= 2024-04-19T23:42:10.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:10 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ic4Wo70Qo2i2MEpBrHVjwWz1q9c=', 'Content-Length': '0'} 2024-04-19T23:42:10.327 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7a32d067539171d-0066230152-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:10.328 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:10.328 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:10.328 DEBUG:boto:Method: DELETE 2024-04-19T23:42:10.328 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:10.328 DEBUG:boto:Data: 2024-04-19T23:42:10.328 DEBUG:boto:Headers: {} 2024-04-19T23:42:10.328 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:10.328 DEBUG:boto:Port: 8000 2024-04-19T23:42:10.328 DEBUG:boto:Params: {} 2024-04-19T23:42:10.329 DEBUG:boto:Token: None 2024-04-19T23:42:10.329 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:42:10 GMT /aktlji-24/obj 2024-04-19T23:42:10.329 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:t5CwG91h+Ht1W0QEgkCeli1QKGU= 2024-04-19T23:42:10.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:10 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:t5CwG91h+Ht1W0QEgkCeli1QKGU=', 'Content-Length': '0'} 2024-04-19T23:42:10.337 DEBUG:boto:Response headers: [('x-amz-version-id', 'itwymWRCQlArLV8pM-nwpxotTn8yF95'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000d8116a795e8861bd-0066230152-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:10.337 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:10.337 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:10.337 DEBUG:boto:Method: HEAD 2024-04-19T23:42:10.337 DEBUG:boto:Path: /aktlji-25/ 2024-04-19T23:42:10.337 DEBUG:boto:Data: 2024-04-19T23:42:10.337 DEBUG:boto:Headers: {} 2024-04-19T23:42:10.337 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:10.337 DEBUG:boto:Port: 8000 2024-04-19T23:42:10.338 DEBUG:boto:Params: {} 2024-04-19T23:42:10.338 DEBUG:boto:Token: None 2024-04-19T23:42:10.338 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:10 GMT /aktlji-25/ 2024-04-19T23:42:10.338 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UEdsHoiB3jwd+DyobR7PNTyRefQ= 2024-04-19T23:42:10.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:10 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UEdsHoiB3jwd+DyobR7PNTyRefQ=', 'Content-Length': '0'} 2024-04-19T23:42:10.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092044439e427c815-0066230152-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:10.341 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:10.341 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:10.341 DEBUG:boto:Method: PUT 2024-04-19T23:42:10.341 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:10.341 DEBUG:boto:Data: 2024-04-19T23:42:10.341 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:42:10.341 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:10.341 DEBUG:boto:Port: 8000 2024-04-19T23:42:10.341 DEBUG:boto:Params: {} 2024-04-19T23:42:10.341 DEBUG:boto:Token: None 2024-04-19T23:42:10.342 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:42:10 GMT /aktlji-25/obj 2024-04-19T23:42:10.342 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BnIUMdAu69fFKNomTn79n6IV5uY= 2024-04-19T23:42:10.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:42:10 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BnIUMdAu69fFKNomTn79n6IV5uY='} 2024-04-19T23:42:10.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-amz-request-id', 'tx0000088f65703a5be498a-0066230152-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:10.356 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:10.357 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:10.357 DEBUG:boto:Method: HEAD 2024-04-19T23:42:10.357 DEBUG:boto:Path: /aktlji-25/ 2024-04-19T23:42:10.357 DEBUG:boto:Data: 2024-04-19T23:42:10.357 DEBUG:boto:Headers: {} 2024-04-19T23:42:10.357 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:10.357 DEBUG:boto:Port: 8000 2024-04-19T23:42:10.357 DEBUG:boto:Params: {} 2024-04-19T23:42:10.357 DEBUG:boto:Token: None 2024-04-19T23:42:10.358 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:10 GMT /aktlji-25/ 2024-04-19T23:42:10.358 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UEdsHoiB3jwd+DyobR7PNTyRefQ= 2024-04-19T23:42:10.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:10 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UEdsHoiB3jwd+DyobR7PNTyRefQ=', 'Content-Length': '0'} 2024-04-19T23:42:10.360 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c958cf7825d9bfb4-0066230152-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:10.360 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:10.361 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:10.362 DEBUG:boto:Method: DELETE 2024-04-19T23:42:10.362 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:10.363 DEBUG:boto:Data: 2024-04-19T23:42:10.363 DEBUG:boto:Headers: {} 2024-04-19T23:42:10.363 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:10.363 DEBUG:boto:Port: 8000 2024-04-19T23:42:10.363 DEBUG:boto:Params: {} 2024-04-19T23:42:10.363 DEBUG:boto:Token: None 2024-04-19T23:42:10.363 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:42:10 GMT /aktlji-25/obj 2024-04-19T23:42:10.364 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/p6aHypbASu2XPvbilCWJK8PeJQ= 2024-04-19T23:42:10.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:10 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/p6aHypbASu2XPvbilCWJK8PeJQ=', 'Content-Length': '0'} 2024-04-19T23:42:10.372 DEBUG:boto:Response headers: [('x-amz-version-id', 'qQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcK'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000d6c3b5b03a8f434d-0066230152-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:10.373 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:10.373 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:10.373 DEBUG:boto:Method: HEAD 2024-04-19T23:42:10.374 DEBUG:boto:Path: /aktlji-26/ 2024-04-19T23:42:10.374 DEBUG:boto:Data: 2024-04-19T23:42:10.374 DEBUG:boto:Headers: {} 2024-04-19T23:42:10.375 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:10.375 DEBUG:boto:Port: 8001 2024-04-19T23:42:10.375 DEBUG:boto:Params: {} 2024-04-19T23:42:10.375 DEBUG:boto:Token: None 2024-04-19T23:42:10.375 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:10 GMT /aktlji-26/ 2024-04-19T23:42:10.376 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Pvcm0hX9CtwbqAbv8xKBKG890fA= 2024-04-19T23:42:10.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:10 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Pvcm0hX9CtwbqAbv8xKBKG890fA=', 'Content-Length': '0'} 2024-04-19T23:42:10.378 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b1fbce174d44abe-0066230152-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:10.378 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:10.379 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:10.379 DEBUG:boto:Method: PUT 2024-04-19T23:42:10.379 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:10.379 DEBUG:boto:Data: 2024-04-19T23:42:10.379 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:42:10.379 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:10.379 DEBUG:boto:Port: 8001 2024-04-19T23:42:10.379 DEBUG:boto:Params: {} 2024-04-19T23:42:10.380 DEBUG:boto:Token: None 2024-04-19T23:42:10.380 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:42:10 GMT /aktlji-26/obj 2024-04-19T23:42:10.380 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Y2AHycBfxk9XJ3Kofk78QehGBQk= 2024-04-19T23:42:10.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:42:10 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y2AHycBfxk9XJ3Kofk78QehGBQk='} 2024-04-19T23:42:10.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-amz-request-id', 'tx000004cc99a78dc53e756-0066230152-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:10.396 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:10.396 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:10.396 DEBUG:boto:Method: HEAD 2024-04-19T23:42:10.396 DEBUG:boto:Path: /aktlji-26/ 2024-04-19T23:42:10.396 DEBUG:boto:Data: 2024-04-19T23:42:10.397 DEBUG:boto:Headers: {} 2024-04-19T23:42:10.397 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:10.397 DEBUG:boto:Port: 8001 2024-04-19T23:42:10.397 DEBUG:boto:Params: {} 2024-04-19T23:42:10.397 DEBUG:boto:Token: None 2024-04-19T23:42:10.397 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:10 GMT /aktlji-26/ 2024-04-19T23:42:10.397 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Pvcm0hX9CtwbqAbv8xKBKG890fA= 2024-04-19T23:42:10.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:10 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Pvcm0hX9CtwbqAbv8xKBKG890fA=', 'Content-Length': '0'} 2024-04-19T23:42:10.399 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001084eb4a4ad375a1-0066230152-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:10.400 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:10.400 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:10.400 DEBUG:boto:Method: DELETE 2024-04-19T23:42:10.400 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:10.400 DEBUG:boto:Data: 2024-04-19T23:42:10.400 DEBUG:boto:Headers: {} 2024-04-19T23:42:10.400 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:10.400 DEBUG:boto:Port: 8001 2024-04-19T23:42:10.400 DEBUG:boto:Params: {} 2024-04-19T23:42:10.401 DEBUG:boto:Token: None 2024-04-19T23:42:10.401 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:42:10 GMT /aktlji-26/obj 2024-04-19T23:42:10.401 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FoA7v00kQy/+G+MBt8CTRHeiT2E= 2024-04-19T23:42:10.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:10 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FoA7v00kQy/+G+MBt8CTRHeiT2E=', 'Content-Length': '0'} 2024-04-19T23:42:10.409 DEBUG:boto:Response headers: [('x-amz-version-id', 'iJzYuqY05U-Eu1l-5iINYoSVqxQDj2z'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000b05f3390b2ea9b5e-0066230152-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:10.409 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-24 2024-04-19T23:42:10.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:10.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:10.410 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:10.477 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:10.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:10.521+0000 7fc6cb109780 1 waiting to reach incremental sync.. 2024-04-19T23:42:15.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:15.523+0000 7fc6cb109780 1 waiting to reach incremental sync.. 2024-04-19T23:42:20.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:20.524+0000 7fc6cb109780 1 waiting to reach incremental sync.. 2024-04-19T23:42:25.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:25.524+0000 7fc6cb109780 1 waiting to reach incremental sync.. 2024-04-19T23:42:30.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:30.526+0000 7fc6cb109780 1 bucket sync caught up with source: 2024-04-19T23:42:30.525 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2024-04-19T23:42:30.525 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2024-04-19T23:42:30.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:30.526+0000 7fc6cb109780 0 bucket checkpoint complete 2024-04-19T23:42:30.543 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-24 2024-04-19T23:42:30.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:30.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:30.543 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:30.610 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:30.611 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:30.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:30.654+0000 7f6261f5d780 1 bucket sync caught up with source: 2024-04-19T23:42:30.653 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000003.15.12, , ] 2024-04-19T23:42:30.653 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.15.12, , ] 2024-04-19T23:42:30.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:30.654+0000 7f6261f5d780 0 bucket checkpoint complete 2024-04-19T23:42:30.667 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-24 2024-04-19T23:42:30.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:30.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:30.668 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:30.730 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:30.730 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:30.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:30.772+0000 7f34bbf2b780 1 waiting to reach incremental sync.. 2024-04-19T23:42:35.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:35.773+0000 7f34bbf2b780 1 waiting to reach incremental sync.. 2024-04-19T23:42:40.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:40.775+0000 7f34bbf2b780 1 bucket sync caught up with source: 2024-04-19T23:42:40.773 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000002.149.6, , ] 2024-04-19T23:42:40.774 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000002.149.6, , ] 2024-04-19T23:42:40.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:40.775+0000 7f34bbf2b780 0 bucket checkpoint complete 2024-04-19T23:42:40.791 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-24 2024-04-19T23:42:40.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:40.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:40.791 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:40.856 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:40.856 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:40.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:40.900+0000 7f8d8057c780 1 bucket sync caught up with source: 2024-04-19T23:42:40.898 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000002.149.6, , ] 2024-04-19T23:42:40.898 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000002.149.6, , ] 2024-04-19T23:42:40.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:40.900+0000 7f8d8057c780 0 bucket checkpoint complete 2024-04-19T23:42:40.912 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-24 2024-04-19T23:42:40.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:40.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:40.913 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:40.974 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:40.974 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:41.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:41.017+0000 7fd0162d0780 1 bucket sync caught up with source: 2024-04-19T23:42:41.015 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000002.43.6, , ] 2024-04-19T23:42:41.015 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000002.43.6, , ] 2024-04-19T23:42:41.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:41.017+0000 7fd0162d0780 0 bucket checkpoint complete 2024-04-19T23:42:41.029 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-24 2024-04-19T23:42:41.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:41.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:41.030 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:41.097 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:41.097 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:41.140+0000 7fccd7acc780 1 bucket sync caught up with source: 2024-04-19T23:42:41.139 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000002.43.6, , ] 2024-04-19T23:42:41.139 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000002.43.6, , ] 2024-04-19T23:42:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:41.140+0000 7fccd7acc780 0 bucket checkpoint complete 2024-04-19T23:42:41.154 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-24 zones={test-zone2, test-zone1} 2024-04-19T23:42:41.154 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.154 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.155 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.155 DEBUG:boto:Path: /aktlji-24/ 2024-04-19T23:42:41.155 DEBUG:boto:Data: 2024-04-19T23:42:41.155 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.155 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.155 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.155 DEBUG:boto:Params: {} 2024-04-19T23:42:41.156 DEBUG:boto:Token: None 2024-04-19T23:42:41.156 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/ 2024-04-19T23:42:41.156 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NKW622tgSulwjN3ei+f9Zj8maT8= 2024-04-19T23:42:41.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NKW622tgSulwjN3ei+f9Zj8maT8=', 'Content-Length': '0'} 2024-04-19T23:42:41.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed06e45e0adde26f-0066230171-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.159 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.159 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.159 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.159 DEBUG:boto:Path: /aktlji-24/ 2024-04-19T23:42:41.160 DEBUG:boto:Data: 2024-04-19T23:42:41.160 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.160 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.160 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.160 DEBUG:boto:Params: {} 2024-04-19T23:42:41.160 DEBUG:boto:Token: None 2024-04-19T23:42:41.160 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/ 2024-04-19T23:42:41.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NKW622tgSulwjN3ei+f9Zj8maT8= 2024-04-19T23:42:41.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NKW622tgSulwjN3ei+f9Zj8maT8=', 'Content-Length': '0'} 2024-04-19T23:42:41.163 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000e0833da40eaad3-0066230171-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.163 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:42:41.164 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.164 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.164 DEBUG:boto:path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.164 DEBUG:boto:auth_path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.164 DEBUG:boto:Method: GET 2024-04-19T23:42:41.164 DEBUG:boto:Path: /aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.164 DEBUG:boto:Data: 2024-04-19T23:42:41.164 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.164 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.164 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.164 DEBUG:boto:Params: {} 2024-04-19T23:42:41.165 DEBUG:boto:Token: None 2024-04-19T23:42:41.165 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/?versions 2024-04-19T23:42:41.165 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE= 2024-04-19T23:42:41.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE=', 'Content-Length': '0'} 2024-04-19T23:42:41.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b96a97d2248365ba-0066230171-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.167 DEBUG:boto:b'aktlji-24999falseobjitwymWRCQlArLV8pM-nwpxotTn8yF95true2024-04-19T23:42:10.334Zrgw-multisite-test-userMultisite Test UserNormalobjeQbBGHtsV9aGova6tr7Fc98bzF02RqJfalse2024-04-19T23:42:10.315Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:41.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:41.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:41.168 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:42:41.168 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.169 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.169 DEBUG:boto:path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.169 DEBUG:boto:auth_path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.169 DEBUG:boto:Method: GET 2024-04-19T23:42:41.169 DEBUG:boto:Path: /aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.169 DEBUG:boto:Data: 2024-04-19T23:42:41.169 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.169 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.169 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.169 DEBUG:boto:Params: {} 2024-04-19T23:42:41.169 DEBUG:boto:Token: None 2024-04-19T23:42:41.170 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/?versions 2024-04-19T23:42:41.170 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE= 2024-04-19T23:42:41.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE=', 'Content-Length': '0'} 2024-04-19T23:42:41.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d15f44e047e6f9-0066230171-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.173 DEBUG:boto:b'aktlji-24999falseobjitwymWRCQlArLV8pM-nwpxotTn8yF95true2024-04-19T23:42:10.334Zrgw-multisite-test-userMultisite Test UserNormalobjeQbBGHtsV9aGova6tr7Fc98bzF02RqJfalse2024-04-19T23:42:10.315Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:41.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:41.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:41.173 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.173 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.173 DEBUG:boto:path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.173 DEBUG:boto:auth_path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.174 DEBUG:boto:Method: GET 2024-04-19T23:42:41.174 DEBUG:boto:Path: /aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.174 DEBUG:boto:Data: 2024-04-19T23:42:41.174 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.174 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.174 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.174 DEBUG:boto:Params: {} 2024-04-19T23:42:41.174 DEBUG:boto:Token: None 2024-04-19T23:42:41.174 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/?versions 2024-04-19T23:42:41.174 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE= 2024-04-19T23:42:41.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE=', 'Content-Length': '0'} 2024-04-19T23:42:41.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068e0e444e5d2fec4-0066230171-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.218 DEBUG:boto:b'aktlji-24999falseobjitwymWRCQlArLV8pM-nwpxotTn8yF95true2024-04-19T23:42:10.334Zrgw-multisite-test-userMultisite Test UserNormalobjeQbBGHtsV9aGova6tr7Fc98bzF02RqJfalse2024-04-19T23:42:10.315Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:41.219 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.219 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.219 DEBUG:boto:path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.219 DEBUG:boto:auth_path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.219 DEBUG:boto:Method: GET 2024-04-19T23:42:41.219 DEBUG:boto:Path: /aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.219 DEBUG:boto:Data: 2024-04-19T23:42:41.219 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.219 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.219 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.219 DEBUG:boto:Params: {} 2024-04-19T23:42:41.220 DEBUG:boto:Token: None 2024-04-19T23:42:41.220 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/?versions 2024-04-19T23:42:41.220 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE= 2024-04-19T23:42:41.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE=', 'Content-Length': '0'} 2024-04-19T23:42:41.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f536160b0eedcdd6-0066230171-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.222 DEBUG:boto:b'aktlji-24999falseobjitwymWRCQlArLV8pM-nwpxotTn8yF95true2024-04-19T23:42:10.334Zrgw-multisite-test-userMultisite Test UserNormalobjeQbBGHtsV9aGova6tr7Fc98bzF02RqJfalse2024-04-19T23:42:10.315Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:41.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:41.223 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.223 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.223 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.223 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:41.223 DEBUG:boto:Data: 2024-04-19T23:42:41.223 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.223 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.223 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.224 DEBUG:boto:Params: {} 2024-04-19T23:42:41.224 DEBUG:boto:Token: None 2024-04-19T23:42:41.224 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj 2024-04-19T23:42:41.224 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo= 2024-04-19T23:42:41.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo=', 'Content-Length': '0'} 2024-04-19T23:42:41.226 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000003a652a8125ff4d21-0066230171-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.226 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.226 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.226 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.227 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:41.227 DEBUG:boto:Data: 2024-04-19T23:42:41.227 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.227 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.227 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.227 DEBUG:boto:Params: {} 2024-04-19T23:42:41.227 DEBUG:boto:Token: None 2024-04-19T23:42:41.227 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj 2024-04-19T23:42:41.227 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo= 2024-04-19T23:42:41.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo=', 'Content-Length': '0'} 2024-04-19T23:42:41.230 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000aa2a2ec8c624db30-0066230171-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:41.230 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.230 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.230 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.230 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.230 DEBUG:boto:Method: GET 2024-04-19T23:42:41.230 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.230 DEBUG:boto:Data: 2024-04-19T23:42:41.231 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.231 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.231 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.231 DEBUG:boto:Params: {} 2024-04-19T23:42:41.231 DEBUG:boto:Token: None 2024-04-19T23:42:41.231 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.231 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY= 2024-04-19T23:42:41.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY=', 'Content-Length': '0'} 2024-04-19T23:42:41.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008e1d96c938b8ce6a-0066230171-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.233 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.233 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.233 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.233 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.234 DEBUG:boto:Method: GET 2024-04-19T23:42:41.234 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.234 DEBUG:boto:Data: 2024-04-19T23:42:41.234 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.234 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.234 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.234 DEBUG:boto:Params: {} 2024-04-19T23:42:41.234 DEBUG:boto:Token: None 2024-04-19T23:42:41.234 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.234 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY= 2024-04-19T23:42:41.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY=', 'Content-Length': '0'} 2024-04-19T23:42:41.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008308a02b4a769f60-0066230171-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.237 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.237 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.237 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.237 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.237 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.237 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.237 DEBUG:boto:Data: 2024-04-19T23:42:41.237 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.237 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.237 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.237 DEBUG:boto:Params: {} 2024-04-19T23:42:41.238 DEBUG:boto:Token: None 2024-04-19T23:42:41.238 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.238 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X84EGQ1l2jl/BMW83dE+XMJV+PY= 2024-04-19T23:42:41.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X84EGQ1l2jl/BMW83dE+XMJV+PY=', 'Content-Length': '0'} 2024-04-19T23:42:41.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dfdd49c072b84942-0066230171-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.240 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.240 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.240 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.240 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.240 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.240 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.241 DEBUG:boto:Data: 2024-04-19T23:42:41.241 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.241 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.241 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.241 DEBUG:boto:Params: {} 2024-04-19T23:42:41.241 DEBUG:boto:Token: None 2024-04-19T23:42:41.241 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.241 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X84EGQ1l2jl/BMW83dE+XMJV+PY= 2024-04-19T23:42:41.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X84EGQ1l2jl/BMW83dE+XMJV+PY=', 'Content-Length': '0'} 2024-04-19T23:42:41.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000039f327252d524014-0066230171-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:41.243 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.244 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.244 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.244 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.244 DEBUG:boto:Method: GET 2024-04-19T23:42:41.244 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.244 DEBUG:boto:Data: 2024-04-19T23:42:41.244 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.244 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.244 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.244 DEBUG:boto:Params: {} 2024-04-19T23:42:41.244 DEBUG:boto:Token: None 2024-04-19T23:42:41.244 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.245 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY= 2024-04-19T23:42:41.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY=', 'Content-Length': '0'} 2024-04-19T23:42:41.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006bf0dd1d74c7b9fd-0066230171-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.247 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.247 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.247 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.247 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.247 DEBUG:boto:Method: GET 2024-04-19T23:42:41.247 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.247 DEBUG:boto:Data: 2024-04-19T23:42:41.248 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.248 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.248 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.248 DEBUG:boto:Params: {} 2024-04-19T23:42:41.248 DEBUG:boto:Token: None 2024-04-19T23:42:41.248 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.248 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY= 2024-04-19T23:42:41.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY=', 'Content-Length': '0'} 2024-04-19T23:42:41.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000887ec8515529b1f6-0066230171-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.250 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.250 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.251 DEBUG:boto:path=/aktlji-24/?prefix=obj 2024-04-19T23:42:41.251 DEBUG:boto:auth_path=/aktlji-24/?prefix=obj 2024-04-19T23:42:41.251 DEBUG:boto:Method: GET 2024-04-19T23:42:41.251 DEBUG:boto:Path: /aktlji-24/?prefix=obj 2024-04-19T23:42:41.251 DEBUG:boto:Data: 2024-04-19T23:42:41.251 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.251 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.251 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.251 DEBUG:boto:Params: {} 2024-04-19T23:42:41.251 DEBUG:boto:Token: None 2024-04-19T23:42:41.252 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/ 2024-04-19T23:42:41.252 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gQYYr746H2PINBtpuLRH40vnL7E= 2024-04-19T23:42:41.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gQYYr746H2PINBtpuLRH40vnL7E=', 'Content-Length': '0'} 2024-04-19T23:42:41.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fce9d4c5c37ca8cb-0066230171-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.254 DEBUG:boto:b'aktlji-24obj1000false' 2024-04-19T23:42:41.255 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.255 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.255 DEBUG:boto:path=/aktlji-24/?prefix=obj 2024-04-19T23:42:41.255 DEBUG:boto:auth_path=/aktlji-24/?prefix=obj 2024-04-19T23:42:41.255 DEBUG:boto:Method: GET 2024-04-19T23:42:41.255 DEBUG:boto:Path: /aktlji-24/?prefix=obj 2024-04-19T23:42:41.255 DEBUG:boto:Data: 2024-04-19T23:42:41.255 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.255 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.255 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.255 DEBUG:boto:Params: {} 2024-04-19T23:42:41.256 DEBUG:boto:Token: None 2024-04-19T23:42:41.256 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/ 2024-04-19T23:42:41.256 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gQYYr746H2PINBtpuLRH40vnL7E= 2024-04-19T23:42:41.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gQYYr746H2PINBtpuLRH40vnL7E=', 'Content-Length': '0'} 2024-04-19T23:42:41.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054b1e1f4820ccba5-0066230171-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.302 DEBUG:boto:b'aktlji-24obj1000false' 2024-04-19T23:42:41.303 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.303 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.303 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.303 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:41.303 DEBUG:boto:Data: 2024-04-19T23:42:41.303 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.303 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.303 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.303 DEBUG:boto:Params: {} 2024-04-19T23:42:41.303 DEBUG:boto:Token: None 2024-04-19T23:42:41.304 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj 2024-04-19T23:42:41.304 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo= 2024-04-19T23:42:41.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo=', 'Content-Length': '0'} 2024-04-19T23:42:41.306 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000009fa6810c5a2d226a-0066230171-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.306 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.306 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.306 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.306 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:41.306 DEBUG:boto:Data: 2024-04-19T23:42:41.306 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.306 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.307 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.307 DEBUG:boto:Params: {} 2024-04-19T23:42:41.307 DEBUG:boto:Token: None 2024-04-19T23:42:41.307 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj 2024-04-19T23:42:41.307 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo= 2024-04-19T23:42:41.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo=', 'Content-Length': '0'} 2024-04-19T23:42:41.310 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000040a25e0dc511a2f-0066230171-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.310 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-24 zones={test-zone2, test-zone1} 2024-04-19T23:42:41.310 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-24 zones={test-zone3, test-zone1} 2024-04-19T23:42:41.310 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.310 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.310 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.310 DEBUG:boto:Path: /aktlji-24/ 2024-04-19T23:42:41.310 DEBUG:boto:Data: 2024-04-19T23:42:41.310 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.311 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.311 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.311 DEBUG:boto:Params: {} 2024-04-19T23:42:41.311 DEBUG:boto:Token: None 2024-04-19T23:42:41.311 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/ 2024-04-19T23:42:41.311 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NKW622tgSulwjN3ei+f9Zj8maT8= 2024-04-19T23:42:41.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NKW622tgSulwjN3ei+f9Zj8maT8=', 'Content-Length': '0'} 2024-04-19T23:42:41.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ee03dac8c637775-0066230171-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.314 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.314 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.314 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.314 DEBUG:boto:Path: /aktlji-24/ 2024-04-19T23:42:41.314 DEBUG:boto:Data: 2024-04-19T23:42:41.314 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.314 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.315 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.315 DEBUG:boto:Params: {} 2024-04-19T23:42:41.315 DEBUG:boto:Token: None 2024-04-19T23:42:41.315 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/ 2024-04-19T23:42:41.315 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NKW622tgSulwjN3ei+f9Zj8maT8= 2024-04-19T23:42:41.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NKW622tgSulwjN3ei+f9Zj8maT8=', 'Content-Length': '0'} 2024-04-19T23:42:41.317 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3c9910f5d8246b5-0066230171-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.317 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:42:41.318 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.318 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.318 DEBUG:boto:path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.318 DEBUG:boto:auth_path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.318 DEBUG:boto:Method: GET 2024-04-19T23:42:41.318 DEBUG:boto:Path: /aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.318 DEBUG:boto:Data: 2024-04-19T23:42:41.318 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.318 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.318 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.318 DEBUG:boto:Params: {} 2024-04-19T23:42:41.318 DEBUG:boto:Token: None 2024-04-19T23:42:41.319 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/?versions 2024-04-19T23:42:41.319 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE= 2024-04-19T23:42:41.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE=', 'Content-Length': '0'} 2024-04-19T23:42:41.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1c7b734567e3a58-0066230171-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.321 DEBUG:boto:b'aktlji-24999falseobjitwymWRCQlArLV8pM-nwpxotTn8yF95true2024-04-19T23:42:10.334Zrgw-multisite-test-userMultisite Test UserNormalobjeQbBGHtsV9aGova6tr7Fc98bzF02RqJfalse2024-04-19T23:42:10.315Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:41.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:41.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:41.322 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:42:41.322 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.322 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.322 DEBUG:boto:path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.322 DEBUG:boto:auth_path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.322 DEBUG:boto:Method: GET 2024-04-19T23:42:41.322 DEBUG:boto:Path: /aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.323 DEBUG:boto:Data: 2024-04-19T23:42:41.323 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.323 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.323 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.323 DEBUG:boto:Params: {} 2024-04-19T23:42:41.323 DEBUG:boto:Token: None 2024-04-19T23:42:41.323 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/?versions 2024-04-19T23:42:41.323 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE= 2024-04-19T23:42:41.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE=', 'Content-Length': '0'} 2024-04-19T23:42:41.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb7a1dfe4ea9069f-0066230171-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.326 DEBUG:boto:b'aktlji-24999falseobjitwymWRCQlArLV8pM-nwpxotTn8yF95true2024-04-19T23:42:10.334Zrgw-multisite-test-userMultisite Test UserNormalobjeQbBGHtsV9aGova6tr7Fc98bzF02RqJfalse2024-04-19T23:42:10.315Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:41.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:41.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:41.327 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.327 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.327 DEBUG:boto:path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.327 DEBUG:boto:auth_path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.327 DEBUG:boto:Method: GET 2024-04-19T23:42:41.327 DEBUG:boto:Path: /aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.327 DEBUG:boto:Data: 2024-04-19T23:42:41.327 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.327 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.327 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.327 DEBUG:boto:Params: {} 2024-04-19T23:42:41.328 DEBUG:boto:Token: None 2024-04-19T23:42:41.328 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/?versions 2024-04-19T23:42:41.328 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE= 2024-04-19T23:42:41.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE=', 'Content-Length': '0'} 2024-04-19T23:42:41.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063d460c708eb7d2a-0066230171-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.331 DEBUG:boto:b'aktlji-24999falseobjitwymWRCQlArLV8pM-nwpxotTn8yF95true2024-04-19T23:42:10.334Zrgw-multisite-test-userMultisite Test UserNormalobjeQbBGHtsV9aGova6tr7Fc98bzF02RqJfalse2024-04-19T23:42:10.315Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:41.331 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.331 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.331 DEBUG:boto:path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.331 DEBUG:boto:auth_path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.331 DEBUG:boto:Method: GET 2024-04-19T23:42:41.331 DEBUG:boto:Path: /aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.331 DEBUG:boto:Data: 2024-04-19T23:42:41.332 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.332 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.332 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.332 DEBUG:boto:Params: {} 2024-04-19T23:42:41.332 DEBUG:boto:Token: None 2024-04-19T23:42:41.332 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/?versions 2024-04-19T23:42:41.332 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE= 2024-04-19T23:42:41.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE=', 'Content-Length': '0'} 2024-04-19T23:42:41.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5ac1ccf5681ee24-0066230171-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.335 DEBUG:boto:b'aktlji-24999falseobjitwymWRCQlArLV8pM-nwpxotTn8yF95true2024-04-19T23:42:10.334Zrgw-multisite-test-userMultisite Test UserNormalobjeQbBGHtsV9aGova6tr7Fc98bzF02RqJfalse2024-04-19T23:42:10.315Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:41.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:41.335 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.335 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.336 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.336 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:41.336 DEBUG:boto:Data: 2024-04-19T23:42:41.336 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.336 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.336 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.336 DEBUG:boto:Params: {} 2024-04-19T23:42:41.336 DEBUG:boto:Token: None 2024-04-19T23:42:41.336 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj 2024-04-19T23:42:41.336 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo= 2024-04-19T23:42:41.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo=', 'Content-Length': '0'} 2024-04-19T23:42:41.339 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000f56f97528cb4c33c-0066230171-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.339 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.339 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.339 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.339 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:41.339 DEBUG:boto:Data: 2024-04-19T23:42:41.339 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.339 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.339 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.339 DEBUG:boto:Params: {} 2024-04-19T23:42:41.340 DEBUG:boto:Token: None 2024-04-19T23:42:41.340 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj 2024-04-19T23:42:41.340 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo= 2024-04-19T23:42:41.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo=', 'Content-Length': '0'} 2024-04-19T23:42:41.342 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000acb3672039a3ed95-0066230171-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:41.342 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.342 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.343 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.343 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.343 DEBUG:boto:Method: GET 2024-04-19T23:42:41.343 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.343 DEBUG:boto:Data: 2024-04-19T23:42:41.343 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.343 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.343 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.343 DEBUG:boto:Params: {} 2024-04-19T23:42:41.343 DEBUG:boto:Token: None 2024-04-19T23:42:41.343 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.343 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY= 2024-04-19T23:42:41.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY=', 'Content-Length': '0'} 2024-04-19T23:42:41.345 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003afe8017c91e0ebf-0066230171-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.345 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.345 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.346 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.346 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.346 DEBUG:boto:Method: GET 2024-04-19T23:42:41.346 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.346 DEBUG:boto:Data: 2024-04-19T23:42:41.346 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.346 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.346 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.346 DEBUG:boto:Params: {} 2024-04-19T23:42:41.346 DEBUG:boto:Token: None 2024-04-19T23:42:41.346 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.347 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY= 2024-04-19T23:42:41.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY=', 'Content-Length': '0'} 2024-04-19T23:42:41.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e37c49cf5bb5fe4f-0066230171-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.349 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.349 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.349 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.349 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.349 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.349 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.349 DEBUG:boto:Data: 2024-04-19T23:42:41.349 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.349 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.349 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.350 DEBUG:boto:Params: {} 2024-04-19T23:42:41.350 DEBUG:boto:Token: None 2024-04-19T23:42:41.350 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.350 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X84EGQ1l2jl/BMW83dE+XMJV+PY= 2024-04-19T23:42:41.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X84EGQ1l2jl/BMW83dE+XMJV+PY=', 'Content-Length': '0'} 2024-04-19T23:42:41.352 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000374046615c98f8a7-0066230171-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.352 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.352 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.352 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.352 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.353 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.353 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.353 DEBUG:boto:Data: 2024-04-19T23:42:41.353 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.353 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.353 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.353 DEBUG:boto:Params: {} 2024-04-19T23:42:41.353 DEBUG:boto:Token: None 2024-04-19T23:42:41.353 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.353 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X84EGQ1l2jl/BMW83dE+XMJV+PY= 2024-04-19T23:42:41.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X84EGQ1l2jl/BMW83dE+XMJV+PY=', 'Content-Length': '0'} 2024-04-19T23:42:41.355 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040574e50c6511d5c-0066230171-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:41.356 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.356 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.356 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.356 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.356 DEBUG:boto:Method: GET 2024-04-19T23:42:41.356 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.356 DEBUG:boto:Data: 2024-04-19T23:42:41.356 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.356 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.356 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.356 DEBUG:boto:Params: {} 2024-04-19T23:42:41.357 DEBUG:boto:Token: None 2024-04-19T23:42:41.357 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.357 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY= 2024-04-19T23:42:41.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY=', 'Content-Length': '0'} 2024-04-19T23:42:41.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d3edade9e2b66d83-0066230171-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.359 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.359 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.359 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.359 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.359 DEBUG:boto:Method: GET 2024-04-19T23:42:41.359 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.359 DEBUG:boto:Data: 2024-04-19T23:42:41.360 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.360 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.360 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.360 DEBUG:boto:Params: {} 2024-04-19T23:42:41.360 DEBUG:boto:Token: None 2024-04-19T23:42:41.360 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.360 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY= 2024-04-19T23:42:41.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY=', 'Content-Length': '0'} 2024-04-19T23:42:41.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005074b3b1f9b16093-0066230171-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.362 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.362 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.362 DEBUG:boto:path=/aktlji-24/?prefix=obj 2024-04-19T23:42:41.363 DEBUG:boto:auth_path=/aktlji-24/?prefix=obj 2024-04-19T23:42:41.363 DEBUG:boto:Method: GET 2024-04-19T23:42:41.363 DEBUG:boto:Path: /aktlji-24/?prefix=obj 2024-04-19T23:42:41.363 DEBUG:boto:Data: 2024-04-19T23:42:41.363 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.363 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.363 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.363 DEBUG:boto:Params: {} 2024-04-19T23:42:41.363 DEBUG:boto:Token: None 2024-04-19T23:42:41.363 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/ 2024-04-19T23:42:41.363 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gQYYr746H2PINBtpuLRH40vnL7E= 2024-04-19T23:42:41.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gQYYr746H2PINBtpuLRH40vnL7E=', 'Content-Length': '0'} 2024-04-19T23:42:41.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee400a71a3dedfd3-0066230171-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.406 DEBUG:boto:b'aktlji-24obj1000false' 2024-04-19T23:42:41.407 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.407 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.407 DEBUG:boto:path=/aktlji-24/?prefix=obj 2024-04-19T23:42:41.407 DEBUG:boto:auth_path=/aktlji-24/?prefix=obj 2024-04-19T23:42:41.407 DEBUG:boto:Method: GET 2024-04-19T23:42:41.407 DEBUG:boto:Path: /aktlji-24/?prefix=obj 2024-04-19T23:42:41.407 DEBUG:boto:Data: 2024-04-19T23:42:41.407 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.407 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.407 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.407 DEBUG:boto:Params: {} 2024-04-19T23:42:41.408 DEBUG:boto:Token: None 2024-04-19T23:42:41.408 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/ 2024-04-19T23:42:41.408 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gQYYr746H2PINBtpuLRH40vnL7E= 2024-04-19T23:42:41.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gQYYr746H2PINBtpuLRH40vnL7E=', 'Content-Length': '0'} 2024-04-19T23:42:41.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec18cf157127a106-0066230171-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.411 DEBUG:boto:b'aktlji-24obj1000false' 2024-04-19T23:42:41.411 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.411 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.411 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.412 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:41.412 DEBUG:boto:Data: 2024-04-19T23:42:41.412 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.412 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.412 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.412 DEBUG:boto:Params: {} 2024-04-19T23:42:41.412 DEBUG:boto:Token: None 2024-04-19T23:42:41.412 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj 2024-04-19T23:42:41.412 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo= 2024-04-19T23:42:41.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo=', 'Content-Length': '0'} 2024-04-19T23:42:41.414 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000003a792c9468d79fb3-0066230171-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.415 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.415 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.415 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.415 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:41.415 DEBUG:boto:Data: 2024-04-19T23:42:41.415 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.415 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:41.415 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.415 DEBUG:boto:Params: {} 2024-04-19T23:42:41.415 DEBUG:boto:Token: None 2024-04-19T23:42:41.415 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj 2024-04-19T23:42:41.416 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo= 2024-04-19T23:42:41.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo=', 'Content-Length': '0'} 2024-04-19T23:42:41.418 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000005ad6bd02a4cc8fa4-0066230171-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.418 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-24 zones={test-zone3, test-zone1} 2024-04-19T23:42:41.418 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-24 zones={test-zone3, test-zone2} 2024-04-19T23:42:41.418 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.418 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.418 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.418 DEBUG:boto:Path: /aktlji-24/ 2024-04-19T23:42:41.418 DEBUG:boto:Data: 2024-04-19T23:42:41.418 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.419 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.419 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.419 DEBUG:boto:Params: {} 2024-04-19T23:42:41.419 DEBUG:boto:Token: None 2024-04-19T23:42:41.419 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/ 2024-04-19T23:42:41.419 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NKW622tgSulwjN3ei+f9Zj8maT8= 2024-04-19T23:42:41.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NKW622tgSulwjN3ei+f9Zj8maT8=', 'Content-Length': '0'} 2024-04-19T23:42:41.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007412754eff626a9-0066230171-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.422 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.422 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.422 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.422 DEBUG:boto:Path: /aktlji-24/ 2024-04-19T23:42:41.422 DEBUG:boto:Data: 2024-04-19T23:42:41.422 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.422 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.422 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.422 DEBUG:boto:Params: {} 2024-04-19T23:42:41.423 DEBUG:boto:Token: None 2024-04-19T23:42:41.423 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/ 2024-04-19T23:42:41.423 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NKW622tgSulwjN3ei+f9Zj8maT8= 2024-04-19T23:42:41.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NKW622tgSulwjN3ei+f9Zj8maT8=', 'Content-Length': '0'} 2024-04-19T23:42:41.425 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030ddf1ab32d311d9-0066230171-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.425 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:42:41.426 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.426 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.426 DEBUG:boto:path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.426 DEBUG:boto:auth_path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.426 DEBUG:boto:Method: GET 2024-04-19T23:42:41.426 DEBUG:boto:Path: /aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.426 DEBUG:boto:Data: 2024-04-19T23:42:41.426 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.426 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.426 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.427 DEBUG:boto:Params: {} 2024-04-19T23:42:41.427 DEBUG:boto:Token: None 2024-04-19T23:42:41.427 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/?versions 2024-04-19T23:42:41.427 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE= 2024-04-19T23:42:41.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE=', 'Content-Length': '0'} 2024-04-19T23:42:41.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1bf61a4104c8d0a-0066230171-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.430 DEBUG:boto:b'aktlji-24999falseobjitwymWRCQlArLV8pM-nwpxotTn8yF95true2024-04-19T23:42:10.334Zrgw-multisite-test-userMultisite Test UserNormalobjeQbBGHtsV9aGova6tr7Fc98bzF02RqJfalse2024-04-19T23:42:10.315Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:41.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:41.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:41.430 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:42:41.430 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.430 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.431 DEBUG:boto:path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.431 DEBUG:boto:auth_path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.431 DEBUG:boto:Method: GET 2024-04-19T23:42:41.431 DEBUG:boto:Path: /aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.431 DEBUG:boto:Data: 2024-04-19T23:42:41.431 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.431 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.431 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.431 DEBUG:boto:Params: {} 2024-04-19T23:42:41.431 DEBUG:boto:Token: None 2024-04-19T23:42:41.431 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/?versions 2024-04-19T23:42:41.432 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE= 2024-04-19T23:42:41.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE=', 'Content-Length': '0'} 2024-04-19T23:42:41.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab0cb48ff2149e44-0066230171-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.434 DEBUG:boto:b'aktlji-24999falseobjitwymWRCQlArLV8pM-nwpxotTn8yF95true2024-04-19T23:42:10.334Zrgw-multisite-test-userMultisite Test UserNormalobjeQbBGHtsV9aGova6tr7Fc98bzF02RqJfalse2024-04-19T23:42:10.315Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:41.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:41.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:41.435 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.435 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.435 DEBUG:boto:path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.435 DEBUG:boto:auth_path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.435 DEBUG:boto:Method: GET 2024-04-19T23:42:41.435 DEBUG:boto:Path: /aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.436 DEBUG:boto:Data: 2024-04-19T23:42:41.436 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.436 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.436 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.436 DEBUG:boto:Params: {} 2024-04-19T23:42:41.436 DEBUG:boto:Token: None 2024-04-19T23:42:41.436 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/?versions 2024-04-19T23:42:41.436 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE= 2024-04-19T23:42:41.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE=', 'Content-Length': '0'} 2024-04-19T23:42:41.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004a003b238256dd8-0066230171-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.439 DEBUG:boto:b'aktlji-24999falseobjitwymWRCQlArLV8pM-nwpxotTn8yF95true2024-04-19T23:42:10.334Zrgw-multisite-test-userMultisite Test UserNormalobjeQbBGHtsV9aGova6tr7Fc98bzF02RqJfalse2024-04-19T23:42:10.315Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:41.439 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.439 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.439 DEBUG:boto:path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.440 DEBUG:boto:auth_path=/aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.440 DEBUG:boto:Method: GET 2024-04-19T23:42:41.440 DEBUG:boto:Path: /aktlji-24/?versions&max-keys=999 2024-04-19T23:42:41.440 DEBUG:boto:Data: 2024-04-19T23:42:41.440 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.440 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.440 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.440 DEBUG:boto:Params: {} 2024-04-19T23:42:41.441 DEBUG:boto:Token: None 2024-04-19T23:42:41.441 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/?versions 2024-04-19T23:42:41.441 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE= 2024-04-19T23:42:41.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:82fcmltwNi2gW91GfEwoEht7hvE=', 'Content-Length': '0'} 2024-04-19T23:42:41.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044b85104aebe05e2-0066230171-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.486 DEBUG:boto:b'aktlji-24999falseobjitwymWRCQlArLV8pM-nwpxotTn8yF95true2024-04-19T23:42:10.334Zrgw-multisite-test-userMultisite Test UserNormalobjeQbBGHtsV9aGova6tr7Fc98bzF02RqJfalse2024-04-19T23:42:10.315Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:41.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:41.487 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.487 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.487 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.487 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:41.487 DEBUG:boto:Data: 2024-04-19T23:42:41.487 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.487 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.487 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.488 DEBUG:boto:Params: {} 2024-04-19T23:42:41.488 DEBUG:boto:Token: None 2024-04-19T23:42:41.488 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj 2024-04-19T23:42:41.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo= 2024-04-19T23:42:41.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo=', 'Content-Length': '0'} 2024-04-19T23:42:41.490 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000b59755fceec21996-0066230171-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.490 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.490 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.491 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.491 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:41.491 DEBUG:boto:Data: 2024-04-19T23:42:41.491 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.491 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.491 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.491 DEBUG:boto:Params: {} 2024-04-19T23:42:41.491 DEBUG:boto:Token: None 2024-04-19T23:42:41.491 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj 2024-04-19T23:42:41.491 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo= 2024-04-19T23:42:41.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo=', 'Content-Length': '0'} 2024-04-19T23:42:41.493 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000076aa99b699fa9437-0066230171-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:41.494 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.494 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.494 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.494 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.494 DEBUG:boto:Method: GET 2024-04-19T23:42:41.494 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.494 DEBUG:boto:Data: 2024-04-19T23:42:41.494 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.495 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.495 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.495 DEBUG:boto:Params: {} 2024-04-19T23:42:41.495 DEBUG:boto:Token: None 2024-04-19T23:42:41.495 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.495 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY= 2024-04-19T23:42:41.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY=', 'Content-Length': '0'} 2024-04-19T23:42:41.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000078e2a9fdee83e1a9-0066230171-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.497 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.498 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.498 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.498 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.498 DEBUG:boto:Method: GET 2024-04-19T23:42:41.498 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.498 DEBUG:boto:Data: 2024-04-19T23:42:41.498 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.498 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.498 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.498 DEBUG:boto:Params: {} 2024-04-19T23:42:41.499 DEBUG:boto:Token: None 2024-04-19T23:42:41.499 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.499 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY= 2024-04-19T23:42:41.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY=', 'Content-Length': '0'} 2024-04-19T23:42:41.501 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f465bde0b5ad224-0066230171-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.501 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.501 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.501 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.501 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.501 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.501 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.501 DEBUG:boto:Data: 2024-04-19T23:42:41.501 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.502 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.502 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.502 DEBUG:boto:Params: {} 2024-04-19T23:42:41.502 DEBUG:boto:Token: None 2024-04-19T23:42:41.502 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.502 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X84EGQ1l2jl/BMW83dE+XMJV+PY= 2024-04-19T23:42:41.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X84EGQ1l2jl/BMW83dE+XMJV+PY=', 'Content-Length': '0'} 2024-04-19T23:42:41.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b62a17663226ff2c-0066230171-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.504 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.504 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.504 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.504 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.504 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.504 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.504 DEBUG:boto:Data: 2024-04-19T23:42:41.505 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.505 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.505 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.505 DEBUG:boto:Params: {} 2024-04-19T23:42:41.505 DEBUG:boto:Token: None 2024-04-19T23:42:41.505 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.505 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X84EGQ1l2jl/BMW83dE+XMJV+PY= 2024-04-19T23:42:41.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X84EGQ1l2jl/BMW83dE+XMJV+PY=', 'Content-Length': '0'} 2024-04-19T23:42:41.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002af55d84f9c8c6ce-0066230171-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:41.507 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.507 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.508 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.508 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.508 DEBUG:boto:Method: GET 2024-04-19T23:42:41.508 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.508 DEBUG:boto:Data: 2024-04-19T23:42:41.508 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.508 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.508 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.508 DEBUG:boto:Params: {} 2024-04-19T23:42:41.508 DEBUG:boto:Token: None 2024-04-19T23:42:41.508 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.509 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY= 2024-04-19T23:42:41.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY=', 'Content-Length': '0'} 2024-04-19T23:42:41.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005444e7909e6b3e4d-0066230171-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.510 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.510 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.511 DEBUG:boto:path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.511 DEBUG:boto:auth_path=/aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.511 DEBUG:boto:Method: GET 2024-04-19T23:42:41.511 DEBUG:boto:Path: /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.511 DEBUG:boto:Data: 2024-04-19T23:42:41.511 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.511 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.511 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.511 DEBUG:boto:Params: {} 2024-04-19T23:42:41.511 DEBUG:boto:Token: None 2024-04-19T23:42:41.511 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj?versionId=eQbBGHtsV9aGova6tr7Fc98bzF02RqJ 2024-04-19T23:42:41.512 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY= 2024-04-19T23:42:41.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lGylLqdHGRxWXibK/GhermW3+LY=', 'Content-Length': '0'} 2024-04-19T23:42:41.513 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'eQbBGHtsV9aGova6tr7Fc98bzF02RqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006592911fdca2c208-0066230171-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.514 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.514 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.514 DEBUG:boto:path=/aktlji-24/?prefix=obj 2024-04-19T23:42:41.514 DEBUG:boto:auth_path=/aktlji-24/?prefix=obj 2024-04-19T23:42:41.514 DEBUG:boto:Method: GET 2024-04-19T23:42:41.514 DEBUG:boto:Path: /aktlji-24/?prefix=obj 2024-04-19T23:42:41.514 DEBUG:boto:Data: 2024-04-19T23:42:41.514 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.515 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.515 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.515 DEBUG:boto:Params: {} 2024-04-19T23:42:41.515 DEBUG:boto:Token: None 2024-04-19T23:42:41.515 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/ 2024-04-19T23:42:41.515 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gQYYr746H2PINBtpuLRH40vnL7E= 2024-04-19T23:42:41.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gQYYr746H2PINBtpuLRH40vnL7E=', 'Content-Length': '0'} 2024-04-19T23:42:41.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6134ba6614c8785-0066230171-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.558 DEBUG:boto:b'aktlji-24obj1000false' 2024-04-19T23:42:41.559 DEBUG:boto:path=/aktlji-24/ 2024-04-19T23:42:41.559 DEBUG:boto:auth_path=/aktlji-24/ 2024-04-19T23:42:41.559 DEBUG:boto:path=/aktlji-24/?prefix=obj 2024-04-19T23:42:41.559 DEBUG:boto:auth_path=/aktlji-24/?prefix=obj 2024-04-19T23:42:41.559 DEBUG:boto:Method: GET 2024-04-19T23:42:41.559 DEBUG:boto:Path: /aktlji-24/?prefix=obj 2024-04-19T23:42:41.559 DEBUG:boto:Data: 2024-04-19T23:42:41.559 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.559 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.559 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.559 DEBUG:boto:Params: {} 2024-04-19T23:42:41.560 DEBUG:boto:Token: None 2024-04-19T23:42:41.560 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/ 2024-04-19T23:42:41.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gQYYr746H2PINBtpuLRH40vnL7E= 2024-04-19T23:42:41.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gQYYr746H2PINBtpuLRH40vnL7E=', 'Content-Length': '0'} 2024-04-19T23:42:41.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f80f3200fafec7de-0066230171-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.563 DEBUG:boto:b'aktlji-24obj1000false' 2024-04-19T23:42:41.563 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.563 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.563 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.563 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:41.563 DEBUG:boto:Data: 2024-04-19T23:42:41.563 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.563 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:41.563 DEBUG:boto:Port: 8001 2024-04-19T23:42:41.563 DEBUG:boto:Params: {} 2024-04-19T23:42:41.564 DEBUG:boto:Token: None 2024-04-19T23:42:41.564 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj 2024-04-19T23:42:41.564 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo= 2024-04-19T23:42:41.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo=', 'Content-Length': '0'} 2024-04-19T23:42:41.567 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000b0ec0e84efad041c-0066230171-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.567 DEBUG:boto:path=/aktlji-24/obj 2024-04-19T23:42:41.567 DEBUG:boto:auth_path=/aktlji-24/obj 2024-04-19T23:42:41.567 DEBUG:boto:Method: HEAD 2024-04-19T23:42:41.567 DEBUG:boto:Path: /aktlji-24/obj 2024-04-19T23:42:41.567 DEBUG:boto:Data: 2024-04-19T23:42:41.567 DEBUG:boto:Headers: {} 2024-04-19T23:42:41.567 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:41.567 DEBUG:boto:Port: 8000 2024-04-19T23:42:41.567 DEBUG:boto:Params: {} 2024-04-19T23:42:41.567 DEBUG:boto:Token: None 2024-04-19T23:42:41.568 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:41 GMT /aktlji-24/obj 2024-04-19T23:42:41.568 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo= 2024-04-19T23:42:41.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:y8iSorcBR0NpL2JYgW2kv1UbFeo=', 'Content-Length': '0'} 2024-04-19T23:42:41.570 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a7aab5aad2517bf4-0066230171-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:41.570 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-24 zones={test-zone3, test-zone2} 2024-04-19T23:42:41.570 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-25 2024-04-19T23:42:41.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:41.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:41.571 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:41.640 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:41.640 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:41.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:41.685+0000 7fcfe4f5f780 1 bucket sync caught up with source: 2024-04-19T23:42:41.684 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000002.17.6, , ] 2024-04-19T23:42:41.684 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000002.17.6, , ] 2024-04-19T23:42:41.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:41.685+0000 7fcfe4f5f780 0 bucket checkpoint complete 2024-04-19T23:42:41.699 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-25 2024-04-19T23:42:41.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:41.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:41.700 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:41.768 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:41.768 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:41.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:41.815+0000 7f1c6dd14780 1 bucket sync caught up with source: 2024-04-19T23:42:41.813 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000002.17.6, , ] 2024-04-19T23:42:41.813 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000002.17.6, , ] 2024-04-19T23:42:41.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:41.815+0000 7f1c6dd14780 0 bucket checkpoint complete 2024-04-19T23:42:41.838 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-25 2024-04-19T23:42:41.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:41.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:41.838 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:41.900 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:41.901 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:41.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:41.944+0000 7fc422e0d780 1 bucket sync caught up with source: 2024-04-19T23:42:41.942 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000003.146.12, , ] 2024-04-19T23:42:41.942 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.146.12, , ] 2024-04-19T23:42:41.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:41.944+0000 7fc422e0d780 0 bucket checkpoint complete 2024-04-19T23:42:41.957 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-25 2024-04-19T23:42:41.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:41.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:41.958 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:42.019 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:42.019 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:42.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:42.062+0000 7f3eb3894780 1 bucket sync caught up with source: 2024-04-19T23:42:42.060 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000003.146.12, , ] 2024-04-19T23:42:42.060 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.146.12, , ] 2024-04-19T23:42:42.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:42.062+0000 7f3eb3894780 0 bucket checkpoint complete 2024-04-19T23:42:42.074 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-25 2024-04-19T23:42:42.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:42.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:42.074 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:42.135 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:42.136 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:42.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:42.178+0000 7f877f033780 1 bucket sync caught up with source: 2024-04-19T23:42:42.176 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000002.54.6, , ] 2024-04-19T23:42:42.176 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000002.54.6, , ] 2024-04-19T23:42:42.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:42.178+0000 7f877f033780 0 bucket checkpoint complete 2024-04-19T23:42:42.190 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-25 2024-04-19T23:42:42.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:42.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:42.191 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:42.259 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:42.259 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:42.302+0000 7f30e8a4e780 1 bucket sync caught up with source: 2024-04-19T23:42:42.301 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000002.54.6, , ] 2024-04-19T23:42:42.301 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000002.54.6, , ] 2024-04-19T23:42:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:42.302+0000 7f30e8a4e780 0 bucket checkpoint complete 2024-04-19T23:42:42.316 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-25 zones={test-zone2, test-zone1} 2024-04-19T23:42:42.316 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.316 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.316 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.316 DEBUG:boto:Path: /aktlji-25/ 2024-04-19T23:42:42.316 DEBUG:boto:Data: 2024-04-19T23:42:42.316 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.316 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.317 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.317 DEBUG:boto:Params: {} 2024-04-19T23:42:42.317 DEBUG:boto:Token: None 2024-04-19T23:42:42.317 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/ 2024-04-19T23:42:42.317 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sKKQn4fsI4aOxW3UDX9HhJ6gxMU= 2024-04-19T23:42:42.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sKKQn4fsI4aOxW3UDX9HhJ6gxMU=', 'Content-Length': '0'} 2024-04-19T23:42:42.320 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068929fb8421f06b1-0066230172-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.320 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.320 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.320 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.320 DEBUG:boto:Path: /aktlji-25/ 2024-04-19T23:42:42.320 DEBUG:boto:Data: 2024-04-19T23:42:42.320 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.320 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.320 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.321 DEBUG:boto:Params: {} 2024-04-19T23:42:42.321 DEBUG:boto:Token: None 2024-04-19T23:42:42.321 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/ 2024-04-19T23:42:42.321 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sKKQn4fsI4aOxW3UDX9HhJ6gxMU= 2024-04-19T23:42:42.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sKKQn4fsI4aOxW3UDX9HhJ6gxMU=', 'Content-Length': '0'} 2024-04-19T23:42:42.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e04da0688f898feb-0066230172-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.324 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:42:42.324 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.324 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.324 DEBUG:boto:path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.324 DEBUG:boto:auth_path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.324 DEBUG:boto:Method: GET 2024-04-19T23:42:42.324 DEBUG:boto:Path: /aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.324 DEBUG:boto:Data: 2024-04-19T23:42:42.325 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.325 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.325 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.325 DEBUG:boto:Params: {} 2024-04-19T23:42:42.325 DEBUG:boto:Token: None 2024-04-19T23:42:42.325 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/?versions 2024-04-19T23:42:42.325 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs= 2024-04-19T23:42:42.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs=', 'Content-Length': '0'} 2024-04-19T23:42:42.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b37eefa2d754736c-0066230172-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.328 DEBUG:boto:b'aktlji-25999falseobjqQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcKtrue2024-04-19T23:42:10.369Zrgw-multisite-test-userMultisite Test UserNormalobjsDOAaiEwNhn-g-9rS9.nRm4a3XfM8sgfalse2024-04-19T23:42:10.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:42.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:42.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:42.328 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:42:42.328 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.329 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.329 DEBUG:boto:path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.329 DEBUG:boto:auth_path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.329 DEBUG:boto:Method: GET 2024-04-19T23:42:42.329 DEBUG:boto:Path: /aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.329 DEBUG:boto:Data: 2024-04-19T23:42:42.329 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.329 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.329 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.329 DEBUG:boto:Params: {} 2024-04-19T23:42:42.329 DEBUG:boto:Token: None 2024-04-19T23:42:42.329 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/?versions 2024-04-19T23:42:42.330 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs= 2024-04-19T23:42:42.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs=', 'Content-Length': '0'} 2024-04-19T23:42:42.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da97cee0dec260bf-0066230172-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.332 DEBUG:boto:b'aktlji-25999falseobjqQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcKtrue2024-04-19T23:42:10.369Zrgw-multisite-test-userMultisite Test UserNormalobjsDOAaiEwNhn-g-9rS9.nRm4a3XfM8sgfalse2024-04-19T23:42:10.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:42.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:42.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:42.333 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.333 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.333 DEBUG:boto:path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.333 DEBUG:boto:auth_path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.333 DEBUG:boto:Method: GET 2024-04-19T23:42:42.333 DEBUG:boto:Path: /aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.333 DEBUG:boto:Data: 2024-04-19T23:42:42.333 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.334 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.334 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.334 DEBUG:boto:Params: {} 2024-04-19T23:42:42.334 DEBUG:boto:Token: None 2024-04-19T23:42:42.334 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/?versions 2024-04-19T23:42:42.334 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs= 2024-04-19T23:42:42.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs=', 'Content-Length': '0'} 2024-04-19T23:42:42.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3434aeae0cd000f-0066230172-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.336 DEBUG:boto:b'aktlji-25999falseobjqQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcKtrue2024-04-19T23:42:10.369Zrgw-multisite-test-userMultisite Test UserNormalobjsDOAaiEwNhn-g-9rS9.nRm4a3XfM8sgfalse2024-04-19T23:42:10.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:42.337 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.337 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.337 DEBUG:boto:path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.337 DEBUG:boto:auth_path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.337 DEBUG:boto:Method: GET 2024-04-19T23:42:42.337 DEBUG:boto:Path: /aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.337 DEBUG:boto:Data: 2024-04-19T23:42:42.337 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.338 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.338 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.338 DEBUG:boto:Params: {} 2024-04-19T23:42:42.338 DEBUG:boto:Token: None 2024-04-19T23:42:42.338 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/?versions 2024-04-19T23:42:42.338 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs= 2024-04-19T23:42:42.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs=', 'Content-Length': '0'} 2024-04-19T23:42:42.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044f78e619d6c5a05-0066230172-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.386 DEBUG:boto:b'aktlji-25999falseobjqQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcKtrue2024-04-19T23:42:10.369Zrgw-multisite-test-userMultisite Test UserNormalobjsDOAaiEwNhn-g-9rS9.nRm4a3XfM8sgfalse2024-04-19T23:42:10.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:42.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:42.387 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.387 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.387 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.387 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:42.387 DEBUG:boto:Data: 2024-04-19T23:42:42.387 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.387 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.387 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.387 DEBUG:boto:Params: {} 2024-04-19T23:42:42.387 DEBUG:boto:Token: None 2024-04-19T23:42:42.388 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj 2024-04-19T23:42:42.388 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg= 2024-04-19T23:42:42.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg=', 'Content-Length': '0'} 2024-04-19T23:42:42.390 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006818fc94607ab372-0066230172-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.390 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.390 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.390 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.390 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:42.390 DEBUG:boto:Data: 2024-04-19T23:42:42.390 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.391 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.391 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.391 DEBUG:boto:Params: {} 2024-04-19T23:42:42.391 DEBUG:boto:Token: None 2024-04-19T23:42:42.391 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj 2024-04-19T23:42:42.391 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg= 2024-04-19T23:42:42.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg=', 'Content-Length': '0'} 2024-04-19T23:42:42.393 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004821acb5951e1d8c-0066230172-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:42.393 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.394 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.394 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.394 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.394 DEBUG:boto:Method: GET 2024-04-19T23:42:42.394 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.394 DEBUG:boto:Data: 2024-04-19T23:42:42.394 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.394 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.394 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.394 DEBUG:boto:Params: {} 2024-04-19T23:42:42.395 DEBUG:boto:Token: None 2024-04-19T23:42:42.395 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.395 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw= 2024-04-19T23:42:42.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw=', 'Content-Length': '0'} 2024-04-19T23:42:42.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b9cecfbb8a17ca7a-0066230172-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.397 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.397 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.397 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.397 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.397 DEBUG:boto:Method: GET 2024-04-19T23:42:42.397 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.397 DEBUG:boto:Data: 2024-04-19T23:42:42.397 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.397 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.398 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.398 DEBUG:boto:Params: {} 2024-04-19T23:42:42.398 DEBUG:boto:Token: None 2024-04-19T23:42:42.398 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.398 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw= 2024-04-19T23:42:42.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw=', 'Content-Length': '0'} 2024-04-19T23:42:42.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc20beba23135f92-0066230172-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.400 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.400 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.400 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.400 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.400 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.400 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.400 DEBUG:boto:Data: 2024-04-19T23:42:42.401 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.401 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.401 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.401 DEBUG:boto:Params: {} 2024-04-19T23:42:42.401 DEBUG:boto:Token: None 2024-04-19T23:42:42.401 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.401 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ZoOHnqCTRFR6Nah/8Yo6hhtFTE= 2024-04-19T23:42:42.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ZoOHnqCTRFR6Nah/8Yo6hhtFTE=', 'Content-Length': '0'} 2024-04-19T23:42:42.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000010f6f5e0c7358917-0066230172-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.403 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.403 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.403 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.403 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.403 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.404 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.404 DEBUG:boto:Data: 2024-04-19T23:42:42.404 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.404 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.404 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.404 DEBUG:boto:Params: {} 2024-04-19T23:42:42.404 DEBUG:boto:Token: None 2024-04-19T23:42:42.404 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.404 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ZoOHnqCTRFR6Nah/8Yo6hhtFTE= 2024-04-19T23:42:42.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ZoOHnqCTRFR6Nah/8Yo6hhtFTE=', 'Content-Length': '0'} 2024-04-19T23:42:42.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005761258cf84d61fb-0066230172-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:42.406 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.407 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.407 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.407 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.407 DEBUG:boto:Method: GET 2024-04-19T23:42:42.407 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.407 DEBUG:boto:Data: 2024-04-19T23:42:42.407 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.407 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.407 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.407 DEBUG:boto:Params: {} 2024-04-19T23:42:42.407 DEBUG:boto:Token: None 2024-04-19T23:42:42.408 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.408 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw= 2024-04-19T23:42:42.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw=', 'Content-Length': '0'} 2024-04-19T23:42:42.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c37ab1634a40489d-0066230172-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.409 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.410 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.410 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.410 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.410 DEBUG:boto:Method: GET 2024-04-19T23:42:42.410 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.410 DEBUG:boto:Data: 2024-04-19T23:42:42.410 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.410 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.410 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.410 DEBUG:boto:Params: {} 2024-04-19T23:42:42.411 DEBUG:boto:Token: None 2024-04-19T23:42:42.411 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.411 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw= 2024-04-19T23:42:42.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw=', 'Content-Length': '0'} 2024-04-19T23:42:42.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b9ecae61dbcb33d-0066230172-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.413 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.413 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.413 DEBUG:boto:path=/aktlji-25/?prefix=obj 2024-04-19T23:42:42.413 DEBUG:boto:auth_path=/aktlji-25/?prefix=obj 2024-04-19T23:42:42.413 DEBUG:boto:Method: GET 2024-04-19T23:42:42.413 DEBUG:boto:Path: /aktlji-25/?prefix=obj 2024-04-19T23:42:42.413 DEBUG:boto:Data: 2024-04-19T23:42:42.413 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.413 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.413 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.414 DEBUG:boto:Params: {} 2024-04-19T23:42:42.414 DEBUG:boto:Token: None 2024-04-19T23:42:42.414 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/ 2024-04-19T23:42:42.414 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NV/vjVzEyQzZ/UErL2fIevsQ/O4= 2024-04-19T23:42:42.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NV/vjVzEyQzZ/UErL2fIevsQ/O4=', 'Content-Length': '0'} 2024-04-19T23:42:42.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f407ef1d99ab0379-0066230172-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.458 DEBUG:boto:b'aktlji-25obj1000false' 2024-04-19T23:42:42.459 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.459 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.459 DEBUG:boto:path=/aktlji-25/?prefix=obj 2024-04-19T23:42:42.459 DEBUG:boto:auth_path=/aktlji-25/?prefix=obj 2024-04-19T23:42:42.459 DEBUG:boto:Method: GET 2024-04-19T23:42:42.459 DEBUG:boto:Path: /aktlji-25/?prefix=obj 2024-04-19T23:42:42.459 DEBUG:boto:Data: 2024-04-19T23:42:42.459 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.459 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.459 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.460 DEBUG:boto:Params: {} 2024-04-19T23:42:42.460 DEBUG:boto:Token: None 2024-04-19T23:42:42.460 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/ 2024-04-19T23:42:42.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NV/vjVzEyQzZ/UErL2fIevsQ/O4= 2024-04-19T23:42:42.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NV/vjVzEyQzZ/UErL2fIevsQ/O4=', 'Content-Length': '0'} 2024-04-19T23:42:42.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e17c32b9bbfb84d-0066230172-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.463 DEBUG:boto:b'aktlji-25obj1000false' 2024-04-19T23:42:42.463 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.463 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.463 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.463 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:42.464 DEBUG:boto:Data: 2024-04-19T23:42:42.464 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.464 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.464 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.464 DEBUG:boto:Params: {} 2024-04-19T23:42:42.464 DEBUG:boto:Token: None 2024-04-19T23:42:42.464 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj 2024-04-19T23:42:42.464 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg= 2024-04-19T23:42:42.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg=', 'Content-Length': '0'} 2024-04-19T23:42:42.466 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000093ef39e9adcb9d47-0066230172-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.466 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.467 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.467 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.467 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:42.467 DEBUG:boto:Data: 2024-04-19T23:42:42.467 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.467 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.467 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.467 DEBUG:boto:Params: {} 2024-04-19T23:42:42.467 DEBUG:boto:Token: None 2024-04-19T23:42:42.467 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj 2024-04-19T23:42:42.468 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg= 2024-04-19T23:42:42.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg=', 'Content-Length': '0'} 2024-04-19T23:42:42.470 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000fc14b195f4deec10-0066230172-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.470 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-25 zones={test-zone2, test-zone1} 2024-04-19T23:42:42.470 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-25 zones={test-zone3, test-zone1} 2024-04-19T23:42:42.470 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.470 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.470 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.471 DEBUG:boto:Path: /aktlji-25/ 2024-04-19T23:42:42.471 DEBUG:boto:Data: 2024-04-19T23:42:42.471 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.471 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.471 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.471 DEBUG:boto:Params: {} 2024-04-19T23:42:42.471 DEBUG:boto:Token: None 2024-04-19T23:42:42.471 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/ 2024-04-19T23:42:42.471 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sKKQn4fsI4aOxW3UDX9HhJ6gxMU= 2024-04-19T23:42:42.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sKKQn4fsI4aOxW3UDX9HhJ6gxMU=', 'Content-Length': '0'} 2024-04-19T23:42:42.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cee2f57ff157fd3-0066230172-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.474 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.474 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.474 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.474 DEBUG:boto:Path: /aktlji-25/ 2024-04-19T23:42:42.474 DEBUG:boto:Data: 2024-04-19T23:42:42.474 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.475 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.475 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.475 DEBUG:boto:Params: {} 2024-04-19T23:42:42.475 DEBUG:boto:Token: None 2024-04-19T23:42:42.475 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/ 2024-04-19T23:42:42.475 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sKKQn4fsI4aOxW3UDX9HhJ6gxMU= 2024-04-19T23:42:42.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sKKQn4fsI4aOxW3UDX9HhJ6gxMU=', 'Content-Length': '0'} 2024-04-19T23:42:42.477 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad6b7f23a324a9d5-0066230172-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.478 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:42:42.478 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.478 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.478 DEBUG:boto:path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.478 DEBUG:boto:auth_path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.478 DEBUG:boto:Method: GET 2024-04-19T23:42:42.478 DEBUG:boto:Path: /aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.478 DEBUG:boto:Data: 2024-04-19T23:42:42.478 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.478 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.479 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.479 DEBUG:boto:Params: {} 2024-04-19T23:42:42.479 DEBUG:boto:Token: None 2024-04-19T23:42:42.479 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/?versions 2024-04-19T23:42:42.479 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs= 2024-04-19T23:42:42.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs=', 'Content-Length': '0'} 2024-04-19T23:42:42.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d9fa1ec000154b9-0066230172-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.482 DEBUG:boto:b'aktlji-25999falseobjqQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcKtrue2024-04-19T23:42:10.369Zrgw-multisite-test-userMultisite Test UserNormalobjsDOAaiEwNhn-g-9rS9.nRm4a3XfM8sgfalse2024-04-19T23:42:10.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:42.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:42.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:42.482 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:42:42.483 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.483 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.483 DEBUG:boto:path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.483 DEBUG:boto:auth_path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.483 DEBUG:boto:Method: GET 2024-04-19T23:42:42.483 DEBUG:boto:Path: /aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.483 DEBUG:boto:Data: 2024-04-19T23:42:42.483 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.483 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.483 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.484 DEBUG:boto:Params: {} 2024-04-19T23:42:42.484 DEBUG:boto:Token: None 2024-04-19T23:42:42.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/?versions 2024-04-19T23:42:42.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs= 2024-04-19T23:42:42.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs=', 'Content-Length': '0'} 2024-04-19T23:42:42.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a979e5a95dc4da86-0066230172-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.487 DEBUG:boto:b'aktlji-25999falseobjqQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcKtrue2024-04-19T23:42:10.369Zrgw-multisite-test-userMultisite Test UserNormalobjsDOAaiEwNhn-g-9rS9.nRm4a3XfM8sgfalse2024-04-19T23:42:10.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:42.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:42.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:42.487 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.487 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.487 DEBUG:boto:path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.488 DEBUG:boto:auth_path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.488 DEBUG:boto:Method: GET 2024-04-19T23:42:42.488 DEBUG:boto:Path: /aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.488 DEBUG:boto:Data: 2024-04-19T23:42:42.488 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.488 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.488 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.488 DEBUG:boto:Params: {} 2024-04-19T23:42:42.488 DEBUG:boto:Token: None 2024-04-19T23:42:42.488 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/?versions 2024-04-19T23:42:42.489 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs= 2024-04-19T23:42:42.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs=', 'Content-Length': '0'} 2024-04-19T23:42:42.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009891943118ec5133-0066230172-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.491 DEBUG:boto:b'aktlji-25999falseobjqQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcKtrue2024-04-19T23:42:10.369Zrgw-multisite-test-userMultisite Test UserNormalobjsDOAaiEwNhn-g-9rS9.nRm4a3XfM8sgfalse2024-04-19T23:42:10.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:42.492 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.492 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.492 DEBUG:boto:path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.492 DEBUG:boto:auth_path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.492 DEBUG:boto:Method: GET 2024-04-19T23:42:42.492 DEBUG:boto:Path: /aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.492 DEBUG:boto:Data: 2024-04-19T23:42:42.492 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.492 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.492 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.493 DEBUG:boto:Params: {} 2024-04-19T23:42:42.493 DEBUG:boto:Token: None 2024-04-19T23:42:42.493 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/?versions 2024-04-19T23:42:42.493 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs= 2024-04-19T23:42:42.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs=', 'Content-Length': '0'} 2024-04-19T23:42:42.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000067785ab07f36cb6-0066230172-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.538 DEBUG:boto:b'aktlji-25999falseobjqQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcKtrue2024-04-19T23:42:10.369Zrgw-multisite-test-userMultisite Test UserNormalobjsDOAaiEwNhn-g-9rS9.nRm4a3XfM8sgfalse2024-04-19T23:42:10.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:42.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:42.539 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.539 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.539 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.539 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:42.539 DEBUG:boto:Data: 2024-04-19T23:42:42.539 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.539 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.539 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.539 DEBUG:boto:Params: {} 2024-04-19T23:42:42.540 DEBUG:boto:Token: None 2024-04-19T23:42:42.540 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj 2024-04-19T23:42:42.540 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg= 2024-04-19T23:42:42.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg=', 'Content-Length': '0'} 2024-04-19T23:42:42.542 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000037d06750471ccb51-0066230172-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.542 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.542 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.542 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.542 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:42.542 DEBUG:boto:Data: 2024-04-19T23:42:42.542 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.543 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.543 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.543 DEBUG:boto:Params: {} 2024-04-19T23:42:42.543 DEBUG:boto:Token: None 2024-04-19T23:42:42.543 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj 2024-04-19T23:42:42.543 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg= 2024-04-19T23:42:42.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg=', 'Content-Length': '0'} 2024-04-19T23:42:42.545 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000504a939f5f1aa38f-0066230172-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:42.546 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.546 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.546 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.546 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.546 DEBUG:boto:Method: GET 2024-04-19T23:42:42.546 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.546 DEBUG:boto:Data: 2024-04-19T23:42:42.546 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.546 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.547 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.547 DEBUG:boto:Params: {} 2024-04-19T23:42:42.547 DEBUG:boto:Token: None 2024-04-19T23:42:42.547 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.547 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw= 2024-04-19T23:42:42.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw=', 'Content-Length': '0'} 2024-04-19T23:42:42.549 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dfca8c29d649782e-0066230172-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.549 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.549 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.549 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.549 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.549 DEBUG:boto:Method: GET 2024-04-19T23:42:42.549 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.550 DEBUG:boto:Data: 2024-04-19T23:42:42.550 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.550 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.550 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.550 DEBUG:boto:Params: {} 2024-04-19T23:42:42.550 DEBUG:boto:Token: None 2024-04-19T23:42:42.550 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.551 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw= 2024-04-19T23:42:42.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw=', 'Content-Length': '0'} 2024-04-19T23:42:42.553 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017f964688c78d9f2-0066230172-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.553 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.553 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.553 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.553 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.553 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.553 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.553 DEBUG:boto:Data: 2024-04-19T23:42:42.553 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.553 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.554 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.554 DEBUG:boto:Params: {} 2024-04-19T23:42:42.554 DEBUG:boto:Token: None 2024-04-19T23:42:42.554 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.554 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ZoOHnqCTRFR6Nah/8Yo6hhtFTE= 2024-04-19T23:42:42.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ZoOHnqCTRFR6Nah/8Yo6hhtFTE=', 'Content-Length': '0'} 2024-04-19T23:42:42.556 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000240a5c1cb821de00-0066230172-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.556 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.556 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.556 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.556 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.557 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.557 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.557 DEBUG:boto:Data: 2024-04-19T23:42:42.557 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.557 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.557 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.557 DEBUG:boto:Params: {} 2024-04-19T23:42:42.557 DEBUG:boto:Token: None 2024-04-19T23:42:42.557 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.557 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ZoOHnqCTRFR6Nah/8Yo6hhtFTE= 2024-04-19T23:42:42.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ZoOHnqCTRFR6Nah/8Yo6hhtFTE=', 'Content-Length': '0'} 2024-04-19T23:42:42.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f9e2cec04b601799-0066230172-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:42.559 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.559 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.560 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.560 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.560 DEBUG:boto:Method: GET 2024-04-19T23:42:42.560 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.560 DEBUG:boto:Data: 2024-04-19T23:42:42.560 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.560 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.560 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.560 DEBUG:boto:Params: {} 2024-04-19T23:42:42.560 DEBUG:boto:Token: None 2024-04-19T23:42:42.560 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.561 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw= 2024-04-19T23:42:42.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw=', 'Content-Length': '0'} 2024-04-19T23:42:42.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006dda3b5eed787f62-0066230172-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.562 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.563 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.563 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.563 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.563 DEBUG:boto:Method: GET 2024-04-19T23:42:42.563 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.563 DEBUG:boto:Data: 2024-04-19T23:42:42.563 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.563 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.563 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.563 DEBUG:boto:Params: {} 2024-04-19T23:42:42.563 DEBUG:boto:Token: None 2024-04-19T23:42:42.563 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.564 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw= 2024-04-19T23:42:42.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw=', 'Content-Length': '0'} 2024-04-19T23:42:42.565 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000004f7660188b687ad-0066230172-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.566 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.566 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.566 DEBUG:boto:path=/aktlji-25/?prefix=obj 2024-04-19T23:42:42.566 DEBUG:boto:auth_path=/aktlji-25/?prefix=obj 2024-04-19T23:42:42.566 DEBUG:boto:Method: GET 2024-04-19T23:42:42.566 DEBUG:boto:Path: /aktlji-25/?prefix=obj 2024-04-19T23:42:42.566 DEBUG:boto:Data: 2024-04-19T23:42:42.566 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.566 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.567 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.567 DEBUG:boto:Params: {} 2024-04-19T23:42:42.567 DEBUG:boto:Token: None 2024-04-19T23:42:42.567 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/ 2024-04-19T23:42:42.567 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NV/vjVzEyQzZ/UErL2fIevsQ/O4= 2024-04-19T23:42:42.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NV/vjVzEyQzZ/UErL2fIevsQ/O4=', 'Content-Length': '0'} 2024-04-19T23:42:42.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007159ab9226b90ab6-0066230172-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.570 DEBUG:boto:b'aktlji-25obj1000false' 2024-04-19T23:42:42.570 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.570 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.570 DEBUG:boto:path=/aktlji-25/?prefix=obj 2024-04-19T23:42:42.571 DEBUG:boto:auth_path=/aktlji-25/?prefix=obj 2024-04-19T23:42:42.571 DEBUG:boto:Method: GET 2024-04-19T23:42:42.571 DEBUG:boto:Path: /aktlji-25/?prefix=obj 2024-04-19T23:42:42.571 DEBUG:boto:Data: 2024-04-19T23:42:42.571 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.571 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.571 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.571 DEBUG:boto:Params: {} 2024-04-19T23:42:42.571 DEBUG:boto:Token: None 2024-04-19T23:42:42.571 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/ 2024-04-19T23:42:42.572 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NV/vjVzEyQzZ/UErL2fIevsQ/O4= 2024-04-19T23:42:42.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NV/vjVzEyQzZ/UErL2fIevsQ/O4=', 'Content-Length': '0'} 2024-04-19T23:42:42.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019fbefd5ee8e3229-0066230172-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.574 DEBUG:boto:b'aktlji-25obj1000false' 2024-04-19T23:42:42.574 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.574 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.574 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.575 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:42.575 DEBUG:boto:Data: 2024-04-19T23:42:42.575 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.575 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.575 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.575 DEBUG:boto:Params: {} 2024-04-19T23:42:42.575 DEBUG:boto:Token: None 2024-04-19T23:42:42.575 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj 2024-04-19T23:42:42.575 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg= 2024-04-19T23:42:42.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg=', 'Content-Length': '0'} 2024-04-19T23:42:42.577 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000067faa37e501e6f51-0066230172-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.577 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.578 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.578 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.578 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:42.578 DEBUG:boto:Data: 2024-04-19T23:42:42.578 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.578 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:42.578 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.578 DEBUG:boto:Params: {} 2024-04-19T23:42:42.578 DEBUG:boto:Token: None 2024-04-19T23:42:42.578 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj 2024-04-19T23:42:42.579 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg= 2024-04-19T23:42:42.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg=', 'Content-Length': '0'} 2024-04-19T23:42:42.581 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000018ed2cf1128eabe1-0066230172-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.581 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-25 zones={test-zone3, test-zone1} 2024-04-19T23:42:42.581 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-25 zones={test-zone3, test-zone2} 2024-04-19T23:42:42.581 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.581 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.581 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.582 DEBUG:boto:Path: /aktlji-25/ 2024-04-19T23:42:42.582 DEBUG:boto:Data: 2024-04-19T23:42:42.582 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.582 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.582 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.582 DEBUG:boto:Params: {} 2024-04-19T23:42:42.582 DEBUG:boto:Token: None 2024-04-19T23:42:42.582 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/ 2024-04-19T23:42:42.582 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sKKQn4fsI4aOxW3UDX9HhJ6gxMU= 2024-04-19T23:42:42.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sKKQn4fsI4aOxW3UDX9HhJ6gxMU=', 'Content-Length': '0'} 2024-04-19T23:42:42.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009204e33435afaa59-0066230172-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.585 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.585 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.585 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.585 DEBUG:boto:Path: /aktlji-25/ 2024-04-19T23:42:42.585 DEBUG:boto:Data: 2024-04-19T23:42:42.586 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.586 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.586 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.586 DEBUG:boto:Params: {} 2024-04-19T23:42:42.586 DEBUG:boto:Token: None 2024-04-19T23:42:42.586 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/ 2024-04-19T23:42:42.586 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sKKQn4fsI4aOxW3UDX9HhJ6gxMU= 2024-04-19T23:42:42.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sKKQn4fsI4aOxW3UDX9HhJ6gxMU=', 'Content-Length': '0'} 2024-04-19T23:42:42.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000429bbf5173ed0b54-0066230172-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.589 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:42:42.589 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.589 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.590 DEBUG:boto:path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.590 DEBUG:boto:auth_path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.590 DEBUG:boto:Method: GET 2024-04-19T23:42:42.590 DEBUG:boto:Path: /aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.590 DEBUG:boto:Data: 2024-04-19T23:42:42.590 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.590 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.590 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.590 DEBUG:boto:Params: {} 2024-04-19T23:42:42.590 DEBUG:boto:Token: None 2024-04-19T23:42:42.590 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/?versions 2024-04-19T23:42:42.591 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs= 2024-04-19T23:42:42.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs=', 'Content-Length': '0'} 2024-04-19T23:42:42.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031060c1cc3f30cf7-0066230172-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.593 DEBUG:boto:b'aktlji-25999falseobjqQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcKtrue2024-04-19T23:42:10.369Zrgw-multisite-test-userMultisite Test UserNormalobjsDOAaiEwNhn-g-9rS9.nRm4a3XfM8sgfalse2024-04-19T23:42:10.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:42.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:42.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:42.594 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:42:42.594 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.594 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.594 DEBUG:boto:path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.594 DEBUG:boto:auth_path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.594 DEBUG:boto:Method: GET 2024-04-19T23:42:42.594 DEBUG:boto:Path: /aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.594 DEBUG:boto:Data: 2024-04-19T23:42:42.595 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.595 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.595 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.595 DEBUG:boto:Params: {} 2024-04-19T23:42:42.595 DEBUG:boto:Token: None 2024-04-19T23:42:42.595 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/?versions 2024-04-19T23:42:42.595 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs= 2024-04-19T23:42:42.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs=', 'Content-Length': '0'} 2024-04-19T23:42:42.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000912cb2885649d569-0066230172-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.598 DEBUG:boto:b'aktlji-25999falseobjqQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcKtrue2024-04-19T23:42:10.369Zrgw-multisite-test-userMultisite Test UserNormalobjsDOAaiEwNhn-g-9rS9.nRm4a3XfM8sgfalse2024-04-19T23:42:10.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:42.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:42.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:42.598 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.598 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.599 DEBUG:boto:path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.599 DEBUG:boto:auth_path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.599 DEBUG:boto:Method: GET 2024-04-19T23:42:42.599 DEBUG:boto:Path: /aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.599 DEBUG:boto:Data: 2024-04-19T23:42:42.599 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.599 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.599 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.599 DEBUG:boto:Params: {} 2024-04-19T23:42:42.599 DEBUG:boto:Token: None 2024-04-19T23:42:42.599 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/?versions 2024-04-19T23:42:42.600 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs= 2024-04-19T23:42:42.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs=', 'Content-Length': '0'} 2024-04-19T23:42:42.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae539321a8756c2f-0066230172-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.602 DEBUG:boto:b'aktlji-25999falseobjqQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcKtrue2024-04-19T23:42:10.369Zrgw-multisite-test-userMultisite Test UserNormalobjsDOAaiEwNhn-g-9rS9.nRm4a3XfM8sgfalse2024-04-19T23:42:10.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:42.602 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.603 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.603 DEBUG:boto:path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.603 DEBUG:boto:auth_path=/aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.603 DEBUG:boto:Method: GET 2024-04-19T23:42:42.603 DEBUG:boto:Path: /aktlji-25/?versions&max-keys=999 2024-04-19T23:42:42.603 DEBUG:boto:Data: 2024-04-19T23:42:42.603 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.603 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.603 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.603 DEBUG:boto:Params: {} 2024-04-19T23:42:42.603 DEBUG:boto:Token: None 2024-04-19T23:42:42.604 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/?versions 2024-04-19T23:42:42.604 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs= 2024-04-19T23:42:42.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N2QFNj+ULipHHQihvl5RjrmBehs=', 'Content-Length': '0'} 2024-04-19T23:42:42.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c8ad7672c5e5479-0066230172-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.606 DEBUG:boto:b'aktlji-25999falseobjqQhjnXxp5qX3KLv4FH-Tr4m5Crp5vcKtrue2024-04-19T23:42:10.369Zrgw-multisite-test-userMultisite Test UserNormalobjsDOAaiEwNhn-g-9rS9.nRm4a3XfM8sgfalse2024-04-19T23:42:10.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:42.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:42.607 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.607 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.607 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.607 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:42.607 DEBUG:boto:Data: 2024-04-19T23:42:42.607 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.607 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.607 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.607 DEBUG:boto:Params: {} 2024-04-19T23:42:42.607 DEBUG:boto:Token: None 2024-04-19T23:42:42.608 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj 2024-04-19T23:42:42.608 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg= 2024-04-19T23:42:42.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg=', 'Content-Length': '0'} 2024-04-19T23:42:42.610 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006a5ab95bab151c48-0066230172-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.610 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.610 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.610 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.610 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:42.610 DEBUG:boto:Data: 2024-04-19T23:42:42.611 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.611 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.611 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.611 DEBUG:boto:Params: {} 2024-04-19T23:42:42.611 DEBUG:boto:Token: None 2024-04-19T23:42:42.611 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj 2024-04-19T23:42:42.611 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg= 2024-04-19T23:42:42.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg=', 'Content-Length': '0'} 2024-04-19T23:42:42.613 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000000ee8a020592f3bd6-0066230172-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:42.614 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.614 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.614 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.614 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.614 DEBUG:boto:Method: GET 2024-04-19T23:42:42.614 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.614 DEBUG:boto:Data: 2024-04-19T23:42:42.614 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.614 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.615 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.615 DEBUG:boto:Params: {} 2024-04-19T23:42:42.615 DEBUG:boto:Token: None 2024-04-19T23:42:42.615 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.615 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw= 2024-04-19T23:42:42.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw=', 'Content-Length': '0'} 2024-04-19T23:42:42.617 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f8172ac591ad3ec-0066230172-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.617 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.617 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.617 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.617 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.617 DEBUG:boto:Method: GET 2024-04-19T23:42:42.617 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.618 DEBUG:boto:Data: 2024-04-19T23:42:42.618 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.618 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.618 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.618 DEBUG:boto:Params: {} 2024-04-19T23:42:42.618 DEBUG:boto:Token: None 2024-04-19T23:42:42.618 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.618 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw= 2024-04-19T23:42:42.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw=', 'Content-Length': '0'} 2024-04-19T23:42:42.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b2aaa6d6f8d12bb-0066230172-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.621 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.621 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.621 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.621 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.621 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.621 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.621 DEBUG:boto:Data: 2024-04-19T23:42:42.621 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.621 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.621 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.621 DEBUG:boto:Params: {} 2024-04-19T23:42:42.622 DEBUG:boto:Token: None 2024-04-19T23:42:42.622 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.622 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ZoOHnqCTRFR6Nah/8Yo6hhtFTE= 2024-04-19T23:42:42.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ZoOHnqCTRFR6Nah/8Yo6hhtFTE=', 'Content-Length': '0'} 2024-04-19T23:42:42.624 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000004603424fe7e47d5-0066230172-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.624 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.625 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.625 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.625 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.625 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.625 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.625 DEBUG:boto:Data: 2024-04-19T23:42:42.625 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.625 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.625 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.625 DEBUG:boto:Params: {} 2024-04-19T23:42:42.625 DEBUG:boto:Token: None 2024-04-19T23:42:42.625 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.626 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0ZoOHnqCTRFR6Nah/8Yo6hhtFTE= 2024-04-19T23:42:42.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0ZoOHnqCTRFR6Nah/8Yo6hhtFTE=', 'Content-Length': '0'} 2024-04-19T23:42:42.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002e151e10a07dd4d5-0066230172-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:42.628 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.628 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.628 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.628 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.628 DEBUG:boto:Method: GET 2024-04-19T23:42:42.628 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.628 DEBUG:boto:Data: 2024-04-19T23:42:42.628 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.628 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.628 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.629 DEBUG:boto:Params: {} 2024-04-19T23:42:42.629 DEBUG:boto:Token: None 2024-04-19T23:42:42.629 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.629 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw= 2024-04-19T23:42:42.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw=', 'Content-Length': '0'} 2024-04-19T23:42:42.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e00074d37bed75e4-0066230172-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.631 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.631 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.631 DEBUG:boto:path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.631 DEBUG:boto:auth_path=/aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.631 DEBUG:boto:Method: GET 2024-04-19T23:42:42.632 DEBUG:boto:Path: /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.632 DEBUG:boto:Data: 2024-04-19T23:42:42.632 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.632 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.632 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.632 DEBUG:boto:Params: {} 2024-04-19T23:42:42.632 DEBUG:boto:Token: None 2024-04-19T23:42:42.632 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj?versionId=sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg 2024-04-19T23:42:42.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw= 2024-04-19T23:42:42.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sZptlidSxL2C/E6xZOo/+qragFw=', 'Content-Length': '0'} 2024-04-19T23:42:42.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'sDOAaiEwNhn-g-9rS9.nRm4a3XfM8sg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c54e497f9840d83a-0066230172-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.634 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.634 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.635 DEBUG:boto:path=/aktlji-25/?prefix=obj 2024-04-19T23:42:42.635 DEBUG:boto:auth_path=/aktlji-25/?prefix=obj 2024-04-19T23:42:42.635 DEBUG:boto:Method: GET 2024-04-19T23:42:42.635 DEBUG:boto:Path: /aktlji-25/?prefix=obj 2024-04-19T23:42:42.635 DEBUG:boto:Data: 2024-04-19T23:42:42.635 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.635 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.635 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.635 DEBUG:boto:Params: {} 2024-04-19T23:42:42.635 DEBUG:boto:Token: None 2024-04-19T23:42:42.635 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/ 2024-04-19T23:42:42.636 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NV/vjVzEyQzZ/UErL2fIevsQ/O4= 2024-04-19T23:42:42.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NV/vjVzEyQzZ/UErL2fIevsQ/O4=', 'Content-Length': '0'} 2024-04-19T23:42:42.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e3062199c301a8b-0066230172-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.678 DEBUG:boto:b'aktlji-25obj1000false' 2024-04-19T23:42:42.679 DEBUG:boto:path=/aktlji-25/ 2024-04-19T23:42:42.679 DEBUG:boto:auth_path=/aktlji-25/ 2024-04-19T23:42:42.679 DEBUG:boto:path=/aktlji-25/?prefix=obj 2024-04-19T23:42:42.679 DEBUG:boto:auth_path=/aktlji-25/?prefix=obj 2024-04-19T23:42:42.679 DEBUG:boto:Method: GET 2024-04-19T23:42:42.679 DEBUG:boto:Path: /aktlji-25/?prefix=obj 2024-04-19T23:42:42.679 DEBUG:boto:Data: 2024-04-19T23:42:42.679 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.679 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.679 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.679 DEBUG:boto:Params: {} 2024-04-19T23:42:42.680 DEBUG:boto:Token: None 2024-04-19T23:42:42.680 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/ 2024-04-19T23:42:42.680 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NV/vjVzEyQzZ/UErL2fIevsQ/O4= 2024-04-19T23:42:42.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NV/vjVzEyQzZ/UErL2fIevsQ/O4=', 'Content-Length': '0'} 2024-04-19T23:42:42.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7c889fd5e97b78b-0066230172-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.683 DEBUG:boto:b'aktlji-25obj1000false' 2024-04-19T23:42:42.683 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.683 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.683 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.683 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:42.683 DEBUG:boto:Data: 2024-04-19T23:42:42.684 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.684 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:42.684 DEBUG:boto:Port: 8001 2024-04-19T23:42:42.684 DEBUG:boto:Params: {} 2024-04-19T23:42:42.684 DEBUG:boto:Token: None 2024-04-19T23:42:42.684 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj 2024-04-19T23:42:42.684 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg= 2024-04-19T23:42:42.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg=', 'Content-Length': '0'} 2024-04-19T23:42:42.686 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000009da9b44fea557c63-0066230172-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.686 DEBUG:boto:path=/aktlji-25/obj 2024-04-19T23:42:42.686 DEBUG:boto:auth_path=/aktlji-25/obj 2024-04-19T23:42:42.687 DEBUG:boto:Method: HEAD 2024-04-19T23:42:42.687 DEBUG:boto:Path: /aktlji-25/obj 2024-04-19T23:42:42.687 DEBUG:boto:Data: 2024-04-19T23:42:42.687 DEBUG:boto:Headers: {} 2024-04-19T23:42:42.687 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:42.687 DEBUG:boto:Port: 8000 2024-04-19T23:42:42.687 DEBUG:boto:Params: {} 2024-04-19T23:42:42.687 DEBUG:boto:Token: None 2024-04-19T23:42:42.687 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:42 GMT /aktlji-25/obj 2024-04-19T23:42:42.687 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg= 2024-04-19T23:42:42.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rA/glNcIAMKC21Y/JV6pj9yHNGg=', 'Content-Length': '0'} 2024-04-19T23:42:42.689 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000b305edcf486ab14d-0066230172-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:42.690 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-25 zones={test-zone3, test-zone2} 2024-04-19T23:42:42.690 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-26 2024-04-19T23:42:42.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:42.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:42.690 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:42.758 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:42.759 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:42.803+0000 7fca40bfa780 1 bucket sync caught up with source: 2024-04-19T23:42:42.801 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000002.21.6, , ] 2024-04-19T23:42:42.801 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000002.21.6, , ] 2024-04-19T23:42:42.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:42.803+0000 7fca40bfa780 0 bucket checkpoint complete 2024-04-19T23:42:42.816 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-26 2024-04-19T23:42:42.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:42.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:42.817 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:42.890 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:42.891 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:42.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:42.987+0000 7fe508b9e780 1 bucket sync caught up with source: 2024-04-19T23:42:42.986 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000002.21.6, , ] 2024-04-19T23:42:42.986 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000002.21.6, , ] 2024-04-19T23:42:42.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:42.987+0000 7fe508b9e780 0 bucket checkpoint complete 2024-04-19T23:42:43.000 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-26 2024-04-19T23:42:43.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:43.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:43.000 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:43.061 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:43.061 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:43.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:43.104+0000 7f4aad329780 1 bucket sync caught up with source: 2024-04-19T23:42:43.102 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000002.147.6, , ] 2024-04-19T23:42:43.102 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000002.147.6, , ] 2024-04-19T23:42:43.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:43.104+0000 7f4aad329780 0 bucket checkpoint complete 2024-04-19T23:42:43.117 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-26 2024-04-19T23:42:43.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:43.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:43.118 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:43.179 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:43.180 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:43.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:43.221+0000 7fda237e3780 1 bucket sync caught up with source: 2024-04-19T23:42:43.219 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000002.147.6, , ] 2024-04-19T23:42:43.220 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000002.147.6, , ] 2024-04-19T23:42:43.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:43.221+0000 7fda237e3780 0 bucket checkpoint complete 2024-04-19T23:42:43.233 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-26 2024-04-19T23:42:43.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:43.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:43.234 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:43.296 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:43.296 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:43.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:43.338+0000 7f0217dc8780 1 bucket sync caught up with source: 2024-04-19T23:42:43.336 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000003.59.12, , ] 2024-04-19T23:42:43.337 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.59.12, , ] 2024-04-19T23:42:43.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:43.338+0000 7f0217dc8780 0 bucket checkpoint complete 2024-04-19T23:42:43.351 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-26 2024-04-19T23:42:43.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:43.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:43.352 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:43.418 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:43.418 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:43.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:43.461+0000 7f5838f8a780 1 bucket sync caught up with source: 2024-04-19T23:42:43.460 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000003.59.12, , ] 2024-04-19T23:42:43.460 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000003.59.12, , ] 2024-04-19T23:42:43.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:43.461+0000 7f5838f8a780 0 bucket checkpoint complete 2024-04-19T23:42:43.475 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-26 zones={test-zone2, test-zone1} 2024-04-19T23:42:43.475 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.475 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.475 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.475 DEBUG:boto:Path: /aktlji-26/ 2024-04-19T23:42:43.475 DEBUG:boto:Data: 2024-04-19T23:42:43.475 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.476 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.476 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.476 DEBUG:boto:Params: {} 2024-04-19T23:42:43.476 DEBUG:boto:Token: None 2024-04-19T23:42:43.476 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/ 2024-04-19T23:42:43.476 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Jq+2plmAskKnfDb0yyaE164Oo+o= 2024-04-19T23:42:43.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Jq+2plmAskKnfDb0yyaE164Oo+o=', 'Content-Length': '0'} 2024-04-19T23:42:43.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c164e8338a00d03f-0066230173-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.479 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.479 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.479 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.479 DEBUG:boto:Path: /aktlji-26/ 2024-04-19T23:42:43.479 DEBUG:boto:Data: 2024-04-19T23:42:43.479 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.479 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.479 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.479 DEBUG:boto:Params: {} 2024-04-19T23:42:43.480 DEBUG:boto:Token: None 2024-04-19T23:42:43.480 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/ 2024-04-19T23:42:43.480 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Jq+2plmAskKnfDb0yyaE164Oo+o= 2024-04-19T23:42:43.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Jq+2plmAskKnfDb0yyaE164Oo+o=', 'Content-Length': '0'} 2024-04-19T23:42:43.483 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093e9931f47d008de-0066230173-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.483 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:42:43.483 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.483 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.483 DEBUG:boto:path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.483 DEBUG:boto:auth_path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.483 DEBUG:boto:Method: GET 2024-04-19T23:42:43.484 DEBUG:boto:Path: /aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.484 DEBUG:boto:Data: 2024-04-19T23:42:43.484 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.484 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.484 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.484 DEBUG:boto:Params: {} 2024-04-19T23:42:43.484 DEBUG:boto:Token: None 2024-04-19T23:42:43.484 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/?versions 2024-04-19T23:42:43.484 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg= 2024-04-19T23:42:43.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg=', 'Content-Length': '0'} 2024-04-19T23:42:43.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8c27182484d0fde-0066230173-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.487 DEBUG:boto:b'aktlji-26999falseobjiJzYuqY05U-Eu1l-5iINYoSVqxQDj2ztrue2024-04-19T23:42:10.406Zrgw-multisite-test-userMultisite Test UserNormalobjaFg1G93maO0BVFcQAVh5xHYwflNTsyLfalse2024-04-19T23:42:10.386Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:43.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:43.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:43.488 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:42:43.488 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.488 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.488 DEBUG:boto:path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.488 DEBUG:boto:auth_path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.488 DEBUG:boto:Method: GET 2024-04-19T23:42:43.488 DEBUG:boto:Path: /aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.488 DEBUG:boto:Data: 2024-04-19T23:42:43.488 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.488 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.489 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.489 DEBUG:boto:Params: {} 2024-04-19T23:42:43.489 DEBUG:boto:Token: None 2024-04-19T23:42:43.489 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/?versions 2024-04-19T23:42:43.489 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg= 2024-04-19T23:42:43.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg=', 'Content-Length': '0'} 2024-04-19T23:42:43.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003456a122f5803619-0066230173-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.534 DEBUG:boto:b'aktlji-26999falseobjiJzYuqY05U-Eu1l-5iINYoSVqxQDj2ztrue2024-04-19T23:42:10.406Zrgw-multisite-test-userMultisite Test UserNormalobjaFg1G93maO0BVFcQAVh5xHYwflNTsyLfalse2024-04-19T23:42:10.386Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:43.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:43.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:43.535 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.535 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.535 DEBUG:boto:path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.536 DEBUG:boto:auth_path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.536 DEBUG:boto:Method: GET 2024-04-19T23:42:43.536 DEBUG:boto:Path: /aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.536 DEBUG:boto:Data: 2024-04-19T23:42:43.536 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.536 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.536 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.536 DEBUG:boto:Params: {} 2024-04-19T23:42:43.536 DEBUG:boto:Token: None 2024-04-19T23:42:43.536 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/?versions 2024-04-19T23:42:43.537 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg= 2024-04-19T23:42:43.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg=', 'Content-Length': '0'} 2024-04-19T23:42:43.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000597168d863707a6c-0066230173-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.539 DEBUG:boto:b'aktlji-26999falseobjiJzYuqY05U-Eu1l-5iINYoSVqxQDj2ztrue2024-04-19T23:42:10.406Zrgw-multisite-test-userMultisite Test UserNormalobjaFg1G93maO0BVFcQAVh5xHYwflNTsyLfalse2024-04-19T23:42:10.386Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:43.539 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.540 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.540 DEBUG:boto:path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.540 DEBUG:boto:auth_path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.540 DEBUG:boto:Method: GET 2024-04-19T23:42:43.540 DEBUG:boto:Path: /aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.540 DEBUG:boto:Data: 2024-04-19T23:42:43.540 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.540 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.540 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.540 DEBUG:boto:Params: {} 2024-04-19T23:42:43.540 DEBUG:boto:Token: None 2024-04-19T23:42:43.541 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/?versions 2024-04-19T23:42:43.541 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg= 2024-04-19T23:42:43.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg=', 'Content-Length': '0'} 2024-04-19T23:42:43.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000729bde7b63e94000-0066230173-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.544 DEBUG:boto:b'aktlji-26999falseobjiJzYuqY05U-Eu1l-5iINYoSVqxQDj2ztrue2024-04-19T23:42:10.406Zrgw-multisite-test-userMultisite Test UserNormalobjaFg1G93maO0BVFcQAVh5xHYwflNTsyLfalse2024-04-19T23:42:10.386Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:43.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:43.544 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.544 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.544 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.545 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:43.545 DEBUG:boto:Data: 2024-04-19T23:42:43.545 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.545 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.545 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.545 DEBUG:boto:Params: {} 2024-04-19T23:42:43.545 DEBUG:boto:Token: None 2024-04-19T23:42:43.545 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj 2024-04-19T23:42:43.545 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw= 2024-04-19T23:42:43.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw=', 'Content-Length': '0'} 2024-04-19T23:42:43.547 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000820eff744510847a-0066230173-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.548 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.548 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.548 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.548 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:43.548 DEBUG:boto:Data: 2024-04-19T23:42:43.548 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.548 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.548 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.548 DEBUG:boto:Params: {} 2024-04-19T23:42:43.548 DEBUG:boto:Token: None 2024-04-19T23:42:43.549 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj 2024-04-19T23:42:43.549 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw= 2024-04-19T23:42:43.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw=', 'Content-Length': '0'} 2024-04-19T23:42:43.551 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000033927826dbd36878-0066230173-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:43.551 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.551 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.551 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.551 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.551 DEBUG:boto:Method: GET 2024-04-19T23:42:43.551 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.552 DEBUG:boto:Data: 2024-04-19T23:42:43.552 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.552 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.552 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.552 DEBUG:boto:Params: {} 2024-04-19T23:42:43.552 DEBUG:boto:Token: None 2024-04-19T23:42:43.552 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.552 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk= 2024-04-19T23:42:43.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk=', 'Content-Length': '0'} 2024-04-19T23:42:43.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3f1c7efbd3767bc-0066230173-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.554 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.554 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.555 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.555 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.555 DEBUG:boto:Method: GET 2024-04-19T23:42:43.555 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.555 DEBUG:boto:Data: 2024-04-19T23:42:43.555 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.555 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.555 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.555 DEBUG:boto:Params: {} 2024-04-19T23:42:43.556 DEBUG:boto:Token: None 2024-04-19T23:42:43.556 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.556 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk= 2024-04-19T23:42:43.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk=', 'Content-Length': '0'} 2024-04-19T23:42:43.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077136d18dc73169c-0066230173-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.558 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.558 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.558 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.558 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.558 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.559 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.559 DEBUG:boto:Data: 2024-04-19T23:42:43.559 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.559 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.559 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.559 DEBUG:boto:Params: {} 2024-04-19T23:42:43.559 DEBUG:boto:Token: None 2024-04-19T23:42:43.559 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.559 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gWqBWEOzjqI6KXHddINDh+QIYf0= 2024-04-19T23:42:43.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gWqBWEOzjqI6KXHddINDh+QIYf0=', 'Content-Length': '0'} 2024-04-19T23:42:43.561 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000311ac53964aa24c9-0066230173-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.562 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.562 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.562 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.562 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.562 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.562 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.562 DEBUG:boto:Data: 2024-04-19T23:42:43.562 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.562 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.563 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.563 DEBUG:boto:Params: {} 2024-04-19T23:42:43.563 DEBUG:boto:Token: None 2024-04-19T23:42:43.563 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.563 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gWqBWEOzjqI6KXHddINDh+QIYf0= 2024-04-19T23:42:43.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gWqBWEOzjqI6KXHddINDh+QIYf0=', 'Content-Length': '0'} 2024-04-19T23:42:43.565 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000464b6a2b692241c6-0066230173-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:43.566 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.566 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.566 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.566 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.566 DEBUG:boto:Method: GET 2024-04-19T23:42:43.566 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.566 DEBUG:boto:Data: 2024-04-19T23:42:43.566 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.566 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.567 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.567 DEBUG:boto:Params: {} 2024-04-19T23:42:43.567 DEBUG:boto:Token: None 2024-04-19T23:42:43.567 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.567 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk= 2024-04-19T23:42:43.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk=', 'Content-Length': '0'} 2024-04-19T23:42:43.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007414a9ada0ce93f6-0066230173-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.569 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.569 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.569 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.570 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.570 DEBUG:boto:Method: GET 2024-04-19T23:42:43.570 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.570 DEBUG:boto:Data: 2024-04-19T23:42:43.570 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.570 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.570 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.570 DEBUG:boto:Params: {} 2024-04-19T23:42:43.570 DEBUG:boto:Token: None 2024-04-19T23:42:43.570 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.571 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk= 2024-04-19T23:42:43.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk=', 'Content-Length': '0'} 2024-04-19T23:42:43.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000014e26a22e5c2ee30-0066230173-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.573 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.573 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.573 DEBUG:boto:path=/aktlji-26/?prefix=obj 2024-04-19T23:42:43.573 DEBUG:boto:auth_path=/aktlji-26/?prefix=obj 2024-04-19T23:42:43.573 DEBUG:boto:Method: GET 2024-04-19T23:42:43.573 DEBUG:boto:Path: /aktlji-26/?prefix=obj 2024-04-19T23:42:43.573 DEBUG:boto:Data: 2024-04-19T23:42:43.573 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.573 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.573 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.574 DEBUG:boto:Params: {} 2024-04-19T23:42:43.574 DEBUG:boto:Token: None 2024-04-19T23:42:43.574 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/ 2024-04-19T23:42:43.574 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RdNHbM3RuEVVQwUShiOuHD+5+SU= 2024-04-19T23:42:43.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RdNHbM3RuEVVQwUShiOuHD+5+SU=', 'Content-Length': '0'} 2024-04-19T23:42:43.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a689498b64541b30-0066230173-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.618 DEBUG:boto:b'aktlji-26obj1000false' 2024-04-19T23:42:43.619 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.619 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.619 DEBUG:boto:path=/aktlji-26/?prefix=obj 2024-04-19T23:42:43.619 DEBUG:boto:auth_path=/aktlji-26/?prefix=obj 2024-04-19T23:42:43.619 DEBUG:boto:Method: GET 2024-04-19T23:42:43.619 DEBUG:boto:Path: /aktlji-26/?prefix=obj 2024-04-19T23:42:43.619 DEBUG:boto:Data: 2024-04-19T23:42:43.619 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.619 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.620 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.620 DEBUG:boto:Params: {} 2024-04-19T23:42:43.620 DEBUG:boto:Token: None 2024-04-19T23:42:43.620 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/ 2024-04-19T23:42:43.620 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RdNHbM3RuEVVQwUShiOuHD+5+SU= 2024-04-19T23:42:43.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RdNHbM3RuEVVQwUShiOuHD+5+SU=', 'Content-Length': '0'} 2024-04-19T23:42:43.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001462cece7463c9c0-0066230173-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.623 DEBUG:boto:b'aktlji-26obj1000false' 2024-04-19T23:42:43.623 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.623 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.624 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.624 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:43.624 DEBUG:boto:Data: 2024-04-19T23:42:43.624 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.624 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.624 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.624 DEBUG:boto:Params: {} 2024-04-19T23:42:43.624 DEBUG:boto:Token: None 2024-04-19T23:42:43.624 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj 2024-04-19T23:42:43.624 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw= 2024-04-19T23:42:43.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw=', 'Content-Length': '0'} 2024-04-19T23:42:43.627 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000f766dbd6a63e610e-0066230173-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.627 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.627 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.627 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.627 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:43.627 DEBUG:boto:Data: 2024-04-19T23:42:43.627 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.627 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.628 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.628 DEBUG:boto:Params: {} 2024-04-19T23:42:43.628 DEBUG:boto:Token: None 2024-04-19T23:42:43.628 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj 2024-04-19T23:42:43.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw= 2024-04-19T23:42:43.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw=', 'Content-Length': '0'} 2024-04-19T23:42:43.630 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000003b7849b73e7ef5d7-0066230173-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.631 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-26 zones={test-zone2, test-zone1} 2024-04-19T23:42:43.631 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-26 zones={test-zone3, test-zone1} 2024-04-19T23:42:43.631 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.631 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.631 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.631 DEBUG:boto:Path: /aktlji-26/ 2024-04-19T23:42:43.631 DEBUG:boto:Data: 2024-04-19T23:42:43.632 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.632 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.632 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.632 DEBUG:boto:Params: {} 2024-04-19T23:42:43.632 DEBUG:boto:Token: None 2024-04-19T23:42:43.632 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/ 2024-04-19T23:42:43.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Jq+2plmAskKnfDb0yyaE164Oo+o= 2024-04-19T23:42:43.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Jq+2plmAskKnfDb0yyaE164Oo+o=', 'Content-Length': '0'} 2024-04-19T23:42:43.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7b23a8a4d7d9bbe-0066230173-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.635 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.635 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.635 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.635 DEBUG:boto:Path: /aktlji-26/ 2024-04-19T23:42:43.635 DEBUG:boto:Data: 2024-04-19T23:42:43.635 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.635 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.635 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.636 DEBUG:boto:Params: {} 2024-04-19T23:42:43.636 DEBUG:boto:Token: None 2024-04-19T23:42:43.636 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/ 2024-04-19T23:42:43.636 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Jq+2plmAskKnfDb0yyaE164Oo+o= 2024-04-19T23:42:43.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Jq+2plmAskKnfDb0yyaE164Oo+o=', 'Content-Length': '0'} 2024-04-19T23:42:43.638 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee8d26ccafaba370-0066230173-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.638 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:42:43.639 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.639 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.639 DEBUG:boto:path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.639 DEBUG:boto:auth_path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.639 DEBUG:boto:Method: GET 2024-04-19T23:42:43.639 DEBUG:boto:Path: /aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.639 DEBUG:boto:Data: 2024-04-19T23:42:43.639 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.639 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.639 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.639 DEBUG:boto:Params: {} 2024-04-19T23:42:43.640 DEBUG:boto:Token: None 2024-04-19T23:42:43.640 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/?versions 2024-04-19T23:42:43.640 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg= 2024-04-19T23:42:43.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg=', 'Content-Length': '0'} 2024-04-19T23:42:43.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c11efa10acf6873-0066230173-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.642 DEBUG:boto:b'aktlji-26999falseobjiJzYuqY05U-Eu1l-5iINYoSVqxQDj2ztrue2024-04-19T23:42:10.406Zrgw-multisite-test-userMultisite Test UserNormalobjaFg1G93maO0BVFcQAVh5xHYwflNTsyLfalse2024-04-19T23:42:10.386Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:43.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:43.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:43.643 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:42:43.643 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.643 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.643 DEBUG:boto:path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.644 DEBUG:boto:auth_path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.644 DEBUG:boto:Method: GET 2024-04-19T23:42:43.644 DEBUG:boto:Path: /aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.644 DEBUG:boto:Data: 2024-04-19T23:42:43.644 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.644 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.644 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.644 DEBUG:boto:Params: {} 2024-04-19T23:42:43.644 DEBUG:boto:Token: None 2024-04-19T23:42:43.644 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/?versions 2024-04-19T23:42:43.644 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg= 2024-04-19T23:42:43.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg=', 'Content-Length': '0'} 2024-04-19T23:42:43.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011424cdeea80a8d0-0066230173-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.690 DEBUG:boto:b'aktlji-26999falseobjiJzYuqY05U-Eu1l-5iINYoSVqxQDj2ztrue2024-04-19T23:42:10.406Zrgw-multisite-test-userMultisite Test UserNormalobjaFg1G93maO0BVFcQAVh5xHYwflNTsyLfalse2024-04-19T23:42:10.386Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:43.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:43.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:43.691 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.691 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.691 DEBUG:boto:path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.691 DEBUG:boto:auth_path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.691 DEBUG:boto:Method: GET 2024-04-19T23:42:43.692 DEBUG:boto:Path: /aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.692 DEBUG:boto:Data: 2024-04-19T23:42:43.692 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.692 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.692 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.692 DEBUG:boto:Params: {} 2024-04-19T23:42:43.692 DEBUG:boto:Token: None 2024-04-19T23:42:43.692 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/?versions 2024-04-19T23:42:43.692 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg= 2024-04-19T23:42:43.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg=', 'Content-Length': '0'} 2024-04-19T23:42:43.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2029f0d4c28a62f-0066230173-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.695 DEBUG:boto:b'aktlji-26999falseobjiJzYuqY05U-Eu1l-5iINYoSVqxQDj2ztrue2024-04-19T23:42:10.406Zrgw-multisite-test-userMultisite Test UserNormalobjaFg1G93maO0BVFcQAVh5xHYwflNTsyLfalse2024-04-19T23:42:10.386Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:43.696 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.696 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.696 DEBUG:boto:path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.696 DEBUG:boto:auth_path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.696 DEBUG:boto:Method: GET 2024-04-19T23:42:43.696 DEBUG:boto:Path: /aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.696 DEBUG:boto:Data: 2024-04-19T23:42:43.696 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.696 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.696 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.697 DEBUG:boto:Params: {} 2024-04-19T23:42:43.697 DEBUG:boto:Token: None 2024-04-19T23:42:43.697 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/?versions 2024-04-19T23:42:43.697 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg= 2024-04-19T23:42:43.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg=', 'Content-Length': '0'} 2024-04-19T23:42:43.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adcb1cbfac6f0d4f-0066230173-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.700 DEBUG:boto:b'aktlji-26999falseobjiJzYuqY05U-Eu1l-5iINYoSVqxQDj2ztrue2024-04-19T23:42:10.406Zrgw-multisite-test-userMultisite Test UserNormalobjaFg1G93maO0BVFcQAVh5xHYwflNTsyLfalse2024-04-19T23:42:10.386Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:43.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:43.700 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.700 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.700 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.700 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:43.700 DEBUG:boto:Data: 2024-04-19T23:42:43.701 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.701 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.701 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.701 DEBUG:boto:Params: {} 2024-04-19T23:42:43.701 DEBUG:boto:Token: None 2024-04-19T23:42:43.701 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj 2024-04-19T23:42:43.701 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw= 2024-04-19T23:42:43.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw=', 'Content-Length': '0'} 2024-04-19T23:42:43.703 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000004384ee088a3c73cf-0066230173-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.703 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.704 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.704 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.704 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:43.704 DEBUG:boto:Data: 2024-04-19T23:42:43.704 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.704 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.704 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.704 DEBUG:boto:Params: {} 2024-04-19T23:42:43.704 DEBUG:boto:Token: None 2024-04-19T23:42:43.705 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj 2024-04-19T23:42:43.705 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw= 2024-04-19T23:42:43.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw=', 'Content-Length': '0'} 2024-04-19T23:42:43.707 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000006ae957e44f6f8ec-0066230173-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:43.707 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.707 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.707 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.708 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.708 DEBUG:boto:Method: GET 2024-04-19T23:42:43.708 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.708 DEBUG:boto:Data: 2024-04-19T23:42:43.708 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.708 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.708 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.708 DEBUG:boto:Params: {} 2024-04-19T23:42:43.708 DEBUG:boto:Token: None 2024-04-19T23:42:43.708 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.709 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk= 2024-04-19T23:42:43.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk=', 'Content-Length': '0'} 2024-04-19T23:42:43.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077e7bf103fc46f81-0066230173-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.711 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.711 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.711 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.711 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.711 DEBUG:boto:Method: GET 2024-04-19T23:42:43.711 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.711 DEBUG:boto:Data: 2024-04-19T23:42:43.711 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.712 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.712 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.712 DEBUG:boto:Params: {} 2024-04-19T23:42:43.712 DEBUG:boto:Token: None 2024-04-19T23:42:43.712 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.712 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk= 2024-04-19T23:42:43.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk=', 'Content-Length': '0'} 2024-04-19T23:42:43.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000040ffc21baa4377b7-0066230173-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.714 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.714 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.714 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.714 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.714 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.715 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.715 DEBUG:boto:Data: 2024-04-19T23:42:43.715 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.715 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.715 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.715 DEBUG:boto:Params: {} 2024-04-19T23:42:43.715 DEBUG:boto:Token: None 2024-04-19T23:42:43.715 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.715 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gWqBWEOzjqI6KXHddINDh+QIYf0= 2024-04-19T23:42:43.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gWqBWEOzjqI6KXHddINDh+QIYf0=', 'Content-Length': '0'} 2024-04-19T23:42:43.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7a3d065c901ed93-0066230173-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.717 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.718 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.718 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.718 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.718 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.718 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.718 DEBUG:boto:Data: 2024-04-19T23:42:43.718 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.718 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.718 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.718 DEBUG:boto:Params: {} 2024-04-19T23:42:43.719 DEBUG:boto:Token: None 2024-04-19T23:42:43.719 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.719 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gWqBWEOzjqI6KXHddINDh+QIYf0= 2024-04-19T23:42:43.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gWqBWEOzjqI6KXHddINDh+QIYf0=', 'Content-Length': '0'} 2024-04-19T23:42:43.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004dc007e83d8f5768-0066230173-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:43.721 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.721 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.721 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.721 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.721 DEBUG:boto:Method: GET 2024-04-19T23:42:43.721 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.721 DEBUG:boto:Data: 2024-04-19T23:42:43.722 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.722 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.722 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.722 DEBUG:boto:Params: {} 2024-04-19T23:42:43.722 DEBUG:boto:Token: None 2024-04-19T23:42:43.722 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.722 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk= 2024-04-19T23:42:43.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk=', 'Content-Length': '0'} 2024-04-19T23:42:43.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000725e1b2f3721fb9-0066230173-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.724 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.725 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.725 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.725 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.725 DEBUG:boto:Method: GET 2024-04-19T23:42:43.725 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.725 DEBUG:boto:Data: 2024-04-19T23:42:43.725 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.725 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.725 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.725 DEBUG:boto:Params: {} 2024-04-19T23:42:43.725 DEBUG:boto:Token: None 2024-04-19T23:42:43.725 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.726 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk= 2024-04-19T23:42:43.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk=', 'Content-Length': '0'} 2024-04-19T23:42:43.727 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b4920664eb7e4e9a-0066230173-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.728 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.728 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.728 DEBUG:boto:path=/aktlji-26/?prefix=obj 2024-04-19T23:42:43.728 DEBUG:boto:auth_path=/aktlji-26/?prefix=obj 2024-04-19T23:42:43.728 DEBUG:boto:Method: GET 2024-04-19T23:42:43.728 DEBUG:boto:Path: /aktlji-26/?prefix=obj 2024-04-19T23:42:43.728 DEBUG:boto:Data: 2024-04-19T23:42:43.728 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.729 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.729 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.729 DEBUG:boto:Params: {} 2024-04-19T23:42:43.729 DEBUG:boto:Token: None 2024-04-19T23:42:43.729 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/ 2024-04-19T23:42:43.729 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RdNHbM3RuEVVQwUShiOuHD+5+SU= 2024-04-19T23:42:43.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RdNHbM3RuEVVQwUShiOuHD+5+SU=', 'Content-Length': '0'} 2024-04-19T23:42:43.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc493f10e875c2d4-0066230173-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.778 DEBUG:boto:b'aktlji-26obj1000false' 2024-04-19T23:42:43.779 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.779 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.779 DEBUG:boto:path=/aktlji-26/?prefix=obj 2024-04-19T23:42:43.779 DEBUG:boto:auth_path=/aktlji-26/?prefix=obj 2024-04-19T23:42:43.780 DEBUG:boto:Method: GET 2024-04-19T23:42:43.780 DEBUG:boto:Path: /aktlji-26/?prefix=obj 2024-04-19T23:42:43.780 DEBUG:boto:Data: 2024-04-19T23:42:43.780 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.780 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.780 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.780 DEBUG:boto:Params: {} 2024-04-19T23:42:43.780 DEBUG:boto:Token: None 2024-04-19T23:42:43.780 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/ 2024-04-19T23:42:43.781 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RdNHbM3RuEVVQwUShiOuHD+5+SU= 2024-04-19T23:42:43.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RdNHbM3RuEVVQwUShiOuHD+5+SU=', 'Content-Length': '0'} 2024-04-19T23:42:43.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af9e685653312dd5-0066230173-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.784 DEBUG:boto:b'aktlji-26obj1000false' 2024-04-19T23:42:43.784 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.784 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.784 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.785 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:43.785 DEBUG:boto:Data: 2024-04-19T23:42:43.785 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.785 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.785 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.785 DEBUG:boto:Params: {} 2024-04-19T23:42:43.785 DEBUG:boto:Token: None 2024-04-19T23:42:43.785 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj 2024-04-19T23:42:43.785 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw= 2024-04-19T23:42:43.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw=', 'Content-Length': '0'} 2024-04-19T23:42:43.788 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000007c57ab164de5bfca-0066230173-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.788 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.788 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.788 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.789 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:43.789 DEBUG:boto:Data: 2024-04-19T23:42:43.789 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.789 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.789 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.789 DEBUG:boto:Params: {} 2024-04-19T23:42:43.789 DEBUG:boto:Token: None 2024-04-19T23:42:43.789 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj 2024-04-19T23:42:43.789 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw= 2024-04-19T23:42:43.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw=', 'Content-Length': '0'} 2024-04-19T23:42:43.792 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000dcdab251c07af4bb-0066230173-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.792 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-26 zones={test-zone3, test-zone1} 2024-04-19T23:42:43.792 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-26 zones={test-zone3, test-zone2} 2024-04-19T23:42:43.792 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.792 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.792 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.792 DEBUG:boto:Path: /aktlji-26/ 2024-04-19T23:42:43.792 DEBUG:boto:Data: 2024-04-19T23:42:43.793 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.793 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.793 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.793 DEBUG:boto:Params: {} 2024-04-19T23:42:43.793 DEBUG:boto:Token: None 2024-04-19T23:42:43.793 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/ 2024-04-19T23:42:43.793 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Jq+2plmAskKnfDb0yyaE164Oo+o= 2024-04-19T23:42:43.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Jq+2plmAskKnfDb0yyaE164Oo+o=', 'Content-Length': '0'} 2024-04-19T23:42:43.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cb4ba8827b2d896-0066230173-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.796 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.796 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.796 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.796 DEBUG:boto:Path: /aktlji-26/ 2024-04-19T23:42:43.796 DEBUG:boto:Data: 2024-04-19T23:42:43.796 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.796 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.796 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.797 DEBUG:boto:Params: {} 2024-04-19T23:42:43.797 DEBUG:boto:Token: None 2024-04-19T23:42:43.797 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/ 2024-04-19T23:42:43.797 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Jq+2plmAskKnfDb0yyaE164Oo+o= 2024-04-19T23:42:43.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Jq+2plmAskKnfDb0yyaE164Oo+o=', 'Content-Length': '0'} 2024-04-19T23:42:43.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d8049a5acc6a27d-0066230173-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.801 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:42:43.801 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.801 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.801 DEBUG:boto:path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.801 DEBUG:boto:auth_path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.801 DEBUG:boto:Method: GET 2024-04-19T23:42:43.801 DEBUG:boto:Path: /aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.801 DEBUG:boto:Data: 2024-04-19T23:42:43.801 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.801 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.802 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.802 DEBUG:boto:Params: {} 2024-04-19T23:42:43.802 DEBUG:boto:Token: None 2024-04-19T23:42:43.802 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/?versions 2024-04-19T23:42:43.802 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg= 2024-04-19T23:42:43.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg=', 'Content-Length': '0'} 2024-04-19T23:42:43.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5721f28f8f257de-0066230173-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.805 DEBUG:boto:b'aktlji-26999falseobjiJzYuqY05U-Eu1l-5iINYoSVqxQDj2ztrue2024-04-19T23:42:10.406Zrgw-multisite-test-userMultisite Test UserNormalobjaFg1G93maO0BVFcQAVh5xHYwflNTsyLfalse2024-04-19T23:42:10.386Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:43.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:43.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:43.805 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:42:43.806 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.806 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.806 DEBUG:boto:path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.806 DEBUG:boto:auth_path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.806 DEBUG:boto:Method: GET 2024-04-19T23:42:43.806 DEBUG:boto:Path: /aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.806 DEBUG:boto:Data: 2024-04-19T23:42:43.806 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.806 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.806 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.806 DEBUG:boto:Params: {} 2024-04-19T23:42:43.807 DEBUG:boto:Token: None 2024-04-19T23:42:43.807 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/?versions 2024-04-19T23:42:43.807 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg= 2024-04-19T23:42:43.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg=', 'Content-Length': '0'} 2024-04-19T23:42:43.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc50348b7a15612c-0066230173-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.810 DEBUG:boto:b'aktlji-26999falseobjiJzYuqY05U-Eu1l-5iINYoSVqxQDj2ztrue2024-04-19T23:42:10.406Zrgw-multisite-test-userMultisite Test UserNormalobjaFg1G93maO0BVFcQAVh5xHYwflNTsyLfalse2024-04-19T23:42:10.386Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:43.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:43.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:42:43.811 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.811 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.812 DEBUG:boto:path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.812 DEBUG:boto:auth_path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.812 DEBUG:boto:Method: GET 2024-04-19T23:42:43.812 DEBUG:boto:Path: /aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.812 DEBUG:boto:Data: 2024-04-19T23:42:43.812 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.812 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.812 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.812 DEBUG:boto:Params: {} 2024-04-19T23:42:43.813 DEBUG:boto:Token: None 2024-04-19T23:42:43.813 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/?versions 2024-04-19T23:42:43.813 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg= 2024-04-19T23:42:43.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg=', 'Content-Length': '0'} 2024-04-19T23:42:43.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ece070f152bbcb3-0066230173-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.815 DEBUG:boto:b'aktlji-26999falseobjiJzYuqY05U-Eu1l-5iINYoSVqxQDj2ztrue2024-04-19T23:42:10.406Zrgw-multisite-test-userMultisite Test UserNormalobjaFg1G93maO0BVFcQAVh5xHYwflNTsyLfalse2024-04-19T23:42:10.386Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:43.816 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.816 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.816 DEBUG:boto:path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.816 DEBUG:boto:auth_path=/aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.816 DEBUG:boto:Method: GET 2024-04-19T23:42:43.816 DEBUG:boto:Path: /aktlji-26/?versions&max-keys=999 2024-04-19T23:42:43.816 DEBUG:boto:Data: 2024-04-19T23:42:43.817 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.817 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.817 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.817 DEBUG:boto:Params: {} 2024-04-19T23:42:43.817 DEBUG:boto:Token: None 2024-04-19T23:42:43.817 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/?versions 2024-04-19T23:42:43.817 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg= 2024-04-19T23:42:43.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ea6VgilT3XqCASIgx4XMmsJLXwg=', 'Content-Length': '0'} 2024-04-19T23:42:43.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038066a8afa60a9f1-0066230173-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.820 DEBUG:boto:b'aktlji-26999falseobjiJzYuqY05U-Eu1l-5iINYoSVqxQDj2ztrue2024-04-19T23:42:10.406Zrgw-multisite-test-userMultisite Test UserNormalobjaFg1G93maO0BVFcQAVh5xHYwflNTsyLfalse2024-04-19T23:42:10.386Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:42:43.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:43.820 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.821 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.821 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.821 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:43.821 DEBUG:boto:Data: 2024-04-19T23:42:43.821 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.821 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.821 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.821 DEBUG:boto:Params: {} 2024-04-19T23:42:43.821 DEBUG:boto:Token: None 2024-04-19T23:42:43.822 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj 2024-04-19T23:42:43.822 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw= 2024-04-19T23:42:43.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw=', 'Content-Length': '0'} 2024-04-19T23:42:43.824 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000005e537b79fc52869d-0066230173-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.824 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.824 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.824 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.825 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:43.825 DEBUG:boto:Data: 2024-04-19T23:42:43.825 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.825 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.825 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.825 DEBUG:boto:Params: {} 2024-04-19T23:42:43.825 DEBUG:boto:Token: None 2024-04-19T23:42:43.825 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj 2024-04-19T23:42:43.826 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw= 2024-04-19T23:42:43.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw=', 'Content-Length': '0'} 2024-04-19T23:42:43.828 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000961bfe5e37444855-0066230173-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:43.828 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.828 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.828 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.829 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.829 DEBUG:boto:Method: GET 2024-04-19T23:42:43.829 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.829 DEBUG:boto:Data: 2024-04-19T23:42:43.829 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.829 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.829 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.829 DEBUG:boto:Params: {} 2024-04-19T23:42:43.829 DEBUG:boto:Token: None 2024-04-19T23:42:43.830 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.830 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk= 2024-04-19T23:42:43.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk=', 'Content-Length': '0'} 2024-04-19T23:42:43.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008df4b5124166ea5a-0066230173-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.832 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.832 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.832 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.832 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.832 DEBUG:boto:Method: GET 2024-04-19T23:42:43.832 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.833 DEBUG:boto:Data: 2024-04-19T23:42:43.833 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.833 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.833 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.833 DEBUG:boto:Params: {} 2024-04-19T23:42:43.833 DEBUG:boto:Token: None 2024-04-19T23:42:43.833 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.833 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk= 2024-04-19T23:42:43.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk=', 'Content-Length': '0'} 2024-04-19T23:42:43.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000037f403f70a914716-0066230173-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.836 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.836 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.836 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.836 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.836 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.836 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.836 DEBUG:boto:Data: 2024-04-19T23:42:43.836 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.836 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.837 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.837 DEBUG:boto:Params: {} 2024-04-19T23:42:43.837 DEBUG:boto:Token: None 2024-04-19T23:42:43.837 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.837 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gWqBWEOzjqI6KXHddINDh+QIYf0= 2024-04-19T23:42:43.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gWqBWEOzjqI6KXHddINDh+QIYf0=', 'Content-Length': '0'} 2024-04-19T23:42:43.839 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004528bbd4d5e55675-0066230173-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.839 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.839 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.840 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.840 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.840 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.840 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.840 DEBUG:boto:Data: 2024-04-19T23:42:43.840 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.840 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.840 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.840 DEBUG:boto:Params: {} 2024-04-19T23:42:43.841 DEBUG:boto:Token: None 2024-04-19T23:42:43.841 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.841 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gWqBWEOzjqI6KXHddINDh+QIYf0= 2024-04-19T23:42:43.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gWqBWEOzjqI6KXHddINDh+QIYf0=', 'Content-Length': '0'} 2024-04-19T23:42:43.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003318bd7bee1df18d-0066230173-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:42:43.843 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.843 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.843 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.843 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.844 DEBUG:boto:Method: GET 2024-04-19T23:42:43.844 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.844 DEBUG:boto:Data: 2024-04-19T23:42:43.844 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.844 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.844 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.844 DEBUG:boto:Params: {} 2024-04-19T23:42:43.844 DEBUG:boto:Token: None 2024-04-19T23:42:43.844 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.845 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk= 2024-04-19T23:42:43.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk=', 'Content-Length': '0'} 2024-04-19T23:42:43.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006c937e125c27a076-0066230173-1106-test-zone3'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.846 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.847 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.847 DEBUG:boto:path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.847 DEBUG:boto:auth_path=/aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.847 DEBUG:boto:Method: GET 2024-04-19T23:42:43.847 DEBUG:boto:Path: /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.847 DEBUG:boto:Data: 2024-04-19T23:42:43.847 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.847 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.847 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.847 DEBUG:boto:Params: {} 2024-04-19T23:42:43.848 DEBUG:boto:Token: None 2024-04-19T23:42:43.848 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj?versionId=aFg1G93maO0BVFcQAVh5xHYwflNTsyL 2024-04-19T23:42:43.848 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk= 2024-04-19T23:42:43.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FkuRo0UnZazN2qZAgZzrHlT3yYk=', 'Content-Length': '0'} 2024-04-19T23:42:43.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:42:10 GMT'), ('x-amz-version-id', 'aFg1G93maO0BVFcQAVh5xHYwflNTsyL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e124a09e52cb00be-0066230173-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.850 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.850 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.850 DEBUG:boto:path=/aktlji-26/?prefix=obj 2024-04-19T23:42:43.851 DEBUG:boto:auth_path=/aktlji-26/?prefix=obj 2024-04-19T23:42:43.851 DEBUG:boto:Method: GET 2024-04-19T23:42:43.851 DEBUG:boto:Path: /aktlji-26/?prefix=obj 2024-04-19T23:42:43.851 DEBUG:boto:Data: 2024-04-19T23:42:43.851 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.851 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.851 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.851 DEBUG:boto:Params: {} 2024-04-19T23:42:43.851 DEBUG:boto:Token: None 2024-04-19T23:42:43.852 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/ 2024-04-19T23:42:43.852 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RdNHbM3RuEVVQwUShiOuHD+5+SU= 2024-04-19T23:42:43.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RdNHbM3RuEVVQwUShiOuHD+5+SU=', 'Content-Length': '0'} 2024-04-19T23:42:43.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dda2488b9c0c104c-0066230173-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.894 DEBUG:boto:b'aktlji-26obj1000false' 2024-04-19T23:42:43.895 DEBUG:boto:path=/aktlji-26/ 2024-04-19T23:42:43.895 DEBUG:boto:auth_path=/aktlji-26/ 2024-04-19T23:42:43.895 DEBUG:boto:path=/aktlji-26/?prefix=obj 2024-04-19T23:42:43.895 DEBUG:boto:auth_path=/aktlji-26/?prefix=obj 2024-04-19T23:42:43.895 DEBUG:boto:Method: GET 2024-04-19T23:42:43.895 DEBUG:boto:Path: /aktlji-26/?prefix=obj 2024-04-19T23:42:43.895 DEBUG:boto:Data: 2024-04-19T23:42:43.895 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.896 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.896 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.896 DEBUG:boto:Params: {} 2024-04-19T23:42:43.896 DEBUG:boto:Token: None 2024-04-19T23:42:43.896 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/ 2024-04-19T23:42:43.896 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RdNHbM3RuEVVQwUShiOuHD+5+SU= 2024-04-19T23:42:43.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RdNHbM3RuEVVQwUShiOuHD+5+SU=', 'Content-Length': '0'} 2024-04-19T23:42:43.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cfad2e3fa6c63c0-0066230173-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.899 DEBUG:boto:b'aktlji-26obj1000false' 2024-04-19T23:42:43.899 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.899 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.899 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.900 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:43.900 DEBUG:boto:Data: 2024-04-19T23:42:43.900 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.900 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.900 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.900 DEBUG:boto:Params: {} 2024-04-19T23:42:43.900 DEBUG:boto:Token: None 2024-04-19T23:42:43.900 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj 2024-04-19T23:42:43.900 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw= 2024-04-19T23:42:43.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw=', 'Content-Length': '0'} 2024-04-19T23:42:43.903 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000007e64e4d22a43a63d-0066230173-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.903 DEBUG:boto:path=/aktlji-26/obj 2024-04-19T23:42:43.903 DEBUG:boto:auth_path=/aktlji-26/obj 2024-04-19T23:42:43.903 DEBUG:boto:Method: HEAD 2024-04-19T23:42:43.903 DEBUG:boto:Path: /aktlji-26/obj 2024-04-19T23:42:43.904 DEBUG:boto:Data: 2024-04-19T23:42:43.904 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.904 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.904 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.904 DEBUG:boto:Params: {} 2024-04-19T23:42:43.904 DEBUG:boto:Token: None 2024-04-19T23:42:43.904 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:43 GMT /aktlji-26/obj 2024-04-19T23:42:43.905 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw= 2024-04-19T23:42:43.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q12NHg+j8jINyDDMCL9cC2bwyEw=', 'Content-Length': '0'} 2024-04-19T23:42:43.907 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c43fea02ef3c87a8-0066230173-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.907 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-26 zones={test-zone3, test-zone2} 2024-04-19T23:42:43.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2024-04-19T23:42:43.909 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-27 2024-04-19T23:42:43.909 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:42:43.909 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:42:43.909 DEBUG:boto:Method: PUT 2024-04-19T23:42:43.909 DEBUG:boto:Path: /aktlji-27/ 2024-04-19T23:42:43.909 DEBUG:boto:Data: 2024-04-19T23:42:43.909 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.910 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.910 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.910 DEBUG:boto:Params: {} 2024-04-19T23:42:43.910 DEBUG:boto:Token: None 2024-04-19T23:42:43.910 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:42:43 GMT /aktlji-27/ 2024-04-19T23:42:43.910 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MTWyPu3+k1+DmNNLPVipTZ9uPLE= 2024-04-19T23:42:43.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MTWyPu3+k1+DmNNLPVipTZ9uPLE=', 'Content-Length': '0'} 2024-04-19T23:42:43.927 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009b0c19528276f3ed-0066230173-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.928 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-28 2024-04-19T23:42:43.928 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:42:43.928 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:42:43.928 DEBUG:boto:Method: PUT 2024-04-19T23:42:43.928 DEBUG:boto:Path: /aktlji-28/ 2024-04-19T23:42:43.928 DEBUG:boto:Data: 2024-04-19T23:42:43.928 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.928 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:43.928 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.928 DEBUG:boto:Params: {} 2024-04-19T23:42:43.929 DEBUG:boto:Token: None 2024-04-19T23:42:43.929 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:42:43 GMT /aktlji-28/ 2024-04-19T23:42:43.929 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:spAhEw3NDSQo1jBleM5wMaSdQxI= 2024-04-19T23:42:43.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:spAhEw3NDSQo1jBleM5wMaSdQxI=', 'Content-Length': '0'} 2024-04-19T23:42:43.958 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000051d21814a90ecdf2-0066230173-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.958 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-29 2024-04-19T23:42:43.958 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:42:43.958 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:42:43.958 DEBUG:boto:Method: PUT 2024-04-19T23:42:43.959 DEBUG:boto:Path: /aktlji-29/ 2024-04-19T23:42:43.959 DEBUG:boto:Data: 2024-04-19T23:42:43.959 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.959 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:43.959 DEBUG:boto:Port: 8001 2024-04-19T23:42:43.959 DEBUG:boto:Params: {} 2024-04-19T23:42:43.959 DEBUG:boto:Token: None 2024-04-19T23:42:43.959 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:42:43 GMT /aktlji-29/ 2024-04-19T23:42:43.959 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KZaLEzs/PdzRG2+eNspStAYzPdU= 2024-04-19T23:42:43.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KZaLEzs/PdzRG2+eNspStAYzPdU=', 'Content-Length': '0'} 2024-04-19T23:42:43.987 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000473e64830579668a-0066230173-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.987 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:42:43.988 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:42:43.988 DEBUG:boto:path=/aktlji-27/?versioning 2024-04-19T23:42:43.988 DEBUG:boto:auth_path=/aktlji-27/?versioning 2024-04-19T23:42:43.988 DEBUG:boto:Method: PUT 2024-04-19T23:42:43.988 DEBUG:boto:Path: /aktlji-27/?versioning 2024-04-19T23:42:43.988 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:42:43.988 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.989 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.989 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.989 DEBUG:boto:Params: {} 2024-04-19T23:42:43.989 DEBUG:boto:Token: None 2024-04-19T23:42:43.989 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:42:43 GMT /aktlji-27/?versioning 2024-04-19T23:42:43.989 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CN1rFNCnIAlKtQVC990v0oQ78SQ= 2024-04-19T23:42:43.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CN1rFNCnIAlKtQVC990v0oQ78SQ=', 'Content-Length': '228'} 2024-04-19T23:42:43.995 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b470442b9b58bd58-0066230173-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:43.995 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:42:43.996 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:42:43.996 DEBUG:boto:path=/aktlji-27/?versioning 2024-04-19T23:42:43.996 DEBUG:boto:auth_path=/aktlji-27/?versioning 2024-04-19T23:42:43.996 DEBUG:boto:Method: PUT 2024-04-19T23:42:43.996 DEBUG:boto:Path: /aktlji-27/?versioning 2024-04-19T23:42:43.996 DEBUG:boto:Data: Suspended Disabled 2024-04-19T23:42:43.996 DEBUG:boto:Headers: {} 2024-04-19T23:42:43.996 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:43.996 DEBUG:boto:Port: 8000 2024-04-19T23:42:43.996 DEBUG:boto:Params: {} 2024-04-19T23:42:43.996 DEBUG:boto:Token: None 2024-04-19T23:42:43.997 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:42:43 GMT /aktlji-27/?versioning 2024-04-19T23:42:43.997 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CN1rFNCnIAlKtQVC990v0oQ78SQ= 2024-04-19T23:42:43.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CN1rFNCnIAlKtQVC990v0oQ78SQ=', 'Content-Length': '230'} 2024-04-19T23:42:44.002 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c2517508da7285c3-0066230174-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:44.003 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:42:44.003 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:42:44.003 DEBUG:boto:path=/aktlji-28/?versioning 2024-04-19T23:42:44.003 DEBUG:boto:auth_path=/aktlji-28/?versioning 2024-04-19T23:42:44.003 DEBUG:boto:Method: PUT 2024-04-19T23:42:44.003 DEBUG:boto:Path: /aktlji-28/?versioning 2024-04-19T23:42:44.003 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:42:44.003 DEBUG:boto:Headers: {} 2024-04-19T23:42:44.003 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:44.003 DEBUG:boto:Port: 8000 2024-04-19T23:42:44.003 DEBUG:boto:Params: {} 2024-04-19T23:42:44.004 DEBUG:boto:Token: None 2024-04-19T23:42:44.004 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:42:44 GMT /aktlji-28/?versioning 2024-04-19T23:42:44.004 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0sABuR/amnJ7HMdT3IxoVUSYgOY= 2024-04-19T23:42:44.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:44 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0sABuR/amnJ7HMdT3IxoVUSYgOY=', 'Content-Length': '228'} 2024-04-19T23:42:44.013 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000011398c786d8497a6-0066230174-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:44.013 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:42:44.014 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:42:44.014 DEBUG:boto:path=/aktlji-28/?versioning 2024-04-19T23:42:44.014 DEBUG:boto:auth_path=/aktlji-28/?versioning 2024-04-19T23:42:44.014 DEBUG:boto:Method: PUT 2024-04-19T23:42:44.014 DEBUG:boto:Path: /aktlji-28/?versioning 2024-04-19T23:42:44.014 DEBUG:boto:Data: Suspended Disabled 2024-04-19T23:42:44.014 DEBUG:boto:Headers: {} 2024-04-19T23:42:44.014 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:44.014 DEBUG:boto:Port: 8000 2024-04-19T23:42:44.014 DEBUG:boto:Params: {} 2024-04-19T23:42:44.015 DEBUG:boto:Token: None 2024-04-19T23:42:44.015 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:42:44 GMT /aktlji-28/?versioning 2024-04-19T23:42:44.015 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0sABuR/amnJ7HMdT3IxoVUSYgOY= 2024-04-19T23:42:44.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:44 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0sABuR/amnJ7HMdT3IxoVUSYgOY=', 'Content-Length': '230'} 2024-04-19T23:42:44.035 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b3438ba2e1fd8b2f-0066230174-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:44.035 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:42:44.036 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:42:44.036 DEBUG:boto:path=/aktlji-29/?versioning 2024-04-19T23:42:44.036 DEBUG:boto:auth_path=/aktlji-29/?versioning 2024-04-19T23:42:44.036 DEBUG:boto:Method: PUT 2024-04-19T23:42:44.036 DEBUG:boto:Path: /aktlji-29/?versioning 2024-04-19T23:42:44.036 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:42:44.036 DEBUG:boto:Headers: {} 2024-04-19T23:42:44.037 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:44.037 DEBUG:boto:Port: 8001 2024-04-19T23:42:44.037 DEBUG:boto:Params: {} 2024-04-19T23:42:44.037 DEBUG:boto:Token: None 2024-04-19T23:42:44.037 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:42:44 GMT /aktlji-29/?versioning 2024-04-19T23:42:44.038 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0KHibBuCJTtHDeDSb8XEowKrZuo= 2024-04-19T23:42:44.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:44 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0KHibBuCJTtHDeDSb8XEowKrZuo=', 'Content-Length': '228'} 2024-04-19T23:42:44.048 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000940d6735a7b2995e-0066230174-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:44.048 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:42:44.048 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:42:44.048 DEBUG:boto:path=/aktlji-29/?versioning 2024-04-19T23:42:44.048 DEBUG:boto:auth_path=/aktlji-29/?versioning 2024-04-19T23:42:44.048 DEBUG:boto:Method: PUT 2024-04-19T23:42:44.049 DEBUG:boto:Path: /aktlji-29/?versioning 2024-04-19T23:42:44.049 DEBUG:boto:Data: Suspended Disabled 2024-04-19T23:42:44.049 DEBUG:boto:Headers: {} 2024-04-19T23:42:44.049 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:44.049 DEBUG:boto:Port: 8001 2024-04-19T23:42:44.049 DEBUG:boto:Params: {} 2024-04-19T23:42:44.049 DEBUG:boto:Token: None 2024-04-19T23:42:44.049 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:42:44 GMT /aktlji-29/?versioning 2024-04-19T23:42:44.049 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0KHibBuCJTtHDeDSb8XEowKrZuo= 2024-04-19T23:42:44.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:44 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0KHibBuCJTtHDeDSb8XEowKrZuo=', 'Content-Length': '230'} 2024-04-19T23:42:44.058 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000e9e5de843d52db4-0066230174-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:44.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:44.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:44.059 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:42:44.117 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:44.117 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:44.153 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:42:44.167 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:42:44.167 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:42:44.167 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:42:44.167 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:42:44.168 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:44.168 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:42:44.168 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:42:44.168 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:42:44.168 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:44.168 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:42:44.168 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713570164.057390_278.1", 2024-04-19T23:42:44.168 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:42:44.057390Z" 2024-04-19T23:42:44.168 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:44.168 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:42:44.168 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:42:44.169 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:42:44.169 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:42:44.169 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:42:44.170 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570164.057390_278.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:44.170 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:42:44.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:44.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:44.171 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:42:44.237 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:44.237 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:44.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:44.293+0000 7ff5401fe780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:42:44.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:44.294+0000 7ff5401fe780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:42:44.307 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:42:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:42:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:42:44.307 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:42:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:42:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:42:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:42:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:42:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:42:44.308 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:42:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:42:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:42:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:44.309 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:42:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:42:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:42:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:42:44.310 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570164.001250_269.1", 2024-04-19T23:42:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:42:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:42:44.001250Z", 2024-04-19T23:42:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:44.311 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:42:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:42:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:42:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:42:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:42:44.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:42:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:42:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:42:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:44.313 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:42:44Z" 2024-04-19T23:42:44.313 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:42:44.314 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570164.001250_269.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:42:44.001250Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:42:44Z" } 2024-04-19T23:42:44.314 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570164.001250_269.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:42:44.001250Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:42:44.314 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570164.057390_278.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:44.315 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570164.001250_269.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:44.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570164.057390_278.1 target=1_1713570164.001250_269.1 2024-04-19T23:42:49.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:49.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:49.316 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:42:49.375 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:49.375 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:49.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:49.411+0000 7f2f90a07780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:42:49.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:49.413+0000 7f2f90a07780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:42:49.426 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:42:49.426 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:42:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:42:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:42:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:42:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:42:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:42:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:42:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:49.427 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:42:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:42:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:42:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:42:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:42:49.429 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:49.429 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:42:49.429 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:49.429 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:42:49.429 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:49.429 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:49.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:49.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:49.429 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:42:49.429 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:49.429 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:49.429 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570164.057390_278.1", 2024-04-19T23:42:49.430 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:49.430 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:42:49.430 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:49.430 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:42:44.057390Z", 2024-04-19T23:42:49.430 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:49.430 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:49.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:49.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:42:49.430 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:42:49.430 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:42:49.430 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:42:49.431 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:42:49.431 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:42:49.431 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:42:49.431 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:42:49.431 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:42:49.431 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:42:49.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:49.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:42:49.431 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:42:49.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:49.431 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:42:49.431 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:42:49.432 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:42:49.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:42:49.432 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:42:49Z" 2024-04-19T23:42:49.432 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:42:49.433 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570164.057390_278.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:42:44.057390Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:42:49Z" } 2024-04-19T23:42:49.433 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570164.057390_278.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:42:44.057390Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:42:49.433 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570164.057390_278.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:49.433 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570164.057390_278.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:49.433 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-19T23:42:49.433 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-19T23:42:49.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:49.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:42:49.434 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:42:49.493 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:49.493 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:49.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:49.530+0000 7fe291b49780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:42:49.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:42:49.531+0000 7fe291b49780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:42:49.544 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-19T23:42:49.544 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-19T23:42:49.544 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-19T23:42:49.544 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-19T23:42:49.544 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-19T23:42:49.544 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:42:49.544 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:42:49.544 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:49.544 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-19T23:42:49.545 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:42:49.545 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-19T23:42:49.545 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:42:49.545 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:42:49.545 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:42:49.545 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:42:49.545 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-19T23:42:49.545 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:42:49.545 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:42:49.545 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:42:49.545 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:42:49.546 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:49.546 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:42:49.546 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-19T23:42:49.546 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:42:49.546 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:42:49.546 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:42:49.546 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:42:49.546 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:42:49.546 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:42:49.546 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-19T23:42:49.546 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:42:49.547 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:42:49.547 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:49.547 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:42:49.547 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-19T23:42:49.547 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:42:49.547 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:42:49.547 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713570164.057390_278.1", 2024-04-19T23:42:49.547 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:42:49.547 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:42:49.547 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:42:49.547 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:42:44.057390Z", 2024-04-19T23:42:49.548 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:42:49.548 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:42:49.548 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:49.548 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:42:49.548 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-19T23:42:49.548 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-19T23:42:49.548 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-19T23:42:49.548 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:42:49.548 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-19T23:42:49.548 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-19T23:42:49.548 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-19T23:42:49.549 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:42:49.549 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-19T23:42:49.549 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:42:49.549 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:42:49.549 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-19T23:42:49.549 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:49.549 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-19T23:42:49.549 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-19T23:42:49.549 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-19T23:42:49.549 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:42:49.550 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-19T23:42:49Z" 2024-04-19T23:42:49.550 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-19T23:42:49.551 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570164.057390_278.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:42:44.057390Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:42:49Z" } 2024-04-19T23:42:49.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570164.057390_278.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:42:44.057390Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:42:49.551 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570164.057390_278.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:49.551 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570164.057390_278.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:42:49.551 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-19T23:42:49.551 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:42:49.551 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:42:49.551 DEBUG:boto:Method: HEAD 2024-04-19T23:42:49.552 DEBUG:boto:Path: /aktlji-27/ 2024-04-19T23:42:49.552 DEBUG:boto:Data: 2024-04-19T23:42:49.552 DEBUG:boto:Headers: {} 2024-04-19T23:42:49.552 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:49.552 DEBUG:boto:Port: 8000 2024-04-19T23:42:49.552 DEBUG:boto:Params: {} 2024-04-19T23:42:49.552 DEBUG:boto:Token: None 2024-04-19T23:42:49.552 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:49 GMT /aktlji-27/ 2024-04-19T23:42:49.552 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ogkHb9WmTOTz+cRSbRLxwAlg3YY= 2024-04-19T23:42:49.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ogkHb9WmTOTz+cRSbRLxwAlg3YY=', 'Content-Length': '0'} 2024-04-19T23:42:49.555 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000886b04be326245d-0066230179-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:49.555 DEBUG:boto:path=/aktlji-27/obj 2024-04-19T23:42:49.555 DEBUG:boto:auth_path=/aktlji-27/obj 2024-04-19T23:42:49.555 DEBUG:boto:Method: PUT 2024-04-19T23:42:49.555 DEBUG:boto:Path: /aktlji-27/obj 2024-04-19T23:42:49.556 DEBUG:boto:Data: 2024-04-19T23:42:49.556 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:42:49.556 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:49.556 DEBUG:boto:Port: 8000 2024-04-19T23:42:49.556 DEBUG:boto:Params: {} 2024-04-19T23:42:49.556 DEBUG:boto:Token: None 2024-04-19T23:42:49.556 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:42:49 GMT /aktlji-27/obj 2024-04-19T23:42:49.556 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aF9JJhfrB/6JCMSatStZAuEKFXo= 2024-04-19T23:42:49.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:42:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aF9JJhfrB/6JCMSatStZAuEKFXo='} 2024-04-19T23:42:49.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000900cb8d1a643005b-0066230179-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:49.599 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:42:49.599 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:42:49.599 DEBUG:boto:Method: HEAD 2024-04-19T23:42:49.599 DEBUG:boto:Path: /aktlji-27/ 2024-04-19T23:42:49.599 DEBUG:boto:Data: 2024-04-19T23:42:49.599 DEBUG:boto:Headers: {} 2024-04-19T23:42:49.599 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:49.599 DEBUG:boto:Port: 8000 2024-04-19T23:42:49.600 DEBUG:boto:Params: {} 2024-04-19T23:42:49.600 DEBUG:boto:Token: None 2024-04-19T23:42:49.600 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:49 GMT /aktlji-27/ 2024-04-19T23:42:49.600 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ogkHb9WmTOTz+cRSbRLxwAlg3YY= 2024-04-19T23:42:49.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ogkHb9WmTOTz+cRSbRLxwAlg3YY=', 'Content-Length': '0'} 2024-04-19T23:42:49.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074f1b014faf96544-0066230179-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:49.603 DEBUG:boto:path=/aktlji-27/obj 2024-04-19T23:42:49.603 DEBUG:boto:auth_path=/aktlji-27/obj 2024-04-19T23:42:49.603 DEBUG:boto:Method: DELETE 2024-04-19T23:42:49.603 DEBUG:boto:Path: /aktlji-27/obj 2024-04-19T23:42:49.603 DEBUG:boto:Data: 2024-04-19T23:42:49.603 DEBUG:boto:Headers: {} 2024-04-19T23:42:49.603 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:42:49.604 DEBUG:boto:Port: 8000 2024-04-19T23:42:49.604 DEBUG:boto:Params: {} 2024-04-19T23:42:49.604 DEBUG:boto:Token: None 2024-04-19T23:42:49.604 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:42:49 GMT /aktlji-27/obj 2024-04-19T23:42:49.604 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bJtIsFDPF0bxWzTir044ylFgqN4= 2024-04-19T23:42:49.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bJtIsFDPF0bxWzTir044ylFgqN4=', 'Content-Length': '0'} 2024-04-19T23:42:49.614 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000205cd7d35d48007b-0066230179-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:49.615 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:42:49.615 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:42:49.615 DEBUG:boto:Method: HEAD 2024-04-19T23:42:49.615 DEBUG:boto:Path: /aktlji-28/ 2024-04-19T23:42:49.615 DEBUG:boto:Data: 2024-04-19T23:42:49.615 DEBUG:boto:Headers: {} 2024-04-19T23:42:49.615 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:49.615 DEBUG:boto:Port: 8000 2024-04-19T23:42:49.615 DEBUG:boto:Params: {} 2024-04-19T23:42:49.615 DEBUG:boto:Token: None 2024-04-19T23:42:49.615 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:49 GMT /aktlji-28/ 2024-04-19T23:42:49.616 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R6vdXUjOosqSB9LTI3LghLj7SfM= 2024-04-19T23:42:49.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R6vdXUjOosqSB9LTI3LghLj7SfM=', 'Content-Length': '0'} 2024-04-19T23:42:49.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bcb8a7010ad8980e-0066230179-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:49.618 DEBUG:boto:path=/aktlji-28/obj 2024-04-19T23:42:49.618 DEBUG:boto:auth_path=/aktlji-28/obj 2024-04-19T23:42:49.619 DEBUG:boto:Method: PUT 2024-04-19T23:42:49.619 DEBUG:boto:Path: /aktlji-28/obj 2024-04-19T23:42:49.619 DEBUG:boto:Data: 2024-04-19T23:42:49.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:42:49.619 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:49.619 DEBUG:boto:Port: 8000 2024-04-19T23:42:49.619 DEBUG:boto:Params: {} 2024-04-19T23:42:49.619 DEBUG:boto:Token: None 2024-04-19T23:42:49.619 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:42:49 GMT /aktlji-28/obj 2024-04-19T23:42:49.619 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MiLyf419hdkzlwbYuD3Q/Jl13xA= 2024-04-19T23:42:49.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:42:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MiLyf419hdkzlwbYuD3Q/Jl13xA='} 2024-04-19T23:42:49.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e3f686c10c4ca13-0066230179-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:49.627 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:42:49.627 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:42:49.628 DEBUG:boto:Method: HEAD 2024-04-19T23:42:49.628 DEBUG:boto:Path: /aktlji-28/ 2024-04-19T23:42:49.628 DEBUG:boto:Data: 2024-04-19T23:42:49.628 DEBUG:boto:Headers: {} 2024-04-19T23:42:49.628 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:49.628 DEBUG:boto:Port: 8000 2024-04-19T23:42:49.628 DEBUG:boto:Params: {} 2024-04-19T23:42:49.628 DEBUG:boto:Token: None 2024-04-19T23:42:49.628 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:49 GMT /aktlji-28/ 2024-04-19T23:42:49.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R6vdXUjOosqSB9LTI3LghLj7SfM= 2024-04-19T23:42:49.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R6vdXUjOosqSB9LTI3LghLj7SfM=', 'Content-Length': '0'} 2024-04-19T23:42:49.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d9ce5b138e654f6-0066230179-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:49.631 DEBUG:boto:path=/aktlji-28/obj 2024-04-19T23:42:49.631 DEBUG:boto:auth_path=/aktlji-28/obj 2024-04-19T23:42:49.631 DEBUG:boto:Method: DELETE 2024-04-19T23:42:49.631 DEBUG:boto:Path: /aktlji-28/obj 2024-04-19T23:42:49.631 DEBUG:boto:Data: 2024-04-19T23:42:49.632 DEBUG:boto:Headers: {} 2024-04-19T23:42:49.632 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:42:49.632 DEBUG:boto:Port: 8000 2024-04-19T23:42:49.632 DEBUG:boto:Params: {} 2024-04-19T23:42:49.632 DEBUG:boto:Token: None 2024-04-19T23:42:49.632 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:42:49 GMT /aktlji-28/obj 2024-04-19T23:42:49.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OiD1t0nfuFeP+Mp+z+OrePz32tc= 2024-04-19T23:42:49.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OiD1t0nfuFeP+Mp+z+OrePz32tc=', 'Content-Length': '0'} 2024-04-19T23:42:49.642 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000e5fa894f2eb2dbde-0066230179-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:49.643 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:42:49.643 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:42:49.643 DEBUG:boto:Method: HEAD 2024-04-19T23:42:49.643 DEBUG:boto:Path: /aktlji-29/ 2024-04-19T23:42:49.643 DEBUG:boto:Data: 2024-04-19T23:42:49.643 DEBUG:boto:Headers: {} 2024-04-19T23:42:49.643 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:49.643 DEBUG:boto:Port: 8001 2024-04-19T23:42:49.643 DEBUG:boto:Params: {} 2024-04-19T23:42:49.643 DEBUG:boto:Token: None 2024-04-19T23:42:49.644 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:49 GMT /aktlji-29/ 2024-04-19T23:42:49.644 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EdccYd7DGk+ZK+qWu3SK4i0pRWU= 2024-04-19T23:42:49.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EdccYd7DGk+ZK+qWu3SK4i0pRWU=', 'Content-Length': '0'} 2024-04-19T23:42:49.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002921ea273ea051c2-0066230179-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:49.646 DEBUG:boto:path=/aktlji-29/obj 2024-04-19T23:42:49.647 DEBUG:boto:auth_path=/aktlji-29/obj 2024-04-19T23:42:49.647 DEBUG:boto:Method: PUT 2024-04-19T23:42:49.647 DEBUG:boto:Path: /aktlji-29/obj 2024-04-19T23:42:49.647 DEBUG:boto:Data: 2024-04-19T23:42:49.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-19T23:42:49.647 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:49.647 DEBUG:boto:Port: 8001 2024-04-19T23:42:49.647 DEBUG:boto:Params: {} 2024-04-19T23:42:49.647 DEBUG:boto:Token: None 2024-04-19T23:42:49.647 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 19 Apr 2024 23:42:49 GMT /aktlji-29/obj 2024-04-19T23:42:49.648 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nx0I2JmvNpvhKHeiqLQzzAFxWu0= 2024-04-19T23:42:49.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:42:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nx0I2JmvNpvhKHeiqLQzzAFxWu0='} 2024-04-19T23:42:49.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084c43c35cb8b1bb0-0066230179-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:49.655 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:42:49.655 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:42:49.655 DEBUG:boto:Method: HEAD 2024-04-19T23:42:49.655 DEBUG:boto:Path: /aktlji-29/ 2024-04-19T23:42:49.655 DEBUG:boto:Data: 2024-04-19T23:42:49.656 DEBUG:boto:Headers: {} 2024-04-19T23:42:49.656 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:49.656 DEBUG:boto:Port: 8001 2024-04-19T23:42:49.656 DEBUG:boto:Params: {} 2024-04-19T23:42:49.656 DEBUG:boto:Token: None 2024-04-19T23:42:49.656 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:42:49 GMT /aktlji-29/ 2024-04-19T23:42:49.656 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EdccYd7DGk+ZK+qWu3SK4i0pRWU= 2024-04-19T23:42:49.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EdccYd7DGk+ZK+qWu3SK4i0pRWU=', 'Content-Length': '0'} 2024-04-19T23:42:49.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aad0338eea93d0bc-0066230179-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:49.659 DEBUG:boto:path=/aktlji-29/obj 2024-04-19T23:42:49.659 DEBUG:boto:auth_path=/aktlji-29/obj 2024-04-19T23:42:49.659 DEBUG:boto:Method: DELETE 2024-04-19T23:42:49.659 DEBUG:boto:Path: /aktlji-29/obj 2024-04-19T23:42:49.659 DEBUG:boto:Data: 2024-04-19T23:42:49.659 DEBUG:boto:Headers: {} 2024-04-19T23:42:49.659 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:42:49.659 DEBUG:boto:Port: 8001 2024-04-19T23:42:49.659 DEBUG:boto:Params: {} 2024-04-19T23:42:49.659 DEBUG:boto:Token: None 2024-04-19T23:42:49.659 DEBUG:boto:StringToSign: DELETE Fri, 19 Apr 2024 23:42:49 GMT /aktlji-29/obj 2024-04-19T23:42:49.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7v3jesQEEtPqrtbfi8Og7nw9wFM= 2024-04-19T23:42:49.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:42:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7v3jesQEEtPqrtbfi8Og7nw9wFM=', 'Content-Length': '0'} 2024-04-19T23:42:49.670 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000036ebfec82398b971-0066230179-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:42:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:42:49.671 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-27 2024-04-19T23:42:49.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:49.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:42:49.671 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-27 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:42:49.739 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:42:49.739 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:42:49.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:49.782+0000 7f3096ff5780 1 waiting to reach incremental sync.. 2024-04-19T23:42:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:54.783+0000 7f3096ff5780 1 waiting to reach incremental sync.. 2024-04-19T23:42:59.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:42:59.784+0000 7f3096ff5780 1 waiting to reach incremental sync.. 2024-04-19T23:43:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:43:04.784+0000 7f3096ff5780 1 waiting to reach incremental sync.. 2024-04-19T23:43:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:43:09.785+0000 7f3096ff5780 1 bucket sync caught up with source: 2024-04-19T23:43:09.785 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000003.25.7, , ] 2024-04-19T23:43:09.785 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000003.25.7, , ] 2024-04-19T23:43:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:43:09.785+0000 7f3096ff5780 0 bucket checkpoint complete 2024-04-19T23:43:09.803 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-27 2024-04-19T23:43:09.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:43:09.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:43:09.804 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-27 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:43:09.870 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:43:09.870 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:43:09.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:43:09.914+0000 7fdecc76d780 1 bucket sync caught up with source: 2024-04-19T23:43:09.913 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000003.25.7, , ] 2024-04-19T23:43:09.913 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.25.7, , ] 2024-04-19T23:43:09.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:43:09.914+0000 7fdecc76d780 0 bucket checkpoint complete 2024-04-19T23:43:09.927 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-27 2024-04-19T23:43:09.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:43:09.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:43:09.928 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-27 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:43:09.990 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:43:09.990 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:43:10.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:43:10.032+0000 7f62902af780 1 waiting to reach incremental sync.. 2024-04-19T23:43:15.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:43:15.033+0000 7f62902af780 1 waiting to reach incremental sync.. 2024-04-19T23:43:20.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:43:20.033+0000 7f62902af780 1 waiting to reach incremental sync.. 2024-04-19T23:43:25.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:43:25.034+0000 7f62902af780 1 waiting to reach incremental sync.. 2024-04-19T23:43:30.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:43:30.035+0000 7f62902af780 1 waiting to reach incremental sync.. 2024-04-19T23:43:35.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:43:35.036+0000 7f62902af780 1 waiting to reach incremental sync.. 2024-04-19T23:43:40.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:43:40.036+0000 7f62902af780 1 waiting to reach incremental sync.. 2024-04-19T23:43:45.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:43:45.037+0000 7f62902af780 1 waiting to reach incremental sync.. 2024-04-19T23:43:50.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:43:50.037+0000 7f62902af780 1 waiting to reach incremental sync.. 2024-04-19T23:43:55.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:43:55.038+0000 7f62902af780 1 waiting to reach incremental sync.. 2024-04-19T23:44:00.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:00.040+0000 7f62902af780 1 bucket sync caught up with source: 2024-04-19T23:44:00.040 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.66.14, , ] 2024-04-19T23:44:00.040 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.66.14, , ] 2024-04-19T23:44:00.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:00.040+0000 7f62902af780 0 bucket checkpoint complete 2024-04-19T23:44:00.057 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-27 2024-04-19T23:44:00.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:00.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:00.058 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-27 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:00.123 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:00.124 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:00.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:00.166+0000 7f677d434780 1 bucket sync caught up with source: 2024-04-19T23:44:00.166 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.66.14, , ] 2024-04-19T23:44:00.166 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.66.14, , ] 2024-04-19T23:44:00.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:00.166+0000 7f677d434780 0 bucket checkpoint complete 2024-04-19T23:44:00.181 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-27 2024-04-19T23:44:00.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:00.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:00.181 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-27 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:00.244 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:00.244 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:00.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:00.286+0000 7f75e4e62780 1 bucket sync caught up with source: 2024-04-19T23:44:00.285 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.43.14, , ] 2024-04-19T23:44:00.285 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.43.14, , ] 2024-04-19T23:44:00.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:00.286+0000 7f75e4e62780 0 bucket checkpoint complete 2024-04-19T23:44:00.300 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-27 2024-04-19T23:44:00.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:00.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:00.301 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-27 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:00.368 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:00.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:00.412+0000 7fdd1689a780 1 bucket sync caught up with source: 2024-04-19T23:44:00.411 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000001.43.14, , ] 2024-04-19T23:44:00.411 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000001.43.14, , ] 2024-04-19T23:44:00.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:00.412+0000 7fdd1689a780 0 bucket checkpoint complete 2024-04-19T23:44:00.426 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-27 zones={test-zone2, test-zone1} 2024-04-19T23:44:00.426 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.427 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.427 DEBUG:boto:Method: HEAD 2024-04-19T23:44:00.427 DEBUG:boto:Path: /aktlji-27/ 2024-04-19T23:44:00.427 DEBUG:boto:Data: 2024-04-19T23:44:00.427 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.427 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:00.427 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.427 DEBUG:boto:Params: {} 2024-04-19T23:44:00.428 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:44:00.428 DEBUG:boto:Token: None 2024-04-19T23:44:00.428 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/ 2024-04-19T23:44:00.428 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tMz1qWQjEiKMq+97wPzgtfkmtxE= 2024-04-19T23:44:00.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tMz1qWQjEiKMq+97wPzgtfkmtxE=', 'Content-Length': '0'} 2024-04-19T23:44:00.464 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fa8fb483f703596-00662301c0-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.464 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.464 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.464 DEBUG:boto:Method: HEAD 2024-04-19T23:44:00.464 DEBUG:boto:Path: /aktlji-27/ 2024-04-19T23:44:00.464 DEBUG:boto:Data: 2024-04-19T23:44:00.464 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.464 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:00.465 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.465 DEBUG:boto:Params: {} 2024-04-19T23:44:00.465 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:44:00.465 DEBUG:boto:Token: None 2024-04-19T23:44:00.465 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/ 2024-04-19T23:44:00.465 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tMz1qWQjEiKMq+97wPzgtfkmtxE= 2024-04-19T23:44:00.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tMz1qWQjEiKMq+97wPzgtfkmtxE=', 'Content-Length': '0'} 2024-04-19T23:44:00.494 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dad9aa262500539a-00662301c0-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.494 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:44:00.494 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.494 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.494 DEBUG:boto:path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.494 DEBUG:boto:auth_path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.494 DEBUG:boto:Method: GET 2024-04-19T23:44:00.495 DEBUG:boto:Path: /aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.495 DEBUG:boto:Data: 2024-04-19T23:44:00.495 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.495 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:00.495 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.495 DEBUG:boto:Params: {} 2024-04-19T23:44:00.495 DEBUG:boto:Token: None 2024-04-19T23:44:00.496 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/?versions 2024-04-19T23:44:00.496 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ= 2024-04-19T23:44:00.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ=', 'Content-Length': '0'} 2024-04-19T23:44:00.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d194abc828e46746-00662301c0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.498 DEBUG:boto:b'aktlji-27999falseobjnulltrue2024-04-19T23:42:49.608Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:00.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:00.499 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:44:00.499 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.499 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.500 DEBUG:boto:path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.500 DEBUG:boto:auth_path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.500 DEBUG:boto:Method: GET 2024-04-19T23:44:00.500 DEBUG:boto:Path: /aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.500 DEBUG:boto:Data: 2024-04-19T23:44:00.500 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.500 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:00.500 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.500 DEBUG:boto:Params: {} 2024-04-19T23:44:00.500 DEBUG:boto:Token: None 2024-04-19T23:44:00.501 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/?versions 2024-04-19T23:44:00.501 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ= 2024-04-19T23:44:00.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ=', 'Content-Length': '0'} 2024-04-19T23:44:00.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000262a595333f3f56f-00662301c0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.503 DEBUG:boto:b'aktlji-27999falseobjnulltrue2024-04-19T23:42:49.608Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:00.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:00.504 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.504 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.504 DEBUG:boto:path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.504 DEBUG:boto:auth_path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.504 DEBUG:boto:Method: GET 2024-04-19T23:44:00.504 DEBUG:boto:Path: /aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.504 DEBUG:boto:Data: 2024-04-19T23:44:00.505 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.505 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:00.505 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.505 DEBUG:boto:Params: {} 2024-04-19T23:44:00.505 DEBUG:boto:Token: None 2024-04-19T23:44:00.505 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/?versions 2024-04-19T23:44:00.505 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ= 2024-04-19T23:44:00.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ=', 'Content-Length': '0'} 2024-04-19T23:44:00.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033bc60c559f9c1f4-00662301c0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.508 DEBUG:boto:b'aktlji-27999falseobjnulltrue2024-04-19T23:42:49.608Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:00.508 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.508 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.508 DEBUG:boto:path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.509 DEBUG:boto:auth_path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.509 DEBUG:boto:Method: GET 2024-04-19T23:44:00.509 DEBUG:boto:Path: /aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.509 DEBUG:boto:Data: 2024-04-19T23:44:00.509 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.509 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:00.509 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.509 DEBUG:boto:Params: {} 2024-04-19T23:44:00.509 DEBUG:boto:Token: None 2024-04-19T23:44:00.509 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/?versions 2024-04-19T23:44:00.510 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ= 2024-04-19T23:44:00.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ=', 'Content-Length': '0'} 2024-04-19T23:44:00.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bb54a52a7700101-00662301c0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.512 DEBUG:boto:b'aktlji-27999falseobjnulltrue2024-04-19T23:42:49.608Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:00.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:44:00.513 DEBUG:boto:path=/aktlji-27/obj 2024-04-19T23:44:00.513 DEBUG:boto:auth_path=/aktlji-27/obj 2024-04-19T23:44:00.513 DEBUG:boto:Method: HEAD 2024-04-19T23:44:00.513 DEBUG:boto:Path: /aktlji-27/obj 2024-04-19T23:44:00.513 DEBUG:boto:Data: 2024-04-19T23:44:00.513 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.513 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:00.514 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.514 DEBUG:boto:Params: {} 2024-04-19T23:44:00.514 DEBUG:boto:Token: None 2024-04-19T23:44:00.514 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/obj 2024-04-19T23:44:00.514 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+U7fyfhsePFaKJEP4yo4DzCMv0A= 2024-04-19T23:44:00.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+U7fyfhsePFaKJEP4yo4DzCMv0A=', 'Content-Length': '0'} 2024-04-19T23:44:00.516 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000002ff183b37235a0dc-00662301c0-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.516 DEBUG:boto:path=/aktlji-27/obj 2024-04-19T23:44:00.516 DEBUG:boto:auth_path=/aktlji-27/obj 2024-04-19T23:44:00.516 DEBUG:boto:Method: HEAD 2024-04-19T23:44:00.516 DEBUG:boto:Path: /aktlji-27/obj 2024-04-19T23:44:00.516 DEBUG:boto:Data: 2024-04-19T23:44:00.516 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.516 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:00.517 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.517 DEBUG:boto:Params: {} 2024-04-19T23:44:00.517 DEBUG:boto:Token: None 2024-04-19T23:44:00.517 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/obj 2024-04-19T23:44:00.517 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+U7fyfhsePFaKJEP4yo4DzCMv0A= 2024-04-19T23:44:00.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+U7fyfhsePFaKJEP4yo4DzCMv0A=', 'Content-Length': '0'} 2024-04-19T23:44:00.519 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000031c4bf4510c04f4a-00662301c0-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.519 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-27 zones={test-zone2, test-zone1} 2024-04-19T23:44:00.519 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-27 zones={test-zone3, test-zone1} 2024-04-19T23:44:00.519 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.519 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.519 DEBUG:boto:Method: HEAD 2024-04-19T23:44:00.519 DEBUG:boto:Path: /aktlji-27/ 2024-04-19T23:44:00.519 DEBUG:boto:Data: 2024-04-19T23:44:00.519 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.519 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:00.519 DEBUG:boto:Port: 8001 2024-04-19T23:44:00.520 DEBUG:boto:Params: {} 2024-04-19T23:44:00.520 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-19T23:44:00.520 DEBUG:boto:Token: None 2024-04-19T23:44:00.520 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/ 2024-04-19T23:44:00.520 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tMz1qWQjEiKMq+97wPzgtfkmtxE= 2024-04-19T23:44:00.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tMz1qWQjEiKMq+97wPzgtfkmtxE=', 'Content-Length': '0'} 2024-04-19T23:44:00.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca1924afc90dc904-00662301c0-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.561 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.561 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.561 DEBUG:boto:Method: HEAD 2024-04-19T23:44:00.561 DEBUG:boto:Path: /aktlji-27/ 2024-04-19T23:44:00.561 DEBUG:boto:Data: 2024-04-19T23:44:00.561 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.561 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:00.561 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.561 DEBUG:boto:Params: {} 2024-04-19T23:44:00.561 DEBUG:boto:Token: None 2024-04-19T23:44:00.562 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/ 2024-04-19T23:44:00.562 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tMz1qWQjEiKMq+97wPzgtfkmtxE= 2024-04-19T23:44:00.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tMz1qWQjEiKMq+97wPzgtfkmtxE=', 'Content-Length': '0'} 2024-04-19T23:44:00.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002df8fe2fb40c7aae-00662301c0-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.564 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:44:00.564 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.564 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.564 DEBUG:boto:path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.564 DEBUG:boto:auth_path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.564 DEBUG:boto:Method: GET 2024-04-19T23:44:00.565 DEBUG:boto:Path: /aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.565 DEBUG:boto:Data: 2024-04-19T23:44:00.565 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.565 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:00.565 DEBUG:boto:Port: 8001 2024-04-19T23:44:00.565 DEBUG:boto:Params: {} 2024-04-19T23:44:00.565 DEBUG:boto:Token: None 2024-04-19T23:44:00.565 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/?versions 2024-04-19T23:44:00.565 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ= 2024-04-19T23:44:00.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ=', 'Content-Length': '0'} 2024-04-19T23:44:00.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9d69e566b13ca7f-00662301c0-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.568 DEBUG:boto:b'aktlji-27999falseobjnulltrue2024-04-19T23:42:49.608Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:00.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:00.568 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:44:00.568 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.568 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.569 DEBUG:boto:path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.569 DEBUG:boto:auth_path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.569 DEBUG:boto:Method: GET 2024-04-19T23:44:00.569 DEBUG:boto:Path: /aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.569 DEBUG:boto:Data: 2024-04-19T23:44:00.569 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.569 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:00.569 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.569 DEBUG:boto:Params: {} 2024-04-19T23:44:00.569 DEBUG:boto:Token: None 2024-04-19T23:44:00.569 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/?versions 2024-04-19T23:44:00.570 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ= 2024-04-19T23:44:00.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ=', 'Content-Length': '0'} 2024-04-19T23:44:00.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc66564a93c0c53e-00662301c0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.572 DEBUG:boto:b'aktlji-27999falseobjnulltrue2024-04-19T23:42:49.608Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:00.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:00.573 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.573 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.573 DEBUG:boto:path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.573 DEBUG:boto:auth_path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.573 DEBUG:boto:Method: GET 2024-04-19T23:44:00.573 DEBUG:boto:Path: /aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.573 DEBUG:boto:Data: 2024-04-19T23:44:00.573 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.573 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:00.573 DEBUG:boto:Port: 8001 2024-04-19T23:44:00.574 DEBUG:boto:Params: {} 2024-04-19T23:44:00.574 DEBUG:boto:Token: None 2024-04-19T23:44:00.574 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/?versions 2024-04-19T23:44:00.574 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ= 2024-04-19T23:44:00.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ=', 'Content-Length': '0'} 2024-04-19T23:44:00.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006cc2fb3736eed02-00662301c0-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.577 DEBUG:boto:b'aktlji-27999falseobjnulltrue2024-04-19T23:42:49.608Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:00.577 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.577 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.577 DEBUG:boto:path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.577 DEBUG:boto:auth_path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.577 DEBUG:boto:Method: GET 2024-04-19T23:44:00.577 DEBUG:boto:Path: /aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.578 DEBUG:boto:Data: 2024-04-19T23:44:00.578 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.578 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:00.578 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.578 DEBUG:boto:Params: {} 2024-04-19T23:44:00.578 DEBUG:boto:Token: None 2024-04-19T23:44:00.578 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/?versions 2024-04-19T23:44:00.578 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ= 2024-04-19T23:44:00.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ=', 'Content-Length': '0'} 2024-04-19T23:44:00.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b38be4a7fc74d5eb-00662301c0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.581 DEBUG:boto:b'aktlji-27999falseobjnulltrue2024-04-19T23:42:49.608Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:00.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:44:00.581 DEBUG:boto:path=/aktlji-27/obj 2024-04-19T23:44:00.581 DEBUG:boto:auth_path=/aktlji-27/obj 2024-04-19T23:44:00.581 DEBUG:boto:Method: HEAD 2024-04-19T23:44:00.581 DEBUG:boto:Path: /aktlji-27/obj 2024-04-19T23:44:00.582 DEBUG:boto:Data: 2024-04-19T23:44:00.582 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.582 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:00.582 DEBUG:boto:Port: 8001 2024-04-19T23:44:00.582 DEBUG:boto:Params: {} 2024-04-19T23:44:00.582 DEBUG:boto:Token: None 2024-04-19T23:44:00.582 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/obj 2024-04-19T23:44:00.582 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+U7fyfhsePFaKJEP4yo4DzCMv0A= 2024-04-19T23:44:00.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+U7fyfhsePFaKJEP4yo4DzCMv0A=', 'Content-Length': '0'} 2024-04-19T23:44:00.584 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006053ed4acab16215-00662301c0-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.584 DEBUG:boto:path=/aktlji-27/obj 2024-04-19T23:44:00.584 DEBUG:boto:auth_path=/aktlji-27/obj 2024-04-19T23:44:00.584 DEBUG:boto:Method: HEAD 2024-04-19T23:44:00.584 DEBUG:boto:Path: /aktlji-27/obj 2024-04-19T23:44:00.584 DEBUG:boto:Data: 2024-04-19T23:44:00.584 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.584 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:00.585 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.585 DEBUG:boto:Params: {} 2024-04-19T23:44:00.585 DEBUG:boto:Token: None 2024-04-19T23:44:00.585 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/obj 2024-04-19T23:44:00.585 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+U7fyfhsePFaKJEP4yo4DzCMv0A= 2024-04-19T23:44:00.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+U7fyfhsePFaKJEP4yo4DzCMv0A=', 'Content-Length': '0'} 2024-04-19T23:44:00.587 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a803e0170ccd452d-00662301c0-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.587 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-27 zones={test-zone3, test-zone1} 2024-04-19T23:44:00.587 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-27 zones={test-zone3, test-zone2} 2024-04-19T23:44:00.587 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.587 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.587 DEBUG:boto:Method: HEAD 2024-04-19T23:44:00.587 DEBUG:boto:Path: /aktlji-27/ 2024-04-19T23:44:00.587 DEBUG:boto:Data: 2024-04-19T23:44:00.587 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.588 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:00.588 DEBUG:boto:Port: 8001 2024-04-19T23:44:00.588 DEBUG:boto:Params: {} 2024-04-19T23:44:00.588 DEBUG:boto:Token: None 2024-04-19T23:44:00.588 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/ 2024-04-19T23:44:00.588 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tMz1qWQjEiKMq+97wPzgtfkmtxE= 2024-04-19T23:44:00.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tMz1qWQjEiKMq+97wPzgtfkmtxE=', 'Content-Length': '0'} 2024-04-19T23:44:00.590 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c7b1d169b7b47a4-00662301c0-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.590 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.591 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.591 DEBUG:boto:Method: HEAD 2024-04-19T23:44:00.591 DEBUG:boto:Path: /aktlji-27/ 2024-04-19T23:44:00.591 DEBUG:boto:Data: 2024-04-19T23:44:00.591 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.591 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:00.591 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.591 DEBUG:boto:Params: {} 2024-04-19T23:44:00.591 DEBUG:boto:Token: None 2024-04-19T23:44:00.591 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/ 2024-04-19T23:44:00.592 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tMz1qWQjEiKMq+97wPzgtfkmtxE= 2024-04-19T23:44:00.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tMz1qWQjEiKMq+97wPzgtfkmtxE=', 'Content-Length': '0'} 2024-04-19T23:44:00.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a2cd71ee30c5ea4-00662301c0-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.594 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:44:00.594 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.595 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.595 DEBUG:boto:path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.595 DEBUG:boto:auth_path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.595 DEBUG:boto:Method: GET 2024-04-19T23:44:00.595 DEBUG:boto:Path: /aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.595 DEBUG:boto:Data: 2024-04-19T23:44:00.595 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.595 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:00.595 DEBUG:boto:Port: 8001 2024-04-19T23:44:00.595 DEBUG:boto:Params: {} 2024-04-19T23:44:00.595 DEBUG:boto:Token: None 2024-04-19T23:44:00.596 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/?versions 2024-04-19T23:44:00.596 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ= 2024-04-19T23:44:00.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ=', 'Content-Length': '0'} 2024-04-19T23:44:00.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf131876010c454a-00662301c0-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.598 DEBUG:boto:b'aktlji-27999falseobjnulltrue2024-04-19T23:42:49.608Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:00.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:00.599 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:44:00.599 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.599 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.599 DEBUG:boto:path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.599 DEBUG:boto:auth_path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.599 DEBUG:boto:Method: GET 2024-04-19T23:44:00.599 DEBUG:boto:Path: /aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.599 DEBUG:boto:Data: 2024-04-19T23:44:00.600 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.600 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:00.600 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.600 DEBUG:boto:Params: {} 2024-04-19T23:44:00.600 DEBUG:boto:Token: None 2024-04-19T23:44:00.600 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/?versions 2024-04-19T23:44:00.600 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ= 2024-04-19T23:44:00.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ=', 'Content-Length': '0'} 2024-04-19T23:44:00.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b68893e7a72d417c-00662301c0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.603 DEBUG:boto:b'aktlji-27999falseobjnulltrue2024-04-19T23:42:49.608Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:00.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:00.603 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.603 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.603 DEBUG:boto:path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.603 DEBUG:boto:auth_path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.603 DEBUG:boto:Method: GET 2024-04-19T23:44:00.604 DEBUG:boto:Path: /aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.604 DEBUG:boto:Data: 2024-04-19T23:44:00.604 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.604 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:00.604 DEBUG:boto:Port: 8001 2024-04-19T23:44:00.604 DEBUG:boto:Params: {} 2024-04-19T23:44:00.604 DEBUG:boto:Token: None 2024-04-19T23:44:00.604 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/?versions 2024-04-19T23:44:00.604 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ= 2024-04-19T23:44:00.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ=', 'Content-Length': '0'} 2024-04-19T23:44:00.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f10aa8bc2170c3b9-00662301c0-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.650 DEBUG:boto:b'aktlji-27999falseobjnulltrue2024-04-19T23:42:49.608Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:00.651 DEBUG:boto:path=/aktlji-27/ 2024-04-19T23:44:00.651 DEBUG:boto:auth_path=/aktlji-27/ 2024-04-19T23:44:00.651 DEBUG:boto:path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.651 DEBUG:boto:auth_path=/aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.651 DEBUG:boto:Method: GET 2024-04-19T23:44:00.651 DEBUG:boto:Path: /aktlji-27/?versions&max-keys=999 2024-04-19T23:44:00.651 DEBUG:boto:Data: 2024-04-19T23:44:00.651 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.652 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:00.652 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.652 DEBUG:boto:Params: {} 2024-04-19T23:44:00.652 DEBUG:boto:Token: None 2024-04-19T23:44:00.652 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/?versions 2024-04-19T23:44:00.652 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ= 2024-04-19T23:44:00.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L6UolRmiFeVTYoHvq+A0ReXCoHQ=', 'Content-Length': '0'} 2024-04-19T23:44:00.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079b604e4d3205e48-00662301c0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.655 DEBUG:boto:b'aktlji-27999falseobjnulltrue2024-04-19T23:42:49.608Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:00.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:44:00.655 DEBUG:boto:path=/aktlji-27/obj 2024-04-19T23:44:00.655 DEBUG:boto:auth_path=/aktlji-27/obj 2024-04-19T23:44:00.655 DEBUG:boto:Method: HEAD 2024-04-19T23:44:00.655 DEBUG:boto:Path: /aktlji-27/obj 2024-04-19T23:44:00.656 DEBUG:boto:Data: 2024-04-19T23:44:00.656 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.656 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:00.656 DEBUG:boto:Port: 8001 2024-04-19T23:44:00.656 DEBUG:boto:Params: {} 2024-04-19T23:44:00.656 DEBUG:boto:Token: None 2024-04-19T23:44:00.656 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/obj 2024-04-19T23:44:00.656 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+U7fyfhsePFaKJEP4yo4DzCMv0A= 2024-04-19T23:44:00.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+U7fyfhsePFaKJEP4yo4DzCMv0A=', 'Content-Length': '0'} 2024-04-19T23:44:00.658 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a04ac268971a5d50-00662301c0-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.658 DEBUG:boto:path=/aktlji-27/obj 2024-04-19T23:44:00.658 DEBUG:boto:auth_path=/aktlji-27/obj 2024-04-19T23:44:00.658 DEBUG:boto:Method: HEAD 2024-04-19T23:44:00.658 DEBUG:boto:Path: /aktlji-27/obj 2024-04-19T23:44:00.658 DEBUG:boto:Data: 2024-04-19T23:44:00.658 DEBUG:boto:Headers: {} 2024-04-19T23:44:00.658 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:00.659 DEBUG:boto:Port: 8000 2024-04-19T23:44:00.659 DEBUG:boto:Params: {} 2024-04-19T23:44:00.659 DEBUG:boto:Token: None 2024-04-19T23:44:00.659 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:00 GMT /aktlji-27/obj 2024-04-19T23:44:00.659 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+U7fyfhsePFaKJEP4yo4DzCMv0A= 2024-04-19T23:44:00.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+U7fyfhsePFaKJEP4yo4DzCMv0A=', 'Content-Length': '0'} 2024-04-19T23:44:00.661 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000583c467059794f1f-00662301c0-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:00.661 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-27 zones={test-zone3, test-zone2} 2024-04-19T23:44:00.661 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-28 2024-04-19T23:44:00.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:00.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:00.662 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-28 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:00.728 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:00.728 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:00.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:00.772+0000 7fddad5ed780 1 bucket sync caught up with source: 2024-04-19T23:44:00.771 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000001.26.14, , ] 2024-04-19T23:44:00.772 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000001.26.14, , ] 2024-04-19T23:44:00.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:00.772+0000 7fddad5ed780 0 bucket checkpoint complete 2024-04-19T23:44:00.787 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-28 2024-04-19T23:44:00.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:00.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:00.787 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-28 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:00.854 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:00.854 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:00.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:00.897+0000 7f057d799780 1 bucket sync caught up with source: 2024-04-19T23:44:00.897 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.26.14, , ] 2024-04-19T23:44:00.897 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.26.14, , ] 2024-04-19T23:44:00.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:00.897+0000 7f057d799780 0 bucket checkpoint complete 2024-04-19T23:44:00.912 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-28 2024-04-19T23:44:00.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:00.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:00.913 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-28 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:00.974 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:00.975 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:01.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:01.016+0000 7f7e79727780 1 bucket sync caught up with source: 2024-04-19T23:44:01.016 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000003.162.7, , ] 2024-04-19T23:44:01.016 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.162.7, , ] 2024-04-19T23:44:01.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:01.016+0000 7f7e79727780 0 bucket checkpoint complete 2024-04-19T23:44:01.031 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-28 2024-04-19T23:44:01.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.031 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-28 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:01.093 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:01.094 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:01.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:01.135+0000 7f31953ea780 1 bucket sync caught up with source: 2024-04-19T23:44:01.135 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000003.162.7, , ] 2024-04-19T23:44:01.135 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.162.7, , ] 2024-04-19T23:44:01.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:01.135+0000 7f31953ea780 0 bucket checkpoint complete 2024-04-19T23:44:01.149 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-28 2024-04-19T23:44:01.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.149 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-28 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:01.211 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:01.211 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:01.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:01.252+0000 7f85781ff780 1 bucket sync caught up with source: 2024-04-19T23:44:01.252 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.67.14, , ] 2024-04-19T23:44:01.252 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.67.14, , ] 2024-04-19T23:44:01.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:01.252+0000 7f85781ff780 0 bucket checkpoint complete 2024-04-19T23:44:01.267 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-28 2024-04-19T23:44:01.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.268 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-28 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:01.335 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:01.335 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:01.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:01.379+0000 7f5807a15780 1 bucket sync caught up with source: 2024-04-19T23:44:01.378 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000001.67.14, , ] 2024-04-19T23:44:01.378 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000001.67.14, , ] 2024-04-19T23:44:01.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:01.379+0000 7f5807a15780 0 bucket checkpoint complete 2024-04-19T23:44:01.413 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-28 zones={test-zone2, test-zone1} 2024-04-19T23:44:01.413 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.414 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.414 DEBUG:boto:Method: HEAD 2024-04-19T23:44:01.414 DEBUG:boto:Path: /aktlji-28/ 2024-04-19T23:44:01.414 DEBUG:boto:Data: 2024-04-19T23:44:01.414 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.414 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:01.414 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.414 DEBUG:boto:Params: {} 2024-04-19T23:44:01.414 DEBUG:boto:Token: None 2024-04-19T23:44:01.415 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/ 2024-04-19T23:44:01.415 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PK8ow+WDBzUlCDPGipUCEHbfitg= 2024-04-19T23:44:01.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PK8ow+WDBzUlCDPGipUCEHbfitg=', 'Content-Length': '0'} 2024-04-19T23:44:01.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009de78c4121b67cf2-00662301c1-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.418 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.418 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.418 DEBUG:boto:Method: HEAD 2024-04-19T23:44:01.418 DEBUG:boto:Path: /aktlji-28/ 2024-04-19T23:44:01.418 DEBUG:boto:Data: 2024-04-19T23:44:01.418 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.418 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:01.419 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.419 DEBUG:boto:Params: {} 2024-04-19T23:44:01.419 DEBUG:boto:Token: None 2024-04-19T23:44:01.419 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/ 2024-04-19T23:44:01.419 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PK8ow+WDBzUlCDPGipUCEHbfitg= 2024-04-19T23:44:01.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PK8ow+WDBzUlCDPGipUCEHbfitg=', 'Content-Length': '0'} 2024-04-19T23:44:01.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1ab57ef498a6149-00662301c1-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.422 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:44:01.422 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.422 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.422 DEBUG:boto:path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.422 DEBUG:boto:auth_path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.422 DEBUG:boto:Method: GET 2024-04-19T23:44:01.422 DEBUG:boto:Path: /aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.422 DEBUG:boto:Data: 2024-04-19T23:44:01.423 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.423 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:01.423 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.423 DEBUG:boto:Params: {} 2024-04-19T23:44:01.423 DEBUG:boto:Token: None 2024-04-19T23:44:01.423 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/?versions 2024-04-19T23:44:01.423 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k= 2024-04-19T23:44:01.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k=', 'Content-Length': '0'} 2024-04-19T23:44:01.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe0e7edbdb1f1653-00662301c1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.426 DEBUG:boto:b'aktlji-28999falseobjnulltrue2024-04-19T23:42:49.636Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:01.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:01.427 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:44:01.427 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.427 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.427 DEBUG:boto:path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.427 DEBUG:boto:auth_path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.427 DEBUG:boto:Method: GET 2024-04-19T23:44:01.427 DEBUG:boto:Path: /aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.427 DEBUG:boto:Data: 2024-04-19T23:44:01.428 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.428 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:01.428 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.428 DEBUG:boto:Params: {} 2024-04-19T23:44:01.428 DEBUG:boto:Token: None 2024-04-19T23:44:01.428 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/?versions 2024-04-19T23:44:01.428 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k= 2024-04-19T23:44:01.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k=', 'Content-Length': '0'} 2024-04-19T23:44:01.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6fd72b0b6f6d51e-00662301c1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.431 DEBUG:boto:b'aktlji-28999falseobjnulltrue2024-04-19T23:42:49.636Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:01.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:01.432 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.432 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.432 DEBUG:boto:path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.432 DEBUG:boto:auth_path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.432 DEBUG:boto:Method: GET 2024-04-19T23:44:01.432 DEBUG:boto:Path: /aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.432 DEBUG:boto:Data: 2024-04-19T23:44:01.432 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.433 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:01.433 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.433 DEBUG:boto:Params: {} 2024-04-19T23:44:01.433 DEBUG:boto:Token: None 2024-04-19T23:44:01.433 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/?versions 2024-04-19T23:44:01.433 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k= 2024-04-19T23:44:01.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k=', 'Content-Length': '0'} 2024-04-19T23:44:01.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000885ec500ea36ccb6-00662301c1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.436 DEBUG:boto:b'aktlji-28999falseobjnulltrue2024-04-19T23:42:49.636Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:01.436 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.437 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.437 DEBUG:boto:path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.437 DEBUG:boto:auth_path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.437 DEBUG:boto:Method: GET 2024-04-19T23:44:01.437 DEBUG:boto:Path: /aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.437 DEBUG:boto:Data: 2024-04-19T23:44:01.437 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.437 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:01.438 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.438 DEBUG:boto:Params: {} 2024-04-19T23:44:01.438 DEBUG:boto:Token: None 2024-04-19T23:44:01.438 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/?versions 2024-04-19T23:44:01.438 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k= 2024-04-19T23:44:01.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k=', 'Content-Length': '0'} 2024-04-19T23:44:01.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000170cf11dc92065f7-00662301c1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.486 DEBUG:boto:b'aktlji-28999falseobjnulltrue2024-04-19T23:42:49.636Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:01.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:44:01.487 DEBUG:boto:path=/aktlji-28/obj 2024-04-19T23:44:01.487 DEBUG:boto:auth_path=/aktlji-28/obj 2024-04-19T23:44:01.487 DEBUG:boto:Method: HEAD 2024-04-19T23:44:01.487 DEBUG:boto:Path: /aktlji-28/obj 2024-04-19T23:44:01.487 DEBUG:boto:Data: 2024-04-19T23:44:01.487 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.487 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:01.487 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.488 DEBUG:boto:Params: {} 2024-04-19T23:44:01.488 DEBUG:boto:Token: None 2024-04-19T23:44:01.488 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/obj 2024-04-19T23:44:01.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WvO4HgfWWd7oOnsBlBgF/Tltads= 2024-04-19T23:44:01.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WvO4HgfWWd7oOnsBlBgF/Tltads=', 'Content-Length': '0'} 2024-04-19T23:44:01.490 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000000bc37602ebe3f395-00662301c1-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.490 DEBUG:boto:path=/aktlji-28/obj 2024-04-19T23:44:01.490 DEBUG:boto:auth_path=/aktlji-28/obj 2024-04-19T23:44:01.490 DEBUG:boto:Method: HEAD 2024-04-19T23:44:01.490 DEBUG:boto:Path: /aktlji-28/obj 2024-04-19T23:44:01.490 DEBUG:boto:Data: 2024-04-19T23:44:01.490 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.491 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:01.491 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.491 DEBUG:boto:Params: {} 2024-04-19T23:44:01.491 DEBUG:boto:Token: None 2024-04-19T23:44:01.491 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/obj 2024-04-19T23:44:01.491 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WvO4HgfWWd7oOnsBlBgF/Tltads= 2024-04-19T23:44:01.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WvO4HgfWWd7oOnsBlBgF/Tltads=', 'Content-Length': '0'} 2024-04-19T23:44:01.493 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006b91f1afc6de61cf-00662301c1-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.493 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-28 zones={test-zone2, test-zone1} 2024-04-19T23:44:01.493 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-28 zones={test-zone3, test-zone1} 2024-04-19T23:44:01.493 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.494 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.494 DEBUG:boto:Method: HEAD 2024-04-19T23:44:01.494 DEBUG:boto:Path: /aktlji-28/ 2024-04-19T23:44:01.494 DEBUG:boto:Data: 2024-04-19T23:44:01.494 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.494 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:01.494 DEBUG:boto:Port: 8001 2024-04-19T23:44:01.494 DEBUG:boto:Params: {} 2024-04-19T23:44:01.494 DEBUG:boto:Token: None 2024-04-19T23:44:01.495 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/ 2024-04-19T23:44:01.495 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PK8ow+WDBzUlCDPGipUCEHbfitg= 2024-04-19T23:44:01.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PK8ow+WDBzUlCDPGipUCEHbfitg=', 'Content-Length': '0'} 2024-04-19T23:44:01.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df5a47a8c2c7a771-00662301c1-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.500 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.500 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.500 DEBUG:boto:Method: HEAD 2024-04-19T23:44:01.500 DEBUG:boto:Path: /aktlji-28/ 2024-04-19T23:44:01.500 DEBUG:boto:Data: 2024-04-19T23:44:01.500 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.500 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:01.500 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.500 DEBUG:boto:Params: {} 2024-04-19T23:44:01.500 DEBUG:boto:Token: None 2024-04-19T23:44:01.501 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/ 2024-04-19T23:44:01.501 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PK8ow+WDBzUlCDPGipUCEHbfitg= 2024-04-19T23:44:01.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PK8ow+WDBzUlCDPGipUCEHbfitg=', 'Content-Length': '0'} 2024-04-19T23:44:01.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ad32bc73071c946-00662301c1-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.503 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:44:01.504 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.504 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.504 DEBUG:boto:path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.504 DEBUG:boto:auth_path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.504 DEBUG:boto:Method: GET 2024-04-19T23:44:01.504 DEBUG:boto:Path: /aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.504 DEBUG:boto:Data: 2024-04-19T23:44:01.504 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.505 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:01.505 DEBUG:boto:Port: 8001 2024-04-19T23:44:01.505 DEBUG:boto:Params: {} 2024-04-19T23:44:01.505 DEBUG:boto:Token: None 2024-04-19T23:44:01.505 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/?versions 2024-04-19T23:44:01.505 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k= 2024-04-19T23:44:01.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k=', 'Content-Length': '0'} 2024-04-19T23:44:01.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008975de070cb4edb-00662301c1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.509 DEBUG:boto:b'aktlji-28999falseobjnulltrue2024-04-19T23:42:49.636Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:01.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:01.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:44:01.510 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.510 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.510 DEBUG:boto:path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.510 DEBUG:boto:auth_path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.510 DEBUG:boto:Method: GET 2024-04-19T23:44:01.510 DEBUG:boto:Path: /aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.510 DEBUG:boto:Data: 2024-04-19T23:44:01.511 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.511 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:01.511 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.511 DEBUG:boto:Params: {} 2024-04-19T23:44:01.511 DEBUG:boto:Token: None 2024-04-19T23:44:01.511 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/?versions 2024-04-19T23:44:01.511 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k= 2024-04-19T23:44:01.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k=', 'Content-Length': '0'} 2024-04-19T23:44:01.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021c552a13c52eb79-00662301c1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.514 DEBUG:boto:b'aktlji-28999falseobjnulltrue2024-04-19T23:42:49.636Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:01.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:01.515 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.515 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.515 DEBUG:boto:path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.515 DEBUG:boto:auth_path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.515 DEBUG:boto:Method: GET 2024-04-19T23:44:01.515 DEBUG:boto:Path: /aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.516 DEBUG:boto:Data: 2024-04-19T23:44:01.516 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.516 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:01.516 DEBUG:boto:Port: 8001 2024-04-19T23:44:01.516 DEBUG:boto:Params: {} 2024-04-19T23:44:01.516 DEBUG:boto:Token: None 2024-04-19T23:44:01.516 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/?versions 2024-04-19T23:44:01.517 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k= 2024-04-19T23:44:01.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k=', 'Content-Length': '0'} 2024-04-19T23:44:01.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098cbde36fda6c92b-00662301c1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.566 DEBUG:boto:b'aktlji-28999falseobjnulltrue2024-04-19T23:42:49.636Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:01.567 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.567 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.567 DEBUG:boto:path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.567 DEBUG:boto:auth_path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.567 DEBUG:boto:Method: GET 2024-04-19T23:44:01.567 DEBUG:boto:Path: /aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.567 DEBUG:boto:Data: 2024-04-19T23:44:01.567 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.568 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:01.568 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.568 DEBUG:boto:Params: {} 2024-04-19T23:44:01.568 DEBUG:boto:Token: None 2024-04-19T23:44:01.568 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/?versions 2024-04-19T23:44:01.568 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k= 2024-04-19T23:44:01.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k=', 'Content-Length': '0'} 2024-04-19T23:44:01.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000137308effcd3baf9-00662301c1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.571 DEBUG:boto:b'aktlji-28999falseobjnulltrue2024-04-19T23:42:49.636Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:01.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:44:01.572 DEBUG:boto:path=/aktlji-28/obj 2024-04-19T23:44:01.572 DEBUG:boto:auth_path=/aktlji-28/obj 2024-04-19T23:44:01.572 DEBUG:boto:Method: HEAD 2024-04-19T23:44:01.572 DEBUG:boto:Path: /aktlji-28/obj 2024-04-19T23:44:01.572 DEBUG:boto:Data: 2024-04-19T23:44:01.572 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.572 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:01.572 DEBUG:boto:Port: 8001 2024-04-19T23:44:01.572 DEBUG:boto:Params: {} 2024-04-19T23:44:01.572 DEBUG:boto:Token: None 2024-04-19T23:44:01.573 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/obj 2024-04-19T23:44:01.573 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WvO4HgfWWd7oOnsBlBgF/Tltads= 2024-04-19T23:44:01.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WvO4HgfWWd7oOnsBlBgF/Tltads=', 'Content-Length': '0'} 2024-04-19T23:44:01.574 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000587fd33b7e0be312-00662301c1-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.574 DEBUG:boto:path=/aktlji-28/obj 2024-04-19T23:44:01.575 DEBUG:boto:auth_path=/aktlji-28/obj 2024-04-19T23:44:01.575 DEBUG:boto:Method: HEAD 2024-04-19T23:44:01.575 DEBUG:boto:Path: /aktlji-28/obj 2024-04-19T23:44:01.575 DEBUG:boto:Data: 2024-04-19T23:44:01.575 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.575 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:01.575 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.575 DEBUG:boto:Params: {} 2024-04-19T23:44:01.575 DEBUG:boto:Token: None 2024-04-19T23:44:01.575 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/obj 2024-04-19T23:44:01.576 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WvO4HgfWWd7oOnsBlBgF/Tltads= 2024-04-19T23:44:01.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WvO4HgfWWd7oOnsBlBgF/Tltads=', 'Content-Length': '0'} 2024-04-19T23:44:01.578 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c8e124d6c6f1c80a-00662301c1-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.578 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-28 zones={test-zone3, test-zone1} 2024-04-19T23:44:01.578 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-28 zones={test-zone3, test-zone2} 2024-04-19T23:44:01.578 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.578 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.578 DEBUG:boto:Method: HEAD 2024-04-19T23:44:01.578 DEBUG:boto:Path: /aktlji-28/ 2024-04-19T23:44:01.578 DEBUG:boto:Data: 2024-04-19T23:44:01.578 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.579 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:01.579 DEBUG:boto:Port: 8001 2024-04-19T23:44:01.579 DEBUG:boto:Params: {} 2024-04-19T23:44:01.579 DEBUG:boto:Token: None 2024-04-19T23:44:01.579 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/ 2024-04-19T23:44:01.579 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PK8ow+WDBzUlCDPGipUCEHbfitg= 2024-04-19T23:44:01.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PK8ow+WDBzUlCDPGipUCEHbfitg=', 'Content-Length': '0'} 2024-04-19T23:44:01.581 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9690195342c6f07-00662301c1-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.582 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.582 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.582 DEBUG:boto:Method: HEAD 2024-04-19T23:44:01.582 DEBUG:boto:Path: /aktlji-28/ 2024-04-19T23:44:01.582 DEBUG:boto:Data: 2024-04-19T23:44:01.582 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.582 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:01.582 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.582 DEBUG:boto:Params: {} 2024-04-19T23:44:01.582 DEBUG:boto:Token: None 2024-04-19T23:44:01.583 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/ 2024-04-19T23:44:01.583 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PK8ow+WDBzUlCDPGipUCEHbfitg= 2024-04-19T23:44:01.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PK8ow+WDBzUlCDPGipUCEHbfitg=', 'Content-Length': '0'} 2024-04-19T23:44:01.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef4557ff2616b446-00662301c1-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.586 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:44:01.586 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.586 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.586 DEBUG:boto:path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.586 DEBUG:boto:auth_path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.586 DEBUG:boto:Method: GET 2024-04-19T23:44:01.586 DEBUG:boto:Path: /aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.587 DEBUG:boto:Data: 2024-04-19T23:44:01.587 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.587 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:01.587 DEBUG:boto:Port: 8001 2024-04-19T23:44:01.587 DEBUG:boto:Params: {} 2024-04-19T23:44:01.587 DEBUG:boto:Token: None 2024-04-19T23:44:01.587 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/?versions 2024-04-19T23:44:01.587 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k= 2024-04-19T23:44:01.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k=', 'Content-Length': '0'} 2024-04-19T23:44:01.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0910f23a4b03ca6-00662301c1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.590 DEBUG:boto:b'aktlji-28999falseobjnulltrue2024-04-19T23:42:49.636Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:01.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:01.590 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:44:01.591 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.591 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.591 DEBUG:boto:path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.591 DEBUG:boto:auth_path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.591 DEBUG:boto:Method: GET 2024-04-19T23:44:01.591 DEBUG:boto:Path: /aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.591 DEBUG:boto:Data: 2024-04-19T23:44:01.591 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.591 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:01.591 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.592 DEBUG:boto:Params: {} 2024-04-19T23:44:01.592 DEBUG:boto:Token: None 2024-04-19T23:44:01.592 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/?versions 2024-04-19T23:44:01.592 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k= 2024-04-19T23:44:01.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k=', 'Content-Length': '0'} 2024-04-19T23:44:01.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce8c932aa3f4ae23-00662301c1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.595 DEBUG:boto:b'aktlji-28999falseobjnulltrue2024-04-19T23:42:49.636Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:01.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:01.595 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.595 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.596 DEBUG:boto:path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.596 DEBUG:boto:auth_path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.596 DEBUG:boto:Method: GET 2024-04-19T23:44:01.596 DEBUG:boto:Path: /aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.596 DEBUG:boto:Data: 2024-04-19T23:44:01.596 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.596 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:01.596 DEBUG:boto:Port: 8001 2024-04-19T23:44:01.596 DEBUG:boto:Params: {} 2024-04-19T23:44:01.597 DEBUG:boto:Token: None 2024-04-19T23:44:01.597 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/?versions 2024-04-19T23:44:01.597 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k= 2024-04-19T23:44:01.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k=', 'Content-Length': '0'} 2024-04-19T23:44:01.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a3702461a12d38c-00662301c1-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.600 DEBUG:boto:b'aktlji-28999falseobjnulltrue2024-04-19T23:42:49.636Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:01.600 DEBUG:boto:path=/aktlji-28/ 2024-04-19T23:44:01.600 DEBUG:boto:auth_path=/aktlji-28/ 2024-04-19T23:44:01.600 DEBUG:boto:path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.600 DEBUG:boto:auth_path=/aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.601 DEBUG:boto:Method: GET 2024-04-19T23:44:01.601 DEBUG:boto:Path: /aktlji-28/?versions&max-keys=999 2024-04-19T23:44:01.601 DEBUG:boto:Data: 2024-04-19T23:44:01.601 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.601 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:01.601 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.601 DEBUG:boto:Params: {} 2024-04-19T23:44:01.601 DEBUG:boto:Token: None 2024-04-19T23:44:01.601 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/?versions 2024-04-19T23:44:01.601 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k= 2024-04-19T23:44:01.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sXusZQmukX0heE/PGY5fUOH8f0k=', 'Content-Length': '0'} 2024-04-19T23:44:01.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c409e9cdbd9e7aa-00662301c1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.604 DEBUG:boto:b'aktlji-28999falseobjnulltrue2024-04-19T23:42:49.636Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:01.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:44:01.604 DEBUG:boto:path=/aktlji-28/obj 2024-04-19T23:44:01.605 DEBUG:boto:auth_path=/aktlji-28/obj 2024-04-19T23:44:01.605 DEBUG:boto:Method: HEAD 2024-04-19T23:44:01.605 DEBUG:boto:Path: /aktlji-28/obj 2024-04-19T23:44:01.605 DEBUG:boto:Data: 2024-04-19T23:44:01.605 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.605 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:01.606 DEBUG:boto:Port: 8001 2024-04-19T23:44:01.606 DEBUG:boto:Params: {} 2024-04-19T23:44:01.606 DEBUG:boto:Token: None 2024-04-19T23:44:01.606 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/obj 2024-04-19T23:44:01.607 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WvO4HgfWWd7oOnsBlBgF/Tltads= 2024-04-19T23:44:01.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WvO4HgfWWd7oOnsBlBgF/Tltads=', 'Content-Length': '0'} 2024-04-19T23:44:01.609 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d97e90cda2b2b521-00662301c1-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.609 DEBUG:boto:path=/aktlji-28/obj 2024-04-19T23:44:01.609 DEBUG:boto:auth_path=/aktlji-28/obj 2024-04-19T23:44:01.609 DEBUG:boto:Method: HEAD 2024-04-19T23:44:01.610 DEBUG:boto:Path: /aktlji-28/obj 2024-04-19T23:44:01.610 DEBUG:boto:Data: 2024-04-19T23:44:01.610 DEBUG:boto:Headers: {} 2024-04-19T23:44:01.610 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:01.610 DEBUG:boto:Port: 8000 2024-04-19T23:44:01.611 DEBUG:boto:Params: {} 2024-04-19T23:44:01.611 DEBUG:boto:Token: None 2024-04-19T23:44:01.611 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:01 GMT /aktlji-28/obj 2024-04-19T23:44:01.611 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WvO4HgfWWd7oOnsBlBgF/Tltads= 2024-04-19T23:44:01.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WvO4HgfWWd7oOnsBlBgF/Tltads=', 'Content-Length': '0'} 2024-04-19T23:44:01.613 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d0dab918cbf4b2f2-00662301c1-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:01.614 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-28 zones={test-zone3, test-zone2} 2024-04-19T23:44:01.614 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-29 2024-04-19T23:44:01.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.614 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-29 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:01.682 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:01.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:01.726+0000 7fd35366d780 1 bucket sync caught up with source: 2024-04-19T23:44:01.725 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000001.18.14, , ] 2024-04-19T23:44:01.725 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000001.18.14, , ] 2024-04-19T23:44:01.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:01.726+0000 7fd35366d780 0 bucket checkpoint complete 2024-04-19T23:44:01.741 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-29 2024-04-19T23:44:01.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.741 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-29 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:01.809 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:01.809 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:01.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:01.852+0000 7f86c7652780 1 bucket sync caught up with source: 2024-04-19T23:44:01.852 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.18.14, , ] 2024-04-19T23:44:01.852 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.18.14, , ] 2024-04-19T23:44:01.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:01.852+0000 7f86c7652780 0 bucket checkpoint complete 2024-04-19T23:44:01.867 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-29 2024-04-19T23:44:01.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.868 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-29 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:01.930 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:01.930 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:01.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:01.971+0000 7f9e0d1b0780 1 bucket sync caught up with source: 2024-04-19T23:44:01.971 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.76.14, , ] 2024-04-19T23:44:01.971 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.76.14, , ] 2024-04-19T23:44:01.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:01.971+0000 7f9e0d1b0780 0 bucket checkpoint complete 2024-04-19T23:44:01.985 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone3 bucket=aktlji-29 2024-04-19T23:44:01.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:01.986 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-29 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:02.048 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:02.048 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:02.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:02.088+0000 7f006f434780 1 bucket sync caught up with source: 2024-04-19T23:44:02.088 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.76.14, , ] 2024-04-19T23:44:02.088 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.76.14, , ] 2024-04-19T23:44:02.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:02.088+0000 7f006f434780 0 bucket checkpoint complete 2024-04-19T23:44:02.102 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone1 bucket=aktlji-29 2024-04-19T23:44:02.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:02.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:02.103 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-29 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:02.164 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:02.164 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:02.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:02.205+0000 7f29e72ad780 1 bucket sync caught up with source: 2024-04-19T23:44:02.204 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000003.54.7, , ] 2024-04-19T23:44:02.204 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000003.54.7, , ] 2024-04-19T23:44:02.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-19T23:44:02.205+0000 7f29e72ad780 0 bucket checkpoint complete 2024-04-19T23:44:02.220 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone3 target=test-zone2 bucket=aktlji-29 2024-04-19T23:44:02.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:02.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-19T23:44:02.220 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-29 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-19T23:44:02.287 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:02.287 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:02.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:02.332+0000 7f7b6081f780 1 bucket sync caught up with source: 2024-04-19T23:44:02.331 INFO:teuthology.orchestra.run.smithi031.stderr: local status: [, , , , , , , , 00000000003.54.7, , ] 2024-04-19T23:44:02.331 INFO:teuthology.orchestra.run.smithi031.stderr: remote markers: [, , , , , , , , 00000000003.54.7, , ] 2024-04-19T23:44:02.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:02.332+0000 7f7b6081f780 0 bucket checkpoint complete 2024-04-19T23:44:02.347 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-29 zones={test-zone2, test-zone1} 2024-04-19T23:44:02.347 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.347 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.348 DEBUG:boto:Method: HEAD 2024-04-19T23:44:02.348 DEBUG:boto:Path: /aktlji-29/ 2024-04-19T23:44:02.348 DEBUG:boto:Data: 2024-04-19T23:44:02.348 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.348 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.348 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.348 DEBUG:boto:Params: {} 2024-04-19T23:44:02.348 DEBUG:boto:Token: None 2024-04-19T23:44:02.348 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/ 2024-04-19T23:44:02.348 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MlaQHXJg49L3M8SDCE5kILrTC4I= 2024-04-19T23:44:02.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MlaQHXJg49L3M8SDCE5kILrTC4I=', 'Content-Length': '0'} 2024-04-19T23:44:02.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016c7d56457f67591-00662301c2-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.351 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.351 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.351 DEBUG:boto:Method: HEAD 2024-04-19T23:44:02.352 DEBUG:boto:Path: /aktlji-29/ 2024-04-19T23:44:02.352 DEBUG:boto:Data: 2024-04-19T23:44:02.352 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.352 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.352 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.352 DEBUG:boto:Params: {} 2024-04-19T23:44:02.352 DEBUG:boto:Token: None 2024-04-19T23:44:02.352 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/ 2024-04-19T23:44:02.352 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MlaQHXJg49L3M8SDCE5kILrTC4I= 2024-04-19T23:44:02.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MlaQHXJg49L3M8SDCE5kILrTC4I=', 'Content-Length': '0'} 2024-04-19T23:44:02.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000396a602f07f898c6-00662301c2-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.356 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:44:02.356 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.356 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.356 DEBUG:boto:path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.356 DEBUG:boto:auth_path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.357 DEBUG:boto:Method: GET 2024-04-19T23:44:02.357 DEBUG:boto:Path: /aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.357 DEBUG:boto:Data: 2024-04-19T23:44:02.357 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.357 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.358 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.358 DEBUG:boto:Params: {} 2024-04-19T23:44:02.358 DEBUG:boto:Token: None 2024-04-19T23:44:02.358 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/?versions 2024-04-19T23:44:02.359 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU= 2024-04-19T23:44:02.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU=', 'Content-Length': '0'} 2024-04-19T23:44:02.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ca573ed522659c6-00662301c2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.361 DEBUG:boto:b'aktlji-29999falseobjnulltrue2024-04-19T23:42:49.664Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:02.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:02.362 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:44:02.362 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.362 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.362 DEBUG:boto:path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.363 DEBUG:boto:auth_path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.363 DEBUG:boto:Method: GET 2024-04-19T23:44:02.363 DEBUG:boto:Path: /aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.363 DEBUG:boto:Data: 2024-04-19T23:44:02.363 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.363 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.363 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.363 DEBUG:boto:Params: {} 2024-04-19T23:44:02.363 DEBUG:boto:Token: None 2024-04-19T23:44:02.363 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/?versions 2024-04-19T23:44:02.363 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU= 2024-04-19T23:44:02.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU=', 'Content-Length': '0'} 2024-04-19T23:44:02.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d91a9c4b5fb5256-00662301c2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.367 DEBUG:boto:b'aktlji-29999falseobjnulltrue2024-04-19T23:42:49.664Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:02.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:02.367 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.367 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.367 DEBUG:boto:path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.368 DEBUG:boto:auth_path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.368 DEBUG:boto:Method: GET 2024-04-19T23:44:02.368 DEBUG:boto:Path: /aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.368 DEBUG:boto:Data: 2024-04-19T23:44:02.368 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.368 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.368 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.368 DEBUG:boto:Params: {} 2024-04-19T23:44:02.368 DEBUG:boto:Token: None 2024-04-19T23:44:02.368 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/?versions 2024-04-19T23:44:02.369 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU= 2024-04-19T23:44:02.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU=', 'Content-Length': '0'} 2024-04-19T23:44:02.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004345da78b0fe1bee-00662301c2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.371 DEBUG:boto:b'aktlji-29999falseobjnulltrue2024-04-19T23:42:49.664Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:02.371 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.371 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.371 DEBUG:boto:path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.372 DEBUG:boto:auth_path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.372 DEBUG:boto:Method: GET 2024-04-19T23:44:02.372 DEBUG:boto:Path: /aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.372 DEBUG:boto:Data: 2024-04-19T23:44:02.372 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.372 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.372 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.372 DEBUG:boto:Params: {} 2024-04-19T23:44:02.372 DEBUG:boto:Token: None 2024-04-19T23:44:02.372 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/?versions 2024-04-19T23:44:02.372 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU= 2024-04-19T23:44:02.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU=', 'Content-Length': '0'} 2024-04-19T23:44:02.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d50ac821198a278d-00662301c2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.375 DEBUG:boto:b'aktlji-29999falseobjnulltrue2024-04-19T23:42:49.664Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:02.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:44:02.375 DEBUG:boto:path=/aktlji-29/obj 2024-04-19T23:44:02.375 DEBUG:boto:auth_path=/aktlji-29/obj 2024-04-19T23:44:02.376 DEBUG:boto:Method: HEAD 2024-04-19T23:44:02.376 DEBUG:boto:Path: /aktlji-29/obj 2024-04-19T23:44:02.376 DEBUG:boto:Data: 2024-04-19T23:44:02.376 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.376 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.376 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.376 DEBUG:boto:Params: {} 2024-04-19T23:44:02.376 DEBUG:boto:Token: None 2024-04-19T23:44:02.376 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/obj 2024-04-19T23:44:02.376 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cWtFsqGLK60f20dI6Q5p/GMnN2A= 2024-04-19T23:44:02.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cWtFsqGLK60f20dI6Q5p/GMnN2A=', 'Content-Length': '0'} 2024-04-19T23:44:02.378 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000f3bd90f8f5ff1f20-00662301c2-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.378 DEBUG:boto:path=/aktlji-29/obj 2024-04-19T23:44:02.378 DEBUG:boto:auth_path=/aktlji-29/obj 2024-04-19T23:44:02.379 DEBUG:boto:Method: HEAD 2024-04-19T23:44:02.379 DEBUG:boto:Path: /aktlji-29/obj 2024-04-19T23:44:02.379 DEBUG:boto:Data: 2024-04-19T23:44:02.379 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.379 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.379 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.379 DEBUG:boto:Params: {} 2024-04-19T23:44:02.379 DEBUG:boto:Token: None 2024-04-19T23:44:02.379 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/obj 2024-04-19T23:44:02.379 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cWtFsqGLK60f20dI6Q5p/GMnN2A= 2024-04-19T23:44:02.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cWtFsqGLK60f20dI6Q5p/GMnN2A=', 'Content-Length': '0'} 2024-04-19T23:44:02.381 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006bb3ec9ab4172ae0-00662301c2-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.381 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-29 zones={test-zone2, test-zone1} 2024-04-19T23:44:02.382 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-29 zones={test-zone3, test-zone1} 2024-04-19T23:44:02.382 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.382 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.382 DEBUG:boto:Method: HEAD 2024-04-19T23:44:02.382 DEBUG:boto:Path: /aktlji-29/ 2024-04-19T23:44:02.382 DEBUG:boto:Data: 2024-04-19T23:44:02.382 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.382 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.382 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.383 DEBUG:boto:Params: {} 2024-04-19T23:44:02.383 DEBUG:boto:Token: None 2024-04-19T23:44:02.383 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/ 2024-04-19T23:44:02.383 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MlaQHXJg49L3M8SDCE5kILrTC4I= 2024-04-19T23:44:02.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MlaQHXJg49L3M8SDCE5kILrTC4I=', 'Content-Length': '0'} 2024-04-19T23:44:02.386 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a1abfdb1beadf81-00662301c2-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.386 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.386 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.386 DEBUG:boto:Method: HEAD 2024-04-19T23:44:02.386 DEBUG:boto:Path: /aktlji-29/ 2024-04-19T23:44:02.386 DEBUG:boto:Data: 2024-04-19T23:44:02.386 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.386 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.387 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.387 DEBUG:boto:Params: {} 2024-04-19T23:44:02.387 DEBUG:boto:Token: None 2024-04-19T23:44:02.387 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/ 2024-04-19T23:44:02.387 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MlaQHXJg49L3M8SDCE5kILrTC4I= 2024-04-19T23:44:02.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MlaQHXJg49L3M8SDCE5kILrTC4I=', 'Content-Length': '0'} 2024-04-19T23:44:02.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d171e0c3d0f2abfb-00662301c2-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:44:02.390 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.390 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.390 DEBUG:boto:path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.390 DEBUG:boto:auth_path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.390 DEBUG:boto:Method: GET 2024-04-19T23:44:02.390 DEBUG:boto:Path: /aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.390 DEBUG:boto:Data: 2024-04-19T23:44:02.390 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.391 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.391 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.391 DEBUG:boto:Params: {} 2024-04-19T23:44:02.391 DEBUG:boto:Token: None 2024-04-19T23:44:02.391 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/?versions 2024-04-19T23:44:02.391 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU= 2024-04-19T23:44:02.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU=', 'Content-Length': '0'} 2024-04-19T23:44:02.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041494b11609aaf79-00662301c2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.394 DEBUG:boto:b'aktlji-29999falseobjnulltrue2024-04-19T23:42:49.664Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:02.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:02.394 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:44:02.394 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.394 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.395 DEBUG:boto:path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.395 DEBUG:boto:auth_path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.395 DEBUG:boto:Method: GET 2024-04-19T23:44:02.395 DEBUG:boto:Path: /aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.395 DEBUG:boto:Data: 2024-04-19T23:44:02.395 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.395 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.396 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.396 DEBUG:boto:Params: {} 2024-04-19T23:44:02.396 DEBUG:boto:Token: None 2024-04-19T23:44:02.396 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/?versions 2024-04-19T23:44:02.396 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU= 2024-04-19T23:44:02.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU=', 'Content-Length': '0'} 2024-04-19T23:44:02.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c564a66a9043f53f-00662301c2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.442 DEBUG:boto:b'aktlji-29999falseobjnulltrue2024-04-19T23:42:49.664Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:02.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:02.443 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.443 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.443 DEBUG:boto:path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.443 DEBUG:boto:auth_path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.443 DEBUG:boto:Method: GET 2024-04-19T23:44:02.443 DEBUG:boto:Path: /aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.444 DEBUG:boto:Data: 2024-04-19T23:44:02.444 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.444 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.444 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.444 DEBUG:boto:Params: {} 2024-04-19T23:44:02.444 DEBUG:boto:Token: None 2024-04-19T23:44:02.444 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/?versions 2024-04-19T23:44:02.444 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU= 2024-04-19T23:44:02.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU=', 'Content-Length': '0'} 2024-04-19T23:44:02.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001409a7acbb985536-00662301c2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.447 DEBUG:boto:b'aktlji-29999falseobjnulltrue2024-04-19T23:42:49.664Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:02.447 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.447 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.448 DEBUG:boto:path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.448 DEBUG:boto:auth_path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.448 DEBUG:boto:Method: GET 2024-04-19T23:44:02.448 DEBUG:boto:Path: /aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.448 DEBUG:boto:Data: 2024-04-19T23:44:02.448 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.448 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.448 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.448 DEBUG:boto:Params: {} 2024-04-19T23:44:02.448 DEBUG:boto:Token: None 2024-04-19T23:44:02.448 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/?versions 2024-04-19T23:44:02.449 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU= 2024-04-19T23:44:02.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU=', 'Content-Length': '0'} 2024-04-19T23:44:02.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094f2033e65a04fe3-00662301c2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.451 DEBUG:boto:b'aktlji-29999falseobjnulltrue2024-04-19T23:42:49.664Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:02.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:44:02.452 DEBUG:boto:path=/aktlji-29/obj 2024-04-19T23:44:02.452 DEBUG:boto:auth_path=/aktlji-29/obj 2024-04-19T23:44:02.452 DEBUG:boto:Method: HEAD 2024-04-19T23:44:02.452 DEBUG:boto:Path: /aktlji-29/obj 2024-04-19T23:44:02.452 DEBUG:boto:Data: 2024-04-19T23:44:02.452 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.452 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.453 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.453 DEBUG:boto:Params: {} 2024-04-19T23:44:02.453 DEBUG:boto:Token: None 2024-04-19T23:44:02.453 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/obj 2024-04-19T23:44:02.453 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cWtFsqGLK60f20dI6Q5p/GMnN2A= 2024-04-19T23:44:02.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cWtFsqGLK60f20dI6Q5p/GMnN2A=', 'Content-Length': '0'} 2024-04-19T23:44:02.454 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000624f57a284c9298e-00662301c2-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.455 DEBUG:boto:path=/aktlji-29/obj 2024-04-19T23:44:02.455 DEBUG:boto:auth_path=/aktlji-29/obj 2024-04-19T23:44:02.455 DEBUG:boto:Method: HEAD 2024-04-19T23:44:02.455 DEBUG:boto:Path: /aktlji-29/obj 2024-04-19T23:44:02.455 DEBUG:boto:Data: 2024-04-19T23:44:02.455 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.455 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.455 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.455 DEBUG:boto:Params: {} 2024-04-19T23:44:02.456 DEBUG:boto:Token: None 2024-04-19T23:44:02.456 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/obj 2024-04-19T23:44:02.456 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cWtFsqGLK60f20dI6Q5p/GMnN2A= 2024-04-19T23:44:02.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cWtFsqGLK60f20dI6Q5p/GMnN2A=', 'Content-Length': '0'} 2024-04-19T23:44:02.458 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000cdb64e02c0a137b7-00662301c2-1100-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.458 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-29 zones={test-zone3, test-zone1} 2024-04-19T23:44:02.458 INFO:tasks.rgw_multi.zone_rados:comparing bucket=aktlji-29 zones={test-zone3, test-zone2} 2024-04-19T23:44:02.458 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.458 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.458 DEBUG:boto:Method: HEAD 2024-04-19T23:44:02.459 DEBUG:boto:Path: /aktlji-29/ 2024-04-19T23:44:02.459 DEBUG:boto:Data: 2024-04-19T23:44:02.459 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.459 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.459 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.459 DEBUG:boto:Params: {} 2024-04-19T23:44:02.459 DEBUG:boto:Token: None 2024-04-19T23:44:02.459 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/ 2024-04-19T23:44:02.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MlaQHXJg49L3M8SDCE5kILrTC4I= 2024-04-19T23:44:02.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MlaQHXJg49L3M8SDCE5kILrTC4I=', 'Content-Length': '0'} 2024-04-19T23:44:02.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006261815bd562e41a-00662301c2-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.462 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.462 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.463 DEBUG:boto:Method: HEAD 2024-04-19T23:44:02.463 DEBUG:boto:Path: /aktlji-29/ 2024-04-19T23:44:02.463 DEBUG:boto:Data: 2024-04-19T23:44:02.463 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.463 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.463 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.463 DEBUG:boto:Params: {} 2024-04-19T23:44:02.463 DEBUG:boto:Token: None 2024-04-19T23:44:02.463 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/ 2024-04-19T23:44:02.464 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MlaQHXJg49L3M8SDCE5kILrTC4I= 2024-04-19T23:44:02.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MlaQHXJg49L3M8SDCE5kILrTC4I=', 'Content-Length': '0'} 2024-04-19T23:44:02.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8abb49dbf11da49-00662301c2-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.466 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-19T23:44:02.466 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.466 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.466 DEBUG:boto:path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.466 DEBUG:boto:auth_path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.466 DEBUG:boto:Method: GET 2024-04-19T23:44:02.467 DEBUG:boto:Path: /aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.467 DEBUG:boto:Data: 2024-04-19T23:44:02.467 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.467 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.467 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.467 DEBUG:boto:Params: {} 2024-04-19T23:44:02.467 DEBUG:boto:Token: None 2024-04-19T23:44:02.467 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/?versions 2024-04-19T23:44:02.467 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU= 2024-04-19T23:44:02.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU=', 'Content-Length': '0'} 2024-04-19T23:44:02.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7137b10e39f157c-00662301c2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.470 DEBUG:boto:b'aktlji-29999falseobjnulltrue2024-04-19T23:42:49.664Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:02.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:02.471 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-19T23:44:02.471 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.471 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.471 DEBUG:boto:path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.471 DEBUG:boto:auth_path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.471 DEBUG:boto:Method: GET 2024-04-19T23:44:02.471 DEBUG:boto:Path: /aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.471 DEBUG:boto:Data: 2024-04-19T23:44:02.471 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.471 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.472 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.472 DEBUG:boto:Params: {} 2024-04-19T23:44:02.472 DEBUG:boto:Token: None 2024-04-19T23:44:02.472 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/?versions 2024-04-19T23:44:02.472 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU= 2024-04-19T23:44:02.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU=', 'Content-Length': '0'} 2024-04-19T23:44:02.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039fe62c970166987-00662301c2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.474 DEBUG:boto:b'aktlji-29999falseobjnulltrue2024-04-19T23:42:49.664Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:02.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-19T23:44:02.475 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.475 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.475 DEBUG:boto:path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.475 DEBUG:boto:auth_path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.475 DEBUG:boto:Method: GET 2024-04-19T23:44:02.475 DEBUG:boto:Path: /aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.475 DEBUG:boto:Data: 2024-04-19T23:44:02.475 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.476 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.476 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.476 DEBUG:boto:Params: {} 2024-04-19T23:44:02.476 DEBUG:boto:Token: None 2024-04-19T23:44:02.476 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/?versions 2024-04-19T23:44:02.476 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU= 2024-04-19T23:44:02.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU=', 'Content-Length': '0'} 2024-04-19T23:44:02.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075b377e95c3ee6f8-00662301c2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.479 DEBUG:boto:b'aktlji-29999falseobjnulltrue2024-04-19T23:42:49.664Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:02.479 DEBUG:boto:path=/aktlji-29/ 2024-04-19T23:44:02.479 DEBUG:boto:auth_path=/aktlji-29/ 2024-04-19T23:44:02.479 DEBUG:boto:path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.479 DEBUG:boto:auth_path=/aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.479 DEBUG:boto:Method: GET 2024-04-19T23:44:02.480 DEBUG:boto:Path: /aktlji-29/?versions&max-keys=999 2024-04-19T23:44:02.480 DEBUG:boto:Data: 2024-04-19T23:44:02.480 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.480 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.480 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.480 DEBUG:boto:Params: {} 2024-04-19T23:44:02.480 DEBUG:boto:Token: None 2024-04-19T23:44:02.480 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/?versions 2024-04-19T23:44:02.481 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU= 2024-04-19T23:44:02.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BUZMFFUhggod45bGyIQKH7E+CdU=', 'Content-Length': '0'} 2024-04-19T23:44:02.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beea7c862d83a366-00662301c2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.530 DEBUG:boto:b'aktlji-29999falseobjnulltrue2024-04-19T23:42:49.664Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-19T23:44:02.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-19T23:44:02.531 DEBUG:boto:path=/aktlji-29/obj 2024-04-19T23:44:02.531 DEBUG:boto:auth_path=/aktlji-29/obj 2024-04-19T23:44:02.531 DEBUG:boto:Method: HEAD 2024-04-19T23:44:02.531 DEBUG:boto:Path: /aktlji-29/obj 2024-04-19T23:44:02.531 DEBUG:boto:Data: 2024-04-19T23:44:02.532 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.532 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.532 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.532 DEBUG:boto:Params: {} 2024-04-19T23:44:02.532 DEBUG:boto:Token: None 2024-04-19T23:44:02.532 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/obj 2024-04-19T23:44:02.532 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cWtFsqGLK60f20dI6Q5p/GMnN2A= 2024-04-19T23:44:02.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cWtFsqGLK60f20dI6Q5p/GMnN2A=', 'Content-Length': '0'} 2024-04-19T23:44:02.534 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000005e3dc5d4da2d9d-00662301c2-1106-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.535 DEBUG:boto:path=/aktlji-29/obj 2024-04-19T23:44:02.535 DEBUG:boto:auth_path=/aktlji-29/obj 2024-04-19T23:44:02.535 DEBUG:boto:Method: HEAD 2024-04-19T23:44:02.535 DEBUG:boto:Path: /aktlji-29/obj 2024-04-19T23:44:02.535 DEBUG:boto:Data: 2024-04-19T23:44:02.535 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.535 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.536 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.536 DEBUG:boto:Params: {} 2024-04-19T23:44:02.536 DEBUG:boto:Token: None 2024-04-19T23:44:02.536 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:44:02 GMT /aktlji-29/obj 2024-04-19T23:44:02.536 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cWtFsqGLK60f20dI6Q5p/GMnN2A= 2024-04-19T23:44:02.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cWtFsqGLK60f20dI6Q5p/GMnN2A=', 'Content-Length': '0'} 2024-04-19T23:44:02.538 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d1b47a2578afe6ee-00662301c2-10d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.538 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=aktlji-29 zones={test-zone3, test-zone2} 2024-04-19T23:44:02.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2024-04-19T23:44:02.540 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-30 2024-04-19T23:44:02.540 DEBUG:boto:path=/aktlji-30/ 2024-04-19T23:44:02.540 DEBUG:boto:auth_path=/aktlji-30/ 2024-04-19T23:44:02.540 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.540 DEBUG:boto:Path: /aktlji-30/ 2024-04-19T23:44:02.540 DEBUG:boto:Data: 2024-04-19T23:44:02.540 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.540 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.540 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.540 DEBUG:boto:Params: {} 2024-04-19T23:44:02.541 DEBUG:boto:Token: None 2024-04-19T23:44:02.541 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT /aktlji-30/ 2024-04-19T23:44:02.541 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7D8GcnkjhV+fMuwTcoK4to+65QQ= 2024-04-19T23:44:02.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7D8GcnkjhV+fMuwTcoK4to+65QQ=', 'Content-Length': '0'} 2024-04-19T23:44:02.559 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000077e8a80e3741c2ee-00662301c2-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.559 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-31 2024-04-19T23:44:02.559 DEBUG:boto:path=/aktlji-31/ 2024-04-19T23:44:02.559 DEBUG:boto:auth_path=/aktlji-31/ 2024-04-19T23:44:02.560 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.560 DEBUG:boto:Path: /aktlji-31/ 2024-04-19T23:44:02.560 DEBUG:boto:Data: 2024-04-19T23:44:02.560 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.560 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.560 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.560 DEBUG:boto:Params: {} 2024-04-19T23:44:02.560 DEBUG:boto:Token: None 2024-04-19T23:44:02.560 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT /aktlji-31/ 2024-04-19T23:44:02.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qntGbYg8RCHYS9M/WPlafUcTiM0= 2024-04-19T23:44:02.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qntGbYg8RCHYS9M/WPlafUcTiM0=', 'Content-Length': '0'} 2024-04-19T23:44:02.592 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000051ef01b67cf83ce4-00662301c2-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.592 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-32 2024-04-19T23:44:02.592 DEBUG:boto:path=/aktlji-32/ 2024-04-19T23:44:02.593 DEBUG:boto:auth_path=/aktlji-32/ 2024-04-19T23:44:02.593 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.593 DEBUG:boto:Path: /aktlji-32/ 2024-04-19T23:44:02.593 DEBUG:boto:Data: 2024-04-19T23:44:02.593 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.593 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.594 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.594 DEBUG:boto:Params: {} 2024-04-19T23:44:02.594 DEBUG:boto:Token: None 2024-04-19T23:44:02.594 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT /aktlji-32/ 2024-04-19T23:44:02.594 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/8tHpwrkTVZvAFrCpggzqPInABY= 2024-04-19T23:44:02.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/8tHpwrkTVZvAFrCpggzqPInABY=', 'Content-Length': '0'} 2024-04-19T23:44:02.624 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008db0226a9688cf29-00662301c2-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.624 DEBUG:boto:path=/aktlji-30/ 2024-04-19T23:44:02.624 DEBUG:boto:auth_path=/aktlji-30/ 2024-04-19T23:44:02.624 DEBUG:boto:path=/aktlji-30/?versioning 2024-04-19T23:44:02.625 DEBUG:boto:auth_path=/aktlji-30/?versioning 2024-04-19T23:44:02.625 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.625 DEBUG:boto:Path: /aktlji-30/?versioning 2024-04-19T23:44:02.625 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:44:02.625 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.625 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.625 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.625 DEBUG:boto:Params: {} 2024-04-19T23:44:02.625 DEBUG:boto:Token: None 2024-04-19T23:44:02.625 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT /aktlji-30/?versioning 2024-04-19T23:44:02.626 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KCAigBYNvdeVrRpNKmr/7HS9uK0= 2024-04-19T23:44:02.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KCAigBYNvdeVrRpNKmr/7HS9uK0=', 'Content-Length': '228'} 2024-04-19T23:44:02.631 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004874b073a8076ccd-00662301c2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.631 DEBUG:boto:path=/aktlji-30/ 2024-04-19T23:44:02.632 DEBUG:boto:auth_path=/aktlji-30/ 2024-04-19T23:44:02.632 DEBUG:boto:path=/aktlji-30/?versioning 2024-04-19T23:44:02.632 DEBUG:boto:auth_path=/aktlji-30/?versioning 2024-04-19T23:44:02.632 DEBUG:boto:Method: GET 2024-04-19T23:44:02.632 DEBUG:boto:Path: /aktlji-30/?versioning 2024-04-19T23:44:02.632 DEBUG:boto:Data: 2024-04-19T23:44:02.632 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.632 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.632 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.632 DEBUG:boto:Params: {} 2024-04-19T23:44:02.632 DEBUG:boto:Token: None 2024-04-19T23:44:02.633 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-30/?versioning 2024-04-19T23:44:02.633 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:44wyeCGFQ0PRwiRczebGDH13dDY= 2024-04-19T23:44:02.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:44wyeCGFQ0PRwiRczebGDH13dDY=', 'Content-Length': '0'} 2024-04-19T23:44:02.634 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000378314381f40d38-00662301c2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.675 DEBUG:boto:EnabledDisabled 2024-04-19T23:44:02.676 DEBUG:boto:path=/aktlji-31/ 2024-04-19T23:44:02.676 DEBUG:boto:auth_path=/aktlji-31/ 2024-04-19T23:44:02.676 DEBUG:boto:path=/aktlji-31/?versioning 2024-04-19T23:44:02.676 DEBUG:boto:auth_path=/aktlji-31/?versioning 2024-04-19T23:44:02.676 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.676 DEBUG:boto:Path: /aktlji-31/?versioning 2024-04-19T23:44:02.676 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:44:02.676 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.676 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.676 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.676 DEBUG:boto:Params: {} 2024-04-19T23:44:02.677 DEBUG:boto:Token: None 2024-04-19T23:44:02.677 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT /aktlji-31/?versioning 2024-04-19T23:44:02.677 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:79gXr3copyTUDMAuNX/EtdUZDVw= 2024-04-19T23:44:02.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:79gXr3copyTUDMAuNX/EtdUZDVw=', 'Content-Length': '228'} 2024-04-19T23:44:02.689 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000577148713ed26599-00662301c2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.689 DEBUG:boto:path=/aktlji-31/ 2024-04-19T23:44:02.689 DEBUG:boto:auth_path=/aktlji-31/ 2024-04-19T23:44:02.690 DEBUG:boto:path=/aktlji-31/?versioning 2024-04-19T23:44:02.690 DEBUG:boto:auth_path=/aktlji-31/?versioning 2024-04-19T23:44:02.690 DEBUG:boto:Method: GET 2024-04-19T23:44:02.690 DEBUG:boto:Path: /aktlji-31/?versioning 2024-04-19T23:44:02.690 DEBUG:boto:Data: 2024-04-19T23:44:02.690 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.690 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.690 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.690 DEBUG:boto:Params: {} 2024-04-19T23:44:02.690 DEBUG:boto:Token: None 2024-04-19T23:44:02.691 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-31/?versioning 2024-04-19T23:44:02.691 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GYsClQ84jMqUe4x8FOM4UnH8ZHs= 2024-04-19T23:44:02.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GYsClQ84jMqUe4x8FOM4UnH8ZHs=', 'Content-Length': '0'} 2024-04-19T23:44:02.692 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000469b78e777d0b870-00662301c2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.692 DEBUG:boto:EnabledDisabled 2024-04-19T23:44:02.692 DEBUG:boto:path=/aktlji-32/ 2024-04-19T23:44:02.692 DEBUG:boto:auth_path=/aktlji-32/ 2024-04-19T23:44:02.692 DEBUG:boto:path=/aktlji-32/?versioning 2024-04-19T23:44:02.692 DEBUG:boto:auth_path=/aktlji-32/?versioning 2024-04-19T23:44:02.693 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.693 DEBUG:boto:Path: /aktlji-32/?versioning 2024-04-19T23:44:02.693 DEBUG:boto:Data: Enabled Disabled 2024-04-19T23:44:02.693 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.693 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.693 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.693 DEBUG:boto:Params: {} 2024-04-19T23:44:02.693 DEBUG:boto:Token: None 2024-04-19T23:44:02.693 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT /aktlji-32/?versioning 2024-04-19T23:44:02.694 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TOkNtdt5HYx+LS9N+h+B3RhNnQU= 2024-04-19T23:44:02.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TOkNtdt5HYx+LS9N+h+B3RhNnQU=', 'Content-Length': '228'} 2024-04-19T23:44:02.702 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e9862edf2ac96b21-00662301c2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.702 DEBUG:boto:path=/aktlji-32/ 2024-04-19T23:44:02.703 DEBUG:boto:auth_path=/aktlji-32/ 2024-04-19T23:44:02.703 DEBUG:boto:path=/aktlji-32/?versioning 2024-04-19T23:44:02.703 DEBUG:boto:auth_path=/aktlji-32/?versioning 2024-04-19T23:44:02.703 DEBUG:boto:Method: GET 2024-04-19T23:44:02.703 DEBUG:boto:Path: /aktlji-32/?versioning 2024-04-19T23:44:02.703 DEBUG:boto:Data: 2024-04-19T23:44:02.703 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.703 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.703 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.703 DEBUG:boto:Params: {} 2024-04-19T23:44:02.703 DEBUG:boto:Token: None 2024-04-19T23:44:02.704 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-32/?versioning 2024-04-19T23:44:02.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Jl5bnUdnE91q946guUFIwZtB9EM= 2024-04-19T23:44:02.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Jl5bnUdnE91q946guUFIwZtB9EM=', 'Content-Length': '0'} 2024-04-19T23:44:02.705 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e3de8c46c044428-00662301c2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.705 DEBUG:boto:EnabledDisabled 2024-04-19T23:44:02.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-04-19T23:44:02.706 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-33 2024-04-19T23:44:02.706 DEBUG:boto:path=/aktlji-33/ 2024-04-19T23:44:02.706 DEBUG:boto:auth_path=/aktlji-33/ 2024-04-19T23:44:02.706 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.706 DEBUG:boto:Path: /aktlji-33/ 2024-04-19T23:44:02.706 DEBUG:boto:Data: 2024-04-19T23:44:02.707 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.707 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.707 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.707 DEBUG:boto:Params: {} 2024-04-19T23:44:02.707 DEBUG:boto:Token: None 2024-04-19T23:44:02.707 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT /aktlji-33/ 2024-04-19T23:44:02.707 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mTiBdSTTgO7FPJeB41WPcV0pmp8= 2024-04-19T23:44:02.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mTiBdSTTgO7FPJeB41WPcV0pmp8=', 'Content-Length': '0'} 2024-04-19T23:44:02.725 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b1e1c43772ade231-00662301c2-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.725 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-34 2024-04-19T23:44:02.725 DEBUG:boto:path=/aktlji-34/ 2024-04-19T23:44:02.725 DEBUG:boto:auth_path=/aktlji-34/ 2024-04-19T23:44:02.725 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.725 DEBUG:boto:Path: /aktlji-34/ 2024-04-19T23:44:02.726 DEBUG:boto:Data: 2024-04-19T23:44:02.726 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.726 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.726 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.726 DEBUG:boto:Params: {} 2024-04-19T23:44:02.726 DEBUG:boto:Token: None 2024-04-19T23:44:02.726 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT /aktlji-34/ 2024-04-19T23:44:02.726 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pCLTuWUprL/MdWlvGacn0OBaglg= 2024-04-19T23:44:02.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pCLTuWUprL/MdWlvGacn0OBaglg=', 'Content-Length': '0'} 2024-04-19T23:44:02.756 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e08bb02863891c05-00662301c2-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.756 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-35 2024-04-19T23:44:02.756 DEBUG:boto:path=/aktlji-35/ 2024-04-19T23:44:02.756 DEBUG:boto:auth_path=/aktlji-35/ 2024-04-19T23:44:02.757 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.757 DEBUG:boto:Path: /aktlji-35/ 2024-04-19T23:44:02.757 DEBUG:boto:Data: 2024-04-19T23:44:02.757 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.757 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.757 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.757 DEBUG:boto:Params: {} 2024-04-19T23:44:02.757 DEBUG:boto:Token: None 2024-04-19T23:44:02.757 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT /aktlji-35/ 2024-04-19T23:44:02.758 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FqYCNB6jcUKc1NMMyYdI+OnpWN4= 2024-04-19T23:44:02.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FqYCNB6jcUKc1NMMyYdI+OnpWN4=', 'Content-Length': '0'} 2024-04-19T23:44:02.785 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009a0be9711fe8f8dd-00662301c2-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.785 DEBUG:boto:path=/aktlji-33/ 2024-04-19T23:44:02.785 DEBUG:boto:auth_path=/aktlji-33/ 2024-04-19T23:44:02.785 DEBUG:boto:path=/aktlji-33/?acl 2024-04-19T23:44:02.785 DEBUG:boto:auth_path=/aktlji-33/?acl 2024-04-19T23:44:02.785 DEBUG:boto:Method: GET 2024-04-19T23:44:02.785 DEBUG:boto:Path: /aktlji-33/?acl 2024-04-19T23:44:02.786 DEBUG:boto:Data: 2024-04-19T23:44:02.786 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.786 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.786 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.786 DEBUG:boto:Params: {} 2024-04-19T23:44:02.786 DEBUG:boto:Token: None 2024-04-19T23:44:02.786 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-33/?acl 2024-04-19T23:44:02.786 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YpFtRl4sWifKllqcNNi7JNGDowM= 2024-04-19T23:44:02.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpFtRl4sWifKllqcNNi7JNGDowM=', 'Content-Length': '0'} 2024-04-19T23:44:02.788 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c00a71d35b473576-00662301c2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.789 DEBUG:boto:path=/aktlji-33/ 2024-04-19T23:44:02.789 DEBUG:boto:auth_path=/aktlji-33/ 2024-04-19T23:44:02.789 DEBUG:boto:path=/aktlji-33/?acl 2024-04-19T23:44:02.789 DEBUG:boto:auth_path=/aktlji-33/?acl 2024-04-19T23:44:02.789 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.789 DEBUG:boto:Path: /aktlji-33/?acl 2024-04-19T23:44:02.789 DEBUG:boto:Data: 2024-04-19T23:44:02.789 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-19T23:44:02.789 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.789 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.789 DEBUG:boto:Params: {} 2024-04-19T23:44:02.790 DEBUG:boto:Token: None 2024-04-19T23:44:02.790 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT x-amz-acl:public-read /aktlji-33/?acl 2024-04-19T23:44:02.790 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ivK6avJcwRBie1LmBxuf8cULFmk= 2024-04-19T23:44:02.790 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ivK6avJcwRBie1LmBxuf8cULFmk=', 'Content-Length': '0'} 2024-04-19T23:44:02.795 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002503c0829e99167c-00662301c2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.796 DEBUG:boto:path=/aktlji-33/ 2024-04-19T23:44:02.796 DEBUG:boto:auth_path=/aktlji-33/ 2024-04-19T23:44:02.796 DEBUG:boto:path=/aktlji-33/?acl 2024-04-19T23:44:02.796 DEBUG:boto:auth_path=/aktlji-33/?acl 2024-04-19T23:44:02.796 DEBUG:boto:Method: GET 2024-04-19T23:44:02.796 DEBUG:boto:Path: /aktlji-33/?acl 2024-04-19T23:44:02.796 DEBUG:boto:Data: 2024-04-19T23:44:02.796 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.796 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.797 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.797 DEBUG:boto:Params: {} 2024-04-19T23:44:02.797 DEBUG:boto:Token: None 2024-04-19T23:44:02.797 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-33/?acl 2024-04-19T23:44:02.797 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:YpFtRl4sWifKllqcNNi7JNGDowM= 2024-04-19T23:44:02.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpFtRl4sWifKllqcNNi7JNGDowM=', 'Content-Length': '0'} 2024-04-19T23:44:02.798 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007da18bb57b320c12-00662301c2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.798 DEBUG:boto:path=/aktlji-34/ 2024-04-19T23:44:02.799 DEBUG:boto:auth_path=/aktlji-34/ 2024-04-19T23:44:02.799 DEBUG:boto:path=/aktlji-34/?acl 2024-04-19T23:44:02.799 DEBUG:boto:auth_path=/aktlji-34/?acl 2024-04-19T23:44:02.799 DEBUG:boto:Method: GET 2024-04-19T23:44:02.799 DEBUG:boto:Path: /aktlji-34/?acl 2024-04-19T23:44:02.799 DEBUG:boto:Data: 2024-04-19T23:44:02.799 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.799 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.799 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.800 DEBUG:boto:Params: {} 2024-04-19T23:44:02.800 DEBUG:boto:Token: None 2024-04-19T23:44:02.800 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-34/?acl 2024-04-19T23:44:02.800 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bAwbTkawejvqmJ0pyciCZNjBwXc= 2024-04-19T23:44:02.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bAwbTkawejvqmJ0pyciCZNjBwXc=', 'Content-Length': '0'} 2024-04-19T23:44:02.802 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f75a111316c97858-00662301c2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.802 DEBUG:boto:path=/aktlji-34/ 2024-04-19T23:44:02.802 DEBUG:boto:auth_path=/aktlji-34/ 2024-04-19T23:44:02.802 DEBUG:boto:path=/aktlji-34/?acl 2024-04-19T23:44:02.802 DEBUG:boto:auth_path=/aktlji-34/?acl 2024-04-19T23:44:02.802 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.803 DEBUG:boto:Path: /aktlji-34/?acl 2024-04-19T23:44:02.803 DEBUG:boto:Data: 2024-04-19T23:44:02.803 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-19T23:44:02.803 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.803 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.803 DEBUG:boto:Params: {} 2024-04-19T23:44:02.803 DEBUG:boto:Token: None 2024-04-19T23:44:02.803 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT x-amz-acl:public-read /aktlji-34/?acl 2024-04-19T23:44:02.803 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gF3uezpfDu81XWXZea4G6K0MpkA= 2024-04-19T23:44:02.804 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gF3uezpfDu81XWXZea4G6K0MpkA=', 'Content-Length': '0'} 2024-04-19T23:44:02.813 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fdc94aafdd6b7d54-00662301c2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.813 DEBUG:boto:path=/aktlji-34/ 2024-04-19T23:44:02.813 DEBUG:boto:auth_path=/aktlji-34/ 2024-04-19T23:44:02.813 DEBUG:boto:path=/aktlji-34/?acl 2024-04-19T23:44:02.814 DEBUG:boto:auth_path=/aktlji-34/?acl 2024-04-19T23:44:02.814 DEBUG:boto:Method: GET 2024-04-19T23:44:02.814 DEBUG:boto:Path: /aktlji-34/?acl 2024-04-19T23:44:02.814 DEBUG:boto:Data: 2024-04-19T23:44:02.814 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.814 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.814 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.814 DEBUG:boto:Params: {} 2024-04-19T23:44:02.814 DEBUG:boto:Token: None 2024-04-19T23:44:02.814 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-34/?acl 2024-04-19T23:44:02.815 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bAwbTkawejvqmJ0pyciCZNjBwXc= 2024-04-19T23:44:02.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bAwbTkawejvqmJ0pyciCZNjBwXc=', 'Content-Length': '0'} 2024-04-19T23:44:02.816 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b558396c10e5a46b-00662301c2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.858 DEBUG:boto:path=/aktlji-35/ 2024-04-19T23:44:02.859 DEBUG:boto:auth_path=/aktlji-35/ 2024-04-19T23:44:02.859 DEBUG:boto:path=/aktlji-35/?acl 2024-04-19T23:44:02.859 DEBUG:boto:auth_path=/aktlji-35/?acl 2024-04-19T23:44:02.859 DEBUG:boto:Method: GET 2024-04-19T23:44:02.859 DEBUG:boto:Path: /aktlji-35/?acl 2024-04-19T23:44:02.859 DEBUG:boto:Data: 2024-04-19T23:44:02.859 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.859 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.859 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.859 DEBUG:boto:Params: {} 2024-04-19T23:44:02.859 DEBUG:boto:Token: None 2024-04-19T23:44:02.860 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-35/?acl 2024-04-19T23:44:02.860 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bWRKS2hULsl6+/WeyJ0Nz88DxxQ= 2024-04-19T23:44:02.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bWRKS2hULsl6+/WeyJ0Nz88DxxQ=', 'Content-Length': '0'} 2024-04-19T23:44:02.861 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002aabbd12b765cde9-00662301c2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.861 DEBUG:boto:path=/aktlji-35/ 2024-04-19T23:44:02.862 DEBUG:boto:auth_path=/aktlji-35/ 2024-04-19T23:44:02.862 DEBUG:boto:path=/aktlji-35/?acl 2024-04-19T23:44:02.862 DEBUG:boto:auth_path=/aktlji-35/?acl 2024-04-19T23:44:02.862 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.862 DEBUG:boto:Path: /aktlji-35/?acl 2024-04-19T23:44:02.862 DEBUG:boto:Data: 2024-04-19T23:44:02.862 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-19T23:44:02.862 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.862 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.862 DEBUG:boto:Params: {} 2024-04-19T23:44:02.862 DEBUG:boto:Token: None 2024-04-19T23:44:02.863 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT x-amz-acl:public-read /aktlji-35/?acl 2024-04-19T23:44:02.863 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FXOX/uxqpDtawo7prLzfgpvUA4I= 2024-04-19T23:44:02.863 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FXOX/uxqpDtawo7prLzfgpvUA4I=', 'Content-Length': '0'} 2024-04-19T23:44:02.880 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c8e2434fd9dc8205-00662301c2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.880 DEBUG:boto:path=/aktlji-35/ 2024-04-19T23:44:02.880 DEBUG:boto:auth_path=/aktlji-35/ 2024-04-19T23:44:02.880 DEBUG:boto:path=/aktlji-35/?acl 2024-04-19T23:44:02.880 DEBUG:boto:auth_path=/aktlji-35/?acl 2024-04-19T23:44:02.880 DEBUG:boto:Method: GET 2024-04-19T23:44:02.880 DEBUG:boto:Path: /aktlji-35/?acl 2024-04-19T23:44:02.881 DEBUG:boto:Data: 2024-04-19T23:44:02.881 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.881 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.881 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.881 DEBUG:boto:Params: {} 2024-04-19T23:44:02.881 DEBUG:boto:Token: None 2024-04-19T23:44:02.881 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-35/?acl 2024-04-19T23:44:02.881 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bWRKS2hULsl6+/WeyJ0Nz88DxxQ= 2024-04-19T23:44:02.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bWRKS2hULsl6+/WeyJ0Nz88DxxQ=', 'Content-Length': '0'} 2024-04-19T23:44:02.883 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000036394de07f6b1f4f-00662301c2-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-04-19T23:44:02.883 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-36 2024-04-19T23:44:02.884 DEBUG:boto:path=/aktlji-36/ 2024-04-19T23:44:02.884 DEBUG:boto:auth_path=/aktlji-36/ 2024-04-19T23:44:02.884 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.884 DEBUG:boto:Path: /aktlji-36/ 2024-04-19T23:44:02.884 DEBUG:boto:Data: 2024-04-19T23:44:02.884 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.884 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.884 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.884 DEBUG:boto:Params: {} 2024-04-19T23:44:02.884 DEBUG:boto:Token: None 2024-04-19T23:44:02.885 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT /aktlji-36/ 2024-04-19T23:44:02.885 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ctvO3s1JBmJNkNCpD8IuF6CStVo= 2024-04-19T23:44:02.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ctvO3s1JBmJNkNCpD8IuF6CStVo=', 'Content-Length': '0'} 2024-04-19T23:44:02.914 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d9f15d5f82904c25-00662301c2-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.914 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-37 2024-04-19T23:44:02.915 DEBUG:boto:path=/aktlji-37/ 2024-04-19T23:44:02.915 DEBUG:boto:auth_path=/aktlji-37/ 2024-04-19T23:44:02.915 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.915 DEBUG:boto:Path: /aktlji-37/ 2024-04-19T23:44:02.915 DEBUG:boto:Data: 2024-04-19T23:44:02.915 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.915 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:02.915 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.915 DEBUG:boto:Params: {} 2024-04-19T23:44:02.915 DEBUG:boto:Token: None 2024-04-19T23:44:02.915 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT /aktlji-37/ 2024-04-19T23:44:02.916 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+nkswD7Oiei3mca428/y2+v+cQY= 2024-04-19T23:44:02.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+nkswD7Oiei3mca428/y2+v+cQY=', 'Content-Length': '0'} 2024-04-19T23:44:02.952 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000020ebfc994679f44a-00662301c2-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.952 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-38 2024-04-19T23:44:02.952 DEBUG:boto:path=/aktlji-38/ 2024-04-19T23:44:02.952 DEBUG:boto:auth_path=/aktlji-38/ 2024-04-19T23:44:02.952 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.952 DEBUG:boto:Path: /aktlji-38/ 2024-04-19T23:44:02.952 DEBUG:boto:Data: 2024-04-19T23:44:02.952 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.952 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:02.953 DEBUG:boto:Port: 8001 2024-04-19T23:44:02.953 DEBUG:boto:Params: {} 2024-04-19T23:44:02.953 DEBUG:boto:Token: None 2024-04-19T23:44:02.954 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:02 GMT /aktlji-38/ 2024-04-19T23:44:02.954 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iS2xV5Z/LxeeSmW8VlA6HV29xOU= 2024-04-19T23:44:02.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iS2xV5Z/LxeeSmW8VlA6HV29xOU=', 'Content-Length': '0'} 2024-04-19T23:44:02.981 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000016dae8d8a072084-00662301c2-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.982 DEBUG:boto:path=/aktlji-36/ 2024-04-19T23:44:02.982 DEBUG:boto:auth_path=/aktlji-36/ 2024-04-19T23:44:02.982 DEBUG:boto:path=/aktlji-36/?cors 2024-04-19T23:44:02.982 DEBUG:boto:auth_path=/aktlji-36/?cors 2024-04-19T23:44:02.982 DEBUG:boto:Method: PUT 2024-04-19T23:44:02.982 DEBUG:boto:Path: /aktlji-36/?cors 2024-04-19T23:44:02.983 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-19T23:44:02.983 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-19T23:44:02.983 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.983 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.983 DEBUG:boto:Params: {} 2024-04-19T23:44:02.983 DEBUG:boto:Token: None 2024-04-19T23:44:02.983 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 19 Apr 2024 23:44:02 GMT /aktlji-36/?cors 2024-04-19T23:44:02.983 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CR586D17rrIMYLfL/7jNrf5X3oU= 2024-04-19T23:44:02.983 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CR586D17rrIMYLfL/7jNrf5X3oU=', 'Content-Length': '218'} 2024-04-19T23:44:02.990 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000827a0a922ab21594-00662301c2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:02.990 DEBUG:boto:path=/aktlji-36/ 2024-04-19T23:44:02.990 DEBUG:boto:auth_path=/aktlji-36/ 2024-04-19T23:44:02.990 DEBUG:boto:path=/aktlji-36/?cors 2024-04-19T23:44:02.991 DEBUG:boto:auth_path=/aktlji-36/?cors 2024-04-19T23:44:02.991 DEBUG:boto:Method: GET 2024-04-19T23:44:02.991 DEBUG:boto:Path: /aktlji-36/?cors 2024-04-19T23:44:02.991 DEBUG:boto:Data: 2024-04-19T23:44:02.991 DEBUG:boto:Headers: {} 2024-04-19T23:44:02.992 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:02.992 DEBUG:boto:Port: 8000 2024-04-19T23:44:02.992 DEBUG:boto:Params: {} 2024-04-19T23:44:02.992 DEBUG:boto:Token: None 2024-04-19T23:44:02.992 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:02 GMT /aktlji-36/?cors 2024-04-19T23:44:02.993 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Xa2u5vGnTdgadUkGF6v3sylNzp4= 2024-04-19T23:44:02.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Xa2u5vGnTdgadUkGF6v3sylNzp4=', 'Content-Length': '0'} 2024-04-19T23:44:02.994 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005594fa2479125a6e-00662301c2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 19 Apr 2024 23:44:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:03.038 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-19T23:44:03.039 DEBUG:boto:path=/aktlji-37/ 2024-04-19T23:44:03.039 DEBUG:boto:auth_path=/aktlji-37/ 2024-04-19T23:44:03.039 DEBUG:boto:path=/aktlji-37/?cors 2024-04-19T23:44:03.039 DEBUG:boto:auth_path=/aktlji-37/?cors 2024-04-19T23:44:03.040 DEBUG:boto:Method: PUT 2024-04-19T23:44:03.040 DEBUG:boto:Path: /aktlji-37/?cors 2024-04-19T23:44:03.040 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-19T23:44:03.040 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-19T23:44:03.040 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:03.040 DEBUG:boto:Port: 8000 2024-04-19T23:44:03.040 DEBUG:boto:Params: {} 2024-04-19T23:44:03.040 DEBUG:boto:Token: None 2024-04-19T23:44:03.041 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 19 Apr 2024 23:44:03 GMT /aktlji-37/?cors 2024-04-19T23:44:03.041 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OC+LGvPhjszyeZsw7j+EQUS+Cwg= 2024-04-19T23:44:03.041 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:03 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OC+LGvPhjszyeZsw7j+EQUS+Cwg=', 'Content-Length': '218'} 2024-04-19T23:44:04.076 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000091fb1d0132704691-00662301c3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:04.076 DEBUG:boto:path=/aktlji-37/ 2024-04-19T23:44:04.076 DEBUG:boto:auth_path=/aktlji-37/ 2024-04-19T23:44:04.076 DEBUG:boto:path=/aktlji-37/?cors 2024-04-19T23:44:04.077 DEBUG:boto:auth_path=/aktlji-37/?cors 2024-04-19T23:44:04.077 DEBUG:boto:Method: GET 2024-04-19T23:44:04.077 DEBUG:boto:Path: /aktlji-37/?cors 2024-04-19T23:44:04.077 DEBUG:boto:Data: 2024-04-19T23:44:04.077 DEBUG:boto:Headers: {} 2024-04-19T23:44:04.077 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:04.077 DEBUG:boto:Port: 8000 2024-04-19T23:44:04.077 DEBUG:boto:Params: {} 2024-04-19T23:44:04.077 DEBUG:boto:Token: None 2024-04-19T23:44:04.078 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:04 GMT /aktlji-37/?cors 2024-04-19T23:44:04.078 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Hy+dqsg0PdQ1rkKw5bEpozmNTUo= 2024-04-19T23:44:04.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Hy+dqsg0PdQ1rkKw5bEpozmNTUo=', 'Content-Length': '0'} 2024-04-19T23:44:04.079 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000052f9d2bdd30f80a-00662301c4-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 19 Apr 2024 23:44:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:04.080 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-19T23:44:04.080 DEBUG:boto:path=/aktlji-38/ 2024-04-19T23:44:04.080 DEBUG:boto:auth_path=/aktlji-38/ 2024-04-19T23:44:04.080 DEBUG:boto:path=/aktlji-38/?cors 2024-04-19T23:44:04.080 DEBUG:boto:auth_path=/aktlji-38/?cors 2024-04-19T23:44:04.080 DEBUG:boto:Method: PUT 2024-04-19T23:44:04.081 DEBUG:boto:Path: /aktlji-38/?cors 2024-04-19T23:44:04.081 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-19T23:44:04.081 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-19T23:44:04.081 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:04.081 DEBUG:boto:Port: 8001 2024-04-19T23:44:04.081 DEBUG:boto:Params: {} 2024-04-19T23:44:04.081 DEBUG:boto:Token: None 2024-04-19T23:44:04.081 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 19 Apr 2024 23:44:04 GMT /aktlji-38/?cors 2024-04-19T23:44:04.081 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MpsuI1+a4wE91QjQGLpaxiwh5k8= 2024-04-19T23:44:04.082 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:04 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MpsuI1+a4wE91QjQGLpaxiwh5k8=', 'Content-Length': '218'} 2024-04-19T23:44:05.092 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fdf064bc6b0d6a99-00662301c4-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:05.093 DEBUG:boto:path=/aktlji-38/ 2024-04-19T23:44:05.093 DEBUG:boto:auth_path=/aktlji-38/ 2024-04-19T23:44:05.093 DEBUG:boto:path=/aktlji-38/?cors 2024-04-19T23:44:05.093 DEBUG:boto:auth_path=/aktlji-38/?cors 2024-04-19T23:44:05.093 DEBUG:boto:Method: GET 2024-04-19T23:44:05.093 DEBUG:boto:Path: /aktlji-38/?cors 2024-04-19T23:44:05.093 DEBUG:boto:Data: 2024-04-19T23:44:05.093 DEBUG:boto:Headers: {} 2024-04-19T23:44:05.093 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:05.094 DEBUG:boto:Port: 8001 2024-04-19T23:44:05.094 DEBUG:boto:Params: {} 2024-04-19T23:44:05.094 DEBUG:boto:Token: None 2024-04-19T23:44:05.094 DEBUG:boto:StringToSign: GET Fri, 19 Apr 2024 23:44:05 GMT /aktlji-38/?cors 2024-04-19T23:44:05.094 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:B4tTPYdk40hbupVugSZRr6xYR4Y= 2024-04-19T23:44:05.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B4tTPYdk40hbupVugSZRr6xYR4Y=', 'Content-Length': '0'} 2024-04-19T23:44:05.095 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000000321d3b3371e5e7-00662301c5-1106-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 19 Apr 2024 23:44:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:05.096 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-19T23:44:05.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-04-19T23:44:05.097 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-39 2024-04-19T23:44:05.097 DEBUG:boto:path=/aktlji-39/ 2024-04-19T23:44:05.097 DEBUG:boto:auth_path=/aktlji-39/ 2024-04-19T23:44:05.097 DEBUG:boto:Method: PUT 2024-04-19T23:44:05.097 DEBUG:boto:Path: /aktlji-39/ 2024-04-19T23:44:05.097 DEBUG:boto:Data: 2024-04-19T23:44:05.097 DEBUG:boto:Headers: {} 2024-04-19T23:44:05.097 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:44:05.097 DEBUG:boto:Port: 8000 2024-04-19T23:44:05.097 DEBUG:boto:Params: {} 2024-04-19T23:44:05.098 DEBUG:boto:Token: None 2024-04-19T23:44:05.098 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:05 GMT /aktlji-39/ 2024-04-19T23:44:05.098 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d8SN9GWgVBNEChRLFfHwoH/weZQ= 2024-04-19T23:44:05.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d8SN9GWgVBNEChRLFfHwoH/weZQ=', 'Content-Length': '0'} 2024-04-19T23:44:05.115 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aa8cda59e075e825-00662301c5-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:05.115 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-40 2024-04-19T23:44:05.115 DEBUG:boto:path=/aktlji-40/ 2024-04-19T23:44:05.115 DEBUG:boto:auth_path=/aktlji-40/ 2024-04-19T23:44:05.116 DEBUG:boto:Method: PUT 2024-04-19T23:44:05.116 DEBUG:boto:Path: /aktlji-40/ 2024-04-19T23:44:05.116 DEBUG:boto:Data: 2024-04-19T23:44:05.116 DEBUG:boto:Headers: {} 2024-04-19T23:44:05.116 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:44:05.116 DEBUG:boto:Port: 8000 2024-04-19T23:44:05.116 DEBUG:boto:Params: {} 2024-04-19T23:44:05.116 DEBUG:boto:Token: None 2024-04-19T23:44:05.116 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:05 GMT /aktlji-40/ 2024-04-19T23:44:05.117 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gPMzWRdUt2uvhWzlDWzfb1b45cY= 2024-04-19T23:44:05.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gPMzWRdUt2uvhWzlDWzfb1b45cY=', 'Content-Length': '0'} 2024-04-19T23:44:05.148 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004faa31999e953b2e-00662301c5-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:05.148 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-41 2024-04-19T23:44:05.149 DEBUG:boto:path=/aktlji-41/ 2024-04-19T23:44:05.149 DEBUG:boto:auth_path=/aktlji-41/ 2024-04-19T23:44:05.149 DEBUG:boto:Method: PUT 2024-04-19T23:44:05.149 DEBUG:boto:Path: /aktlji-41/ 2024-04-19T23:44:05.149 DEBUG:boto:Data: 2024-04-19T23:44:05.149 DEBUG:boto:Headers: {} 2024-04-19T23:44:05.149 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:44:05.149 DEBUG:boto:Port: 8001 2024-04-19T23:44:05.149 DEBUG:boto:Params: {} 2024-04-19T23:44:05.149 DEBUG:boto:Token: None 2024-04-19T23:44:05.150 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:44:05 GMT /aktlji-41/ 2024-04-19T23:44:05.150 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PaAvvMSr9lbfioqobDMMH3gj+Yo= 2024-04-19T23:44:05.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PaAvvMSr9lbfioqobDMMH3gj+Yo=', 'Content-Length': '0'} 2024-04-19T23:44:05.178 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dbcce448e20eca0c-00662301c5-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:44:05.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:05.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:05.179 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:44:05.257 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:05.257 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:05.325 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:44:05.340 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:44:05.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:44:05.340 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:44:05.340 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:44:05.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:44:05.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:44:05.340 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:44:05.340 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:44:05.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:44:05.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:44:05.341 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713570245.168285_359.1", 2024-04-19T23:44:05.341 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:44:05.168285Z" 2024-04-19T23:44:05.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:44:05.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:44:05.341 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:44:05.341 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:44:05.341 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:44:05.341 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:44:05.342 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:05.343 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:44:05.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:05.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:05.343 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:44:05.399 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:05.400 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:05.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:05.434+0000 7f21d3c02780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:05.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:05.435+0000 7f21d3c02780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:05.450 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:44:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:44:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:44:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:44:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:44:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:44:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:44:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:44:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:44:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:44:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:44:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:44:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:44:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:44:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:44:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:44:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:44:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:44:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:44:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:44:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:44:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:44:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:44:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:44:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:44:05Z" 2024-04-19T23:44:05.457 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:44:05.458 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:44:05Z" } 2024-04-19T23:44:05.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:44:05.458 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:05.458 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:05.458 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.554348_295.1 2024-04-19T23:44:10.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:10.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:10.460 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:44:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:10.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:10.554+0000 7fc4963de780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:10.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:10.555+0000 7fc4963de780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:10.569 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:44:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:44:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:44:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:44:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:44:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:44:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:44:10.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:10.571 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:44:10.571 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:10.571 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:10.571 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:44:10.571 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:10.571 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:44:10.571 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:10.571 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:44:10.571 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:10.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:10.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:10.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:10.572 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:44:10.572 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:10.572 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:10.572 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:44:10.572 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:10.572 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:10.572 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:10.572 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:44:10.572 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:10.572 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:10.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:10.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:10.573 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:44:10.573 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:10.573 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:10.573 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.609968_303.1", 2024-04-19T23:44:10.573 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:10.573 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:10.573 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:10.573 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-19T23:44:10.573 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:10.573 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:10.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:10.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:10.574 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:44:10.574 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:10.574 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:10.574 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:44:10.574 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:10.574 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:10.574 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:10.574 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:44:10.575 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:10.575 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:10.575 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:10.575 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:44:10.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:10.575 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:44:10.575 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:44:10.575 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:44:10.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:10.575 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:44:10Z" 2024-04-19T23:44:10.575 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:44:10.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.609968_303.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.609968Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:44:10Z" } 2024-04-19T23:44:10.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:44:10.577 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:10.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:10.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.609968_303.1 2024-04-19T23:44:14.513 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-19T23:44:14.513+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-19T23:44:15.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:15.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:15.579 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:44:15.673 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:15.673 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:15.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:15.723+0000 7f58e4fc9780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:15.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:15.724+0000 7f58e4fc9780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:15.738 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:44:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:44:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:44:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:44:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:44:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:44:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:44:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:15.738 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:44:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:44:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:44:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:44:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:15.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:44:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:44:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:44:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:15.740 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:15.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:15.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:15.741 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:44:15.741 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:15.741 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:15.741 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:44:15.741 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:15.741 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:15.741 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:15.741 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:44:15.741 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:15.742 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:15.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:15.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:15.742 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:44:15.742 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:15.742 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:15.742 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:44:15.742 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:15.742 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:15.742 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:15.742 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:44:15.743 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:15.743 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:15.743 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:15.743 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:44:15.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:15.743 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:44:15.743 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:44:15.743 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:44:15.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:15.743 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:44:15Z" 2024-04-19T23:44:15.743 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:44:15.744 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:44:15Z" } 2024-04-19T23:44:15.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:44:15.745 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:15.745 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:15.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-19T23:44:20.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:20.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:20.746 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:44:20.806 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:20.806 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:20.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:20.843+0000 7f3841b67780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:20.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:20.844+0000 7f3841b67780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:20.857 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:44:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:44:20.857 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:44:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:44:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:44:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:44:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:44:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:44:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:20.858 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:44:20.859 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:20.859 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:44:20.859 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:20.859 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:44:20.859 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:20.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:20.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:20.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:20.859 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:44:20.859 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:20.859 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:20.860 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:44:20.860 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:20.860 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:20.860 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:20.860 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:44:20.860 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:20.860 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:20.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:20.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:20.860 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:44:20.860 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:20.860 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:20.861 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:44:20.862 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:20.862 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:20.862 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:20.862 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:44:20.862 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:20.862 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:20.862 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:20.862 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:44:20.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:20.862 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:44:20.862 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:44:20.863 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:44:20.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:20.863 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:44:20Z" 2024-04-19T23:44:20.863 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:44:20.864 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:44:20Z" } 2024-04-19T23:44:20.864 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:44:20.864 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:20.864 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:20.864 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-19T23:44:25.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:25.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:25.866 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:44:25.961 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:25.962 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:25.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:25.997+0000 7f252c371780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:25.998+0000 7f252c371780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:26.013 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:44:26.013 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:44:26.013 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:44:26.013 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:44:26.013 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:44:26.013 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:44:26.013 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:26.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:26.013 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:44:26.013 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:26.014 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:44:26.014 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:26.014 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:26.014 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:44:26.014 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:26.014 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:44:26.014 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:26.014 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:44:26.014 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:26.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:26.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:26.015 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:26.015 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:44:26.015 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:26.015 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:26.015 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:44:26.015 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:26.015 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:26.015 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:26.016 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:44:26.016 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:26.016 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:26.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:26.016 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:26.016 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:44:26.016 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:26.016 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:26.016 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:44:26.016 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:26.017 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:26.018 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:26.018 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:44:26.018 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:26.018 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:26.018 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:26.018 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:44:26.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:26.018 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:44:26.018 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:44:26.018 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:44:26.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:26.019 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:44:26Z" 2024-04-19T23:44:26.019 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:44:26.019 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:44:26Z" } 2024-04-19T23:44:26.020 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:44:26.020 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:26.020 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:26.020 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.554348_295.1 2024-04-19T23:44:31.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:31.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:31.021 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:44:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:31.078 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:31.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:31.113+0000 7f624950c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:31.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:31.114+0000 7f624950c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:31.128 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:44:31.128 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:44:31.128 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:44:31.128 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:44:31.128 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:44:31.128 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:44:31.128 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:31.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:31.128 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:44:31.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:31.129 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:44:31.129 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:31.129 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:31.129 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:44:31.129 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:31.129 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:44:31.129 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:31.129 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:44:31.129 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:31.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:31.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:31.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:31.130 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:44:31.130 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:31.130 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:31.130 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:44:31.130 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:31.130 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:31.130 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:31.130 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:44:31.130 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:31.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:31.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:31.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:31.131 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:44:31.133 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:31.134 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:31.134 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.738086_316.1", 2024-04-19T23:44:31.134 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:31.134 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:31.134 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:31.135 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-19T23:44:31.135 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:31.135 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:31.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:31.135 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:31.135 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:44:31.135 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:31.135 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:31.135 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:44:31.136 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:31.136 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:31.136 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:31.136 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:44:31.136 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:31.136 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:31.136 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:31.136 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:44:31.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:31.136 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:44:31.136 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:44:31.137 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:44:31.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:31.137 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:44:31Z" 2024-04-19T23:44:31.137 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:44:31.138 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.738086_316.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.738086Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:44:31Z" } 2024-04-19T23:44:31.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:44:31.138 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:31.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:31.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.738086_316.1 2024-04-19T23:44:36.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:36.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:36.141 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:44:36.219 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:36.219 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:36.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:36.307+0000 7f45351eb780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:36.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:36.308+0000 7f45351eb780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:36.322 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:44:36.322 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:44:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:44:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:44:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:44:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:44:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:44:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:44:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:36.323 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:44:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:44:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:44:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:36.324 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:44:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:44:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:44:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:36.325 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:44:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:44:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:44:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:36.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:44:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:44:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:44:36.327 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:44:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:44:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:44:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:44:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:36.328 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:44:36Z" 2024-04-19T23:44:36.328 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:44:36.329 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:44:36Z" } 2024-04-19T23:44:36.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:44:36.330 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:36.330 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:36.330 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-19T23:44:41.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:41.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:41.331 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:44:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:41.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:41.426+0000 7f0f69542780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:41.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:41.428+0000 7f0f69542780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:41.442 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:44:41.442 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:44:41.442 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:44:41.442 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:44:41.442 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:44:41.442 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:44:41.442 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:41.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:41.442 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:44:41.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:41.443 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:44:41.443 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:41.443 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:41.443 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:44:41.443 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:41.443 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:44:41.443 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:41.443 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:44:41.443 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:41.443 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:41.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:41.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:41.444 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:44:41.444 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:41.444 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:41.444 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:44:41.444 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:41.444 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:41.444 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:41.444 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:44:41.444 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:41.444 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:41.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:41.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:41.445 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:44:41.445 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:41.445 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:41.445 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:44:41.445 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:41.445 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:41.445 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:41.445 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:44:41.445 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:41.445 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:41.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:41.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:41.446 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:44:41.446 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:41.446 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:41.446 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:44:41.446 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:41.446 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:41.446 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:41.446 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:44:41.446 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:41.446 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:41.447 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:41.447 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:44:41.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:41.447 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:44:41.447 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:44:41.447 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:44:41.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:41.447 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:44:41Z" 2024-04-19T23:44:41.447 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:44:41.448 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:44:41Z" } 2024-04-19T23:44:41.448 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:44:41.449 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:41.449 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:41.449 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-19T23:44:46.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:46.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:46.451 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:44:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:46.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:46.547+0000 7f25a18d4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:46.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:46.548+0000 7f25a18d4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:46.566 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:44:46.567 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:44:46.567 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:44:46.567 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:44:46.567 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:44:46.567 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:44:46.567 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:46.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:46.567 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:44:46.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:46.568 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:44:46.568 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:46.568 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:46.568 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:44:46.568 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:46.568 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:44:46.568 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:46.568 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:44:46.568 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:46.568 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:46.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:46.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:46.569 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:44:46.569 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:46.569 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:46.569 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:44:46.569 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:46.569 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:46.569 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:46.569 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:44:46.569 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:46.569 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:46.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:46.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:46.570 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:44:46.570 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:46.570 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:46.570 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:44:46.570 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:46.570 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:46.570 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:46.570 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:44:46.570 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:46.570 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:46.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:46.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:46.571 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:44:46.571 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:46.571 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:46.571 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:44:46.571 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:46.571 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:46.571 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:46.571 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:44:46.571 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:46.572 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:46.572 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:46.572 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:44:46.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:46.572 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:44:46.572 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:44:46.572 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:44:46.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:46.572 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:44:46Z" 2024-04-19T23:44:46.572 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:44:46.574 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:44:46Z" } 2024-04-19T23:44:46.574 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:44:46.574 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:46.574 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:46.574 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-19T23:44:51.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:51.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:51.576 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:44:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:51.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:51.675+0000 7fd0d03f2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:51.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:51.676+0000 7fd0d03f2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:51.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:51.691 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:51.692 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.577447_300.1", 2024-04-19T23:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:51.693 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-19T23:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:44:51.694 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:44:51.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:51.696 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:44:51Z" 2024-04-19T23:44:51.696 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:44:51.697 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.577447_300.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.577447Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:44:51Z" } 2024-04-19T23:44:51.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:44:51.697 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:51.697 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:51.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.577447_300.1 2024-04-19T23:44:56.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:56.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:44:56.699 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:44:56.777 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:44:56.777 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:44:56.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:56.877+0000 7f545bb59780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:44:56.878+0000 7f545bb59780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:44:56.893 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:44:56.893 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:44:56.893 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:44:56.893 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:44:56.893 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:44:56.893 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:44:56.893 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:56.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:56.894 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:44:56.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:56.894 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:44:56.894 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:56.894 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:56.894 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:44:56.894 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:56.894 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:44:56.894 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:44:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:44:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:44:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:56.896 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:56.896 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:56.896 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:44:56.896 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:56.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:56.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:56.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:56.896 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:44:56.896 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:56.896 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:56.897 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.572795_298.1", 2024-04-19T23:44:56.897 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:56.897 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:56.897 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:56.897 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-19T23:44:56.897 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:56.897 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:56.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:56.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:44:56.898 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:44:56.898 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:44:56.898 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:44:56.898 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:44:56.898 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:44:56.899 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:44:56.899 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:44:56.899 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:44:56.899 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:44:56.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:56.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:44:56.899 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:44:56.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:56.900 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:44:56.900 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:44:56.900 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:44:56.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:44:56.900 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:44:56Z" 2024-04-19T23:44:56.900 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:44:56.902 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.572795_298.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.572795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:44:56Z" } 2024-04-19T23:44:56.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:44:56.902 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:56.902 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:44:56.902 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.572795_298.1 2024-04-19T23:45:01.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:01.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:01.904 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:45:01.962 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:45:01.962 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:45:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:01.996+0000 7fc161f2e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:01.997+0000 7fc161f2e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:02.011 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:45:02.011 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:45:02.011 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:45:02.011 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:45:02.012 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:45:02.012 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:45:02.012 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:02.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:02.012 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:45:02.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:02.012 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:45:02.012 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:02.012 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:02.012 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:45:02.013 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:02.013 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:45:02.013 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:02.013 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:45:02.013 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:02.013 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:02.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:02.013 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:02.013 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:45:02.013 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:02.013 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:02.014 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:45:02.014 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:02.014 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:02.014 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:02.014 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:45:02.014 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:02.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:02.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:02.014 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:02.014 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:45:02.014 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:02.015 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:02.015 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.572795_298.1", 2024-04-19T23:45:02.015 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:02.015 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:02.015 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:02.015 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-19T23:45:02.015 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:02.015 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:02.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:02.015 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:02.015 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:45:02.015 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:02.016 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:02.016 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:45:02.016 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:02.016 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:02.016 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:02.016 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:45:02.016 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:02.016 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:02.016 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:02.016 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:45:02.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:02.017 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:45:02.017 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:45:02.017 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:45:02.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:02.017 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:45:01Z" 2024-04-19T23:45:02.017 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:45:02.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.572795_298.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.572795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:45:01Z" } 2024-04-19T23:45:02.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:45:02.018 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:02.018 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:02.019 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.572795_298.1 2024-04-19T23:45:07.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:07.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:07.020 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:45:07.079 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:45:07.079 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:45:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:07.114+0000 7fa3cca2d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:07.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:07.115+0000 7fa3cca2d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:07.129 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:45:07.129 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:45:07.129 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:45:07.129 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:45:07.129 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:45:07.129 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:45:07.129 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:07.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:07.130 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:45:07.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:07.130 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:45:07.130 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:07.130 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:07.130 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:45:07.130 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:07.130 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:45:07.130 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:07.130 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:45:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:45:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:45:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:07.131 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:45:07.132 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:07.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:07.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:07.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:07.132 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:45:07.132 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:07.132 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:07.132 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:45:07.132 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:07.132 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:07.132 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:07.133 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:45:07.133 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:07.133 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:07.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:07.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:07.133 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:45:07.133 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:07.133 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:07.133 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:45:07.133 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:07.133 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:07.133 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:07.134 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:45:07.134 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:07.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:07.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:07.134 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:45:07.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:07.134 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:45:07.134 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:45:07.134 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:45:07.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:07.134 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:45:07Z" 2024-04-19T23:45:07.135 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:45:07.135 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:45:07Z" } 2024-04-19T23:45:07.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:45:07.136 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:07.136 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:07.136 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.576049_299.1 2024-04-19T23:45:12.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:12.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:12.137 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:45:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:45:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:45:12.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:12.231+0000 7f7e2bf5a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:12.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:12.232+0000 7f7e2bf5a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:12.248 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:45:12.248 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:45:12.248 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:45:12.248 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:45:12.248 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:45:12.248 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:45:12.248 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:12.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:12.248 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:45:12.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:12.249 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:45:12.249 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:12.249 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:12.249 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:45:12.249 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:12.249 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:45:12.249 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:12.249 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:45:12.249 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:12.249 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:12.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:12.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:12.250 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:45:12.250 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:12.250 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:12.250 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:45:12.250 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:12.250 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:12.251 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:12.251 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:45:12.251 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:12.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:12.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:12.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:12.252 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:45:12.252 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:12.252 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:12.252 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.580723_301.1", 2024-04-19T23:45:12.252 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:12.252 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:12.252 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:12.252 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-19T23:45:12.252 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:12.253 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:12.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:12.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:12.253 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:45:12.253 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:12.254 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:12.254 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:45:12.254 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:12.254 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:12.255 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:12.255 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:45:12.255 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:12.255 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:12.255 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:12.255 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:45:12.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:12.256 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:45:12.256 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:45:12.256 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:45:12.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:12.256 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:45:12Z" 2024-04-19T23:45:12.256 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:45:12.257 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.580723_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.580723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:45:12Z" } 2024-04-19T23:45:12.257 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:45:12.258 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:12.258 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:12.258 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.580723_301.1 2024-04-19T23:45:17.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:17.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:17.259 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:45:17.318 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:45:17.319 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:45:17.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:17.355+0000 7f512f4c5780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:17.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:17.356+0000 7f512f4c5780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:17.370 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:45:17.370 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:45:17.371 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:45:17.371 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:45:17.371 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:45:17.371 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:45:17.371 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:17.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:17.371 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:45:17.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:17.371 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:45:17.372 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:17.372 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:17.372 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:45:17.372 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:17.372 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:45:17.372 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:17.372 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:45:17.372 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:17.372 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:17.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:17.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:17.373 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:45:17.373 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:17.373 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:17.373 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:45:17.374 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:17.374 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:17.374 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:17.374 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:45:17.374 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:17.375 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:17.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:17.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:17.375 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:45:17.375 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:17.376 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:17.376 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.555632_296.1", 2024-04-19T23:45:17.376 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:17.376 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:17.376 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:17.377 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-19T23:45:17.377 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:17.377 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:17.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:17.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:17.378 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:45:17.378 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:17.378 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:17.378 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:45:17.378 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:17.378 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:17.379 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:17.379 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:45:17.379 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:17.379 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:17.379 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:17.380 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:45:17.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:17.380 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:45:17.380 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:45:17.380 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:45:17.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:17.381 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:45:17Z" 2024-04-19T23:45:17.381 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:45:17.382 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.555632_296.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.555632Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:45:17Z" } 2024-04-19T23:45:17.383 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:45:17.383 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:17.383 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:17.383 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.555632_296.1 2024-04-19T23:45:22.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:22.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:22.385 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:45:22.443 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:45:22.444 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:45:22.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:22.479+0000 7f23e5659780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:22.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:22.480+0000 7f23e5659780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:22.494 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:45:22.495 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:45:22.495 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:45:22.495 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:45:22.495 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:45:22.495 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:45:22.495 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:22.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:22.495 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:45:22.495 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:22.495 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:45:22.496 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:22.496 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:22.496 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:45:22.496 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:22.496 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:45:22.496 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:22.496 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:45:22.496 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:22.496 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:22.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:22.496 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:22.497 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:45:22.497 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:22.497 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:22.497 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:45:22.497 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:22.497 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:22.497 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:22.497 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:45:22.497 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:22.497 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:22.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:22.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:22.498 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:45:22.498 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:22.498 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:22.498 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:45:22.498 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:22.498 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:22.498 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:22.498 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:45:22.498 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:45:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:45:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:22.499 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:45:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:45:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:45:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:45:22.500 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:45:22.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:22.501 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:45:22Z" 2024-04-19T23:45:22.501 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:45:22.502 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:45:22Z" } 2024-04-19T23:45:22.502 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:45:22.502 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:22.502 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:22.502 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.554348_295.1 2024-04-19T23:45:27.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:27.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:27.504 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:45:27.563 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:45:27.564 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:45:27.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:27.695+0000 7f3ad3e8d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:27.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:27.696+0000 7f3ad3e8d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:27.709 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:45:27.709 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:45:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:45:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:45:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:45:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:45:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:45:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:45:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:27.710 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:45:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:45:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:45:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:45:27.711 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:45:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:45:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:27.712 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.572795_298.1", 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:45:27.713 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:45:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:45:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:45:27.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:45:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:45:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:45:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:45:27Z" 2024-04-19T23:45:27.715 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:45:27.716 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.572795_298.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.572795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:45:27Z" } 2024-04-19T23:45:27.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:45:27.716 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:27.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:27.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.572795_298.1 2024-04-19T23:45:32.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:32.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:32.718 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:45:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:45:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:45:32.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:32.813+0000 7fed69e7e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:32.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:32.814+0000 7fed69e7e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:32.828 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:45:32.828 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:45:32.828 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:45:32.828 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:45:32.828 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:45:32.828 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:45:32.828 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:32.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:32.829 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:45:32.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:32.829 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:45:32.829 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:32.829 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:32.829 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:45:32.829 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:32.829 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:45:32.829 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:32.829 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:45:32.830 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:32.830 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:32.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:32.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:32.830 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:45:32.830 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:32.830 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:32.830 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:45:32.830 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:32.830 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:32.831 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:32.831 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:45:32.831 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:32.831 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:32.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:32.831 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:32.831 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:45:32.831 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:32.831 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:32.831 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:45:32.831 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:32.831 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:45:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:45:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:32.833 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:45:32.833 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:32.833 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:32.833 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:32.833 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:45:32.833 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:32.833 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:32.833 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:32.833 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:45:32.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:32.833 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:45:32.834 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:45:32.834 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:45:32.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:32.834 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:45:32Z" 2024-04-19T23:45:32.834 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:45:32.835 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:45:32Z" } 2024-04-19T23:45:32.835 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:45:32.835 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:32.835 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:32.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-19T23:45:37.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:37.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:37.837 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:45:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:45:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:45:37.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:37.932+0000 7faea99b9780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:37.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:37.933+0000 7faea99b9780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:37.947 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:45:37.948 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:45:37.948 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:45:37.948 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:45:37.948 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:45:37.948 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:45:37.948 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:37.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:37.948 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:45:37.948 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:37.948 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:45:37.948 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:37.949 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:37.949 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:45:37.949 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:37.949 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:45:37.949 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:37.949 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:45:37.949 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:37.949 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:37.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:37.949 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:37.949 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:45:37.950 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:37.950 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:37.950 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:45:37.950 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:37.950 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:37.950 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:37.950 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:45:37.950 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:37.950 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:37.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:37.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:37.950 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:45:37.951 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:37.951 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:37.951 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.577447_300.1", 2024-04-19T23:45:37.951 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:37.951 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:37.951 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:37.951 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-19T23:45:37.951 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:37.951 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:37.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:37.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:37.952 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:45:37.952 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:37.952 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:37.952 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:45:37.952 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:37.952 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:37.952 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:37.952 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:45:37.952 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:37.952 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:37.953 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:37.953 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:45:37.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:37.953 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:45:37.953 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:45:37.953 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:45:37.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:37.953 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:45:37Z" 2024-04-19T23:45:37.953 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:45:37.954 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.577447_300.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.577447Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:45:37Z" } 2024-04-19T23:45:37.954 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:45:37.954 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:37.955 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:37.955 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.577447_300.1 2024-04-19T23:45:42.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:42.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:42.956 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:45:43.015 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:45:43.016 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:45:43.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:43.052+0000 7fc481dd1780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:43.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:43.053+0000 7fc481dd1780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:43.068 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:45:43.068 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:45:43.068 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:45:43.068 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:45:43.068 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:45:43.068 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:45:43.068 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:43.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:43.068 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:43.069 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:43.070 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:45:43.070 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:43.070 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:43.070 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:45:43.070 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:43.070 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:43.070 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:43.070 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:45:43.070 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:43.070 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:43.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:43.071 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:43.071 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:45:43.071 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:43.071 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:43.071 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:45:43.071 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:43.071 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:43.071 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:43.071 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:45:43.071 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:43.071 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:43.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:43.072 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:43.072 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:45:43.072 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:43.072 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:43.072 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:45:43.072 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:43.072 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:43.072 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:43.072 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:45:43.073 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:43.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:43.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:43.073 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:45:43.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:43.073 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:45:43.073 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:45:43.073 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:45:43.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:43.073 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:45:43Z" 2024-04-19T23:45:43.073 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:45:43.074 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:45:43Z" } 2024-04-19T23:45:43.075 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:45:43.075 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:43.075 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:43.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-19T23:45:48.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:48.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:48.076 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:45:48.136 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:45:48.136 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:45:48.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:48.172+0000 7f73478fb780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:48.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:48.173+0000 7f73478fb780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:45:48.188 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:45:48.188 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:45:48.188 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:45:48.188 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:45:48.188 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:45:48.188 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:48.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:48.188 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:45:48.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:48.188 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:45:48.188 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:48.189 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:48.189 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:45:48.189 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:48.189 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:45:48.189 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:48.189 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:45:48.189 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:48.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:48.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:48.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:48.189 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:45:48.189 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:48.190 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:48.190 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:45:48.190 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:48.190 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:48.190 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:48.190 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:45:48.190 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:48.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:48.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:48.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:48.191 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:45:48.191 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:48.191 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:48.191 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.698420_310.1", 2024-04-19T23:45:48.191 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:48.191 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:48.191 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:48.191 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-19T23:45:48.191 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:48.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:48.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:48.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:48.192 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:45:48.192 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:48.192 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:48.192 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:45:48.192 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:48.192 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:48.192 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:48.192 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:45:48.192 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:48.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:48.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:48.193 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:45:48.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:48.193 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:45:48.193 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:45:48.193 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:45:48.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:48.193 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:45:48Z" 2024-04-19T23:45:48.193 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:45:48.194 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.698420_310.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.698420Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:45:48Z" } 2024-04-19T23:45:48.194 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.698420_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.698420Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:45:48.194 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:48.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.698420_310.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:48.195 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.698420_310.1 2024-04-19T23:45:53.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:53.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:53.197 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:45:53.259 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:45:53.260 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:45:53.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:53.329+0000 7efe6fe54780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:53.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:53.330+0000 7efe6fe54780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:53.344 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:45:53.344 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:45:53.344 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:45:53.344 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:45:53.344 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:45:53.345 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:45:53.345 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:53.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:53.345 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:45:53.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:53.345 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:45:53.345 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:53.345 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:53.345 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:45:53.345 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:53.345 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:45:53.346 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:53.346 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:45:53.346 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:53.346 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:53.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:53.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:53.346 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:45:53.346 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:53.346 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:53.346 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:45:53.347 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:53.347 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:53.347 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:53.347 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:45:53.347 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:53.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:53.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:53.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:53.347 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:45:53.347 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:53.347 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:53.347 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.580723_301.1", 2024-04-19T23:45:53.348 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:53.348 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:53.348 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:53.348 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-19T23:45:53.348 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:53.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:53.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:53.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:53.348 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:45:53.348 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:53.348 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:53.349 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:45:53.349 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:53.349 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:53.349 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:53.349 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:45:53.349 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:53.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:53.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:53.349 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:45:53.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:53.349 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:45:53.349 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:45:53.350 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:45:53.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:53.350 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:45:53Z" 2024-04-19T23:45:53.350 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:45:53.351 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.580723_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.580723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:45:53Z" } 2024-04-19T23:45:53.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:45:53.351 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:53.351 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:53.351 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.580723_301.1 2024-04-19T23:45:58.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:58.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:45:58.353 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:45:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:45:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:45:58.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:58.501+0000 7f2da4ca4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:58.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:45:58.503+0000 7f2da4ca4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:45:58.517 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:45:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:45:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:45:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:45:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:45:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:45:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:58.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:45:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:45:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:45:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:45:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:58.518 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:45:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:45:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:45:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:58.519 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:45:58.520 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:58.520 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:58.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:58.520 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:58.520 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:45:58.520 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:58.520 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:58.520 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:45:58.520 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:58.520 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:58.520 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:45:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:45:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:45:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:45:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:45:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:45:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:45:58.521 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:45:58.522 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:45:58.522 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:45:58.522 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:45:58.522 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:58.522 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:45:58.522 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:45:58.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:58.522 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:45:58.522 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:45:58.522 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:45:58.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:45:58.523 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:45:58Z" 2024-04-19T23:45:58.523 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:45:58.524 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:45:58Z" } 2024-04-19T23:45:58.524 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:45:58.524 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:58.524 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:45:58.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.554348_295.1 2024-04-19T23:46:03.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:03.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:03.525 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:46:03.586 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:46:03.586 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:46:03.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:03.622+0000 7f635cdb3780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:03.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:03.623+0000 7f635cdb3780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:03.637 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:46:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:46:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:46:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:46:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:46:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:46:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:46:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:46:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:46:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:46:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:46:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:46:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:46:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:46:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:46:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.606676_302.1", 2024-04-19T23:46:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-19T23:46:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:46:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:46:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:46:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:46:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:46:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:46:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:46:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:46:03Z" 2024-04-19T23:46:03.643 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:46:03.644 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.606676_302.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.606676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:46:03Z" } 2024-04-19T23:46:03.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:46:03.645 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:03.645 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:03.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.606676_302.1 2024-04-19T23:46:08.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:08.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:08.646 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:46:08.705 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:46:08.705 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:46:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:08.741+0000 7f550a29b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:08.742+0000 7f550a29b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:08.757 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:46:08.757 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:46:08.757 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:46:08.757 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:46:08.757 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:46:08.757 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:46:08.757 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:08.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:08.758 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:46:08.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:08.758 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:46:08.758 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:08.758 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:08.758 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:46:08.758 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:08.758 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:46:08.758 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:08.759 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:46:08.759 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:08.759 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:08.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:08.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:08.759 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:46:08.759 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:08.759 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:08.759 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:46:08.759 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:08.759 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:08.759 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:08.760 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:46:08.760 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:08.760 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:08.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:08.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:08.760 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:46:08.760 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:08.760 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:08.760 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.555632_296.1", 2024-04-19T23:46:08.760 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:08.760 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:08.761 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:08.761 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-19T23:46:08.761 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:08.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:08.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:08.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:08.761 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:46:08.761 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:08.761 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:08.761 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:46:08.761 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:08.761 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:08.762 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:08.762 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:46:08.762 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:08.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:08.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:08.762 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:46:08.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:08.762 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:46:08.762 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:46:08.762 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:46:08.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:08.763 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:46:08Z" 2024-04-19T23:46:08.763 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:46:08.764 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.555632_296.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.555632Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:46:08Z" } 2024-04-19T23:46:08.764 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:46:08.764 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:08.764 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:08.764 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.555632_296.1 2024-04-19T23:46:13.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:13.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:13.765 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:46:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:46:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:46:13.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:13.861+0000 7efd7bc0c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:13.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:13.862+0000 7efd7bc0c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:13.878 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:46:13.878 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:46:13.878 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:46:13.879 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:46:13.879 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:46:13.879 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:46:13.879 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:13.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:13.879 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:46:13.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:13.880 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:46:13.880 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:13.880 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:13.880 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:46:13.880 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:13.880 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:46:13.880 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:13.880 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:46:13.880 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:13.880 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:13.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:13.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:13.881 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:46:13.881 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:13.881 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:13.881 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:46:13.881 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:13.881 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:13.882 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:13.882 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:46:13.882 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:13.882 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:13.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:13.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:13.882 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:46:13.882 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:13.882 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:13.882 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:46:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:46:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:46:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:13.883 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:46:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:46:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:46:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:13.884 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:46:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:46:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:46:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:13.885 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:46:13Z" 2024-04-19T23:46:13.885 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:46:13.886 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:46:13Z" } 2024-04-19T23:46:13.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:46:13.886 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:13.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:13.886 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-19T23:46:18.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:18.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:18.888 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:46:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:46:18.948 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:46:18.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:18.983+0000 7ff240d17780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:18.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:18.985+0000 7ff240d17780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:18.999 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:46:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:46:18.999 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:46:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:46:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:46:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:46:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:46:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:46:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:46:19.000 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:46:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:46:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:46:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:19.001 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:46:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:46:19.002 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:46:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:46:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:19.003 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:46:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:46:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:46:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:19.004 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:46:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:46:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:46:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:46:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:46:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:19.005 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:46:18Z" 2024-04-19T23:46:19.005 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:46:19.006 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:46:18Z" } 2024-04-19T23:46:19.007 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:46:19.007 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:19.007 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:19.007 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-19T23:46:24.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:24.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:24.009 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:46:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:46:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:46:24.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:24.105+0000 7f18fd80d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:24.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:24.106+0000 7f18fd80d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:24.121 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:46:24.121 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:46:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:46:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:46:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:46:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:46:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:46:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:46:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:46:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:46:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:46:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:46:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:46:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:46:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:46:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:46:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:46:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:46:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:46:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:46:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:46:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:46:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:46:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:46:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:46:24Z" 2024-04-19T23:46:24.128 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:46:24.129 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:46:24Z" } 2024-04-19T23:46:24.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:46:24.129 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:24.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:24.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-19T23:46:29.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:29.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:29.131 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:46:29.191 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:46:29.191 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:46:29.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:29.227+0000 7f2b76b3d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:29.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:29.229+0000 7f2b76b3d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:29.243 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:46:29.243 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:46:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:46:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:46:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:46:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:46:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:46:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:46:29.244 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:46:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:46:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:46:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:29.245 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:46:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:46:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:46:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:29.246 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:46:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:46:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:46:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:29.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:46:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:46:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:46:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:29.248 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:46:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:46:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:46:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:46:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:29.249 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:46:29Z" 2024-04-19T23:46:29.250 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:46:29.251 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:46:29Z" } 2024-04-19T23:46:29.251 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:46:29.251 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:29.251 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:29.251 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.554348_295.1 2024-04-19T23:46:34.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:34.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:34.253 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:46:34.367 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:46:34.368 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:46:34.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:34.412+0000 7f744977b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:34.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:34.413+0000 7f744977b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:34.427 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:46:34.427 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:46:34.427 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:46:34.428 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:46:34.428 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:46:34.428 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:46:34.428 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:34.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:34.428 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:46:34.428 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:34.428 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:46:34.428 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:34.428 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:34.429 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:46:34.429 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:34.429 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:46:34.429 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:34.429 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:46:34.429 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:34.429 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:34.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:34.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:34.429 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:46:34.430 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:34.430 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:34.430 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:46:34.430 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:34.430 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:34.430 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:34.430 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:46:34.430 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:34.430 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:34.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:34.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:46:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-19T23:46:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-19T23:46:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:46:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:46:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:46:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:34.433 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:34.433 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:46:34.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:34.433 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:46:34.433 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:46:34.433 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:46:34.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:34.433 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:46:34Z" 2024-04-19T23:46:34.433 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:46:34.434 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:46:34Z" } 2024-04-19T23:46:34.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:46:34.435 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:34.435 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:34.435 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.682721_308.1 2024-04-19T23:46:39.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:39.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:39.436 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:46:39.494 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:46:39.495 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:46:39.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:39.530+0000 7f82e4399780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:39.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:39.531+0000 7f82e4399780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:39.546 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:46:39.546 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:46:39.546 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:46:39.547 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:46:39.547 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:46:39.547 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:46:39.547 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:39.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:39.547 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:46:39.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:39.547 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:46:39.547 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:39.547 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:39.547 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:46:39.547 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:39.548 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:46:39.548 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:39.548 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:46:39.548 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:39.548 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:39.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:39.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:39.548 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:46:39.548 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:39.548 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:39.548 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:46:39.548 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:39.549 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:39.549 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:39.549 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:46:39.549 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:39.549 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:39.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:39.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:39.549 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:46:39.549 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:39.549 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:39.549 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:46:39.550 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:39.550 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:39.550 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:39.550 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:46:39.550 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:39.550 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:39.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:39.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:39.550 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:46:39.550 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:46:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:46:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:46:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:46:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:46:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:46:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:46:39Z" 2024-04-19T23:46:39.552 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:46:39.553 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:46:39Z" } 2024-04-19T23:46:39.553 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:46:39.553 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:39.554 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:39.554 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-19T23:46:44.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:44.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:44.555 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:46:44.616 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:46:44.617 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:46:44.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:44.652+0000 7efc02203780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:44.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:44.653+0000 7efc02203780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:44.668 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:46:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:46:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:46:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:46:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:46:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:46:44.668 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:46:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:46:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:46:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:46:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:46:44.669 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:46:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:46:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:44.670 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:46:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:46:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.687754_309.1", 2024-04-19T23:46:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:44.671 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-19T23:46:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:46:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:46:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:44.672 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:46:44Z" 2024-04-19T23:46:44.673 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:46:44.674 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.687754_309.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.687754Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:46:44Z" } 2024-04-19T23:46:44.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.687754_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.687754Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:46:44.675 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:44.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.687754_309.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:44.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.687754_309.1 2024-04-19T23:46:49.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:49.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:49.676 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:46:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:46:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:46:49.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:49.772+0000 7fe9b490c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:49.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:49.773+0000 7fe9b490c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:49.787 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:46:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:46:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:46:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:46:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:46:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:46:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:46:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:46:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:46:49.788 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:46:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:46:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:46:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:49.789 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:46:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:46:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:46:49.790 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:46:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:46:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:49.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:46:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:46:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:46:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:49.792 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:46:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:46:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:46:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:46:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:49.793 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:46:49Z" 2024-04-19T23:46:49.793 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:46:49.794 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:46:49Z" } 2024-04-19T23:46:49.794 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:46:49.794 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:49.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:49.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-19T23:46:54.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:54.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:54.796 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:46:54.855 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:46:54.856 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:46:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:54.892+0000 7fadb325b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:46:54.893+0000 7fadb325b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:46:54.908 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:46:54.908 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:46:54.908 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:46:54.908 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:46:54.908 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:46:54.908 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:46:54.908 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:54.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:54.908 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:46:54.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:54.909 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:46:54.909 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:54.909 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:54.909 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:46:54.909 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:54.909 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:46:54.909 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:54.909 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:46:54.909 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:54.909 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:54.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:54.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:54.910 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:46:54.910 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:54.910 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:54.910 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:46:54.910 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:54.910 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:54.910 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:54.910 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:46:54.910 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:54.911 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:54.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:54.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:54.911 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:46:54.911 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:54.911 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:54.911 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.632785_307.1", 2024-04-19T23:46:54.912 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:54.912 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:54.912 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:54.912 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-19T23:46:54.912 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:54.912 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:54.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:54.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:46:54.912 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:46:54.912 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:46:54.912 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:46:54.913 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:46:54.913 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:46:54.913 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:46:54.913 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:46:54.913 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:46:54.913 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:46:54.913 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:54.913 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:46:54.913 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:46:54.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:54.913 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:46:54.913 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:46:54.914 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:46:54.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:46:54.914 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:46:54Z" 2024-04-19T23:46:54.914 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:46:54.915 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.632785_307.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.632785Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:46:54Z" } 2024-04-19T23:46:54.915 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:46:54.915 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:54.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:46:54.915 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.632785_307.1 2024-04-19T23:46:59.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:59.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:46:59.917 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:46:59.976 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:46:59.976 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:47:00.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:00.013+0000 7f552eecf780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:00.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:00.019+0000 7f552eecf780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:00.110 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:47:00.111 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:47:00.111 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:47:00.111 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:47:00.111 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:47:00.111 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:47:00.112 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:00.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:00.112 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:47:00.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:00.112 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:47:00.113 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:00.113 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:00.113 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:47:00.113 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:00.113 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:47:00.114 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:00.114 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:47:00.114 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:00.114 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:00.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:00.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:00.115 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:47:00.115 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:00.115 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:00.115 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:47:00.116 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:00.116 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:00.116 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:00.116 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:47:00.116 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:00.116 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:00.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:00.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:00.117 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:47:00.117 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:00.117 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:00.117 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.738086_316.1", 2024-04-19T23:47:00.117 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:00.117 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:00.117 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:00.117 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-19T23:47:00.117 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:00.117 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:00.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:00.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:00.118 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:47:00.118 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:00.118 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:00.118 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:47:00.118 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:00.118 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:00.118 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:00.118 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:47:00.118 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:00.119 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:00.119 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:00.119 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:47:00.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:00.119 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:47:00.119 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:47:00.119 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:47:00.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:00.119 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:47:00Z" 2024-04-19T23:47:00.119 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:47:00.120 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.738086_316.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.738086Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:47:00Z" } 2024-04-19T23:47:00.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:47:00.120 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:00.120 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:00.120 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.738086_316.1 2024-04-19T23:47:05.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:05.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:05.122 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:47:05.187 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:47:05.187 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:47:05.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:05.318+0000 7ff4e0f37780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:05.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:05.319+0000 7ff4e0f37780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:05.334 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:47:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:47:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:47:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:47:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:47:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:47:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:47:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:47:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:47:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:47:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:47:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:47:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:47:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:47:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:47:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:47:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:47:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:47:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:47:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:47:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:47:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:47:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:47:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:47:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:47:05Z" 2024-04-19T23:47:05.340 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:47:05.341 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:47:05Z" } 2024-04-19T23:47:05.341 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:47:05.342 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:05.342 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:05.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-19T23:47:10.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:10.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:10.343 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:47:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:47:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:47:10.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:10.439+0000 7fb553fd7780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:10.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:10.440+0000 7fb553fd7780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:10.455 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:47:10.455 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:47:10.455 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:47:10.455 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:47:10.455 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:47:10.455 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:47:10.455 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:10.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:10.456 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:47:10.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:10.456 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:47:10.456 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:10.456 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:10.456 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:47:10.456 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:10.456 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:47:10.456 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:10.456 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:47:10.456 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:10.457 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:10.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:10.457 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:10.457 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:47:10.457 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:10.457 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:10.457 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:47:10.457 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:10.457 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:10.457 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:10.457 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:47:10.457 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:10.458 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:10.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:10.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:10.458 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:47:10.458 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:10.458 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:10.458 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.614143_305.1", 2024-04-19T23:47:10.458 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:10.458 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:10.458 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:10.458 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-19T23:47:10.458 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:10.459 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:10.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:10.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:10.459 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:47:10.459 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:10.459 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:10.459 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:47:10.459 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:10.459 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:10.460 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:10.460 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:47:10.460 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:10.460 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:10.460 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:10.460 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:47:10.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:10.460 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:47:10.460 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:47:10.460 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:47:10.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:10.461 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:47:10Z" 2024-04-19T23:47:10.461 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:47:10.462 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.614143_305.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.614143Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:47:10Z" } 2024-04-19T23:47:10.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:47:10.463 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:10.463 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:10.463 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.614143_305.1 2024-04-19T23:47:15.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:15.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:15.465 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:47:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:47:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:47:15.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:15.589+0000 7f0d671de780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:15.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:15.590+0000 7f0d671de780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:15.606 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:47:15.606 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:47:15.606 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:47:15.606 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:47:15.606 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:47:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:47:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:47:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:47:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:47:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:47:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:15.608 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:47:15.609 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:15.609 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:15.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:15.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:15.609 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:47:15.609 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:15.609 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:15.609 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:47:15.609 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:15.609 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:15.609 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:15.610 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:47:15.610 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:15.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:15.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:15.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:15.610 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:47:15.610 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:15.610 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:15.610 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:47:15.610 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:15.610 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:15.611 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:15.611 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:47:15.611 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:15.611 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:15.611 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:15.611 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:47:15.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:15.611 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:47:15.611 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:47:15.611 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:47:15.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:15.611 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:47:15Z" 2024-04-19T23:47:15.612 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:47:15.612 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:47:15Z" } 2024-04-19T23:47:15.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:47:15.613 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:15.613 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:15.613 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.576049_299.1 2024-04-19T23:47:20.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:20.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:20.614 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:47:20.674 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:47:20.674 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:47:20.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:20.710+0000 7f3b2a4a5780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:20.711+0000 7f3b2a4a5780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:47:20.727 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:47:20.727 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:47:20.727 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:47:20.727 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:47:20.727 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:47:20.727 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:20.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:20.727 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:47:20.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:20.727 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:47:20.728 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:20.728 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:20.728 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:47:20.728 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:20.728 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:47:20.728 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:20.728 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:47:20.728 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:20.728 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:20.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:20.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:20.729 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:47:20.729 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:20.729 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:20.729 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:47:20.729 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:20.729 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:20.729 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:20.729 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:47:20.730 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:20.730 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:20.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:20.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:20.730 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:47:20.730 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:20.730 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:20.730 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.720922_313.1", 2024-04-19T23:47:20.730 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:20.731 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:20.731 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:20.731 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.720922Z", 2024-04-19T23:47:20.731 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:20.731 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:20.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:20.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:20.731 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:47:20.731 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:20.731 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:20.731 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:47:20.731 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:20.732 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:20.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:20.732 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:47:20.732 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:20.732 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:20.732 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:20.732 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:47:20.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:20.732 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:47:20.732 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:47:20.732 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:47:20.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:20.733 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:47:20Z" 2024-04-19T23:47:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:47:20.734 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.720922_313.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.720922Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:47:20Z" } 2024-04-19T23:47:20.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.720922_313.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.720922Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:47:20.734 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:20.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.720922_313.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:20.734 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.720922_313.1 2024-04-19T23:47:25.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:25.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:25.736 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:47:25.841 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:47:25.842 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:47:25.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:25.915+0000 7fdbcc1b2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:25.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:25.916+0000 7fdbcc1b2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:25.931 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:47:25.931 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:47:25.931 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:47:25.931 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:47:25.931 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:47:25.931 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:47:25.931 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:25.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:25.932 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:47:25.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:25.932 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:47:25.932 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:25.932 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:25.932 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:47:25.932 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:25.932 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:47:25.932 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:25.932 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:47:25.932 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:25.933 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:25.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:25.933 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:25.933 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:47:25.933 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:25.933 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:25.933 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:47:25.933 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:25.933 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:25.933 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:25.933 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:47:25.933 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:25.934 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:25.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:25.934 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:25.934 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:47:25.934 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:25.934 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:25.934 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.577447_300.1", 2024-04-19T23:47:25.934 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:25.934 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:25.934 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:25.935 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-19T23:47:25.935 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:25.935 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:25.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:25.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:25.935 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:47:25.935 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:25.935 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:25.935 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:47:25.935 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:25.935 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:25.936 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:25.936 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:47:25.936 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:25.936 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:25.936 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:25.936 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:47:25.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:25.936 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:47:25.936 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:47:25.936 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:47:25.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:25.937 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:47:25Z" 2024-04-19T23:47:25.937 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:47:25.938 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.577447_300.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.577447Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:47:25Z" } 2024-04-19T23:47:25.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:47:25.938 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:25.938 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:25.938 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.577447_300.1 2024-04-19T23:47:30.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:30.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:30.940 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:47:30.999 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:47:30.999 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:47:31.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:31.035+0000 7f31ca362780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:31.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:31.036+0000 7f31ca362780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:31.052 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:47:31.052 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:47:31.052 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:47:31.052 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:47:31.052 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:47:31.052 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:47:31.052 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:31.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:31.052 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:47:31.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:31.053 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:47:31.053 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:31.053 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:31.053 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:47:31.053 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:31.053 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:47:31.053 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:31.053 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:47:31.053 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:31.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:31.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:31.054 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:31.054 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:47:31.054 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:31.054 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:31.054 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:47:31.054 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:31.054 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:31.054 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:31.054 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:47:31.055 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:31.055 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:31.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:31.055 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:31.055 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:47:31.055 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:31.055 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:31.055 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.580723_301.1", 2024-04-19T23:47:31.055 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:31.055 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:31.055 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:31.056 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-19T23:47:31.056 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:31.056 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:31.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:31.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:31.056 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:47:31.056 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:31.056 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:31.056 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:47:31.056 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:31.056 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:31.056 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:47:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:47:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:47:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:47:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:47:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:31.058 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:47:31Z" 2024-04-19T23:47:31.058 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:47:31.059 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.580723_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.580723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:47:31Z" } 2024-04-19T23:47:31.059 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:47:31.059 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:31.059 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:31.059 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.580723_301.1 2024-04-19T23:47:36.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:36.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:36.060 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:47:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:47:36.119 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:47:36.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:36.155+0000 7fe152c7a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:36.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:36.156+0000 7fe152c7a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:36.171 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:47:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:47:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:47:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:47:36.171 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:47:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:47:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:47:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:47:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:47:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:36.172 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:47:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:47:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:47:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:36.173 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:47:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:47:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:36.174 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:47:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.577447_300.1", 2024-04-19T23:47:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-19T23:47:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:36.175 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:47:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:47:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:47:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:47:36.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:36.177 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:47:36.177 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:47:36.177 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:47:36.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:36.177 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:47:36Z" 2024-04-19T23:47:36.177 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:47:36.178 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.577447_300.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.577447Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:47:36Z" } 2024-04-19T23:47:36.178 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:47:36.178 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:36.178 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:36.178 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.577447_300.1 2024-04-19T23:47:41.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:41.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:41.180 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:47:41.240 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:47:41.240 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:47:41.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:41.276+0000 7fcc7c794780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:41.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:41.277+0000 7fcc7c794780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:41.293 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:47:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:47:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:47:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:47:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:47:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:47:41.293 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:47:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:47:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:47:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:47:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:41.294 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:47:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:47:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:47:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:41.295 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:47:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:47:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.580723_301.1", 2024-04-19T23:47:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:41.296 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-19T23:47:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:47:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:47:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:41.297 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:47:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:47:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:47:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:47:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:47:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:41.298 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:47:41Z" 2024-04-19T23:47:41.299 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:47:41.299 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.580723_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.580723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:47:41Z" } 2024-04-19T23:47:41.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:47:41.300 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:41.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:41.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.580723_301.1 2024-04-19T23:47:46.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:46.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:46.302 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:47:46.362 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:47:46.362 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:47:46.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:46.400+0000 7fa40903b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:46.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:46.401+0000 7fa40903b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:46.417 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:47:46.417 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:47:46.417 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:47:46.417 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:47:46.417 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:47:46.417 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:47:46.418 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:46.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:46.418 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:47:46.418 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:46.418 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:47:46.418 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:46.418 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:46.418 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:47:46.418 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:46.418 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:47:46.418 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:46.419 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:47:46.419 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:46.419 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:46.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:46.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:46.419 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:47:46.419 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:46.419 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:46.419 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:47:46.419 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:46.419 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:46.420 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:46.420 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:47:46.420 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:46.420 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:46.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:46.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:46.420 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:47:46.420 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:46.420 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:46.420 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.580723_301.1", 2024-04-19T23:47:46.420 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:46.421 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:46.421 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:46.421 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-19T23:47:46.421 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:46.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:46.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:46.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:46.421 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:47:46.421 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:46.421 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:46.422 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:47:46.422 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:46.422 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:46.422 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:46.422 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:47:46.422 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:46.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:46.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:46.422 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:47:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:47:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:47:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:47:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:47:46Z" 2024-04-19T23:47:46.423 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:47:46.424 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.580723_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.580723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:47:46Z" } 2024-04-19T23:47:46.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:47:46.424 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:46.425 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:46.425 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.580723_301.1 2024-04-19T23:47:51.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:51.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:51.426 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:47:51.484 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:47:51.484 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:47:51.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:51.521+0000 7f65a5ea4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:51.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:51.523+0000 7f65a5ea4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:51.538 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:47:51.538 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:47:51.538 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:47:51.538 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:47:51.538 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:47:51.538 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:47:51.538 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:51.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:51.538 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:47:51.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:51.539 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:47:51.539 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:51.539 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:51.539 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:47:51.539 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:51.539 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:47:51.539 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:51.539 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:47:51.539 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:51.539 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:51.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:51.539 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:51.540 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:47:51.541 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:47:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:47:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:47:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:47:51.543 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:47:51.543 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:47:51.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:51.543 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:47:51Z" 2024-04-19T23:47:51.543 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:47:51.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:47:51Z" } 2024-04-19T23:47:51.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:47:51.544 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:51.544 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:51.544 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-19T23:47:56.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:56.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:47:56.546 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:47:56.621 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:47:56.621 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:47:56.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:56.731+0000 7fdcdb6d5780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:56.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:47:56.733+0000 7fdcdb6d5780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:47:56.747 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:47:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:47:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:47:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:47:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:47:56.747 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:47:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:47:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:47:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:47:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:47:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:56.748 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:47:56.749 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:56.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:56.751 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:47:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:47:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:47:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:47:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:47:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:47:56.752 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:47:56Z" 2024-04-19T23:47:56.752 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:47:56.753 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:47:56Z" } 2024-04-19T23:47:56.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:47:56.753 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:56.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:47:56.754 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.797017_325.1 2024-04-19T23:48:01.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:01.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:01.755 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:48:01.810 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:48:01.810 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:48:01.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:01.844+0000 7f28eba46780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:01.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:01.845+0000 7f28eba46780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:01.860 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:48:01.860 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:48:01.860 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:48:01.860 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:48:01.860 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:48:01.860 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:48:01.861 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:01.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:01.861 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:48:01.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:01.861 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:48:01.861 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:01.861 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:01.861 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:48:01.861 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:01.861 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:48:01.861 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:01.861 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:48:01.862 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:01.862 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:01.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:01.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:01.862 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:48:01.862 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:01.862 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:01.862 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:48:01.862 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:01.862 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:01.862 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:01.863 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:48:01.863 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:01.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:01.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:01.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:01.863 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:48:01.863 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:01.863 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:01.863 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.572795_298.1", 2024-04-19T23:48:01.863 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:01.863 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:01.864 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:01.865 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:48:01.865 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:01.865 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:01.865 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:01.865 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:48:01.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:01.865 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:48:01.865 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:48:01.865 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:48:01.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:01.865 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:48:01Z" 2024-04-19T23:48:01.865 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:48:01.866 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.572795_298.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.572795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:48:01Z" } 2024-04-19T23:48:01.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:48:01.867 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:01.867 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:01.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.572795_298.1 2024-04-19T23:48:06.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:06.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:06.868 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:48:06.926 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:48:06.927 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:48:06.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:06.963+0000 7f1c2a392780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:06.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:06.964+0000 7f1c2a392780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:06.980 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:48:06.980 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:48:06.980 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:48:06.980 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:48:06.980 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:48:06.980 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:48:06.980 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:06.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:06.980 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:48:06.980 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:06.981 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:48:06.981 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:06.981 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:06.981 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:48:06.981 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:06.981 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:48:06.981 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:06.981 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:48:06.981 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:06.981 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:06.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:06.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:06.982 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:48:06.982 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:06.982 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:06.982 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:48:06.982 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:06.982 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:06.982 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:06.982 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:48:06.982 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:06.982 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:06.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:06.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:06.983 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:48:06.983 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:06.983 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:06.983 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:48:06.983 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:06.983 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:06.983 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:06.983 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:48:06.983 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:06.983 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:06.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:06.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:06.984 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:48:06.984 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:06.984 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:06.984 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:48:06.984 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:06.984 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:06.984 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:06.984 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:48:06.984 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:06.985 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:06.985 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:06.985 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:48:06.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:06.985 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:48:06.985 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:48:06.985 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:48:06.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:06.985 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:48:06Z" 2024-04-19T23:48:06.985 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:48:06.986 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:48:06Z" } 2024-04-19T23:48:06.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:48:06.987 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:06.987 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:06.987 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-19T23:48:11.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:11.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:11.988 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:48:12.048 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:48:12.048 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:48:12.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:12.084+0000 7f422ef7d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:12.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:12.086+0000 7f422ef7d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:12.101 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:48:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:48:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:48:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:48:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:48:12.101 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:48:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:48:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:48:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:48:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:48:12.102 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:48:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:48:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:48:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:12.103 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:48:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:48:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.632785_307.1", 2024-04-19T23:48:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:12.104 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-19T23:48:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:48:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:48:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:12.105 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:48:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:48:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:48:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:48:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:48:12.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:12.107 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:48:12Z" 2024-04-19T23:48:12.107 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:48:12.107 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.632785_307.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.632785Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:48:12Z" } 2024-04-19T23:48:12.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:48:12.108 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:12.108 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:12.108 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.632785_307.1 2024-04-19T23:48:17.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:17.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:17.109 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:48:17.167 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:48:17.168 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:48:17.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:17.203+0000 7f51b1229780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:17.204+0000 7f51b1229780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:17.220 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:48:17.220 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:48:17.220 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:48:17.220 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:48:17.221 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:48:17.221 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:48:17.221 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:17.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:17.221 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:48:17.221 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:17.221 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:48:17.221 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:17.221 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:17.221 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:48:17.221 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:17.222 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:48:17.222 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:17.222 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:48:17.222 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:17.222 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:17.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:17.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:17.222 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:48:17.222 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:17.222 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:17.222 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:48:17.223 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:17.223 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:17.223 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:17.223 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:48:17.223 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:17.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:17.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:17.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:17.224 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:48:17.224 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:17.225 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:17.225 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.720922_313.1", 2024-04-19T23:48:17.225 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:17.226 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:17.226 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:17.227 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.720922Z", 2024-04-19T23:48:17.227 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:17.228 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:17.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:17.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:17.229 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:48:17.229 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:17.229 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:17.230 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:48:17.230 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:17.230 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:17.230 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:48:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:48:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:48:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:48:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:48:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:17.231 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:48:17Z" 2024-04-19T23:48:17.231 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:48:17.233 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.720922_313.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.720922Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:48:17Z" } 2024-04-19T23:48:17.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.720922_313.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.720922Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:48:17.233 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:17.233 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.720922_313.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:17.233 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.720922_313.1 2024-04-19T23:48:22.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:22.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:22.235 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:48:22.295 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:48:22.295 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:48:22.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:22.331+0000 7f315381f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:22.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:22.333+0000 7f315381f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:22.348 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:48:22.348 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:48:22.348 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:48:22.348 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:48:22.349 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:48:22.349 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:48:22.349 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:22.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:22.349 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:48:22.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:22.349 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:48:22.349 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:22.349 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:22.349 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:48:22.349 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:22.350 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:48:22.350 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:22.350 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:48:22.350 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:22.350 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:22.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:22.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:22.350 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:48:22.350 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:22.350 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:22.350 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:48:22.351 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:22.351 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:22.351 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:22.351 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:48:22.351 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:22.351 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:22.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:22.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:22.351 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:48:22.351 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:22.351 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:22.351 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.555632_296.1", 2024-04-19T23:48:22.352 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:22.352 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:22.352 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:22.352 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-19T23:48:22.352 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:22.352 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:22.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:22.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:22.352 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:48:22.352 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:22.352 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:22.353 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:48:22.353 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:22.353 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:22.353 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:22.353 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:48:22.353 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:22.353 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:22.353 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:22.353 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:48:22.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:22.353 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:48:22.353 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:48:22.354 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:48:22.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:22.354 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:48:22Z" 2024-04-19T23:48:22.354 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:48:22.355 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.555632_296.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.555632Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:48:22Z" } 2024-04-19T23:48:22.355 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:48:22.355 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:22.355 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:22.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.555632_296.1 2024-04-19T23:48:27.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:27.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:27.357 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:48:27.416 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:48:27.417 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:48:27.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:27.453+0000 7f9285b8c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:27.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:27.455+0000 7f9285b8c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:27.470 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:48:27.470 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:48:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:48:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:48:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:48:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:48:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:48:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:48:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:27.472 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:48:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:48:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:48:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:48:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:48:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:27.474 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:48:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:48:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:48:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:48:27.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:48:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:48:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:48:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:27.476 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:48:27Z" 2024-04-19T23:48:27.476 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:48:27.477 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:48:27Z" } 2024-04-19T23:48:27.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:48:27.477 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:27.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:27.477 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.576049_299.1 2024-04-19T23:48:32.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:32.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:32.478 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:48:32.583 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:48:32.583 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:48:32.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:32.619+0000 7f04418ab780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:32.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:32.620+0000 7f04418ab780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:32.636 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:48:32.636 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:48:32.636 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:48:32.636 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:48:32.636 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:48:32.637 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:48:32.637 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:32.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:32.637 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:48:32.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:32.637 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:48:32.637 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:32.637 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:32.637 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:48:32.637 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:32.637 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:48:32.638 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:32.638 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:48:32.638 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:32.638 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:32.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:32.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:32.638 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:48:32.638 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:32.638 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:32.638 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:48:32.638 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.577447_300.1", 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:32.639 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:32.640 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:32.640 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-19T23:48:32.640 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:32.640 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:32.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:32.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:32.640 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:48:32.640 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:32.640 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:32.640 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:48:32.640 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:32.641 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:32.641 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:32.641 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:48:32.641 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:32.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:32.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:32.641 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:48:32.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:32.641 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:48:32.641 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:48:32.641 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:48:32.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:32.642 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:48:32Z" 2024-04-19T23:48:32.642 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:48:32.643 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.577447_300.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.577447Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:48:32Z" } 2024-04-19T23:48:32.643 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:48:32.643 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:32.643 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:32.643 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.577447_300.1 2024-04-19T23:48:37.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:37.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:37.645 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:48:37.705 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:48:37.705 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:48:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:37.742+0000 7f989d0f7780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:37.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:37.743+0000 7f989d0f7780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:37.759 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:48:37.759 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:48:37.759 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:48:37.759 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:48:37.759 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:48:37.759 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:48:37.759 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:37.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:37.759 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:48:37.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:37.760 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:48:37.760 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:37.760 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:37.760 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:48:37.760 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:37.760 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:48:37.760 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:37.760 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:48:37.760 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:37.760 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:37.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:37.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:37.761 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:48:37.761 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:37.761 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:37.761 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:48:37.761 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:37.761 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:37.761 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:37.761 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:48:37.761 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:37.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:37.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:37.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:37.762 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:48:37.762 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:37.762 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:37.762 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.868452_328.1", 2024-04-19T23:48:37.762 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:37.762 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:37.762 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:37.763 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-19T23:48:37.763 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:37.763 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:37.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:37.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:37.763 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:48:37.763 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:37.763 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:37.763 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:48:37.763 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:37.763 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:37.763 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:37.764 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:48:37.764 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:37.764 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:37.765 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:37.765 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:48:37.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:37.765 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:48:37.765 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:48:37.765 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:48:37.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:37.765 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:48:37Z" 2024-04-19T23:48:37.765 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:48:37.766 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.868452_328.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.868452Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:48:37Z" } 2024-04-19T23:48:37.767 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.868452_328.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.868452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:48:37.767 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:37.767 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.868452_328.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:37.767 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.868452_328.1 2024-04-19T23:48:42.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:42.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:42.768 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:48:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:48:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:48:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:42.863+0000 7f96a49ee780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:42.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:42.865+0000 7f96a49ee780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:42.880 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:48:42.880 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:48:42.880 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:48:42.881 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:48:42.881 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:48:42.881 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:48:42.881 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:42.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:42.881 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:48:42.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:42.881 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:48:42.881 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:42.881 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:42.881 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:48:42.882 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:42.882 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:48:42.882 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:42.882 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:48:42.882 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:42.882 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:42.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:42.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:42.882 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:48:42.883 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:42.883 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:42.883 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:48:42.883 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:42.883 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:42.883 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:42.883 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:48:42.883 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:42.883 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:42.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:42.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:42.884 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:48:42.884 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:42.884 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:42.884 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.609968_303.1", 2024-04-19T23:48:42.884 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:42.884 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:42.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:42.884 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-19T23:48:42.884 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:42.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:42.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:42.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:42.885 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:48:42.885 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:42.885 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:42.885 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:48:42.885 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:42.885 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:42.885 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:42.885 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:48:42.885 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:42.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:42.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:42.885 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:48:42.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:42.886 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:48:42.886 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:48:42.886 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:48:42.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:42.886 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:48:42Z" 2024-04-19T23:48:42.886 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:48:42.887 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.609968_303.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.609968Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:48:42Z" } 2024-04-19T23:48:42.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:48:42.887 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:42.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:42.888 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.609968_303.1 2024-04-19T23:48:47.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:47.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:47.889 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:48:47.949 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:48:47.949 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:48:47.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:47.984+0000 7f9987691780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:47.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:47.986+0000 7f9987691780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:48.001 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:48:48.001 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:48:48.001 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:48:48.001 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:48:48.001 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:48:48.002 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:48:48.002 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:48.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:48.002 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:48:48.002 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:48.002 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:48:48.002 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:48.002 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:48.002 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:48:48.002 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:48.003 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:48:48.003 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:48.003 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:48:48.003 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:48.003 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:48.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:48.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:48.003 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:48:48.003 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:48.003 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:48.003 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:48:48.003 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:48.004 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:48.004 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:48.004 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:48:48.004 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:48.004 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:48.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:48.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:48.004 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:48:48.004 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:48.004 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:48.004 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.717624_312.1", 2024-04-19T23:48:48.004 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:48.005 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:48.005 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:48.005 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-19T23:48:48.005 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:48.005 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:48.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:48.005 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:48.005 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:48:48.005 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:48.005 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:48.005 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:48:48.006 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:48.006 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:48.006 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:48.006 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:48:48.006 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:48.006 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:48.006 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:48.006 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:48:48.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:48.006 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:48:48.006 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:48:48.007 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:48:48.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:48.007 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:48:47Z" 2024-04-19T23:48:48.007 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:48:48.008 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.717624_312.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.717624Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:48:47Z" } 2024-04-19T23:48:48.008 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.717624_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.717624Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:48:48.008 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:48.008 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.717624_312.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:48.008 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.717624_312.1 2024-04-19T23:48:53.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:53.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:53.010 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:48:53.069 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:48:53.070 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:48:53.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:53.105+0000 7fa00a404780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:53.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:53.107+0000 7fa00a404780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:53.122 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:48:53.122 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:48:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:48:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:48:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:48:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:48:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:48:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:48:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:48:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:48:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:48:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:48:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:48:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:48:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:48:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.717624_312.1", 2024-04-19T23:48:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-19T23:48:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:48:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:48:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:48:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:48:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:53.128 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:48:53.128 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:48:53.128 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:48:53.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:53.128 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:48:53Z" 2024-04-19T23:48:53.128 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:48:53.129 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.717624_312.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.717624Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:48:53Z" } 2024-04-19T23:48:53.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.717624_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.717624Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:48:53.129 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:53.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.717624_312.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:53.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.717624_312.1 2024-04-19T23:48:58.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:58.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:48:58.131 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:48:58.212 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:48:58.212 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:48:58.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:58.314+0000 7fc2dd18d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:58.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:48:58.315+0000 7fc2dd18d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:48:58.331 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:48:58.331 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:48:58.331 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:48:58.331 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:48:58.332 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:48:58.332 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:48:58.332 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:58.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:58.332 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:48:58.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:58.332 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:48:58.332 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:58.332 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:58.333 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:48:58.333 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:58.333 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:48:58.333 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:58.333 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:48:58.333 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:58.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:58.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:58.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:58.334 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:48:58.334 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:58.334 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:58.334 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:48:58.334 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:58.334 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:58.334 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:58.334 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:48:58.334 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:58.335 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:58.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:58.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:58.335 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:48:58.335 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:58.335 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:58.335 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.555632_296.1", 2024-04-19T23:48:58.335 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:58.335 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:58.336 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:58.336 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-19T23:48:58.336 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:58.336 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:58.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:58.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:48:58.336 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:48:58.336 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:48:58.336 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:48:58.337 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:48:58.337 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:48:58.337 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:48:58.337 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:48:58.337 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:48:58.337 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:48:58.337 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:58.337 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:48:58.337 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:48:58.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:58.338 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:48:58.338 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:48:58.338 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:48:58.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:48:58.338 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:48:58Z" 2024-04-19T23:48:58.338 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:48:58.339 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.555632_296.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.555632Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:48:58Z" } 2024-04-19T23:48:58.340 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:48:58.340 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:58.340 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:48:58.340 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.555632_296.1 2024-04-19T23:49:03.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:03.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:03.341 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:49:03.407 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:49:03.407 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:49:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:03.442+0000 7fe3db9eb780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:03.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:03.443+0000 7fe3db9eb780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:03.458 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:49:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:49:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:49:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:49:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:49:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:49:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:49:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:49:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:49:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:49:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:49:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:49:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:49:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:49:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:49:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:49:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:49:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:49:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:49:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:49:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:49:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:49:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:49:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:49:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:49:03Z" 2024-04-19T23:49:03.464 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:49:03.465 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:49:03Z" } 2024-04-19T23:49:03.465 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:49:03.465 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:03.465 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:03.465 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.554348_295.1 2024-04-19T23:49:08.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:08.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:08.467 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:49:08.524 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:49:08.524 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:49:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:08.561+0000 7faefff54780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:08.563+0000 7faefff54780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:08.579 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:49:08.580 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:49:08.580 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:49:08.580 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:49:08.580 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:49:08.580 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:49:08.580 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:08.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:08.580 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:49:08.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:08.580 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:49:08.580 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:08.581 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:08.581 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:49:08.581 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:08.581 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:49:08.581 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:08.581 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:49:08.581 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:08.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:08.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:08.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:08.581 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:49:08.582 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:08.582 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:08.582 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:49:08.582 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:08.582 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:08.582 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:08.582 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:49:08.582 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:08.582 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:08.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:08.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:08.583 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:49:08.583 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:08.583 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:08.583 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:49:08.583 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:08.583 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:08.583 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:08.583 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:49:08.583 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:08.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:08.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:08.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:08.584 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:49:08.584 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:08.584 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:08.584 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:49:08.584 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:08.584 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:08.584 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:08.584 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:49:08.584 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:08.585 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:08.585 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:08.585 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:49:08.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:08.585 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:49:08.585 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:49:08.585 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:49:08.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:08.585 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:49:08Z" 2024-04-19T23:49:08.585 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:49:08.586 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:49:08Z" } 2024-04-19T23:49:08.587 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:49:08.587 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:08.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:08.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-19T23:49:13.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2024-04-19T23:49:13.592 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-42 2024-04-19T23:49:13.593 DEBUG:boto:path=/aktlji-42/ 2024-04-19T23:49:13.593 DEBUG:boto:auth_path=/aktlji-42/ 2024-04-19T23:49:13.593 DEBUG:boto:Method: PUT 2024-04-19T23:49:13.593 DEBUG:boto:Path: /aktlji-42/ 2024-04-19T23:49:13.593 DEBUG:boto:Data: 2024-04-19T23:49:13.593 DEBUG:boto:Headers: {} 2024-04-19T23:49:13.593 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:49:13.593 DEBUG:boto:Port: 8000 2024-04-19T23:49:13.593 DEBUG:boto:Params: {} 2024-04-19T23:49:13.594 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:49:13.594 DEBUG:boto:Token: None 2024-04-19T23:49:13.594 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:49:13 GMT /aktlji-42/ 2024-04-19T23:49:13.594 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NlQ1vw3o1KldTw18QhGAZ4d9iH0= 2024-04-19T23:49:13.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:49:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NlQ1vw3o1KldTw18QhGAZ4d9iH0=', 'Content-Length': '0'} 2024-04-19T23:49:13.634 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002c8bc13064425a82-00662302f9-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:49:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:49:13.635 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-43 2024-04-19T23:49:13.635 DEBUG:boto:path=/aktlji-43/ 2024-04-19T23:49:13.635 DEBUG:boto:auth_path=/aktlji-43/ 2024-04-19T23:49:13.635 DEBUG:boto:Method: PUT 2024-04-19T23:49:13.635 DEBUG:boto:Path: /aktlji-43/ 2024-04-19T23:49:13.635 DEBUG:boto:Data: 2024-04-19T23:49:13.635 DEBUG:boto:Headers: {} 2024-04-19T23:49:13.635 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:49:13.635 DEBUG:boto:Port: 8000 2024-04-19T23:49:13.635 DEBUG:boto:Params: {} 2024-04-19T23:49:13.636 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:49:13.636 DEBUG:boto:Token: None 2024-04-19T23:49:13.636 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:49:13 GMT /aktlji-43/ 2024-04-19T23:49:13.636 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Y6Bz09C2QlItxMkZmUCYPaAPFeU= 2024-04-19T23:49:13.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:49:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y6Bz09C2QlItxMkZmUCYPaAPFeU=', 'Content-Length': '0'} 2024-04-19T23:49:13.702 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000de191e880de62143-00662302f9-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:49:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:49:13.702 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-44 2024-04-19T23:49:13.703 DEBUG:boto:path=/aktlji-44/ 2024-04-19T23:49:13.703 DEBUG:boto:auth_path=/aktlji-44/ 2024-04-19T23:49:13.703 DEBUG:boto:Method: PUT 2024-04-19T23:49:13.703 DEBUG:boto:Path: /aktlji-44/ 2024-04-19T23:49:13.703 DEBUG:boto:Data: 2024-04-19T23:49:13.703 DEBUG:boto:Headers: {} 2024-04-19T23:49:13.703 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:49:13.703 DEBUG:boto:Port: 8001 2024-04-19T23:49:13.703 DEBUG:boto:Params: {} 2024-04-19T23:49:13.704 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-19T23:49:13.704 DEBUG:boto:Token: None 2024-04-19T23:49:13.704 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:49:13 GMT /aktlji-44/ 2024-04-19T23:49:13.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cs02GKKGT24bVlhiOle5zAkwxTI= 2024-04-19T23:49:13.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:49:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cs02GKKGT24bVlhiOle5zAkwxTI=', 'Content-Length': '0'} 2024-04-19T23:49:13.777 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000af4dd71f0197ccf7-00662302f9-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:49:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:49:13.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:13.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:13.777 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:49:13.841 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:49:13.841 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:49:13.886 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:49:13.901 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:49:13.901 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:49:13.901 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:49:13.901 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:49:13.902 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:49:13.902 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:49:13.902 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:49:13.902 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:49:13.902 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:49:13.902 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:49:13.902 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713570553.765147_405.1", 2024-04-19T23:49:13.902 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:49:13.765147Z" 2024-04-19T23:49:13.906 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:49:13.906 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:49:13.906 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:49:13.906 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:49:13.906 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:49:13.906 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:49:13.907 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:13.907 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:49:13.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:13.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:13.908 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:49:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:49:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:49:14.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:14.026+0000 7f265e6ac780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:14.028+0000 7f265e6ac780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:14.047 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:49:14.047 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:49:14.047 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:49:14.048 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:49:14.048 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:49:14.048 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:49:14.048 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:14.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:14.048 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:49:14.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:14.048 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:49:14.048 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:14.048 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:14.049 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:49:14.049 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:14.049 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:49:14.049 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:14.049 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:49:14.049 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:14.049 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:14.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:14.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:14.050 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:49:14.050 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:14.050 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:14.050 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:49:14.050 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:14.050 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:14.050 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:14.050 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:49:14.050 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:14.050 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:14.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:14.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:14.051 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:49:14.051 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:14.051 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:14.051 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:49:14.051 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:14.051 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:14.051 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:14.052 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:49:14.052 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:14.052 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:14.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:14.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:14.052 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:49:14.052 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:14.052 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:14.052 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:49:14.052 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:14.053 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:14.053 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:14.053 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:49:14.053 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:14.053 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:14.053 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:14.053 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:49:14.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:14.053 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:49:14.053 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:49:14.053 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:49:14.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:14.054 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:49:14Z" 2024-04-19T23:49:14.054 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:49:14.055 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:49:14Z" } 2024-04-19T23:49:14.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:49:14.055 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:14.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:14.055 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.629562_306.1 2024-04-19T23:49:14.522 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-19T23:49:14.520+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-19T23:49:19.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:19.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:19.056 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:49:19.179 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:49:19.179 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:49:19.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:19.221+0000 7fcffc384780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:19.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:19.222+0000 7fcffc384780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:19.239 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:49:19.239 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:49:19.239 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:49:19.239 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:49:19.239 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:49:19.239 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:49:19.239 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:19.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:19.239 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:49:19.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:19.240 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:49:19.240 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:19.240 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:19.240 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:49:19.240 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:19.240 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:49:19.240 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:19.240 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:49:19.240 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:19.240 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:49:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:49:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:49:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:19.241 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:49:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.606676_302.1", 2024-04-19T23:49:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-19T23:49:19.242 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:19.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:19.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:19.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:19.243 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:49:19.243 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:19.243 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:19.243 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:49:19.243 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:19.243 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:19.243 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:19.243 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:49:19.243 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:19.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:19.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:19.244 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:49:19.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:19.244 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:49:19.244 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:49:19.244 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:49:19.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:19.244 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:49:19Z" 2024-04-19T23:49:19.244 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:49:19.245 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.606676_302.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.606676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:49:19Z" } 2024-04-19T23:49:19.246 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:49:19.246 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:19.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:19.246 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.606676_302.1 2024-04-19T23:49:24.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:24.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:24.248 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:49:24.304 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:49:24.304 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:49:24.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:24.340+0000 7f7cea89c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:24.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:24.341+0000 7f7cea89c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:24.357 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:49:24.357 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:49:24.357 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:49:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:49:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:49:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:49:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:49:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:49:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:49:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:49:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:49:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:49:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:49:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:49:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:49:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:49:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:49:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:49:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:49:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:49:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:49:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:49:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:49:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:49:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:49:24Z" 2024-04-19T23:49:24.363 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:49:24.364 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:49:24Z" } 2024-04-19T23:49:24.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:49:24.365 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:24.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:24.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.576049_299.1 2024-04-19T23:49:29.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:29.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:29.367 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:49:29.426 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:49:29.426 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:49:29.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:29.463+0000 7fedbda88780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:29.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:29.464+0000 7fedbda88780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:29.479 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:49:29.479 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:49:29.480 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:49:29.480 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:49:29.480 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:49:29.480 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:49:29.480 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:29.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:29.480 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:49:29.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:29.480 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:49:29.480 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:29.480 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:29.480 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:49:29.481 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:29.481 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:49:29.481 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:29.481 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:49:29.481 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:29.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:29.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:29.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:29.481 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:49:29.481 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:29.481 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:29.481 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:49:29.482 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:29.482 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:29.482 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:29.482 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:49:29.482 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:29.482 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:29.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:29.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:29.482 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:49:29.482 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:29.483 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:29.483 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.555632_296.1", 2024-04-19T23:49:29.483 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:29.483 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:29.483 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:29.483 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-19T23:49:29.483 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:29.483 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:29.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:29.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:29.483 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:49:29.484 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:29.484 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:29.484 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:49:29.484 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:29.484 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:29.484 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:29.484 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:49:29.484 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:29.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:29.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:29.484 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:49:29.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:29.485 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:49:29.485 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:49:29.485 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:49:29.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:29.485 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:49:29Z" 2024-04-19T23:49:29.485 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:49:29.486 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.555632_296.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.555632Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:49:29Z" } 2024-04-19T23:49:29.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:49:29.486 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:29.486 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:29.486 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.555632_296.1 2024-04-19T23:49:34.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:34.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:34.488 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:49:34.547 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:49:34.547 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:49:34.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:34.587+0000 7f1ffee77780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:34.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:34.588+0000 7f1ffee77780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:34.604 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:49:34.604 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:49:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:49:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:49:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:49:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:49:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:49:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:49:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:34.605 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:49:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:49:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:49:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:49:34.606 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:49:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:49:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:49:34.607 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.580723_301.1", 2024-04-19T23:49:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-19T23:49:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:34.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:49:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:49:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:49:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:34.609 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:49:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:49:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:49:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:49:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:34.610 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:49:34Z" 2024-04-19T23:49:34.610 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:49:34.611 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.580723_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.580723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:49:34Z" } 2024-04-19T23:49:34.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:49:34.611 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:34.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:34.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.580723_301.1 2024-04-19T23:49:39.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:39.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:39.613 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:49:39.673 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:49:39.673 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:49:39.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:39.709+0000 7fd8cb34a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:39.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:39.710+0000 7fd8cb34a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:39.725 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:49:39.726 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:49:39.726 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:49:39.726 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:49:39.726 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:49:39.726 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:49:39.726 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:39.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:39.726 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:49:39.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:39.727 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:49:39.727 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:39.727 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:39.727 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:49:39.727 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:39.727 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:49:39.727 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:39.727 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:49:39.727 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:39.727 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:39.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:39.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:39.728 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:49:39.728 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:39.728 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:39.728 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:49:39.728 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:39.728 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:39.728 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:39.728 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:49:39.728 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:39.728 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:39.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:39.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:39.729 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:49:39.729 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:39.729 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:39.729 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.580723_301.1", 2024-04-19T23:49:39.729 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:39.729 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:39.729 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:39.729 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-19T23:49:39.729 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:39.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:39.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:49:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:49:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:49:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:49:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:49:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:49:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:49:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:49:39Z" 2024-04-19T23:49:39.731 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:49:39.732 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.580723_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.580723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:49:39Z" } 2024-04-19T23:49:39.732 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:49:39.732 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:39.733 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:39.733 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.580723_301.1 2024-04-19T23:49:44.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:44.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:44.734 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:49:44.794 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:49:44.794 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:49:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:44.830+0000 7fc3b3042780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:44.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:44.831+0000 7fc3b3042780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:44.918 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:49:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:49:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:49:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:49:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:49:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:49:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:44.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:49:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:49:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:49:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:49:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:44.920 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:49:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:49:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:44.921 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:49:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:49:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:44.922 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:49:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:49:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:49:44.923 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:49:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:49:44.924 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:49:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:49:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:44.925 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:49:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:49:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:49:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:44.926 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:49:44Z" 2024-04-19T23:49:44.926 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:49:44.926 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:49:44Z" } 2024-04-19T23:49:44.926 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:49:44.927 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:44.927 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:44.927 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-19T23:49:49.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:49.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:49.929 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:49:49.989 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:49:49.990 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:49:50.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:50.026+0000 7fb7a6811780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:50.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:50.027+0000 7fb7a6811780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:50.043 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:49:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:49:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:49:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:49:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:49:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:49:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:50.043 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:49:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:49:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:49:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:49:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:49:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:50.044 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:49:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:49:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:49:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:50.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:49:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.606676_302.1", 2024-04-19T23:49:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:50.046 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:49:50.047 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:49:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:49:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:49:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:49:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:50.048 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:49:50Z" 2024-04-19T23:49:50.048 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:49:50.049 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.606676_302.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.606676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:49:50Z" } 2024-04-19T23:49:50.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:49:50.050 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:50.050 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:50.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.606676_302.1 2024-04-19T23:49:55.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:55.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:49:55.051 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:49:55.110 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:49:55.110 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:49:55.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:55.146+0000 7ff6c9a4f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:55.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:49:55.147+0000 7ff6c9a4f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:49:55.163 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:49:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:49:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:49:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:49:55.163 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:49:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:49:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:49:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:49:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:49:55.164 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:49:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:49:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:49:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:55.165 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:49:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:49:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:49:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:55.166 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.808187_326.1", 2024-04-19T23:49:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-19T23:49:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:49:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:49:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:49:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:49:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:49:55.169 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:49:55Z" 2024-04-19T23:49:55.169 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:49:55.169 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.808187_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.808187Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:49:55Z" } 2024-04-19T23:49:55.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.808187_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.808187Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:49:55.170 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:55.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.808187_326.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:49:55.170 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.808187_326.1 2024-04-19T23:50:00.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:00.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:00.172 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:50:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:50:00.231 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:50:00.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:00.267+0000 7f09780fd780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:00.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:00.268+0000 7f09780fd780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:00.284 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:50:00.284 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:50:00.284 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:50:00.285 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:50:00.285 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:50:00.285 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:50:00.285 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:00.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:00.285 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:50:00.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:00.285 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:50:00.285 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:00.285 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:00.285 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:50:00.286 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:00.286 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:50:00.286 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:00.286 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:50:00.286 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:00.286 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:00.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:00.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:00.286 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:50:00.286 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:00.286 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:00.287 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:50:00.287 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:00.287 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:00.287 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:00.287 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:50:00.287 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:00.287 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:00.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:00.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:00.287 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:50:00.287 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:00.288 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:00.288 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.614143_305.1", 2024-04-19T23:50:00.288 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:00.288 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:00.288 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:00.288 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-19T23:50:00.288 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:00.288 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:00.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:00.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:00.288 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:50:00.289 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:00.289 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:00.289 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:50:00.289 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:00.289 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:00.289 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:00.289 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:50:00.289 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:00.289 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:00.289 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:00.289 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:50:00.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:00.290 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:50:00.290 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:50:00.290 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:50:00.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:00.290 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:50:00Z" 2024-04-19T23:50:00.290 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:50:00.291 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.614143_305.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.614143Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:50:00Z" } 2024-04-19T23:50:00.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:50:00.291 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:00.291 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:00.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.614143_305.1 2024-04-19T23:50:05.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:05.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:05.292 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:50:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:50:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:50:05.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:05.386+0000 7f9d1faa4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:05.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:05.387+0000 7f9d1faa4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:05.403 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:50:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:50:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:50:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:50:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:50:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:50:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:50:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:50:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:50:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:50:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:50:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:50:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:50:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:50:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:50:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.609968_303.1", 2024-04-19T23:50:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-19T23:50:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:50:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:50:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:05.408 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:50:05.408 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:05.408 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:05.408 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:05.408 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:50:05.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:05.408 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:50:05.408 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:50:05.408 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:50:05.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:05.408 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:50:05Z" 2024-04-19T23:50:05.409 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:50:05.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.609968_303.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.609968Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:50:05Z" } 2024-04-19T23:50:05.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:50:05.410 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:05.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:05.410 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.609968_303.1 2024-04-19T23:50:10.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:10.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:10.412 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:50:10.471 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:50:10.472 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:50:10.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:10.509+0000 7fbe8f642780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:10.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:10.511+0000 7fbe8f642780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:10.527 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:50:10.527 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:50:10.527 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:50:10.528 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:50:10.528 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:50:10.528 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:50:10.528 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:10.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:10.528 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:50:10.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:10.528 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:50:10.528 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:10.528 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:10.528 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:50:10.529 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:10.529 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:50:10.529 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:10.529 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:50:10.529 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:10.529 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:10.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:10.529 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:10.529 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:50:10.529 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:10.529 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:10.530 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:50:10.530 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:10.530 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:10.530 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:10.530 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:50:10.530 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:10.530 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:10.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:10.531 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:10.531 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:50:10.531 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:10.531 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:10.531 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.580723_301.1", 2024-04-19T23:50:10.531 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:10.531 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:10.531 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:10.531 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-19T23:50:10.531 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:10.531 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:10.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:10.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:10.532 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:50:10.532 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:10.532 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:10.532 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:50:10.532 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:10.532 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:10.532 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:10.532 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:50:10.532 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:10.533 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:10.533 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:10.533 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:50:10.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:10.533 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:50:10.533 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:50:10.533 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:50:10.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:10.533 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:50:10Z" 2024-04-19T23:50:10.533 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:50:10.534 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.580723_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.580723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:50:10Z" } 2024-04-19T23:50:10.535 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:50:10.535 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:10.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:10.535 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.580723_301.1 2024-04-19T23:50:15.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:15.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:15.536 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:50:15.598 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:50:15.598 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:50:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:15.635+0000 7f41469a4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:15.636+0000 7f41469a4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:15.653 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:50:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:50:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:50:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:50:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:50:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:50:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:15.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:50:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:50:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:50:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:50:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:50:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:15.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:50:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:50:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:50:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:15.655 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:50:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-19T23:50:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-19T23:50:15.656 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:50:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:50:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:50:15.657 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:50:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:50:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:50:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:50:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:15.658 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:50:15Z" 2024-04-19T23:50:15.658 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:50:15.659 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:50:15Z" } 2024-04-19T23:50:15.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:50:15.660 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:15.660 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:15.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.797017_325.1 2024-04-19T23:50:20.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:20.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:20.661 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:50:20.720 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:50:20.720 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:50:20.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:20.754+0000 7f205cf8a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:20.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:20.755+0000 7f205cf8a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:20.772 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:50:20.772 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:50:20.772 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:50:20.772 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:50:20.772 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:50:20.773 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:50:20.773 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:20.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:20.773 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:50:20.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:20.773 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:50:20.773 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:20.773 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:20.773 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:50:20.773 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:20.773 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:50:20.774 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:20.774 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:50:20.774 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:20.774 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:20.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:20.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:20.774 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:50:20.774 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:20.774 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:20.774 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:50:20.775 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:20.775 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:20.775 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:20.775 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:50:20.775 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:20.775 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:20.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:20.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:20.775 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:50:20.775 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:20.775 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:20.776 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570244.063607_345.1", 2024-04-19T23:50:20.776 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:20.776 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:20.776 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:20.776 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:04.063607Z", 2024-04-19T23:50:20.776 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:20.776 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:20.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:20.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:20.776 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:50:20.776 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:20.777 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:20.777 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:50:20.777 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:20.777 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:20.777 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:20.777 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:50:20.777 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:20.777 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:20.777 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:20.777 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:50:20.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:20.778 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:50:20.778 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:50:20.778 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:50:20.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:20.778 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:50:20Z" 2024-04-19T23:50:20.778 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:50:20.779 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570244.063607_345.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:04.063607Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:50:20Z" } 2024-04-19T23:50:20.779 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570244.063607_345.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:04.063607Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:50:20.779 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:20.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570244.063607_345.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:20.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570244.063607_345.1 2024-04-19T23:50:25.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:25.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:25.782 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:50:25.864 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:50:25.864 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:50:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:25.900+0000 7f5095aa4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:25.901+0000 7f5095aa4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:25.917 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:50:25.918 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:50:25.918 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:50:25.918 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:50:25.918 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:50:25.918 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:50:25.918 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:25.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:25.918 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:50:25.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:25.918 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:50:25.919 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:25.919 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:25.919 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:50:25.919 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:25.919 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:50:25.919 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:25.919 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:50:25.919 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:25.919 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:25.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:25.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:25.920 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:50:25.920 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:25.920 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:25.920 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:50:25.920 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:25.920 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:25.920 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:25.920 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:50:25.920 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:25.920 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:25.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:25.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:25.921 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:50:25.921 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:25.921 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:25.921 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.698420_310.1", 2024-04-19T23:50:25.921 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:25.921 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:25.921 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:25.921 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-19T23:50:25.921 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:25.921 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:25.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:25.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:25.922 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:50:25.922 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:25.922 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:25.922 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:50:25.922 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:25.922 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:25.922 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:25.922 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:50:25.922 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:25.923 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:25.923 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:25.923 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:50:25.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:25.923 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:50:25.923 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:50:25.923 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:50:25.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:25.923 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:50:25Z" 2024-04-19T23:50:25.923 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:50:25.924 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.698420_310.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.698420Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:50:25Z" } 2024-04-19T23:50:25.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.698420_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.698420Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:50:25.925 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:25.925 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.698420_310.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:25.925 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.698420_310.1 2024-04-19T23:50:30.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:30.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:30.927 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:50:31.004 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:50:31.004 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:50:31.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:31.040+0000 7f00d3c62780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:31.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:31.041+0000 7f00d3c62780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:31.057 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:50:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:50:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:50:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:50:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:50:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:50:31.057 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:31.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:31.058 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:50:31.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:31.058 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:50:31.058 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:31.058 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:31.058 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:50:31.058 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:31.058 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:50:31.058 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:31.058 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:50:31.059 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:31.059 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:31.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:31.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:31.059 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:50:31.059 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:31.059 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:31.059 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:50:31.059 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:31.059 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:31.059 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:31.059 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:50:31.060 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:31.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:31.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:31.060 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:31.060 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:50:31.060 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:31.060 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:31.060 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.572795_298.1", 2024-04-19T23:50:31.060 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:31.060 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:31.060 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:31.061 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-19T23:50:31.061 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:31.061 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:31.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:31.061 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:31.061 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:50:31.061 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:31.061 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:31.061 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:50:31.061 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:31.061 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:31.062 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:31.062 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:50:31.062 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:31.062 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:31.062 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:31.062 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:50:31.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:31.062 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:50:31.062 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:50:31.062 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:50:31.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:31.063 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:50:31Z" 2024-04-19T23:50:31.063 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:50:31.064 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.572795_298.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.572795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:50:31Z" } 2024-04-19T23:50:31.064 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:50:31.064 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:31.064 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:31.064 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.572795_298.1 2024-04-19T23:50:36.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:36.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:36.066 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:50:36.125 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:50:36.125 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:50:36.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:36.161+0000 7fd1aac5c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:36.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:36.162+0000 7fd1aac5c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:36.179 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:50:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:50:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:50:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:50:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:50:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:50:36.179 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:50:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:50:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:50:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:50:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:36.180 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:50:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:50:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:50:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:36.181 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:50:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:50:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.701491_311.1", 2024-04-19T23:50:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:36.182 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.701491Z", 2024-04-19T23:50:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:50:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:50:36.183 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:50:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:50:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:50:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:50:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:50:36.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:36.185 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:50:36Z" 2024-04-19T23:50:36.185 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:50:36.185 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.701491_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.701491Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:50:36Z" } 2024-04-19T23:50:36.186 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.701491_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.701491Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:50:36.186 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:36.186 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.701491_311.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:36.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.701491_311.1 2024-04-19T23:50:41.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:41.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:41.187 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:50:41.246 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:50:41.246 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:50:41.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:41.282+0000 7f243c0b8780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:41.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:41.284+0000 7f243c0b8780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:41.300 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:50:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:50:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:50:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:50:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:50:41.300 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:50:41.301 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:50:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:50:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:50:41.302 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:50:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.717624_312.1", 2024-04-19T23:50:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-19T23:50:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:50:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:50:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:50:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:50:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:50:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:50:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:50:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:41.305 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:50:41Z" 2024-04-19T23:50:41.305 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:50:41.306 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.717624_312.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.717624Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:50:41Z" } 2024-04-19T23:50:41.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.717624_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.717624Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:50:41.306 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:41.306 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.717624_312.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:41.306 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.717624_312.1 2024-04-19T23:50:46.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:46.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:46.308 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:50:46.369 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:50:46.369 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:50:46.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:46.405+0000 7f8daa416780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:46.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:46.406+0000 7f8daa416780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:46.422 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:50:46.422 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:50:46.422 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:50:46.422 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:50:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:50:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:50:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:50:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:50:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:50:46.423 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:46.424 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:50:46.424 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:46.424 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:50:46.424 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:46.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:46.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:46.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:46.424 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:50:46.424 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:46.424 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:46.424 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:50:46.424 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:46.425 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:46.425 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:46.425 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:50:46.425 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:46.425 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:46.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:46.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:46.425 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:50:46.425 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:46.425 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:46.425 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.769121_320.1", 2024-04-19T23:50:46.425 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:46.426 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:46.426 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:46.426 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.769121Z", 2024-04-19T23:50:46.426 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:46.426 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:46.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:46.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:46.426 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:50:46.426 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:46.426 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:46.426 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:50:46.427 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:46.427 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:46.427 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:46.427 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:50:46.427 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:46.427 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:46.427 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:46.427 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:50:46.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:46.427 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:50:46.427 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:50:46.428 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:50:46.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:46.428 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:50:46Z" 2024-04-19T23:50:46.428 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:50:46.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.769121_320.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.769121Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:50:46Z" } 2024-04-19T23:50:46.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.769121_320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.769121Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:50:46.429 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:46.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.769121_320.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:46.429 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.769121_320.1 2024-04-19T23:50:51.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:51.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:51.431 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:50:51.490 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:50:51.490 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:50:51.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:51.525+0000 7fbc69b11780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:51.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:51.526+0000 7fbc69b11780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:51.543 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:50:51.543 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:50:51.543 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:50:51.543 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:50:51.544 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:50:51.544 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:50:51.544 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:51.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:51.544 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:50:51.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:51.544 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:50:51.544 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:51.544 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:51.544 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:50:51.544 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:51.544 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:50:51.545 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:51.545 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:50:51.545 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:51.545 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:51.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:51.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:51.545 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:50:51.545 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:51.545 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:51.545 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:50:51.545 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:51.545 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:51.546 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:51.546 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:50:51.546 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:51.546 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:51.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:51.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:51.546 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:50:51.546 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:51.547 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:51.547 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.687754_309.1", 2024-04-19T23:50:51.547 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:51.547 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:51.547 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:51.547 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-19T23:50:51.547 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:51.547 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:51.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:51.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:51.547 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:50:51.547 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:51.548 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:51.548 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:50:51.548 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:51.548 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:51.548 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:51.548 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:50:51.548 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:51.548 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:51.548 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:51.548 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:50:51.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:51.549 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:50:51.549 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:50:51.549 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:50:51.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:51.549 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:50:51Z" 2024-04-19T23:50:51.549 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:50:51.550 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.687754_309.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.687754Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:50:51Z" } 2024-04-19T23:50:51.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.687754_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.687754Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:50:51.550 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:51.550 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.687754_309.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:51.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.687754_309.1 2024-04-19T23:50:56.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:56.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:50:56.552 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:50:56.613 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:50:56.613 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:50:56.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:56.650+0000 7fd4facbd780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:56.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:50:56.651+0000 7fd4facbd780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:50:56.667 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:50:56.668 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:50:56.668 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:50:56.668 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:50:56.668 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:50:56.668 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:50:56.668 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:56.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:56.668 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:50:56.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:56.669 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:50:56.669 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:56.669 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:56.669 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:50:56.669 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:56.669 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:50:56.669 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:56.669 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:50:56.669 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:56.669 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:56.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:56.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:56.670 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:50:56.670 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:56.670 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:56.670 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:50:56.670 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:56.670 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:56.670 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:56.670 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:50:56.670 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:56.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:56.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:56.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:56.671 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:50:56.671 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:56.671 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:56.671 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:50:56.671 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:56.671 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:56.671 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:56.672 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:50:56.672 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:56.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:56.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:56.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:50:56.672 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:50:56.672 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:50:56.672 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:50:56.672 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:50:56.672 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:50:56.672 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:50:56.672 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:50:56.673 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:50:56.673 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:50:56.673 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:56.673 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:50:56.673 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:50:56.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:56.673 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:50:56.673 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:50:56.673 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:50:56.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:50:56.673 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:50:56Z" 2024-04-19T23:50:56.673 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:50:56.674 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:50:56Z" } 2024-04-19T23:50:56.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:50:56.675 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:56.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:50:56.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.629562_306.1 2024-04-19T23:51:01.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:01.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:01.677 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:51:01.779 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:51:01.779 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:51:01.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:01.815+0000 7fc794cd9780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:01.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:01.816+0000 7fc794cd9780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:01.835 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:51:01.835 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:51:01.835 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:51:01.835 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:51:01.835 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:51:01.835 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:51:01.835 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:01.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:01.835 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:51:01.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:01.836 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:51:01.836 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:01.836 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:01.836 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:51:01.836 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:01.836 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:51:01.836 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:01.836 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:51:01.836 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:01.836 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:01.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:01.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:01.837 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:51:01.837 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:01.837 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:01.837 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:51:01.837 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:01.837 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:01.837 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:01.837 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:51:01.837 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:01.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:51:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.580723_301.1", 2024-04-19T23:51:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-19T23:51:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:01.838 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:01.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:01.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:01.839 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:51:01.839 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:01.839 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:01.839 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:51:01.839 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:01.839 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:01.839 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:01.839 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:51:01.839 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:01.839 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:01.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:01.840 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:51:01.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:01.840 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:51:01.840 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:51:01.840 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:51:01.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:01.840 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:51:01Z" 2024-04-19T23:51:01.840 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:51:01.841 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.580723_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.580723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:51:01Z" } 2024-04-19T23:51:01.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:51:01.842 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:01.842 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:01.842 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.580723_301.1 2024-04-19T23:51:06.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:06.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:06.843 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:51:06.902 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:51:06.902 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:51:06.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:06.939+0000 7f4d6aa88780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:06.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:06.940+0000 7f4d6aa88780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:06.956 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:51:06.956 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:51:06.956 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:51:06.956 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:51:06.956 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:51:06.956 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:51:06.956 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:06.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:06.957 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:51:06.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:06.957 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:51:06.957 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:06.957 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:06.957 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:51:06.957 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:06.957 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:51:06.957 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:06.957 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:51:06.958 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:06.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:06.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:06.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:06.958 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:51:06.958 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:06.958 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:06.958 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:51:06.958 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:06.958 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:06.958 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:06.959 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:51:06.959 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:06.959 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:06.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:06.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:06.959 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:51:06.959 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:06.959 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:06.959 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.725126_315.1", 2024-04-19T23:51:06.959 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:06.959 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:06.960 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:06.960 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.725126Z", 2024-04-19T23:51:06.960 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:06.960 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:06.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:06.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:06.960 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:51:06.960 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:06.960 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:06.960 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:51:06.960 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:06.961 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:06.961 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:06.961 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:51:06.961 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:06.961 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:06.961 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:06.961 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:51:06.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:06.961 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:51:06.962 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:51:06.962 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:51:06.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:06.962 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:51:06Z" 2024-04-19T23:51:06.962 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:51:06.963 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.725126_315.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.725126Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:51:06Z" } 2024-04-19T23:51:06.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.725126_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.725126Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:51:06.963 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:06.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.725126_315.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:06.963 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.725126_315.1 2024-04-19T23:51:11.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:11.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:11.965 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:51:12.023 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:51:12.024 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:51:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:12.058+0000 7fc782330780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:12.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:12.059+0000 7fc782330780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:12.076 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:51:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:51:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:51:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:51:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:51:12.076 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:51:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:51:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:51:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:51:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:51:12.077 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:51:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:51:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:51:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:12.078 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:51:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:51:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.606676_302.1", 2024-04-19T23:51:12.079 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-19T23:51:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:51:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:12.080 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:51:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:51:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:51:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:12.081 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:51:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:51:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:51:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:12.082 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:51:12Z" 2024-04-19T23:51:12.082 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:51:12.083 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.606676_302.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.606676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:51:12Z" } 2024-04-19T23:51:12.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:51:12.083 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:12.083 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:12.083 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.606676_302.1 2024-04-19T23:51:17.084 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:17.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:17.084 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:51:17.145 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:51:17.146 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:51:17.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:17.184+0000 7f170feab780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:17.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:17.185+0000 7f170feab780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:17.202 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:51:17.202 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:51:17.202 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:51:17.203 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:51:17.203 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:51:17.203 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:51:17.203 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:17.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:17.203 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:51:17.203 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:17.203 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:51:17.203 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:17.204 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:17.204 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:51:17.204 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:17.204 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:51:17.204 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:17.204 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:51:17.204 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:17.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:17.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:17.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:17.205 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:51:17.205 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:17.205 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:17.205 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:51:17.205 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:17.205 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:17.205 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:17.206 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:51:17.206 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:17.206 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:17.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:17.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:17.206 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:51:17.206 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:17.206 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:17.207 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.725126_315.1", 2024-04-19T23:51:17.207 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:17.207 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:17.207 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:17.207 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.725126Z", 2024-04-19T23:51:17.207 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:17.207 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:17.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:17.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:17.207 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:51:17.207 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:17.208 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:17.208 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:51:17.208 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:17.208 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:17.208 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:17.208 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:51:17.208 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:17.208 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:17.208 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:17.209 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:51:17.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:17.209 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:51:17.209 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:51:17.209 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:51:17.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:17.209 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:51:17Z" 2024-04-19T23:51:17.210 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:51:17.211 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.725126_315.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.725126Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:51:17Z" } 2024-04-19T23:51:17.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.725126_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.725126Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:51:17.211 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:17.211 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.725126_315.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:17.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.725126_315.1 2024-04-19T23:51:22.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:22.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:22.213 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:51:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:51:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:51:22.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:22.308+0000 7f768de75780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:22.309+0000 7f768de75780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:22.327 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:51:22.327 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:51:22.327 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:51:22.327 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:51:22.327 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:51:22.327 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:51:22.327 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:22.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:22.328 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:51:22.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:22.328 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:51:22.328 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:22.328 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:22.328 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:51:22.328 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:22.328 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:51:22.328 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:22.328 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:51:22.329 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:22.329 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:22.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:22.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:22.329 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:51:22.329 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:22.329 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:22.329 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:51:22.329 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:22.329 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:22.330 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:22.330 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:51:22.330 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:22.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:22.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:22.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:22.330 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:51:22.330 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:22.330 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:22.330 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:51:22.330 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:22.330 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:22.331 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:22.331 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:51:22.331 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:22.331 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:22.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:22.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:22.331 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:51:22.331 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:22.331 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:22.331 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:51:22.331 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:22.331 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:51:22Z" 2024-04-19T23:51:22.332 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:51:22.333 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:51:22Z" } 2024-04-19T23:51:22.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:51:22.334 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:22.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:22.334 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.559046_297.1 2024-04-19T23:51:27.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:27.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:27.336 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:51:27.414 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:51:27.414 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:51:27.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:27.546+0000 7f84a4f06780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:27.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:27.548+0000 7f84a4f06780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:27.566 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:51:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:51:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:51:27.566 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:51:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:51:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:51:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:51:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:51:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:51:27.567 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:27.568 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.609968_303.1", 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:27.569 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-19T23:51:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:51:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:51:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:51:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:51:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:51:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:51:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:51:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:27.571 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:51:27Z" 2024-04-19T23:51:27.572 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:51:27.572 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.609968_303.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.609968Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:51:27Z" } 2024-04-19T23:51:27.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:51:27.573 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:27.573 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:27.573 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.609968_303.1 2024-04-19T23:51:32.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:32.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:32.574 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:51:32.635 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:51:32.635 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:51:32.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:32.670+0000 7f3a733ca780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:32.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:32.671+0000 7f3a733ca780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:32.689 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:51:32.689 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:51:32.689 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:51:32.689 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:51:32.690 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:51:32.690 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:51:32.690 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:32.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:32.690 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:51:32.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:32.691 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:51:32.691 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:32.691 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:32.691 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:51:32.691 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:32.692 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:51:32.692 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:32.692 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:51:32.692 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:32.692 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:32.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:32.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:32.693 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:51:32.693 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:32.693 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:32.693 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:51:32.693 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:32.694 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:32.694 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:32.694 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:51:32.694 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:32.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:32.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:32.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:32.695 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:51:32.695 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:32.695 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:32.696 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:51:32.696 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:32.696 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:32.696 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:32.696 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:51:32.696 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:32.696 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:32.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:32.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:32.697 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:51:32.697 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:32.697 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:32.697 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:51:32.697 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:32.697 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:32.697 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:32.698 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:51:32.698 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:32.698 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:32.698 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:32.698 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:51:32.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:32.699 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:51:32.699 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:51:32.699 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:51:32.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:32.699 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:51:32Z" 2024-04-19T23:51:32.699 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:51:32.701 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:51:32Z" } 2024-04-19T23:51:32.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:51:32.701 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:32.702 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:32.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-19T23:51:37.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:37.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:37.703 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:51:37.763 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:51:37.764 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:51:37.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:37.800+0000 7f897f354780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:37.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:37.801+0000 7f897f354780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:37.817 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:51:37.817 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:51:37.817 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:51:37.817 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:51:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:51:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:51:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:51:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:51:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:37.818 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:51:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:51:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:51:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:51:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:37.819 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:51:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:51:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:51:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:37.820 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:51:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:51:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:51:37.821 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:51:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:51:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:37.822 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:51:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:51:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:51:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:51:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:37.823 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:51:37Z" 2024-04-19T23:51:37.823 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:51:37.824 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:51:37Z" } 2024-04-19T23:51:37.824 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:51:37.824 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:37.824 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:37.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-19T23:51:42.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:42.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:42.826 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:51:42.886 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:51:42.932 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:51:42.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:42.922+0000 7f90b1bfd780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:42.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:42.923+0000 7f90b1bfd780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:42.940 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:51:42.940 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:51:42.940 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:51:42.940 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:51:42.940 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:51:42.940 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:51:42.940 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:42.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:42.940 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:51:42.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:42.941 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:51:42.941 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:42.941 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:42.941 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:51:42.941 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:42.941 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:51:42.941 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:42.941 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:51:42.941 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:42.941 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:42.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:42.942 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:42.942 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:51:42.942 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:42.942 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:42.942 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:51:42.942 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:42.942 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:42.942 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:42.942 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:51:42.942 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.577447_300.1", 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:42.943 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:42.944 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:42.945 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:51:42.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:42.945 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:51:42.945 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:51:42.945 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:51:42.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:42.945 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:51:42Z" 2024-04-19T23:51:42.945 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:51:42.946 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.577447_300.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.577447Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:51:42Z" } 2024-04-19T23:51:42.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:51:42.946 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:42.946 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:42.947 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.577447_300.1 2024-04-19T23:51:47.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:47.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:47.948 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:51:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:51:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:51:48.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:48.044+0000 7f2b6648b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:48.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:48.045+0000 7f2b6648b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:48.061 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:51:48.061 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:51:48.061 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:51:48.062 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:51:48.062 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:51:48.062 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:51:48.062 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:48.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:48.062 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:51:48.062 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:48.062 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:51:48.062 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:48.062 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:48.062 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:51:48.063 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:48.063 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:51:48.063 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:48.063 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:51:48.063 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:48.063 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:48.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:48.063 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:48.063 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:51:48.063 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:48.063 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:48.063 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:51:48.064 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:48.064 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:48.064 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:48.064 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:51:48.064 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:48.064 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:48.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:48.064 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:48.064 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:51:48.064 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:48.064 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:48.064 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.572795_298.1", 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:51:48.065 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:48.066 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:48.066 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:48.066 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:51:48.066 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:48.066 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:48.066 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:48.066 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:51:48.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:48.066 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:51:48.066 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:51:48.066 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:51:48.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:48.067 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:51:48Z" 2024-04-19T23:51:48.067 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:51:48.067 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.572795_298.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.572795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:51:48Z" } 2024-04-19T23:51:48.068 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:51:48.068 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:48.068 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:48.068 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.572795_298.1 2024-04-19T23:51:53.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:53.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:53.069 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:51:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:51:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:51:53.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:53.166+0000 7f3314e24780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:53.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:53.167+0000 7f3314e24780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:53.184 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:51:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:51:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:51:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:51:53.184 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:51:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:51:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:51:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:51:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:51:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:51:53.185 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:51:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:51:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:51:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:53.186 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.738086_316.1", 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:53.187 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:53.188 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:51:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:51:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:51:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:51:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:51:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:53.189 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:51:53Z" 2024-04-19T23:51:53.189 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:51:53.190 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.738086_316.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.738086Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:51:53Z" } 2024-04-19T23:51:53.191 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:51:53.191 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:53.191 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:53.191 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.738086_316.1 2024-04-19T23:51:58.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:58.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:51:58.192 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:51:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:51:58.292 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:51:58.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:58.368+0000 7f697e645780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:58.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:51:58.370+0000 7f697e645780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:51:58.385 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:51:58.385 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:51:58.386 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:51:58.386 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:51:58.386 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:51:58.386 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:51:58.386 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:58.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:58.386 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:51:58.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:58.386 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:51:58.386 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:58.386 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:58.387 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:51:58.387 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:58.387 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:51:58.387 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:58.387 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:51:58.387 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:58.387 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:58.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:58.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:58.387 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:51:58.387 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:58.388 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.572795_298.1", 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:51:58.389 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:51:58.390 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:51:58.390 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:51:58.390 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:51:58.390 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:51:58.390 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:51:58.390 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:58.390 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:51:58.390 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:51:58.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:58.390 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:51:58.390 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:51:58.391 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:51:58.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:51:58.391 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:51:58Z" 2024-04-19T23:51:58.391 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:51:58.392 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.572795_298.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.572795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:51:58Z" } 2024-04-19T23:51:58.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:51:58.392 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:58.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:51:58.392 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.572795_298.1 2024-04-19T23:52:03.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:03.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:03.394 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:52:03.452 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:52:03.452 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:52:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:03.489+0000 7f3172ad0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:03.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:03.490+0000 7f3172ad0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:03.508 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:52:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:52:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:52:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:52:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:52:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:52:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:52:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:52:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:52:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:52:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:52:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:52:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:52:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:52:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:52:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:52:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:52:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:52:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:52:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:52:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:52:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:52:03Z" 2024-04-19T23:52:03.513 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:52:03.514 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:52:03Z" } 2024-04-19T23:52:03.514 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:52:03.514 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:03.515 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:03.515 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-19T23:52:08.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:08.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:08.516 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:52:08.575 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:52:08.575 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:52:08.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:08.615+0000 7f3a10365780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:08.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:08.617+0000 7f3a10365780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:08.634 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:52:08.634 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:52:08.634 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:52:08.634 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:52:08.634 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:52:08.634 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:52:08.634 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:08.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:08.634 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:52:08.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:08.635 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:52:08.635 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:08.635 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:08.635 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:52:08.635 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:08.635 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:52:08.635 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:08.635 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:52:08.635 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:08.635 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:08.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:08.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:08.636 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:52:08.636 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:08.636 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:08.636 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:52:08.636 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:08.636 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:08.636 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:08.636 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:52:08.636 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:08.637 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:08.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:08.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:08.637 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:52:08.637 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:08.637 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:08.637 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.687754_309.1", 2024-04-19T23:52:08.637 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:08.637 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:08.637 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:08.637 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-19T23:52:08.637 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:08.638 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:08.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:08.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:08.638 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:52:08.638 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:08.638 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:08.638 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:52:08.638 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:08.638 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:08.638 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:08.638 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:52:08.638 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:08.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:08.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:08.639 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:52:08.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:08.639 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:52:08.639 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:52:08.639 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:52:08.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:08.639 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:52:08Z" 2024-04-19T23:52:08.639 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:52:08.640 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.687754_309.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.687754Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:52:08Z" } 2024-04-19T23:52:08.640 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.687754_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.687754Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:52:08.640 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:08.641 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.687754_309.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:08.641 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.687754_309.1 2024-04-19T23:52:13.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:13.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:13.642 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:52:13.701 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:52:13.701 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:52:13.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:13.736+0000 7fc6dcb6d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:13.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:13.738+0000 7fc6dcb6d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:13.754 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:52:13.754 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:52:13.755 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:52:13.755 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:52:13.755 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:52:13.755 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:52:13.755 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:13.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:13.755 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:52:13.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:13.755 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:52:13.755 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:13.755 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:13.755 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:52:13.756 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:13.757 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:13.757 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:13.757 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:52:13.757 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:13.757 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:13.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:13.757 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:13.757 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:52:13.757 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:13.757 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:13.757 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-19T23:52:13.757 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:13.758 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:13.758 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:13.758 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-19T23:52:13.758 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:13.758 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:13.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:13.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:13.758 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:52:13.758 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:13.758 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:13.758 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:52:13.758 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:13.759 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:52:13Z" 2024-04-19T23:52:13.760 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:52:13.760 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:52:13Z" } 2024-04-19T23:52:13.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:52:13.761 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:13.761 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:13.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.682721_308.1 2024-04-19T23:52:18.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:18.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:18.763 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:52:18.827 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:52:18.827 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:52:18.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:18.863+0000 7f177eb4f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:18.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:18.865+0000 7f177eb4f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:18.881 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:52:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:52:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:52:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:52:18.881 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:52:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:52:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:52:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:52:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:52:18.882 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:52:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:52:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:52:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:18.883 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:52:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:52:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:52:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-19T23:52:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-19T23:52:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:52:18.885 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:52:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:52:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:18.886 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:52:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:52:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:52:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:52:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:18.887 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:52:18Z" 2024-04-19T23:52:18.887 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:52:18.888 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:52:18Z" } 2024-04-19T23:52:18.888 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:52:18.888 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:18.888 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:18.888 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.682721_308.1 2024-04-19T23:52:23.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:23.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:23.890 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:52:23.948 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:52:23.948 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:52:23.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:23.983+0000 7fe4a5175780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:23.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:23.984+0000 7fe4a5175780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:24.001 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:52:24.001 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:52:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:52:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:52:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:52:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:52:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:52:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:52:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:52:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:52:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:52:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:52:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:52:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:52:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:52:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:52:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:52:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:52:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:52:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:52:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:52:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:52:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:52:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:52:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:52:23Z" 2024-04-19T23:52:24.007 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:52:24.008 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:52:23Z" } 2024-04-19T23:52:24.008 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:52:24.008 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:24.009 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:24.009 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.559046_297.1 2024-04-19T23:52:29.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:29.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:29.010 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:52:29.097 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:52:29.098 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:52:29.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:29.208+0000 7f094fcd4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:29.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:29.209+0000 7f094fcd4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:29.226 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:52:29.226 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:52:29.226 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:52:29.226 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:52:29.226 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:52:29.226 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:52:29.226 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:29.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:29.227 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:52:29.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:29.227 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:52:29.227 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:29.227 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:29.227 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:52:29.227 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:29.227 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:52:29.227 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:29.227 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:52:29.227 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:29.228 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:29.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:29.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:29.228 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:52:29.228 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:29.228 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:29.228 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:52:29.228 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:29.228 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:29.228 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:29.228 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:52:29.228 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:52:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.572795_298.1", 2024-04-19T23:52:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-19T23:52:29.229 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:52:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:52:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:29.230 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:52:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:52:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:52:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:52:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:52:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:29.231 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:52:29Z" 2024-04-19T23:52:29.231 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:52:29.232 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.572795_298.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.572795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:52:29Z" } 2024-04-19T23:52:29.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:52:29.233 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:29.233 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:29.233 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.572795_298.1 2024-04-19T23:52:34.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:34.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:34.234 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:52:34.291 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:52:34.291 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:52:34.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:34.326+0000 7f4342985780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:34.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:34.327+0000 7f4342985780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:34.344 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:52:34.344 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:52:34.344 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:52:34.344 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:52:34.344 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:52:34.344 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:52:34.344 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:34.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:34.345 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:52:34.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:34.345 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:52:34.345 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:34.345 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:34.345 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:52:34.345 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:34.345 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:52:34.345 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:34.345 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:52:34.345 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:34.346 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:34.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:34.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:34.346 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:52:34.346 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:34.346 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:34.346 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:52:34.346 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:34.346 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:34.346 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:34.346 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:52:34.346 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:34.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:34.348 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:52:34.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:34.349 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:52:34.349 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:52:34.349 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:52:34.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:34.349 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:52:34Z" 2024-04-19T23:52:34.349 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:52:34.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:52:34Z" } 2024-04-19T23:52:34.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:52:34.350 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:34.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:34.350 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.629562_306.1 2024-04-19T23:52:39.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:39.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:39.351 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:52:39.419 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:52:39.419 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:52:39.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:39.455+0000 7f17a7357780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:39.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:39.456+0000 7f17a7357780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:39.473 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:52:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:52:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:52:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:52:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:52:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:52:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:39.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:52:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:52:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:52:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:52:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:39.474 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:52:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:52:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:52:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:39.475 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:52:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:52:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:52:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:39.476 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:52:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:52:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:52:39.477 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:52:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:52:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:52:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:52:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:52:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:39.479 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:52:39Z" 2024-04-19T23:52:39.479 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:52:39.480 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:52:39Z" } 2024-04-19T23:52:39.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:52:39.480 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:39.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:39.480 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-19T23:52:44.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:44.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:44.482 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:52:44.540 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:52:44.540 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:52:44.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:44.579+0000 7fe0b2336780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:44.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:44.580+0000 7fe0b2336780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:44.597 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:52:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:52:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:52:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:52:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:52:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:52:44.597 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:52:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:52:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:52:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:52:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:52:44.598 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:52:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:52:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:44.599 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:52:44.600 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:52:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:52:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:52:44.601 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:52:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:52:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:52:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:52:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:44.602 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:52:44Z" 2024-04-19T23:52:44.602 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:52:44.603 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:52:44Z" } 2024-04-19T23:52:44.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:52:44.604 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:44.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:44.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.576049_299.1 2024-04-19T23:52:49.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:49.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:49.606 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:52:49.666 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:52:49.666 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:52:49.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:49.703+0000 7f5c1e902780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:49.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:49.704+0000 7f5c1e902780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:49.721 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:52:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:52:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:52:49.722 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:52:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:52:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:52:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:52:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:49.723 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:52:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:52:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:52:49.724 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:49.725 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:52:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:52:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:49.726 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:52:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:52:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:49.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:52:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:52:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:49.728 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:52:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:49.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:52:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:52:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:49.730 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:52:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:52:49.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:52:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:52:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:52:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:49.732 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:52:49Z" 2024-04-19T23:52:49.732 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:52:49.733 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:52:49Z" } 2024-04-19T23:52:49.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:52:49.734 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:49.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:49.735 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-19T23:52:54.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:54.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:54.737 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:52:54.796 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:52:54.797 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:52:54.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:54.833+0000 7f8f13c74780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:54.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:54.835+0000 7f8f13c74780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:54.852 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:52:54.852 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:52:54.852 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:52:54.852 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:52:54.852 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:52:54.852 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:52:54.853 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:54.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:54.853 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:52:54.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:54.853 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:52:54.853 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:54.853 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:54.853 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:52:54.853 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:54.853 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:52:54.853 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:54.853 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:52:54.854 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:54.854 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:54.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:54.854 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:54.854 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:52:54.854 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:54.854 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:54.854 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:52:54.854 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:54.854 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:54.855 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:54.855 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:52:54.855 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:54.855 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:54.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:54.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:54.855 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:52:54.855 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:54.855 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:54.855 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:52:54.856 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:54.857 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:54.857 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:54.857 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:52:54.857 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:54.857 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:54.857 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:54.857 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:52:54.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:54.857 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:52:54.857 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:52:54.857 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:52:54.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:54.858 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:52:54Z" 2024-04-19T23:52:54.858 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:52:54.859 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:52:54Z" } 2024-04-19T23:52:54.859 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:52:54.859 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:54.859 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:54.859 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.629562_306.1 2024-04-19T23:52:59.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:59.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:52:59.860 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:52:59.922 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:52:59.922 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:52:59.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:59.958+0000 7f9da9240780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:59.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:52:59.960+0000 7f9da9240780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:52:59.981 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:52:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:52:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:52:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:52:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:52:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:52:59.981 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:52:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:52:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:52:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:52:59.982 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:52:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:52:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:52:59.983 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:52:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:59.984 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.632785_307.1", 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:52:59.985 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:52:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:52:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:52:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:52:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:52:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:52:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:52:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:52:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:52:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:52:59.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:52:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:52:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:52:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:52:59.987 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:52:59Z" 2024-04-19T23:52:59.987 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:52:59.988 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.632785_307.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.632785Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:52:59Z" } 2024-04-19T23:52:59.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:52:59.988 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:59.988 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:52:59.988 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.632785_307.1 2024-04-19T23:53:04.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:04.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:04.990 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:53:05.048 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:53:05.048 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:53:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:05.091+0000 7fc971500780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:05.092+0000 7fc971500780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:05.108 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:53:05.109 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:53:05.109 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:53:05.109 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:53:05.109 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:53:05.109 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:53:05.109 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:05.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:05.109 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:53:05.109 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:05.109 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:53:05.109 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:05.110 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:05.110 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:53:05.110 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:05.110 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:53:05.110 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:05.110 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:53:05.110 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:05.110 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:05.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:05.110 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:05.110 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:53:05.111 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:05.111 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:05.111 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:53:05.111 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:05.111 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:05.111 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:05.111 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:53:05.111 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:05.111 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:05.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:05.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:05.111 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:53:05.112 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:05.112 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:05.112 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:53:05.112 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:05.112 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:05.112 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:05.112 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:53:05.112 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:05.112 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:05.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:05.112 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:05.113 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:53:05.113 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:05.113 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:05.113 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:53:05.113 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:05.113 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:05.113 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:05.113 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:53:05.113 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:05.113 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:05.113 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:05.113 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:53:05.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:05.114 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:53:05.114 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:53:05.114 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:53:05.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:05.114 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:53:05Z" 2024-04-19T23:53:05.114 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:53:05.115 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:53:05Z" } 2024-04-19T23:53:05.115 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:53:05.115 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:05.115 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:05.115 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.629562_306.1 2024-04-19T23:53:10.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:10.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:10.117 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:53:10.175 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:53:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:53:10.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:10.222+0000 7f52b705a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:10.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:10.223+0000 7f52b705a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:10.240 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:53:10.241 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:53:10.241 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:53:10.241 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:53:10.241 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:53:10.241 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:53:10.241 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:10.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:10.241 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:53:10.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:10.241 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:53:10.242 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:10.242 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:10.242 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:53:10.242 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:10.242 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:53:10.242 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:10.242 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:53:10.242 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:10.242 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:10.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:10.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:10.243 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:53:10.243 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:10.243 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:10.243 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:53:10.243 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:10.243 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:10.243 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:10.243 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:53:10.243 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:10.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:10.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:10.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:10.244 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:53:10.244 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:10.244 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:10.244 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.572795_298.1", 2024-04-19T23:53:10.244 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:10.244 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:10.244 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:10.244 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-19T23:53:10.244 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:10.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:10.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:10.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:10.245 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:53:10.245 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:10.245 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:10.245 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:53:10.245 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:10.245 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:10.245 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:10.245 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:53:10.245 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:10.245 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:10.246 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:10.246 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:53:10.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:10.246 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:53:10.246 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:53:10.246 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:53:10.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:10.246 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:53:10Z" 2024-04-19T23:53:10.246 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:53:10.251 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.572795_298.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.572795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:53:10Z" } 2024-04-19T23:53:10.251 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:53:10.251 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:10.251 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:10.251 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.572795_298.1 2024-04-19T23:53:15.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:15.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:15.253 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:53:15.311 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:53:15.311 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:53:15.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:15.355+0000 7f3439753780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:15.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:15.356+0000 7f3439753780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:15.373 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:53:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:53:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:53:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:53:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:53:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:53:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:53:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:53:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:53:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:53:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:53:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:53:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:53:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:53:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:53:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:53:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:53:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:53:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:53:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:53:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:53:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:53:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:53:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:53:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:53:15Z" 2024-04-19T23:53:15.379 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:53:15.380 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:53:15Z" } 2024-04-19T23:53:15.380 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:53:15.380 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:15.380 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:15.380 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.559046_297.1 2024-04-19T23:53:20.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:20.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:20.382 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:53:20.443 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:53:20.443 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:53:20.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:20.477+0000 7fa57b407780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:20.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:20.478+0000 7fa57b407780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:20.496 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:53:20.496 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:53:20.496 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:53:20.497 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:53:20.497 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:53:20.497 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:53:20.497 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:20.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:20.497 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:53:20.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:20.497 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:53:20.497 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:20.497 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:20.498 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:53:20.498 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:20.498 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:53:20.498 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:20.498 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:53:20.498 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:20.498 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:20.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:20.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:20.499 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:53:20.499 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:20.499 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:20.499 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:53:20.499 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:20.499 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:20.499 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:20.499 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:53:20.499 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:20.499 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:20.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:20.500 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:20.500 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:53:20.500 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:20.500 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:20.500 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.580723_301.1", 2024-04-19T23:53:20.500 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:20.500 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:20.500 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:20.500 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-19T23:53:20.501 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:20.501 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:20.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:20.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:20.501 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:53:20.501 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:20.501 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:20.501 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:53:20.501 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:20.501 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:20.501 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:20.501 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:53:20.502 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:20.502 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:20.502 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:20.502 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:53:20.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:20.502 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:53:20.502 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:53:20.502 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:53:20.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:20.502 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:53:20Z" 2024-04-19T23:53:20.502 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:53:20.504 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.580723_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.580723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:53:20Z" } 2024-04-19T23:53:20.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:53:20.504 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:20.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:20.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.580723_301.1 2024-04-19T23:53:25.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:25.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:25.506 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:53:25.565 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:53:25.565 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:53:25.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:25.604+0000 7f41ba102780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:25.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:25.605+0000 7f41ba102780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:25.623 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:53:25.623 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:53:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:53:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:53:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:53:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:53:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:53:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:53:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:25.625 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:25.625 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:53:25.625 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:25.625 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:53:25.625 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:25.625 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:53:25.625 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:25.625 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:25.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:25.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:25.625 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:53:25.626 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:25.626 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:25.626 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:53:25.626 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:25.626 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:25.626 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:25.626 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:53:25.626 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:25.626 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:25.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:25.627 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:25.627 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:53:25.627 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:25.627 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:25.627 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.611175_304.1", 2024-04-19T23:53:25.627 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:25.627 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:25.627 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:25.627 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.611175Z", 2024-04-19T23:53:25.627 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:25.627 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:25.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:25.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:25.628 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:53:25.628 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:25.628 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:25.628 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:53:25.628 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:25.628 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:25.628 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:25.629 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:53:25.629 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:25.629 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:25.629 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:25.629 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:53:25.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:25.629 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:53:25.629 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:53:25.629 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:53:25.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:25.629 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:53:25Z" 2024-04-19T23:53:25.630 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:53:25.631 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.611175_304.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.611175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:53:25Z" } 2024-04-19T23:53:25.631 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.611175_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.611175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:53:25.631 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:25.631 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.611175_304.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:25.631 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.611175_304.1 2024-04-19T23:53:30.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:30.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:30.633 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:53:30.692 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:53:30.692 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:53:30.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:30.729+0000 7f03bae86780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:30.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:30.731+0000 7f03bae86780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:30.750 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:53:30.751 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:53:30.751 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:53:30.751 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:53:30.751 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:53:30.751 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:53:30.751 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:30.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:30.751 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:53:30.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:30.751 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:53:30.751 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:30.751 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:30.752 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:53:30.752 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:30.752 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:53:30.752 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:30.752 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:53:30.752 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:30.752 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:30.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:30.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:30.752 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:53:30.752 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:30.752 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:30.753 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:53:30.753 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:30.753 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:30.753 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:30.753 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:53:30.753 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:30.753 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:30.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:30.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:30.753 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:53:30.753 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:30.753 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:30.754 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-19T23:53:30.754 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:30.754 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:30.754 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:30.754 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-19T23:53:30.754 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:30.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:30.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:30.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:30.754 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:53:30.754 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:30.755 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:30.755 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:53:30.755 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:30.755 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:30.755 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:30.755 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:53:30.755 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:30.755 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:30.755 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:30.755 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:53:30.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:30.756 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:53:30.756 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:53:30.756 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:53:30.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:30.756 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:53:30Z" 2024-04-19T23:53:30.756 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:53:30.757 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:53:30Z" } 2024-04-19T23:53:30.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:53:30.757 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:30.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:30.757 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.682721_308.1 2024-04-19T23:53:35.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:35.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:35.759 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:53:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:53:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:53:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:35.862+0000 7f21259ef780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:35.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:35.863+0000 7f21259ef780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:35.880 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:53:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:53:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:53:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:53:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:53:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:53:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:53:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:53:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:35.881 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:53:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:53:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:53:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:53:35.882 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:35.883 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.606676_302.1", 2024-04-19T23:53:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-19T23:53:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:53:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:35.884 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:35.885 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:53:35Z" 2024-04-19T23:53:35.886 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:53:35.886 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.606676_302.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.606676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:53:35Z" } 2024-04-19T23:53:35.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:53:35.887 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:35.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:35.887 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.606676_302.1 2024-04-19T23:53:40.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:40.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:40.888 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:53:40.947 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:53:40.947 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:53:40.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:40.983+0000 7f35e1756780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:40.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:40.984+0000 7f35e1756780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:41.003 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:53:41.003 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:53:41.003 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:53:41.003 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:53:41.003 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:53:41.003 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:53:41.003 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:41.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:41.004 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:53:41.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:41.004 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:53:41.004 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:41.004 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:41.004 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:53:41.004 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:41.004 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:53:41.004 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:41.004 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:53:41.004 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:41.005 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:41.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:41.006 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:41.006 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:53:41.006 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:41.006 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:41.006 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:53:41.006 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:41.006 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:41.006 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:41.006 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:53:41.006 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:41.007 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:41.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:41.008 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:53:41.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:41.008 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:53:41.008 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:53:41.008 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:53:41.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:41.008 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:53:40Z" 2024-04-19T23:53:41.008 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:53:41.009 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:53:40Z" } 2024-04-19T23:53:41.009 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:53:41.009 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:41.009 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:41.010 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-19T23:53:46.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:46.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:46.011 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:53:46.070 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:53:46.070 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:53:46.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:46.105+0000 7f9497317780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:46.107+0000 7f9497317780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:46.125 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:53:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:53:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:53:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:53:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:53:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:53:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:53:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:46.125 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:53:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:53:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:53:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:53:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:46.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:53:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:53:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:53:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:46.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:53:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.555632_296.1", 2024-04-19T23:53:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-19T23:53:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:46.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:53:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:53:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:53:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:53:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:53:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:53:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:53:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:46.130 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:53:46Z" 2024-04-19T23:53:46.130 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:53:46.131 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.555632_296.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.555632Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:53:46Z" } 2024-04-19T23:53:46.131 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:53:46.131 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:46.131 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:46.132 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.555632_296.1 2024-04-19T23:53:51.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:51.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:51.132 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:53:51.191 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:53:51.191 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:53:51.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:51.239+0000 7fddac538780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:51.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:51.241+0000 7fddac538780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:51.260 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:53:51.260 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:53:51.260 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:53:51.260 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:53:51.260 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:53:51.260 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:53:51.260 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:51.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:51.260 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:53:51.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:51.261 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:53:51.261 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:51.261 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:51.261 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:53:51.261 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:51.261 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:53:51.261 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:51.261 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:53:51.261 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:51.261 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:51.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:51.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:51.262 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:53:51.262 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:51.262 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:51.262 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:53:51.262 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:51.262 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:51.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:51.262 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:53:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:53:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.632785_307.1", 2024-04-19T23:53:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-19T23:53:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:53:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:53:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:51.264 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:53:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:53:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:53:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:53:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:53:51.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:51.266 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:53:51Z" 2024-04-19T23:53:51.266 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:53:51.266 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.632785_307.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.632785Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:53:51Z" } 2024-04-19T23:53:51.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:53:51.267 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:51.267 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:51.267 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.632785_307.1 2024-04-19T23:53:56.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:56.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:53:56.268 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:53:56.327 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:53:56.327 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:53:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:56.377+0000 7f3dcf350780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:56.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:53:56.378+0000 7f3dcf350780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:53:56.396 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:53:56.396 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:53:56.396 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:53:56.396 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:53:56.397 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:53:56.397 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:53:56.397 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:56.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:56.397 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:53:56.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:56.397 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:53:56.397 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:56.397 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:56.397 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:53:56.397 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:56.398 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:53:56.398 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:56.398 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:53:56.398 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:56.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:56.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:56.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:56.398 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:53:56.398 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:56.398 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:56.399 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:53:56.399 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:56.399 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:56.399 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:56.399 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:53:56.399 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:56.399 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:56.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:56.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:56.399 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:53:56.399 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:56.399 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:53:56.400 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:53:56.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:53:56.402 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:53:56Z" 2024-04-19T23:53:56.402 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:53:56.402 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:53:56Z" } 2024-04-19T23:53:56.403 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:53:56.403 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:56.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:53:56.403 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.576049_299.1 2024-04-19T23:54:01.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:01.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:01.404 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:01.464 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:01.464 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:01.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:01.500+0000 7fb3aaf16780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:01.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:01.501+0000 7fb3aaf16780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:01.520 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:54:01.520 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:54:01.520 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:54:01.520 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:54:01.520 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:54:01.520 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:54:01.521 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:01.522 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:01.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:01.522 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:01.522 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:54:01.522 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:01.522 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:01.522 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:54:01.522 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:01.522 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:01.522 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:01.522 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:54:01.523 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:01.523 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:01.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:01.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:01.523 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:54:01.523 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:01.523 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:01.523 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:54:01.523 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:01.523 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:01.523 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:01.523 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:54:01.524 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:01.525 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:01.525 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:01.525 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:54:01.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:01.525 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:54:01.525 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:54:01.525 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:54:01.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:01.525 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:54:01Z" 2024-04-19T23:54:01.525 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:54:01.526 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:54:01Z" } 2024-04-19T23:54:01.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:54:01.527 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:01.527 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:01.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.629562_306.1 2024-04-19T23:54:06.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:06.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:06.529 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:06.592 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:06.593 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:06.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:06.631+0000 7fc0e81b0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:06.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:06.632+0000 7fc0e81b0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:06.650 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:54:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:54:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:54:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:54:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:54:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:54:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:54:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:54:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:54:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:54:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:54:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:54:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:54:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:54:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.632785_307.1", 2024-04-19T23:54:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-19T23:54:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:54:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:54:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:54:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:54:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:54:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:54:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:54:06Z" 2024-04-19T23:54:06.656 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:54:06.657 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.632785_307.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.632785Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:54:06Z" } 2024-04-19T23:54:06.657 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:54:06.657 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:06.657 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:06.657 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.632785_307.1 2024-04-19T23:54:11.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:11.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:11.659 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:11.718 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:11.719 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:11.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:11.756+0000 7f12f50d1780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:11.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:11.757+0000 7f12f50d1780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:11.775 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:54:11.775 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:54:11.775 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:54:11.775 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:54:11.775 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:54:11.775 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:54:11.775 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:11.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:11.776 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:54:11.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:11.776 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:54:11.776 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:11.776 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:11.776 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:11.776 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:11.776 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:54:11.777 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:11.777 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:54:11.777 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:11.777 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:11.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:11.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:11.777 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:54:11.777 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:11.777 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:11.777 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:54:11.777 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:11.778 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:11.778 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:11.778 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:54:11.778 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:11.778 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:11.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:11.778 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:11.778 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:54:11.778 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:11.778 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:11.778 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-19T23:54:11.779 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:11.779 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:11.779 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:11.779 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-19T23:54:11.779 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:11.779 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:11.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:11.779 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:11.779 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:54:11.779 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:11.780 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:11.780 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:11.780 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:11.780 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:11.780 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:11.780 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:54:11.780 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:11.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:11.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:11.780 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:54:11.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:11.780 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:54:11.781 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:54:11.781 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:54:11.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:11.781 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:54:11Z" 2024-04-19T23:54:11.781 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:54:11.782 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:54:11Z" } 2024-04-19T23:54:11.782 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:54:11.782 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:11.782 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:11.782 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.682721_308.1 2024-04-19T23:54:14.527 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-19T23:54:14.525+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-19T23:54:16.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:16.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:16.784 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:16.844 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:16.844 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:16.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:16.984+0000 7f0416f14780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:16.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:16.993+0000 7f0416f14780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:17.039 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:54:17.040 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:54:17.040 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:54:17.040 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:54:17.040 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:54:17.040 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:54:17.040 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:17.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:17.040 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:54:17.040 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:17.040 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:54:17.041 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:17.041 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:17.041 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:17.041 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:17.041 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:54:17.041 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:17.041 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:54:17.041 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:17.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:17.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:17.042 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:17.042 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:54:17.042 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:17.042 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:17.042 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:54:17.042 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:17.042 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:17.042 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:17.042 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:54:17.042 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:17.043 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:17.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:17.043 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:17.043 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:54:17.043 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:17.043 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:17.043 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:54:17.043 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:17.043 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:17.043 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:17.043 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:54:17.044 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:17.044 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:17.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:17.044 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:17.044 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:54:17.044 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:17.044 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:17.044 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:17.044 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:17.044 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:17.044 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:17.044 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:54:17.045 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:17.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:17.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:17.045 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:54:17.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:17.045 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:54:17.045 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:54:17.045 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:54:17.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:17.045 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:54:16Z" 2024-04-19T23:54:17.045 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:54:17.046 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:54:16Z" } 2024-04-19T23:54:17.047 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:54:17.047 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:17.047 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:17.047 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.576049_299.1 2024-04-19T23:54:22.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2024-04-19T23:54:22.053 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-45 2024-04-19T23:54:22.053 DEBUG:boto:path=/aktlji-45/ 2024-04-19T23:54:22.054 DEBUG:boto:auth_path=/aktlji-45/ 2024-04-19T23:54:22.054 DEBUG:boto:Method: PUT 2024-04-19T23:54:22.054 DEBUG:boto:Path: /aktlji-45/ 2024-04-19T23:54:22.054 DEBUG:boto:Data: 2024-04-19T23:54:22.054 DEBUG:boto:Headers: {} 2024-04-19T23:54:22.054 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:54:22.054 DEBUG:boto:Port: 8000 2024-04-19T23:54:22.054 DEBUG:boto:Params: {} 2024-04-19T23:54:22.055 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:54:22.055 DEBUG:boto:Token: None 2024-04-19T23:54:22.055 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:54:22 GMT /aktlji-45/ 2024-04-19T23:54:22.055 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:C/5ve7yvmI6QZ7M1gbSTYCHFPj8= 2024-04-19T23:54:22.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:C/5ve7yvmI6QZ7M1gbSTYCHFPj8=', 'Content-Length': '0'} 2024-04-19T23:54:22.103 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d9b57a0acbec57fb-006623042e-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:54:22.103 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-46 2024-04-19T23:54:22.104 DEBUG:boto:path=/aktlji-46/ 2024-04-19T23:54:22.104 DEBUG:boto:auth_path=/aktlji-46/ 2024-04-19T23:54:22.104 DEBUG:boto:Method: PUT 2024-04-19T23:54:22.104 DEBUG:boto:Path: /aktlji-46/ 2024-04-19T23:54:22.104 DEBUG:boto:Data: 2024-04-19T23:54:22.104 DEBUG:boto:Headers: {} 2024-04-19T23:54:22.104 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:54:22.104 DEBUG:boto:Port: 8000 2024-04-19T23:54:22.104 DEBUG:boto:Params: {} 2024-04-19T23:54:22.105 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-19T23:54:22.105 DEBUG:boto:Token: None 2024-04-19T23:54:22.105 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:54:22 GMT /aktlji-46/ 2024-04-19T23:54:22.105 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XHarvH3wDkIJW/6kRgGMEno6I8Q= 2024-04-19T23:54:22.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XHarvH3wDkIJW/6kRgGMEno6I8Q=', 'Content-Length': '0'} 2024-04-19T23:54:22.205 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002b82244dac32d6d3-006623042e-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:54:22.206 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-47 2024-04-19T23:54:22.206 DEBUG:boto:path=/aktlji-47/ 2024-04-19T23:54:22.206 DEBUG:boto:auth_path=/aktlji-47/ 2024-04-19T23:54:22.206 DEBUG:boto:Method: PUT 2024-04-19T23:54:22.206 DEBUG:boto:Path: /aktlji-47/ 2024-04-19T23:54:22.206 DEBUG:boto:Data: 2024-04-19T23:54:22.206 DEBUG:boto:Headers: {} 2024-04-19T23:54:22.206 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:54:22.206 DEBUG:boto:Port: 8001 2024-04-19T23:54:22.207 DEBUG:boto:Params: {} 2024-04-19T23:54:22.207 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-19T23:54:22.207 DEBUG:boto:Token: None 2024-04-19T23:54:22.207 DEBUG:boto:StringToSign: PUT Fri, 19 Apr 2024 23:54:22 GMT /aktlji-47/ 2024-04-19T23:54:22.207 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DcJ9oHBOcIgVx4uu5pi/I7L56Rk= 2024-04-19T23:54:22.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DcJ9oHBOcIgVx4uu5pi/I7L56Rk=', 'Content-Length': '0'} 2024-04-19T23:54:22.275 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e0b8f71b1bedf772-006623042e-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:54:22.275 DEBUG:boto:path=/aktlji-45/ 2024-04-19T23:54:22.275 DEBUG:boto:auth_path=/aktlji-45/ 2024-04-19T23:54:22.275 DEBUG:boto:Method: HEAD 2024-04-19T23:54:22.275 DEBUG:boto:Path: /aktlji-45/ 2024-04-19T23:54:22.276 DEBUG:boto:Data: 2024-04-19T23:54:22.276 DEBUG:boto:Headers: {} 2024-04-19T23:54:22.276 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:54:22.276 DEBUG:boto:Port: 8000 2024-04-19T23:54:22.276 DEBUG:boto:Params: {} 2024-04-19T23:54:22.276 DEBUG:boto:Token: None 2024-04-19T23:54:22.276 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:54:22 GMT /aktlji-45/ 2024-04-19T23:54:22.276 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9z+sXwF8si1DD04e46fegyCcTnw= 2024-04-19T23:54:22.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9z+sXwF8si1DD04e46fegyCcTnw=', 'Content-Length': '0'} 2024-04-19T23:54:22.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000504d4fdb4637c0ea-006623042e-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:54:22.279 DEBUG:boto:path=/aktlji-45/key 2024-04-19T23:54:22.279 DEBUG:boto:auth_path=/aktlji-45/key 2024-04-19T23:54:22.279 DEBUG:boto:Method: PUT 2024-04-19T23:54:22.280 DEBUG:boto:Path: /aktlji-45/key 2024-04-19T23:54:22.280 DEBUG:boto:Data: 2024-04-19T23:54:22.280 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-19T23:54:22.280 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-19T23:54:22.280 DEBUG:boto:Port: 8000 2024-04-19T23:54:22.280 DEBUG:boto:Params: {} 2024-04-19T23:54:22.280 DEBUG:boto:Token: None 2024-04-19T23:54:22.280 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 19 Apr 2024 23:54:22 GMT /aktlji-45/key 2024-04-19T23:54:22.280 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Vd31iakfXXWGP83F6rNVB6QWMQo= 2024-04-19T23:54:22.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Vd31iakfXXWGP83F6rNVB6QWMQo='} 2024-04-19T23:54:22.287 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a86c97d4411e40c-006623042e-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:54:22.287 DEBUG:boto:path=/aktlji-46/ 2024-04-19T23:54:22.288 DEBUG:boto:auth_path=/aktlji-46/ 2024-04-19T23:54:22.288 DEBUG:boto:Method: HEAD 2024-04-19T23:54:22.288 DEBUG:boto:Path: /aktlji-46/ 2024-04-19T23:54:22.288 DEBUG:boto:Data: 2024-04-19T23:54:22.288 DEBUG:boto:Headers: {} 2024-04-19T23:54:22.288 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:54:22.288 DEBUG:boto:Port: 8000 2024-04-19T23:54:22.288 DEBUG:boto:Params: {} 2024-04-19T23:54:22.289 DEBUG:boto:Token: None 2024-04-19T23:54:22.289 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:54:22 GMT /aktlji-46/ 2024-04-19T23:54:22.289 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:soFx5r61eGhSdhlOdHt8t/oeH98= 2024-04-19T23:54:22.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:soFx5r61eGhSdhlOdHt8t/oeH98=', 'Content-Length': '0'} 2024-04-19T23:54:22.309 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006dc6394c2aa02ccf-006623042e-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:54:22.310 DEBUG:boto:path=/aktlji-46/key 2024-04-19T23:54:22.310 DEBUG:boto:auth_path=/aktlji-46/key 2024-04-19T23:54:22.310 DEBUG:boto:Method: PUT 2024-04-19T23:54:22.310 DEBUG:boto:Path: /aktlji-46/key 2024-04-19T23:54:22.310 DEBUG:boto:Data: 2024-04-19T23:54:22.310 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-19T23:54:22.310 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-19T23:54:22.310 DEBUG:boto:Port: 8000 2024-04-19T23:54:22.310 DEBUG:boto:Params: {} 2024-04-19T23:54:22.311 DEBUG:boto:Token: None 2024-04-19T23:54:22.311 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 19 Apr 2024 23:54:22 GMT /aktlji-46/key 2024-04-19T23:54:22.311 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1mh1Q4bI35b5sKn4JGH1yw5ZWOw= 2024-04-19T23:54:22.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1mh1Q4bI35b5sKn4JGH1yw5ZWOw='} 2024-04-19T23:54:22.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec27c9101f6712fe-006623042e-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:54:22.338 DEBUG:boto:path=/aktlji-47/ 2024-04-19T23:54:22.339 DEBUG:boto:auth_path=/aktlji-47/ 2024-04-19T23:54:22.339 DEBUG:boto:Method: HEAD 2024-04-19T23:54:22.339 DEBUG:boto:Path: /aktlji-47/ 2024-04-19T23:54:22.339 DEBUG:boto:Data: 2024-04-19T23:54:22.339 DEBUG:boto:Headers: {} 2024-04-19T23:54:22.339 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:54:22.339 DEBUG:boto:Port: 8001 2024-04-19T23:54:22.339 DEBUG:boto:Params: {} 2024-04-19T23:54:22.339 DEBUG:boto:Token: None 2024-04-19T23:54:22.340 DEBUG:boto:StringToSign: HEAD Fri, 19 Apr 2024 23:54:22 GMT /aktlji-47/ 2024-04-19T23:54:22.340 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tF81CjUAu1YlvWGVtulKaqqE6hE= 2024-04-19T23:54:22.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tF81CjUAu1YlvWGVtulKaqqE6hE=', 'Content-Length': '0'} 2024-04-19T23:54:22.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060a1b4d1ae47246b-006623042e-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:54:22.343 DEBUG:boto:path=/aktlji-47/key 2024-04-19T23:54:22.343 DEBUG:boto:auth_path=/aktlji-47/key 2024-04-19T23:54:22.343 DEBUG:boto:Method: PUT 2024-04-19T23:54:22.343 DEBUG:boto:Path: /aktlji-47/key 2024-04-19T23:54:22.343 DEBUG:boto:Data: 2024-04-19T23:54:22.343 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-19T23:54:22.344 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-19T23:54:22.344 DEBUG:boto:Port: 8001 2024-04-19T23:54:22.344 DEBUG:boto:Params: {} 2024-04-19T23:54:22.344 DEBUG:boto:Token: None 2024-04-19T23:54:22.344 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 19 Apr 2024 23:54:22 GMT /aktlji-47/key 2024-04-19T23:54:22.344 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ApjnyE/+777brPOGMV3K9WGbZOk= 2024-04-19T23:54:22.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ApjnyE/+777brPOGMV3K9WGbZOk='} 2024-04-19T23:54:22.353 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000193dc8ff69b733e5-006623042e-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-19T23:54:22.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:22.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:22.354 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:22.413 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:22.414 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:22.457 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-19T23:54:22.471 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-19T23:54:22.471 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:54:22.471 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:22.471 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-19T23:54:22.472 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:54:22.472 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:54:22.472 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-19T23:54:22.472 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-19T23:54:22.472 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:54:22.472 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:54:22.472 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713570862.263117_452.1", 2024-04-19T23:54:22.472 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:54:22.263117Z" 2024-04-19T23:54:22.472 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-19T23:54:22.472 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-19T23:54:22.472 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:22.473 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-19T23:54:22.473 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-19T23:54:22.473 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-19T23:54:22.474 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:22.474 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-19T23:54:22.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:22.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:22.474 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:22.532 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:22.532 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:22.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:22.567+0000 7f10af769780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:22.569+0000 7f10af769780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:22.591 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:54:22.591 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:54:22.591 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:54:22.591 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:54:22.591 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:54:22.591 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:54:22.592 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:22.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:22.592 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:54:22.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:22.592 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:54:22.592 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:22.592 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:22.592 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:22.592 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:22.593 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:54:22.593 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:22.593 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:54:22.593 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:22.593 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:22.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:22.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:22.593 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:54:22.594 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:22.594 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:22.594 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:54:22.594 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:22.594 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:22.594 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:22.594 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:54:22.594 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:22.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:22.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:22.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:22.595 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:54:22.595 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:22.595 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:22.595 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.632785_307.1", 2024-04-19T23:54:22.595 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:22.595 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:22.595 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:22.595 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-19T23:54:22.595 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:22.596 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:22.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:22.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:22.596 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:54:22.596 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:22.596 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:22.596 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:22.596 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:22.596 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:22.596 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:22.596 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:54:22.597 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:22.597 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:22.597 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:22.597 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:54:22.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:22.597 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:54:22.597 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:54:22.598 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:54:22.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:22.598 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:54:22Z" 2024-04-19T23:54:22.598 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:54:22.599 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.632785_307.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.632785Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:54:22Z" } 2024-04-19T23:54:22.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:54:22.599 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:22.599 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:22.599 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.632785_307.1 2024-04-19T23:54:27.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:27.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:27.601 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:27.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:27.721+0000 7f047fdf7780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:27.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:27.722+0000 7f047fdf7780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:27.745 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:54:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:54:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:54:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:54:27.745 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:54:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:54:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:54:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:54:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:27.746 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:54:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:54:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:54:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:54:27.747 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:54:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:54:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.555632_296.1", 2024-04-19T23:54:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:27.749 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:54:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:54:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:54:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:54:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:54:27.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:27.751 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:54:27Z" 2024-04-19T23:54:27.751 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:54:27.751 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.555632_296.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.555632Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:54:27Z" } 2024-04-19T23:54:27.752 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:54:27.752 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:27.752 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:27.752 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.555632_296.1 2024-04-19T23:54:32.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:32.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:32.754 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:32.812 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:32.812 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:32.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:32.859+0000 7fa9ee78c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:32.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:32.860+0000 7fa9ee78c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:32.880 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:54:32.881 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:54:32.881 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:54:32.881 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:54:32.881 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:54:32.881 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:54:32.881 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:32.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:32.881 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:54:32.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:54:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:54:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:54:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:32.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:32.883 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:54:32.883 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:32.883 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:32.883 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:54:32.883 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:32.883 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:32.883 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:32.883 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:54:32.883 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:32.883 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:32.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:32.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:32.884 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:54:32.884 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:32.884 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:32.884 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-19T23:54:32.884 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:32.884 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:32.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:32.884 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-19T23:54:32.884 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:32.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:32.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:32.886 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:54:32.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:32.886 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:54:32.886 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:54:32.886 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:54:32.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:32.886 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:54:32Z" 2024-04-19T23:54:32.886 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:54:32.887 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:54:32Z" } 2024-04-19T23:54:32.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:54:32.887 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:32.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:32.887 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.797017_325.1 2024-04-19T23:54:37.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:37.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:37.889 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:37.948 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:37.949 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:37.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:37.994+0000 7f4157ed3780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:37.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:37.995+0000 7f4157ed3780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:38.064 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:54:38.064 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:54:38.064 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:54:38.065 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:54:38.065 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:54:38.065 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:54:38.065 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:38.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:38.065 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:54:38.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:38.065 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:54:38.065 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:38.065 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:38.066 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:38.066 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:38.066 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:54:38.066 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:38.066 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:54:38.066 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:38.066 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:38.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:38.066 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:38.066 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:54:38.066 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:38.067 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:38.067 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:54:38.067 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:38.067 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:38.067 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:38.067 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:54:38.067 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:38.067 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:38.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:38.067 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:38.068 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:54:38.068 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:38.068 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:38.068 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.572795_298.1", 2024-04-19T23:54:38.068 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:38.068 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:38.068 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:38.068 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-19T23:54:38.068 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:38.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:38.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:54:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:54:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:38.069 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:54:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:54:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:54:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:54:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:38.070 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:54:37Z" 2024-04-19T23:54:38.070 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:54:38.071 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.572795_298.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.572795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:54:37Z" } 2024-04-19T23:54:38.071 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:54:38.071 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:38.071 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:38.071 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.572795_298.1 2024-04-19T23:54:43.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:43.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:43.074 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:43.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:43.234+0000 7f2cb08b2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:43.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:43.235+0000 7f2cb08b2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:43.253 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:54:43.253 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:54:43.253 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:54:43.253 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:54:43.253 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:54:43.253 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:54:43.253 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:43.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:43.253 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:54:43.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:43.253 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:54:43.254 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:43.254 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:43.254 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:43.254 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:43.254 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:54:43.254 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:43.254 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:54:43.254 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:43.254 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:43.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:43.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:43.256 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:54:43.256 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:43.256 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:43.256 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:54:43.256 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:43.256 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:43.256 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:43.256 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:54:43.256 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:43.256 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:43.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:43.256 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:54:43.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:43.258 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:54:43.258 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:54:43.258 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:54:43.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:43.258 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:54:43Z" 2024-04-19T23:54:43.258 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:54:43.259 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:54:43Z" } 2024-04-19T23:54:43.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:54:43.259 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:43.260 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:43.260 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.554348_295.1 2024-04-19T23:54:48.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:48.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:48.261 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:48.337 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:48.337 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:48.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:48.486+0000 7f68737c0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:48.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:48.488+0000 7f68737c0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:48.531 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:54:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:54:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:54:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:54:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:54:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:54:48.531 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:54:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:54:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:54:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:54:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:48.533 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:48.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:48.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:48.533 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:54:48.533 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:48.533 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:48.533 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:54:48.533 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:48.533 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:48.533 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:48.533 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:54:48.533 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:48.534 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:48.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:48.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:48.534 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:54:48.534 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:48.534 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:48.534 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:54:48.534 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:48.534 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:48.534 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:48.534 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:54:48.534 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:48.535 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:48.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:48.535 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:48.535 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:54:48.535 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:48.535 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:48.535 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:48.535 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:48.535 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:48.535 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:48.535 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:54:48.536 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:48.536 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:48.536 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:48.536 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:54:48.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:48.536 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:54:48.536 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:54:48.536 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:54:48.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:48.536 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:54:48Z" 2024-04-19T23:54:48.537 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:54:48.537 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:54:48Z" } 2024-04-19T23:54:48.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:54:48.538 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:48.538 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:48.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.629562_306.1 2024-04-19T23:54:53.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:53.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:53.539 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:53.605 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:53.605 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:53.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:53.654+0000 7f3c926bb780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:53.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:53.655+0000 7f3c926bb780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:53.677 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:54:53.677 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:54:53.677 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:54:53.677 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:54:53.677 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:54:53.677 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:54:53.677 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:53.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:53.678 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:54:53.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:53.678 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:54:53.678 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:53.678 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:53.678 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:53.678 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:53.678 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:54:53.678 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:53.678 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:54:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:54:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:54:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:53.679 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:54:53.680 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:53.680 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:53.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:53.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:53.680 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:54:53.680 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:53.680 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:53.680 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.793769_324.1", 2024-04-19T23:54:53.680 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:53.680 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:53.680 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:53.681 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.793769Z", 2024-04-19T23:54:53.681 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:53.681 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:53.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:53.681 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:53.681 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:54:53.681 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:53.681 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:53.681 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:53.681 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:53.681 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:53.681 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:53.682 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:54:53.682 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:53.682 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:53.682 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:53.682 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:54:53.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:53.682 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:54:53.682 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:54:53.682 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:54:53.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:53.683 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:54:53Z" 2024-04-19T23:54:53.683 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:54:53.683 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.793769_324.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.793769Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:54:53Z" } 2024-04-19T23:54:53.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.793769_324.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.793769Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:54:53.684 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:53.684 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.793769_324.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:53.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.793769_324.1 2024-04-19T23:54:58.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:58.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:54:58.686 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:54:58.745 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:54:58.745 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:54:58.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:58.793+0000 7f4c25dc0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:58.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:54:58.794+0000 7f4c25dc0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:54:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:54:58.872 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:54:58.872 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:54:58.872 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:54:58.873 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:54:58.873 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:54:58.873 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:58.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:58.873 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:54:58.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:58.873 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:54:58.873 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:58.873 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:58.873 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:54:58.874 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:58.874 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:54:58.874 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:58.874 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:54:58.874 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:58.874 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:58.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:58.874 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:58.874 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:54:58.874 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:58.874 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:58.874 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:54:58.875 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:58.875 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:58.875 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:58.875 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:54:58.875 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:58.875 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:58.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:58.875 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:58.875 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:54:58.875 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:58.875 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:58.875 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.555632_296.1", 2024-04-19T23:54:58.876 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:58.876 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:58.876 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:58.876 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-19T23:54:58.876 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:58.876 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:58.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:58.876 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:54:58.876 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:54:58.876 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:54:58.876 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:54:58.877 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:54:58.877 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:54:58.877 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:54:58.877 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:54:58.877 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:54:58.877 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:54:58.877 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:58.877 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:54:58.878 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:54:58.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:58.878 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:54:58.878 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:54:58.878 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:54:58.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:54:58.878 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:54:58Z" 2024-04-19T23:54:58.878 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:54:58.878 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.555632_296.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.555632Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:54:58Z" } 2024-04-19T23:54:58.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:54:58.879 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:58.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:54:58.879 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.555632_296.1 2024-04-19T23:55:03.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:03.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:03.880 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:55:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:55:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:55:04.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:04.104+0000 7f2737aa0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:04.106+0000 7f2737aa0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:04.127 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:55:04.128 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:55:04.128 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:55:04.128 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:55:04.128 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:55:04.128 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:55:04.128 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:04.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:04.128 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:55:04.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:04.128 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:55:04.129 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:04.129 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:04.129 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:55:04.129 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:04.129 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:55:04.129 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:04.129 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:55:04.129 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:04.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:04.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:04.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:04.129 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:55:04.130 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:04.130 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:04.130 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:55:04.130 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:04.130 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:04.130 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:04.130 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:55:04.130 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:04.130 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:04.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:04.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.717624_312.1", 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:04.131 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:04.132 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:55:04.133 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:55:04.133 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:55:04.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:04.133 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:55:04Z" 2024-04-19T23:55:04.133 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:55:04.134 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.717624_312.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.717624Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:55:04Z" } 2024-04-19T23:55:04.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.717624_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.717624Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:55:04.134 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:04.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.717624_312.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:04.134 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.717624_312.1 2024-04-19T23:55:09.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:09.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:09.136 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:55:09.193 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:55:09.193 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:55:09.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:09.238+0000 7f9faff91780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:09.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:09.239+0000 7f9faff91780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:09.260 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:55:09.260 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:55:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:55:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:55:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:55:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:55:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:55:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:55:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:55:09.262 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:09.262 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:55:09.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:09.262 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:55:09.262 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:09.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:09.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:09.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:09.262 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:55:09.262 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:09.262 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:09.263 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:55:09.263 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:09.263 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:09.263 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:09.263 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:55:09.263 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:09.263 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:09.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:09.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:09.263 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:55:09.263 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:09.264 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:09.264 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.738086_316.1", 2024-04-19T23:55:09.264 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:09.264 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:09.264 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:09.264 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-19T23:55:09.264 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:09.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:09.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:09.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:09.264 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:55:09.264 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:09.265 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:09.265 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:55:09.265 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:09.265 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:09.265 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:09.265 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:55:09.265 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:09.265 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:09.265 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:09.265 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:55:09.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:09.266 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:55:09.266 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:55:09.266 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:55:09.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:09.266 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:55:09Z" 2024-04-19T23:55:09.266 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:55:09.267 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.738086_316.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.738086Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:55:09Z" } 2024-04-19T23:55:09.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:55:09.267 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:09.267 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:09.267 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.738086_316.1 2024-04-19T23:55:14.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:14.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:14.269 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:55:14.327 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:55:14.328 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:55:14.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:14.367+0000 7f9bf8d28780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:14.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:14.368+0000 7f9bf8d28780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:14.389 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:55:14.389 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:55:14.389 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:55:14.389 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:55:14.390 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:55:14.390 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:55:14.390 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:14.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:14.390 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:55:14.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:14.390 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:55:14.390 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:14.390 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:14.390 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:55:14.390 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:14.390 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:55:14.391 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:14.391 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:55:14.391 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:14.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:14.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:14.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:14.391 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:55:14.391 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:14.391 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:14.392 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:55:14.392 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:14.392 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:14.392 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:14.392 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:55:14.392 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:14.392 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:14.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:14.392 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:14.392 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:55:14.392 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:14.393 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:14.394 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:55:14.394 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:14.394 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:14.394 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:14.394 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:55:14.394 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:14.394 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:14.394 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:14.394 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:55:14.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:14.395 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:55:14.395 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:55:14.395 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:55:14.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:14.395 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:55:14Z" 2024-04-19T23:55:14.395 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:55:14.396 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:55:14Z" } 2024-04-19T23:55:14.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:55:14.396 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:14.396 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:14.396 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.576049_299.1 2024-04-19T23:55:19.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:19.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:19.398 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:55:19.458 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:55:19.458 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:55:19.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:19.536+0000 7f1a552ba780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:19.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:19.546+0000 7f1a552ba780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:19.658 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:55:19.659 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:55:19.659 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:55:19.659 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:55:19.659 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:55:19.659 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:55:19.659 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:19.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:19.660 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:55:19.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:19.660 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:55:19.660 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:19.660 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:19.660 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:55:19.660 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:19.661 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:55:19.661 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:19.661 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:55:19.661 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:19.661 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:19.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:19.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:19.662 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:55:19.662 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:19.662 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:19.662 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:55:19.662 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:19.662 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:19.663 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:19.663 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:55:19.663 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:19.663 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:19.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:19.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:19.664 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:55:19.665 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:19.665 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:19.665 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.614143_305.1", 2024-04-19T23:55:19.665 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:19.665 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:19.665 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:19.665 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-19T23:55:19.665 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:19.665 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:19.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:19.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:19.666 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:55:19.666 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:19.666 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:19.666 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:55:19.666 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:19.666 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:19.666 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:19.666 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:55:19.666 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:19.667 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:19.667 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:19.667 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:55:19.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:19.667 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:55:19.667 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:55:19.667 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:55:19.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:19.667 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:55:19Z" 2024-04-19T23:55:19.668 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:55:19.668 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.614143_305.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.614143Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:55:19Z" } 2024-04-19T23:55:19.669 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:55:19.669 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:19.669 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:19.669 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.614143_305.1 2024-04-19T23:55:24.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:24.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:24.670 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:55:24.725 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:55:24.725 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:55:24.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:24.783+0000 7ff6da420780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:24.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:24.785+0000 7ff6da420780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:24.811 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:55:24.811 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:55:24.811 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:55:24.811 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:55:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:55:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:55:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:55:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:55:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:55:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:55:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:55:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:55:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:55:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:55:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:55:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.606676_302.1", 2024-04-19T23:55:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-19T23:55:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:55:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:55:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:55:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:55:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:55:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:55:24.817 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:55:24.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:24.817 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:55:24Z" 2024-04-19T23:55:24.817 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:55:24.818 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.606676_302.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.606676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:55:24Z" } 2024-04-19T23:55:24.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:55:24.818 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:24.818 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:24.818 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.606676_302.1 2024-04-19T23:55:29.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:29.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:29.819 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:55:29.877 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:55:29.877 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:55:29.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:29.919+0000 7f3f8a453780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:29.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:29.920+0000 7f3f8a453780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:29.938 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:55:29.938 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:55:29.938 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:55:29.938 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:55:29.939 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:55:29.939 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:55:29.939 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:29.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:29.939 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:55:29.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:29.939 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:55:29.939 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:29.939 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:29.939 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:55:29.939 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:29.940 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:55:29.940 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:29.940 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:55:29.940 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:29.940 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:29.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:29.940 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:29.940 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:55:29.940 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:29.940 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:29.940 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:55:29.940 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:29.941 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:29.941 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:29.941 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:55:29.941 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:29.941 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:29.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:29.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:29.941 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:55:29.941 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:29.941 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:29.941 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:55:29.941 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:29.942 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:29.942 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:29.942 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:55:29.942 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:29.942 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:29.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:29.942 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:29.942 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:55:29.942 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:29.942 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:29.942 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:55:29.943 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:29.943 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:29.943 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:29.943 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:55:29.943 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:29.943 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:29.943 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:29.943 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:55:29.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:29.943 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:55:29.943 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:55:29.943 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:55:29.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:29.944 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:55:29Z" 2024-04-19T23:55:29.944 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:55:29.945 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:55:29Z" } 2024-04-19T23:55:29.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:55:29.945 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:29.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:29.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.576049_299.1 2024-04-19T23:55:34.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:34.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:34.946 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:55:35.006 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:55:35.007 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:55:35.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:35.047+0000 7f559c411780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:35.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:35.048+0000 7f559c411780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:35.066 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:55:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:55:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:55:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:55:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:55:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:55:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:55:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:55:35.067 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:55:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:55:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:55:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:35.068 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:55:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.580723_301.1", 2024-04-19T23:55:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-19T23:55:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:35.070 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:55:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:55:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:55:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:55:35.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:55:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:55:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:55:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:35.072 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:55:35Z" 2024-04-19T23:55:35.072 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:55:35.073 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.580723_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.580723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:55:35Z" } 2024-04-19T23:55:35.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:55:35.073 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:35.073 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:35.073 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.580723_301.1 2024-04-19T23:55:40.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:40.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:40.074 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:55:40.133 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:55:40.133 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:55:40.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:40.180+0000 7fec463cc780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:40.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:40.181+0000 7fec463cc780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:40.208 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:55:40.208 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:55:40.208 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:55:40.208 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:55:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:55:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:55:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:55:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:55:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:55:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:55:40.210 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:40.210 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:55:40.210 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:40.210 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:40.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:40.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:40.210 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:55:40.210 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:40.210 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:40.210 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:55:40.210 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:40.211 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:40.211 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:40.211 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:55:40.211 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:40.211 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:40.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:40.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:40.211 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:55:40.211 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:40.211 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:40.211 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.606676_302.1", 2024-04-19T23:55:40.211 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:40.212 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:55:40Z" 2024-04-19T23:55:40.213 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:55:40.214 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.606676_302.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.606676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:55:40Z" } 2024-04-19T23:55:40.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:55:40.215 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:40.215 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:40.215 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.606676_302.1 2024-04-19T23:55:45.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:45.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:45.216 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:55:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:55:45.275 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:55:45.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:45.332+0000 7fc16e212780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:45.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:45.333+0000 7fc16e212780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:45.357 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:55:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:55:45.357 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:55:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:55:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:55:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:55:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:55:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:45.358 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:55:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:55:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:55:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:55:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:45.359 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:55:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:55:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:55:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:45.360 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:55:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.698420_310.1", 2024-04-19T23:55:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-19T23:55:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:45.361 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:55:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:55:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:55:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:45.362 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:55:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:55:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:55:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:55:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:45.363 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:55:45Z" 2024-04-19T23:55:45.363 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:55:45.364 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.698420_310.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.698420Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:55:45Z" } 2024-04-19T23:55:45.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.698420_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.698420Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:55:45.364 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:45.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.698420_310.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:45.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.698420_310.1 2024-04-19T23:55:50.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:50.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:50.366 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:55:50.426 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:55:50.426 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:55:50.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:50.469+0000 7f19f2ea2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:50.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:50.470+0000 7f19f2ea2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:50.496 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:55:50.497 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:55:50.497 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:55:50.497 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:55:50.497 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:55:50.497 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:55:50.497 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:50.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:50.497 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:55:50.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:50.497 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:55:50.498 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:50.498 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:50.498 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:55:50.498 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:50.498 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:55:50.498 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:50.498 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:55:50.498 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:50.498 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:50.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:50.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:50.502 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:55:50.502 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:50.503 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:50.503 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:55:50.503 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:50.503 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:50.503 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:50.503 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:55:50.503 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:50.503 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:50.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:50.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:50.503 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:55:50.503 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:50.504 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:50.504 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-19T23:55:50.504 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:50.504 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:50.504 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:50.504 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-19T23:55:50.504 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:50.504 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:50.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:50.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:50.504 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:55:50.504 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:50.505 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:50.505 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:55:50.505 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:50.505 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:50.505 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:50.505 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:55:50.505 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:50.505 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:50.505 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:50.505 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:55:50.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:50.505 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:55:50.506 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:55:50.506 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:55:50.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:50.506 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:55:50Z" 2024-04-19T23:55:50.506 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:55:50.507 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:55:50Z" } 2024-04-19T23:55:50.507 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:55:50.507 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:50.507 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:50.507 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.682721_308.1 2024-04-19T23:55:55.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:55.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:55:55.509 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:55:55.635 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:55:55.635 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:55:55.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:55.715+0000 7fde48e63780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:55.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:55:55.716+0000 7fde48e63780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:55:55.741 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:55:55.741 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:55:55.741 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:55:55.741 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:55:55.741 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:55:55.742 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:55:55.742 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:55.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:55.742 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:55:55.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:55.742 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:55:55.742 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:55.742 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:55.742 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:55:55.742 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:55.742 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:55:55.742 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:55.743 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:55:55.743 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:55.743 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:55.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:55.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:55.743 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:55:55.743 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:55.743 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:55.743 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:55:55.743 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:55.743 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:55.744 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:55.744 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:55:55.744 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:55.744 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:55.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:55.744 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:55.744 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:55:55.744 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:55.744 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:55.744 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:55:55.744 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:55.745 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:55.745 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:55.745 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:55:55.745 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:55.745 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:55.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:55.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:55:55.745 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:55:55.745 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:55:55.745 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:55:55.745 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:55:55.745 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:55:55.746 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:55:55Z" 2024-04-19T23:55:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:55:55.747 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:55:55Z" } 2024-04-19T23:55:55.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:55:55.748 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:55.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:55:55.748 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.576049_299.1 2024-04-19T23:56:00.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:00.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:00.749 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:56:00.807 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:56:00.808 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:56:00.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:00.853+0000 7f922d11a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:00.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:00.854+0000 7f922d11a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:00.875 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:56:00.875 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:56:00.876 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:56:00.876 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:56:00.876 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:56:00.876 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:56:00.876 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:00.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:00.876 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:56:00.876 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:00.876 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:56:00.876 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:00.876 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:00.877 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:56:00.878 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:00.878 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:00.878 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:00.878 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:56:00.878 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:00.878 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:00.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:00.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:00.878 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:56:00.878 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:00.879 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:00.879 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.555632_296.1", 2024-04-19T23:56:00.879 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:00.879 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:00.879 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:00.879 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-19T23:56:00.879 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:00.879 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:00.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:00.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:00.879 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:56:00.879 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:00.880 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:00.880 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:56:00.880 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:00.880 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:00.880 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:00.880 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:56:00.880 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:00.880 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:00.880 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:00.880 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:56:00.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:00.880 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:56:00.881 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:56:00.881 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:56:00.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:00.881 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:56:00Z" 2024-04-19T23:56:00.881 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:56:00.882 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.555632_296.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.555632Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:56:00Z" } 2024-04-19T23:56:00.882 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:56:00.882 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:00.882 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:00.882 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.555632_296.1 2024-04-19T23:56:05.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:05.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:05.884 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:56:05.945 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:56:05.945 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:56:05.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:05.995+0000 7f451a72f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:05.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:05.996+0000 7f451a72f780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:06.016 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:56:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:56:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:56:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:56:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:56:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:56:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:56:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:56:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:56:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:56:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:56:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:56:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:56:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:56:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:56:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-19T23:56:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-19T23:56:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:56:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:56:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:56:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:56:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:56:05Z" 2024-04-19T23:56:06.021 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:56:06.022 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:56:05Z" } 2024-04-19T23:56:06.022 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:56:06.022 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:06.022 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:06.022 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.682721_308.1 2024-04-19T23:56:11.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:11.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:11.024 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:56:11.083 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:56:11.083 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:56:11.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:11.123+0000 7f4d8c83e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:11.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:11.124+0000 7f4d8c83e780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:11.144 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:56:11.144 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:56:11.144 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:56:11.144 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:56:11.144 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:56:11.144 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:56:11.144 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:11.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:11.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:11.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:11.146 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:56:11.146 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:11.146 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:11.146 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:56:11.146 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:11.146 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:11.146 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:11.146 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:56:11.146 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:11.146 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.555632_296.1", 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:11.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:11.148 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:56:11.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:11.149 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:56:11.149 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:56:11.149 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:56:11.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:11.149 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:56:11Z" 2024-04-19T23:56:11.149 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:56:11.150 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.555632_296.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.555632Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:56:11Z" } 2024-04-19T23:56:11.150 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:56:11.150 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:11.151 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:11.151 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.555632_296.1 2024-04-19T23:56:16.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:16.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:16.151 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:56:16.210 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:56:16.211 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:56:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:16.253+0000 7fcf34b68780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:16.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:16.254+0000 7fcf34b68780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:16.272 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:56:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:56:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:56:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:56:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:56:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:56:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:56:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:56:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:16.273 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:56:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:56:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:56:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:16.274 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:56:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:56:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:56:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:16.275 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:56:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.614143_305.1", 2024-04-19T23:56:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-19T23:56:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:16.276 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:56:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:56:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:56:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:56:16.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:56:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:56:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:56:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:16.278 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:56:16Z" 2024-04-19T23:56:16.278 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:56:16.279 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.614143_305.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.614143Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:56:16Z" } 2024-04-19T23:56:16.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:56:16.279 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:16.279 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:16.279 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.614143_305.1 2024-04-19T23:56:21.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:21.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:21.281 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:56:21.340 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:56:21.340 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:56:21.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:21.380+0000 7f5015a1b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:21.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:21.382+0000 7f5015a1b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:21.402 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:56:21.402 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:56:21.402 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:56:21.402 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:56:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:56:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:56:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:56:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:56:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:56:21.403 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:21.404 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.717624_312.1", 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:21.405 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-19T23:56:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:56:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:56:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:21.406 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:56:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:56:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:56:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:56:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:56:21.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:21.408 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:56:21Z" 2024-04-19T23:56:21.408 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:56:21.408 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.717624_312.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.717624Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:56:21Z" } 2024-04-19T23:56:21.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.717624_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.717624Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:56:21.409 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:21.409 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.717624_312.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:21.409 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.717624_312.1 2024-04-19T23:56:26.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:26.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:26.410 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:56:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:56:26.469 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:56:26.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:26.509+0000 7f0b3eefa780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:26.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:26.511+0000 7f0b3eefa780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:26.532 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:56:26.533 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:56:26.533 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:56:26.533 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:56:26.533 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:56:26.533 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:56:26.533 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:26.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:26.533 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:56:26.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:26.533 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:56:26.533 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:26.534 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:26.534 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:56:26.534 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:26.534 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:56:26.534 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:26.534 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:56:26.534 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:26.534 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:26.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:26.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:26.534 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:56:26.535 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:26.536 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:26.536 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.609968_303.1", 2024-04-19T23:56:26.536 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:26.536 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:26.536 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:26.536 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-19T23:56:26.536 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:26.536 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:26.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:26.536 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:26.536 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:56:26.536 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:26.537 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:26.537 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:56:26.537 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:26.537 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:26.537 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:26.537 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:56:26.537 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:26.537 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:26.537 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:26.537 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:56:26.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:26.537 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:56:26.538 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:56:26.538 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:56:26.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:26.538 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:56:26Z" 2024-04-19T23:56:26.538 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:56:26.539 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.609968_303.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.609968Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:56:26Z" } 2024-04-19T23:56:26.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:56:26.539 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:26.539 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:26.539 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.609968_303.1 2024-04-19T23:56:31.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:31.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:31.541 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:56:31.601 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:56:31.601 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:56:31.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:31.644+0000 7f038ecbc780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:31.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:31.645+0000 7f038ecbc780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:31.664 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:56:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:56:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:56:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:56:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:56:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:56:31.664 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:31.665 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:56:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:56:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:31.666 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.572795_298.1", 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-19T23:56:31.667 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:56:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:56:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:31.668 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:56:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:56:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:56:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:56:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:56:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:31.669 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:56:31Z" 2024-04-19T23:56:31.669 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:56:31.670 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.572795_298.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.572795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:56:31Z" } 2024-04-19T23:56:31.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:56:31.670 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:31.671 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:31.671 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.572795_298.1 2024-04-19T23:56:36.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:36.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:36.672 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:56:36.732 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:56:36.732 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:56:36.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:36.771+0000 7f2c05316780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:36.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:36.772+0000 7f2c05316780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:36.792 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:56:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:56:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:56:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:56:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:56:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:56:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:56:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:36.793 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:56:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:56:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:56:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:56:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:36.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:56:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:56:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:56:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:36.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:56:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.555632_296.1", 2024-04-19T23:56:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-19T23:56:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:36.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:56:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:56:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:56:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:56:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:56:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:56:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:56:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:36.798 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:56:36Z" 2024-04-19T23:56:36.798 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:56:36.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.555632_296.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.555632Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:56:36Z" } 2024-04-19T23:56:36.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:56:36.800 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:36.800 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:36.800 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.555632_296.1 2024-04-19T23:56:41.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:41.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:41.801 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:56:41.860 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:56:41.860 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:56:41.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:41.900+0000 7f22ec4f4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:41.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:41.901+0000 7f22ec4f4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:42.013 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:56:42.013 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:56:42.014 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:56:42.014 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:56:42.014 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:56:42.014 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:56:42.014 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:42.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:42.014 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:56:42.014 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:56:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:56:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:56:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:56:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:56:42.017 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:42.017 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:42.017 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.611175_304.1", 2024-04-19T23:56:42.017 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:42.017 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:42.017 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:42.017 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.611175Z", 2024-04-19T23:56:42.017 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:42.017 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:42.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:42.017 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:42.017 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:56:42.018 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:42.018 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:42.018 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:56:42.018 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:42.018 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:42.018 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:42.018 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:56:42.018 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:42.018 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:42.018 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:42.018 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:56:42.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:42.019 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:56:42.019 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:56:42.019 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:56:42.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:42.019 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:56:41Z" 2024-04-19T23:56:42.019 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:56:42.020 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.611175_304.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.611175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:56:41Z" } 2024-04-19T23:56:42.020 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.611175_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.611175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:56:42.020 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:42.020 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.611175_304.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:42.020 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.611175_304.1 2024-04-19T23:56:47.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:47.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:47.021 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:56:47.113 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:56:47.113 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:56:47.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:47.194+0000 7fcd9efa6780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:47.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:47.195+0000 7fcd9efa6780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:47.218 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:56:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:56:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:56:47.218 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:56:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:56:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:56:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:47.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:56:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:56:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:56:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:56:47.221 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:56:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:56:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:47.222 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:56:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:56:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:56:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.609968_303.1", 2024-04-19T23:56:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-19T23:56:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:47.224 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:47.225 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:56:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:56:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:56:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:47.226 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:56:47Z" 2024-04-19T23:56:47.226 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:56:47.227 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.609968_303.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.609968Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:56:47Z" } 2024-04-19T23:56:47.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:56:47.227 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:47.227 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:47.227 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.609968_303.1 2024-04-19T23:56:52.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:52.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:52.229 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:56:52.289 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:56:52.289 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:56:52.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:52.341+0000 7faef03e3780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:52.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:52.343+0000 7faef03e3780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:52.369 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:56:52.370 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:56:52.370 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:56:52.370 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:56:52.370 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:56:52.370 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:56:52.370 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:52.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:52.370 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:56:52.370 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:52.370 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:56:52.371 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:52.371 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:52.371 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:56:52.371 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:52.371 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:56:52.371 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:52.371 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:56:52.371 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:52.371 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:52.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:52.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:52.372 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.632785_307.1", 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:56:52.373 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:52.374 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:52.374 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:56:52.374 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:52.374 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:52.374 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:52.374 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:56:52.374 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:52.374 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:52.374 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:52.374 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:56:52.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:52.375 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:56:52.375 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:56:52.375 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:56:52.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:52.375 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:56:52Z" 2024-04-19T23:56:52.375 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:56:52.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.632785_307.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.632785Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:56:52Z" } 2024-04-19T23:56:52.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:56:52.376 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:52.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:52.376 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.632785_307.1 2024-04-19T23:56:57.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:57.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:56:57.378 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:56:57.438 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:56:57.438 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:56:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:57.481+0000 7ff169710780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:57.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:56:57.482+0000 7ff169710780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:56:57.500 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:56:57.500 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:56:57.500 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:56:57.500 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:56:57.501 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:56:57.501 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:56:57.501 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:57.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:57.501 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:56:57.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:57.501 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:56:57.501 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:57.501 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:57.501 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:56:57.502 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:57.502 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:56:57.502 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:57.502 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:56:57.502 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:57.502 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:57.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:57.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:57.502 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:56:57.502 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:57.502 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:57.503 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:56:57.503 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:57.503 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:57.503 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:57.503 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:56:57.503 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:57.503 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:57.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:57.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:57.503 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:56:57.503 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:57.503 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:56:57.504 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:56:57.505 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:56:57.505 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:56:57.505 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:56:57.505 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:56:57.505 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:56:57.505 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:57.505 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:56:57.505 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:56:57.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:57.505 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:56:57.505 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:56:57.505 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:56:57.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:56:57.506 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:56:57Z" 2024-04-19T23:56:57.506 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:56:57.507 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:56:57Z" } 2024-04-19T23:56:57.507 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:56:57.507 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:57.507 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:56:57.507 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.682721_308.1 2024-04-19T23:57:02.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:02.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:02.508 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:57:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:57:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:57:02.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:02.631+0000 7f5c9cbfc780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:02.632+0000 7f5c9cbfc780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:02.655 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:57:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:57:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:57:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:57:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:57:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:57:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:57:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:57:02.655 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:02.656 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:57:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:57:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:57:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:02.657 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:02.658 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:57:02.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:02.660 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:57:02Z" 2024-04-19T23:57:02.660 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:57:02.660 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:57:02Z" } 2024-04-19T23:57:02.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:57:02.661 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:02.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:02.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.576049_299.1 2024-04-19T23:57:07.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:07.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:07.662 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:57:07.723 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:57:07.723 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:57:07.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:07.776+0000 7fab12a78780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:07.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:07.777+0000 7fab12a78780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:07.798 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:57:07.798 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:57:07.798 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:57:07.798 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:57:07.799 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:57:07.799 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:57:07.799 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:07.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:07.799 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:57:07.799 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:07.799 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:57:07.800 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:07.800 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:07.800 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:57:07.800 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:07.800 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:57:07.800 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:07.800 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:57:07.800 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:07.801 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:07.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:07.801 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:07.801 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:57:07.801 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:07.801 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:07.801 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:57:07.801 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:07.802 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:07.802 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:07.802 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:57:07.802 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:07.802 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:07.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:07.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:07.802 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:57:07.803 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:07.803 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:07.803 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.632785_307.1", 2024-04-19T23:57:07.803 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:07.803 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:07.803 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:07.803 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-19T23:57:07.803 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:07.803 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:07.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:07.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:07.804 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:57:07.804 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:07.804 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:07.804 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:57:07.804 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:07.804 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:07.805 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:07.805 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:57:07.805 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:07.805 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:07.805 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:07.805 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:57:07.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:07.805 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:57:07.806 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:57:07.806 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:57:07.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:07.806 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:57:07Z" 2024-04-19T23:57:07.806 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:57:07.807 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.632785_307.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.632785Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:57:07Z" } 2024-04-19T23:57:07.807 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:57:07.808 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:07.808 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:07.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.632785_307.1 2024-04-19T23:57:12.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:12.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:12.810 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:57:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:57:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:57:12.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:12.918+0000 7f44df41c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:12.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:12.920+0000 7f44df41c780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:12.942 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:57:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:57:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:57:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:57:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:57:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:57:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:57:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:57:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:57:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:57:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:57:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:57:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:57:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:57:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:57:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:57:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:57:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:57:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:57:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:57:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:57:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:57:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:57:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:57:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:57:12Z" 2024-04-19T23:57:12.948 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:57:12.949 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:57:12Z" } 2024-04-19T23:57:12.949 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:57:12.949 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:12.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:12.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.559046_297.1 2024-04-19T23:57:17.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:17.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:17.951 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:57:18.025 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:57:18.025 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:57:18.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:18.192+0000 7fcea8339780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:18.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:18.195+0000 7fcea8339780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:18.223 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:57:18.223 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:57:18.223 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:57:18.223 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:57:18.223 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:57:18.223 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:57:18.223 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:18.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:18.223 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:57:18.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:18.224 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:57:18.224 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:18.224 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:18.224 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:57:18.224 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:18.224 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:57:18.224 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:18.224 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:57:18.224 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:18.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:18.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:18.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:18.225 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:57:18.225 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:18.225 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:18.225 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:57:18.225 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:18.225 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:18.225 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:18.225 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:57:18.226 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:18.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:18.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:18.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:18.226 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:57:18.226 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:18.226 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:18.226 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:57:18.226 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:18.226 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:18.226 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:18.226 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:57:18.227 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:18.228 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:18.228 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:18.228 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:57:18.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:18.228 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:57:18.228 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:57:18.228 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:57:18.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:18.228 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:57:18Z" 2024-04-19T23:57:18.228 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:57:18.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:57:18Z" } 2024-04-19T23:57:18.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:57:18.229 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:18.229 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:18.230 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.629562_306.1 2024-04-19T23:57:23.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:23.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:23.231 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:57:23.287 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:57:23.287 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:57:23.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:23.326+0000 7f8cb1624780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:23.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:23.327+0000 7f8cb1624780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:23.347 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:57:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:57:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:57:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:57:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:57:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:57:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:57:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:57:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:57:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:57:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:57:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:57:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:57:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:57:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:57:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:57:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:57:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:57:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:57:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:57:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:57:23Z" 2024-04-19T23:57:23.353 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:57:23.353 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:57:23Z" } 2024-04-19T23:57:23.354 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:57:23.354 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:23.354 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:23.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.554348_295.1 2024-04-19T23:57:28.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:28.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:28.356 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:57:28.420 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:57:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:57:28.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:28.485+0000 7f830d0ff780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:28.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:28.486+0000 7f830d0ff780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:28.531 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:57:28.531 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:57:28.531 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:57:28.531 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:57:28.531 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:57:28.531 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:57:28.531 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:28.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:28.531 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:57:28.531 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:28.531 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:57:28.532 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:28.533 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:28.533 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:57:28.533 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:28.533 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:28.533 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:28.533 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:57:28.533 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:28.533 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:28.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:28.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:28.533 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:57:28.534 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:28.534 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:28.534 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:57:28.534 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:28.534 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:28.534 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:28.534 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:57:28.534 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:28.534 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:28.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:28.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:57:28.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:28.536 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:57:28.536 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:57:28.536 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:57:28.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:28.536 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:57:28Z" 2024-04-19T23:57:28.536 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:57:28.537 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:57:28Z" } 2024-04-19T23:57:28.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:57:28.537 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:28.537 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:28.537 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.629562_306.1 2024-04-19T23:57:33.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:33.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:33.539 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:57:33.601 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:57:33.601 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:57:33.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:33.648+0000 7f742f504780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:33.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:33.649+0000 7f742f504780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:33.670 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:57:33.670 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:57:33.671 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:57:33.671 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:57:33.671 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:57:33.671 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:57:33.671 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:33.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:33.671 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:57:33.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:33.671 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:57:33.671 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:33.671 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:33.671 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:57:33.672 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:33.672 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:57:33.672 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:33.672 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:57:33.672 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:33.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:33.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:33.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:33.672 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:57:33.672 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:33.672 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:33.672 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:57:33.673 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:33.674 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:33.674 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:33.674 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:57:33.674 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:33.674 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:33.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:33.674 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:33.674 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:57:33.674 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:33.674 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:33.674 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:57:33.675 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:33.675 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:33.675 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:33.675 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:57:33.675 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:33.675 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:33.675 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:33.675 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:57:33.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:33.675 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:57:33.675 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:57:33.675 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:57:33.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:33.676 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:57:33Z" 2024-04-19T23:57:33.676 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:57:33.677 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:57:33Z" } 2024-04-19T23:57:33.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:57:33.677 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:33.677 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:33.677 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.554348_295.1 2024-04-19T23:57:38.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:38.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:38.679 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:57:38.740 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:57:38.740 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:57:38.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:38.808+0000 7f378b848780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:38.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:38.810+0000 7f378b848780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:38.837 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:57:38.837 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:57:38.837 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:57:38.837 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:57:38.837 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:57:38.838 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:57:38.838 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:38.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:38.838 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:57:38.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:38.838 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:57:38.838 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:38.838 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:38.838 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:57:38.838 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:38.838 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:57:38.839 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:38.839 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:57:38.839 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:38.839 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:38.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:38.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:38.839 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:57:38.839 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:38.839 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:38.839 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:57:38.839 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:38.839 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:38.840 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:38.840 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:57:38.840 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:38.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:38.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:38.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:38.840 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:57:38.840 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:38.840 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:38.840 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:57:38.840 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:57:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:57:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:57:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:38.842 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:38.842 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:38.842 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:57:38.842 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:38.842 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:38.842 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:38.842 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:57:38.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:38.842 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:57:38.842 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:57:38.842 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:57:38.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:38.843 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:57:38Z" 2024-04-19T23:57:38.843 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:57:38.844 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:57:38Z" } 2024-04-19T23:57:38.844 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:57:38.844 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:38.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:38.844 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.559046_297.1 2024-04-19T23:57:43.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:43.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:43.846 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:57:43.904 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:57:43.904 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:57:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:43.962+0000 7f27d40b0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:43.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:43.963+0000 7f27d40b0780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:43.983 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:57:43.983 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:57:43.983 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:57:43.983 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:57:43.984 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:57:43.984 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:57:43.984 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:43.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:43.984 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:57:43.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:43.984 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:57:43.984 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:43.984 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:43.984 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:57:43.984 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:43.985 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:57:43.985 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:43.985 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:57:43.985 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:43.985 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:43.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:43.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:43.985 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:57:43.985 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:43.985 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:43.985 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:57:43.985 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:43.986 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:43.986 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:43.986 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:57:43.986 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:43.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:43.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:43.986 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:43.986 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:57:43.986 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:43.986 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:43.986 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.577447_300.1", 2024-04-19T23:57:43.986 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:43.987 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:57:43Z" 2024-04-19T23:57:43.988 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:57:43.989 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.577447_300.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.577447Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:57:43Z" } 2024-04-19T23:57:43.989 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:57:43.990 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:43.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:43.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.577447_300.1 2024-04-19T23:57:48.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:48.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:48.991 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:57:49.050 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:57:49.050 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:57:49.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:49.107+0000 7f8e1b47a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:49.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:49.108+0000 7f8e1b47a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:49.135 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:57:49.135 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:57:49.135 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:57:49.135 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:57:49.135 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:57:49.135 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:57:49.135 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:49.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:49.135 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:57:49.136 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:49.136 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:57:49.136 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:49.136 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:49.136 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:57:49.136 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:49.136 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:57:49.136 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:49.136 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:57:49.136 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:49.136 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:49.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:57:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:57:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:57:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:49.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:49.138 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:49.138 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:57:49.138 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:49.138 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:49.138 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-19T23:57:49.138 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:49.138 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:49.138 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:49.138 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-19T23:57:49.138 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:49.139 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:49.140 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:49.140 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:57:49.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:49.140 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:57:49.140 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:57:49.140 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:57:49.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:49.140 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:57:49Z" 2024-04-19T23:57:49.140 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:57:49.141 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:57:49Z" } 2024-04-19T23:57:49.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:57:49.141 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:49.141 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:49.141 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.629562_306.1 2024-04-19T23:57:54.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:54.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:54.143 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:57:54.202 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:57:54.202 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:57:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:54.243+0000 7f54a639d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:54.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:54.244+0000 7f54a639d780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:54.265 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:57:54.265 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:57:54.265 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:57:54.265 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:57:54.265 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:57:54.265 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:57:54.265 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:54.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:54.265 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:57:54.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:54.265 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:57:54.265 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:54.266 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:54.266 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:57:54.266 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:54.266 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:57:54.266 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:54.266 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:57:54.266 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:54.266 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:54.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:54.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:54.267 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:57:54.267 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:54.267 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:54.267 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:57:54.267 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:54.267 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:54.267 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:54.267 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:57:54.267 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:54.267 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:54.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:54.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:54.268 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:57:54.268 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:54.268 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:54.268 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:57:54.268 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:54.268 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:54.268 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:54.268 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:57:54.268 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:54.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:54.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:54.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:54.269 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:57:54.269 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:54.269 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:54.269 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:57:54.269 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:54.269 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:54.269 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:54.269 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:57:54.270 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:54.270 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:54.270 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:54.270 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:57:54.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:54.270 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:57:54.270 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:57:54.270 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:57:54.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:54.270 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:57:54Z" 2024-04-19T23:57:54.271 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:57:54.271 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:57:54Z" } 2024-04-19T23:57:54.271 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:57:54.272 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:54.272 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:54.272 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.559046_297.1 2024-04-19T23:57:59.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:59.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:57:59.274 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:57:59.333 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:57:59.333 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:57:59.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:59.399+0000 7f917a8ae780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:59.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:57:59.415+0000 7f917a8ae780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:57:59.511 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:57:59.511 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:57:59.511 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:57:59.511 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:57:59.511 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:57:59.511 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:57:59.511 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:59.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:59.511 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:57:59.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:59.512 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:57:59.512 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:59.512 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:59.512 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:57:59.512 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:59.512 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:57:59.512 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:59.512 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:57:59.512 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:59.512 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:59.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:59.513 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:57:59.514 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:59.514 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:59.514 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:57:59.514 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:59.514 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:59.514 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:59.514 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:57:59.514 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:59.514 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:59.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:59.514 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:57:59.514 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:57:59.515 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:57:59.515 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:57:59.515 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:57:59.515 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:57:59.515 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:57:59.515 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:57:59.515 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:57:59.515 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:57:59.515 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:59.515 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:57:59.515 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:57:59.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:59.516 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:57:59.516 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:57:59.516 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:57:59.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:57:59.516 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:57:59Z" 2024-04-19T23:57:59.516 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:57:59.517 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:57:59Z" } 2024-04-19T23:57:59.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:57:59.517 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:59.517 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:57:59.518 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.554348_295.1 2024-04-19T23:58:04.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:04.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:04.519 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:58:04.580 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:58:04.580 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:58:04.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:04.688+0000 7f97ddc50780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:04.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:04.696+0000 7f97ddc50780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:04.779 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:58:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:58:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:58:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:58:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:58:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:58:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:04.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:58:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:58:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:58:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:58:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:58:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:04.780 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:04.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:04.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:04.782 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:58:04.782 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:04.782 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:04.782 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-19T23:58:04.782 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:04.782 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:04.782 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:04.782 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-19T23:58:04.782 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:04.782 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:04.783 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:58:04.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:04.784 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:58:04.784 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:58:04.784 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:58:04.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:04.784 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:58:04Z" 2024-04-19T23:58:04.784 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:58:04.785 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:58:04Z" } 2024-04-19T23:58:04.785 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:58:04.785 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:04.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:04.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.682721_308.1 2024-04-19T23:58:09.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:09.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:09.787 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:58:09.887 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:58:09.888 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:58:10.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:10.005+0000 7fe74ac4b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:10.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:10.006+0000 7fe74ac4b780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:10.025 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:58:10.025 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:58:10.025 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:58:10.025 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:58:10.025 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:58:10.025 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:58:10.025 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:10.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:10.025 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:58:10.025 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:10.026 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:58:10.026 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:10.026 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:10.026 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:58:10.026 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:10.026 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:58:10.026 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:10.026 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:58:10.026 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:10.026 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:10.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:10.026 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:10.027 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:58:10.027 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:10.027 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:10.027 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:58:10.027 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:10.027 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:10.027 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:10.027 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:58:10.027 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:10.027 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:10.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:10.028 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:10.028 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:58:10.028 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:10.028 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:10.028 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.687754_309.1", 2024-04-19T23:58:10.028 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:10.028 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:10.028 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:10.028 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-19T23:58:10.028 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:10.028 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:10.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:10.029 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:58:10.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:10.030 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:58:10.030 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:58:10.030 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:58:10.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:10.030 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:58:10Z" 2024-04-19T23:58:10.030 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:58:10.031 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.687754_309.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.687754Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:58:10Z" } 2024-04-19T23:58:10.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.687754_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.687754Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:58:10.031 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:10.031 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.687754_309.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:10.031 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.687754_309.1 2024-04-19T23:58:15.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:15.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:15.033 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:58:15.089 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:58:15.089 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:58:15.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:15.139+0000 7f1dfef6a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:15.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:15.140+0000 7f1dfef6a780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:15.161 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:58:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:58:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:58:15.161 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:58:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:58:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:58:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:58:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:58:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:15.162 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:58:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:58:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:58:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:58:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:15.163 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:58:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:58:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:15.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:58:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:58:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:58:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:15.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:58:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:58:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:58:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:15.166 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:58:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:58:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:58:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:58:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:15.167 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:58:15Z" 2024-04-19T23:58:15.167 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:58:15.168 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:58:15Z" } 2024-04-19T23:58:15.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:58:15.168 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:15.168 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:15.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.576049_299.1 2024-04-19T23:58:20.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:20.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:20.169 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:58:20.228 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:58:20.228 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:58:20.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:20.277+0000 7f1d976f5780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:20.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:20.278+0000 7f1d976f5780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:20.300 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:58:20.300 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:58:20.300 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:58:20.301 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:58:20.301 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:58:20.301 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:58:20.301 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:20.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:20.301 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:58:20.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:20.301 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:58:20.301 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:20.301 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:20.301 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:58:20.301 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:20.302 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:58:20.302 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:20.302 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:58:20.302 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:20.302 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:20.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:20.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:20.302 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:58:20.302 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:20.302 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:20.302 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.609968_303.1", 2024-04-19T23:58:20.303 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:20.304 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:20.304 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:20.304 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-19T23:58:20.304 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:20.304 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:20.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:20.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:20.304 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:58:20.304 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:20.304 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:20.304 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:58:20.304 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:20.305 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:20.305 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:20.305 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:58:20.305 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:20.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:20.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:20.305 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:58:20.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:20.306 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:58:20.306 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:58:20.306 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:58:20.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:20.306 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:58:20Z" 2024-04-19T23:58:20.306 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:58:20.307 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.609968_303.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.609968Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:58:20Z" } 2024-04-19T23:58:20.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:58:20.307 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:20.307 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:20.307 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.609968_303.1 2024-04-19T23:58:25.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:25.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:25.309 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:58:25.367 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:58:25.367 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:58:25.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:25.414+0000 7fc80e0a4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:25.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:25.415+0000 7fc80e0a4780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:25.434 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:58:25.434 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:58:25.434 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:58:25.435 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:58:25.435 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:58:25.435 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:58:25.435 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:25.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:25.435 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:58:25.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:25.435 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:58:25.435 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:25.435 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:25.435 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:58:25.435 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:25.436 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:58:25.436 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:25.436 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:58:25.436 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:25.436 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:25.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:25.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:25.436 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:58:25.436 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:25.436 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:25.436 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:58:25.437 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:25.437 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:25.437 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:25.437 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:58:25.437 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:25.437 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:25.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:25.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:25.437 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:58:25.437 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:25.437 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:25.437 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.606676_302.1", 2024-04-19T23:58:25.438 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:25.438 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:25.438 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:25.438 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-19T23:58:25.438 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:25.438 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:25.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:25.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:25.438 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:58:25.438 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:25.438 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:25.439 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:58:25.439 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:25.439 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:25.439 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:25.439 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:58:25.439 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:25.439 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:25.439 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:25.439 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:58:25.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:25.439 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:58:25.440 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:58:25.440 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:58:25.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:25.440 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:58:25Z" 2024-04-19T23:58:25.440 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:58:25.441 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.606676_302.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.606676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:58:25Z" } 2024-04-19T23:58:25.441 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:58:25.441 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:25.441 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:25.441 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.606676_302.1 2024-04-19T23:58:30.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:30.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:30.443 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:58:30.503 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:58:30.503 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:58:30.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:30.560+0000 7fc85ecb1780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:30.561+0000 7fc85ecb1780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:30.590 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:58:30.590 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:58:30.590 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:58:30.590 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:58:30.590 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:58:30.590 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:58:30.590 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:30.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:30.590 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:58:30.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:30.591 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:58:30.592 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:30.592 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:30.592 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:58:30.592 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:30.592 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:30.592 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:30.592 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:58:30.592 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:30.592 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:30.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:30.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:30.592 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.554348_295.1", 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:58:30.593 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:30.594 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:30.594 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:58:30.594 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:30.594 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:30.594 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:30.594 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:58:30.594 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:30.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:30.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:30.594 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:58:30.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:30.594 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:58:30.595 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:58:30.595 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:58:30.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:30.595 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:58:30Z" 2024-04-19T23:58:30.595 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:58:30.596 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.554348_295.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.554348Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:58:30Z" } 2024-04-19T23:58:30.596 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:58:30.596 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:30.596 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:30.596 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.554348_295.1 2024-04-19T23:58:35.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:35.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:35.598 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:58:35.654 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:58:35.654 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:58:35.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:35.697+0000 7f2b6e422780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:35.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:35.698+0000 7f2b6e422780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:35.721 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:58:35.721 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:58:35.721 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:58:35.721 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:58:35.721 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:58:35.722 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:58:35.722 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:35.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:35.722 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:58:35.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:35.722 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:58:35.722 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:35.722 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:35.722 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:58:35.722 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:35.722 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:58:35.723 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:35.723 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:58:35.723 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:35.723 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:35.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:35.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:35.723 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:58:35.723 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:35.723 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:35.723 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:58:35.723 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:35.724 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:35.724 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:35.724 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:58:35.724 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:35.724 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:35.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:35.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:35.724 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:58:35.724 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:35.724 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:35.724 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.742452_317.1", 2024-04-19T23:58:35.724 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:35.725 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:35.725 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:35.725 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.742452Z", 2024-04-19T23:58:35.725 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:35.725 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:35.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:35.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:35.725 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:58:35.725 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:35.725 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:35.725 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:58:35.726 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:35.726 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:35.726 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:35.726 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:58:35.726 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:35.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:35.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:35.726 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:58:35.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:35.726 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:58:35.726 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:58:35.727 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:58:35.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:35.727 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:58:35Z" 2024-04-19T23:58:35.727 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:58:35.728 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.742452_317.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.742452Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:58:35Z" } 2024-04-19T23:58:35.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.742452_317.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.742452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:58:35.728 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:35.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.742452_317.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:35.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.742452_317.1 2024-04-19T23:58:40.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:40.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:40.730 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:58:40.790 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:58:40.790 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:58:40.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:40.841+0000 7fb7b96bc780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:40.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:40.843+0000 7fb7b96bc780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:40.870 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:58:40.871 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:58:40.871 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:58:40.871 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:58:40.871 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:58:40.871 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:58:40.871 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:40.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:40.871 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:58:40.871 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:40.871 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:58:40.871 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:40.871 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:40.872 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:58:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:58:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:58:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.793769_324.1", 2024-04-19T23:58:40.874 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:40.874 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:40.874 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:40.874 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.793769Z", 2024-04-19T23:58:40.874 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:40.874 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:40.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:40.874 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:40.874 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:58:40.874 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:40.874 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:58:40.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:40.876 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:58:40Z" 2024-04-19T23:58:40.876 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:58:40.876 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.793769_324.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.793769Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:58:40Z" } 2024-04-19T23:58:40.877 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.793769_324.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.793769Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:58:40.877 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:40.877 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.793769_324.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:40.877 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.793769_324.1 2024-04-19T23:58:45.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:45.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:45.878 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:58:45.938 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:58:45.938 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:58:45.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:45.996+0000 7f6e952c7780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:45.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:45.998+0000 7f6e952c7780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:46.025 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:58:46.025 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:58:46.025 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:58:46.025 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:58:46.026 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:58:46.026 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:58:46.026 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:46.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:46.026 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:58:46.026 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:46.026 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:58:46.026 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:46.026 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:46.027 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:58:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:58:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:58:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.609968_303.1", 2024-04-19T23:58:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-19T23:58:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:58:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:58:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:58:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:58:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:58:46.031 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:58:46.031 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:58:46.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:46.031 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:58:45Z" 2024-04-19T23:58:46.031 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:58:46.032 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.609968_303.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.609968Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:58:45Z" } 2024-04-19T23:58:46.032 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:58:46.032 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:46.032 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:46.032 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.609968_303.1 2024-04-19T23:58:51.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:51.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:51.034 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:58:51.093 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:58:51.093 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:58:51.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:51.151+0000 7fe2b0a20780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:51.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:51.152+0000 7fe2b0a20780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:51.176 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:58:51.176 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:58:51.176 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:58:51.176 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:58:51.176 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:58:51.176 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:58:51.176 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:51.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:51.177 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:58:51.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:51.177 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:58:51.177 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:51.177 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:51.177 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:58:51.177 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:51.177 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:58:51.177 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:51.177 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:58:51.177 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:51.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:51.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:51.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:51.178 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:58:51.178 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:51.178 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:51.178 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:58:51.178 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:51.178 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:51.178 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:51.178 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:58:51.178 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:51.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:51.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:51.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:51.179 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:58:51.179 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:51.179 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:51.179 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.606676_302.1", 2024-04-19T23:58:51.179 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:51.179 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:51.179 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:51.179 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-19T23:58:51.179 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:51.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:51.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:51.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:51.180 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:58:51.180 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:51.180 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:51.180 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:58:51.180 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:51.180 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:51.180 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:51.180 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:58:51.181 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:51.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:51.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:51.181 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:58:51.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:51.181 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:58:51.181 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:58:51.181 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:58:51.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:51.181 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:58:51Z" 2024-04-19T23:58:51.181 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:58:51.182 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.606676_302.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.606676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:58:51Z" } 2024-04-19T23:58:51.182 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:58:51.182 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:51.183 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:51.183 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.606676_302.1 2024-04-19T23:58:56.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:56.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:58:56.185 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:58:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:58:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:58:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:56.280+0000 7f6311a32780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:58:56.281+0000 7f6311a32780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:58:56.300 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:58:56.300 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:58:56.301 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:56.302 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:58:56.302 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:56.302 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:58:56.302 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:56.302 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:56.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:56.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:56.302 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:58:56.302 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:56.302 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:56.302 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:58:56.302 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:56.303 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:56.303 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:56.303 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:58:56.303 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:56.303 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:56.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:56.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:56.303 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:58:56.303 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:56.303 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:56.303 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.614143_305.1", 2024-04-19T23:58:56.303 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:58:56.304 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:58:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:58:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:58:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:58:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:58:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:58:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:58:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:58:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:58:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:58:56.305 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:58:56Z" 2024-04-19T23:58:56.306 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:58:56.306 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.614143_305.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.614143Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:58:56Z" } 2024-04-19T23:58:56.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:58:56.307 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:56.307 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:58:56.307 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.614143_305.1 2024-04-19T23:59:01.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:59:01.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:59:01.308 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:59:01.368 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:59:01.369 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:59:01.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:59:01.541+0000 7f0b5eb80780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:59:01.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:59:01.543+0000 7f0b5eb80780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:59:01.576 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:59:01.576 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:59:01.576 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:59:01.576 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:59:01.576 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:59:01.576 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:59:01.576 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:01.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:01.576 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:59:01.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:01.576 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:59:01.577 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:01.577 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:01.577 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:59:01.577 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:01.577 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:59:01.577 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:01.577 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:59:01.577 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:01.577 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:01.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:01.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:01.578 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:59:01.578 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:01.578 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:01.578 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:59:01.578 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:01.578 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:01.578 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:01.578 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:59:01.579 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:01.579 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:01.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:01.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:01.579 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:59:01.579 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:01.579 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:01.579 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.606676_302.1", 2024-04-19T23:59:01.579 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:01.579 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:01.579 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:01.580 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-19T23:59:01.580 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:01.580 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:01.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:01.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:01.580 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:59:01.580 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:01.580 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:01.580 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:59:01.580 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:01.581 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:01.581 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:01.581 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:59:01.581 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:01.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:01.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:01.581 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:59:01.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:01.581 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:59:01.581 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:59:01.581 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:59:01.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:01.582 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:59:01Z" 2024-04-19T23:59:01.582 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:59:01.582 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.606676_302.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.606676Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:59:01Z" } 2024-04-19T23:59:01.583 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:59:01.583 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:59:01.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:59:01.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.606676_302.1 2024-04-19T23:59:06.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:59:06.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:59:06.585 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:59:06.660 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:59:06.660 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:59:06.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:59:06.698+0000 7f60ed7e2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:59:06.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:59:06.699+0000 7f60ed7e2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:59:06.718 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:59:06.718 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:59:06.718 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:59:06.718 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:59:06.718 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:59:06.718 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:59:06.718 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:06.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:06.718 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:59:06.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:06.719 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:59:06.719 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:06.719 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:06.719 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:59:06.719 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:06.719 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:59:06.719 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:06.719 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:59:06.719 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:06.719 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:06.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:06.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.577447_300.1", 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:06.721 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:59:06.722 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:06.722 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:06.722 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:59:06.722 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:06.722 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:06.722 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:06.722 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:59:06.722 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:06.722 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:06.722 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:06.722 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:59:06.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:06.723 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:59:06.723 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:59:06.723 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:59:06.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:06.723 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:59:06Z" 2024-04-19T23:59:06.723 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:59:06.724 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.577447_300.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.577447Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:59:06Z" } 2024-04-19T23:59:06.724 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:59:06.724 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:59:06.724 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:59:06.724 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.577447_300.1 2024-04-19T23:59:11.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:59:11.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:59:11.726 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:59:11.794 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:59:11.795 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:59:11.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:59:11.848+0000 7eff81138780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:59:11.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:59:11.849+0000 7eff81138780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:59:11.872 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:59:11.872 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:59:11.872 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:59:11.873 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:59:11.873 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:59:11.873 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:59:11.873 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:11.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:11.873 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:59:11.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:11.873 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:59:11.873 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:11.873 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:11.873 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:59:11.873 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:11.874 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:59:11.874 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:11.874 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:59:11.874 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:11.874 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:11.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:11.874 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:11.874 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:59:11.874 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:11.874 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:11.874 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:59:11.874 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:11.875 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:11.875 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:11.875 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:59:11.875 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:11.875 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:11.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:11.875 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:11.875 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:59:11.875 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:11.875 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:11.875 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.572795_298.1", 2024-04-19T23:59:11.875 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:11.876 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:11.877 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:11.877 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:59:11.877 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:11.877 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:11.877 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:11.877 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:59:11.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:11.877 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:59:11.877 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:59:11.877 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:59:11.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:11.877 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:59:11Z" 2024-04-19T23:59:11.878 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:59:11.878 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.572795_298.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.572795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:59:11Z" } 2024-04-19T23:59:11.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:59:11.879 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:59:11.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:59:11.879 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.572795_298.1 2024-04-19T23:59:14.531 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-19T23:59:14.529+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-19T23:59:16.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:59:16.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:59:16.881 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:59:16.935 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:59:16.935 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:59:16.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:59:16.991+0000 7f4597930780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:59:16.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:59:16.992+0000 7f4597930780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:59:17.018 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:59:17.019 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:59:17.019 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:59:17.019 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:59:17.019 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:59:17.019 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:59:17.019 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:17.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:17.019 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:59:17.019 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:17.019 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:59:17.019 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:17.020 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:17.020 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:59:17.020 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:17.020 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:59:17.020 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:17.020 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:59:17.020 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:17.020 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:17.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:17.020 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:17.020 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:59:17.020 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:17.021 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:17.022 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.576049_299.1", 2024-04-19T23:59:17.022 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:17.022 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:17.022 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:17.022 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-19T23:59:17.022 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:17.022 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:17.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:17.022 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:17.022 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:59:17.022 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:17.023 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:17.023 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:59:17.023 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:17.023 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:17.023 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:17.023 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:59:17.023 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:17.023 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:17.023 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:17.023 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:59:17.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:17.024 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:59:17.024 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:59:17.024 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:59:17.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:17.024 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:59:16Z" 2024-04-19T23:59:17.024 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:59:17.025 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.576049_299.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.576049Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:59:16Z" } 2024-04-19T23:59:17.025 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:59:17.025 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:59:17.025 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:59:17.025 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.576049_299.1 2024-04-19T23:59:22.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:59:22.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:59:22.027 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:59:22.086 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:59:22.086 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:59:22.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:59:22.134+0000 7f6761611780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:59:22.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:59:22.135+0000 7f6761611780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:59:22.159 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:59:22.159 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:59:22.159 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:59:22.159 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:59:22.159 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:22.160 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:59:22.161 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:22.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:22.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:22.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:22.161 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:59:22.161 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:22.161 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:22.161 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:59:22.161 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:22.161 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:22.161 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:22.161 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:59:22.162 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:22.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:22.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:22.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:22.162 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:59:22.162 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:22.162 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:22.162 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.614143_305.1", 2024-04-19T23:59:22.162 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:22.163 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:59:22Z" 2024-04-19T23:59:22.164 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:59:22.165 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.614143_305.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.614143Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:59:22Z" } 2024-04-19T23:59:22.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:59:22.166 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:59:22.166 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:59:22.166 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.614143_305.1 2024-04-19T23:59:27.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:59:27.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-19T23:59:27.167 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-19T23:59:27.226 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:59:27.226 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:59:27.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:59:27.269+0000 7f612d2c2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:59:27.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-19T23:59:27.270+0000 7f612d2c2780 20 RGW-SYNC:meta: read sync status 2024-04-19T23:59:27.290 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:59:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-19T23:59:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-19T23:59:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-19T23:59:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-19T23:59:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:59:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-19T23:59:27.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-19T23:59:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-19T23:59:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-19T23:59:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-19T23:59:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:27.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-19T23:59:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-19T23:59:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-19T23:59:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-19T23:59:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.559046_297.1", 2024-04-19T23:59:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-19T23:59:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:27.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-19T23:59:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-19T23:59:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-19T23:59:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-19T23:59:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-19T23:59:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-19T23:59:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-19T23:59:27.294 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:59:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-19T23:59:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-19T23:59:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-19T23:59:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:27.295 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-19T23:59:27Z" 2024-04-19T23:59:27.295 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:59:27.296 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.559046_297.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.559046Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-19T23:59:27Z" } 2024-04-19T23:59:27.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-19T23:59:27.297 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:59:27.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-19T23:59:27.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.559046_297.1 2024-04-19T23:59:32.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2024-04-19T23:59:32.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:59:32.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:59:32.303 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-19T23:59:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:59:32.363 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:59:32.388 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:59:32.388 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-19T23:59:32.388 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zone2", 2024-04-19T23:59:32.388 INFO:teuthology.orchestra.run.smithi031.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-04-19T23:59:32.388 INFO:teuthology.orchestra.run.smithi031.stdout: "control_pool": "test-zone2.rgw.control", 2024-04-19T23:59:32.388 INFO:teuthology.orchestra.run.smithi031.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-04-19T23:59:32.389 INFO:teuthology.orchestra.run.smithi031.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-04-19T23:59:32.389 INFO:teuthology.orchestra.run.smithi031.stdout: "log_pool": "test-zone2.rgw.log", 2024-04-19T23:59:32.389 INFO:teuthology.orchestra.run.smithi031.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-04-19T23:59:32.389 INFO:teuthology.orchestra.run.smithi031.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-04-19T23:59:32.389 INFO:teuthology.orchestra.run.smithi031.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-04-19T23:59:32.389 INFO:teuthology.orchestra.run.smithi031.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-04-19T23:59:32.389 INFO:teuthology.orchestra.run.smithi031.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-04-19T23:59:32.389 INFO:teuthology.orchestra.run.smithi031.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-04-19T23:59:32.389 INFO:teuthology.orchestra.run.smithi031.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-04-19T23:59:32.389 INFO:teuthology.orchestra.run.smithi031.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-04-19T23:59:32.389 INFO:teuthology.orchestra.run.smithi031.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-04-19T23:59:32.390 INFO:teuthology.orchestra.run.smithi031.stdout: "system_key": { 2024-04-19T23:59:32.390 INFO:teuthology.orchestra.run.smithi031.stdout: "access_key": "TWO98NBPIQJFTC6N", 2024-04-19T23:59:32.390 INFO:teuthology.orchestra.run.smithi031.stdout: "secret_key": "WVtqgebnqURqV1qdDSut7FusWVqSrgr8" 2024-04-19T23:59:32.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.390 INFO:teuthology.orchestra.run.smithi031.stdout: "placement_pools": [ 2024-04-19T23:59:32.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:32.390 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "default-placement", 2024-04-19T23:59:32.390 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-19T23:59:32.390 INFO:teuthology.orchestra.run.smithi031.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-04-19T23:59:32.390 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_classes": { 2024-04-19T23:59:32.390 INFO:teuthology.orchestra.run.smithi031.stdout: "STANDARD": { 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout: "compression_type": "random" 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout: "index_type": 0, 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout: "inline_data": true 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-04-19T23:59:32.391 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:59:32.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:59:32.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-19T23:59:32.393 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-19T23:59:32.447 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-19T23:59:32.447 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-19T23:59:32.521 INFO:teuthology.orchestra.run.smithi031.stderr:Sending period to new master zone 73f46f02-c7ab-4426-a267-2273af911da5 2024-04-19T23:59:32.555 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-19T23:59:32.555 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:59:32.555 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 4, 2024-04-19T23:59:32.555 INFO:teuthology.orchestra.run.smithi031.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-19T23:59:32.555 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": [], 2024-04-19T23:59:32.555 INFO:teuthology.orchestra.run.smithi031.stdout: "period_map": { 2024-04-19T23:59:32.555 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-19T23:59:32.556 INFO:teuthology.orchestra.run.smithi031.stdout: "zonegroups": [ 2024-04-19T23:59:32.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:32.556 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "test-zonegroup", 2024-04-19T23:59:32.556 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zonegroup", 2024-04-19T23:59:32.556 INFO:teuthology.orchestra.run.smithi031.stdout: "api_name": "", 2024-04-19T23:59:32.556 INFO:teuthology.orchestra.run.smithi031.stdout: "is_master": true, 2024-04-19T23:59:32.556 INFO:teuthology.orchestra.run.smithi031.stdout: "endpoints": [ 2024-04-19T23:59:32.556 INFO:teuthology.orchestra.run.smithi031.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-19T23:59:32.556 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:59:32.556 INFO:teuthology.orchestra.run.smithi031.stdout: "hostnames": [], 2024-04-19T23:59:32.556 INFO:teuthology.orchestra.run.smithi031.stdout: "hostnames_s3website": [], 2024-04-19T23:59:32.556 INFO:teuthology.orchestra.run.smithi031.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: "zones": [ 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zone2", 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: "endpoints": [ 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: "log_meta": false, 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: "log_data": true, 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_index_max_shards": 11, 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: "read_only": false, 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_type": "", 2024-04-19T23:59:32.557 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from_all": false, 2024-04-19T23:59:32.558 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from": [], 2024-04-19T23:59:32.558 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_zone": "", 2024-04-19T23:59:32.558 INFO:teuthology.orchestra.run.smithi031.stdout: "supported_features": [ 2024-04-19T23:59:32.558 INFO:teuthology.orchestra.run.smithi031.stdout: "compress-encrypted", 2024-04-19T23:59:32.558 INFO:teuthology.orchestra.run.smithi031.stdout: "resharding" 2024-04-19T23:59:32.558 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:59:32.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:32.558 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:59:32.558 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zone1", 2024-04-19T23:59:32.558 INFO:teuthology.orchestra.run.smithi031.stdout: "endpoints": [ 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: "log_meta": false, 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: "log_data": true, 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_index_max_shards": 11, 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: "read_only": false, 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_type": "", 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from_all": true, 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from": [], 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_zone": "", 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: "supported_features": [ 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: "compress-encrypted", 2024-04-19T23:59:32.559 INFO:teuthology.orchestra.run.smithi031.stdout: "resharding" 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zone3", 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: "endpoints": [ 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: "log_meta": false, 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: "log_data": true, 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_index_max_shards": 11, 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: "read_only": false, 2024-04-19T23:59:32.560 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_type": "", 2024-04-19T23:59:32.561 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from_all": true, 2024-04-19T23:59:32.561 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from": [], 2024-04-19T23:59:32.561 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_zone": "", 2024-04-19T23:59:32.561 INFO:teuthology.orchestra.run.smithi031.stdout: "supported_features": [ 2024-04-19T23:59:32.561 INFO:teuthology.orchestra.run.smithi031.stdout: "compress-encrypted", 2024-04-19T23:59:32.561 INFO:teuthology.orchestra.run.smithi031.stdout: "resharding" 2024-04-19T23:59:32.561 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:59:32.561 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:32.561 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:59:32.561 INFO:teuthology.orchestra.run.smithi031.stdout: "placement_targets": [ 2024-04-19T23:59:32.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:32.561 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default-placement", 2024-04-19T23:59:32.562 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-19T23:59:32.562 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_classes": [ 2024-04-19T23:59:32.562 INFO:teuthology.orchestra.run.smithi031.stdout: "STANDARD" 2024-04-19T23:59:32.562 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:59:32.562 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:32.562 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:59:32.562 INFO:teuthology.orchestra.run.smithi031.stdout: "default_placement": "default-placement", 2024-04-19T23:59:32.562 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:59:32.562 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_policy": { 2024-04-19T23:59:32.562 INFO:teuthology.orchestra.run.smithi031.stdout: "groups": [] 2024-04-19T23:59:32.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.562 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled_features": [ 2024-04-19T23:59:32.563 INFO:teuthology.orchestra.run.smithi031.stdout: "resharding" 2024-04-19T23:59:32.563 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:59:32.563 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:32.563 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-19T23:59:32.563 INFO:teuthology.orchestra.run.smithi031.stdout: "short_zone_ids": [ 2024-04-19T23:59:32.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:32.563 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-19T23:59:32.563 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 2010863401 2024-04-19T23:59:32.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:32.563 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:59:32.563 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 1221149675 2024-04-19T23:59:32.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-19T23:59:32.575 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-19T23:59:32.575 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 1478903101 2024-04-19T23:59:32.575 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:32.575 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-19T23:59:32.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.575 INFO:teuthology.orchestra.run.smithi031.stdout: "master_zonegroup": "test-zonegroup", 2024-04-19T23:59:32.575 INFO:teuthology.orchestra.run.smithi031.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-19T23:59:32.575 INFO:teuthology.orchestra.run.smithi031.stdout: "period_config": { 2024-04-19T23:59:32.576 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_quota": { 2024-04-19T23:59:32.576 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false, 2024-04-19T23:59:32.576 INFO:teuthology.orchestra.run.smithi031.stdout: "check_on_raw": false, 2024-04-19T23:59:32.576 INFO:teuthology.orchestra.run.smithi031.stdout: "max_size": -1, 2024-04-19T23:59:32.576 INFO:teuthology.orchestra.run.smithi031.stdout: "max_size_kb": 0, 2024-04-19T23:59:32.576 INFO:teuthology.orchestra.run.smithi031.stdout: "max_objects": -1 2024-04-19T23:59:32.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.576 INFO:teuthology.orchestra.run.smithi031.stdout: "user_quota": { 2024-04-19T23:59:32.576 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false, 2024-04-19T23:59:32.576 INFO:teuthology.orchestra.run.smithi031.stdout: "check_on_raw": false, 2024-04-19T23:59:32.576 INFO:teuthology.orchestra.run.smithi031.stdout: "max_size": -1, 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max_size_kb": 0, 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max_objects": -1 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: "user_ratelimit": { 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_ops": 0, 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_ops": 0, 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_bytes": 0, 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_bytes": 0, 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_ratelimit": { 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_ops": 0, 2024-04-19T23:59:32.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_ops": 0, 2024-04-19T23:59:32.578 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_bytes": 0, 2024-04-19T23:59:32.578 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_bytes": 0, 2024-04-19T23:59:32.578 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false 2024-04-19T23:59:32.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.578 INFO:teuthology.orchestra.run.smithi031.stdout: "anonymous_ratelimit": { 2024-04-19T23:59:32.578 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_ops": 0, 2024-04-19T23:59:32.578 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_ops": 0, 2024-04-19T23:59:32.578 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_bytes": 0, 2024-04-19T23:59:32.578 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_bytes": 0, 2024-04-19T23:59:32.578 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false 2024-04-19T23:59:32.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-19T23:59:32.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-19T23:59:32.579 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-19T23:59:32.579 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_name": "test-realm", 2024-04-19T23:59:32.579 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-19T23:59:32.579 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-19T23:59:32.580 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2024-04-20T00:00:32.581 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-48 2024-04-20T00:00:32.581 DEBUG:boto:path=/aktlji-48/ 2024-04-20T00:00:32.582 DEBUG:boto:auth_path=/aktlji-48/ 2024-04-20T00:00:32.582 DEBUG:boto:Method: PUT 2024-04-20T00:00:32.582 DEBUG:boto:Path: /aktlji-48/ 2024-04-20T00:00:32.582 DEBUG:boto:Data: 2024-04-20T00:00:32.582 DEBUG:boto:Headers: {} 2024-04-20T00:00:32.582 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:00:32.582 DEBUG:boto:Port: 8000 2024-04-20T00:00:32.582 DEBUG:boto:Params: {} 2024-04-20T00:00:32.583 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:00:32.583 DEBUG:boto:Token: None 2024-04-20T00:00:32.583 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:00:32 GMT /aktlji-48/ 2024-04-20T00:00:32.583 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9v2636GG6Ua7LLdBB0Jctb4PvWw= 2024-04-20T00:00:32.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:00:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9v2636GG6Ua7LLdBB0Jctb4PvWw=', 'Content-Length': '0'} 2024-04-20T00:00:32.658 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b062126b2bc8d60a-00662305a0-153e-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:00:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:00:32.659 DEBUG:boto:path=/aktlji-48/testredirect 2024-04-20T00:00:32.659 DEBUG:boto:auth_path=/aktlji-48/testredirect 2024-04-20T00:00:32.659 DEBUG:boto:Method: PUT 2024-04-20T00:00:32.659 DEBUG:boto:Path: /aktlji-48/testredirect 2024-04-20T00:00:32.659 DEBUG:boto:Data: 2024-04-20T00:00:32.660 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-20T00:00:32.660 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:00:32.660 DEBUG:boto:Port: 8000 2024-04-20T00:00:32.660 DEBUG:boto:Params: {} 2024-04-20T00:00:32.660 DEBUG:boto:Token: None 2024-04-20T00:00:32.660 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Sat, 20 Apr 2024 00:00:32 GMT /aktlji-48/testredirect 2024-04-20T00:00:32.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GeDC9kyOg3OrEFAmrztSNFpi8X0= 2024-04-20T00:00:32.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:00:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GeDC9kyOg3OrEFAmrztSNFpi8X0='} 2024-04-20T00:00:32.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000277fc3e1cc2baf7e-00662305a0-153e-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:00:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:00:32.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:32.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:32.673 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:00:32.736 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:00:32.736 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:00:32.774 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:00:32.790 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:00:32.790 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:00:32.790 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:00:32.790 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:00:32.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:00:32.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:00:32.791 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:00:32.791 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:00:32.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:00:32.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:00:32.791 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:00:32.791 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:00:32.655231Z" 2024-04-20T00:00:32.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:00:32.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:00:32.791 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:00:32.791 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:00:32.792 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:00:32.792 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:00:32.793 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:32.793 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:00:32.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:32.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:32.793 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:00:32.899 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:00:32.899 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:00:32.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:00:32.968+0000 7f3feb595780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:00:32.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:00:32.969+0000 7f3feb595780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:00:32.993 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:00:32.993 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:00:32.993 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:00:32.993 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:00:32.993 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:00:32.993 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:00:32.993 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:32.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:32.994 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:00:32.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:32.994 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:00:32.994 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:32.994 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:32.994 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:00:32.994 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:32.994 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:00:32.994 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:32.994 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:00:32.995 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:32.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:32.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:32.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:32.995 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:00:32.995 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:32.995 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:32.995 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:00:32.995 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:32.995 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:32.995 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:32.996 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:00:32.996 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:32.996 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:32.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:32.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:32.996 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:00:32.996 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:32.996 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:32.996 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-20T00:00:32.997 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:32.997 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:32.997 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:32.997 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T00:00:32.997 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:32.997 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:32.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:32.997 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:32.997 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:00:32.997 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:32.997 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:32.998 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:00:32.998 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:32.998 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:32.998 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:32.998 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:00:32.998 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:32.998 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:32.998 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:32.998 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:00:32.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:32.998 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:00:32.998 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:00:32.999 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:00:32.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:32.999 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:00:32Z" 2024-04-20T00:00:32.999 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:00:33.000 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:00:32Z" } 2024-04-20T00:00:33.000 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:00:33.000 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:33.000 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:33.000 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.682721_308.1 2024-04-20T00:00:38.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:38.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:38.001 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:00:38.059 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:00:38.060 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:00:38.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:00:38.099+0000 7f65752f3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:00:38.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:00:38.100+0000 7f65752f3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:00:38.123 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:00:38.123 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:00:38.123 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:00:38.123 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:00:38.123 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:00:38.124 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:00:38.124 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:38.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:38.124 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:00:38.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:38.124 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:00:38.124 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:38.124 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:38.124 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:00:38.124 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:38.125 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:00:38.125 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:38.125 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:00:38.125 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:38.125 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:38.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:38.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:38.125 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:00:38.125 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:38.125 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:38.126 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:00:38.126 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:38.126 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:38.126 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:38.126 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:00:38.126 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:38.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:38.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:38.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:38.126 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:00:38.126 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:38.127 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:38.127 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.808187_326.1", 2024-04-20T00:00:38.127 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:38.127 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:38.127 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:38.127 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T00:00:38.127 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:38.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:38.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:38.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:38.128 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:00:38.128 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:38.128 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:38.128 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:00:38.128 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:38.128 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:38.128 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:38.128 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:00:38.129 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:38.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:38.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:38.129 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:00:38.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:38.129 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:00:38.129 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:00:38.129 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:00:38.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:38.129 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:00:38Z" 2024-04-20T00:00:38.129 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:00:38.131 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.808187_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.808187Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:00:38Z" } 2024-04-20T00:00:38.131 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.808187_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.808187Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:00:38.131 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:38.131 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.808187_326.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:38.131 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.808187_326.1 2024-04-20T00:00:43.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:43.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:43.132 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:00:43.191 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:00:43.191 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:00:43.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:00:43.244+0000 7f83eac58780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:00:43.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:00:43.246+0000 7f83eac58780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:00:43.266 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:00:43.266 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:00:43.266 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:00:43.266 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:00:43.266 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:00:43.266 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:00:43.266 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:43.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:43.267 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:00:43.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:43.267 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:00:43.267 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:43.267 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:43.267 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:00:43.267 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:43.267 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:00:43.267 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:43.267 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:00:43.268 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:43.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:43.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:43.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:43.268 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:00:43.268 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:43.268 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:43.268 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:00:43.268 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:43.268 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:43.268 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:43.269 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:00:43.269 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:43.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:43.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:43.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:43.269 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:00:43.269 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:43.269 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:43.269 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.687754_309.1", 2024-04-20T00:00:43.269 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:43.269 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:43.270 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:43.270 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-20T00:00:43.270 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:43.270 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:43.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:43.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:43.270 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:00:43.270 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:43.270 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:43.270 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:00:43.270 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:43.271 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:43.271 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:43.271 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:00:43.271 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:43.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:43.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:43.271 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:00:43.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:43.271 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:00:43.271 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:00:43.271 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:00:43.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:43.272 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:00:43Z" 2024-04-20T00:00:43.272 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:00:43.273 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.687754_309.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.687754Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:00:43Z" } 2024-04-20T00:00:43.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.687754_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.687754Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:00:43.273 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:43.273 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.687754_309.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:43.273 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.687754_309.1 2024-04-20T00:00:48.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:48.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:48.275 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:00:48.336 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:00:48.336 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:00:48.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:00:48.372+0000 7f8e33a59780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:00:48.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:00:48.373+0000 7f8e33a59780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:00:48.393 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:00:48.393 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:00:48.393 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:00:48.393 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:00:48.393 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:00:48.393 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:00:48.393 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:48.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:48.394 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:00:48.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:48.394 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:00:48.394 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:48.394 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:48.394 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:00:48.394 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:48.394 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:00:48.395 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:48.395 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:00:48.395 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:48.395 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:48.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:48.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:48.395 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:00:48.395 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:48.395 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:48.395 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:00:48.396 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:48.396 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:48.396 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:48.396 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:00:48.396 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:48.396 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:48.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:48.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:48.396 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:00:48.396 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:48.396 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:48.397 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-20T00:00:48.397 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:48.397 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:48.397 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:48.397 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T00:00:48.397 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:48.397 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:48.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:48.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:48.397 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:00:48.397 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:48.397 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:48.398 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:00:48.398 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:48.398 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:48.398 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:48.398 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:00:48.398 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:48.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:48.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:48.398 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:00:48.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:48.398 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:00:48.399 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:00:48.399 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:00:48.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:48.399 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:00:48Z" 2024-04-20T00:00:48.399 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:00:48.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:00:48Z" } 2024-04-20T00:00:48.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:00:48.400 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:48.400 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:48.400 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.682721_308.1 2024-04-20T00:00:53.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:53.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:53.402 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:00:53.460 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:00:53.461 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:00:53.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:00:53.500+0000 7f6e9f7f0780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:00:53.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:00:53.501+0000 7f6e9f7f0780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:00:53.522 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:00:53.523 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:00:53.523 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:00:53.523 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:00:53.523 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:00:53.523 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:00:53.523 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:53.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:53.523 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:00:53.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:53.523 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:00:53.524 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:53.524 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:53.524 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:00:53.524 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:53.524 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:00:53.524 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:53.524 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:00:53.524 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:53.524 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:53.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:53.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:53.525 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:00:53.525 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:53.525 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:53.525 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:00:53.525 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:53.525 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:53.525 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:53.525 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:00:53.525 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:53.525 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:53.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:53.526 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:53.526 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:00:53.526 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:53.526 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:53.526 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.738086_316.1", 2024-04-20T00:00:53.526 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:53.526 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:53.526 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:53.526 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T00:00:53.526 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:53.526 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:53.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:53.527 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:53.527 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:00:53.527 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:53.527 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:53.527 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:00:53.527 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:53.527 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:53.527 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:53.527 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:00:53.527 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:53.528 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:53.528 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:53.528 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:00:53.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:53.528 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:00:53.528 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:00:53.528 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:00:53.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:53.528 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:00:53Z" 2024-04-20T00:00:53.528 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:00:53.529 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.738086_316.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.738086Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:00:53Z" } 2024-04-20T00:00:53.529 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:00:53.530 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:53.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:53.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.738086_316.1 2024-04-20T00:00:58.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:58.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:00:58.532 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:00:58.638 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:00:58.638 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:00:58.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:00:58.725+0000 7fa6b9581780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:00:58.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:00:58.726+0000 7fa6b9581780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:00:58.746 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:00:58.746 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:00:58.746 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:00:58.746 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:00:58.746 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:00:58.746 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:00:58.746 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:58.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:58.747 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:00:58.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:58.747 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:00:58.747 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:58.747 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:58.747 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:00:58.747 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:58.747 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:00:58.747 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:58.747 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:00:58.747 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:58.748 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:58.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:58.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:58.748 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:00:58.748 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:58.748 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:58.748 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:00:58.748 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:58.748 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:58.748 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:58.748 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:00:58.749 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:58.749 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:58.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:58.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:58.749 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:00:58.749 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:58.749 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:58.749 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.632785_307.1", 2024-04-20T00:00:58.749 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:58.749 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:58.749 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:58.749 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T00:00:58.750 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:58.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:58.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:58.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:00:58.750 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:00:58.750 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:00:58.750 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:00:58.750 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:00:58.750 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:00:58.750 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:00:58.750 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:00:58.751 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:00:58.751 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:00:58.751 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:58.751 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:00:58.751 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:00:58.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:58.751 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:00:58.751 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:00:58.751 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:00:58.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:00:58.752 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:00:58Z" 2024-04-20T00:00:58.752 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:00:58.753 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.632785_307.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.632785Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:00:58Z" } 2024-04-20T00:00:58.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:00:58.753 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:58.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:00:58.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.632785_307.1 2024-04-20T00:01:03.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:03.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:03.754 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:01:03.855 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:01:03.856 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:01:03.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:03.920+0000 7f8b0523e780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:03.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:03.922+0000 7f8b0523e780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:03.962 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:01:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:01:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:01:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:01:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:01:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:01:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:01:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:01:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:01:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:01:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:01:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:01:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:01:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:01:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:01:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-20T00:01:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T00:01:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:01:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:01:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:01:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:01:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:01:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:01:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:01:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:01:03Z" 2024-04-20T00:01:03.969 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:01:03.970 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:01:03Z" } 2024-04-20T00:01:03.970 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:01:03.970 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:03.970 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:03.970 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.682721_308.1 2024-04-20T00:01:08.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:08.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:08.971 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:01:09.035 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:01:09.035 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:01:09.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:09.080+0000 7f654f3dd780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:09.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:09.081+0000 7f654f3dd780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:09.119 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:01:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:01:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:01:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:01:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:01:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:01:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:01:09.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:01:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:01:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:01:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:01:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:09.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:01:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:01:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:01:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:09.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:01:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-20T00:01:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T00:01:09.122 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:01:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:01:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:09.123 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:01:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:01:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:01:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:01:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:01:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:09.124 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:01:09Z" 2024-04-20T00:01:09.124 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:01:09.125 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:01:09Z" } 2024-04-20T00:01:09.125 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:01:09.126 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:09.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:09.126 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.629562_306.1 2024-04-20T00:01:14.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:14.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:14.127 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:01:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:01:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:01:14.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:14.230+0000 7f7acefe5780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:14.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:14.232+0000 7f7acefe5780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:14.259 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:01:14.259 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:01:14.259 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:01:14.260 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:01:14.260 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:01:14.260 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:01:14.260 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:14.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:14.260 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:01:14.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:14.260 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:01:14.260 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:14.260 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:14.260 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:01:14.261 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:14.261 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:01:14.261 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:14.261 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:01:14.261 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:14.261 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:14.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:14.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:14.261 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:01:14.261 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:14.261 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:14.261 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:01:14.262 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:14.262 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:14.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:14.262 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:01:14.262 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:14.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:14.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:14.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:14.262 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:01:14.262 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:14.263 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:14.263 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.738086_316.1", 2024-04-20T00:01:14.263 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:14.263 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:14.263 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:14.263 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T00:01:14.263 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:14.263 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:14.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:14.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:14.263 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:01:14.263 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:14.264 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:14.264 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:01:14.264 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:14.264 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:14.264 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:14.264 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:01:14.264 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:14.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:14.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:14.264 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:01:14.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:14.264 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:01:14.265 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:01:14.265 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:01:14.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:14.265 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:01:14Z" 2024-04-20T00:01:14.265 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:01:14.266 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.738086_316.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.738086Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:01:14Z" } 2024-04-20T00:01:14.266 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:01:14.266 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:14.266 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:14.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.738086_316.1 2024-04-20T00:01:19.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:19.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:19.268 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:01:19.327 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:01:19.328 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:01:19.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:19.366+0000 7f00467bb780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:19.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:19.367+0000 7f00467bb780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:19.391 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:01:19.391 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:01:19.391 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:01:19.391 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:01:19.391 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:01:19.391 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:01:19.391 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:19.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:19.391 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:01:19.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:19.392 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:01:19.392 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:19.392 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:19.392 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:01:19.392 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:19.392 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:01:19.392 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:19.392 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:01:19.392 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:19.392 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:19.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:19.393 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:19.393 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:01:19.393 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:19.393 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:19.393 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:01:19.393 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:19.393 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:19.393 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:19.393 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:01:19.393 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:19.393 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:19.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:19.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:19.394 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:01:19.394 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:19.394 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:19.394 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-20T00:01:19.394 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:19.394 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:19.394 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:19.394 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T00:01:19.395 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:19.395 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:19.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:19.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:19.395 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:01:19.395 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:19.395 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:19.395 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:01:19.395 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:19.395 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:19.395 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:19.396 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:01:19.396 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:19.396 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:19.396 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:19.396 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:01:19.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:19.396 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:01:19.396 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:01:19.396 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:01:19.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:19.396 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:01:19Z" 2024-04-20T00:01:19.397 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:01:19.397 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:01:19Z" } 2024-04-20T00:01:19.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:01:19.398 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:19.398 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:19.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.629562_306.1 2024-04-20T00:01:24.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:24.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:24.399 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:01:24.459 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:01:24.459 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:01:24.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:24.499+0000 7fd9d5d9b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:24.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:24.501+0000 7fd9d5d9b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:24.520 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:01:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:01:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:01:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:01:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:01:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:01:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:01:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:01:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:01:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:01:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:01:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:01:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:01:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:01:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:01:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-20T00:01:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T00:01:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:01:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:01:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:01:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:01:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:01:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:01:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:01:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:01:24Z" 2024-04-20T00:01:24.526 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:01:24.527 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:01:24Z" } 2024-04-20T00:01:24.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:01:24.527 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:24.527 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:24.528 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T00:01:29.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:29.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:29.529 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:01:29.588 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:01:29.588 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:01:29.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:29.646+0000 7f872ce5a780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:29.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:29.647+0000 7f872ce5a780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:29.674 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:01:29.674 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:01:29.674 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:01:29.674 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:01:29.675 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:01:29.675 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:01:29.675 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:29.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:29.675 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:01:29.675 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:29.675 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:01:29.676 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:29.676 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:29.676 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:01:29.676 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:29.676 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:01:29.676 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:29.676 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:01:29.677 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:29.677 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:29.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:29.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:29.677 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:01:29.677 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:29.677 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:29.677 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:01:29.677 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:29.678 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:29.678 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:29.678 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:01:29.678 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:29.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:29.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:29.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:29.678 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:01:29.678 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:29.679 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:29.679 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.793769_324.1", 2024-04-20T00:01:29.679 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:29.679 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:29.679 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:29.679 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.793769Z", 2024-04-20T00:01:29.679 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:29.680 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:29.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:29.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:29.680 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:01:29.680 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:29.680 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:29.680 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:01:29.681 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:29.681 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:29.681 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:29.681 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:01:29.681 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:29.682 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:29.682 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:29.682 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:01:29.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:29.682 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:01:29.682 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:01:29.682 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:01:29.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:29.683 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:01:29Z" 2024-04-20T00:01:29.683 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:01:29.685 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.793769_324.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.793769Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:01:29Z" } 2024-04-20T00:01:29.685 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.793769_324.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.793769Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:01:29.685 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:29.685 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.793769_324.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:29.685 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.793769_324.1 2024-04-20T00:01:34.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:34.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:34.687 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:01:34.746 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:01:34.746 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:01:34.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:34.794+0000 7f574a63b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:34.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:34.795+0000 7f574a63b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:34.818 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:01:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:01:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:01:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:01:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:01:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:01:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:01:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:01:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:01:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:01:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:01:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:01:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:01:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:01:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:01:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-20T00:01:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T00:01:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:01:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:01:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:01:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:01:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:01:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:01:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:01:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:01:34Z" 2024-04-20T00:01:34.824 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:01:34.825 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:01:34Z" } 2024-04-20T00:01:34.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:01:34.825 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:34.825 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:34.826 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.629562_306.1 2024-04-20T00:01:39.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:39.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:39.827 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:01:39.887 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:01:39.887 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:01:39.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:39.947+0000 7fe94d7dc780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:39.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:39.949+0000 7fe94d7dc780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:39.977 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:01:39.978 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:01:39.978 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:01:39.978 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:01:39.978 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:01:39.978 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:01:39.978 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:39.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:39.978 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:01:39.978 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:39.978 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:01:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:01:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:01:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:01:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:01:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:01:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:01:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:39.981 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:01:39.981 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:39.981 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:39.981 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.698420_310.1", 2024-04-20T00:01:39.981 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:39.981 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:39.981 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:39.981 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-20T00:01:39.981 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:39.981 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:39.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:39.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:39.982 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:01:39.982 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:39.982 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:39.982 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:01:39.982 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:39.982 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:39.982 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:39.982 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:01:39.982 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:39.982 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:39.982 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:39.983 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:01:39.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:39.983 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:01:39.983 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:01:39.983 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:01:39.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:39.983 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:01:39Z" 2024-04-20T00:01:39.983 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:01:39.984 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.698420_310.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.698420Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:01:39Z" } 2024-04-20T00:01:39.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.698420_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.698420Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:01:39.984 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:39.984 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.698420_310.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:39.984 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.698420_310.1 2024-04-20T00:01:44.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:44.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:44.986 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:01:45.073 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:01:45.073 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:01:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:45.169+0000 7ff97cecd780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:45.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:45.170+0000 7ff97cecd780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:45.189 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:01:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:01:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:01:45.189 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:01:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:01:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:01:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:01:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:01:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:01:45.190 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:01:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:01:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:01:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:01:45.191 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:01:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:01:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:45.192 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.629562_306.1", 2024-04-20T00:01:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T00:01:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:01:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:45.193 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:01:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:01:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:01:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:01:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:01:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:01:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:45.195 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:01:45Z" 2024-04-20T00:01:45.195 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:01:45.195 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.629562_306.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.629562Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:01:45Z" } 2024-04-20T00:01:45.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:01:45.196 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:45.196 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:45.196 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.629562_306.1 2024-04-20T00:01:50.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:50.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:50.198 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:01:50.254 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:01:50.255 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:01:50.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:50.309+0000 7fd927cb2780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:50.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:50.310+0000 7fd927cb2780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:50.336 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:01:50.336 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:01:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:01:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:01:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:01:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:01:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:01:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:01:50.337 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:01:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:01:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:01:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:01:50.338 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:01:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:01:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:01:50.339 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.682721_308.1", 2024-04-20T00:01:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T00:01:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:50.340 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:01:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:01:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:01:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:01:50.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:01:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:01:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:01:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:01:50Z" 2024-04-20T00:01:50.342 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:01:50.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.682721_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.682721Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:01:50Z" } 2024-04-20T00:01:50.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:01:50.343 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:50.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:50.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.682721_308.1 2024-04-20T00:01:55.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:55.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:01:55.345 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:01:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:01:55.405 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:01:55.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:55.445+0000 7fd3e20b9780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:55.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:01:55.446+0000 7fd3e20b9780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:01:55.466 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:01:55.466 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:01:55.466 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:01:55.466 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:01:55.466 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:01:55.466 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:01:55.466 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:55.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:55.466 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:01:55.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:55.467 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:01:55.467 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:55.467 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:55.467 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:01:55.467 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:55.467 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:01:55.467 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:55.467 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:01:55.467 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:55.468 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:55.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:55.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:55.468 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:01:55.468 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:55.468 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:55.468 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:01:55.468 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:55.468 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:55.468 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:55.468 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:01:55.468 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:55.469 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:55.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:55.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:55.469 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:01:55.469 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:55.469 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:55.469 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.738086_316.1", 2024-04-20T00:01:55.469 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:55.469 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:55.469 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:55.469 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T00:01:55.469 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:55.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:55.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:55.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:01:55.470 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:01:55.470 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:01:55.470 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:01:55.470 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:01:55.470 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:01:55.471 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:01:55.471 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:01:55.471 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:01:55.471 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:01:55.471 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:55.471 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:01:55.471 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:01:55.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:55.471 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:01:55.472 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:01:55.472 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:01:55.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:01:55.472 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:01:55Z" 2024-04-20T00:01:55.472 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:01:55.473 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.738086_316.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.738086Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:01:55Z" } 2024-04-20T00:01:55.473 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:01:55.474 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:55.474 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:01:55.474 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.738086_316.1 2024-04-20T00:02:00.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:00.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:00.475 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:02:00.534 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:02:00.534 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:02:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:00.590+0000 7f1665e40780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:00.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:00.591+0000 7f1665e40780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:00.616 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:02:00.616 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:02:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:02:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:02:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:02:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:02:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:02:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:02:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:00.617 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:02:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:02:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:02:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:02:00.618 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:02:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:02:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:00.619 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:02:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.701491_311.1", 2024-04-20T00:02:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.701491Z", 2024-04-20T00:02:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:00.620 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:02:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:02:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:02:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:00.621 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:02:00.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:00.622 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:02:00.622 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:02:00.622 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:02:00.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:00.622 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:02:00Z" 2024-04-20T00:02:00.622 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:02:00.623 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.701491_311.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.701491Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:02:00Z" } 2024-04-20T00:02:00.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.701491_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.701491Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:02:00.623 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:00.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.701491_311.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:00.623 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.701491_311.1 2024-04-20T00:02:05.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:05.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:05.625 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:02:05.683 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:02:05.684 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:02:05.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:05.740+0000 7f0c1ab45780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:05.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:05.742+0000 7f0c1ab45780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:05.771 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:02:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:02:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:02:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:02:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:02:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:02:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:02:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:02:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:02:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:02:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:02:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:02:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:02:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:02:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:02:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.632785_307.1", 2024-04-20T00:02:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T00:02:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:02:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:02:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:02:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:02:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:02:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:02:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:02:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:02:05Z" 2024-04-20T00:02:05.780 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:02:05.781 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.632785_307.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.632785Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:02:05Z" } 2024-04-20T00:02:05.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:02:05.782 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:05.782 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:05.782 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.632785_307.1 2024-04-20T00:02:10.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:10.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:10.784 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:02:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:02:10.844 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:02:10.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:10.902+0000 7fe1363e6780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:10.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:10.903+0000 7fe1363e6780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:10.924 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:02:10.924 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:02:10.924 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:02:10.925 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:02:10.925 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:02:10.925 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:02:10.925 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:10.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:10.925 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:02:10.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:10.925 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:02:10.926 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:10.926 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:10.926 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:02:10.926 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:10.926 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:02:10.926 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:10.926 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:02:10.927 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:10.927 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:10.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:10.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:10.927 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:02:10.927 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:10.927 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:10.927 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:02:10.928 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:10.928 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:10.928 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:10.928 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:02:10.928 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:10.928 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:10.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:10.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:10.928 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:02:10.928 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:10.929 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:10.929 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.614143_305.1", 2024-04-20T00:02:10.929 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:10.929 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:10.929 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:10.929 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T00:02:10.929 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:10.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:10.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:10.930 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:10.930 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:02:10.930 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:10.930 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:10.930 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:02:10.930 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:10.930 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:10.931 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:10.931 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:02:10.931 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:10.931 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:10.931 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:10.931 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:02:10.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:10.931 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:02:10.932 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:02:10.932 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:02:10.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:10.932 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:02:10Z" 2024-04-20T00:02:10.932 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:02:10.933 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.614143_305.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.614143Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:02:10Z" } 2024-04-20T00:02:10.933 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:02:10.933 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:10.933 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:10.934 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.614143_305.1 2024-04-20T00:02:15.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:15.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:15.935 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:02:15.996 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:02:15.996 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:02:16.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:16.161+0000 7f08d49a3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:16.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:16.165+0000 7f08d49a3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:16.195 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:02:16.195 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:02:16.196 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:02:16.196 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:02:16.196 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:02:16.196 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:02:16.196 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:16.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:16.196 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:02:16.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:16.196 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:02:16.197 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:16.197 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:16.197 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:02:16.197 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:16.197 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:02:16.197 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:16.197 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:02:16.197 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:16.197 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:16.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:16.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:16.198 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:02:16.198 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:16.198 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:16.198 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:02:16.198 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:16.198 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:16.198 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:16.199 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:02:16.199 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:16.199 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:16.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:16.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:16.199 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:02:16.199 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:16.199 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:16.199 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.698420_310.1", 2024-04-20T00:02:16.200 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:16.200 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:16.200 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:16.200 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-20T00:02:16.200 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:16.200 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:16.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:16.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:16.200 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:02:16.200 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:16.201 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:16.201 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:02:16.201 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:16.201 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:16.201 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:16.201 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:02:16.201 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:16.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:16.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:16.201 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:02:16.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:16.202 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:02:16.202 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:02:16.202 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:02:16.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:16.202 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:02:16Z" 2024-04-20T00:02:16.202 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:02:16.203 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.698420_310.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.698420Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:02:16Z" } 2024-04-20T00:02:16.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.698420_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.698420Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:02:16.203 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:16.204 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.698420_310.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:16.204 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.698420_310.1 2024-04-20T00:02:21.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:21.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:21.206 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:02:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:02:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:02:21.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:21.310+0000 7f9e8c6b9780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:21.311+0000 7f9e8c6b9780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:21.331 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:02:21.331 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:02:21.332 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:02:21.332 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:02:21.332 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:02:21.332 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:02:21.332 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:21.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:21.332 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:02:21.332 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:21.332 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:02:21.332 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:21.332 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:21.333 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:02:21.333 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:21.333 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:02:21.333 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:21.333 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:02:21.333 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:21.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:21.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:21.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:21.333 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:02:21.334 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:21.334 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:21.334 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:02:21.334 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:21.334 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:21.334 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:21.334 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:02:21.334 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:21.334 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:21.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:21.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:21.335 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:02:21.335 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:21.335 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:21.335 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.973078_341.1", 2024-04-20T00:02:21.335 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:21.335 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:21.335 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:21.335 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.973078Z", 2024-04-20T00:02:21.335 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:21.335 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:21.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:21.336 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:21.336 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:02:21.336 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:21.336 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:21.336 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:02:21.336 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:21.336 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:21.336 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:21.336 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:02:21.336 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:21.336 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:21.337 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:21.337 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:02:21.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:21.337 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:02:21.337 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:02:21.337 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:02:21.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:21.337 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:02:21Z" 2024-04-20T00:02:21.337 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:02:21.338 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.973078_341.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.973078Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:02:21Z" } 2024-04-20T00:02:21.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.973078_341.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.973078Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:02:21.338 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:21.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.973078_341.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:21.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.973078_341.1 2024-04-20T00:02:26.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:26.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:26.340 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:02:26.400 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:02:26.400 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:02:26.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:26.453+0000 7f9e20b6f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:26.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:26.454+0000 7f9e20b6f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:26.478 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:02:26.478 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:02:26.478 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:02:26.479 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:02:26.479 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:02:26.479 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:02:26.479 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:26.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:26.479 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:02:26.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:26.479 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:02:26.480 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:26.480 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:26.480 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:02:26.480 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:26.480 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:02:26.480 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:26.480 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:02:26.480 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:26.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:26.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:26.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:26.481 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:02:26.481 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:26.481 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:26.481 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:02:26.481 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:26.481 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:26.481 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:26.482 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:02:26.482 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:26.482 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:26.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:26.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:26.482 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:02:26.482 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:26.482 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:26.482 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-20T00:02:26.482 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:26.483 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:26.483 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:26.483 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T00:02:26.483 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:26.483 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:26.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:26.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:26.483 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:02:26.483 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:26.484 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:26.484 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:02:26.484 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:26.484 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:26.484 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:26.484 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:02:26.484 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:26.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:26.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:26.484 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:02:26.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:26.485 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:02:26.485 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:02:26.485 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:02:26.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:26.485 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:02:26Z" 2024-04-20T00:02:26.485 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:02:26.486 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:02:26Z" } 2024-04-20T00:02:26.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:02:26.486 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:26.487 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:26.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T00:02:31.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:31.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:31.489 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:02:31.548 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:02:31.548 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:02:31.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:31.591+0000 7fef5a1b3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:31.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:31.592+0000 7fef5a1b3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:31.615 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:02:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:02:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:02:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:02:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:02:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:02:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:31.615 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:02:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:02:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:02:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:02:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:02:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:31.616 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:02:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:02:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:31.617 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:02:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:02:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.907433_331.1", 2024-04-20T00:02:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:31.618 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.907433Z", 2024-04-20T00:02:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:02:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:02:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:02:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:02:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:02:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:02:31.620 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:02:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:31.621 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:02:31Z" 2024-04-20T00:02:31.621 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:02:31.622 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.907433_331.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.907433Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:02:31Z" } 2024-04-20T00:02:31.622 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.907433_331.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.907433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:02:31.622 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:31.622 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.907433_331.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:31.622 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.907433_331.1 2024-04-20T00:02:36.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:36.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:36.624 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:02:36.683 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:02:36.683 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:02:36.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:36.735+0000 7fd737fd8780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:36.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:36.736+0000 7fd737fd8780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:36.758 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:02:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:02:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:02:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:02:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:02:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:02:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:36.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:02:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:02:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:02:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:02:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:02:36.759 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:02:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:02:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:36.760 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:02:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:02:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.987431_342.1", 2024-04-20T00:02:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:36.761 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.987431Z", 2024-04-20T00:02:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:02:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:36.762 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:02:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:02:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:02:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:02:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:02:36.763 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:02:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:36.764 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:02:36Z" 2024-04-20T00:02:36.764 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:02:36.765 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.987431_342.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.987431Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:02:36Z" } 2024-04-20T00:02:36.765 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.987431_342.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.987431Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:02:36.765 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:36.765 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.987431_342.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:36.765 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.987431_342.1 2024-04-20T00:02:41.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:41.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:41.767 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:02:41.840 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:02:41.841 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:02:41.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:41.884+0000 7f6d01543780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:41.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:41.887+0000 7f6d01543780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:41.907 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:02:41.908 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:02:41.908 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:02:41.908 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:02:41.908 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:02:41.908 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:02:41.908 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:41.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:41.908 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:02:41.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:41.908 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:02:41.908 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:41.909 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:41.909 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:02:41.909 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:41.909 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:02:41.909 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:41.909 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:02:41.909 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:41.909 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:41.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:41.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:41.909 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:02:41.910 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:41.910 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:41.910 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:02:41.910 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:41.910 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:41.910 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:41.910 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:02:41.910 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:41.910 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:41.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:41.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:41.911 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:02:41.911 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:41.911 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:41.911 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.868452_328.1", 2024-04-20T00:02:41.911 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:41.911 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:41.911 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:41.911 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T00:02:41.911 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:41.911 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:41.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:41.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:41.912 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:02:41.912 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:41.912 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:41.912 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:02:41.912 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:41.912 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:41.912 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:41.912 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:02:41.912 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:41.913 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:41.913 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:41.913 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:02:41.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:41.913 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:02:41.913 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:02:41.913 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:02:41.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:41.914 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:02:41Z" 2024-04-20T00:02:41.914 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:02:41.915 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.868452_328.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.868452Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:02:41Z" } 2024-04-20T00:02:41.915 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.868452_328.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.868452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:02:41.916 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:41.916 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.868452_328.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:41.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.868452_328.1 2024-04-20T00:02:46.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:46.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:46.918 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:02:46.977 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:02:46.978 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:02:47.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:47.016+0000 7fb95287e780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:47.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:47.017+0000 7fb95287e780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:47.038 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:02:47.038 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:02:47.038 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:02:47.038 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:02:47.038 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:02:47.041 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:02:47.042 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:47.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:47.042 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:02:47.042 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:47.042 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:02:47.042 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:47.042 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:47.042 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:02:47.042 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:47.042 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:02:47.042 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:47.043 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:02:47.043 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:47.043 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:47.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:47.043 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:47.043 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:02:47.043 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:47.043 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:47.043 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:02:47.043 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:47.043 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:47.043 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:47.044 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:02:47.044 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:47.044 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:47.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:47.044 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:47.044 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:02:47.044 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:47.044 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:47.044 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.987431_342.1", 2024-04-20T00:02:47.044 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:47.045 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:47.045 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:47.045 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.987431Z", 2024-04-20T00:02:47.045 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:47.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:47.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:47.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:47.045 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:02:47.045 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:47.045 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:47.045 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:02:47.046 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:47.046 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:47.046 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:47.046 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:02:47.046 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:47.046 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:47.046 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:47.046 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:02:47.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:47.046 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:02:47.046 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:02:47.047 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:02:47.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:47.047 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:02:47Z" 2024-04-20T00:02:47.047 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:02:47.048 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.987431_342.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.987431Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:02:47Z" } 2024-04-20T00:02:47.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.987431_342.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.987431Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:02:47.048 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:47.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.987431_342.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:47.048 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.987431_342.1 2024-04-20T00:02:52.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:52.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:52.050 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:02:52.108 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:02:52.108 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:02:52.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:52.147+0000 7f8729515780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:52.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:52.148+0000 7f8729515780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:52.169 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:02:52.169 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:02:52.169 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:02:52.169 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:02:52.169 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:02:52.169 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:02:52.169 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:52.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:52.170 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:02:52.170 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:52.170 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:02:52.170 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:52.170 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:52.170 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:02:52.170 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:52.170 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:02:52.170 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:52.170 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:02:52.171 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:52.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:52.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:52.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:52.171 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:02:52.171 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:52.171 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:52.171 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:02:52.171 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:52.171 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:52.171 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:52.172 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:02:52.172 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:52.172 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:52.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:52.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:52.172 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:02:52.172 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:52.172 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:52.172 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.968897_339.1", 2024-04-20T00:02:52.172 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:52.172 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:52.173 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:52.173 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.968897Z", 2024-04-20T00:02:52.173 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:52.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:52.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:52.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:52.173 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:02:52.173 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:52.173 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:52.173 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:02:52.173 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:52.173 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:52.174 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:52.174 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:02:52.174 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:52.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:52.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:52.174 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:02:52.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:52.174 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:02:52.174 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:02:52.174 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:02:52.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:52.175 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:02:52Z" 2024-04-20T00:02:52.175 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:02:52.175 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.968897_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.968897Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:02:52Z" } 2024-04-20T00:02:52.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.968897_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.968897Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:02:52.176 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:52.176 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.968897_339.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:52.176 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.968897_339.1 2024-04-20T00:02:57.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:57.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:02:57.177 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:02:57.237 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:02:57.237 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:02:57.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:57.287+0000 7f1cf2f02780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:57.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:02:57.288+0000 7f1cf2f02780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:02:57.316 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:02:57.316 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:02:57.316 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:02:57.316 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:02:57.317 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:02:57.317 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:02:57.317 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:57.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:57.317 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:02:57.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:57.317 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:02:57.317 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:57.317 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:57.317 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:02:57.317 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:57.317 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:02:57.318 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:57.318 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:02:57.318 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:57.318 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:57.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:57.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:57.318 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:02:57.318 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:57.318 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:57.318 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:02:57.318 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:57.319 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:57.319 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:57.319 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:02:57.319 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:57.319 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:57.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:57.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:57.319 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:02:57.319 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:57.319 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:57.319 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.868452_328.1", 2024-04-20T00:02:57.319 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:57.320 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:57.320 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:57.320 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T00:02:57.320 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:57.320 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:57.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:57.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:02:57.320 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:02:57.320 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:02:57.320 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:02:57.321 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:02:57.321 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:02:57.321 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:02:57.321 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:02:57.321 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:02:57.321 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:02:57.321 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:57.322 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:02:57.322 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:02:57.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:57.322 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:02:57.322 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:02:57.322 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:02:57.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:02:57.322 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:02:57Z" 2024-04-20T00:02:57.322 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:02:57.323 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.868452_328.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.868452Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:02:57Z" } 2024-04-20T00:02:57.323 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.868452_328.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.868452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:02:57.323 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:57.324 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.868452_328.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:02:57.324 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.868452_328.1 2024-04-20T00:03:02.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:02.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:02.325 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:03:02.384 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:03:02.384 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:03:02.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:02.433+0000 7f61025d0780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:02.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:02.434+0000 7f61025d0780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:02.462 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:03:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:03:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:03:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:03:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:03:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:03:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:03:02.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:03:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:03:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:03:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:03:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:02.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:03:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:03:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:03:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:03:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-20T00:03:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T00:03:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:02.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:03:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:03:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:03:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:02.466 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:03:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:03:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:03:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:03:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:02.467 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:03:02Z" 2024-04-20T00:03:02.467 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:03:02.468 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:03:02Z" } 2024-04-20T00:03:02.468 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:03:02.468 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:02.469 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:02.469 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T00:03:07.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:07.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:07.470 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:03:07.545 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:03:07.545 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:03:07.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:07.679+0000 7f2a8f94d780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:07.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:07.680+0000 7f2a8f94d780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:07.715 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:03:07.715 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:03:07.715 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:03:07.716 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:03:07.716 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:03:07.716 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:03:07.716 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:07.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:07.716 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:03:07.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:07.716 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:03:07.716 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:07.716 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:07.716 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:03:07.716 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:07.717 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:03:07.717 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:07.717 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:03:07.717 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:07.717 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:07.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:07.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:07.717 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:03:07.717 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:07.717 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:07.717 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:03:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:03:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:03:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-20T00:03:07.719 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:07.719 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:07.719 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:07.719 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T00:03:07.719 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:07.719 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:07.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:07.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:07.719 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:03:07.719 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:07.719 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:07.719 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:03:07.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:07.721 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:03:07Z" 2024-04-20T00:03:07.721 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:03:07.721 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:03:07Z" } 2024-04-20T00:03:07.722 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:03:07.722 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:07.722 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:07.722 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T00:03:12.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:12.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:12.724 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:03:12.782 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:03:12.782 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:03:12.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:12.833+0000 7f54499ef780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:12.834+0000 7f54499ef780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:12.863 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:03:12.863 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:03:12.863 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:03:12.863 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:03:12.863 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:03:12.863 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:03:12.863 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:12.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:12.863 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:03:12.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:12.864 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:03:12.864 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:12.864 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:12.864 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:03:12.864 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:12.864 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:03:12.864 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:12.864 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:03:12.864 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:12.864 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:03:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:03:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:03:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:12.865 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:03:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.868452_328.1", 2024-04-20T00:03:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T00:03:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:12.866 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:12.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:12.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:12.867 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:03:12.867 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:12.867 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:12.867 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:03:12.867 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:12.867 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:12.867 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:12.867 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:03:12.867 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:12.867 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:12.868 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:12.868 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:03:12.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:12.868 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:03:12.868 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:03:12.868 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:03:12.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:12.868 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:03:12Z" 2024-04-20T00:03:12.868 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:03:12.869 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.868452_328.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.868452Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:03:12Z" } 2024-04-20T00:03:12.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.868452_328.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.868452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:03:12.869 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:12.869 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.868452_328.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:12.869 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.868452_328.1 2024-04-20T00:03:17.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:17.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:17.871 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:03:17.931 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:03:17.931 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:03:17.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:17.977+0000 7fe94e2ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:17.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:17.978+0000 7fe94e2ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:18.007 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:03:18.007 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:03:18.007 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:03:18.007 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:03:18.007 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:03:18.007 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:03:18.007 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:18.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:18.007 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:03:18.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:18.008 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:03:18.008 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:18.008 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:18.008 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:03:18.008 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:18.008 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:03:18.008 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:18.008 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:03:18.008 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:18.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:18.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:18.009 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:18.009 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:03:18.009 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:18.009 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:18.009 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:03:18.009 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:18.009 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:18.009 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:18.009 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:03:18.009 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:18.010 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:18.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:18.010 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:18.010 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:03:18.010 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:18.010 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:18.010 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.907433_331.1", 2024-04-20T00:03:18.010 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:18.010 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:18.010 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:18.011 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.907433Z", 2024-04-20T00:03:18.011 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:18.011 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:18.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:18.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:18.011 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:03:18.011 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:18.011 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:18.011 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:03:18.011 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:18.011 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:18.012 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:18.012 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:03:18.012 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:18.012 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:18.012 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:18.012 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:03:18.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:18.012 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:03:18.012 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:03:18.012 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:03:18.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:18.013 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:03:17Z" 2024-04-20T00:03:18.013 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:03:18.013 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.907433_331.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.907433Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:03:17Z" } 2024-04-20T00:03:18.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.907433_331.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.907433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:03:18.014 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:18.014 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.907433_331.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:18.014 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.907433_331.1 2024-04-20T00:03:23.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:23.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:23.016 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:03:23.075 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:03:23.075 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:03:23.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:23.129+0000 7f6671dda780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:23.130+0000 7f6671dda780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:23.153 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:03:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:03:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:03:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:03:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:03:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:03:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:03:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:03:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:03:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:03:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:03:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:03:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:03:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:03:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:03:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.811395_327.1", 2024-04-20T00:03:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T00:03:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:03:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:03:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:03:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:03:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:03:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:03:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:03:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:03:23Z" 2024-04-20T00:03:23.160 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:03:23.160 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.811395_327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.811395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:03:23Z" } 2024-04-20T00:03:23.161 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.811395_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.811395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:03:23.161 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:23.161 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.811395_327.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:23.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.811395_327.1 2024-04-20T00:03:28.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:28.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:28.163 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:03:28.230 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:03:28.230 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:03:28.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:28.387+0000 7f1cfc42c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:28.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:28.388+0000 7f1cfc42c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:28.418 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:03:28.418 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:03:28.418 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:03:28.418 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:03:28.419 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:03:28.419 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:03:28.419 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:28.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:28.419 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:03:28.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:28.419 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:03:28.419 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:28.419 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:28.419 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:03:28.419 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:28.420 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:03:28.420 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:28.420 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:03:28.420 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:28.420 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:28.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:28.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:28.420 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:03:28.420 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:28.420 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:28.420 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:03:28.421 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:28.421 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:28.421 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:28.421 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:03:28.421 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:28.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:28.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:28.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:28.421 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:03:28.421 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:28.421 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:28.421 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.907433_331.1", 2024-04-20T00:03:28.422 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:28.422 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:28.422 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:28.422 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.907433Z", 2024-04-20T00:03:28.422 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:28.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:28.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:28.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:28.422 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:03:28.422 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:28.422 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:28.423 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:03:28.423 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:28.423 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:28.423 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:28.423 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:03:28.423 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:28.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:28.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:28.423 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:03:28.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:28.423 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:03:28.423 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:03:28.424 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:03:28.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:28.424 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:03:28Z" 2024-04-20T00:03:28.424 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:03:28.425 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.907433_331.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.907433Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:03:28Z" } 2024-04-20T00:03:28.425 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.907433_331.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.907433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:03:28.425 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:28.425 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.907433_331.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:28.425 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.907433_331.1 2024-04-20T00:03:33.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:33.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:33.427 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:03:33.485 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:03:33.485 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:03:33.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:33.530+0000 7fbf9ba21780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:33.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:33.531+0000 7fbf9ba21780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:33.553 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:03:33.553 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:03:33.553 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:03:33.553 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:03:33.553 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:03:33.553 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:03:33.554 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:33.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:33.554 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:03:33.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:33.554 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:03:33.554 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:33.554 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:33.554 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:03:33.554 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:33.554 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:03:33.555 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:33.555 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:03:33.555 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:33.555 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:33.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:33.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:33.555 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:03:33.555 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:33.555 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:33.555 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:03:33.555 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:33.555 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:33.556 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:33.556 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:03:33.556 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:33.556 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:33.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:33.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:33.556 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:03:33.556 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:33.556 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:33.556 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.907433_331.1", 2024-04-20T00:03:33.556 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:33.556 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.907433Z", 2024-04-20T00:03:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:03:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:03:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:33.557 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:33.558 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:33.558 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:03:33.558 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:33.558 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:33.558 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:33.558 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:03:33.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:33.558 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:03:33.558 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:03:33.558 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:03:33.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:33.559 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:03:33Z" 2024-04-20T00:03:33.559 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:03:33.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.907433_331.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.907433Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:03:33Z" } 2024-04-20T00:03:33.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.907433_331.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.907433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:03:33.560 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:33.560 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.907433_331.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:33.560 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.907433_331.1 2024-04-20T00:03:38.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:38.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:38.562 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:03:38.620 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:03:38.620 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:03:38.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:38.672+0000 7f0064f67780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:38.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:38.673+0000 7f0064f67780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:38.694 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:03:38.694 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:03:38.694 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:03:38.694 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:03:38.694 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:03:38.695 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:03:38.695 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:38.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:38.695 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:03:38.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:38.695 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:03:38.695 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:38.695 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:38.695 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:03:38.695 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:38.695 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:03:38.696 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:38.696 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:03:38.696 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:38.696 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:38.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:38.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:38.696 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:03:38.696 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:38.696 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:38.696 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:03:38.696 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:38.696 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:38.697 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:38.697 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:03:38.697 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:38.697 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:38.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:38.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:38.697 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:03:38.697 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:38.697 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:38.697 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570245.107533_348.1", 2024-04-20T00:03:38.697 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:38.698 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:38.698 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:38.698 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:05.107533Z", 2024-04-20T00:03:38.698 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:38.698 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:38.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:38.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:38.698 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:03:38.698 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:38.698 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:38.698 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:03:38.699 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:38.699 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:38.699 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:38.699 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:03:38.699 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:38.699 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:38.699 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:38.699 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:03:38.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:38.699 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:03:38.700 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:03:38.700 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:03:38.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:38.700 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:03:38Z" 2024-04-20T00:03:38.700 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:03:38.701 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570245.107533_348.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:05.107533Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:03:38Z" } 2024-04-20T00:03:38.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570245.107533_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:05.107533Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:03:38.701 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:38.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.107533_348.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:38.701 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570245.107533_348.1 2024-04-20T00:03:43.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:43.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:43.703 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:03:43.762 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:03:43.762 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:03:43.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:43.816+0000 7f9f3923e780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:43.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:43.817+0000 7f9f3923e780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:43.848 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:03:43.848 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:03:43.848 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:03:43.848 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:03:43.848 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:03:43.848 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:03:43.849 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:43.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:43.849 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:03:43.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:43.849 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:03:43.849 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:43.849 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:43.849 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:03:43.849 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:43.849 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:03:43.849 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:03:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:03:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:03:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:43.850 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:43.851 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:03:43.851 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:43.851 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:43.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:43.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:43.851 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:03:43.851 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:43.851 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:43.851 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.808187_326.1", 2024-04-20T00:03:43.851 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:43.851 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:43.851 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:43.852 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T00:03:43.852 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:43.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:43.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:43.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:43.852 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:03:43.852 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:43.852 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:43.852 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:03:43.852 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:43.852 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:43.852 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:43.853 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:03:43.853 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:43.853 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:43.853 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:43.853 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:03:43.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:43.853 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:03:43.853 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:03:43.853 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:03:43.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:43.853 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:03:43Z" 2024-04-20T00:03:43.853 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:03:43.854 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.808187_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.808187Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:03:43Z" } 2024-04-20T00:03:43.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.808187_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.808187Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:03:43.855 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:43.855 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.808187_326.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:43.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.808187_326.1 2024-04-20T00:03:48.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:48.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:48.857 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:03:48.960 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:03:48.960 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:03:49.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:49.025+0000 7fb866639780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:49.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:49.026+0000 7fb866639780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:49.053 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:03:49.053 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:03:49.053 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:03:49.053 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:03:49.053 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:03:49.054 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:03:49.054 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:49.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:49.054 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:03:49.054 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:49.054 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:03:49.054 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:49.054 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:49.054 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:03:49.054 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:49.054 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:03:49.055 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:49.055 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:03:49.055 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:49.055 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:49.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:49.055 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:49.055 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:03:49.055 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:49.055 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:49.055 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:03:49.055 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:49.056 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:49.056 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:49.056 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:03:49.056 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:49.056 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:49.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:49.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:49.056 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:03:49.056 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:49.056 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:49.056 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.970082_340.1", 2024-04-20T00:03:49.057 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:49.057 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:49.057 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:49.057 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.970082Z", 2024-04-20T00:03:49.057 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:49.057 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:49.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:49.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:49.057 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:03:49.057 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:49.057 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:49.057 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:03:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:03:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:03:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:03:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:03:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:03:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:03:49Z" 2024-04-20T00:03:49.059 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:03:49.060 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.970082_340.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.970082Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:03:49Z" } 2024-04-20T00:03:49.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.970082_340.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.970082Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:03:49.060 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:49.060 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.970082_340.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:49.061 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.970082_340.1 2024-04-20T00:03:54.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:54.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:54.062 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:03:54.122 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:03:54.122 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:03:54.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:54.162+0000 7f6acc971780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:54.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:54.163+0000 7f6acc971780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:54.185 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:03:54.185 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:03:54.185 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:03:54.185 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:03:54.185 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:03:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:03:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:03:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:03:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:03:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:03:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:03:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:03:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:03:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:03:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:54.189 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:03:54.189 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:54.189 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:54.189 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-20T00:03:54.189 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:54.189 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:54.189 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:54.189 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T00:03:54.189 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:54.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:54.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:54.190 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:54.190 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:03:54.190 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:54.190 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:54.190 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:03:54.190 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:54.190 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:54.190 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:54.190 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:03:54.190 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:54.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:54.191 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:54.191 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:03:54.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:54.191 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:03:54.191 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:03:54.191 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:03:54.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:54.191 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:03:54Z" 2024-04-20T00:03:54.191 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:03:54.192 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:03:54Z" } 2024-04-20T00:03:54.192 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:03:54.193 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:54.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:54.193 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T00:03:59.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:59.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:03:59.194 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:03:59.252 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:03:59.252 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:03:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:59.291+0000 7f689f865780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:59.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:03:59.292+0000 7f689f865780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:03:59.318 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:03:59.318 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:03:59.319 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:03:59.319 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:03:59.319 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:03:59.319 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:03:59.319 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:59.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:59.319 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:03:59.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:59.319 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:03:59.319 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:59.320 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:59.320 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:03:59.320 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:59.320 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:03:59.320 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:59.320 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:03:59.320 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:59.320 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:59.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:59.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:59.320 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:03:59.320 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:59.321 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:59.321 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:03:59.321 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:59.321 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:59.321 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:59.321 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:03:59.321 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:59.321 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:59.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:59.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:59.321 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:03:59.321 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:59.322 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:59.322 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570245.087989_346.1", 2024-04-20T00:03:59.322 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:59.322 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:59.322 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:59.322 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:05.087989Z", 2024-04-20T00:03:59.322 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:59.322 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:59.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:59.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:03:59.323 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:03:59.323 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:03:59.323 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:03:59.323 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:03:59.323 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:03:59.323 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:03:59.323 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:03:59.323 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:03:59.323 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:03:59.323 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:59.323 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:03:59.323 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:03:59.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:59.324 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:03:59.324 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:03:59.324 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:03:59.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:03:59.324 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:03:59Z" 2024-04-20T00:03:59.324 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:03:59.325 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570245.087989_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:05.087989Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:03:59Z" } 2024-04-20T00:03:59.325 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570245.087989_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:05.087989Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:03:59.325 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:59.326 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.087989_346.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:03:59.326 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570245.087989_346.1 2024-04-20T00:04:04.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:04.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:04.327 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:04:04.386 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:04:04.387 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:04:04.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:04.438+0000 7f3e1685f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:04.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:04.439+0000 7f3e1685f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:04.460 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:04:04.460 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:04:04.460 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:04:04.460 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:04:04.460 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:04:04.460 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:04:04.461 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:04.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:04.461 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:04:04.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:04.461 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:04:04.461 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:04.461 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:04.461 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:04:04.461 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:04.461 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:04:04.462 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:04.462 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:04:04.462 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:04.462 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:04.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:04.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:04.462 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:04:04.462 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:04.462 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:04.462 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:04:04.462 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:04.463 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:04.463 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:04.463 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:04:04.463 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:04.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:04.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:04.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:04.463 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:04:04.463 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:04.463 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:04.464 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.877554_329.1", 2024-04-20T00:04:04.464 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:04.464 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:04.464 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:04.464 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.877554Z", 2024-04-20T00:04:04.464 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:04.464 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:04.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:04.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:04.464 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:04:04.465 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:04.465 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:04.465 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:04:04.465 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:04.465 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:04.465 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:04.465 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:04:04.465 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:04.465 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:04.465 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:04.465 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:04:04.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:04.466 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:04:04.466 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:04:04.466 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:04:04.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:04.466 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:04:04Z" 2024-04-20T00:04:04.466 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:04:04.467 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.877554_329.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.877554Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:04:04Z" } 2024-04-20T00:04:04.467 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.877554_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.877554Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:04:04.467 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:04.468 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.877554_329.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:04.468 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.877554_329.1 2024-04-20T00:04:09.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:09.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:09.469 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:04:09.536 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:04:09.536 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:04:09.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:09.592+0000 7f5e1be85780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:09.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:09.593+0000 7f5e1be85780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:09.619 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:04:09.619 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:04:09.619 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:04:09.620 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:04:09.620 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:04:09.620 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:04:09.620 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:09.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:09.620 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:04:09.620 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:09.620 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:04:09.620 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:09.620 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:09.620 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:04:09.621 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:09.621 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:04:09.621 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:09.621 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:04:09.621 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:09.621 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:09.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:09.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:09.621 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:04:09.621 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:09.621 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:09.622 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:04:09.622 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:09.622 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:09.622 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:09.622 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:04:09.622 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:09.622 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:09.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:09.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:09.622 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:04:09.623 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:09.623 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:09.623 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.811395_327.1", 2024-04-20T00:04:09.623 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:09.623 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:09.623 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:09.623 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T00:04:09.623 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:09.623 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:09.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:09.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:09.624 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:04:09.624 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:09.624 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:09.624 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:04:09.624 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:09.624 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:09.624 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:09.624 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:04:09.624 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:09.624 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:09.624 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:09.625 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:04:09.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:09.625 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:04:09.625 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:04:09.625 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:04:09.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:09.625 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:04:09Z" 2024-04-20T00:04:09.625 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:04:09.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.811395_327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.811395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:04:09Z" } 2024-04-20T00:04:09.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.811395_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.811395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:04:09.627 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:09.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.811395_327.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:09.627 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.811395_327.1 2024-04-20T00:04:14.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:14.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:14.629 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:04:14.691 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:04:14.691 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:04:14.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:14.730+0000 7fe5c642b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:14.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:14.731+0000 7fe5c642b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:14.752 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:04:14.752 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:04:14.752 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:04:14.752 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:04:14.752 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:04:14.752 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:04:14.752 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:14.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:14.753 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:04:14.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:14.753 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:04:14.753 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:14.753 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:14.753 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:04:14.753 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:14.753 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:04:14.753 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:14.753 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:04:14.753 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:14.753 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:14.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:14.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:14.754 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:04:14.754 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:14.754 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:14.754 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:04:14.754 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:14.754 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:14.754 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:14.754 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:04:14.754 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:14.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:14.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:14.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:14.755 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:04:14.755 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:14.755 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:14.755 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.936115_334.1", 2024-04-20T00:04:14.755 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:14.755 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:14.755 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:14.755 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.936115Z", 2024-04-20T00:04:14.756 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:14.756 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:14.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:14.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:14.756 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:04:14.756 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:14.756 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:14.756 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:04:14.756 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:14.756 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:14.756 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:14.756 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:04:14.757 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:14.757 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:14.757 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:14.757 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:04:14.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:14.757 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:04:14.757 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:04:14.757 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:04:14.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:14.757 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:04:14Z" 2024-04-20T00:04:14.757 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:04:14.758 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.936115_334.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.936115Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:04:14Z" } 2024-04-20T00:04:14.759 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.936115_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.936115Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:04:14.759 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:14.759 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.936115_334.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:14.759 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.936115_334.1 2024-04-20T00:04:19.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:19.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:19.760 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:04:19.818 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:04:19.818 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:04:19.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:19.858+0000 7fc240e36780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:19.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:19.859+0000 7fc240e36780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:19.879 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:04:19.879 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:04:19.879 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:04:19.879 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:04:19.880 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:04:19.880 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:04:19.880 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:19.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:19.880 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:04:19.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:19.880 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:04:19.880 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:19.880 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:19.880 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:04:19.880 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:19.880 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:04:19.881 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:19.881 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:04:19.881 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:19.881 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:19.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:19.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:19.881 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:04:19.881 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:19.881 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:19.881 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:04:19.881 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:19.881 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:19.882 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:19.882 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:04:19.882 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:19.882 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:19.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:19.882 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:19.882 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:04:19.882 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:19.882 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:19.882 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-20T00:04:19.882 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:19.883 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:19.883 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:19.883 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T00:04:19.883 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:19.883 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:19.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:19.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:19.883 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:04:19.883 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:19.883 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:19.883 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:04:19.883 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:19.884 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:19.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:19.884 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:04:19.884 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:19.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:19.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:19.884 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:04:19.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:19.884 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:04:19.884 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:04:19.884 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:04:19.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:19.885 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:04:19Z" 2024-04-20T00:04:19.885 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:04:19.886 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:04:19Z" } 2024-04-20T00:04:19.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:04:19.886 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:19.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:19.886 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T00:04:24.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:24.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:24.888 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:04:24.947 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:04:24.947 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:04:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:25.000+0000 7ff740778780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:25.002+0000 7ff740778780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:25.023 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:04:25.023 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:04:25.023 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:04:25.023 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:04:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:04:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:04:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:04:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:04:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:04:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:04:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:04:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:04:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:04:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:04:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:04:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.868452_328.1", 2024-04-20T00:04:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T00:04:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:04:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:04:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:04:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:04:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:04:25.029 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:04:25.029 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:04:25.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:25.029 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:04:25Z" 2024-04-20T00:04:25.029 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:04:25.030 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.868452_328.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.868452Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:04:25Z" } 2024-04-20T00:04:25.030 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.868452_328.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.868452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:04:25.030 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:25.030 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.868452_328.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:25.030 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.868452_328.1 2024-04-20T00:04:30.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:30.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:30.032 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:04:30.087 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:04:30.087 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:04:30.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:30.136+0000 7fc795779780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:30.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:30.137+0000 7fc795779780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:30.156 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:04:30.156 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:04:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:04:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:04:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:04:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:04:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:04:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:04:30.157 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:04:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:04:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:04:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:30.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:04:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:04:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:04:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:30.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:04:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570245.114997_351.1", 2024-04-20T00:04:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:05.114997Z", 2024-04-20T00:04:30.160 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:04:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:04:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:04:30.161 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:04:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:04:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:04:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:04:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:30.162 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:04:30Z" 2024-04-20T00:04:30.162 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:04:30.163 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570245.114997_351.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:05.114997Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:04:30Z" } 2024-04-20T00:04:30.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570245.114997_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:05.114997Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:04:30.164 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:30.164 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.114997_351.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:30.164 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570245.114997_351.1 2024-04-20T00:04:35.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:35.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:35.165 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:04:35.226 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:04:35.226 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:04:35.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:35.277+0000 7f2585b1b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:35.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:35.279+0000 7f2585b1b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:35.300 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:04:35.301 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:04:35.301 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:04:35.301 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:04:35.301 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:04:35.301 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:04:35.301 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:35.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:35.301 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:04:35.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:35.301 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:04:35.302 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:35.302 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:35.302 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:04:35.302 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:35.302 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:04:35.302 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:35.302 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:04:35.302 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:35.302 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:35.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:35.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:35.303 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:04:35.303 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:35.303 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:35.303 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:04:35.303 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:35.303 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:35.303 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:35.303 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:04:35.303 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:35.303 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:35.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:35.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:35.304 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:04:35.304 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:35.304 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:35.304 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-20T00:04:35.304 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:35.304 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:35.305 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:35.305 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T00:04:35.305 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:35.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:35.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:35.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:35.306 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:04:35.306 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:35.306 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:35.306 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:04:35.307 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:35.307 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:35.307 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:35.307 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:04:35.307 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:35.308 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:35.308 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:35.308 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:04:35.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:35.308 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:04:35.308 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:04:35.309 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:04:35.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:35.309 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:04:35Z" 2024-04-20T00:04:35.309 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:04:35.310 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:04:35Z" } 2024-04-20T00:04:35.311 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:04:35.311 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:35.311 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:35.311 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T00:04:40.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:40.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:40.313 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:04:40.373 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:04:40.373 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:04:40.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:40.420+0000 7ff5a313b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:40.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:40.421+0000 7ff5a313b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:40.442 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:04:40.442 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:04:40.442 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:04:40.442 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:04:40.442 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:04:40.443 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:04:40.443 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:40.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:40.443 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:04:40.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:40.443 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:04:40.443 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:40.443 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:40.443 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:04:40.443 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:40.444 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:04:40.444 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:40.444 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:04:40.444 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:40.444 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:40.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:40.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:40.444 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:04:40.444 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:40.444 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:40.444 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:04:40.445 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:40.445 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:40.445 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:40.445 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:04:40.445 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:40.445 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:40.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:40.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:40.445 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:04:40.445 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:40.446 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:40.446 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.808187_326.1", 2024-04-20T00:04:40.446 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:40.446 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:40.446 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:40.446 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T00:04:40.446 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:40.446 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:40.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:40.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:40.447 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:04:40.447 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:40.447 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:40.447 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:04:40.447 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:40.447 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:40.447 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:40.447 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:04:40.447 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:40.447 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:40.447 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:40.448 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:04:40.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:40.448 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:04:40.448 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:04:40.448 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:04:40.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:40.448 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:04:40Z" 2024-04-20T00:04:40.448 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:04:40.449 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.808187_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.808187Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:04:40Z" } 2024-04-20T00:04:40.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.808187_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.808187Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:04:40.449 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:40.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.808187_326.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:40.450 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.808187_326.1 2024-04-20T00:04:45.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:45.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:45.451 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:04:45.527 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:04:45.527 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:04:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:45.647+0000 7fb19d54a780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:45.648+0000 7fb19d54a780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:45.675 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:04:45.676 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:04:45.676 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:04:45.676 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:04:45.676 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:04:45.676 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:04:45.676 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:45.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:45.676 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:04:45.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:45.677 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:04:45.677 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:45.677 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:45.677 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:04:45.677 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:45.677 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:04:45.677 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:45.677 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:04:45.677 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:45.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:45.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:45.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:45.678 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:04:45.678 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:45.678 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:45.678 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:04:45.678 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:45.678 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:45.678 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:45.679 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:04:45.679 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:45.679 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:45.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:45.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:45.679 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:04:45.679 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:45.679 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:45.679 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.914458_333.1", 2024-04-20T00:04:45.679 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:45.680 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:45.680 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:45.680 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.914458Z", 2024-04-20T00:04:45.680 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:45.680 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:45.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:45.680 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:45.680 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:04:45.680 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:45.680 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:45.681 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:04:45.681 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:45.681 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:45.681 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:45.681 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:04:45.681 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:45.681 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:45.681 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:45.681 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:04:45.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:45.682 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:04:45.682 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:04:45.682 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:04:45.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:45.682 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:04:45Z" 2024-04-20T00:04:45.682 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:04:45.683 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.914458_333.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.914458Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:04:45Z" } 2024-04-20T00:04:45.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.914458_333.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.914458Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:04:45.683 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:45.684 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.914458_333.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:45.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.914458_333.1 2024-04-20T00:04:50.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:50.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:50.685 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:04:50.745 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:04:50.745 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:04:50.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:50.797+0000 7fdb93f6c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:50.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:50.799+0000 7fdb93f6c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:50.823 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:04:50.823 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:04:50.823 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:04:50.823 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:04:50.823 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:04:50.824 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:04:50.824 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:50.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:50.824 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:04:50.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:50.824 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:04:50.824 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:50.824 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:50.824 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:04:50.824 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:50.824 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:04:50.824 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:04:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:04:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:04:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:04:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:04:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:50.827 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:50.827 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.811395_327.1", 2024-04-20T00:04:50.827 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:50.827 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:50.827 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:50.827 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T00:04:50.827 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:50.827 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:50.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:50.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:50.827 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:04:50.827 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:50.828 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:50.828 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:04:50.828 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:50.828 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:50.828 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:50.828 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:04:50.828 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:50.828 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:50.828 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:50.828 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:04:50.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:50.829 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:04:50.829 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:04:50.829 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:04:50.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:50.829 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:04:50Z" 2024-04-20T00:04:50.829 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:04:50.830 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.811395_327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.811395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:04:50Z" } 2024-04-20T00:04:50.830 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.811395_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.811395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:04:50.830 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:50.830 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.811395_327.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:50.830 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.811395_327.1 2024-04-20T00:04:55.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:55.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:04:55.834 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:04:55.893 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:04:55.893 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:04:55.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:55.933+0000 7f21b8fc0780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:55.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:04:55.934+0000 7f21b8fc0780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:04:55.954 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:04:55.954 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:04:55.955 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:04:55.955 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:04:55.955 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:04:55.955 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:04:55.955 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:55.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:55.955 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:04:55.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:55.955 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:04:55.955 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:55.955 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:55.956 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:04:55.956 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:55.956 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:04:55.956 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:55.956 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:04:55.956 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:55.956 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:55.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:55.956 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:55.956 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:04:55.956 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:55.956 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:55.957 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:04:55.957 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:55.957 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:55.957 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:55.957 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:04:55.957 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:55.957 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:55.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:55.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:55.957 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:04:55.957 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:55.958 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:55.958 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-20T00:04:55.958 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:55.958 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:55.958 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:55.958 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T00:04:55.958 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:55.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:55.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:55.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:04:55.958 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:04:55.959 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:04:55.959 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:04:55.959 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:04:55.959 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:04:55.959 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:04:55.959 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:04:55.959 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:04:55.959 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:04:55.959 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:55.959 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:04:55.959 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:04:55.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:55.960 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:04:55.960 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:04:55.960 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:04:55.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:04:55.960 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:04:55Z" 2024-04-20T00:04:55.960 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:04:55.961 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:04:55Z" } 2024-04-20T00:04:55.961 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:04:55.961 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:55.961 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:04:55.961 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T00:05:00.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:00.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:00.963 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:01.022 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:01.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:01.075+0000 7febce631780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:01.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:01.077+0000 7febce631780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:01.099 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:05:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:05:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:05:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:05:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:05:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:05:01.099 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:05:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:05:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:05:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:01.100 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:05:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:05:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:05:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:01.101 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:05:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:05:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.909976_332.1", 2024-04-20T00:05:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:01.102 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.909976Z", 2024-04-20T00:05:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:05:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:01.103 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:05:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:05:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:05:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:05:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:05:01.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:01.105 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:05:01Z" 2024-04-20T00:05:01.105 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:05:01.105 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.909976_332.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.909976Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:05:01Z" } 2024-04-20T00:05:01.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.909976_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.909976Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:05:01.106 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:01.106 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.909976_332.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:01.106 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.909976_332.1 2024-04-20T00:05:06.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:06.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:06.107 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:06.163 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:06.163 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:06.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:06.217+0000 7f22535f3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:06.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:06.218+0000 7f22535f3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:06.238 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:05:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:05:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:05:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:05:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:05:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:05:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:05:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:05:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:05:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:05:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:05:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:05:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:05:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:05:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.877554_329.1", 2024-04-20T00:05:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.877554Z", 2024-04-20T00:05:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:05:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:05:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:05:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:05:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:05:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:05:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:05:06Z" 2024-04-20T00:05:06.245 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:05:06.246 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.877554_329.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.877554Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:05:06Z" } 2024-04-20T00:05:06.246 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.877554_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.877554Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:05:06.246 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:06.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.877554_329.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:06.246 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.877554_329.1 2024-04-20T00:05:11.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:11.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:11.248 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:11.306 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:11.307 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:11.349+0000 7f001c795780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:11.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:11.351+0000 7f001c795780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:11.372 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:05:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:05:11.372 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:05:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:05:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:05:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:05:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:05:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:05:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:11.373 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:11.374 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:11.374 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:11.374 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:05:11.374 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:11.374 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:05:11.374 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:11.374 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:11.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:11.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:11.374 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:05:11.375 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:11.375 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:11.375 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:05:11.375 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:11.375 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:11.375 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:11.375 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:05:11.375 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:11.375 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:11.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:11.375 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:11.376 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:05:11.376 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:11.376 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:11.376 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.808187_326.1", 2024-04-20T00:05:11.376 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:11.376 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:11.376 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:11.376 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T00:05:11.376 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:11.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:05:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:05:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:11.378 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:11.378 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:11.378 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:05:11.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:11.378 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:05:11.378 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:05:11.378 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:05:11.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:11.378 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:05:11Z" 2024-04-20T00:05:11.378 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:05:11.379 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.808187_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.808187Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:05:11Z" } 2024-04-20T00:05:11.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.808187_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.808187Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:05:11.380 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:11.380 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.808187_326.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:11.380 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.808187_326.1 2024-04-20T00:05:16.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:16.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:16.381 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:16.440 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:16.440 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:16.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:16.483+0000 7f367c78e780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:16.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:16.485+0000 7f367c78e780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:16.508 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:05:16.508 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:05:16.508 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:05:16.509 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:05:16.509 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:05:16.509 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:05:16.509 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:16.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:16.509 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:05:16.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:16.510 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:05:16.510 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:16.510 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:16.510 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:16.511 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:16.511 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:05:16.511 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:16.511 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:05:16.511 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:16.511 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:16.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:16.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:16.512 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:05:16.512 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:16.512 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:16.512 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:05:16.512 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:16.512 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:16.512 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:16.513 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:05:16.513 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:16.513 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:16.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:16.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:16.513 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:05:16.513 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:16.513 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:16.514 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.942889_337.1", 2024-04-20T00:05:16.514 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:16.514 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:16.514 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:16.514 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.942889Z", 2024-04-20T00:05:16.514 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:16.514 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:16.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:16.515 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:16.515 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:05:16.515 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:16.515 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:16.515 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:16.515 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:16.516 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:16.516 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:16.516 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:05:16.516 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:16.516 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:16.516 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:16.516 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:05:16.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:16.517 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:05:16.517 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:05:16.517 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:05:16.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:16.517 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:05:16Z" 2024-04-20T00:05:16.517 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:05:16.520 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.942889_337.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.942889Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:05:16Z" } 2024-04-20T00:05:16.520 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.942889_337.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.942889Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:05:16.521 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:16.521 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.942889_337.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:16.521 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.942889_337.1 2024-04-20T00:05:21.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:21.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:21.523 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:21.582 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:21.582 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:21.620+0000 7f81dc95c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:21.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:21.622+0000 7f81dc95c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:21.643 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:05:21.643 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:05:21.643 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:05:21.643 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:05:21.644 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:05:21.644 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:05:21.644 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:21.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:21.644 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:05:21.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:21.644 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:05:21.644 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:21.644 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:21.644 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:21.644 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:21.645 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:05:21.645 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:21.645 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:05:21.645 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:21.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:21.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:21.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:21.645 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:05:21.645 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:21.645 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:21.645 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:05:21.645 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:05:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:05:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.811395_327.1", 2024-04-20T00:05:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:21.647 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:21.647 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:21.647 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T00:05:21.647 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:21.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:21.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:21.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:21.647 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:05:21.647 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:21.647 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:21.647 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:05:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:05:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:05:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:05:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:05:21.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:21.649 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:05:21Z" 2024-04-20T00:05:21.649 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:05:21.650 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.811395_327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.811395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:05:21Z" } 2024-04-20T00:05:21.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.811395_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.811395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:05:21.650 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:21.650 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.811395_327.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:21.650 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.811395_327.1 2024-04-20T00:05:26.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:26.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:26.652 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:26.708 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:26.708 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:26.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:26.765+0000 7ff339dda780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:26.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:26.766+0000 7ff339dda780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:26.788 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:05:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:05:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:05:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:05:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:05:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:05:26.788 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:05:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:05:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:26.789 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:05:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:05:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:05:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:26.790 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:05:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:05:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:05:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:26.791 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-20T00:05:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T00:05:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:05:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:26.792 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:05:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:05:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:26.793 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:05:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:05:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:05:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:26.794 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:05:26Z" 2024-04-20T00:05:26.794 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:05:26.795 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:05:26Z" } 2024-04-20T00:05:26.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:05:26.795 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:26.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:26.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T00:05:31.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:31.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:31.797 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:31.855 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:31.855 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:31.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:31.904+0000 7f2ce65af780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:31.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:31.906+0000 7f2ce65af780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:31.936 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:05:31.936 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:05:31.936 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:05:31.936 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:05:31.936 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:05:31.936 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:05:31.936 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:31.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:31.936 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:05:31.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:31.937 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:05:31.937 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:31.937 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:31.937 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:31.937 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:31.937 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:05:31.937 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:31.937 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:05:31.937 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:31.937 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:31.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:31.938 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:31.938 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:05:31.938 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:31.938 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:31.938 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:05:31.938 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:31.938 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:31.938 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:31.938 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:05:31.938 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:31.938 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:31.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:31.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:31.939 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:05:31.939 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:31.939 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:31.939 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.877554_329.1", 2024-04-20T00:05:31.939 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:31.939 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:31.939 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:31.939 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.877554Z", 2024-04-20T00:05:31.939 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:31.940 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:31.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:31.940 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:31.940 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:05:31.940 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:31.940 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:31.940 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:31.940 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:31.940 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:31.940 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:31.940 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:05:31.940 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:31.941 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:31.941 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:31.941 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:05:31.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:31.941 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:05:31.941 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:05:31.941 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:05:31.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:31.941 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:05:31Z" 2024-04-20T00:05:31.941 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:05:31.942 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.877554_329.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.877554Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:05:31Z" } 2024-04-20T00:05:31.942 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.877554_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.877554Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:05:31.942 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:31.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.877554_329.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:31.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.877554_329.1 2024-04-20T00:05:36.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:36.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:36.944 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:37.003 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:37.003 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:37.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:37.056+0000 7fe6ff346780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:37.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:37.058+0000 7fe6ff346780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:37.080 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:05:37.080 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:05:37.080 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:05:37.080 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:05:37.080 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:05:37.080 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:05:37.080 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:37.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:37.081 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:05:37.081 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:37.081 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:05:37.081 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:37.081 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:37.081 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:37.081 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:37.081 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:05:37.081 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:37.082 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:05:37.082 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:37.082 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:37.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:37.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:37.082 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:05:37.082 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:37.082 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:37.082 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:05:37.082 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:37.083 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:37.083 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:37.083 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:05:37.083 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:37.083 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:37.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:37.083 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:37.083 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:05:37.083 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:37.083 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:37.083 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.968897_339.1", 2024-04-20T00:05:37.084 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:37.084 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:37.084 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:37.084 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.968897Z", 2024-04-20T00:05:37.084 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:37.084 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:37.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:37.084 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:37.084 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:05:37.084 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:37.084 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:05:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:05:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:05:37.085 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:05:37.086 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:05:37.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:37.086 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:05:37Z" 2024-04-20T00:05:37.086 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:05:37.087 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.968897_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.968897Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:05:37Z" } 2024-04-20T00:05:37.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.968897_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.968897Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:05:37.087 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:37.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.968897_339.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:37.087 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.968897_339.1 2024-04-20T00:05:42.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2024-04-20T00:05:42.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:42.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:42.095 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:42.155 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:42.155 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:42.192 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:05:42.208 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:05:42.208 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:05:42.208 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:42.208 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:05:42.208 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:05:42.208 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:05:42.208 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:05:42.208 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:05:42.208 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:05:42.208 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:05:42.208 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:05:42.209 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:00:32.655231Z" 2024-04-20T00:05:42.209 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:05:42.209 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:05:42.209 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:42.209 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:05:42.209 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:05:42.209 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:05:42.210 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:42.210 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:05:42.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:42.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:42.211 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:42.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:42.364+0000 7f876fbe0780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:42.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:42.365+0000 7f876fbe0780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:42.385 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:05:42.385 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:05:42.385 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:05:42.386 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:05:42.386 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:05:42.386 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:05:42.386 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:42.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:42.386 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:05:42.386 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:42.386 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:05:42.386 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:42.386 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:42.387 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:42.387 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:42.387 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:05:42.387 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:42.387 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:05:42.387 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:42.387 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:42.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:42.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:42.387 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:05:42.388 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:42.388 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:42.388 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:05:42.388 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:42.388 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:42.388 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:42.388 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:05:42.388 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:42.388 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:42.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:42.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:42.389 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:05:42.389 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:42.389 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:42.389 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.909976_332.1", 2024-04-20T00:05:42.389 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:42.389 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:42.389 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:42.389 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.909976Z", 2024-04-20T00:05:42.389 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:42.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:42.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:42.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:42.390 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:05:42.390 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:42.390 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:42.390 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:42.390 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:42.390 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:42.390 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:42.390 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:05:42.390 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:42.390 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:42.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:42.391 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:05:42.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:42.391 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:05:42.391 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:05:42.391 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:05:42.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:42.391 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:05:42Z" 2024-04-20T00:05:42.391 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:05:42.392 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.909976_332.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.909976Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:05:42Z" } 2024-04-20T00:05:42.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.909976_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.909976Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:05:42.392 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:42.393 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.909976_332.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:42.393 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.909976_332.1 2024-04-20T00:05:47.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:47.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:47.394 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:47.453 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:47.453 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:47.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:47.492+0000 7fad0611d780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:47.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:47.493+0000 7fad0611d780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:47.513 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:05:47.513 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:05:47.513 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:05:47.514 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:05:47.514 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:05:47.514 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:05:47.514 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:47.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:47.514 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:05:47.514 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:47.514 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:05:47.514 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:47.514 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:47.514 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:47.515 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:47.515 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:05:47.515 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:47.515 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:05:47.515 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:47.515 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:47.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:47.515 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:47.515 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:05:47.515 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:47.515 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:47.515 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:05:47.516 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:47.516 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:47.516 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:47.516 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:05:47.516 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:47.516 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:47.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:47.516 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:47.516 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:05:47.516 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:47.516 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:47.517 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.868452_328.1", 2024-04-20T00:05:47.517 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:47.517 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:47.517 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:47.517 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T00:05:47.517 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:47.517 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:47.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:47.517 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:47.517 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:05:47.517 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:47.518 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:47.518 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:47.518 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:47.518 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:47.518 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:47.518 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:05:47.518 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:47.518 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:47.518 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:47.518 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:05:47.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:47.518 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:05:47.519 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:05:47.519 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:05:47.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:47.519 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:05:47Z" 2024-04-20T00:05:47.519 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:05:47.520 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.868452_328.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.868452Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:05:47Z" } 2024-04-20T00:05:47.520 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.868452_328.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.868452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:05:47.520 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:47.520 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.868452_328.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:47.520 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.868452_328.1 2024-04-20T00:05:52.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:52.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:52.521 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:52.581 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:52.581 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:52.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:52.640+0000 7f05c3ec9780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:52.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:52.641+0000 7f05c3ec9780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:52.664 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:05:52.665 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:05:52.665 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:05:52.665 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:05:52.665 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:05:52.665 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:05:52.665 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:52.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:52.665 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:05:52.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:52.665 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:05:52.666 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:52.666 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:52.666 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:52.666 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:52.666 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:05:52.666 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:52.666 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:05:52.666 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:52.666 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:52.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:52.668 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:05:52.668 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:52.668 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:52.668 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.811395_327.1", 2024-04-20T00:05:52.668 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:52.668 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:52.668 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:52.668 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T00:05:52.668 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:52.668 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:52.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:52.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:52.669 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:05:52.669 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:52.669 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:52.669 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:52.669 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:52.669 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:52.669 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:52.669 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:05:52.669 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:52.669 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:52.669 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:52.669 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:05:52.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:52.670 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:05:52.670 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:05:52.670 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:05:52.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:52.670 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:05:52Z" 2024-04-20T00:05:52.670 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:05:52.671 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.811395_327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.811395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:05:52Z" } 2024-04-20T00:05:52.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.811395_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.811395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:05:52.671 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:52.671 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.811395_327.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:52.672 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.811395_327.1 2024-04-20T00:05:57.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:57.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:05:57.673 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:05:57.733 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:05:57.733 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:05:57.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:57.778+0000 7efe06873780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:57.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:05:57.780+0000 7efe06873780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:05:57.802 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:05:57.802 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:05:57.802 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:05:57.802 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:05:57.802 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:05:57.802 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:05:57.802 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:57.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:57.802 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:05:57.802 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:57.803 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:05:57.803 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:57.803 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:57.803 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:05:57.803 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:57.803 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:05:57.803 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:57.803 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:05:57.803 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:57.803 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:57.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:57.803 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:57.804 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:05:57.804 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:57.804 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:57.804 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:05:57.804 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:57.804 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:57.804 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:57.804 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:05:57.804 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:57.804 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:57.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:57.805 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:57.805 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:05:57.805 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:57.805 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:57.805 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.868452_328.1", 2024-04-20T00:05:57.805 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:57.805 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:57.805 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:57.805 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T00:05:57.805 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:57.805 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:05:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:05:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:05:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:05:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:05:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:05:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:05:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:05:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:05:57.806 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:05:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:05:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:05:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:05:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:05:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:05:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:05:57.807 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:05:57Z" 2024-04-20T00:05:57.807 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:05:57.808 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.868452_328.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.868452Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:05:57Z" } 2024-04-20T00:05:57.808 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.868452_328.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.868452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:05:57.809 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:57.809 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.868452_328.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:05:57.809 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.868452_328.1 2024-04-20T00:06:02.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:02.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:02.811 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:06:02.870 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:02.870 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:03.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:02.998+0000 7fe818820780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:03.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:03.001+0000 7fe818820780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:03.046 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:06:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:06:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:06:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:06:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:06:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:06:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:06:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:06:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:06:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:06:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:06:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:06:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:06:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:06:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.811395_327.1", 2024-04-20T00:06:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T00:06:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:06:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:06:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:06:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:06:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:06:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:06:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:06:03Z" 2024-04-20T00:06:03.052 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:06:03.053 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.811395_327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.811395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:06:03Z" } 2024-04-20T00:06:03.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.811395_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.811395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:06:03.053 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:03.053 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.811395_327.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:03.054 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.811395_327.1 2024-04-20T00:06:08.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:08.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:08.055 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:06:08.114 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:08.114 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:08.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:08.160+0000 7fb8cdd62780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:08.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:08.162+0000 7fb8cdd62780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:08.185 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:06:08.185 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:06:08.185 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:06:08.185 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:06:08.185 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:06:08.186 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:08.186 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:08.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:08.186 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:06:08.186 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:08.186 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:06:08.186 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:08.186 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:08.186 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:06:08.186 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:08.186 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:06:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:06:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:06:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:06:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:08.187 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:08.188 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:08.188 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:06:08.188 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:08.188 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:08.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:08.188 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:08.188 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:06:08.188 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:08.188 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:08.188 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.808187_326.1", 2024-04-20T00:06:08.188 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:08.189 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:08.189 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:08.189 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T00:06:08.189 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:08.189 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:08.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:08.189 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:08.189 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:06:08.189 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:08.189 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:08.189 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:06:08.189 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:08.190 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:08.190 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:08.190 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:06:08.190 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:08.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:08.190 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:08.190 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:08.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:08.190 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:06:08.190 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:06:08.191 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:06:08.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:08.191 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:06:08Z" 2024-04-20T00:06:08.191 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:06:08.192 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.808187_326.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.808187Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:06:08Z" } 2024-04-20T00:06:08.192 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.808187_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.808187Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:06:08.192 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:08.192 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.808187_326.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:08.192 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.808187_326.1 2024-04-20T00:06:13.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:13.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:13.194 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:06:13.252 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:13.252 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:13.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:13.310+0000 7f984be42780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:13.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:13.311+0000 7f984be42780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:13.338 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:06:13.338 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:06:13.338 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:06:13.339 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:06:13.339 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:06:13.339 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:13.339 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:13.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:13.339 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:06:13.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:13.339 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:06:13.339 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:13.339 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:13.339 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:06:13.339 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:06:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:06:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:06:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:06:13.340 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:13.341 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:13.341 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:13.341 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:06:13.341 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:13.341 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:13.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:13.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:13.341 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:06:13.341 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:13.341 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:13.341 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.968897_339.1", 2024-04-20T00:06:13.341 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:13.342 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:13.342 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:13.342 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.968897Z", 2024-04-20T00:06:13.342 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:13.342 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:13.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:13.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:13.342 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:06:13.342 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:13.342 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:13.342 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:06:13.342 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:13.343 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:13.343 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:13.343 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:06:13.343 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:13.343 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:13.343 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:13.343 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:13.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:13.343 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:06:13.343 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:06:13.343 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:06:13.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:13.344 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:06:13Z" 2024-04-20T00:06:13.344 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:06:13.344 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.968897_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.968897Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:06:13Z" } 2024-04-20T00:06:13.345 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.968897_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.968897Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:06:13.345 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:13.345 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.968897_339.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:13.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.968897_339.1 2024-04-20T00:06:18.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:18.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:18.346 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:06:18.403 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:18.404 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:18.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:18.464+0000 7ff1d6fc4780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:18.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:18.465+0000 7ff1d6fc4780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:18.489 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:06:18.489 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:06:18.489 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:06:18.489 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:06:18.489 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:06:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:06:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:06:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:06:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:06:18.490 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:18.491 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:06:18.491 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:18.491 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:18.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:18.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:18.491 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:06:18.491 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:18.491 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:18.491 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:06:18.491 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:18.491 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:18.492 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:18.492 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:06:18.492 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:18.492 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:18.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:18.492 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:18.492 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:06:18.492 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:18.492 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:18.492 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.797017_325.1", 2024-04-20T00:06:18.492 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:18.492 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:18.493 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:18.493 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T00:06:18.493 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:18.493 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:18.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:18.493 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:18.493 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:06:18.493 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:18.493 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:18.493 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:06:18.493 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:18.493 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:18.494 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:18.494 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:06:18.494 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:18.494 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:18.494 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:18.494 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:18.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:18.494 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:06:18.494 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:06:18.494 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:06:18.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:18.494 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:06:18Z" 2024-04-20T00:06:18.495 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:06:18.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.797017_325.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.797017Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:06:18Z" } 2024-04-20T00:06:18.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:06:18.496 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:18.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:18.496 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T00:06:23.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:23.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:23.497 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:06:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:23.596+0000 7f8383709780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:23.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:23.597+0000 7f8383709780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:23.690 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:06:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:06:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:06:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:06:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:06:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:06:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:06:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:06:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:06:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:06:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:06:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:06:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:06:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:06:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713570242.942889_337.1", 2024-04-20T00:06:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:44:02.942889Z", 2024-04-20T00:06:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:06:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:06:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:06:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:06:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:06:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:06:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:06:23Z" 2024-04-20T00:06:23.696 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:06:23.697 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713570242.942889_337.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:44:02.942889Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:06:23Z" } 2024-04-20T00:06:23.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.942889_337.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.942889Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:06:23.698 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:23.698 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.942889_337.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:23.698 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.942889_337.1 2024-04-20T00:06:28.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:28.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:28.699 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:06:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:28.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:28.797+0000 7fa954b68780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:28.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:28.798+0000 7fa954b68780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:28.820 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:06:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:06:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:06:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:06:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:06:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:28.820 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:06:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:06:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:06:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:28.821 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:06:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:06:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:28.822 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:06:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:06:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:06:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:28.823 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:06:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:06:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:28.824 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:06:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:06:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:06:28.825 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:06:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:28.826 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:06:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:06:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:06:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:28.827 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:06:28Z" 2024-04-20T00:06:28.827 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:06:28.828 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:06:28Z" } 2024-04-20T00:06:28.828 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:06:28.828 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:28.828 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:28.829 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:06:28.829 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:06:28.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:28.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:28.829 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:06:28.889 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:28.889 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:28.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:06:28.922+0000 7f3f6dcec780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:28.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:06:28.923+0000 7f3f6dcec780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:28.940 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:06:28.940 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:06:28.940 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:06:28.940 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:06:28.940 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:06:28.940 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:28.941 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:06:28.941 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:28.941 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:06:28.941 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:28.941 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:06:28.941 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:06:28.941 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:06:28.941 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:06:28.942 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:06:28.942 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:06:28.942 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:06:28.942 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:06:28.942 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:06:28.942 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:28.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:28.943 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:28.943 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:06:28.943 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:06:28.943 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:06:28.943 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:06:28.943 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:06:28.943 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:06:28.943 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:06:28.943 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:06:28.944 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:06:28.944 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:28.944 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:28.944 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:28.944 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:06:28.944 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:06:28.944 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:06:28.944 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:06:28.944 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:06:28.945 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:06:28.945 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:06:28.945 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:06:28.945 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:06:28.945 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:28.945 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:28.945 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:28.945 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:06:28.946 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:06:28.946 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:06:28.946 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:06:28.946 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:06:28.946 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:06:28.946 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:06:28.946 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:06:28.946 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:06:28.946 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:28.947 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:28.947 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:28.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:28.947 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:06:28.947 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:06:28.947 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:06:28.947 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:28.947 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:06:28Z" 2024-04-20T00:06:28.947 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:06:28.948 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:06:28Z" } 2024-04-20T00:06:28.949 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:06:28.949 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:28.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:28.949 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:06:28.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'TWO98NBPIQJFTC6N', '--secret', 'WVtqgebnqURqV1qdDSut7FusWVqSrgr8', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:28.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'TWO98NBPIQJFTC6N', '--secret', 'WVtqgebnqURqV1qdDSut7FusWVqSrgr8', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:28.950 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key TWO98NBPIQJFTC6N --secret WVtqgebnqURqV1qdDSut7FusWVqSrgr8 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:29.010 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:NOTICE: set zone's realm_id=0542eeed-b7db-454d-a499-a81b016de7e3 2024-04-20T00:06:29.048 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:06:29.048 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "160fb3c3-e977-4293-878c-2367de8cdff6", 2024-04-20T00:06:29.048 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "remove", 2024-04-20T00:06:29.048 INFO:teuthology.orchestra.run.smithi031.stdout: "domain_root": "remove.rgw.meta:root", 2024-04-20T00:06:29.048 INFO:teuthology.orchestra.run.smithi031.stdout: "control_pool": "remove.rgw.control", 2024-04-20T00:06:29.048 INFO:teuthology.orchestra.run.smithi031.stdout: "gc_pool": "remove.rgw.log:gc", 2024-04-20T00:06:29.049 INFO:teuthology.orchestra.run.smithi031.stdout: "lc_pool": "remove.rgw.log:lc", 2024-04-20T00:06:29.049 INFO:teuthology.orchestra.run.smithi031.stdout: "log_pool": "remove.rgw.log", 2024-04-20T00:06:29.049 INFO:teuthology.orchestra.run.smithi031.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-04-20T00:06:29.049 INFO:teuthology.orchestra.run.smithi031.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-04-20T00:06:29.049 INFO:teuthology.orchestra.run.smithi031.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-04-20T00:06:29.049 INFO:teuthology.orchestra.run.smithi031.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-04-20T00:06:29.049 INFO:teuthology.orchestra.run.smithi031.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-04-20T00:06:29.049 INFO:teuthology.orchestra.run.smithi031.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-04-20T00:06:29.049 INFO:teuthology.orchestra.run.smithi031.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-04-20T00:06:29.049 INFO:teuthology.orchestra.run.smithi031.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-04-20T00:06:29.050 INFO:teuthology.orchestra.run.smithi031.stdout: "otp_pool": "remove.rgw.otp", 2024-04-20T00:06:29.050 INFO:teuthology.orchestra.run.smithi031.stdout: "system_key": { 2024-04-20T00:06:29.050 INFO:teuthology.orchestra.run.smithi031.stdout: "access_key": "TWO98NBPIQJFTC6N", 2024-04-20T00:06:29.050 INFO:teuthology.orchestra.run.smithi031.stdout: "secret_key": "WVtqgebnqURqV1qdDSut7FusWVqSrgr8" 2024-04-20T00:06:29.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:29.050 INFO:teuthology.orchestra.run.smithi031.stdout: "placement_pools": [ 2024-04-20T00:06:29.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:29.050 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "default-placement", 2024-04-20T00:06:29.051 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:29.051 INFO:teuthology.orchestra.run.smithi031.stdout: "index_pool": "remove.rgw.buckets.index", 2024-04-20T00:06:29.051 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_classes": { 2024-04-20T00:06:29.051 INFO:teuthology.orchestra.run.smithi031.stdout: "STANDARD": { 2024-04-20T00:06:29.051 INFO:teuthology.orchestra.run.smithi031.stdout: "data_pool": "remove.rgw.buckets.data" 2024-04-20T00:06:29.051 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:29.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:29.051 INFO:teuthology.orchestra.run.smithi031.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-04-20T00:06:29.051 INFO:teuthology.orchestra.run.smithi031.stdout: "index_type": 0, 2024-04-20T00:06:29.052 INFO:teuthology.orchestra.run.smithi031.stdout: "inline_data": true 2024-04-20T00:06:29.052 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:29.052 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:29.052 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-20T00:06:29.052 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:29.052 INFO:teuthology.orchestra.run.smithi031.stdout: "notif_pool": "remove.rgw.log:notif" 2024-04-20T00:06:29.052 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:06:29.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:29.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:29.054 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:29.108 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:29.108 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:29.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:29.120+0000 7f645ea09780 0 period (f494ba41-e721-4411-8938-a297d95f7246 does not have zone 160fb3c3-e977-4293-878c-2367de8cdff6 configured 2024-04-20T00:06:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:Sending period to new master zone 73f46f02-c7ab-4426-a267-2273af911da5 2024-04-20T00:06:35.901 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:06:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 5, 2024-04-20T00:06:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T00:06:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": [], 2024-04-20T00:06:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "period_map": { 2024-04-20T00:06:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "zonegroups": [ 2024-04-20T00:06:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "test-zonegroup", 2024-04-20T00:06:35.902 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zonegroup", 2024-04-20T00:06:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "api_name": "", 2024-04-20T00:06:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "is_master": true, 2024-04-20T00:06:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "endpoints": [ 2024-04-20T00:06:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T00:06:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-20T00:06:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "hostnames": [], 2024-04-20T00:06:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "hostnames_s3website": [], 2024-04-20T00:06:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: "zones": [ 2024-04-20T00:06:35.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T00:06:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zone2", 2024-04-20T00:06:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "endpoints": [ 2024-04-20T00:06:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T00:06:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-20T00:06:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "log_meta": false, 2024-04-20T00:06:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "log_data": true, 2024-04-20T00:06:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "read_only": false, 2024-04-20T00:06:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_type": "", 2024-04-20T00:06:35.904 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from_all": false, 2024-04-20T00:06:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from": [], 2024-04-20T00:06:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_zone": "", 2024-04-20T00:06:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "supported_features": [ 2024-04-20T00:06:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "compress-encrypted", 2024-04-20T00:06:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "resharding" 2024-04-20T00:06:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "160fb3c3-e977-4293-878c-2367de8cdff6", 2024-04-20T00:06:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "remove", 2024-04-20T00:06:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "endpoints": [], 2024-04-20T00:06:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: "log_meta": false, 2024-04-20T00:06:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "log_data": true, 2024-04-20T00:06:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "read_only": false, 2024-04-20T00:06:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_type": "", 2024-04-20T00:06:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from_all": true, 2024-04-20T00:06:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from": [], 2024-04-20T00:06:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_zone": "", 2024-04-20T00:06:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "supported_features": [ 2024-04-20T00:06:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "compress-encrypted", 2024-04-20T00:06:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: "resharding" 2024-04-20T00:06:35.906 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zone1", 2024-04-20T00:06:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "endpoints": [ 2024-04-20T00:06:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T00:06:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-20T00:06:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "log_meta": false, 2024-04-20T00:06:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "log_data": true, 2024-04-20T00:06:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:35.907 INFO:teuthology.orchestra.run.smithi031.stdout: "read_only": false, 2024-04-20T00:06:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_type": "", 2024-04-20T00:06:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from_all": true, 2024-04-20T00:06:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from": [], 2024-04-20T00:06:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_zone": "", 2024-04-20T00:06:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "supported_features": [ 2024-04-20T00:06:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "compress-encrypted", 2024-04-20T00:06:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "resharding" 2024-04-20T00:06:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T00:06:35.908 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test-zone3", 2024-04-20T00:06:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "endpoints": [ 2024-04-20T00:06:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T00:06:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-20T00:06:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "log_meta": false, 2024-04-20T00:06:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "log_data": true, 2024-04-20T00:06:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:35.909 INFO:teuthology.orchestra.run.smithi031.stdout: "read_only": false, 2024-04-20T00:06:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "tier_type": "", 2024-04-20T00:06:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from_all": true, 2024-04-20T00:06:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_from": [], 2024-04-20T00:06:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_zone": "", 2024-04-20T00:06:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: "supported_features": [ 2024-04-20T00:06:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: "compress-encrypted", 2024-04-20T00:06:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: "resharding" 2024-04-20T00:06:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:35.911 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-20T00:06:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "placement_targets": [ 2024-04-20T00:06:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default-placement", 2024-04-20T00:06:35.912 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-20T00:06:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_classes": [ 2024-04-20T00:06:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "STANDARD" 2024-04-20T00:06:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-20T00:06:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "default_placement": "default-placement", 2024-04-20T00:06:35.917 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_policy": { 2024-04-20T00:06:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: "groups": [] 2024-04-20T00:06:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled_features": [ 2024-04-20T00:06:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: "resharding" 2024-04-20T00:06:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-20T00:06:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: "short_zone_ids": [ 2024-04-20T00:06:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:35.918 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T00:06:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 2010863401 2024-04-20T00:06:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "160fb3c3-e977-4293-878c-2367de8cdff6", 2024-04-20T00:06:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 2924758009 2024-04-20T00:06:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 1221149675 2024-04-20T00:06:35.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T00:06:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 1478903101 2024-04-20T00:06:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T00:06:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "period_config": { 2024-04-20T00:06:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_quota": { 2024-04-20T00:06:35.920 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false, 2024-04-20T00:06:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "check_on_raw": false, 2024-04-20T00:06:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max_size": -1, 2024-04-20T00:06:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max_size_kb": 0, 2024-04-20T00:06:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max_objects": -1 2024-04-20T00:06:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "user_quota": { 2024-04-20T00:06:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false, 2024-04-20T00:06:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "check_on_raw": false, 2024-04-20T00:06:35.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max_size": -1, 2024-04-20T00:06:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max_size_kb": 0, 2024-04-20T00:06:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max_objects": -1 2024-04-20T00:06:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "user_ratelimit": { 2024-04-20T00:06:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_ops": 0, 2024-04-20T00:06:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_ops": 0, 2024-04-20T00:06:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_bytes": 0, 2024-04-20T00:06:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_bytes": 0, 2024-04-20T00:06:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false 2024-04-20T00:06:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.922 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket_ratelimit": { 2024-04-20T00:06:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_ops": 0, 2024-04-20T00:06:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_ops": 0, 2024-04-20T00:06:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_bytes": 0, 2024-04-20T00:06:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_bytes": 0, 2024-04-20T00:06:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false 2024-04-20T00:06:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "anonymous_ratelimit": { 2024-04-20T00:06:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_ops": 0, 2024-04-20T00:06:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_ops": 0, 2024-04-20T00:06:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max_read_bytes": 0, 2024-04-20T00:06:35.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max_write_bytes": 0, 2024-04-20T00:06:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": false 2024-04-20T00:06:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_name": "test-realm", 2024-04-20T00:06:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:35.924 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:06:35.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:35.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:35.925 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:35.993 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:35.993 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:36.026 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:06:36.026 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T00:06:36.027 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T00:06:36.027 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T00:06:36.027 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T00:06:36.027 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:36.027 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T00:06:36.027 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.027 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T00:06:36.027 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T00:06:36.027 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:36.027 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T00:06:36.027 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.028 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T00:06:36.028 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T00:06:36.028 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:36.028 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T00:06:36.028 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.028 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:36.028 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T00:06:36.028 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:36.028 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:36.028 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:36.028 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T00:06:36.028 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:36.029 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:36.029 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:36.029 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:36.029 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.029 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.029 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:36.029 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T00:06:36.029 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:36.029 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T00:06:36.029 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.030 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:36.030 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T00:06:36.030 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:36.030 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:36.030 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:36.030 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T00:06:36.030 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:36.030 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:36.030 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:36.030 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:36.030 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.031 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.031 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.031 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T00:06:36.031 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T00:06:36.031 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:36.031 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T00:06:36.031 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.031 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:36.031 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T00:06:36.031 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:36.031 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:36.032 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:36.032 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T00:06:36.032 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:36.032 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:36.032 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:36.032 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:36.032 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.032 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.032 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.032 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.032 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T00:06:36.032 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.033 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T00:06:36.033 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T00:06:36.033 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T00:06:36.033 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T00:06:36.033 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.033 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.033 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.033 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T00:06:36.033 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:36.033 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T00:06:36.033 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [] 2024-04-20T00:06:36.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.034 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T00:06:36.034 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.034 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.034 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:06:36.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:36.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:36.035 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:36.112 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:36.113 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:36.142 INFO:teuthology.orchestra.run.smithi019.stderr:zone name remove not found in zonegroup test-zonegroup 2024-04-20T00:06:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-20T00:06:36.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:36.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:36.156 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:36.211 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:36.211 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:36.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:36.240+0000 7ff858db5780 0 Removed zone from zonegroup test-zonegroup 2024-04-20T00:06:36.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:36.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:36.257 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:36.348 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:36.349 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:36.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:06:36.387+0000 7f81182a4780 0 WARNING: can't generate connection for zone 160fb3c3-e977-4293-878c-2367de8cdff6 id remove: no endpoints defined 2024-04-20T00:06:36.538 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:06:36.538 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:36.538 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 6, 2024-04-20T00:06:36.538 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T00:06:36.538 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T00:06:36.538 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T00:06:36.538 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:36.538 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T00:06:36.538 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.538 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T00:06:36.539 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T00:06:36.539 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T00:06:36.539 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T00:06:36.539 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:36.539 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T00:06:36.539 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.539 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T00:06:36.539 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T00:06:36.539 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:36.539 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T00:06:36.539 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.540 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T00:06:36.540 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T00:06:36.540 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:36.540 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T00:06:36.540 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.540 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:36.540 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T00:06:36.540 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:36.540 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:36.540 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:36.540 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T00:06:36.541 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:36.541 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:36.541 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:36.541 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:36.541 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.541 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.541 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.541 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.541 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:36.541 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T00:06:36.541 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:36.542 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T00:06:36.542 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.542 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:36.542 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T00:06:36.542 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:36.542 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:36.542 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:36.542 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T00:06:36.542 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:36.543 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:36.543 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:36.543 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:36.543 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.543 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.543 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.543 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.543 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T00:06:36.543 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T00:06:36.543 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:36.544 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T00:06:36.544 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.544 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:36.544 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T00:06:36.544 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:36.544 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:36.544 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:36.544 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T00:06:36.544 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:36.544 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:36.544 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:36.544 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T00:06:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T00:06:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T00:06:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T00:06:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T00:06:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.546 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.546 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T00:06:36.546 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:36.546 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T00:06:36.546 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [] 2024-04-20T00:06:36.546 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.546 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T00:06:36.546 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.546 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.546 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.546 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.547 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T00:06:36.547 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.547 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T00:06:36.547 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T00:06:36.547 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.547 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.547 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:36.547 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T00:06:36.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.571 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.571 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T00:06:36.571 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T00:06:36.571 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.571 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.571 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T00:06:36.571 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:36.572 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T00:06:36.572 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T00:06:36.572 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T00:06:36.572 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T00:06:36.572 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T00:06:36.572 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T00:06:36.572 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T00:06:36.572 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.572 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T00:06:36.572 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T00:06:36.572 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T00:06:36.572 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T00:06:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T00:06:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T00:06:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T00:06:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T00:06:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T00:06:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T00:06:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T00:06:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T00:06:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T00:06:36.573 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T00:06:36.574 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T00:06:36.574 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T00:06:36.574 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T00:06:36.574 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T00:06:36.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.574 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T00:06:36.574 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T00:06:36.574 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T00:06:36.574 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T00:06:36.574 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T00:06:36.575 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T00:06:36.575 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.575 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:36.575 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T00:06:36.575 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:06:36.575 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:06:36.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2024-04-20T00:06:36.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:36.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:36.577 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:36.641 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:36.642 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:36.676 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:06:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "9992c60f-a9bd-463f-b974-12df6c614b6a", 2024-04-20T00:06:36.676 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "remove_zg", 2024-04-20T00:06:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "remove_zg", 2024-04-20T00:06:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": false, 2024-04-20T00:06:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [], 2024-04-20T00:06:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T00:06:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T00:06:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "", 2024-04-20T00:06:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [], 2024-04-20T00:06:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T00:06:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.677 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T00:06:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T00:06:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [] 2024-04-20T00:06:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T00:06:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T00:06:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [] 2024-04-20T00:06:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T00:06:36.678 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.679 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.679 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:06:36.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:36.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:36.680 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:36.745 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:36.745 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:36.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:06:36.789+0000 7fa04fb13780 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2024-04-20T00:06:36.799 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:06:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "c2105248-25cc-430a-935c-21c1680bfe4c", 2024-04-20T00:06:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "remove", 2024-04-20T00:06:36.799 INFO:teuthology.orchestra.run.smithi019.stdout: "domain_root": "remove.rgw.meta:root", 2024-04-20T00:06:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "control_pool": "remove.rgw.control", 2024-04-20T00:06:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "gc_pool": "remove.rgw.log:gc", 2024-04-20T00:06:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "lc_pool": "remove.rgw.log:lc", 2024-04-20T00:06:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "log_pool": "remove.rgw.log", 2024-04-20T00:06:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-04-20T00:06:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-04-20T00:06:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-04-20T00:06:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-04-20T00:06:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-04-20T00:06:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-04-20T00:06:36.800 INFO:teuthology.orchestra.run.smithi019.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "otp_pool": "remove.rgw.otp", 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "system_key": { 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "access_key": "", 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "secret_key": "" 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_pools": [ 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "default-placement", 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "index_pool": "remove.rgw.buckets.index", 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": { 2024-04-20T00:06:36.801 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD": { 2024-04-20T00:06:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "data_pool": "remove.rgw.buckets.data" 2024-04-20T00:06:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-04-20T00:06:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "index_type": 0, 2024-04-20T00:06:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "inline_data": true 2024-04-20T00:06:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:36.802 INFO:teuthology.orchestra.run.smithi019.stdout: "notif_pool": "remove.rgw.log:notif" 2024-04-20T00:06:36.803 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:06:36.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:36.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:36.804 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:36.872 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:36.872 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:36.976 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:06:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 7, 2024-04-20T00:06:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T00:06:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T00:06:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T00:06:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:36.976 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T00:06:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "9992c60f-a9bd-463f-b974-12df6c614b6a", 2024-04-20T00:06:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "remove_zg", 2024-04-20T00:06:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "remove_zg", 2024-04-20T00:06:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": false, 2024-04-20T00:06:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [], 2024-04-20T00:06:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T00:06:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T00:06:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "c2105248-25cc-430a-935c-21c1680bfe4c", 2024-04-20T00:06:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T00:06:36.977 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "c2105248-25cc-430a-935c-21c1680bfe4c", 2024-04-20T00:06:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "remove", 2024-04-20T00:06:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [], 2024-04-20T00:06:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": false, 2024-04-20T00:06:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T00:06:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:36.978 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T00:06:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T00:06:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T00:06:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T00:06:36.979 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T00:06:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T00:06:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T00:06:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [] 2024-04-20T00:06:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T00:06:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.980 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T00:06:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T00:06:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T00:06:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T00:06:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T00:06:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T00:06:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T00:06:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:36.981 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T00:06:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T00:06:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T00:06:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T00:06:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T00:06:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:36.982 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T00:06:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T00:06:36.983 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T00:06:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T00:06:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T00:06:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:36.984 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:36.985 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T00:06:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T00:06:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T00:06:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T00:06:37.015 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T00:06:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T00:06:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T00:06:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T00:06:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T00:06:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T00:06:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T00:06:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T00:06:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [] 2024-04-20T00:06:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T00:06:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T00:06:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.018 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T00:06:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T00:06:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T00:06:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T00:06:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T00:06:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "c2105248-25cc-430a-935c-21c1680bfe4c", 2024-04-20T00:06:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2717934044 2024-04-20T00:06:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T00:06:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T00:06:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T00:06:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T00:06:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T00:06:37.020 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T00:06:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T00:06:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T00:06:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T00:06:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T00:06:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T00:06:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T00:06:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T00:06:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T00:06:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T00:06:37.021 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T00:06:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T00:06:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T00:06:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T00:06:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T00:06:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T00:06:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T00:06:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T00:06:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T00:06:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T00:06:37.022 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T00:06:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T00:06:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T00:06:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T00:06:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T00:06:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T00:06:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T00:06:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T00:06:37.023 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:06:37.024 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:06:37.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:37.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:37.025 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:37.117 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:37.117 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:37.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:06:37.186+0000 7fb3b187d780 0 NOTICE: removed master_zone of zonegroup remove_zg 2024-04-20T00:06:37.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:06:37.189+0000 7fb3b187d780 0 Removed zone from zonegroup remove_zg 2024-04-20T00:06:37.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:37.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:37.217 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:37.298 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:37.298 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:37.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:06:37.388+0000 7f922ef17780 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-04-20T00:06:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-20T00:06:37.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:37.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:37.401 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:37.467 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:37.467 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:37.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:37.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:06:37.504 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:06:37.574 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:37.574 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:37.661 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:06:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 8, 2024-04-20T00:06:37.661 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T00:06:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T00:06:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T00:06:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T00:06:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T00:06:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T00:06:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T00:06:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T00:06:37.662 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T00:06:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T00:06:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T00:06:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T00:06:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T00:06:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T00:06:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T00:06:37.663 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T00:06:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T00:06:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:37.664 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T00:06:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T00:06:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:37.665 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T00:06:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T00:06:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:37.666 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T00:06:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T00:06:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T00:06:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T00:06:37.667 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T00:06:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T00:06:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T00:06:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T00:06:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T00:06:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T00:06:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T00:06:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T00:06:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T00:06:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T00:06:37.668 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T00:06:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T00:06:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T00:06:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T00:06:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T00:06:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:37.669 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T00:06:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T00:06:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [] 2024-04-20T00:06:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T00:06:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T00:06:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T00:06:37.670 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T00:06:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T00:06:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T00:06:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:37.671 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T00:06:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:37.681 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T00:06:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T00:06:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:06:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T00:06:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:06:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T00:06:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T00:06:37.682 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T00:06:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T00:06:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T00:06:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T00:06:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T00:06:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T00:06:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T00:06:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T00:06:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T00:06:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T00:06:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T00:06:37.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T00:06:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T00:06:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T00:06:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T00:06:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T00:06:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T00:06:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T00:06:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T00:06:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T00:06:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T00:06:37.684 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T00:06:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T00:06:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T00:06:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T00:06:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T00:06:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T00:06:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T00:06:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T00:06:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T00:06:37.685 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T00:06:37.686 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:06:37.686 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:06:37.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-04-20T00:06:37.687 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-49 2024-04-20T00:06:37.688 DEBUG:boto:path=/aktlji-49/ 2024-04-20T00:06:37.688 DEBUG:boto:auth_path=/aktlji-49/ 2024-04-20T00:06:37.688 DEBUG:boto:Method: PUT 2024-04-20T00:06:37.688 DEBUG:boto:Path: /aktlji-49/ 2024-04-20T00:06:37.688 DEBUG:boto:Data: 2024-04-20T00:06:37.688 DEBUG:boto:Headers: {} 2024-04-20T00:06:37.688 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:06:37.689 DEBUG:boto:Port: 8000 2024-04-20T00:06:37.689 DEBUG:boto:Params: {} 2024-04-20T00:06:37.689 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:06:37.689 DEBUG:boto:Token: None 2024-04-20T00:06:37.690 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:37 GMT /aktlji-49/ 2024-04-20T00:06:37.690 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vnzlwVP2nVxufq+VmAHGLSxza2Y= 2024-04-20T00:06:37.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vnzlwVP2nVxufq+VmAHGLSxza2Y=', 'Content-Length': '0'} 2024-04-20T00:06:37.929 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d6def8e3b5d40acc-006623070d-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:06:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:37.930 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-50 2024-04-20T00:06:37.930 DEBUG:boto:path=/aktlji-50/ 2024-04-20T00:06:37.930 DEBUG:boto:auth_path=/aktlji-50/ 2024-04-20T00:06:37.930 DEBUG:boto:Method: PUT 2024-04-20T00:06:37.930 DEBUG:boto:Path: /aktlji-50/ 2024-04-20T00:06:37.930 DEBUG:boto:Data: 2024-04-20T00:06:37.931 DEBUG:boto:Headers: {} 2024-04-20T00:06:37.931 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:06:37.931 DEBUG:boto:Port: 8000 2024-04-20T00:06:37.931 DEBUG:boto:Params: {} 2024-04-20T00:06:37.931 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:06:37.931 DEBUG:boto:Token: None 2024-04-20T00:06:37.931 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:37 GMT /aktlji-50/ 2024-04-20T00:06:37.932 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/rtLPJOWO3TadTXpr5nogQv3fqE= 2024-04-20T00:06:37.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/rtLPJOWO3TadTXpr5nogQv3fqE=', 'Content-Length': '0'} 2024-04-20T00:06:38.195 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ad5a25383d1d2930-006623070e-1bf8-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:06:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:38.195 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-51 2024-04-20T00:06:38.196 DEBUG:boto:path=/aktlji-51/ 2024-04-20T00:06:38.196 DEBUG:boto:auth_path=/aktlji-51/ 2024-04-20T00:06:38.196 DEBUG:boto:Method: PUT 2024-04-20T00:06:38.196 DEBUG:boto:Path: /aktlji-51/ 2024-04-20T00:06:38.196 DEBUG:boto:Data: 2024-04-20T00:06:38.196 DEBUG:boto:Headers: {} 2024-04-20T00:06:38.196 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:06:38.196 DEBUG:boto:Port: 8001 2024-04-20T00:06:38.196 DEBUG:boto:Params: {} 2024-04-20T00:06:38.196 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T00:06:38.197 DEBUG:boto:Token: None 2024-04-20T00:06:38.197 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:38 GMT /aktlji-51/ 2024-04-20T00:06:38.197 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iRzWQ9asIRIz3hxgrqAYOiVTW/U= 2024-04-20T00:06:38.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iRzWQ9asIRIz3hxgrqAYOiVTW/U=', 'Content-Length': '0'} 2024-04-20T00:06:38.292 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d7143b11861674a9-006623070e-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:06:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:38.292 DEBUG:boto:path=/aktlji-49/ 2024-04-20T00:06:38.292 DEBUG:boto:auth_path=/aktlji-49/ 2024-04-20T00:06:38.292 DEBUG:boto:path=/aktlji-49/?website 2024-04-20T00:06:38.292 DEBUG:boto:auth_path=/aktlji-49/?website 2024-04-20T00:06:38.292 DEBUG:boto:Method: PUT 2024-04-20T00:06:38.293 DEBUG:boto:Path: /aktlji-49/?website 2024-04-20T00:06:38.293 DEBUG:boto:Data: index.htmlerror.html 2024-04-20T00:06:38.293 DEBUG:boto:Headers: {} 2024-04-20T00:06:38.293 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:06:38.293 DEBUG:boto:Port: 8000 2024-04-20T00:06:38.293 DEBUG:boto:Params: {} 2024-04-20T00:06:38.293 DEBUG:boto:Token: None 2024-04-20T00:06:38.293 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:38 GMT /aktlji-49/?website 2024-04-20T00:06:38.293 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IHsW0pqXj+9PRWc+QZ5vwjkIDOI= 2024-04-20T00:06:38.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IHsW0pqXj+9PRWc+QZ5vwjkIDOI=', 'Content-Length': '241'} 2024-04-20T00:06:38.294 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000082be82f2c816433f-006623070e-15b6-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 00:06:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:38.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2024-04-20T00:06:38.296 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-52 2024-04-20T00:06:38.296 DEBUG:boto:path=/aktlji-52/ 2024-04-20T00:06:38.296 DEBUG:boto:auth_path=/aktlji-52/ 2024-04-20T00:06:38.296 DEBUG:boto:Method: PUT 2024-04-20T00:06:38.296 DEBUG:boto:Path: /aktlji-52/ 2024-04-20T00:06:38.297 DEBUG:boto:Data: 2024-04-20T00:06:38.297 DEBUG:boto:Headers: {} 2024-04-20T00:06:38.297 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:06:38.297 DEBUG:boto:Port: 8000 2024-04-20T00:06:38.297 DEBUG:boto:Params: {} 2024-04-20T00:06:38.297 DEBUG:boto:Token: None 2024-04-20T00:06:38.297 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:38 GMT /aktlji-52/ 2024-04-20T00:06:38.297 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:7ShLmarWTsPIQA8uGSMQ5VOHLLs= 2024-04-20T00:06:38.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7ShLmarWTsPIQA8uGSMQ5VOHLLs=', 'Content-Length': '0'} 2024-04-20T00:06:38.321 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001e54dd53b1ca570b-006623070e-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:06:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:38.321 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-53 2024-04-20T00:06:38.322 DEBUG:boto:path=/aktlji-53/ 2024-04-20T00:06:38.322 DEBUG:boto:auth_path=/aktlji-53/ 2024-04-20T00:06:38.322 DEBUG:boto:Method: PUT 2024-04-20T00:06:38.322 DEBUG:boto:Path: /aktlji-53/ 2024-04-20T00:06:38.322 DEBUG:boto:Data: 2024-04-20T00:06:38.322 DEBUG:boto:Headers: {} 2024-04-20T00:06:38.322 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:06:38.322 DEBUG:boto:Port: 8000 2024-04-20T00:06:38.323 DEBUG:boto:Params: {} 2024-04-20T00:06:38.323 DEBUG:boto:Token: None 2024-04-20T00:06:38.323 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:38 GMT /aktlji-53/ 2024-04-20T00:06:38.323 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Mk4xfhjHlgvBmF8LJyDapGz+YjI= 2024-04-20T00:06:38.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Mk4xfhjHlgvBmF8LJyDapGz+YjI=', 'Content-Length': '0'} 2024-04-20T00:06:39.058 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fb7d7d0f6948e914-006623070e-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:06:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:39.059 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-54 2024-04-20T00:06:39.059 DEBUG:boto:path=/aktlji-54/ 2024-04-20T00:06:39.059 DEBUG:boto:auth_path=/aktlji-54/ 2024-04-20T00:06:39.059 DEBUG:boto:Method: PUT 2024-04-20T00:06:39.059 DEBUG:boto:Path: /aktlji-54/ 2024-04-20T00:06:39.059 DEBUG:boto:Data: 2024-04-20T00:06:39.059 DEBUG:boto:Headers: {} 2024-04-20T00:06:39.059 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:06:39.059 DEBUG:boto:Port: 8001 2024-04-20T00:06:39.060 DEBUG:boto:Params: {} 2024-04-20T00:06:39.060 DEBUG:boto:Token: None 2024-04-20T00:06:39.060 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:39 GMT /aktlji-54/ 2024-04-20T00:06:39.060 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FCsJtx0eLGHVwmLKaJ1a/LnIZC0= 2024-04-20T00:06:39.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FCsJtx0eLGHVwmLKaJ1a/LnIZC0=', 'Content-Length': '0'} 2024-04-20T00:06:39.092 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e1311d023a0e21b3-006623070f-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:06:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:39.092 DEBUG:boto:path=/aktlji-52/ 2024-04-20T00:06:39.092 DEBUG:boto:auth_path=/aktlji-52/ 2024-04-20T00:06:39.092 DEBUG:boto:path=/aktlji-52/?policy 2024-04-20T00:06:39.093 DEBUG:boto:auth_path=/aktlji-52/?policy 2024-04-20T00:06:39.093 DEBUG:boto:Method: PUT 2024-04-20T00:06:39.093 DEBUG:boto:Path: /aktlji-52/?policy 2024-04-20T00:06:39.093 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-20T00:06:39.093 DEBUG:boto:Headers: {} 2024-04-20T00:06:39.093 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:06:39.093 DEBUG:boto:Port: 8000 2024-04-20T00:06:39.093 DEBUG:boto:Params: {} 2024-04-20T00:06:39.093 DEBUG:boto:Token: None 2024-04-20T00:06:39.093 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:39 GMT /aktlji-52/?policy 2024-04-20T00:06:39.094 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QXRP+DHb0Vfo2AWSfL3+Nwdj9tI= 2024-04-20T00:06:39.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QXRP+DHb0Vfo2AWSfL3+Nwdj9tI=', 'Content-Length': '97'} 2024-04-20T00:06:39.100 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c1d4d0b6ee550f8f-006623070f-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:06:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:39.100 DEBUG:boto:path=/aktlji-52/ 2024-04-20T00:06:39.100 DEBUG:boto:auth_path=/aktlji-52/ 2024-04-20T00:06:39.100 DEBUG:boto:path=/aktlji-52/?policy 2024-04-20T00:06:39.100 DEBUG:boto:auth_path=/aktlji-52/?policy 2024-04-20T00:06:39.101 DEBUG:boto:Method: GET 2024-04-20T00:06:39.101 DEBUG:boto:Path: /aktlji-52/?policy 2024-04-20T00:06:39.101 DEBUG:boto:Data: 2024-04-20T00:06:39.101 DEBUG:boto:Headers: {} 2024-04-20T00:06:39.101 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:06:39.101 DEBUG:boto:Port: 8000 2024-04-20T00:06:39.101 DEBUG:boto:Params: {} 2024-04-20T00:06:39.101 DEBUG:boto:Token: None 2024-04-20T00:06:39.101 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 00:06:39 GMT /aktlji-52/?policy 2024-04-20T00:06:39.101 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oQWSAatYEczxZIcaCE1fsv5Ohck= 2024-04-20T00:06:39.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oQWSAatYEczxZIcaCE1fsv5Ohck=', 'Content-Length': '0'} 2024-04-20T00:06:39.103 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000095bbff98fc25925b-006623070f-15b6-test-zone1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Sat, 20 Apr 2024 00:06:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:39.103 DEBUG:boto:path=/aktlji-53/ 2024-04-20T00:06:39.103 DEBUG:boto:auth_path=/aktlji-53/ 2024-04-20T00:06:39.103 DEBUG:boto:path=/aktlji-53/?policy 2024-04-20T00:06:39.103 DEBUG:boto:auth_path=/aktlji-53/?policy 2024-04-20T00:06:39.103 DEBUG:boto:Method: PUT 2024-04-20T00:06:39.103 DEBUG:boto:Path: /aktlji-53/?policy 2024-04-20T00:06:39.103 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-20T00:06:39.103 DEBUG:boto:Headers: {} 2024-04-20T00:06:39.103 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:06:39.103 DEBUG:boto:Port: 8000 2024-04-20T00:06:39.104 DEBUG:boto:Params: {} 2024-04-20T00:06:39.104 DEBUG:boto:Token: None 2024-04-20T00:06:39.104 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:39 GMT /aktlji-53/?policy 2024-04-20T00:06:39.104 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vPbVaEbC8TDr+UOUtEzclEDH9rk= 2024-04-20T00:06:39.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vPbVaEbC8TDr+UOUtEzclEDH9rk=', 'Content-Length': '97'} 2024-04-20T00:06:40.117 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000198537790be54fcd-006623070f-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:40.117 DEBUG:boto:path=/aktlji-53/ 2024-04-20T00:06:40.117 DEBUG:boto:auth_path=/aktlji-53/ 2024-04-20T00:06:40.118 DEBUG:boto:path=/aktlji-53/?policy 2024-04-20T00:06:40.118 DEBUG:boto:auth_path=/aktlji-53/?policy 2024-04-20T00:06:40.118 DEBUG:boto:Method: GET 2024-04-20T00:06:40.118 DEBUG:boto:Path: /aktlji-53/?policy 2024-04-20T00:06:40.118 DEBUG:boto:Data: 2024-04-20T00:06:40.118 DEBUG:boto:Headers: {} 2024-04-20T00:06:40.118 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:06:40.118 DEBUG:boto:Port: 8000 2024-04-20T00:06:40.118 DEBUG:boto:Params: {} 2024-04-20T00:06:40.119 DEBUG:boto:Token: None 2024-04-20T00:06:40.119 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 00:06:40 GMT /aktlji-53/?policy 2024-04-20T00:06:40.119 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Re5+xzQNn1/rTRzVFmeGez8IFRE= 2024-04-20T00:06:40.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Re5+xzQNn1/rTRzVFmeGez8IFRE=', 'Content-Length': '0'} 2024-04-20T00:06:40.120 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009dd7ee939f83203b-0066230710-1bfc-test-zone2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Sat, 20 Apr 2024 00:06:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:40.120 DEBUG:boto:path=/aktlji-54/ 2024-04-20T00:06:40.121 DEBUG:boto:auth_path=/aktlji-54/ 2024-04-20T00:06:40.121 DEBUG:boto:path=/aktlji-54/?policy 2024-04-20T00:06:40.121 DEBUG:boto:auth_path=/aktlji-54/?policy 2024-04-20T00:06:40.121 DEBUG:boto:Method: PUT 2024-04-20T00:06:40.121 DEBUG:boto:Path: /aktlji-54/?policy 2024-04-20T00:06:40.121 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-20T00:06:40.121 DEBUG:boto:Headers: {} 2024-04-20T00:06:40.121 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:06:40.121 DEBUG:boto:Port: 8001 2024-04-20T00:06:40.121 DEBUG:boto:Params: {} 2024-04-20T00:06:40.122 DEBUG:boto:Token: None 2024-04-20T00:06:40.122 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:40 GMT /aktlji-54/?policy 2024-04-20T00:06:40.122 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cLdB3cmtTS1PFcnHEgWTJUkqSzY= 2024-04-20T00:06:40.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cLdB3cmtTS1PFcnHEgWTJUkqSzY=', 'Content-Length': '97'} 2024-04-20T00:06:41.134 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a32d478156b062da-0066230710-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:06:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:41.134 DEBUG:boto:path=/aktlji-54/ 2024-04-20T00:06:41.134 DEBUG:boto:auth_path=/aktlji-54/ 2024-04-20T00:06:41.134 DEBUG:boto:path=/aktlji-54/?policy 2024-04-20T00:06:41.134 DEBUG:boto:auth_path=/aktlji-54/?policy 2024-04-20T00:06:41.134 DEBUG:boto:Method: GET 2024-04-20T00:06:41.135 DEBUG:boto:Path: /aktlji-54/?policy 2024-04-20T00:06:41.135 DEBUG:boto:Data: 2024-04-20T00:06:41.135 DEBUG:boto:Headers: {} 2024-04-20T00:06:41.135 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:06:41.135 DEBUG:boto:Port: 8001 2024-04-20T00:06:41.135 DEBUG:boto:Params: {} 2024-04-20T00:06:41.135 DEBUG:boto:Token: None 2024-04-20T00:06:41.136 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 00:06:41 GMT /aktlji-54/?policy 2024-04-20T00:06:41.136 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:K+WUWrqJJaWe6uxX3G4ezfkm6BE= 2024-04-20T00:06:41.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K+WUWrqJJaWe6uxX3G4ezfkm6BE=', 'Content-Length': '0'} 2024-04-20T00:06:41.137 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003cde08b336a3201e-0066230711-15ba-test-zone3'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Sat, 20 Apr 2024 00:06:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:41.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2024-04-20T00:06:41.138 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-55 2024-04-20T00:06:41.138 DEBUG:boto:path=/aktlji-55/ 2024-04-20T00:06:41.139 DEBUG:boto:auth_path=/aktlji-55/ 2024-04-20T00:06:41.139 DEBUG:boto:Method: PUT 2024-04-20T00:06:41.139 DEBUG:boto:Path: /aktlji-55/ 2024-04-20T00:06:41.139 DEBUG:boto:Data: 2024-04-20T00:06:41.139 DEBUG:boto:Headers: {} 2024-04-20T00:06:41.139 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:06:41.139 DEBUG:boto:Port: 8000 2024-04-20T00:06:41.139 DEBUG:boto:Params: {} 2024-04-20T00:06:41.139 DEBUG:boto:Token: None 2024-04-20T00:06:41.140 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:41 GMT /aktlji-55/ 2024-04-20T00:06:41.140 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+5KPnXY2rfd2jt3X+akmKE1pcc8= 2024-04-20T00:06:41.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+5KPnXY2rfd2jt3X+akmKE1pcc8=', 'Content-Length': '0'} 2024-04-20T00:06:41.159 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002afe204477f910b2-0066230711-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:06:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:41.160 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-56 2024-04-20T00:06:41.160 DEBUG:boto:path=/aktlji-56/ 2024-04-20T00:06:41.160 DEBUG:boto:auth_path=/aktlji-56/ 2024-04-20T00:06:41.160 DEBUG:boto:Method: PUT 2024-04-20T00:06:41.160 DEBUG:boto:Path: /aktlji-56/ 2024-04-20T00:06:41.160 DEBUG:boto:Data: 2024-04-20T00:06:41.161 DEBUG:boto:Headers: {} 2024-04-20T00:06:41.161 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:06:41.161 DEBUG:boto:Port: 8000 2024-04-20T00:06:41.161 DEBUG:boto:Params: {} 2024-04-20T00:06:41.161 DEBUG:boto:Token: None 2024-04-20T00:06:41.161 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:41 GMT /aktlji-56/ 2024-04-20T00:06:41.161 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jiHKUF81rP8trOToOtuspjXkhQo= 2024-04-20T00:06:41.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jiHKUF81rP8trOToOtuspjXkhQo=', 'Content-Length': '0'} 2024-04-20T00:06:41.209 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d5869ccd9b32e11-0066230711-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:06:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:41.209 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-57 2024-04-20T00:06:41.209 DEBUG:boto:path=/aktlji-57/ 2024-04-20T00:06:41.209 DEBUG:boto:auth_path=/aktlji-57/ 2024-04-20T00:06:41.209 DEBUG:boto:Method: PUT 2024-04-20T00:06:41.209 DEBUG:boto:Path: /aktlji-57/ 2024-04-20T00:06:41.209 DEBUG:boto:Data: 2024-04-20T00:06:41.209 DEBUG:boto:Headers: {} 2024-04-20T00:06:41.210 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:06:41.210 DEBUG:boto:Port: 8001 2024-04-20T00:06:41.210 DEBUG:boto:Params: {} 2024-04-20T00:06:41.210 DEBUG:boto:Token: None 2024-04-20T00:06:41.210 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:06:41 GMT /aktlji-57/ 2024-04-20T00:06:41.210 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rbM7pVaE/9RCizk4mqycHUOF7mg= 2024-04-20T00:06:41.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rbM7pVaE/9RCizk4mqycHUOF7mg=', 'Content-Length': '0'} 2024-04-20T00:06:41.243 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000056145bc34cd8efe7-0066230711-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:06:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:06:41.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:41.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:41.243 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:06:41.300 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:41.300 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:41.338 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:06:41.352 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:06:41.352 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:41.352 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:06:41.353 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:06:41.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:41.353 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:41.353 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:06:41.353 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:06:41.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:41.353 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:41.353 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713571601.228671_641.1", 2024-04-20T00:06:41.353 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:06:41.228671Z" 2024-04-20T00:06:41.353 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:06:41.353 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:06:41.354 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:06:41.354 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:06:41.354 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:06:41.354 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:06:41.355 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:41.355 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:06:41.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:41.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:41.355 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:06:41.413 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:41.413 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:41.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:41.477+0000 7f1993e6f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:41.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:41.478+0000 7f1993e6f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:41.505 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:06:41.505 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:06:41.505 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:06:41.505 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:06:41.506 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:06:41.506 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:41.506 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:41.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:41.506 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:06:41.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:41.506 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:06:41.506 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:41.506 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:41.506 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:06:41.507 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:41.507 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:06:41.507 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:41.507 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:06:41.507 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:41.507 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:41.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:41.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:41.507 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:06:41.507 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:41.507 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:06:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:06:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:06:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:41.508 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:06:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:06:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:06:41.509 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:41.510 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:41.510 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:06:41.510 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:41.510 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:41.510 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:41.510 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:06:41.510 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:41.510 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:41.510 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:41.510 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:41.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:41.511 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:06:41.511 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:06:41.511 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:06:41.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:41.511 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:06:41Z" 2024-04-20T00:06:41.511 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:06:41.512 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:06:41Z" } 2024-04-20T00:06:41.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:06:41.512 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:41.512 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:41.512 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:06:46.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:46.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:46.514 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:06:46.573 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:46.574 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:46.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:46.616+0000 7f050aed5780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:46.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:46.617+0000 7f050aed5780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:46.637 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:06:46.637 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:06:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:06:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:06:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:06:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:06:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:06:46.638 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:06:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:06:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:06:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:46.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:06:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:06:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:06:46.640 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:06:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:06:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:46.641 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:06:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:06:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:46.642 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:06:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:06:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:46.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:06:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:06:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:06:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:46.644 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:06:46Z" 2024-04-20T00:06:46.644 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:06:46.645 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:06:46Z" } 2024-04-20T00:06:46.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:06:46.645 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:46.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:46.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:06:51.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:51.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:51.647 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:06:51.707 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:51.707 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:51.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:51.760+0000 7fcfd2bdd780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:51.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:51.761+0000 7fcfd2bdd780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:51.785 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:06:51.785 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:06:51.785 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:06:51.785 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:06:51.786 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:06:51.786 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:51.786 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:51.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:51.786 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:06:51.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:51.786 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:06:51.786 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:51.786 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:51.786 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:06:51.786 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:51.787 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:06:51.787 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:51.787 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:06:51.787 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:51.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:51.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:51.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:51.787 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:06:51.787 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:51.787 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:51.787 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:06:51.788 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:51.788 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:51.788 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:51.788 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:06:51.788 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:51.788 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:51.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:51.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:51.788 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:06:51.788 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:51.788 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:51.789 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:06:51.789 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:51.789 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:51.789 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:51.789 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:06:51.789 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:51.789 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:51.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:51.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:51.789 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:06:51.790 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:51.790 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:51.790 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:06:51.790 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:51.790 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:51.790 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:51.790 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:06:51.790 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:51.790 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:51.790 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:51.791 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:51.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:51.791 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:06:51.791 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:06:51.791 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:06:51.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:51.791 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:06:51Z" 2024-04-20T00:06:51.791 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:06:51.792 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:06:51Z" } 2024-04-20T00:06:51.792 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:06:51.793 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:51.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:51.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:06:56.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:56.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:06:56.794 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:06:56.854 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:06:56.855 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:06:56.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:56.910+0000 7f86e4b27780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:56.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:06:56.911+0000 7f86e4b27780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:06:56.933 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:06:56.933 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:06:56.933 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:06:56.933 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:06:56.933 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:56.936 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:56.936 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:06:56.936 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:56.936 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:56.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:56.936 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:56.936 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:06:56.936 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:56.936 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:56.936 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:06:56.936 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:56.936 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:56.937 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:56.937 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:06:56.937 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:56.937 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:56.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:56.937 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:06:56.937 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:06:56.937 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:06:56.937 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:06:56.937 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:06:56.937 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:06:56.937 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:06:56.938 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:06:56.938 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:06:56.938 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:06:56.938 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:56.938 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:06:56.938 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:06:56.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:56.938 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:06:56.938 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:06:56.938 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:06:56.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:06:56.939 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:06:56Z" 2024-04-20T00:06:56.939 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:06:56.939 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:06:56Z" } 2024-04-20T00:06:56.940 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:06:56.940 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:56.940 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:06:56.940 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:07:01.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:01.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:01.942 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:07:02.002 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:07:02.002 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:07:02.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:02.043+0000 7fae588b3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:02.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:02.044+0000 7fae588b3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:02.072 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:07:02.072 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:07:02.072 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:07:02.072 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:07:02.072 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:07:02.072 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:07:02.072 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:02.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:02.072 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:07:02.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:02.073 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:07:02.073 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:02.073 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:02.073 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:07:02.073 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:02.073 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:07:02.073 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:02.073 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:07:02.073 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:02.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:02.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:02.074 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:02.074 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:07:02.074 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:02.074 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:02.074 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:07:02.074 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:02.074 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:02.074 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:02.074 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:07:02.074 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:02.075 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:02.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:02.075 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:02.075 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:07:02.075 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:02.075 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:02.075 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:07:02.075 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:02.075 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:02.075 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:02.075 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:07:02.076 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:02.076 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:02.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:02.076 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:02.076 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:07:02.076 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:02.076 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:02.076 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:07:02.076 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:02.076 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:02.076 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:02.077 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:07:02.077 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:02.077 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:02.077 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:02.077 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:07:02.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:02.077 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:07:02.077 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:07:02.077 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:07:02.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:02.077 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:07:02Z" 2024-04-20T00:07:02.077 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:07:02.078 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:07:02Z" } 2024-04-20T00:07:02.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:07:02.079 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:02.079 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:02.079 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:07:07.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:07.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:07.080 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:07:07.140 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:07:07.140 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:07:07.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:07.196+0000 7f83382e8780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:07.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:07.197+0000 7f83382e8780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:07.221 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:07:07.221 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:07:07.222 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:07:07.222 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:07:07.222 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:07:07.222 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:07:07.222 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:07.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:07.222 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:07:07.222 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:07.222 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:07:07.223 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:07.223 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:07.223 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:07:07.223 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:07.223 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:07:07.223 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:07.223 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:07:07.223 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:07.223 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:07.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:07.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:07.224 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:07:07.224 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:07.224 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:07.224 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:07:07.224 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:07.224 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:07.224 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:07.224 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:07:07.224 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:07.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:07.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:07.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:07.225 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:07:07.225 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:07.225 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:07.225 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:07:07.225 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:07.225 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:07.225 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:07.225 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:07:07.226 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:07.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:07.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:07.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:07.226 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:07:07.226 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:07.226 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:07.226 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:07:07.226 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:07.226 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:07.226 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:07.227 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:07:07.227 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:07.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:07.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:07.227 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:07:07.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:07.227 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:07:07.227 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:07:07.227 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:07:07.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:07.227 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:07:07Z" 2024-04-20T00:07:07.228 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:07:07.228 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:07:07Z" } 2024-04-20T00:07:07.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:07:07.229 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:07.229 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:07.229 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:07:12.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:12.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:12.230 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:07:12.290 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:07:12.291 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:07:12.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:12.344+0000 7fc909adf780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:12.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:12.345+0000 7fc909adf780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:12.367 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:07:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:07:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:07:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:07:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:07:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:07:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:07:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:07:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:07:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:07:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:07:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:07:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:07:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:07:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:07:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:07:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:07:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:07:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:07:12.373 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:12.373 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:12.373 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:12.373 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:07:12.373 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:12.373 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:12.373 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:12.373 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:07:12.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:12.373 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:07:12.373 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:07:12.374 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:07:12.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:12.374 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:07:12Z" 2024-04-20T00:07:12.374 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:07:12.375 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:07:12Z" } 2024-04-20T00:07:12.375 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:07:12.375 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:12.375 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:12.375 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:07:17.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:17.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:17.377 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:07:17.437 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:07:17.437 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:07:17.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:17.474+0000 7fda18899780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:17.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:17.476+0000 7fda18899780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:17.497 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:07:17.497 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:07:17.498 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:07:17.498 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:07:17.498 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:07:17.498 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:07:17.498 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:17.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:17.498 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:07:17.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:17.498 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:07:17.499 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:17.499 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:17.499 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:07:17.499 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:17.499 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:07:17.499 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:17.499 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:07:17.499 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:17.499 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:17.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:17.499 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:17.500 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:07:17.500 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:17.500 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:17.500 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:07:17.500 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:17.500 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:17.500 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:17.500 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:07:17.500 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:17.500 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:17.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:17.501 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:17.501 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:07:17.501 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:17.501 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:17.501 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:07:17.501 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:17.501 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:17.501 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:17.501 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:07:17.501 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:17.501 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:17.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:17.502 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:17.502 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:07:17.502 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:17.502 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:17.502 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:07:17.502 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:17.502 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:17.502 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:17.502 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:07:17.503 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:17.503 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:17.503 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:17.503 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:07:17.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:17.503 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:07:17.503 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:07:17.503 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:07:17.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:17.503 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:07:17Z" 2024-04-20T00:07:17.503 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:07:17.504 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:07:17Z" } 2024-04-20T00:07:17.505 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:07:17.505 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:17.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:17.505 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:07:22.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:22.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:22.507 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:07:22.566 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:07:22.567 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:07:22.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:22.614+0000 7f42def7e780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:22.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:22.615+0000 7f42def7e780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:22.642 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:07:22.642 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:07:22.642 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:07:22.642 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:07:22.642 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:07:22.643 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:07:22.643 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:22.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:22.643 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:07:22.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:22.643 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:07:22.643 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:22.643 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:22.643 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:07:22.644 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:22.644 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:07:22.644 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:22.644 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:07:22.644 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:22.644 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:22.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:22.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:22.644 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:07:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:07:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:07:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:22.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:07:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:07:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:22.646 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:07:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:07:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:07:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:07:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:07:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:07:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:07:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:07:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:07:22Z" 2024-04-20T00:07:22.649 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:07:22.650 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:07:22Z" } 2024-04-20T00:07:22.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:07:22.651 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:22.651 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:22.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:07:27.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:27.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:27.653 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:07:27.715 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:07:27.715 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:07:27.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:27.760+0000 7f6be4d6a780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:27.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:27.761+0000 7f6be4d6a780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:27.781 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:07:27.781 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:07:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:07:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:07:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:07:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:07:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:07:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:27.782 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:07:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:07:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:07:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:07:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:27.783 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:07:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:07:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:27.784 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:07:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:07:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:07:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:07:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:07:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:07:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:07:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:07:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:07:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:07:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:07:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:07:27Z" 2024-04-20T00:07:27.788 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:07:27.789 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:07:27Z" } 2024-04-20T00:07:27.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:07:27.789 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:27.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:27.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:07:32.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:32.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:32.791 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:07:32.849 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:07:32.849 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:07:32.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:32.888+0000 7f17b730c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:32.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:32.889+0000 7f17b730c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:32.909 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:07:32.909 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:07:32.909 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:07:32.909 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:07:32.909 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:07:32.909 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:07:32.909 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:32.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:32.910 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:07:32.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:32.910 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:07:32.910 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:32.910 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:32.910 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:07:32.910 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:32.910 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:07:32.910 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:32.910 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:07:32.911 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:32.911 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:32.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:32.911 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:32.911 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:07:32.911 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:32.911 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:32.911 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:07:32.911 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:32.911 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:32.912 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:32.912 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:07:32.912 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:32.912 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:32.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:32.912 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:32.912 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:07:32.912 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:32.912 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:32.912 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:07:32.912 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:07:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:07:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:07:32.913 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:32.914 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:32.914 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:32.914 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:07:32.914 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:32.914 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:32.914 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:32.914 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:07:32.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:32.914 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:07:32.914 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:07:32.914 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:07:32.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:32.915 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:07:32Z" 2024-04-20T00:07:32.915 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:07:32.916 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:07:32Z" } 2024-04-20T00:07:32.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:07:32.916 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:32.916 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:32.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:07:37.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:37.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:37.917 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:07:37.977 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:07:37.978 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:07:38.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:38.017+0000 7fb0f09c5780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:38.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:38.018+0000 7fb0f09c5780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:38.043 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:07:38.043 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:07:38.043 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:07:38.043 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:07:38.043 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:07:38.043 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:07:38.043 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:38.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:38.044 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:07:38.044 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:38.044 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:07:38.044 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:38.044 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:38.044 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:07:38.044 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:38.044 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:07:38.044 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:38.044 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:07:38.045 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:38.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:38.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:38.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:38.045 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:07:38.045 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:38.045 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:38.045 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:07:38.045 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:38.045 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:38.046 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:38.046 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:07:38.046 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:38.046 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:38.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:38.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:38.046 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:07:38.046 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:38.046 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:38.046 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:07:38.046 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:38.047 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:38.047 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:38.047 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:07:38.047 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:38.047 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:38.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:38.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:38.047 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:07:38.047 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:38.047 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:38.047 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:07:38.048 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:38.048 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:38.048 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:38.048 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:07:38.048 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:38.048 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:38.048 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:38.048 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:07:38.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:38.048 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:07:38.048 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:07:38.048 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:07:38.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:38.049 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:07:38Z" 2024-04-20T00:07:38.049 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:07:38.050 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:07:38Z" } 2024-04-20T00:07:38.050 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:07:38.050 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:38.050 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:38.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:07:43.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:43.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:43.052 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:07:43.110 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:07:43.110 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:07:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:43.168+0000 7f20d47e5780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:43.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:43.169+0000 7f20d47e5780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:43.197 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:07:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:07:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:07:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:07:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:07:43.197 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:07:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:07:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:07:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:07:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:07:43.198 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:43.199 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:07:43.199 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:43.199 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:43.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:43.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:43.199 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:07:43.199 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:43.199 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:43.199 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:07:43.199 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:43.199 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:43.200 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:43.200 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:07:43.200 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:43.200 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:43.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:43.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:43.200 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:07:43.200 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:43.200 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:43.200 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:07:43.201 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:43.201 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:43.201 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:43.201 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:07:43.201 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:43.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:43.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:43.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:43.201 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:07:43.201 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:43.201 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:43.202 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:07:43.202 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:43.202 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:43.202 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:43.202 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:07:43.202 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:43.202 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:43.202 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:43.202 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:07:43.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:43.202 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:07:43.203 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:07:43.203 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:07:43.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:43.203 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:07:43Z" 2024-04-20T00:07:43.203 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:07:43.204 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:07:43Z" } 2024-04-20T00:07:43.204 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:07:43.204 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:43.204 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:43.204 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:07:48.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:48.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:48.206 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:07:48.266 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:07:48.266 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:07:48.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:48.305+0000 7fdef4114780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:48.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:48.306+0000 7fdef4114780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:48.326 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:07:48.326 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:07:48.326 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:07:48.326 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:07:48.326 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:07:48.327 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:07:48.327 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:48.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:48.327 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:07:48.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:48.327 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:07:48.327 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:48.327 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:48.327 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:07:48.327 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:48.328 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:07:48.328 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:48.328 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:07:48.328 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:48.328 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:48.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:48.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:48.328 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:07:48.328 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:48.328 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:48.329 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:07:48.329 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:48.329 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:48.329 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:48.329 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:07:48.329 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:48.329 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:48.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:48.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:48.329 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:07:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:07:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:07:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:48.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:07:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:07:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:07:48.331 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:48.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:48.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:48.332 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:07:48.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:48.332 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:07:48.332 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:07:48.332 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:07:48.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:48.332 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:07:48Z" 2024-04-20T00:07:48.332 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:07:48.333 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:07:48Z" } 2024-04-20T00:07:48.334 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:07:48.334 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:48.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:48.334 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:07:53.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:53.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:53.335 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:07:53.394 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:07:53.395 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:07:53.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:53.435+0000 7fdf295a8780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:53.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:53.436+0000 7fdf295a8780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:53.464 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:07:53.464 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:07:53.464 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:07:53.464 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:07:53.465 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:07:53.465 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:07:53.465 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:53.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:53.465 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:07:53.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:53.465 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:07:53.465 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:53.465 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:53.465 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:07:53.466 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:53.466 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:07:53.466 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:53.466 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:07:53.466 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:53.466 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:53.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:53.466 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:53.466 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:07:53.467 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:53.467 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:53.467 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:07:53.467 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:53.467 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:53.467 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:53.467 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:07:53.467 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:53.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:53.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:53.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:53.468 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:07:53.468 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:53.468 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:53.468 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:07:53.468 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:53.468 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:53.468 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:53.468 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:07:53.468 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:53.469 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:53.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:53.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:53.469 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:07:53.469 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:53.469 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:53.469 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:07:53.469 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:53.469 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:53.469 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:53.470 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:07:53.470 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:53.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:53.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:53.470 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:07:53.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:53.470 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:07:53.470 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:07:53.470 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:07:53.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:53.471 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:07:53Z" 2024-04-20T00:07:53.471 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:07:53.472 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:07:53Z" } 2024-04-20T00:07:53.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:07:53.472 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:53.472 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:53.472 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:07:58.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:58.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:07:58.474 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:07:58.533 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:07:58.533 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:07:58.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:58.591+0000 7fb7af6bc780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:58.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:07:58.592+0000 7fb7af6bc780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:07:58.620 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:07:58.620 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:07:58.620 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:07:58.621 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:07:58.621 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:07:58.621 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:07:58.621 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:58.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:58.621 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:07:58.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:58.621 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:07:58.621 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:58.622 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:58.622 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:07:58.622 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:58.622 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:07:58.622 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:58.622 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:07:58.622 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:58.622 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:58.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:58.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:58.623 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:07:58.623 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:58.623 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:58.623 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:07:58.623 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:58.623 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:58.623 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:58.623 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:07:58.623 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:58.623 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:58.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:07:58.625 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:07:58.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:58.626 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:07:58.626 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:07:58.626 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:07:58.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:07:58.626 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:07:58Z" 2024-04-20T00:07:58.626 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:07:58.627 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:07:58Z" } 2024-04-20T00:07:58.628 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:07:58.628 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:58.628 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:07:58.628 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:08:03.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:03.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:03.630 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:08:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:08:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:08:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:03.727+0000 7fd227e22780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:03.728+0000 7fd227e22780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:03.749 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:08:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:08:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:08:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:08:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:08:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:08:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:08:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:08:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:08:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:08:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:08:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:08:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:08:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:08:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:08:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:08:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:08:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:08:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:08:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:08:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:08:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:08:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:08:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:08:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:08:03Z" 2024-04-20T00:08:03.755 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:08:03.756 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:08:03Z" } 2024-04-20T00:08:03.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:08:03.756 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:03.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:03.757 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:08:08.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:08.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:08.758 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:08:08.817 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:08:08.817 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:08:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:08.869+0000 7fbcae659780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:08.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:08.870+0000 7fbcae659780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:08.894 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:08:08.894 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:08:08.894 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:08:08.894 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:08:08.894 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:08:08.894 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:08:08.894 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:08.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:08.895 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:08:08.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:08.895 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:08:08.895 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:08.895 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:08.895 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:08:08.895 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:08.895 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:08:08.895 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:08.896 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:08:08.896 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:08.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:08.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:08.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:08.896 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:08:08.896 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:08.896 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:08.896 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:08:08.896 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:08.897 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:08.897 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:08.897 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:08:08.897 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:08.897 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:08.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:08.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:08.897 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:08:08.897 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:08.897 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:08.897 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:08:08.898 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:08.898 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:08.898 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:08.898 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:08:08.898 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:08.898 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:08.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:08.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:08.898 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:08:08.898 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:08.899 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:08.899 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:08:08.899 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:08.899 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:08.899 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:08.899 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:08:08.899 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:08.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:08.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:08.899 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:08:08.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:08.900 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:08:08.900 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:08:08.900 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:08:08.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:08.900 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:08:08Z" 2024-04-20T00:08:08.900 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:08:08.901 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:08:08Z" } 2024-04-20T00:08:08.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:08:08.901 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:08.901 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:08.901 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:08:13.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:13.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:13.903 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:08:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:08:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:08:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:14.004+0000 7fb59434d780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:14.006+0000 7fb59434d780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:14.026 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:08:14.027 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:08:14.027 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:08:14.027 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:08:14.027 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:08:14.027 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:08:14.028 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:14.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:14.028 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:08:14.028 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:14.028 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:08:14.028 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:14.028 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:14.029 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:08:14.029 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:14.029 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:08:14.029 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:14.029 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:08:14.029 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:14.030 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:14.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:14.030 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:14.030 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:08:14.031 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:14.031 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:14.031 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:08:14.031 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:14.031 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:14.031 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:14.031 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:08:14.032 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:14.032 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:14.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:14.032 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:14.032 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:08:14.032 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:14.033 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:14.033 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:08:14.033 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:14.033 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:14.033 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:14.033 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:08:14.033 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:14.034 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:14.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:14.034 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:14.034 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:08:14.034 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:14.034 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:14.034 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:08:14.035 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:14.035 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:14.035 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:14.035 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:08:14.035 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:14.035 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:14.035 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:14.036 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:08:14.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:14.036 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:08:14.036 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:08:14.036 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:08:14.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:14.036 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:08:14Z" 2024-04-20T00:08:14.037 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:08:14.038 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:08:14Z" } 2024-04-20T00:08:14.038 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:08:14.038 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:14.039 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:14.039 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:08:19.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:19.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:19.040 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:08:19.099 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:08:19.099 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:08:19.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:19.149+0000 7f61cbc61780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:19.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:19.150+0000 7f61cbc61780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:19.175 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:08:19.176 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:08:19.176 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:08:19.176 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:08:19.176 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:08:19.176 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:08:19.176 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:19.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:19.176 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:08:19.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:19.176 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:08:19.177 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:19.177 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:19.177 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:08:19.177 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:19.177 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:08:19.177 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:19.177 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:08:19.177 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:19.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:19.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:19.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:19.178 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:08:19.178 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:19.178 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:19.178 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:08:19.178 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:19.178 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:19.178 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:19.178 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:08:19.178 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:19.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:19.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:19.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:19.179 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:08:19.179 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:19.179 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:19.179 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:08:19.179 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:19.179 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:19.179 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:19.179 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:08:19.179 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:19.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:19.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:19.180 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:19.180 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:08:19.180 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:19.180 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:19.180 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:08:19.180 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:19.180 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:19.180 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:19.180 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:08:19.181 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:19.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:19.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:19.181 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:08:19.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:19.181 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:08:19.181 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:08:19.181 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:08:19.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:19.181 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:08:19Z" 2024-04-20T00:08:19.181 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:08:19.182 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:08:19Z" } 2024-04-20T00:08:19.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:08:19.183 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:19.183 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:19.183 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:08:24.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:24.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:24.185 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:08:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:08:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:08:24.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:24.289+0000 7f099d08f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:24.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:24.290+0000 7f099d08f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:24.311 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:08:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:08:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:08:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:08:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:08:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:08:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:08:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:08:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:08:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:08:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:08:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:08:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:08:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:08:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:08:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:08:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:08:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:08:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:08:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:08:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:08:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:08:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:08:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:08:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:08:24Z" 2024-04-20T00:08:24.316 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:08:24.317 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:08:24Z" } 2024-04-20T00:08:24.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:08:24.318 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:24.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:24.318 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:08:29.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:29.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:29.320 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:08:29.378 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:08:29.379 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:08:29.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:29.425+0000 7fc2ca07a780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:29.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:29.426+0000 7fc2ca07a780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:29.452 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:08:29.452 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:08:29.452 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:08:29.452 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:08:29.452 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:08:29.452 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:08:29.453 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:29.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:29.453 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:08:29.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:29.453 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:08:29.453 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:29.453 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:29.453 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:08:29.453 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:29.453 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:08:29.453 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:29.454 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:08:29.454 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:29.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:29.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:29.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:29.454 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:08:29.454 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:29.454 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:29.454 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:08:29.454 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:29.454 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:29.455 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:29.455 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:08:29.455 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:29.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:29.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:29.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:29.455 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:08:29.455 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:29.455 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:29.455 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:08:29.455 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:29.456 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:29.456 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:29.456 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:08:29.456 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:29.456 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:29.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:29.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:29.456 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:08:29.456 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:29.456 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:29.456 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:08:29.456 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:29.457 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:29.457 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:29.457 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:08:29.457 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:29.457 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:29.457 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:29.457 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:08:29.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:29.457 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:08:29.457 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:08:29.457 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:08:29.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:29.458 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:08:29Z" 2024-04-20T00:08:29.458 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:08:29.459 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:08:29Z" } 2024-04-20T00:08:29.459 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:08:29.459 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:29.459 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:29.459 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:08:34.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:34.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:34.460 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:08:34.519 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:08:34.519 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:08:34.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:34.572+0000 7fd3ea5e4780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:34.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:34.573+0000 7fd3ea5e4780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:34.597 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:08:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:08:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:08:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:08:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:08:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:08:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:08:34.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:08:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:08:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:08:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:08:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:34.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:08:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:08:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:08:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:34.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:08:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:08:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:08:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:34.600 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:08:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:08:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:08:34.601 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:08:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:08:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:08:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:08:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:34.602 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:08:34Z" 2024-04-20T00:08:34.602 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:08:34.603 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:08:34Z" } 2024-04-20T00:08:34.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:08:34.604 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:34.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:34.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:08:39.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:39.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:39.605 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:08:39.666 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:08:39.666 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:08:39.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:39.703+0000 7f38a7870780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:39.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:39.705+0000 7f38a7870780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:39.730 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:08:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:08:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:08:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:08:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:08:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:08:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:39.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:08:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:08:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:08:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:08:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:08:39.731 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:39.732 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:39.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:39.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:39.732 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:08:39.732 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:39.732 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:39.732 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:08:39.732 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:39.732 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:39.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:39.732 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:08:39.732 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:39.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:39.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:39.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:39.733 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:08:39.733 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:39.733 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:39.733 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571232.655231_514.1", 2024-04-20T00:08:39.733 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:39.733 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:39.733 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:39.733 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T00:08:39.733 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:39.734 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:39.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:39.734 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:39.734 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:08:39.734 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:39.734 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:39.734 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:08:39.734 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:39.734 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:39.734 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:39.734 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:08:39.735 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:39.735 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:39.735 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:39.735 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:08:39.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:39.735 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:08:39.735 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:08:39.735 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:08:39.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:39.735 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:08:39Z" 2024-04-20T00:08:39.735 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:08:39.736 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571232.655231_514.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:00:32.655231Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:08:39Z" } 2024-04-20T00:08:39.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:08:39.737 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:39.737 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:39.737 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T00:08:44.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:44.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:44.738 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:08:44.798 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:08:44.798 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:08:44.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:44.847+0000 7fa697f25780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:44.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:44.848+0000 7fa697f25780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:44.871 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:08:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:08:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:08:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:08:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:08:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:08:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:44.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:08:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:08:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:08:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:44.872 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:08:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:08:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:08:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:44.873 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:08:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:44.874 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:08:44.875 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:08:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:44.876 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571598.305342_610.1", 2024-04-20T00:08:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:38.305342Z", 2024-04-20T00:08:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:44.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:08:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:44.878 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:08:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:08:44.879 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:08:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:08:44.880 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:08:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:08:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:44.881 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:08:44Z" 2024-04-20T00:08:44.881 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:08:44.883 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571598.305342_610.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:38.305342Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:08:44Z" } 2024-04-20T00:08:44.883 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.305342_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.305342Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:08:44.884 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:44.884 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.305342_610.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:44.884 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.305342_610.1 2024-04-20T00:08:49.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:49.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:49.885 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:08:49.943 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:08:49.944 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:08:50.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:50.001+0000 7fee43b91780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:50.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:50.003+0000 7fee43b91780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:50.024 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:08:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:08:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:08:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:08:50.024 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:08:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:08:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:08:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:08:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:08:50.025 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:08:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:08:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:08:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:50.026 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:08:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:08:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:08:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:50.027 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571597.910175_581.1", 2024-04-20T00:08:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:37.910175Z", 2024-04-20T00:08:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:08:50.028 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:08:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:08:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:08:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:08:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:08:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:08:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:08:50Z" 2024-04-20T00:08:50.030 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:08:50.031 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571597.910175_581.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:37.910175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:08:50Z" } 2024-04-20T00:08:50.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571597.910175_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:37.910175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:08:50.031 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:50.031 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571597.910175_581.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:50.031 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571597.910175_581.1 2024-04-20T00:08:55.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:55.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:08:55.033 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:08:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:08:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:08:55.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:55.151+0000 7fbf654f3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:55.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:08:55.153+0000 7fbf654f3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:08:55.176 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:08:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:08:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:08:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:08:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:08:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:08:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:08:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:08:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:08:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:08:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:08:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:55.178 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:08:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:08:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:08:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:55.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:08:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571598.253928_606.1", 2024-04-20T00:08:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:38.253928Z", 2024-04-20T00:08:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:55.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:08:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:08:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:08:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:08:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:08:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:08:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:08:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:08:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:08:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:08:55.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:08:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:08:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:08:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:08:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:08:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:08:55.182 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:08:55Z" 2024-04-20T00:08:55.182 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:08:55.183 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571598.253928_606.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:38.253928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:08:55Z" } 2024-04-20T00:08:55.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.253928_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.253928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:08:55.183 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:55.184 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.253928_606.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:08:55.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.253928_606.1 2024-04-20T00:09:00.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:00.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:00.185 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:09:00.243 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:09:00.244 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:09:00.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:00.283+0000 7f4f0977f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:00.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:00.284+0000 7f4f0977f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:00.307 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:09:00.307 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:09:00.307 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:09:00.307 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:09:00.307 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:09:00.307 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:09:00.307 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:00.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:00.308 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:09:00.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:00.308 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:09:00.308 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:00.308 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:00.308 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:09:00.308 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:00.308 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:09:00.308 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:00.309 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:09:00.309 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:00.309 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:00.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:00.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:00.309 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:09:00.309 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:00.309 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:00.309 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:09:00.309 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:00.309 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:00.310 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:00.310 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:09:00.310 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:00.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:00.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:00.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:00.310 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:09:00.310 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:00.310 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:00.311 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571598.253928_606.1", 2024-04-20T00:09:00.311 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:00.311 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:00.311 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:00.311 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:38.253928Z", 2024-04-20T00:09:00.311 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:00.311 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:00.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:00.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:00.311 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:09:00.311 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:00.311 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:00.312 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:09:00.312 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:00.312 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:00.312 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:00.312 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:09:00.312 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:00.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:00.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:00.312 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:09:00.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:00.313 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:09:00.313 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:09:00.313 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:09:00.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:00.313 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:09:00Z" 2024-04-20T00:09:00.313 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:09:00.314 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571598.253928_606.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:38.253928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:09:00Z" } 2024-04-20T00:09:00.314 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.253928_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.253928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:09:00.314 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:00.314 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.253928_606.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:00.314 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.253928_606.1 2024-04-20T00:09:05.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:05.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:05.316 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:09:05.375 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:09:05.376 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:09:05.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:05.424+0000 7fe3dc27d780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:05.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:05.426+0000 7fe3dc27d780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:05.450 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:09:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:09:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:09:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:09:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:09:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:09:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:09:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:09:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:09:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:09:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:09:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:09:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:09:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:09:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:09:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571598.253928_606.1", 2024-04-20T00:09:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:38.253928Z", 2024-04-20T00:09:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:09:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:09:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:09:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:09:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:09:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:09:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:09:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:09:05Z" 2024-04-20T00:09:05.457 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:09:05.458 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571598.253928_606.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:38.253928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:09:05Z" } 2024-04-20T00:09:05.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.253928_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.253928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:09:05.458 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:05.458 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.253928_606.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:05.458 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.253928_606.1 2024-04-20T00:09:10.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:10.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:10.460 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:09:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:09:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:09:10.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:10.568+0000 7f84023ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:10.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:10.569+0000 7f84023ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:10.601 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:09:10.601 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:09:10.601 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:09:10.602 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:09:10.602 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:09:10.602 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:09:10.602 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:10.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:10.602 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:09:10.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:10.602 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:09:10.602 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:10.602 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:10.603 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:09:10.603 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:10.603 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:09:10.603 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:10.603 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:09:10.603 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:10.603 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:10.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:10.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:10.603 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:09:10.604 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:10.604 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:10.604 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:09:10.604 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:10.604 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:10.604 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:10.604 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:09:10.604 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:10.604 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:10.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:10.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:10.605 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:09:10.605 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:10.605 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:10.605 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571598.263539_607.1", 2024-04-20T00:09:10.605 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:10.605 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:10.605 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:10.605 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:38.263539Z", 2024-04-20T00:09:10.605 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:10.605 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:10.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:10.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:10.606 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:09:10.606 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:10.606 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:10.606 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:09:10.606 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:10.606 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:10.606 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:10.606 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:09:10.606 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:10.606 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:10.607 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:10.607 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:09:10.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:10.607 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:09:10.607 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:09:10.607 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:09:10.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:10.607 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:09:10Z" 2024-04-20T00:09:10.607 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:09:10.608 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571598.263539_607.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:38.263539Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:09:10Z" } 2024-04-20T00:09:10.608 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.263539_607.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.263539Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:09:10.609 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:10.609 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.263539_607.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:10.609 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.263539_607.1 2024-04-20T00:09:15.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:15.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:15.610 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:09:15.670 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:09:15.671 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:09:15.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:15.731+0000 7f6b7cbfe780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:15.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:15.732+0000 7f6b7cbfe780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:15.759 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:09:15.759 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:09:15.759 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:09:15.759 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:09:15.759 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:09:15.759 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:09:15.760 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:15.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:15.760 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:09:15.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:15.760 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:09:15.760 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:15.760 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:15.760 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:09:15.760 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:15.760 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:09:15.760 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:15.760 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:09:15.761 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:15.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:15.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:15.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:15.761 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:09:15.761 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:15.761 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:15.761 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:09:15.761 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:15.761 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:09:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:09:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571598.253928_606.1", 2024-04-20T00:09:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:38.253928Z", 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:15.764 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:15.764 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:09:15.764 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:15.764 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:15.764 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:15.764 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:09:15.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:15.764 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:09:15.764 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:09:15.764 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:09:15.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:15.765 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:09:15Z" 2024-04-20T00:09:15.765 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:09:15.766 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571598.253928_606.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:38.253928Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:09:15Z" } 2024-04-20T00:09:15.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.253928_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.253928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:09:15.766 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:15.766 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.253928_606.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:15.766 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.253928_606.1 2024-04-20T00:09:20.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:20.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:20.768 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:09:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:09:20.838 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:09:20.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:20.942+0000 7f64c3101780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:20.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:20.943+0000 7f64c3101780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:20.980 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:09:20.980 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:09:20.980 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:09:20.980 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:09:20.981 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:09:20.981 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:09:20.981 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:20.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:20.981 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:09:20.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:20.981 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:09:20.981 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:20.981 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:20.982 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:09:20.982 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:20.982 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:09:20.982 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:20.982 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:09:20.982 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:20.982 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:20.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:20.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:20.983 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:09:20.983 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:20.983 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:20.983 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:09:20.983 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:20.984 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:20.984 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:20.984 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:09:20.984 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:20.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:20.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:20.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:20.984 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:09:20.984 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:20.984 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:20.985 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571597.910175_581.1", 2024-04-20T00:09:20.985 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:20.985 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:20.985 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:20.985 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:37.910175Z", 2024-04-20T00:09:20.985 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:20.985 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:20.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:20.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:20.986 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:09:20.986 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:20.986 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:20.986 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:09:20.986 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:20.986 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:20.986 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:20.986 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:09:20.986 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:20.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:20.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:20.987 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:09:20.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:20.987 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:09:20.987 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:09:20.987 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:09:20.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:20.987 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:09:20Z" 2024-04-20T00:09:20.987 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:09:20.988 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571597.910175_581.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:37.910175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:09:20Z" } 2024-04-20T00:09:20.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571597.910175_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:37.910175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:09:20.989 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:20.989 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571597.910175_581.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:20.989 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571597.910175_581.1 2024-04-20T00:09:25.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:25.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:25.990 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:09:26.048 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:09:26.049 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:09:26.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:26.104+0000 7f73b1a27780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:26.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:26.106+0000 7f73b1a27780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:26.127 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:09:26.127 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:09:26.127 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:09:26.127 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:09:26.127 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:09:26.127 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:09:26.127 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:26.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:26.127 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:09:26.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:26.128 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:09:26.128 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:26.128 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:26.128 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:09:26.128 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:26.128 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:09:26.128 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:26.128 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:09:26.128 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:26.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:26.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:26.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:26.129 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:09:26.129 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:26.129 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:26.129 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:09:26.129 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:26.129 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:26.129 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:26.129 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:09:26.129 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:26.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:26.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:26.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:26.130 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:09:26.130 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:26.130 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:26.130 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571598.266030_608.1", 2024-04-20T00:09:26.130 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:26.130 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:26.130 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:26.130 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:38.266030Z", 2024-04-20T00:09:26.130 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:26.130 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:26.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:26.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:26.131 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:09:26.131 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:26.131 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:26.131 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:09:26.131 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:26.131 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:26.131 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:26.131 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:09:26.131 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:26.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:26.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:26.132 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:09:26.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:26.132 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:09:26.132 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:09:26.132 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:09:26.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:26.132 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:09:26Z" 2024-04-20T00:09:26.132 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:09:26.133 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571598.266030_608.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:38.266030Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:09:26Z" } 2024-04-20T00:09:26.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.266030_608.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.266030Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:09:26.134 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:26.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.266030_608.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:26.134 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.266030_608.1 2024-04-20T00:09:31.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:31.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:31.136 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:09:31.195 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:09:31.195 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:09:31.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:31.246+0000 7fefd66c4780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:31.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:31.247+0000 7fefd66c4780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:31.275 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:09:31.275 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:09:31.275 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:09:31.275 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:09:31.275 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:09:31.275 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:09:31.275 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:31.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:31.276 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:09:31.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:31.276 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:09:31.276 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:31.276 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:31.276 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:09:31.276 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:31.276 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:09:31.276 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:31.276 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:09:31.276 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:31.277 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:31.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:31.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:31.278 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:09:31.278 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:31.278 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:31.278 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571597.919367_583.1", 2024-04-20T00:09:31.278 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:31.278 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:31.278 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:31.278 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:37.919367Z", 2024-04-20T00:09:31.278 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:31.279 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:31.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:31.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:31.279 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:09:31.279 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:31.279 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:31.279 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:09:31.279 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:31.279 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:31.279 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:31.279 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:09:31.279 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:31.280 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:31.280 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:31.280 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:09:31.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:31.280 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:09:31.280 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:09:31.280 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:09:31.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:31.280 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:09:31Z" 2024-04-20T00:09:31.280 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:09:31.281 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571597.919367_583.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:37.919367Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:09:31Z" } 2024-04-20T00:09:31.281 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571597.919367_583.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:37.919367Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:09:31.281 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:31.282 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571597.919367_583.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:31.282 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571597.919367_583.1 2024-04-20T00:09:36.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:36.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:36.284 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:09:36.342 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:09:36.342 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:09:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:36.398+0000 7f56f928b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:36.399+0000 7f56f928b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:36.420 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:09:36.420 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:09:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:09:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:09:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:09:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:09:36.421 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:09:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:09:36.422 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:09:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:09:36.423 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:09:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:36.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:09:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:09:36.425 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:09:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:09:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:36.427 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571598.106288_599.1", 2024-04-20T00:09:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:38.106288Z", 2024-04-20T00:09:36.428 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:09:36.429 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:09:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:36.430 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:09:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:09:36.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:09:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:09:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:09:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:36.432 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:09:36Z" 2024-04-20T00:09:36.433 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:09:36.434 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571598.106288_599.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:38.106288Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:09:36Z" } 2024-04-20T00:09:36.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.106288_599.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.106288Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:09:36.434 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:36.435 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.106288_599.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:36.435 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.106288_599.1 2024-04-20T00:09:41.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:41.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:41.437 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:09:41.497 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:09:41.497 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:09:41.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:41.540+0000 7ff1867e5780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:41.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:41.541+0000 7ff1867e5780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:41.562 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:09:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:09:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:09:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:09:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:09:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:09:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:41.562 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:09:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:09:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:09:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:09:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:09:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:41.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:41.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:09:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571598.305342_610.1", 2024-04-20T00:09:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:38.305342Z", 2024-04-20T00:09:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:41.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:09:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:09:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:09:41.566 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:09:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:09:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:09:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:09:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:41.567 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:09:41Z" 2024-04-20T00:09:41.567 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:09:41.568 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571598.305342_610.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:38.305342Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:09:41Z" } 2024-04-20T00:09:41.568 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.305342_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.305342Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:09:41.568 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:41.568 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.305342_610.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:41.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.305342_610.1 2024-04-20T00:09:46.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:46.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:46.570 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:09:46.631 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:09:46.631 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:09:46.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:46.677+0000 7f441d8e6780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:46.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:46.678+0000 7f441d8e6780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:46.701 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:09:46.701 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:09:46.701 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:09:46.701 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:09:46.701 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:09:46.701 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:09:46.701 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:46.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:46.701 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:09:46.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:46.702 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:09:46.702 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:46.702 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:46.702 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:09:46.702 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:46.702 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:09:46.702 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:46.702 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:09:46.702 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:46.702 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:46.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:46.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:46.703 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:09:46.703 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:46.703 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:46.703 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:09:46.703 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:46.703 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:46.703 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:46.703 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:09:46.703 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:46.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:46.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:46.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:46.704 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:09:46.704 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:46.704 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:46.704 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571597.910175_581.1", 2024-04-20T00:09:46.704 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:46.704 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:46.704 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:46.704 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:37.910175Z", 2024-04-20T00:09:46.704 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:46.704 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:46.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:46.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:46.705 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:09:46.705 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:46.705 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:46.705 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:09:46.705 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:46.705 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:46.705 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:46.705 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:09:46.705 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:46.705 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:46.706 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:46.706 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:09:46.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:46.706 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:09:46.706 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:09:46.706 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:09:46.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:46.706 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:09:46Z" 2024-04-20T00:09:46.706 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:09:46.707 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571597.910175_581.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:37.910175Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:09:46Z" } 2024-04-20T00:09:46.707 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571597.910175_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:37.910175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:09:46.707 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:46.707 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571597.910175_581.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:46.708 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571597.910175_581.1 2024-04-20T00:09:51.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:51.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:51.709 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:09:51.768 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:09:51.768 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:09:51.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:51.828+0000 7fbb26bca780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:51.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:51.830+0000 7fbb26bca780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:51.857 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:09:51.857 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:09:51.858 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:09:51.858 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:09:51.858 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:09:51.858 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:09:51.858 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:51.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:51.858 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:09:51.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:51.858 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:09:51.858 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:51.859 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:51.859 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:09:51.859 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:51.859 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:09:51.859 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:51.859 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:09:51.859 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:51.860 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:51.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:51.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:51.860 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:09:51.860 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:51.860 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:51.861 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:09:51.861 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:51.861 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:51.861 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:51.861 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:09:51.861 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:51.861 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:51.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:51.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:51.861 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:09:51.861 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:51.862 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:51.862 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571598.086361_594.1", 2024-04-20T00:09:51.862 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:51.862 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:51.862 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:51.862 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:38.086361Z", 2024-04-20T00:09:51.862 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:51.862 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:51.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:51.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:51.862 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:09:51.863 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:51.863 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:51.863 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:09:51.863 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:51.863 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:51.863 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:51.863 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:09:51.863 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:51.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:51.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:51.863 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:09:51.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:51.864 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:09:51.864 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:09:51.864 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:09:51.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:51.864 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:09:51Z" 2024-04-20T00:09:51.864 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:09:51.865 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571598.086361_594.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:38.086361Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:09:51Z" } 2024-04-20T00:09:51.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.086361_594.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.086361Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:09:51.865 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:51.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.086361_594.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:51.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.086361_594.1 2024-04-20T00:09:56.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:56.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:09:56.867 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:09:56.926 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:09:56.926 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:09:56.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:56.981+0000 7f5878b87780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:56.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:09:56.983+0000 7f5878b87780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:09:57.011 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:09:57.011 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:09:57.011 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:09:57.011 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:09:57.011 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:09:57.011 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:09:57.011 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:57.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:57.011 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:09:57.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:57.011 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:09:57.012 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:57.012 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:57.012 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:09:57.012 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:57.012 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:09:57.012 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:57.012 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:09:57.012 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:57.012 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:57.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:57.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:57.012 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:09:57.013 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:57.013 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:57.013 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:09:57.013 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:57.013 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:57.013 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:57.013 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:09:57.013 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:57.013 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:57.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:57.013 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:57.013 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:09:57.014 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:57.014 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:57.014 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571598.962301_616.1", 2024-04-20T00:09:57.014 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:57.014 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:57.014 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:57.014 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:38.962301Z", 2024-04-20T00:09:57.014 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:57.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:57.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:57.014 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:09:57.014 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:09:57.015 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:09:57.015 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:09:57.015 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:09:57.015 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:09:57.015 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:09:57.015 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:09:57.015 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:09:57.015 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:09:57.015 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:57.015 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:09:57.015 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:09:57.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:57.016 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:09:57.016 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:09:57.016 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:09:57.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:09:57.016 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:09:56Z" 2024-04-20T00:09:57.016 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:09:57.017 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571598.962301_616.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:38.962301Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:09:56Z" } 2024-04-20T00:09:57.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.962301_616.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.962301Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:09:57.017 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:57.017 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.962301_616.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:09:57.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.962301_616.1 2024-04-20T00:10:02.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:02.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:02.019 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:02.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:10:02.118+0000 7f6f8c4b8780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:10:02.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:10:02.119+0000 7f6f8c4b8780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:10:02.140 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:10:02.140 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:10:02.140 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:10:02.140 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:10:02.140 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:10:02.140 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:10:02.141 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:10:02.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:02.141 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:10:02.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:02.141 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:10:02.141 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:02.141 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:10:02.141 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:10:02.141 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:02.141 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:10:02.141 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:02.142 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:10:02.142 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:10:02.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:02.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:02.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:02.142 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:10:02.142 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:02.142 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:10:02.142 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:10:02.142 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:02.143 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:02.143 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:02.143 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:10:02.143 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:10:02.143 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:02.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:02.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:02.143 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:10:02.143 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:02.143 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:10:02.143 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713571601.228671_641.1", 2024-04-20T00:10:02.144 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:02.144 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:02.144 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:02.144 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:06:41.228671Z", 2024-04-20T00:10:02.144 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:10:02.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:02.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:02.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:02.144 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:10:02.144 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:02.144 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:10:02.144 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:10:02.145 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:02.145 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:02.145 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:02.145 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:10:02.145 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:10:02.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:02.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:02.145 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:10:02.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:02.145 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:10:02.145 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:10:02.145 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:10:02.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:02.146 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:10:02Z" 2024-04-20T00:10:02.146 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:10:02.147 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571601.228671_641.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:41.228671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:10:02Z" } 2024-04-20T00:10:02.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571601.228671_641.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:41.228671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:10:02.147 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:10:02.147 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:10:02.147 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:10:02.147 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:10:02.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:02.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:02.148 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:02.207 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:02.207 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:02.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:10:02.239+0000 7f1323785780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:10:02.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:10:02.240+0000 7f1323785780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:10:02.257 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:10:02.258 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:10:02.258 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:10:02.258 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:10:02.258 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:10:02.258 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:10:02.258 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:10:02.258 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:10:02.258 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:10:02.258 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:10:02.259 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:10:02.259 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:10:02.259 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:10:02.259 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:10:02.259 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:10:02.259 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:10:02.259 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:10:02.259 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:10:02.259 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:10:02.259 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:10:02.260 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:10:02.260 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:10:02.260 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:10:02.260 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:10:02.260 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:10:02.260 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:10:02.260 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:10:02.260 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:10:02.260 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:10:02.260 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:10:02.260 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:10:02.261 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:10:02.261 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:10:02.261 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:10:02.261 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:10:02.261 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:10:02.261 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:10:02.261 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713571601.228671_641.1", 2024-04-20T00:10:02.261 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:10:02.261 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:10:02.261 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:10:02.261 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:06:41.228671Z", 2024-04-20T00:10:02.261 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:10:02.262 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:10:02.262 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:10:02.262 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:10:02.262 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:10:02.262 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:10:02.262 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:10:02.262 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:10:02.262 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:10:02.262 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:10:02.262 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:10:02.262 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:10:02.263 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:10:02.263 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:10:02.263 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:10:02.263 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:10:02.263 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:10:02.263 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:10:02.263 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:10:02.263 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:10:02.263 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:10:02.263 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:10:02Z" 2024-04-20T00:10:02.263 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:10:02.264 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713571601.228671_641.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:06:41.228671Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:10:02Z" } 2024-04-20T00:10:02.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571601.228671_641.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:41.228671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:10:02.265 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:10:02.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:10:02.265 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:10:02.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'aktlji-55', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:10:02.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'aktlji-55', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:10:02.265 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket aktlji-55 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:10:02.321 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:02.321 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:02.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'aktlji-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:10:02.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'aktlji-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:10:02.403 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket aktlji-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:10:02.461 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:02.461 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:02.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'aktlji-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:10:02.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'aktlji-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:10:02.541 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket aktlji-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:10:02.597 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:02.597 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:02.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:02.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:02.679 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:02.730 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:02.730 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:02.791 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:10:02.791 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:6", 2024-04-20T00:10:02.791 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.29", 2024-04-20T00:10:02.792 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000078", 2024-04-20T00:10:02.792 INFO:teuthology.orchestra.run.smithi019.stdout: "meta.history", 2024-04-20T00:10:02.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:3", 2024-04-20T00:10:02.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:5", 2024-04-20T00:10:02.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10", 2024-04-20T00:10:02.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:6", 2024-04-20T00:10:02.792 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.23", 2024-04-20T00:10:02.792 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000070", 2024-04-20T00:10:02.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:1", 2024-04-20T00:10:02.792 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000104", 2024-04-20T00:10:02.792 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:8", 2024-04-20T00:10:02.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:5", 2024-04-20T00:10:02.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:8", 2024-04-20T00:10:02.793 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000026", 2024-04-20T00:10:02.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23", 2024-04-20T00:10:02.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:1", 2024-04-20T00:10:02.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:0", 2024-04-20T00:10:02.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26", 2024-04-20T00:10:02.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:3", 2024-04-20T00:10:02.793 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.8", 2024-04-20T00:10:02.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:7", 2024-04-20T00:10:02.793 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14", 2024-04-20T00:10:02.794 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000028", 2024-04-20T00:10:02.794 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:6", 2024-04-20T00:10:02.794 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:8", 2024-04-20T00:10:02.794 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:6", 2024-04-20T00:10:02.794 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20", 2024-04-20T00:10:02.794 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:10", 2024-04-20T00:10:02.794 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:2", 2024-04-20T00:10:02.794 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:3", 2024-04-20T00:10:02.794 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:1", 2024-04-20T00:10:02.794 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.1", 2024-04-20T00:10:02.794 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:0", 2024-04-20T00:10:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:4", 2024-04-20T00:10:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:2", 2024-04-20T00:10:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.30", 2024-04-20T00:10:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13", 2024-04-20T00:10:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:2", 2024-04-20T00:10:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:6", 2024-04-20T00:10:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:3", 2024-04-20T00:10:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000040", 2024-04-20T00:10:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:2", 2024-04-20T00:10:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.0", 2024-04-20T00:10:02.795 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.6", 2024-04-20T00:10:02.796 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19", 2024-04-20T00:10:02.796 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:6", 2024-04-20T00:10:02.796 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:5", 2024-04-20T00:10:02.796 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:1", 2024-04-20T00:10:02.796 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:9", 2024-04-20T00:10:02.796 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000015", 2024-04-20T00:10:02.796 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:3", 2024-04-20T00:10:02.796 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:2", 2024-04-20T00:10:02.796 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:2", 2024-04-20T00:10:02.796 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:7", 2024-04-20T00:10:02.796 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:1", 2024-04-20T00:10:02.796 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000069", 2024-04-20T00:10:02.797 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:10", 2024-04-20T00:10:02.797 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000095", 2024-04-20T00:10:02.797 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000003", 2024-04-20T00:10:02.797 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29", 2024-04-20T00:10:02.797 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000047", 2024-04-20T00:10:02.797 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:0", 2024-04-20T00:10:02.797 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:9", 2024-04-20T00:10:02.797 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:4", 2024-04-20T00:10:02.797 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:4", 2024-04-20T00:10:02.797 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:1", 2024-04-20T00:10:02.797 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:10", 2024-04-20T00:10:02.798 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000035", 2024-04-20T00:10:02.798 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:2", 2024-04-20T00:10:02.798 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000037", 2024-04-20T00:10:02.798 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:4", 2024-04-20T00:10:02.798 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:4", 2024-04-20T00:10:02.798 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.0.retry", 2024-04-20T00:10:02.798 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:3", 2024-04-20T00:10:02.798 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:4", 2024-04-20T00:10:02.798 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:6", 2024-04-20T00:10:02.798 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:3", 2024-04-20T00:10:02.799 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:2", 2024-04-20T00:10:02.799 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000024", 2024-04-20T00:10:02.799 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:1", 2024-04-20T00:10:02.799 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:8", 2024-04-20T00:10:02.799 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:10", 2024-04-20T00:10:02.799 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000000", 2024-04-20T00:10:02.799 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:4", 2024-04-20T00:10:02.799 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:1", 2024-04-20T00:10:02.799 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000031", 2024-04-20T00:10:02.799 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:1", 2024-04-20T00:10:02.800 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:1", 2024-04-20T00:10:02.800 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:2", 2024-04-20T00:10:02.800 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:0", 2024-04-20T00:10:02.800 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:2", 2024-04-20T00:10:02.800 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.3", 2024-04-20T00:10:02.800 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:6", 2024-04-20T00:10:02.800 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:10", 2024-04-20T00:10:02.800 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:10", 2024-04-20T00:10:02.800 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:7", 2024-04-20T00:10:02.800 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:3", 2024-04-20T00:10:02.800 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:6", 2024-04-20T00:10:02.800 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:5", 2024-04-20T00:10:02.801 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:9", 2024-04-20T00:10:02.801 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:0", 2024-04-20T00:10:02.801 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:7", 2024-04-20T00:10:02.801 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:0", 2024-04-20T00:10:02.801 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:7", 2024-04-20T00:10:02.801 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:7", 2024-04-20T00:10:02.801 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:1", 2024-04-20T00:10:02.801 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:10", 2024-04-20T00:10:02.801 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:4", 2024-04-20T00:10:02.801 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:2", 2024-04-20T00:10:02.801 INFO:teuthology.orchestra.run.smithi019.stdout: "data_loggenerations_metadata", 2024-04-20T00:10:02.802 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:3", 2024-04-20T00:10:02.802 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:10", 2024-04-20T00:10:02.802 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12", 2024-04-20T00:10:02.802 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:5", 2024-04-20T00:10:02.802 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:8", 2024-04-20T00:10:02.802 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:6", 2024-04-20T00:10:02.802 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:3", 2024-04-20T00:10:02.802 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:3", 2024-04-20T00:10:02.802 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:1", 2024-04-20T00:10:02.802 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.2.7JqVP+4zZdZ.0", 2024-04-20T00:10:02.802 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000076", 2024-04-20T00:10:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:7", 2024-04-20T00:10:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000113", 2024-04-20T00:10:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22", 2024-04-20T00:10:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000005", 2024-04-20T00:10:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:5", 2024-04-20T00:10:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000011", 2024-04-20T00:10:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:10", 2024-04-20T00:10:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000058", 2024-04-20T00:10:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000052", 2024-04-20T00:10:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:4", 2024-04-20T00:10:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:9", 2024-04-20T00:10:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.21", 2024-04-20T00:10:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:1", 2024-04-20T00:10:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:7", 2024-04-20T00:10:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:0", 2024-04-20T00:10:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:0", 2024-04-20T00:10:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:9", 2024-04-20T00:10:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:3", 2024-04-20T00:10:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:10", 2024-04-20T00:10:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25", 2024-04-20T00:10:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46", 2024-04-20T00:10:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:3", 2024-04-20T00:10:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:4", 2024-04-20T00:10:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18", 2024-04-20T00:10:02.805 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:9", 2024-04-20T00:10:02.805 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:9", 2024-04-20T00:10:02.805 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000088", 2024-04-20T00:10:02.805 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000080", 2024-04-20T00:10:02.805 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:4", 2024-04-20T00:10:02.805 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:3", 2024-04-20T00:10:02.805 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000090", 2024-04-20T00:10:02.805 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22", 2024-04-20T00:10:02.805 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:1", 2024-04-20T00:10:02.805 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:4", 2024-04-20T00:10:02.805 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000110", 2024-04-20T00:10:02.806 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:9", 2024-04-20T00:10:02.806 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:5", 2024-04-20T00:10:02.806 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.0", 2024-04-20T00:10:02.806 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T00:10:02.806 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:4", 2024-04-20T00:10:02.806 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000096", 2024-04-20T00:10:02.806 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:1", 2024-04-20T00:10:02.806 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:6", 2024-04-20T00:10:02.806 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:1", 2024-04-20T00:10:02.806 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:2", 2024-04-20T00:10:02.806 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000087", 2024-04-20T00:10:02.807 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000008", 2024-04-20T00:10:02.807 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:0", 2024-04-20T00:10:02.807 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:6", 2024-04-20T00:10:02.807 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:7", 2024-04-20T00:10:02.807 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:3", 2024-04-20T00:10:02.807 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:2", 2024-04-20T00:10:02.807 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:8", 2024-04-20T00:10:02.807 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:9", 2024-04-20T00:10:02.807 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000006", 2024-04-20T00:10:02.807 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:1", 2024-04-20T00:10:02.807 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000029", 2024-04-20T00:10:02.808 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:7", 2024-04-20T00:10:02.808 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:6", 2024-04-20T00:10:02.808 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000089", 2024-04-20T00:10:02.808 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000022", 2024-04-20T00:10:02.808 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:7", 2024-04-20T00:10:02.808 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:7", 2024-04-20T00:10:02.808 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:6", 2024-04-20T00:10:02.808 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:2", 2024-04-20T00:10:02.808 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:10", 2024-04-20T00:10:02.808 INFO:teuthology.orchestra.run.smithi019.stdout: "bilog.trim", 2024-04-20T00:10:02.808 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:3", 2024-04-20T00:10:02.808 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:2", 2024-04-20T00:10:02.809 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000016", 2024-04-20T00:10:02.809 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:4", 2024-04-20T00:10:02.809 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:5", 2024-04-20T00:10:02.809 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:9", 2024-04-20T00:10:02.809 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000041", 2024-04-20T00:10:02.809 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10", 2024-04-20T00:10:02.809 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:8", 2024-04-20T00:10:02.809 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:7", 2024-04-20T00:10:02.809 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:1", 2024-04-20T00:10:02.809 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27", 2024-04-20T00:10:02.809 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25", 2024-04-20T00:10:02.810 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000018", 2024-04-20T00:10:02.810 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:6", 2024-04-20T00:10:02.810 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:6", 2024-04-20T00:10:02.810 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000092", 2024-04-20T00:10:02.810 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:9", 2024-04-20T00:10:02.810 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:6", 2024-04-20T00:10:02.810 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:2", 2024-04-20T00:10:02.810 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:8", 2024-04-20T00:10:02.810 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.27", 2024-04-20T00:10:02.810 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.4", 2024-04-20T00:10:02.810 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:8", 2024-04-20T00:10:02.811 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:3", 2024-04-20T00:10:02.811 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:8", 2024-04-20T00:10:02.811 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:4", 2024-04-20T00:10:02.811 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:8", 2024-04-20T00:10:02.811 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:6", 2024-04-20T00:10:02.811 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000014", 2024-04-20T00:10:02.811 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:4", 2024-04-20T00:10:02.811 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28", 2024-04-20T00:10:02.811 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000112", 2024-04-20T00:10:02.811 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:2", 2024-04-20T00:10:02.811 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:9", 2024-04-20T00:10:02.811 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000007", 2024-04-20T00:10:02.812 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000021", 2024-04-20T00:10:02.812 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:1", 2024-04-20T00:10:02.812 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:3", 2024-04-20T00:10:02.812 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:6", 2024-04-20T00:10:02.812 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.16", 2024-04-20T00:10:02.812 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000064", 2024-04-20T00:10:02.812 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16", 2024-04-20T00:10:02.812 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:8", 2024-04-20T00:10:02.812 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000071", 2024-04-20T00:10:02.812 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:10", 2024-04-20T00:10:02.812 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:3", 2024-04-20T00:10:02.813 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000074", 2024-04-20T00:10:02.813 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:5", 2024-04-20T00:10:02.813 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:8", 2024-04-20T00:10:02.813 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:6", 2024-04-20T00:10:02.813 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:8", 2024-04-20T00:10:02.813 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:6", 2024-04-20T00:10:02.813 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:7", 2024-04-20T00:10:02.813 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:8", 2024-04-20T00:10:02.813 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:10", 2024-04-20T00:10:02.813 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:7", 2024-04-20T00:10:02.813 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26", 2024-04-20T00:10:02.814 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:0", 2024-04-20T00:10:02.814 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000081", 2024-04-20T00:10:02.814 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.3.retry", 2024-04-20T00:10:02.814 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:5", 2024-04-20T00:10:02.814 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000009", 2024-04-20T00:10:02.814 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:6", 2024-04-20T00:10:02.814 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000121", 2024-04-20T00:10:02.814 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:3", 2024-04-20T00:10:02.814 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:3", 2024-04-20T00:10:02.814 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.1", 2024-04-20T00:10:02.815 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17", 2024-04-20T00:10:02.815 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:4", 2024-04-20T00:10:02.815 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:8", 2024-04-20T00:10:02.815 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.2", 2024-04-20T00:10:02.815 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:8", 2024-04-20T00:10:02.815 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:6", 2024-04-20T00:10:02.815 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.17", 2024-04-20T00:10:02.815 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000125", 2024-04-20T00:10:02.815 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000082", 2024-04-20T00:10:02.815 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:7", 2024-04-20T00:10:02.815 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:2", 2024-04-20T00:10:02.816 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:0", 2024-04-20T00:10:02.816 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:6", 2024-04-20T00:10:02.816 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:6", 2024-04-20T00:10:02.816 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24", 2024-04-20T00:10:02.816 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:6", 2024-04-20T00:10:02.816 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:3", 2024-04-20T00:10:02.816 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000105", 2024-04-20T00:10:02.816 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000059", 2024-04-20T00:10:02.816 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:8", 2024-04-20T00:10:02.816 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:10", 2024-04-20T00:10:02.816 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:7", 2024-04-20T00:10:02.817 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000077", 2024-04-20T00:10:02.817 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:8", 2024-04-20T00:10:02.817 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000032", 2024-04-20T00:10:02.817 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:8", 2024-04-20T00:10:02.817 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:2", 2024-04-20T00:10:02.817 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:7", 2024-04-20T00:10:02.817 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:1", 2024-04-20T00:10:02.818 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:7", 2024-04-20T00:10:02.818 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:9", 2024-04-20T00:10:02.818 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:0", 2024-04-20T00:10:02.818 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:3", 2024-04-20T00:10:02.818 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.24", 2024-04-20T00:10:02.818 INFO:teuthology.orchestra.run.smithi019.stdout: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.2", 2024-04-20T00:10:02.818 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:10", 2024-04-20T00:10:02.818 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:1", 2024-04-20T00:10:02.818 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:4", 2024-04-20T00:10:02.818 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:3", 2024-04-20T00:10:02.818 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:5", 2024-04-20T00:10:02.819 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:4", 2024-04-20T00:10:02.819 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:1", 2024-04-20T00:10:02.819 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:7", 2024-04-20T00:10:02.819 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000053", 2024-04-20T00:10:02.819 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:1", 2024-04-20T00:10:02.819 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:9", 2024-04-20T00:10:02.819 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:7", 2024-04-20T00:10:02.819 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:6", 2024-04-20T00:10:02.819 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27", 2024-04-20T00:10:02.819 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:7", 2024-04-20T00:10:02.819 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000091", 2024-04-20T00:10:02.820 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:9", 2024-04-20T00:10:02.820 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:3", 2024-04-20T00:10:02.820 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:9", 2024-04-20T00:10:02.820 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:10", 2024-04-20T00:10:02.820 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:2", 2024-04-20T00:10:02.820 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000065", 2024-04-20T00:10:02.820 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:4", 2024-04-20T00:10:02.820 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.1.retry", 2024-04-20T00:10:02.820 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000083", 2024-04-20T00:10:02.820 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000010", 2024-04-20T00:10:02.820 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:9", 2024-04-20T00:10:02.821 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:3", 2024-04-20T00:10:02.821 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:9", 2024-04-20T00:10:02.821 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:8", 2024-04-20T00:10:02.821 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:5", 2024-04-20T00:10:02.821 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:0", 2024-04-20T00:10:02.821 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:10", 2024-04-20T00:10:02.821 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:3", 2024-04-20T00:10:02.821 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:0", 2024-04-20T00:10:02.821 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.7", 2024-04-20T00:10:02.821 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.3", 2024-04-20T00:10:02.821 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.3.retry", 2024-04-20T00:10:02.822 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:0", 2024-04-20T00:10:02.822 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:3", 2024-04-20T00:10:02.822 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:3", 2024-04-20T00:10:02.822 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:6", 2024-04-20T00:10:02.822 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:6", 2024-04-20T00:10:02.822 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.1.e8rVRID2eg2.0", 2024-04-20T00:10:02.822 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:0", 2024-04-20T00:10:02.822 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:8", 2024-04-20T00:10:02.822 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:3", 2024-04-20T00:10:02.822 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:2", 2024-04-20T00:10:02.823 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:1", 2024-04-20T00:10:02.823 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:6", 2024-04-20T00:10:02.823 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:0", 2024-04-20T00:10:02.823 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:0", 2024-04-20T00:10:02.823 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:5", 2024-04-20T00:10:02.823 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:4", 2024-04-20T00:10:02.823 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:5", 2024-04-20T00:10:02.823 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000045", 2024-04-20T00:10:02.823 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11", 2024-04-20T00:10:02.823 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000002", 2024-04-20T00:10:02.823 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:6", 2024-04-20T00:10:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.12", 2024-04-20T00:10:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:4", 2024-04-20T00:10:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:6", 2024-04-20T00:10:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:1", 2024-04-20T00:10:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:8", 2024-04-20T00:10:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000116", 2024-04-20T00:10:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:4", 2024-04-20T00:10:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:6", 2024-04-20T00:10:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.3", 2024-04-20T00:10:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:6", 2024-04-20T00:10:02.824 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.25", 2024-04-20T00:10:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:0", 2024-04-20T00:10:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000034", 2024-04-20T00:10:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:9", 2024-04-20T00:10:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:1", 2024-04-20T00:10:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:10", 2024-04-20T00:10:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:4", 2024-04-20T00:10:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:4", 2024-04-20T00:10:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:0", 2024-04-20T00:10:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:1", 2024-04-20T00:10:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000101", 2024-04-20T00:10:02.825 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:2", 2024-04-20T00:10:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:10", 2024-04-20T00:10:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:7", 2024-04-20T00:10:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:0", 2024-04-20T00:10:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45", 2024-04-20T00:10:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.13", 2024-04-20T00:10:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:1", 2024-04-20T00:10:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:10", 2024-04-20T00:10:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:2", 2024-04-20T00:10:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:7", 2024-04-20T00:10:02.826 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:3", 2024-04-20T00:10:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:8", 2024-04-20T00:10:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000079", 2024-04-20T00:10:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:0", 2024-04-20T00:10:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000049", 2024-04-20T00:10:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000117", 2024-04-20T00:10:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000044", 2024-04-20T00:10:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:5", 2024-04-20T00:10:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:7", 2024-04-20T00:10:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:10", 2024-04-20T00:10:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:9", 2024-04-20T00:10:02.827 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:5", 2024-04-20T00:10:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:4", 2024-04-20T00:10:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:5", 2024-04-20T00:10:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:8", 2024-04-20T00:10:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.14", 2024-04-20T00:10:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000066", 2024-04-20T00:10:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:10", 2024-04-20T00:10:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:0", 2024-04-20T00:10:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:1", 2024-04-20T00:10:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:7", 2024-04-20T00:10:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:2", 2024-04-20T00:10:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:4", 2024-04-20T00:10:02.828 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:2", 2024-04-20T00:10:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000068", 2024-04-20T00:10:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:8", 2024-04-20T00:10:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000085", 2024-04-20T00:10:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:9", 2024-04-20T00:10:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:10", 2024-04-20T00:10:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:3", 2024-04-20T00:10:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:3", 2024-04-20T00:10:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:5", 2024-04-20T00:10:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:1", 2024-04-20T00:10:02.829 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:9", 2024-04-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:8", 2024-04-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:7", 2024-04-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:5", 2024-04-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000073", 2024-04-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000038", 2024-04-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20", 2024-04-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19", 2024-04-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:8", 2024-04-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:1", 2024-04-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45", 2024-04-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:9", 2024-04-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:9", 2024-04-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.10", 2024-04-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000118", 2024-04-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:8", 2024-04-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:5", 2024-04-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:2", 2024-04-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:5", 2024-04-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000036", 2024-04-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:6", 2024-04-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:9", 2024-04-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:0", 2024-04-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:8", 2024-04-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:10", 2024-04-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000103", 2024-04-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:5", 2024-04-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:10", 2024-04-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23", 2024-04-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:0", 2024-04-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:5", 2024-04-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:10", 2024-04-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000119", 2024-04-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:4", 2024-04-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:5", 2024-04-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:7", 2024-04-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000098", 2024-04-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.0", 2024-04-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:5", 2024-04-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:4", 2024-04-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11", 2024-04-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.2.retry", 2024-04-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:4", 2024-04-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:5", 2024-04-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:4", 2024-04-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:5", 2024-04-20T00:10:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:2", 2024-04-20T00:10:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:2", 2024-04-20T00:10:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000027", 2024-04-20T00:10:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47", 2024-04-20T00:10:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:3", 2024-04-20T00:10:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:0", 2024-04-20T00:10:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:7", 2024-04-20T00:10:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000019", 2024-04-20T00:10:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:1", 2024-04-20T00:10:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:9", 2024-04-20T00:10:02.834 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000039", 2024-04-20T00:10:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:1", 2024-04-20T00:10:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:3", 2024-04-20T00:10:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:0", 2024-04-20T00:10:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:1", 2024-04-20T00:10:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:10", 2024-04-20T00:10:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.2.retry", 2024-04-20T00:10:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:1", 2024-04-20T00:10:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:8", 2024-04-20T00:10:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:0", 2024-04-20T00:10:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:10", 2024-04-20T00:10:02.835 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000100", 2024-04-20T00:10:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:0", 2024-04-20T00:10:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:8", 2024-04-20T00:10:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:10", 2024-04-20T00:10:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000093", 2024-04-20T00:10:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:5", 2024-04-20T00:10:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:6", 2024-04-20T00:10:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:8", 2024-04-20T00:10:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:5", 2024-04-20T00:10:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:10", 2024-04-20T00:10:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:8", 2024-04-20T00:10:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000004", 2024-04-20T00:10:02.836 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000063", 2024-04-20T00:10:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000122", 2024-04-20T00:10:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000057", 2024-04-20T00:10:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:1", 2024-04-20T00:10:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:8", 2024-04-20T00:10:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:3", 2024-04-20T00:10:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000054", 2024-04-20T00:10:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:3", 2024-04-20T00:10:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000114", 2024-04-20T00:10:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:0", 2024-04-20T00:10:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.2", 2024-04-20T00:10:02.837 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:5", 2024-04-20T00:10:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.0", 2024-04-20T00:10:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:0", 2024-04-20T00:10:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:4", 2024-04-20T00:10:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:3", 2024-04-20T00:10:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000012", 2024-04-20T00:10:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:7", 2024-04-20T00:10:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.3", 2024-04-20T00:10:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:9", 2024-04-20T00:10:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:0", 2024-04-20T00:10:02.838 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:6", 2024-04-20T00:10:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14", 2024-04-20T00:10:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:10", 2024-04-20T00:10:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000084", 2024-04-20T00:10:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000043", 2024-04-20T00:10:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:7", 2024-04-20T00:10:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:7", 2024-04-20T00:10:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.11", 2024-04-20T00:10:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:1", 2024-04-20T00:10:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:5", 2024-04-20T00:10:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:0", 2024-04-20T00:10:02.839 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.0.nl0urTltrWn.0", 2024-04-20T00:10:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:0", 2024-04-20T00:10:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:2", 2024-04-20T00:10:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:9", 2024-04-20T00:10:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000111", 2024-04-20T00:10:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:6", 2024-04-20T00:10:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000048", 2024-04-20T00:10:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46", 2024-04-20T00:10:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000020", 2024-04-20T00:10:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:9", 2024-04-20T00:10:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:2", 2024-04-20T00:10:02.840 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:2", 2024-04-20T00:10:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18", 2024-04-20T00:10:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:5", 2024-04-20T00:10:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:3", 2024-04-20T00:10:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000099", 2024-04-20T00:10:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000056", 2024-04-20T00:10:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:0", 2024-04-20T00:10:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:1", 2024-04-20T00:10:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:4", 2024-04-20T00:10:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:3", 2024-04-20T00:10:02.841 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:10", 2024-04-20T00:10:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.19", 2024-04-20T00:10:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:9", 2024-04-20T00:10:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.28", 2024-04-20T00:10:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:9", 2024-04-20T00:10:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.0.retry", 2024-04-20T00:10:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000072", 2024-04-20T00:10:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:7", 2024-04-20T00:10:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:2", 2024-04-20T00:10:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:0", 2024-04-20T00:10:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:2", 2024-04-20T00:10:02.842 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15", 2024-04-20T00:10:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:10", 2024-04-20T00:10:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:2", 2024-04-20T00:10:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000062", 2024-04-20T00:10:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.2", 2024-04-20T00:10:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000115", 2024-04-20T00:10:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:9", 2024-04-20T00:10:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.1", 2024-04-20T00:10:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:1", 2024-04-20T00:10:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:8", 2024-04-20T00:10:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000033", 2024-04-20T00:10:02.843 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:1", 2024-04-20T00:10:02.844 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:9", 2024-04-20T00:10:02.844 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:5", 2024-04-20T00:10:02.844 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:9", 2024-04-20T00:10:02.844 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:7", 2024-04-20T00:10:02.844 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:7", 2024-04-20T00:10:02.844 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:6", 2024-04-20T00:10:02.844 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:10", 2024-04-20T00:10:02.844 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1", 2024-04-20T00:10:02.844 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:10", 2024-04-20T00:10:02.844 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17", 2024-04-20T00:10:02.844 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:6", 2024-04-20T00:10:02.845 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:3", 2024-04-20T00:10:02.845 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000025", 2024-04-20T00:10:02.845 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000123", 2024-04-20T00:10:02.845 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:2", 2024-04-20T00:10:02.845 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.31", 2024-04-20T00:10:02.845 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:9", 2024-04-20T00:10:02.845 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:0", 2024-04-20T00:10:02.845 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.18", 2024-04-20T00:10:02.845 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:5", 2024-04-20T00:10:02.845 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:7", 2024-04-20T00:10:02.845 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000108", 2024-04-20T00:10:02.846 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000094", 2024-04-20T00:10:02.846 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:4", 2024-04-20T00:10:02.846 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:6", 2024-04-20T00:10:02.846 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000060", 2024-04-20T00:10:02.846 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:2", 2024-04-20T00:10:02.846 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:5", 2024-04-20T00:10:02.846 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:0", 2024-04-20T00:10:02.846 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13", 2024-04-20T00:10:02.846 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:0", 2024-04-20T00:10:02.846 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000109", 2024-04-20T00:10:02.846 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.15", 2024-04-20T00:10:02.847 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:10", 2024-04-20T00:10:02.847 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:7", 2024-04-20T00:10:02.847 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000013", 2024-04-20T00:10:02.847 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:9", 2024-04-20T00:10:02.847 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:5", 2024-04-20T00:10:02.847 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28", 2024-04-20T00:10:02.847 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:5", 2024-04-20T00:10:02.847 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:2", 2024-04-20T00:10:02.847 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.3", 2024-04-20T00:10:02.847 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.0", 2024-04-20T00:10:02.847 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:5", 2024-04-20T00:10:02.848 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000051", 2024-04-20T00:10:02.848 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:8", 2024-04-20T00:10:02.848 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:3", 2024-04-20T00:10:02.848 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:8", 2024-04-20T00:10:02.848 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000106", 2024-04-20T00:10:02.848 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:10", 2024-04-20T00:10:02.848 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:5", 2024-04-20T00:10:02.848 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:8", 2024-04-20T00:10:02.848 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:9", 2024-04-20T00:10:02.848 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000017", 2024-04-20T00:10:02.848 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:9", 2024-04-20T00:10:02.849 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16", 2024-04-20T00:10:02.849 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:4", 2024-04-20T00:10:02.849 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:9", 2024-04-20T00:10:02.849 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:0", 2024-04-20T00:10:02.849 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:5", 2024-04-20T00:10:02.849 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:10", 2024-04-20T00:10:02.849 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:7", 2024-04-20T00:10:02.849 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000023", 2024-04-20T00:10:02.849 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15", 2024-04-20T00:10:02.849 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:6", 2024-04-20T00:10:02.849 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:9", 2024-04-20T00:10:02.850 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:8", 2024-04-20T00:10:02.850 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:5", 2024-04-20T00:10:02.850 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.1.retry", 2024-04-20T00:10:02.850 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000107", 2024-04-20T00:10:02.850 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:8", 2024-04-20T00:10:02.850 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:4", 2024-04-20T00:10:02.850 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000046", 2024-04-20T00:10:02.850 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:0", 2024-04-20T00:10:02.850 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:1", 2024-04-20T00:10:02.850 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.26", 2024-04-20T00:10:02.850 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:8", 2024-04-20T00:10:02.851 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:4", 2024-04-20T00:10:02.851 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47", 2024-04-20T00:10:02.851 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:2", 2024-04-20T00:10:02.851 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.22", 2024-04-20T00:10:02.851 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:0", 2024-04-20T00:10:02.851 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:0", 2024-04-20T00:10:02.851 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.1", 2024-04-20T00:10:02.851 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:9", 2024-04-20T00:10:02.851 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:4", 2024-04-20T00:10:02.851 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.2", 2024-04-20T00:10:02.851 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:7", 2024-04-20T00:10:02.852 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.3.Ul7Ylj3XnqY.0", 2024-04-20T00:10:02.852 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000067", 2024-04-20T00:10:02.852 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:7", 2024-04-20T00:10:02.852 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:8", 2024-04-20T00:10:02.852 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:7", 2024-04-20T00:10:02.852 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:4", 2024-04-20T00:10:02.852 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:7", 2024-04-20T00:10:02.852 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29", 2024-04-20T00:10:02.852 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:8", 2024-04-20T00:10:02.852 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:2", 2024-04-20T00:10:02.852 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:8", 2024-04-20T00:10:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:2", 2024-04-20T00:10:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:5", 2024-04-20T00:10:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:4", 2024-04-20T00:10:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:0", 2024-04-20T00:10:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:2", 2024-04-20T00:10:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.9", 2024-04-20T00:10:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000086", 2024-04-20T00:10:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:8", 2024-04-20T00:10:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21", 2024-04-20T00:10:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.5", 2024-04-20T00:10:02.853 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21", 2024-04-20T00:10:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:4", 2024-04-20T00:10:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:1", 2024-04-20T00:10:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:9", 2024-04-20T00:10:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:0", 2024-04-20T00:10:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:6", 2024-04-20T00:10:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.20", 2024-04-20T00:10:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24", 2024-04-20T00:10:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:1", 2024-04-20T00:10:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:10", 2024-04-20T00:10:02.854 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:9", 2024-04-20T00:10:02.855 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:7", 2024-04-20T00:10:02.855 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:10", 2024-04-20T00:10:02.855 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:5", 2024-04-20T00:10:02.855 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:1", 2024-04-20T00:10:02.855 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:2", 2024-04-20T00:10:02.855 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:5", 2024-04-20T00:10:02.855 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:2", 2024-04-20T00:10:02.855 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000050", 2024-04-20T00:10:02.855 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000102", 2024-04-20T00:10:02.855 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:4", 2024-04-20T00:10:02.855 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:3", 2024-04-20T00:10:02.856 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:9", 2024-04-20T00:10:02.856 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:6", 2024-04-20T00:10:02.856 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:10", 2024-04-20T00:10:02.856 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12", 2024-04-20T00:10:02.856 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:4", 2024-04-20T00:10:02.856 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:9", 2024-04-20T00:10:02.856 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:4", 2024-04-20T00:10:02.856 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:1", 2024-04-20T00:10:02.856 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:0", 2024-04-20T00:10:02.856 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:5", 2024-04-20T00:10:02.856 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000124", 2024-04-20T00:10:02.856 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:10", 2024-04-20T00:10:02.857 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:5", 2024-04-20T00:10:02.857 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:3", 2024-04-20T00:10:02.857 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:9", 2024-04-20T00:10:02.857 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:2", 2024-04-20T00:10:02.857 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:6", 2024-04-20T00:10:02.857 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000001", 2024-04-20T00:10:02.857 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:1", 2024-04-20T00:10:02.857 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000126", 2024-04-20T00:10:02.857 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:8", 2024-04-20T00:10:02.857 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:5", 2024-04-20T00:10:02.857 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:7", 2024-04-20T00:10:02.858 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:3", 2024-04-20T00:10:02.858 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000055", 2024-04-20T00:10:02.858 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:10", 2024-04-20T00:10:02.858 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:7", 2024-04-20T00:10:02.858 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000061", 2024-04-20T00:10:02.858 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000097", 2024-04-20T00:10:02.858 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:4", 2024-04-20T00:10:02.858 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:10", 2024-04-20T00:10:02.858 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:4", 2024-04-20T00:10:02.858 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000042", 2024-04-20T00:10:02.858 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:2", 2024-04-20T00:10:02.859 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:1", 2024-04-20T00:10:02.859 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:9", 2024-04-20T00:10:02.859 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T00:10:02.859 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:7", 2024-04-20T00:10:02.859 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:10", 2024-04-20T00:10:02.859 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:2", 2024-04-20T00:10:02.859 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:6", 2024-04-20T00:10:02.859 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:7", 2024-04-20T00:10:02.859 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:6", 2024-04-20T00:10:02.859 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000075", 2024-04-20T00:10:02.859 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:2", 2024-04-20T00:10:02.860 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:3", 2024-04-20T00:10:02.860 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000120", 2024-04-20T00:10:02.860 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000030", 2024-04-20T00:10:02.860 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:10", 2024-04-20T00:10:02.860 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:3", 2024-04-20T00:10:02.860 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:0", 2024-04-20T00:10:02.860 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:4", 2024-04-20T00:10:02.860 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:10" 2024-04-20T00:10:02.860 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:10:02.860 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-20T00:10:02.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:02.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:02.862 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:02.918 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:02.918 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:03.008 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-04-20T00:10:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:6", 2024-04-20T00:10:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.29", 2024-04-20T00:10:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000078", 2024-04-20T00:10:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "meta.history", 2024-04-20T00:10:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:3", 2024-04-20T00:10:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:5", 2024-04-20T00:10:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10", 2024-04-20T00:10:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:9", 2024-04-20T00:10:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:6", 2024-04-20T00:10:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.23", 2024-04-20T00:10:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000070", 2024-04-20T00:10:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:1", 2024-04-20T00:10:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:1", 2024-04-20T00:10:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000104", 2024-04-20T00:10:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:8", 2024-04-20T00:10:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:9", 2024-04-20T00:10:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000026", 2024-04-20T00:10:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23", 2024-04-20T00:10:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:1", 2024-04-20T00:10:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:0", 2024-04-20T00:10:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.8", 2024-04-20T00:10:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:7", 2024-04-20T00:10:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000028", 2024-04-20T00:10:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:8", 2024-04-20T00:10:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:8", 2024-04-20T00:10:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:6", 2024-04-20T00:10:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:3", 2024-04-20T00:10:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:6", 2024-04-20T00:10:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:10", 2024-04-20T00:10:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19", 2024-04-20T00:10:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:4", 2024-04-20T00:10:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:2", 2024-04-20T00:10:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.0.retry", 2024-04-20T00:10:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:3", 2024-04-20T00:10:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.1", 2024-04-20T00:10:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.30", 2024-04-20T00:10:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:10:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13", 2024-04-20T00:10:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:6", 2024-04-20T00:10:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:7", 2024-04-20T00:10:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000040", 2024-04-20T00:10:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:7", 2024-04-20T00:10:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:2", 2024-04-20T00:10:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.0", 2024-04-20T00:10:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.6", 2024-04-20T00:10:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:1", 2024-04-20T00:10:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:8", 2024-04-20T00:10:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19", 2024-04-20T00:10:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:6", 2024-04-20T00:10:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:1", 2024-04-20T00:10:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:6", 2024-04-20T00:10:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:8", 2024-04-20T00:10:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:2", 2024-04-20T00:10:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000015", 2024-04-20T00:10:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:3", 2024-04-20T00:10:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000069", 2024-04-20T00:10:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000095", 2024-04-20T00:10:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:0", 2024-04-20T00:10:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000003", 2024-04-20T00:10:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29", 2024-04-20T00:10:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000047", 2024-04-20T00:10:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:0", 2024-04-20T00:10:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:9", 2024-04-20T00:10:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:5", 2024-04-20T00:10:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:9", 2024-04-20T00:10:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:10", 2024-04-20T00:10:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000035", 2024-04-20T00:10:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:8", 2024-04-20T00:10:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000037", 2024-04-20T00:10:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:4", 2024-04-20T00:10:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:3", 2024-04-20T00:10:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:1", 2024-04-20T00:10:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:9", 2024-04-20T00:10:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:5", 2024-04-20T00:10:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:3", 2024-04-20T00:10:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "data_log.2.DWMttCqmkiM.0", 2024-04-20T00:10:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:2", 2024-04-20T00:10:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:7", 2024-04-20T00:10:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:1", 2024-04-20T00:10:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:1", 2024-04-20T00:10:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000024", 2024-04-20T00:10:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:2", 2024-04-20T00:10:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:5", 2024-04-20T00:10:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:1", 2024-04-20T00:10:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:10", 2024-04-20T00:10:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:10", 2024-04-20T00:10:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000000", 2024-04-20T00:10:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:1", 2024-04-20T00:10:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000031", 2024-04-20T00:10:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:0", 2024-04-20T00:10:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "data_log.3", 2024-04-20T00:10:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:6", 2024-04-20T00:10:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:7", 2024-04-20T00:10:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:4", 2024-04-20T00:10:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:6", 2024-04-20T00:10:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:5", 2024-04-20T00:10:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:4", 2024-04-20T00:10:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:5", 2024-04-20T00:10:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:9", 2024-04-20T00:10:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:5", 2024-04-20T00:10:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:3", 2024-04-20T00:10:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:0", 2024-04-20T00:10:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:7", 2024-04-20T00:10:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:0", 2024-04-20T00:10:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:7", 2024-04-20T00:10:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:10", 2024-04-20T00:10:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:3", 2024-04-20T00:10:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:2", 2024-04-20T00:10:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "data_loggenerations_metadata", 2024-04-20T00:10:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:6", 2024-04-20T00:10:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:6", 2024-04-20T00:10:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12", 2024-04-20T00:10:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:6", 2024-04-20T00:10:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:9", 2024-04-20T00:10:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:1", 2024-04-20T00:10:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000076", 2024-04-20T00:10:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:4", 2024-04-20T00:10:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000113", 2024-04-20T00:10:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22", 2024-04-20T00:10:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000005", 2024-04-20T00:10:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000011", 2024-04-20T00:10:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:10", 2024-04-20T00:10:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000058", 2024-04-20T00:10:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000052", 2024-04-20T00:10:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.21", 2024-04-20T00:10:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:9", 2024-04-20T00:10:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:3", 2024-04-20T00:10:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:1", 2024-04-20T00:10:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:9", 2024-04-20T00:10:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:8", 2024-04-20T00:10:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:2", 2024-04-20T00:10:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:7", 2024-04-20T00:10:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:0", 2024-04-20T00:10:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:0", 2024-04-20T00:10:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:9", 2024-04-20T00:10:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:8", 2024-04-20T00:10:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27", 2024-04-20T00:10:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25", 2024-04-20T00:10:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:4", 2024-04-20T00:10:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:8", 2024-04-20T00:10:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:6", 2024-04-20T00:10:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:8", 2024-04-20T00:10:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:9", 2024-04-20T00:10:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:9", 2024-04-20T00:10:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000088", 2024-04-20T00:10:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000080", 2024-04-20T00:10:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:4", 2024-04-20T00:10:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000090", 2024-04-20T00:10:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:5", 2024-04-20T00:10:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:7", 2024-04-20T00:10:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000110", 2024-04-20T00:10:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:9", 2024-04-20T00:10:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:9", 2024-04-20T00:10:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:0", 2024-04-20T00:10:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.0", 2024-04-20T00:10:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:4", 2024-04-20T00:10:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:4", 2024-04-20T00:10:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000096", 2024-04-20T00:10:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:6", 2024-04-20T00:10:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:1", 2024-04-20T00:10:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:2", 2024-04-20T00:10:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000087", 2024-04-20T00:10:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000008", 2024-04-20T00:10:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:0", 2024-04-20T00:10:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:3", 2024-04-20T00:10:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:7", 2024-04-20T00:10:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:3", 2024-04-20T00:10:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:3", 2024-04-20T00:10:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:8", 2024-04-20T00:10:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:9", 2024-04-20T00:10:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:1", 2024-04-20T00:10:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000006", 2024-04-20T00:10:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29", 2024-04-20T00:10:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000029", 2024-04-20T00:10:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:7", 2024-04-20T00:10:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:9", 2024-04-20T00:10:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:10", 2024-04-20T00:10:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000089", 2024-04-20T00:10:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000022", 2024-04-20T00:10:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.0", 2024-04-20T00:10:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:9", 2024-04-20T00:10:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:7", 2024-04-20T00:10:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:7", 2024-04-20T00:10:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "bilog.trim", 2024-04-20T00:10:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:6", 2024-04-20T00:10:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:2", 2024-04-20T00:10:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13", 2024-04-20T00:10:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000016", 2024-04-20T00:10:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:6", 2024-04-20T00:10:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:8", 2024-04-20T00:10:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:2", 2024-04-20T00:10:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000041", 2024-04-20T00:10:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:10", 2024-04-20T00:10:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:1", 2024-04-20T00:10:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:8", 2024-04-20T00:10:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000018", 2024-04-20T00:10:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:6", 2024-04-20T00:10:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000092", 2024-04-20T00:10:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:8", 2024-04-20T00:10:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:0", 2024-04-20T00:10:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:2", 2024-04-20T00:10:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:10", 2024-04-20T00:10:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:5", 2024-04-20T00:10:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.27", 2024-04-20T00:10:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.4", 2024-04-20T00:10:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:3", 2024-04-20T00:10:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:0", 2024-04-20T00:10:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:6", 2024-04-20T00:10:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:8", 2024-04-20T00:10:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000014", 2024-04-20T00:10:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:4", 2024-04-20T00:10:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:2", 2024-04-20T00:10:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000112", 2024-04-20T00:10:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:9", 2024-04-20T00:10:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000007", 2024-04-20T00:10:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000021", 2024-04-20T00:10:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:1", 2024-04-20T00:10:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:1", 2024-04-20T00:10:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:3", 2024-04-20T00:10:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.16", 2024-04-20T00:10:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:0", 2024-04-20T00:10:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000064", 2024-04-20T00:10:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:0", 2024-04-20T00:10:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000071", 2024-04-20T00:10:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:1", 2024-04-20T00:10:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:10", 2024-04-20T00:10:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:3", 2024-04-20T00:10:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000074", 2024-04-20T00:10:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:5", 2024-04-20T00:10:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:8", 2024-04-20T00:10:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:8", 2024-04-20T00:10:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:6", 2024-04-20T00:10:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:7", 2024-04-20T00:10:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:8", 2024-04-20T00:10:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26", 2024-04-20T00:10:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:1", 2024-04-20T00:10:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:6", 2024-04-20T00:10:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000081", 2024-04-20T00:10:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.3.retry", 2024-04-20T00:10:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000009", 2024-04-20T00:10:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:6", 2024-04-20T00:10:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000121", 2024-04-20T00:10:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:3", 2024-04-20T00:10:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "data_log.1", 2024-04-20T00:10:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17", 2024-04-20T00:10:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:9", 2024-04-20T00:10:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:4", 2024-04-20T00:10:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:6", 2024-04-20T00:10:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:8", 2024-04-20T00:10:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:2", 2024-04-20T00:10:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.2", 2024-04-20T00:10:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:6", 2024-04-20T00:10:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.17", 2024-04-20T00:10:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000125", 2024-04-20T00:10:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000082", 2024-04-20T00:10:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:2", 2024-04-20T00:10:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:8", 2024-04-20T00:10:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24", 2024-04-20T00:10:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:6", 2024-04-20T00:10:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:3", 2024-04-20T00:10:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000105", 2024-04-20T00:10:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000059", 2024-04-20T00:10:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:1", 2024-04-20T00:10:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:7", 2024-04-20T00:10:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:2", 2024-04-20T00:10:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:5", 2024-04-20T00:10:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000077", 2024-04-20T00:10:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:10", 2024-04-20T00:10:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000032", 2024-04-20T00:10:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:7", 2024-04-20T00:10:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:3", 2024-04-20T00:10:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:1", 2024-04-20T00:10:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:5", 2024-04-20T00:10:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:7", 2024-04-20T00:10:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:9", 2024-04-20T00:10:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:0", 2024-04-20T00:10:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.24", 2024-04-20T00:10:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:10", 2024-04-20T00:10:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.2", 2024-04-20T00:10:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:10", 2024-04-20T00:10:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:4", 2024-04-20T00:10:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:3", 2024-04-20T00:10:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:4", 2024-04-20T00:10:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:4", 2024-04-20T00:10:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:0", 2024-04-20T00:10:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:7", 2024-04-20T00:10:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000053", 2024-04-20T00:10:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:4", 2024-04-20T00:10:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:1", 2024-04-20T00:10:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:9", 2024-04-20T00:10:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27", 2024-04-20T00:10:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000091", 2024-04-20T00:10:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46", 2024-04-20T00:10:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:3", 2024-04-20T00:10:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:2", 2024-04-20T00:10:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:9", 2024-04-20T00:10:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000065", 2024-04-20T00:10:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.1.retry", 2024-04-20T00:10:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000083", 2024-04-20T00:10:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000010", 2024-04-20T00:10:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:8", 2024-04-20T00:10:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:3", 2024-04-20T00:10:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.7", 2024-04-20T00:10:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25", 2024-04-20T00:10:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.3", 2024-04-20T00:10:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:9", 2024-04-20T00:10:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:3", 2024-04-20T00:10:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:8", 2024-04-20T00:10:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:1", 2024-04-20T00:10:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:6", 2024-04-20T00:10:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:6", 2024-04-20T00:10:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:4", 2024-04-20T00:10:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:1", 2024-04-20T00:10:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:0", 2024-04-20T00:10:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:8", 2024-04-20T00:10:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:9", 2024-04-20T00:10:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:2", 2024-04-20T00:10:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:9", 2024-04-20T00:10:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:0", 2024-04-20T00:10:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:5", 2024-04-20T00:10:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "mdlog.sync-status.shard.2", 2024-04-20T00:10:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:10", 2024-04-20T00:10:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:4", 2024-04-20T00:10:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11", 2024-04-20T00:10:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000045", 2024-04-20T00:10:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:10", 2024-04-20T00:10:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000002", 2024-04-20T00:10:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:6", 2024-04-20T00:10:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.12", 2024-04-20T00:10:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:4", 2024-04-20T00:10:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:7", 2024-04-20T00:10:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:2", 2024-04-20T00:10:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:1", 2024-04-20T00:10:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000116", 2024-04-20T00:10:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:4", 2024-04-20T00:10:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:8", 2024-04-20T00:10:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.3", 2024-04-20T00:10:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:6", 2024-04-20T00:10:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.25", 2024-04-20T00:10:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000034", 2024-04-20T00:10:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:0", 2024-04-20T00:10:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:4", 2024-04-20T00:10:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:4", 2024-04-20T00:10:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:7", 2024-04-20T00:10:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:1", 2024-04-20T00:10:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000101", 2024-04-20T00:10:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:2", 2024-04-20T00:10:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:10", 2024-04-20T00:10:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:7", 2024-04-20T00:10:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45", 2024-04-20T00:10:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:0", 2024-04-20T00:10:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.13", 2024-04-20T00:10:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "mdlog.sync-status.shard.1", 2024-04-20T00:10:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17", 2024-04-20T00:10:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:3", 2024-04-20T00:10:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:3", 2024-04-20T00:10:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000079", 2024-04-20T00:10:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000049", 2024-04-20T00:10:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000117", 2024-04-20T00:10:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000044", 2024-04-20T00:10:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24", 2024-04-20T00:10:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:1", 2024-04-20T00:10:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:10", 2024-04-20T00:10:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:1", 2024-04-20T00:10:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:9", 2024-04-20T00:10:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:5", 2024-04-20T00:10:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:4", 2024-04-20T00:10:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:5", 2024-04-20T00:10:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:8", 2024-04-20T00:10:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.14", 2024-04-20T00:10:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000066", 2024-04-20T00:10:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:10", 2024-04-20T00:10:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:7", 2024-04-20T00:10:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:1", 2024-04-20T00:10:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:7", 2024-04-20T00:10:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000068", 2024-04-20T00:10:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:8", 2024-04-20T00:10:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000085", 2024-04-20T00:10:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:5", 2024-04-20T00:10:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:9", 2024-04-20T00:10:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:10", 2024-04-20T00:10:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:3", 2024-04-20T00:10:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:5", 2024-04-20T00:10:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:1", 2024-04-20T00:10:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:9", 2024-04-20T00:10:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:1", 2024-04-20T00:10:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:7", 2024-04-20T00:10:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:5", 2024-04-20T00:10:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000073", 2024-04-20T00:10:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:7", 2024-04-20T00:10:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:2", 2024-04-20T00:10:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28", 2024-04-20T00:10:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000038", 2024-04-20T00:10:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20", 2024-04-20T00:10:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:8", 2024-04-20T00:10:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:5", 2024-04-20T00:10:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.10", 2024-04-20T00:10:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.3", 2024-04-20T00:10:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000118", 2024-04-20T00:10:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:5", 2024-04-20T00:10:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:0", 2024-04-20T00:10:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000036", 2024-04-20T00:10:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:6", 2024-04-20T00:10:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:6", 2024-04-20T00:10:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:0", 2024-04-20T00:10:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:0", 2024-04-20T00:10:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:6", 2024-04-20T00:10:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:9", 2024-04-20T00:10:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000103", 2024-04-20T00:10:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:10", 2024-04-20T00:10:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18", 2024-04-20T00:10:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "mdlog.sync-status.shard.3", 2024-04-20T00:10:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:0", 2024-04-20T00:10:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:3", 2024-04-20T00:10:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000119", 2024-04-20T00:10:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:10", 2024-04-20T00:10:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:5", 2024-04-20T00:10:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000098", 2024-04-20T00:10:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:10", 2024-04-20T00:10:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:5", 2024-04-20T00:10:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:4", 2024-04-20T00:10:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11", 2024-04-20T00:10:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.2.retry", 2024-04-20T00:10:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:4", 2024-04-20T00:10:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "mdlog.sync-status.shard.0", 2024-04-20T00:10:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45", 2024-04-20T00:10:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:5", 2024-04-20T00:10:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:1", 2024-04-20T00:10:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:6", 2024-04-20T00:10:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:5", 2024-04-20T00:10:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:2", 2024-04-20T00:10:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000027", 2024-04-20T00:10:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47", 2024-04-20T00:10:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:3", 2024-04-20T00:10:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "mdlog.sync-status", 2024-04-20T00:10:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000019", 2024-04-20T00:10:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.2.retry", 2024-04-20T00:10:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:1", 2024-04-20T00:10:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000039", 2024-04-20T00:10:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:1", 2024-04-20T00:10:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:3", 2024-04-20T00:10:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10", 2024-04-20T00:10:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:3", 2024-04-20T00:10:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:8", 2024-04-20T00:10:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:3", 2024-04-20T00:10:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:1", 2024-04-20T00:10:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:7", 2024-04-20T00:10:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:0", 2024-04-20T00:10:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:10", 2024-04-20T00:10:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000100", 2024-04-20T00:10:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:6", 2024-04-20T00:10:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:7", 2024-04-20T00:10:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:10", 2024-04-20T00:10:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:1", 2024-04-20T00:10:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:10", 2024-04-20T00:10:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:10", 2024-04-20T00:10:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:4", 2024-04-20T00:10:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000093", 2024-04-20T00:10:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:5", 2024-04-20T00:10:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:3", 2024-04-20T00:10:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:1", 2024-04-20T00:10:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:6", 2024-04-20T00:10:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:10", 2024-04-20T00:10:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22", 2024-04-20T00:10:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:8", 2024-04-20T00:10:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000004", 2024-04-20T00:10:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000063", 2024-04-20T00:10:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000122", 2024-04-20T00:10:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:10", 2024-04-20T00:10:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000057", 2024-04-20T00:10:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:1", 2024-04-20T00:10:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:0", 2024-04-20T00:10:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:9", 2024-04-20T00:10:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:8", 2024-04-20T00:10:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:8", 2024-04-20T00:10:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.1.retry", 2024-04-20T00:10:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000054", 2024-04-20T00:10:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000114", 2024-04-20T00:10:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:2", 2024-04-20T00:10:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:0", 2024-04-20T00:10:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.2", 2024-04-20T00:10:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:5", 2024-04-20T00:10:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "data_log.0", 2024-04-20T00:10:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:4", 2024-04-20T00:10:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000012", 2024-04-20T00:10:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:3", 2024-04-20T00:10:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:7", 2024-04-20T00:10:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:6", 2024-04-20T00:10:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:9", 2024-04-20T00:10:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:0", 2024-04-20T00:10:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14", 2024-04-20T00:10:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:8", 2024-04-20T00:10:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000084", 2024-04-20T00:10:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000043", 2024-04-20T00:10:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47", 2024-04-20T00:10:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:7", 2024-04-20T00:10:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.11", 2024-04-20T00:10:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12", 2024-04-20T00:10:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:6", 2024-04-20T00:10:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:9", 2024-04-20T00:10:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:5", 2024-04-20T00:10:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:0", 2024-04-20T00:10:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:0", 2024-04-20T00:10:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:10", 2024-04-20T00:10:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:2", 2024-04-20T00:10:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:5", 2024-04-20T00:10:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000111", 2024-04-20T00:10:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:8", 2024-04-20T00:10:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:8", 2024-04-20T00:10:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000048", 2024-04-20T00:10:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:0", 2024-04-20T00:10:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46", 2024-04-20T00:10:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:2", 2024-04-20T00:10:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000020", 2024-04-20T00:10:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:9", 2024-04-20T00:10:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:3", 2024-04-20T00:10:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:2", 2024-04-20T00:10:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18", 2024-04-20T00:10:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:10", 2024-04-20T00:10:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:0", 2024-04-20T00:10:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:3", 2024-04-20T00:10:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000099", 2024-04-20T00:10:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000056", 2024-04-20T00:10:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:1", 2024-04-20T00:10:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:4", 2024-04-20T00:10:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "data_log.3.wJbFlsc9Vqh.0", 2024-04-20T00:10:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:10", 2024-04-20T00:10:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:7", 2024-04-20T00:10:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:10", 2024-04-20T00:10:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.19", 2024-04-20T00:10:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:4", 2024-04-20T00:10:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.28", 2024-04-20T00:10:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:2", 2024-04-20T00:10:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.0.retry", 2024-04-20T00:10:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:4", 2024-04-20T00:10:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000072", 2024-04-20T00:10:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:7", 2024-04-20T00:10:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:2", 2024-04-20T00:10:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:2", 2024-04-20T00:10:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000062", 2024-04-20T00:10:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:1", 2024-04-20T00:10:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:3", 2024-04-20T00:10:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000115", 2024-04-20T00:10:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:7", 2024-04-20T00:10:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:9", 2024-04-20T00:10:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.1", 2024-04-20T00:10:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:8", 2024-04-20T00:10:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000033", 2024-04-20T00:10:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21", 2024-04-20T00:10:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:1", 2024-04-20T00:10:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:9", 2024-04-20T00:10:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:5", 2024-04-20T00:10:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:6", 2024-04-20T00:10:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:4", 2024-04-20T00:10:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:3", 2024-04-20T00:10:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:6", 2024-04-20T00:10:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:9", 2024-04-20T00:10:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:2", 2024-04-20T00:10:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:5", 2024-04-20T00:10:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:2", 2024-04-20T00:10:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:5", 2024-04-20T00:10:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:3", 2024-04-20T00:10:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000025", 2024-04-20T00:10:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000123", 2024-04-20T00:10:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:2", 2024-04-20T00:10:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.31", 2024-04-20T00:10:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.18", 2024-04-20T00:10:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:8", 2024-04-20T00:10:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:8", 2024-04-20T00:10:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:7", 2024-04-20T00:10:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000108", 2024-04-20T00:10:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:2", 2024-04-20T00:10:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:5", 2024-04-20T00:10:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:0", 2024-04-20T00:10:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000094", 2024-04-20T00:10:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:4", 2024-04-20T00:10:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000060", 2024-04-20T00:10:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:2", 2024-04-20T00:10:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:7", 2024-04-20T00:10:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:0", 2024-04-20T00:10:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:8", 2024-04-20T00:10:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:0", 2024-04-20T00:10:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000109", 2024-04-20T00:10:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.15", 2024-04-20T00:10:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:10", 2024-04-20T00:10:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000013", 2024-04-20T00:10:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:9", 2024-04-20T00:10:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:0", 2024-04-20T00:10:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:5", 2024-04-20T00:10:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28", 2024-04-20T00:10:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:4", 2024-04-20T00:10:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:3", 2024-04-20T00:10:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:5", 2024-04-20T00:10:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:2", 2024-04-20T00:10:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26", 2024-04-20T00:10:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.3", 2024-04-20T00:10:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.0", 2024-04-20T00:10:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:5", 2024-04-20T00:10:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000051", 2024-04-20T00:10:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:8", 2024-04-20T00:10:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:10", 2024-04-20T00:10:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "data_log.0.T0tGY/bKh7w.0", 2024-04-20T00:10:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:6", 2024-04-20T00:10:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000106", 2024-04-20T00:10:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:1", 2024-04-20T00:10:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14", 2024-04-20T00:10:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:8", 2024-04-20T00:10:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:9", 2024-04-20T00:10:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000017", 2024-04-20T00:10:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:0", 2024-04-20T00:10:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:2", 2024-04-20T00:10:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:9", 2024-04-20T00:10:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16", 2024-04-20T00:10:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:4", 2024-04-20T00:10:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:6", 2024-04-20T00:10:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:9", 2024-04-20T00:10:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:5", 2024-04-20T00:10:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:0", 2024-04-20T00:10:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:10", 2024-04-20T00:10:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:10", 2024-04-20T00:10:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:4", 2024-04-20T00:10:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:7", 2024-04-20T00:10:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000023", 2024-04-20T00:10:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:1", 2024-04-20T00:10:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:7", 2024-04-20T00:10:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15", 2024-04-20T00:10:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:6", 2024-04-20T00:10:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:9", 2024-04-20T00:10:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.2", 2024-04-20T00:10:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:8", 2024-04-20T00:10:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:0", 2024-04-20T00:10:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000107", 2024-04-20T00:10:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000046", 2024-04-20T00:10:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:5", 2024-04-20T00:10:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:1", 2024-04-20T00:10:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:0", 2024-04-20T00:10:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.26", 2024-04-20T00:10:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:5", 2024-04-20T00:10:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:3", 2024-04-20T00:10:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20", 2024-04-20T00:10:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.22", 2024-04-20T00:10:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:8", 2024-04-20T00:10:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:0", 2024-04-20T00:10:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:0", 2024-04-20T00:10:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:1", 2024-04-20T00:10:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:9", 2024-04-20T00:10:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:4", 2024-04-20T00:10:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:10", 2024-04-20T00:10:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "data_log.2", 2024-04-20T00:10:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:7", 2024-04-20T00:10:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000067", 2024-04-20T00:10:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:8", 2024-04-20T00:10:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:4", 2024-04-20T00:10:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:7", 2024-04-20T00:10:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:7", 2024-04-20T00:10:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:2", 2024-04-20T00:10:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:8", 2024-04-20T00:10:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "data_log.1.fX1+jKwmIGM.0", 2024-04-20T00:10:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:2", 2024-04-20T00:10:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:5", 2024-04-20T00:10:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:4", 2024-04-20T00:10:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:7", 2024-04-20T00:10:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:2", 2024-04-20T00:10:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.9", 2024-04-20T00:10:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000086", 2024-04-20T00:10:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:8", 2024-04-20T00:10:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:7", 2024-04-20T00:10:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:2", 2024-04-20T00:10:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.5", 2024-04-20T00:10:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21", 2024-04-20T00:10:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:5", 2024-04-20T00:10:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.3.retry", 2024-04-20T00:10:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:4", 2024-04-20T00:10:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:6", 2024-04-20T00:10:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:4", 2024-04-20T00:10:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:9", 2024-04-20T00:10:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:0", 2024-04-20T00:10:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:6", 2024-04-20T00:10:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "sync.error-log.20", 2024-04-20T00:10:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:3", 2024-04-20T00:10:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:1", 2024-04-20T00:10:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:9", 2024-04-20T00:10:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23", 2024-04-20T00:10:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16", 2024-04-20T00:10:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:7", 2024-04-20T00:10:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:5", 2024-04-20T00:10:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:10", 2024-04-20T00:10:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:5", 2024-04-20T00:10:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:10", 2024-04-20T00:10:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:5", 2024-04-20T00:10:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:2", 2024-04-20T00:10:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000050", 2024-04-20T00:10:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:2", 2024-04-20T00:10:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000102", 2024-04-20T00:10:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:3", 2024-04-20T00:10:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:10", 2024-04-20T00:10:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:10", 2024-04-20T00:10:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:2", 2024-04-20T00:10:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:7", 2024-04-20T00:10:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:0", 2024-04-20T00:10:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:5", 2024-04-20T00:10:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000124", 2024-04-20T00:10:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.1", 2024-04-20T00:10:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15", 2024-04-20T00:10:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:10", 2024-04-20T00:10:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:9", 2024-04-20T00:10:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:5", 2024-04-20T00:10:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:3", 2024-04-20T00:10:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:2", 2024-04-20T00:10:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:6", 2024-04-20T00:10:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000001", 2024-04-20T00:10:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000126", 2024-04-20T00:10:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:8", 2024-04-20T00:10:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:5", 2024-04-20T00:10:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000055", 2024-04-20T00:10:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:4", 2024-04-20T00:10:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000061", 2024-04-20T00:10:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:7", 2024-04-20T00:10:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:7", 2024-04-20T00:10:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:3", 2024-04-20T00:10:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000097", 2024-04-20T00:10:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:4", 2024-04-20T00:10:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:10", 2024-04-20T00:10:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:3", 2024-04-20T00:10:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:4", 2024-04-20T00:10:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:4", 2024-04-20T00:10:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000042", 2024-04-20T00:10:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:9", 2024-04-20T00:10:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:2", 2024-04-20T00:10:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:4", 2024-04-20T00:10:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "datalog.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T00:10:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:10", 2024-04-20T00:10:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:0", 2024-04-20T00:10:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:6", 2024-04-20T00:10:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:7", 2024-04-20T00:10:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:6", 2024-04-20T00:10:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000075", 2024-04-20T00:10:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:2", 2024-04-20T00:10:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:3", 2024-04-20T00:10:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:2", 2024-04-20T00:10:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000120", 2024-04-20T00:10:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "obj_delete_at_hint.0000000030", 2024-04-20T00:10:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:10", 2024-04-20T00:10:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:3", 2024-04-20T00:10:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:0", 2024-04-20T00:10:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:4", 2024-04-20T00:10:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:10", 2024-04-20T00:10:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:4" 2024-04-20T00:10:03.076 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-04-20T00:10:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-20T00:10:03.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:03.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:03.077 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:03.135 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:03.135 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:03.199 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:10:03.199 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.29", 2024-04-20T00:10:03.199 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000078", 2024-04-20T00:10:03.199 INFO:teuthology.orchestra.run.smithi019.stdout: "meta.history", 2024-04-20T00:10:03.200 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:9", 2024-04-20T00:10:03.200 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.23", 2024-04-20T00:10:03.200 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000070", 2024-04-20T00:10:03.200 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:1", 2024-04-20T00:10:03.200 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000104", 2024-04-20T00:10:03.200 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:8", 2024-04-20T00:10:03.200 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:9", 2024-04-20T00:10:03.200 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:8", 2024-04-20T00:10:03.200 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:5", 2024-04-20T00:10:03.200 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:8", 2024-04-20T00:10:03.200 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000026", 2024-04-20T00:10:03.201 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26", 2024-04-20T00:10:03.201 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:3", 2024-04-20T00:10:03.201 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.8", 2024-04-20T00:10:03.201 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14", 2024-04-20T00:10:03.201 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000028", 2024-04-20T00:10:03.201 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:6", 2024-04-20T00:10:03.201 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:8", 2024-04-20T00:10:03.201 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:3", 2024-04-20T00:10:03.201 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.2.XuvZF2h1atf.0", 2024-04-20T00:10:03.201 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20", 2024-04-20T00:10:03.201 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:6", 2024-04-20T00:10:03.201 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.1.jWKpb7Z2XkW.0", 2024-04-20T00:10:03.202 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19", 2024-04-20T00:10:03.202 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:4", 2024-04-20T00:10:03.202 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.0.retry", 2024-04-20T00:10:03.202 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:1", 2024-04-20T00:10:03.202 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:0", 2024-04-20T00:10:03.202 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:4", 2024-04-20T00:10:03.202 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:2", 2024-04-20T00:10:03.202 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.30", 2024-04-20T00:10:03.202 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T00:10:03.202 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:2", 2024-04-20T00:10:03.202 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:6", 2024-04-20T00:10:03.203 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:6", 2024-04-20T00:10:03.203 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:3", 2024-04-20T00:10:03.203 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:7", 2024-04-20T00:10:03.203 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000040", 2024-04-20T00:10:03.203 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:7", 2024-04-20T00:10:03.203 INFO:teuthology.orchestra.run.smithi019.stdout: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.0", 2024-04-20T00:10:03.203 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.6", 2024-04-20T00:10:03.203 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:1", 2024-04-20T00:10:03.203 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:8", 2024-04-20T00:10:03.203 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:5", 2024-04-20T00:10:03.203 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:6", 2024-04-20T00:10:03.204 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:9", 2024-04-20T00:10:03.204 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:8", 2024-04-20T00:10:03.204 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:2", 2024-04-20T00:10:03.204 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000015", 2024-04-20T00:10:03.204 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:2", 2024-04-20T00:10:03.204 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:2", 2024-04-20T00:10:03.204 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:7", 2024-04-20T00:10:03.204 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:1", 2024-04-20T00:10:03.204 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000069", 2024-04-20T00:10:03.204 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:10", 2024-04-20T00:10:03.204 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000095", 2024-04-20T00:10:03.204 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:0", 2024-04-20T00:10:03.205 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000003", 2024-04-20T00:10:03.205 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000047", 2024-04-20T00:10:03.205 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:5", 2024-04-20T00:10:03.205 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:9", 2024-04-20T00:10:03.205 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:4", 2024-04-20T00:10:03.205 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:1", 2024-04-20T00:10:03.205 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000035", 2024-04-20T00:10:03.205 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:8", 2024-04-20T00:10:03.205 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:2", 2024-04-20T00:10:03.205 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000037", 2024-04-20T00:10:03.205 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:4", 2024-04-20T00:10:03.205 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.0.retry", 2024-04-20T00:10:03.206 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:1", 2024-04-20T00:10:03.206 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:4", 2024-04-20T00:10:03.206 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:6", 2024-04-20T00:10:03.206 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:9", 2024-04-20T00:10:03.206 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:5", 2024-04-20T00:10:03.206 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:3", 2024-04-20T00:10:03.206 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:3", 2024-04-20T00:10:03.206 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:7", 2024-04-20T00:10:03.206 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:1", 2024-04-20T00:10:03.206 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:1", 2024-04-20T00:10:03.206 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000024", 2024-04-20T00:10:03.207 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:2", 2024-04-20T00:10:03.207 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:5", 2024-04-20T00:10:03.207 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:8", 2024-04-20T00:10:03.207 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:10", 2024-04-20T00:10:03.207 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000000", 2024-04-20T00:10:03.207 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:4", 2024-04-20T00:10:03.207 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000031", 2024-04-20T00:10:03.207 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:1", 2024-04-20T00:10:03.207 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:1", 2024-04-20T00:10:03.207 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:2", 2024-04-20T00:10:03.207 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:2", 2024-04-20T00:10:03.208 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.3", 2024-04-20T00:10:03.208 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:1", 2024-04-20T00:10:03.208 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:7", 2024-04-20T00:10:03.208 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:10", 2024-04-20T00:10:03.208 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:10", 2024-04-20T00:10:03.208 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:3", 2024-04-20T00:10:03.208 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:4", 2024-04-20T00:10:03.208 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:5", 2024-04-20T00:10:03.208 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:4", 2024-04-20T00:10:03.208 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:5", 2024-04-20T00:10:03.208 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:3", 2024-04-20T00:10:03.209 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:7", 2024-04-20T00:10:03.209 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:1", 2024-04-20T00:10:03.209 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:3", 2024-04-20T00:10:03.209 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:4", 2024-04-20T00:10:03.209 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:2", 2024-04-20T00:10:03.209 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:2", 2024-04-20T00:10:03.209 INFO:teuthology.orchestra.run.smithi019.stdout: "data_loggenerations_metadata", 2024-04-20T00:10:03.209 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:3", 2024-04-20T00:10:03.209 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:6", 2024-04-20T00:10:03.209 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:10", 2024-04-20T00:10:03.209 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:6", 2024-04-20T00:10:03.210 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:5", 2024-04-20T00:10:03.210 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:8", 2024-04-20T00:10:03.210 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:9", 2024-04-20T00:10:03.210 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:3", 2024-04-20T00:10:03.210 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:3", 2024-04-20T00:10:03.210 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000076", 2024-04-20T00:10:03.210 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:4", 2024-04-20T00:10:03.210 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:7", 2024-04-20T00:10:03.210 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000113", 2024-04-20T00:10:03.210 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000005", 2024-04-20T00:10:03.210 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:5", 2024-04-20T00:10:03.211 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:5", 2024-04-20T00:10:03.211 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000011", 2024-04-20T00:10:03.211 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000058", 2024-04-20T00:10:03.211 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000052", 2024-04-20T00:10:03.211 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:4", 2024-04-20T00:10:03.211 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:9", 2024-04-20T00:10:03.211 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.21", 2024-04-20T00:10:03.211 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:9", 2024-04-20T00:10:03.211 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:3", 2024-04-20T00:10:03.211 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:9", 2024-04-20T00:10:03.211 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:8", 2024-04-20T00:10:03.211 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:2", 2024-04-20T00:10:03.212 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:3", 2024-04-20T00:10:03.212 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:10", 2024-04-20T00:10:03.212 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:8", 2024-04-20T00:10:03.212 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27", 2024-04-20T00:10:03.212 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46", 2024-04-20T00:10:03.212 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:4", 2024-04-20T00:10:03.212 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:8", 2024-04-20T00:10:03.212 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:6", 2024-04-20T00:10:03.212 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:3", 2024-04-20T00:10:03.212 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:4", 2024-04-20T00:10:03.212 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:8", 2024-04-20T00:10:03.213 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18", 2024-04-20T00:10:03.213 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000088", 2024-04-20T00:10:03.213 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000080", 2024-04-20T00:10:03.213 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:3", 2024-04-20T00:10:03.213 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000090", 2024-04-20T00:10:03.213 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22", 2024-04-20T00:10:03.213 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:1", 2024-04-20T00:10:03.213 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:5", 2024-04-20T00:10:03.213 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:4", 2024-04-20T00:10:03.213 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:7", 2024-04-20T00:10:03.213 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000110", 2024-04-20T00:10:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:9", 2024-04-20T00:10:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:0", 2024-04-20T00:10:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.0", 2024-04-20T00:10:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:4", 2024-04-20T00:10:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T00:10:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000096", 2024-04-20T00:10:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:1", 2024-04-20T00:10:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000087", 2024-04-20T00:10:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000008", 2024-04-20T00:10:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:3", 2024-04-20T00:10:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:6", 2024-04-20T00:10:03.214 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:2", 2024-04-20T00:10:03.215 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:3", 2024-04-20T00:10:03.215 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:1", 2024-04-20T00:10:03.215 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.0.JXRWqBwqaXn.0", 2024-04-20T00:10:03.215 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000006", 2024-04-20T00:10:03.215 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:1", 2024-04-20T00:10:03.215 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29", 2024-04-20T00:10:03.215 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000029", 2024-04-20T00:10:03.215 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:7", 2024-04-20T00:10:03.215 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:9", 2024-04-20T00:10:03.215 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:7", 2024-04-20T00:10:03.215 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:6", 2024-04-20T00:10:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:10", 2024-04-20T00:10:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000089", 2024-04-20T00:10:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000022", 2024-04-20T00:10:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.0", 2024-04-20T00:10:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:9", 2024-04-20T00:10:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:7", 2024-04-20T00:10:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1", 2024-04-20T00:10:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:7", 2024-04-20T00:10:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:6", 2024-04-20T00:10:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:2", 2024-04-20T00:10:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:10", 2024-04-20T00:10:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "bilog.trim", 2024-04-20T00:10:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:6", 2024-04-20T00:10:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:3", 2024-04-20T00:10:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13", 2024-04-20T00:10:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000016", 2024-04-20T00:10:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:4", 2024-04-20T00:10:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:5", 2024-04-20T00:10:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:9", 2024-04-20T00:10:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:6", 2024-04-20T00:10:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:8", 2024-04-20T00:10:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:2", 2024-04-20T00:10:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000041", 2024-04-20T00:10:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10", 2024-04-20T00:10:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:10", 2024-04-20T00:10:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:1", 2024-04-20T00:10:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:7", 2024-04-20T00:10:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:1", 2024-04-20T00:10:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27", 2024-04-20T00:10:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25", 2024-04-20T00:10:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000018", 2024-04-20T00:10:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:6", 2024-04-20T00:10:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000092", 2024-04-20T00:10:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:9", 2024-04-20T00:10:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:6", 2024-04-20T00:10:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:2", 2024-04-20T00:10:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:0", 2024-04-20T00:10:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:2", 2024-04-20T00:10:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:10", 2024-04-20T00:10:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:5", 2024-04-20T00:10:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.27", 2024-04-20T00:10:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:0", 2024-04-20T00:10:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.4", 2024-04-20T00:10:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:8", 2024-04-20T00:10:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:0", 2024-04-20T00:10:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:6", 2024-04-20T00:10:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:8", 2024-04-20T00:10:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:4", 2024-04-20T00:10:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:6", 2024-04-20T00:10:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000014", 2024-04-20T00:10:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:2", 2024-04-20T00:10:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:3", 2024-04-20T00:10:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:6", 2024-04-20T00:10:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28", 2024-04-20T00:10:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000112", 2024-04-20T00:10:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:2", 2024-04-20T00:10:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000007", 2024-04-20T00:10:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000021", 2024-04-20T00:10:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:1", 2024-04-20T00:10:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:6", 2024-04-20T00:10:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.16", 2024-04-20T00:10:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:0", 2024-04-20T00:10:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000064", 2024-04-20T00:10:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:0", 2024-04-20T00:10:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16", 2024-04-20T00:10:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:8", 2024-04-20T00:10:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000071", 2024-04-20T00:10:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:1", 2024-04-20T00:10:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000074", 2024-04-20T00:10:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:8", 2024-04-20T00:10:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:8", 2024-04-20T00:10:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:6", 2024-04-20T00:10:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:10", 2024-04-20T00:10:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:7", 2024-04-20T00:10:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:1", 2024-04-20T00:10:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:0", 2024-04-20T00:10:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:6", 2024-04-20T00:10:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000081", 2024-04-20T00:10:03.223 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:5", 2024-04-20T00:10:03.223 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000009", 2024-04-20T00:10:03.223 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000121", 2024-04-20T00:10:03.223 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:3", 2024-04-20T00:10:03.223 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.1", 2024-04-20T00:10:03.223 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:9", 2024-04-20T00:10:03.223 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:6", 2024-04-20T00:10:03.223 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:2", 2024-04-20T00:10:03.223 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:8", 2024-04-20T00:10:03.223 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.17", 2024-04-20T00:10:03.223 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000125", 2024-04-20T00:10:03.223 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000082", 2024-04-20T00:10:03.224 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:7", 2024-04-20T00:10:03.224 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:0", 2024-04-20T00:10:03.224 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:8", 2024-04-20T00:10:03.224 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:7", 2024-04-20T00:10:03.224 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:6", 2024-04-20T00:10:03.224 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:6", 2024-04-20T00:10:03.224 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000105", 2024-04-20T00:10:03.224 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000059", 2024-04-20T00:10:03.224 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:8", 2024-04-20T00:10:03.224 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:1", 2024-04-20T00:10:03.224 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:10", 2024-04-20T00:10:03.225 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:2", 2024-04-20T00:10:03.225 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:5", 2024-04-20T00:10:03.225 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000077", 2024-04-20T00:10:03.225 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:10", 2024-04-20T00:10:03.225 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:8", 2024-04-20T00:10:03.225 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000032", 2024-04-20T00:10:03.225 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:8", 2024-04-20T00:10:03.225 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:2", 2024-04-20T00:10:03.225 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:3", 2024-04-20T00:10:03.225 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:5", 2024-04-20T00:10:03.225 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:3", 2024-04-20T00:10:03.225 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.24", 2024-04-20T00:10:03.226 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:10", 2024-04-20T00:10:03.226 INFO:teuthology.orchestra.run.smithi019.stdout: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.2", 2024-04-20T00:10:03.226 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:1", 2024-04-20T00:10:03.226 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:4", 2024-04-20T00:10:03.226 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:5", 2024-04-20T00:10:03.226 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:0", 2024-04-20T00:10:03.226 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:1", 2024-04-20T00:10:03.226 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000053", 2024-04-20T00:10:03.226 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:4", 2024-04-20T00:10:03.226 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:7", 2024-04-20T00:10:03.226 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:6", 2024-04-20T00:10:03.227 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:7", 2024-04-20T00:10:03.227 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000091", 2024-04-20T00:10:03.227 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46", 2024-04-20T00:10:03.227 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:9", 2024-04-20T00:10:03.227 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:2", 2024-04-20T00:10:03.227 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:10", 2024-04-20T00:10:03.227 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:2", 2024-04-20T00:10:03.227 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000065", 2024-04-20T00:10:03.227 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:4", 2024-04-20T00:10:03.227 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000083", 2024-04-20T00:10:03.227 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000010", 2024-04-20T00:10:03.228 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:9", 2024-04-20T00:10:03.228 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:3", 2024-04-20T00:10:03.228 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:9", 2024-04-20T00:10:03.228 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:5", 2024-04-20T00:10:03.228 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:0", 2024-04-20T00:10:03.228 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:10", 2024-04-20T00:10:03.228 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:0", 2024-04-20T00:10:03.228 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.7", 2024-04-20T00:10:03.228 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25", 2024-04-20T00:10:03.228 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.3", 2024-04-20T00:10:03.228 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.3.retry", 2024-04-20T00:10:03.229 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:9", 2024-04-20T00:10:03.229 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:0", 2024-04-20T00:10:03.229 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:3", 2024-04-20T00:10:03.229 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:8", 2024-04-20T00:10:03.229 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:1", 2024-04-20T00:10:03.229 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:6", 2024-04-20T00:10:03.229 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:6", 2024-04-20T00:10:03.229 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:4", 2024-04-20T00:10:03.229 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:1", 2024-04-20T00:10:03.229 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:0", 2024-04-20T00:10:03.229 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:0", 2024-04-20T00:10:03.230 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:9", 2024-04-20T00:10:03.230 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:3", 2024-04-20T00:10:03.230 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:1", 2024-04-20T00:10:03.230 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:6", 2024-04-20T00:10:03.230 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:9", 2024-04-20T00:10:03.230 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:0", 2024-04-20T00:10:03.230 INFO:teuthology.orchestra.run.smithi019.stdout: "mdlog.sync-status.shard.2", 2024-04-20T00:10:03.230 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:10", 2024-04-20T00:10:03.230 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:5", 2024-04-20T00:10:03.230 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11", 2024-04-20T00:10:03.230 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000045", 2024-04-20T00:10:03.231 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11", 2024-04-20T00:10:03.231 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:10", 2024-04-20T00:10:03.231 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:9", 2024-04-20T00:10:03.231 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000002", 2024-04-20T00:10:03.231 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.12", 2024-04-20T00:10:03.231 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:7", 2024-04-20T00:10:03.231 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:6", 2024-04-20T00:10:03.231 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:2", 2024-04-20T00:10:03.231 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:8", 2024-04-20T00:10:03.231 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000116", 2024-04-20T00:10:03.231 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:8", 2024-04-20T00:10:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:6", 2024-04-20T00:10:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.3", 2024-04-20T00:10:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.25", 2024-04-20T00:10:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:0", 2024-04-20T00:10:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000034", 2024-04-20T00:10:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:9", 2024-04-20T00:10:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:0", 2024-04-20T00:10:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.3.fX54X+2Z0es.0", 2024-04-20T00:10:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:4", 2024-04-20T00:10:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:1", 2024-04-20T00:10:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:10", 2024-04-20T00:10:03.233 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:7", 2024-04-20T00:10:03.233 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:4", 2024-04-20T00:10:03.233 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:0", 2024-04-20T00:10:03.233 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000101", 2024-04-20T00:10:03.233 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:0", 2024-04-20T00:10:03.233 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:0", 2024-04-20T00:10:03.233 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.13", 2024-04-20T00:10:03.233 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:1", 2024-04-20T00:10:03.233 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:10", 2024-04-20T00:10:03.233 INFO:teuthology.orchestra.run.smithi019.stdout: "mdlog.sync-status.shard.1", 2024-04-20T00:10:03.233 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17", 2024-04-20T00:10:03.233 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:2", 2024-04-20T00:10:03.234 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:7", 2024-04-20T00:10:03.234 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:3", 2024-04-20T00:10:03.234 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:8", 2024-04-20T00:10:03.234 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000079", 2024-04-20T00:10:03.234 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:0", 2024-04-20T00:10:03.234 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000049", 2024-04-20T00:10:03.234 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000117", 2024-04-20T00:10:03.234 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000044", 2024-04-20T00:10:03.234 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:5", 2024-04-20T00:10:03.234 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24", 2024-04-20T00:10:03.234 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:1", 2024-04-20T00:10:03.235 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:7", 2024-04-20T00:10:03.235 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:1", 2024-04-20T00:10:03.235 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.14", 2024-04-20T00:10:03.235 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000066", 2024-04-20T00:10:03.235 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:7", 2024-04-20T00:10:03.235 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:0", 2024-04-20T00:10:03.235 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:2", 2024-04-20T00:10:03.235 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:4", 2024-04-20T00:10:03.235 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:2", 2024-04-20T00:10:03.236 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000068", 2024-04-20T00:10:03.236 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000085", 2024-04-20T00:10:03.236 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:5", 2024-04-20T00:10:03.236 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:9", 2024-04-20T00:10:03.236 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:9", 2024-04-20T00:10:03.236 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:3", 2024-04-20T00:10:03.237 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:8", 2024-04-20T00:10:03.237 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:1", 2024-04-20T00:10:03.237 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000073", 2024-04-20T00:10:03.237 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:7", 2024-04-20T00:10:03.237 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:2", 2024-04-20T00:10:03.237 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28", 2024-04-20T00:10:03.237 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000038", 2024-04-20T00:10:03.237 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19", 2024-04-20T00:10:03.237 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:1", 2024-04-20T00:10:03.238 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45", 2024-04-20T00:10:03.238 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:5", 2024-04-20T00:10:03.238 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:9", 2024-04-20T00:10:03.238 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.10", 2024-04-20T00:10:03.238 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.3", 2024-04-20T00:10:03.238 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000118", 2024-04-20T00:10:03.238 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:2", 2024-04-20T00:10:03.238 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:0", 2024-04-20T00:10:03.238 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:5", 2024-04-20T00:10:03.238 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000036", 2024-04-20T00:10:03.238 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:6", 2024-04-20T00:10:03.239 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:9", 2024-04-20T00:10:03.239 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:0", 2024-04-20T00:10:03.239 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:8", 2024-04-20T00:10:03.239 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:10", 2024-04-20T00:10:03.239 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:6", 2024-04-20T00:10:03.239 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:9", 2024-04-20T00:10:03.239 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000103", 2024-04-20T00:10:03.239 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:5", 2024-04-20T00:10:03.239 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:10", 2024-04-20T00:10:03.239 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18", 2024-04-20T00:10:03.239 INFO:teuthology.orchestra.run.smithi019.stdout: "mdlog.sync-status.shard.3", 2024-04-20T00:10:03.240 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:10", 2024-04-20T00:10:03.240 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23", 2024-04-20T00:10:03.240 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:3", 2024-04-20T00:10:03.240 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:5", 2024-04-20T00:10:03.240 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:10", 2024-04-20T00:10:03.240 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000119", 2024-04-20T00:10:03.240 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:10", 2024-04-20T00:10:03.240 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:4", 2024-04-20T00:10:03.240 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:5", 2024-04-20T00:10:03.240 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:5", 2024-04-20T00:10:03.240 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:7", 2024-04-20T00:10:03.240 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000098", 2024-04-20T00:10:03.241 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:10", 2024-04-20T00:10:03.241 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:10", 2024-04-20T00:10:03.241 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.0", 2024-04-20T00:10:03.241 INFO:teuthology.orchestra.run.smithi019.stdout: "mdlog.sync-status.shard.0", 2024-04-20T00:10:03.241 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45", 2024-04-20T00:10:03.241 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:4", 2024-04-20T00:10:03.241 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:1", 2024-04-20T00:10:03.241 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:6", 2024-04-20T00:10:03.241 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000027", 2024-04-20T00:10:03.241 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:0", 2024-04-20T00:10:03.241 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:7", 2024-04-20T00:10:03.242 INFO:teuthology.orchestra.run.smithi019.stdout: "mdlog.sync-status", 2024-04-20T00:10:03.242 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000019", 2024-04-20T00:10:03.242 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.2.retry", 2024-04-20T00:10:03.242 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:9", 2024-04-20T00:10:03.242 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000039", 2024-04-20T00:10:03.242 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:0", 2024-04-20T00:10:03.242 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10", 2024-04-20T00:10:03.242 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:3", 2024-04-20T00:10:03.242 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:1", 2024-04-20T00:10:03.242 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:8", 2024-04-20T00:10:03.242 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:10", 2024-04-20T00:10:03.243 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.2.retry", 2024-04-20T00:10:03.243 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:3", 2024-04-20T00:10:03.243 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:8", 2024-04-20T00:10:03.243 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:0", 2024-04-20T00:10:03.243 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:7", 2024-04-20T00:10:03.243 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:0", 2024-04-20T00:10:03.243 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000100", 2024-04-20T00:10:03.243 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:0", 2024-04-20T00:10:03.243 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:6", 2024-04-20T00:10:03.243 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:7", 2024-04-20T00:10:03.243 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:10", 2024-04-20T00:10:03.243 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:8", 2024-04-20T00:10:03.244 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:1", 2024-04-20T00:10:03.244 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:10", 2024-04-20T00:10:03.244 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:4", 2024-04-20T00:10:03.244 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000093", 2024-04-20T00:10:03.244 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:3", 2024-04-20T00:10:03.244 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:1", 2024-04-20T00:10:03.244 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:8", 2024-04-20T00:10:03.244 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:5", 2024-04-20T00:10:03.244 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22", 2024-04-20T00:10:03.244 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000004", 2024-04-20T00:10:03.244 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000063", 2024-04-20T00:10:03.245 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000122", 2024-04-20T00:10:03.245 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:10", 2024-04-20T00:10:03.245 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000057", 2024-04-20T00:10:03.245 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:0", 2024-04-20T00:10:03.245 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:9", 2024-04-20T00:10:03.245 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:8", 2024-04-20T00:10:03.245 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.1.retry", 2024-04-20T00:10:03.245 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:3", 2024-04-20T00:10:03.245 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000054", 2024-04-20T00:10:03.245 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:3", 2024-04-20T00:10:03.245 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000114", 2024-04-20T00:10:03.246 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:2", 2024-04-20T00:10:03.246 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.2", 2024-04-20T00:10:03.246 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.0", 2024-04-20T00:10:03.246 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:3", 2024-04-20T00:10:03.246 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000012", 2024-04-20T00:10:03.246 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:3", 2024-04-20T00:10:03.246 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.3", 2024-04-20T00:10:03.246 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:6", 2024-04-20T00:10:03.246 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:10", 2024-04-20T00:10:03.246 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:8", 2024-04-20T00:10:03.246 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000084", 2024-04-20T00:10:03.246 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000043", 2024-04-20T00:10:03.247 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47", 2024-04-20T00:10:03.247 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:7", 2024-04-20T00:10:03.247 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.11", 2024-04-20T00:10:03.247 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12", 2024-04-20T00:10:03.247 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:6", 2024-04-20T00:10:03.247 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:9", 2024-04-20T00:10:03.247 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:1", 2024-04-20T00:10:03.247 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:10", 2024-04-20T00:10:03.247 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:9", 2024-04-20T00:10:03.247 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:5", 2024-04-20T00:10:03.247 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000111", 2024-04-20T00:10:03.248 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:8", 2024-04-20T00:10:03.248 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:6", 2024-04-20T00:10:03.248 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:8", 2024-04-20T00:10:03.248 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000048", 2024-04-20T00:10:03.248 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:0", 2024-04-20T00:10:03.248 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:2", 2024-04-20T00:10:03.248 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000020", 2024-04-20T00:10:03.248 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:3", 2024-04-20T00:10:03.248 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:2", 2024-04-20T00:10:03.248 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:10", 2024-04-20T00:10:03.248 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:0", 2024-04-20T00:10:03.248 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:5", 2024-04-20T00:10:03.249 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:3", 2024-04-20T00:10:03.249 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:3", 2024-04-20T00:10:03.249 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000099", 2024-04-20T00:10:03.249 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000056", 2024-04-20T00:10:03.249 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:0", 2024-04-20T00:10:03.249 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:10", 2024-04-20T00:10:03.249 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:3", 2024-04-20T00:10:03.249 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:7", 2024-04-20T00:10:03.249 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.19", 2024-04-20T00:10:03.249 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:4", 2024-04-20T00:10:03.249 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:9", 2024-04-20T00:10:03.250 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.28", 2024-04-20T00:10:03.250 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:2", 2024-04-20T00:10:03.250 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:9", 2024-04-20T00:10:03.250 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:4", 2024-04-20T00:10:03.250 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000072", 2024-04-20T00:10:03.250 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:0", 2024-04-20T00:10:03.250 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15", 2024-04-20T00:10:03.250 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:10", 2024-04-20T00:10:03.250 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:2", 2024-04-20T00:10:03.250 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000062", 2024-04-20T00:10:03.250 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:1", 2024-04-20T00:10:03.250 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.2", 2024-04-20T00:10:03.251 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:3", 2024-04-20T00:10:03.251 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000115", 2024-04-20T00:10:03.251 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:7", 2024-04-20T00:10:03.251 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.1", 2024-04-20T00:10:03.251 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:1", 2024-04-20T00:10:03.251 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000033", 2024-04-20T00:10:03.251 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21", 2024-04-20T00:10:03.251 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:5", 2024-04-20T00:10:03.251 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:9", 2024-04-20T00:10:03.251 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:5", 2024-04-20T00:10:03.251 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:6", 2024-04-20T00:10:03.252 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:4", 2024-04-20T00:10:03.252 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:7", 2024-04-20T00:10:03.252 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:3", 2024-04-20T00:10:03.252 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:7", 2024-04-20T00:10:03.252 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:9", 2024-04-20T00:10:03.252 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:10", 2024-04-20T00:10:03.252 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:2", 2024-04-20T00:10:03.252 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:5", 2024-04-20T00:10:03.252 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:2", 2024-04-20T00:10:03.252 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:5", 2024-04-20T00:10:03.252 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17", 2024-04-20T00:10:03.253 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:6", 2024-04-20T00:10:03.253 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000025", 2024-04-20T00:10:03.253 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000123", 2024-04-20T00:10:03.253 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.31", 2024-04-20T00:10:03.253 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:9", 2024-04-20T00:10:03.253 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:0", 2024-04-20T00:10:03.253 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.18", 2024-04-20T00:10:03.253 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:8", 2024-04-20T00:10:03.253 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:5", 2024-04-20T00:10:03.253 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:8", 2024-04-20T00:10:03.253 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000108", 2024-04-20T00:10:03.253 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:2", 2024-04-20T00:10:03.254 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:5", 2024-04-20T00:10:03.254 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:0", 2024-04-20T00:10:03.254 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000094", 2024-04-20T00:10:03.254 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:6", 2024-04-20T00:10:03.254 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000060", 2024-04-20T00:10:03.254 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:7", 2024-04-20T00:10:03.254 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:5", 2024-04-20T00:10:03.254 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:8", 2024-04-20T00:10:03.254 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13", 2024-04-20T00:10:03.254 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000109", 2024-04-20T00:10:03.254 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.15", 2024-04-20T00:10:03.255 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:7", 2024-04-20T00:10:03.255 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000013", 2024-04-20T00:10:03.255 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:0", 2024-04-20T00:10:03.255 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:4", 2024-04-20T00:10:03.255 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:3", 2024-04-20T00:10:03.255 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26", 2024-04-20T00:10:03.255 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000051", 2024-04-20T00:10:03.255 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:3", 2024-04-20T00:10:03.255 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:8", 2024-04-20T00:10:03.255 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:10", 2024-04-20T00:10:03.255 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:6", 2024-04-20T00:10:03.256 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000106", 2024-04-20T00:10:03.256 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:1", 2024-04-20T00:10:03.256 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:10", 2024-04-20T00:10:03.256 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:5", 2024-04-20T00:10:03.256 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14", 2024-04-20T00:10:03.256 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000017", 2024-04-20T00:10:03.256 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:0", 2024-04-20T00:10:03.256 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:2", 2024-04-20T00:10:03.256 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:6", 2024-04-20T00:10:03.256 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:0", 2024-04-20T00:10:03.256 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:0", 2024-04-20T00:10:03.256 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:10", 2024-04-20T00:10:03.257 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:4", 2024-04-20T00:10:03.257 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000023", 2024-04-20T00:10:03.257 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:1", 2024-04-20T00:10:03.257 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:7", 2024-04-20T00:10:03.257 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:9", 2024-04-20T00:10:03.257 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.2", 2024-04-20T00:10:03.257 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:9", 2024-04-20T00:10:03.257 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:4", 2024-04-20T00:10:03.257 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:5", 2024-04-20T00:10:03.257 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.1.retry", 2024-04-20T00:10:03.257 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:0", 2024-04-20T00:10:03.258 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000107", 2024-04-20T00:10:03.258 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:8", 2024-04-20T00:10:03.258 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:4", 2024-04-20T00:10:03.258 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000046", 2024-04-20T00:10:03.258 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:5", 2024-04-20T00:10:03.258 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:0", 2024-04-20T00:10:03.258 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:0", 2024-04-20T00:10:03.258 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.26", 2024-04-20T00:10:03.258 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:5", 2024-04-20T00:10:03.258 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:3", 2024-04-20T00:10:03.258 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20", 2024-04-20T00:10:03.259 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:8", 2024-04-20T00:10:03.259 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:4", 2024-04-20T00:10:03.259 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47", 2024-04-20T00:10:03.259 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:2", 2024-04-20T00:10:03.259 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.22", 2024-04-20T00:10:03.259 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:8", 2024-04-20T00:10:03.259 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:1", 2024-04-20T00:10:03.259 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.1", 2024-04-20T00:10:03.259 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:10", 2024-04-20T00:10:03.259 INFO:teuthology.orchestra.run.smithi019.stdout: "data_log.2", 2024-04-20T00:10:03.259 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:7", 2024-04-20T00:10:03.259 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:7", 2024-04-20T00:10:03.260 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000067", 2024-04-20T00:10:03.260 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:7", 2024-04-20T00:10:03.260 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:4", 2024-04-20T00:10:03.260 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:4", 2024-04-20T00:10:03.260 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29", 2024-04-20T00:10:03.260 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:8", 2024-04-20T00:10:03.260 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:7", 2024-04-20T00:10:03.260 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:0", 2024-04-20T00:10:03.260 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.9", 2024-04-20T00:10:03.260 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000086", 2024-04-20T00:10:03.260 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:7", 2024-04-20T00:10:03.261 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:2", 2024-04-20T00:10:03.261 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21", 2024-04-20T00:10:03.261 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.5", 2024-04-20T00:10:03.261 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:5", 2024-04-20T00:10:03.261 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.3.retry", 2024-04-20T00:10:03.261 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:6", 2024-04-20T00:10:03.261 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:4", 2024-04-20T00:10:03.261 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:1", 2024-04-20T00:10:03.261 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:9", 2024-04-20T00:10:03.261 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:9", 2024-04-20T00:10:03.261 INFO:teuthology.orchestra.run.smithi019.stdout: "sync.error-log.20", 2024-04-20T00:10:03.261 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24", 2024-04-20T00:10:03.262 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:3", 2024-04-20T00:10:03.262 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:10", 2024-04-20T00:10:03.262 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23", 2024-04-20T00:10:03.262 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16", 2024-04-20T00:10:03.262 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:5", 2024-04-20T00:10:03.262 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:5", 2024-04-20T00:10:03.262 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:10", 2024-04-20T00:10:03.262 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:5", 2024-04-20T00:10:03.262 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:2", 2024-04-20T00:10:03.262 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000050", 2024-04-20T00:10:03.262 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:2", 2024-04-20T00:10:03.263 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000102", 2024-04-20T00:10:03.263 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:4", 2024-04-20T00:10:03.263 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:9", 2024-04-20T00:10:03.263 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:6", 2024-04-20T00:10:03.263 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:10", 2024-04-20T00:10:03.263 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12", 2024-04-20T00:10:03.263 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:4", 2024-04-20T00:10:03.263 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:9", 2024-04-20T00:10:03.263 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:2", 2024-04-20T00:10:03.263 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:4", 2024-04-20T00:10:03.263 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:1", 2024-04-20T00:10:03.264 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:7", 2024-04-20T00:10:03.264 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000124", 2024-04-20T00:10:03.264 INFO:teuthology.orchestra.run.smithi019.stdout: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.1", 2024-04-20T00:10:03.264 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15", 2024-04-20T00:10:03.264 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:9", 2024-04-20T00:10:03.264 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:5", 2024-04-20T00:10:03.264 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:5", 2024-04-20T00:10:03.264 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:9", 2024-04-20T00:10:03.264 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:8", 2024-04-20T00:10:03.264 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000001", 2024-04-20T00:10:03.264 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:1", 2024-04-20T00:10:03.265 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000126", 2024-04-20T00:10:03.265 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:7", 2024-04-20T00:10:03.265 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000055", 2024-04-20T00:10:03.265 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:10", 2024-04-20T00:10:03.265 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:4", 2024-04-20T00:10:03.265 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:7", 2024-04-20T00:10:03.265 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000061", 2024-04-20T00:10:03.265 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:7", 2024-04-20T00:10:03.265 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:7", 2024-04-20T00:10:03.265 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:3", 2024-04-20T00:10:03.265 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000097", 2024-04-20T00:10:03.265 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:4", 2024-04-20T00:10:03.266 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:4", 2024-04-20T00:10:03.266 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:3", 2024-04-20T00:10:03.266 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:4", 2024-04-20T00:10:03.266 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:2", 2024-04-20T00:10:03.266 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000042", 2024-04-20T00:10:03.266 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:9", 2024-04-20T00:10:03.266 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:1", 2024-04-20T00:10:03.266 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:9", 2024-04-20T00:10:03.266 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:4", 2024-04-20T00:10:03.266 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:7", 2024-04-20T00:10:03.266 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:0", 2024-04-20T00:10:03.267 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:6", 2024-04-20T00:10:03.267 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:2", 2024-04-20T00:10:03.267 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:6", 2024-04-20T00:10:03.267 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000075", 2024-04-20T00:10:03.267 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:2", 2024-04-20T00:10:03.267 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000120", 2024-04-20T00:10:03.267 INFO:teuthology.orchestra.run.smithi019.stdout: "obj_delete_at_hint.0000000030", 2024-04-20T00:10:03.267 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:10", 2024-04-20T00:10:03.267 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:4", 2024-04-20T00:10:03.267 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:10" 2024-04-20T00:10:03.267 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:10:03.268 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-04-20T00:10:03.268 DEBUG:rgw_multi.tests:data checkpoint: source=test-zone1 target=test-zone2 2024-04-20T00:10:03.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:03.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:03.269 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:03.326 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:03.385 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:10:03.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:10:03.385 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004737", 2024-04-20T00:10:03.385 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:10:02.651783Z" 2024-04-20T00:10:03.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:10:03.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:10:03.385 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000008823", 2024-04-20T00:10:03.385 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:10:02.653237Z" 2024-04-20T00:10:03.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:10:03.386 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:10:03.386 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000009809", 2024-04-20T00:10:03.386 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:10:02.654389Z" 2024-04-20T00:10:03.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:10:03.386 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:10:03.386 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-04-20T00:10:03.386 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:10:02.655689Z" 2024-04-20T00:10:03.386 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:10:03.386 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:10:03.387 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:10:03.387 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T00:10:03.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:03.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:03.388 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:03.449 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:03.449 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:03.533 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:10:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:10:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:10:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:10:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:10:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:10:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:10:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:10:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:10:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:10:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:10:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:10:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:10:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:10:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:10:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:10:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:10:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:10:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:10:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:10:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:10:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:10:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:10:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:10:03Z" 2024-04-20T00:10:03.538 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:10:03.539 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:10:03Z" } 2024-04-20T00:10:03.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:10:03.539 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:10:03.540 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:10:03.540 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:10:08.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:08.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:08.541 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:08.604 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:08.604 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:08.668 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:10:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:10:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:10:08.668 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:10:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:10:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:10:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:10:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:10:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:10:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:08.669 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:10:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:10:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:10:08.670 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:10:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:10:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:10:08.671 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:10:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:10:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:10:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:08.672 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:10:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:10:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:10:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:10:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:10:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:08.673 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:10:08Z" 2024-04-20T00:10:08.673 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:10:08.674 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:10:08Z" } 2024-04-20T00:10:08.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:10:08.675 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:10:08.675 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:10:08.675 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:10:13.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:13.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:13.676 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:13.737 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:13.737 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:13.808 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:10:13.809 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:10:13.809 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:10:13.809 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:10:13.809 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:10:13.809 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:10:13.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:13.809 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:10:13.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:13.809 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:10:13.810 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:13.810 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:13.810 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:10:13.810 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:13.810 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:13.810 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:13.810 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:10:13.810 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:13.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:13.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:13.811 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:10:13.811 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:13.811 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:13.811 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:10:13.811 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:13.811 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:13.811 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:13.811 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:10:13.811 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:13.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:13.811 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:13.812 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:10:13.812 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:13.812 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:13.812 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:10:13.812 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:13.812 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:13.812 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:13.812 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:10:13.812 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:13.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:13.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:13.813 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:10:13.813 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:13.813 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:13.813 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:10:13.813 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:13.813 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:13.813 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:13.813 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:10:13.813 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:13.813 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:13.813 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:10:13.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:13.814 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:10:13.814 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:10:13.814 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:10:13.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:13.814 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:10:13Z" 2024-04-20T00:10:13.814 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:10:13.815 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:10:13Z" } 2024-04-20T00:10:13.815 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:10:13.815 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:10:13.815 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:10:13.816 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:10:18.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:18.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:18.817 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:18.956 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:10:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:10:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:10:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:10:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:10:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:10:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:10:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:10:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:18.957 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:10:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:10:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:10:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:18.958 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:10:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:10:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:10:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:18.959 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:10:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:10:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:10:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:18.960 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:10:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:10:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:10:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:10:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:10:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:10:18.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:18.962 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:10:18Z" 2024-04-20T00:10:18.962 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:10:18.962 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:10:18Z" } 2024-04-20T00:10:18.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:10:18.963 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:10:18.963 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:10:18.963 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:10:23.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:23.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:23.964 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:24.026 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:24.026 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:24.086 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:10:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:10:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:10:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:10:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:10:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:10:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:10:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:10:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:10:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:10:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:10:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:10:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:10:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:10:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:10:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:10:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:10:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:10:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:10:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:10:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:10:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:10:24.092 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:10:24.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:24.092 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:10:24Z" 2024-04-20T00:10:24.092 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:10:24.093 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:10:24Z" } 2024-04-20T00:10:24.093 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:10:24.093 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:10:24.093 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:10:24.093 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:10:29.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:29.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:29.095 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:29.160 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:29.160 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:30.258 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:10:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:10:30.258 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:10:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:10:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:10:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:10:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:10:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:10:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:10:30.259 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:10:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:10:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:10:30.260 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:10:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:10:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:10:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:30.261 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:10:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:10:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:10:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:30.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:10:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:10:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:10:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:10:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:10:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:30.263 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:10:30Z" 2024-04-20T00:10:30.263 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:10:30.264 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:10:30Z" } 2024-04-20T00:10:30.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:10:30.265 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:10:30.265 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:10:30.265 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:10:35.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:35.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:35.266 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:35.403 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:10:35.403 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:10:35.403 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:10:35.403 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:10:35.404 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:10:35.404 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:10:35.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:35.404 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:10:35.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:35.404 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:10:35.404 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:35.404 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:35.404 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:10:35.404 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:35.405 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:35.405 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:35.405 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:10:35.405 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:35.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:35.405 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:35.405 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:10:35.405 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:35.405 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:35.405 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:10:35.405 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:35.406 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:35.406 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:35.406 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:10:35.406 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:35.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:35.406 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:35.406 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:10:35.406 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:35.406 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:35.406 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:10:35.407 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:35.407 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:35.407 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:35.407 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:10:35.407 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:35.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:35.407 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:35.407 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:10:35.407 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:35.407 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:10:35.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:35.409 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:10:35Z" 2024-04-20T00:10:35.409 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:10:35.410 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:10:35Z" } 2024-04-20T00:10:35.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:10:35.410 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:10:35.410 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:10:35.410 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:10:40.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:40.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:40.412 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:40.474 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:40.474 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:40.548 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:10:40.548 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:10:40.548 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:10:40.548 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:10:40.548 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:10:40.548 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:10:40.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:40.549 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:10:40.549 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:40.549 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:10:40.549 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:40.549 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:40.549 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:10:40.549 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:40.549 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:40.549 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:40.549 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:10:40.549 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:40.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:40.550 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:40.550 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:10:40.550 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:40.550 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:40.550 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:10:40.550 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:40.550 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:40.550 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:40.550 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:10:40.550 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:40.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:40.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:40.551 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:10:40.551 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:40.551 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:40.551 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:10:40.551 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:40.551 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:40.551 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:40.551 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:10:40.551 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:10:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:10:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:10:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:40.553 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:10:40.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:40.553 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:10:40.553 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:10:40.553 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:10:40.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:40.553 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:10:40Z" 2024-04-20T00:10:40.553 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:10:40.554 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:10:40Z" } 2024-04-20T00:10:40.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:10:40.554 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:10:40.554 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:10:40.554 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:10:45.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:45.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:45.556 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:45.617 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:45.617 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:45.690 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:10:45.690 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:10:45.690 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:10:45.690 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:10:45.690 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:10:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:10:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:10:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:10:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:10:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:10:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:10:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:10:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:10:45.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:45.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:45.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:45.693 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:10:45.693 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:45.693 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:45.693 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:10:45.693 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:45.693 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:45.693 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:45.693 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:10:45.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:45.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:45.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:45.694 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:10:45.694 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:45.694 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:45.694 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:10:45.694 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:45.694 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:45.694 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:45.694 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:10:45.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:45.695 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:45.695 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:10:45.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:45.695 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:10:45.695 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:10:45.695 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:10:45.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:45.695 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:10:45Z" 2024-04-20T00:10:45.695 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:10:45.696 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:10:45Z" } 2024-04-20T00:10:45.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:10:45.697 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:10:45.697 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:10:45.697 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:10:50.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:50.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:50.698 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:50.760 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:50.761 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:50.847 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:10:50.847 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:10:50.848 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:10:50.848 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:10:50.848 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:10:50.848 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:10:50.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:50.848 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:10:50.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:50.848 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:10:50.849 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:50.849 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:50.849 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:10:50.849 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:50.849 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:50.849 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:50.850 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:10:50.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:50.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:50.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:50.850 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:10:50.850 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:50.850 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:50.850 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:10:50.850 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:50.850 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:50.851 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:50.851 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:10:50.851 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:50.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:50.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:50.851 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:10:50.851 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:50.851 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:50.852 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:10:50.852 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:50.852 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:50.852 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:50.852 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:10:50.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:50.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:50.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:50.852 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:10:50.852 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:50.852 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:10:50.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:50.854 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:10:50Z" 2024-04-20T00:10:50.854 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:10:50.855 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:10:50Z" } 2024-04-20T00:10:50.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:10:50.855 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:10:50.855 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:10:50.855 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:10:55.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:55.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:10:55.857 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:10:55.920 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:10:55.920 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:10:55.981 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:10:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:10:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:10:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:10:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:10:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:10:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:10:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:10:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:10:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:10:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:55.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:55.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:55.983 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:10:55.983 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:55.983 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:55.983 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:10:55.983 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:55.983 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:55.983 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:55.983 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:10:55.983 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:55.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:55.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:55.984 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:10:55.984 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:55.984 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:55.984 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:10:55.984 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:55.985 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:55.985 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:55.985 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:10:55.985 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:55.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:55.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:10:55.985 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:10:55.985 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:10:55.985 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:10:55.985 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:10:55.985 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:10:55.986 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:10:55.986 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:10:55.986 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:10:55.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:55.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:10:55.986 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:10:55.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:55.986 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:10:55.986 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:10:55.986 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:10:55.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:10:55.987 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:10:55Z" 2024-04-20T00:10:55.987 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:10:55.988 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:10:55Z" } 2024-04-20T00:10:55.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:10:55.988 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:10:55.988 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:10:55.989 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:11:00.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:00.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:00.990 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:11:01.051 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:11:01.051 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:11:01.129 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:11:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:11:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:11:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:11:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:11:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:11:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:11:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:11:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:01.130 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:11:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:11:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:11:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:01.131 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:11:01.132 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:11:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:11:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:11:01.133 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:11:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:11:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:11:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:11:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:11:01.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:01.135 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:11:01Z" 2024-04-20T00:11:01.135 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:11:01.135 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:11:01Z" } 2024-04-20T00:11:01.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:11:01.136 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:11:01.136 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:11:01.136 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:11:06.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:06.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:06.138 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:11:06.200 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:11:06.200 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:11:06.267 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:11:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:11:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:11:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:11:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:11:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:11:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:11:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:11:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:11:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:11:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:11:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:11:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:11:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:11:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:11:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:11:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:11:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:11:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:11:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:11:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:11:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:11:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:11:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:11:06Z" 2024-04-20T00:11:06.272 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:11:06.273 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:11:06Z" } 2024-04-20T00:11:06.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:11:06.273 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:11:06.273 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:11:06.273 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:11:11.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:11.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:11.275 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:11:11.336 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:11:11.337 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:11:11.407 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:11:11.408 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:11:11.408 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:11:11.408 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:11:11.408 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:11:11.408 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:11:11.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:11.408 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:11:11.408 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:11.409 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:11:11.409 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:11.409 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:11.409 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:11:11.409 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:11.409 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:11.409 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:11.409 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:11:11.409 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:11.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:11.410 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:11.410 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:11:11.410 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:11.410 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:11.410 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:11:11.410 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:11.410 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:11.410 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:11.410 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:11:11.410 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:11.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:11.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:11.411 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:11:11.411 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:11.411 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:11.411 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:11:11.411 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:11.411 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:11.411 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:11.411 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:11:11.411 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:11.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:11.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:11.412 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:11:11.412 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:11.412 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:11.412 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:11:11.412 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:11.412 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:11.412 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:11.412 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:11:11.412 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:11.412 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:11.413 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:11:11.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:11.413 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:11:11.413 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:11:11.413 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:11:11.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:11.413 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:11:11Z" 2024-04-20T00:11:11.413 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:11:11.414 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:11:11Z" } 2024-04-20T00:11:11.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:11:11.414 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:11:11.414 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:11:11.415 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:11:16.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:16.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:16.416 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:11:16.477 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:11:16.477 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:11:16.550 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:11:16.550 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:11:16.551 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:11:16.551 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:11:16.551 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:11:16.551 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:11:16.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:16.551 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:11:16.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:16.551 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:11:16.551 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:16.551 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:16.551 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:11:16.552 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:16.552 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:16.552 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:16.552 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:11:16.552 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:16.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:16.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:16.552 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:11:16.552 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:16.552 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:16.552 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:11:16.552 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:16.553 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:16.553 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:16.553 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:11:16.553 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:16.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:16.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:16.553 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:11:16.553 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:16.553 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:16.553 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:11:16.553 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:16.553 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:11:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:11:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:11:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:11:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:11:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:11:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:11:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:11:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:11:16Z" 2024-04-20T00:11:16.555 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:11:16.556 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:11:16Z" } 2024-04-20T00:11:16.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:11:16.557 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:11:16.557 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:11:16.557 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:11:21.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:21.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:21.558 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:11:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:11:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:11:21.690 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:11:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:11:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:11:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:11:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:11:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:11:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:21.690 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:11:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:11:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:11:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:11:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:21.691 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:11:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:11:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:11:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:21.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:11:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:11:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:11:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:21.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:11:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:11:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:11:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:21.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:11:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:11:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:11:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:11:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:21.695 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:11:21Z" 2024-04-20T00:11:21.695 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:11:21.696 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:11:21Z" } 2024-04-20T00:11:21.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:11:21.696 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:11:21.697 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:11:21.697 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:11:26.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:26.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:26.698 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:11:26.762 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:11:26.762 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:11:26.837 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:11:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:11:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:11:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:11:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:11:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:11:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:26.837 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:11:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:11:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:11:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:11:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:26.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:11:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:11:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:11:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:26.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:11:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:11:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:11:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:26.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:11:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:11:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:11:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:26.841 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:11:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:11:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:11:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:11:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:26.842 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:11:26Z" 2024-04-20T00:11:26.842 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:11:26.843 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:11:26Z" } 2024-04-20T00:11:26.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:11:26.843 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:11:26.843 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:11:26.844 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:11:31.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:31.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:31.846 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:11:31.907 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:11:31.907 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:11:31.992 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:11:31.992 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:11:31.993 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:11:31.993 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:11:31.993 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:11:31.993 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:11:31.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:31.993 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:11:31.993 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:31.993 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:11:31.993 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:31.993 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:31.993 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:11:31.994 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:31.994 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:31.994 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:31.994 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:11:31.994 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:31.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:31.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:31.994 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:11:31.994 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:31.994 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:31.994 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:11:31.995 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:31.995 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:31.995 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:31.995 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:11:31.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:31.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:31.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:31.995 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:11:31.995 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:31.995 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:31.995 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:11:31.995 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:31.996 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:31.996 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:31.996 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:11:31.996 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:31.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:31.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:31.996 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:11:31.996 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:31.996 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:31.996 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:11:31.996 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:31.996 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:31.997 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:31.997 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:11:31.997 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:31.997 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:31.997 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:11:31.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:31.997 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:11:31.997 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:11:31.997 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:11:31.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:31.997 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:11:31Z" 2024-04-20T00:11:31.998 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:11:31.998 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:11:31Z" } 2024-04-20T00:11:31.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:11:31.999 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:11:31.999 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:11:31.999 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:11:37.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:37.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:37.001 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:11:37.061 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:11:37.061 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:11:37.129 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:11:37.130 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:11:37.130 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:11:37.130 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:11:37.130 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:11:37.130 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:11:37.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:37.130 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:11:37.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:37.130 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:11:37.130 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:37.131 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:37.131 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:11:37.131 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:37.131 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:37.131 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:37.131 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:11:37.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:37.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:37.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:37.131 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:37.132 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:37.133 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:11:37.133 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:37.133 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:37.133 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:37.133 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:11:37.133 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:37.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:37.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:37.133 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:11:37.133 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:37.133 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:37.134 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:11:37.134 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:37.134 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:37.134 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:37.134 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:11:37.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:37.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:37.134 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:11:37.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:37.134 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:11:37.134 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:11:37.135 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:11:37.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:37.135 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:11:37Z" 2024-04-20T00:11:37.135 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:11:37.136 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:11:37Z" } 2024-04-20T00:11:37.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:11:37.136 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:11:37.136 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:11:37.136 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:11:42.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:42.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:42.138 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:11:42.200 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:11:42.201 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:11:42.289 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:11:42.290 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:11:42.290 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:11:42.290 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:11:42.290 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:11:42.290 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:11:42.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:42.290 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:11:42.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:42.290 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:11:42.291 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:42.291 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:42.291 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:11:42.291 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:42.291 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:42.291 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:42.291 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:11:42.291 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:42.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:42.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:42.291 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:11:42.292 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:42.292 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:42.292 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:11:42.292 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:42.292 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:42.292 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:42.292 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:11:42.292 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:42.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:42.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:42.292 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:11:42.293 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:42.293 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:42.293 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:11:42.293 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:42.293 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:42.293 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:42.293 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:11:42.293 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:42.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:42.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:42.293 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:11:42.293 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:42.294 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:42.294 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:11:42.294 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:42.294 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:42.294 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:42.294 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:11:42.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:42.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:42.294 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:11:42.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:42.295 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:11:42.295 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:11:42.295 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:11:42.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:42.295 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:11:42Z" 2024-04-20T00:11:42.295 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:11:42.296 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:11:42Z" } 2024-04-20T00:11:42.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:11:42.296 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:11:42.296 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:11:42.296 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:11:47.297 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:47.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:47.298 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:11:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:11:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:11:47.424 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:11:47.424 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:11:47.424 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:11:47.424 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:11:47.424 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:11:47.424 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:11:47.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:47.424 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:11:47.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:47.425 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:11:47.425 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:47.425 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:47.425 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:11:47.425 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:47.425 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:47.425 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:47.425 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:11:47.425 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:47.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:47.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:47.426 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:11:47.426 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:47.426 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:47.426 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:11:47.426 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:47.426 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:47.426 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:47.426 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:11:47.426 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:47.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:47.426 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:47.427 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:11:47.427 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:47.427 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:47.427 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:11:47.427 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:47.427 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:47.427 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:47.427 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:11:47.427 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:47.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:47.427 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:47.428 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:11:47.429 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:11:47.429 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:11:47.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:47.429 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:11:47Z" 2024-04-20T00:11:47.429 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:11:47.430 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:11:47Z" } 2024-04-20T00:11:47.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:11:47.430 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:11:47.430 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:11:47.430 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:11:52.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:52.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:52.432 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:11:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:11:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:11:52.576 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:11:52.577 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:11:52.577 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:11:52.577 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:11:52.577 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:11:52.577 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:11:52.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:52.577 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:11:52.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:52.577 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:11:52.577 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:52.577 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:52.578 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:11:52.578 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:52.578 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:52.578 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:52.578 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:11:52.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:52.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:52.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:52.578 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:11:52.578 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:52.578 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:52.579 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:11:52.579 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:52.579 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:52.579 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:52.579 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:11:52.579 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:52.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:52.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:52.579 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:11:52.579 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:52.579 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:52.580 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:11:52.580 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:52.580 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:52.580 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:52.580 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:11:52.580 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:52.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:52.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:52.580 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:11:52.580 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:52.581 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:52.581 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:11:52.581 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:52.581 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:52.581 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:52.581 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:11:52.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:52.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:52.581 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:11:52.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:52.581 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:11:52.581 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:11:52.582 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:11:52.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:52.582 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:11:52Z" 2024-04-20T00:11:52.582 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:11:52.583 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:11:52Z" } 2024-04-20T00:11:52.583 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:11:52.583 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:11:52.583 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:11:52.583 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:11:57.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:57.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:11:57.585 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:11:57.647 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:11:57.647 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:11:57.725 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:11:57.725 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:11:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:11:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:11:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:11:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:11:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:11:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:11:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:11:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:11:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:11:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:11:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:11:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:11:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:57.729 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:11:57.729 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:57.729 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:57.729 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:57.729 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:11:57.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:57.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:57.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:11:57.729 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:11:57.729 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:11:57.730 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:11:57.730 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:11:57.730 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:11:57.730 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:11:57.730 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:11:57.730 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:11:57.730 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:57.730 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:11:57.730 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:11:57.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:57.731 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:11:57.731 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:11:57.731 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:11:57.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:11:57.731 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:11:57Z" 2024-04-20T00:11:57.731 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:11:57.732 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:11:57Z" } 2024-04-20T00:11:57.732 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:11:57.732 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:11:57.732 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:11:57.733 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:12:02.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:02.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:02.735 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:12:02.796 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:12:02.797 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:12:02.856 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:12:02.856 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:12:02.856 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:12:02.856 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:12:02.856 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:12:02.856 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:12:02.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:02.856 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:12:02.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:02.857 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:12:02.857 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:02.857 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:02.857 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:12:02.857 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:02.857 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:02.857 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:02.857 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:12:02.857 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:02.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:02.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:02.858 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:12:02.858 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:02.858 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:02.858 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:12:02.858 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:02.858 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:02.858 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:02.858 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:12:02.858 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:02.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:02.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:02.859 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:12:02.859 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:02.859 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:02.859 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:12:02.859 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:02.859 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:02.859 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:02.859 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:12:02.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:02.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:02.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:02.860 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:12:02.860 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:02.860 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:02.860 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:12:02.860 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:02.860 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:02.860 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:02.860 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:12:02.860 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:02.860 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:02.860 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:12:02.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:02.861 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:12:02.861 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:12:02.861 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:12:02.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:02.861 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:12:02Z" 2024-04-20T00:12:02.861 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:12:02.862 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:12:02Z" } 2024-04-20T00:12:02.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:12:02.862 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:12:02.862 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:12:02.863 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:12:07.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:07.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:07.865 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:12:07.925 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:12:07.925 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:12:08.009 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:12:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:12:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:12:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:12:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:12:08.009 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:12:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:12:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:12:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:12:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:08.010 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:12:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:12:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:12:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:08.011 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:12:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:12:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:12:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:12:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:12:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:12:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:12:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:12:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:12:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:12:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:12:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:12:07Z" 2024-04-20T00:12:08.014 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:12:08.015 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:12:07Z" } 2024-04-20T00:12:08.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:12:08.016 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:12:08.016 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:12:08.016 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:12:13.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:13.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:13.017 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:12:13.079 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:12:13.079 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:12:13.146 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:12:13.146 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:12:13.146 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:12:13.147 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:12:13.147 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:12:13.147 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:12:13.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:13.147 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:12:13.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:13.147 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:12:13.147 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:13.147 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:13.147 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:12:13.147 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:13.147 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:13.148 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:13.148 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:12:13.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:13.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:13.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:13.148 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:12:13.148 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:13.148 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:13.148 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:12:13.148 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:13.148 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:13.149 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:13.149 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:12:13.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:13.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:13.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:13.149 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:12:13.149 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:13.149 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:13.149 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:12:13.149 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:13.150 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:13.150 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:13.150 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:12:13.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:13.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:13.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:13.150 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:12:13.150 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:13.150 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:13.150 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:12:13.150 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:13.151 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:13.151 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:13.151 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:12:13.151 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:13.151 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:13.151 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:12:13.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:13.151 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:12:13.151 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:12:13.151 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:12:13.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:13.151 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:12:13Z" 2024-04-20T00:12:13.152 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:12:13.152 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:12:13Z" } 2024-04-20T00:12:13.153 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:12:13.153 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:12:13.153 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:12:13.153 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:12:18.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:18.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:18.154 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:12:18.216 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:12:18.216 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:12:18.292 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:12:18.292 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:12:18.292 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:12:18.292 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:12:18.292 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:12:18.293 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:12:18.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:18.293 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:12:18.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:18.293 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:12:18.293 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:18.293 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:18.293 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:12:18.293 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:18.293 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:18.293 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:18.293 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:12:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:12:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:12:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:12:18.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:18.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:18.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:18.295 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:12:18.295 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:18.295 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:18.295 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:12:18.295 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:18.295 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:18.295 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:18.295 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:12:18.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:18.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:18.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:18.296 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:12:18.296 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:18.296 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:18.296 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:12:18.296 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:18.296 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:18.296 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:18.296 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:12:18.296 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:18.297 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:18.297 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:12:18.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:18.297 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:12:18.297 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:12:18.297 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:12:18.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:18.297 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:12:18Z" 2024-04-20T00:12:18.297 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:12:18.298 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:12:18Z" } 2024-04-20T00:12:18.298 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:12:18.298 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:12:18.299 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:12:18.299 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:12:23.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:23.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:23.300 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:12:23.361 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:12:23.362 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:12:23.429 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:12:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:12:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:12:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:12:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:12:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:12:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:12:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:12:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:12:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:12:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:12:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:12:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:12:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:12:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:12:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:12:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:12:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:12:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:12:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:12:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:12:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:12:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:12:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:12:23Z" 2024-04-20T00:12:23.434 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:12:23.435 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:12:23Z" } 2024-04-20T00:12:23.435 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:12:23.435 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:12:23.435 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:12:23.435 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:12:28.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:28.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:28.437 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:12:28.498 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:12:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:12:28.561 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:12:28.561 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:12:28.561 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:12:28.561 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:12:28.561 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:12:28.562 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:12:28.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:28.562 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:12:28.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:28.562 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:12:28.562 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:28.562 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:28.562 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:12:28.562 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:28.562 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:28.563 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:28.563 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:12:28.563 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:28.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:28.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:28.563 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:12:28.563 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:28.563 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:28.563 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:12:28.563 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:28.563 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:28.563 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:28.564 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:12:28.564 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:28.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:28.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:28.564 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:12:28.564 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:28.564 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:28.564 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:12:28.564 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:28.564 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:28.564 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:28.564 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:12:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:12:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:12:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:12:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:28.566 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:28.566 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:12:28.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:28.566 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:12:28.566 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:12:28.566 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:12:28.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:28.566 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:12:28Z" 2024-04-20T00:12:28.566 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:12:28.567 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:12:28Z" } 2024-04-20T00:12:28.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:12:28.568 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:12:28.568 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:12:28.568 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:12:33.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:33.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:33.569 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:12:33.630 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:12:33.630 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:12:33.719 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:12:33.719 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:12:33.719 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:12:33.719 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:12:33.719 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:12:33.719 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:12:33.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:33.719 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:12:33.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:33.720 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:12:33.720 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:33.720 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:33.720 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:12:33.720 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:33.720 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:33.720 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:33.720 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:12:33.720 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:33.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:33.720 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:33.721 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:12:33.721 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:33.721 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:33.721 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:12:33.721 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:33.721 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:33.721 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:33.721 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:12:33.721 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:33.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:33.721 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:33.722 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:12:33.722 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:33.722 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:33.722 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:12:33.722 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:33.722 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:33.722 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:33.722 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:12:33.722 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:33.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:33.722 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:33.723 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:12:33.724 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:12:33.724 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:12:33.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:33.724 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:12:33Z" 2024-04-20T00:12:33.724 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:12:33.725 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:12:33Z" } 2024-04-20T00:12:33.725 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:12:33.725 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:12:33.725 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:12:33.725 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:12:38.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:38.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:38.727 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:12:38.789 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:12:38.789 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:12:38.864 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:12:38.864 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:12:38.864 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:12:38.864 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:12:38.864 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:12:38.864 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:12:38.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:38.865 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:12:38.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:38.865 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:12:38.865 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:38.865 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:38.866 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:12:38.866 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:38.866 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:38.866 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:38.866 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:12:38.866 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:38.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:38.866 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:38.866 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:12:38.867 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:38.867 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:38.867 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:12:38.867 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:38.867 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:38.867 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:38.868 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:12:38.868 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:38.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:38.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:38.868 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:12:38.868 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:38.868 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:38.868 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:12:38.869 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:38.869 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:38.869 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:38.869 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:12:38.869 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:38.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:38.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:38.869 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:12:38.869 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:38.870 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:38.870 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:12:38.870 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:38.870 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:38.870 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:38.871 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:12:38.871 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:38.871 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:38.871 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:12:38.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:38.871 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:12:38.872 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:12:38.872 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:12:38.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:38.872 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:12:38Z" 2024-04-20T00:12:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:12:38.873 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:12:38Z" } 2024-04-20T00:12:38.874 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:12:38.874 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:12:38.874 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:12:38.874 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:12:43.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:43.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:43.875 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:12:43.938 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:12:43.939 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:12:44.002 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:44.003 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:44.004 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:44.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:44.006 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:44.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:44.008 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:44.009 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:12:43Z" 2024-04-20T00:12:44.010 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:12:44.011 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:12:43Z" } 2024-04-20T00:12:44.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:12:44.012 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:12:44.012 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:12:44.012 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:12:49.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:49.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:49.013 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:12:49.077 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:12:49.077 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:12:49.149 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:12:49.149 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:12:49.149 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:12:49.149 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:12:49.149 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:12:49.149 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:12:49.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:49.150 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:12:49.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:49.150 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:12:49.150 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:49.150 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:49.150 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:12:49.150 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:49.150 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:49.150 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:49.150 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:12:49.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:49.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:49.152 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:12:49.152 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:49.152 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:49.152 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:12:49.152 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:49.152 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:49.152 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:49.152 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:12:49.152 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:49.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:49.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:49.152 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:12:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:12:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:12:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:12:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:49.153 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:12:49.154 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:12:49.154 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:12:49.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:49.154 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:12:49Z" 2024-04-20T00:12:49.154 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:12:49.155 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:12:49Z" } 2024-04-20T00:12:49.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:12:49.155 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:12:49.155 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:12:49.155 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:12:54.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:54.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:54.156 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:12:54.218 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:12:54.219 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:12:54.289 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:12:54.289 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:12:54.289 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:12:54.289 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:12:54.289 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:12:54.289 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:12:54.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:54.290 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:12:54.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:54.290 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:12:54.290 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:54.290 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:54.290 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:12:54.290 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:54.290 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:54.290 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:54.290 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:12:54.291 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:54.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:54.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:54.291 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:12:54.291 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:54.291 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:54.291 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:12:54.291 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:54.291 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:54.291 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:54.291 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:12:54.291 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:54.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:54.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:54.292 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:12:54.292 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:54.292 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:54.292 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:12:54.292 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:54.292 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:54.292 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:54.292 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:12:54.292 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:12:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:12:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:12:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:54.293 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:54.294 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:12:54.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:54.294 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:12:54.294 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:12:54.294 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:12:54.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:54.294 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:12:54Z" 2024-04-20T00:12:54.294 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:12:54.295 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:12:54Z" } 2024-04-20T00:12:54.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:12:54.295 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:12:54.295 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:12:54.296 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:12:59.297 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:59.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:12:59.324 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:12:59.386 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:12:59.386 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:12:59.459 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:12:59.459 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:12:59.459 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:12:59.459 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:12:59.459 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:12:59.460 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:12:59.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:59.460 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:12:59.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:59.460 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:12:59.460 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:59.460 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:59.460 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:12:59.460 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:59.460 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:59.460 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:59.460 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:12:59.461 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:59.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:59.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:59.461 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:12:59.461 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:59.461 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:59.461 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:12:59.461 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:59.461 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:59.461 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:59.461 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:12:59.461 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:59.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:59.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:59.462 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:12:59.462 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:59.462 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:59.462 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:12:59.462 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:59.462 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:59.462 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:59.462 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:59.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:12:59.464 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:12:59.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:59.464 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:12:59.464 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:12:59.464 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:12:59.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:12:59.464 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:12:59Z" 2024-04-20T00:12:59.464 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:12:59.465 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:12:59Z" } 2024-04-20T00:12:59.465 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:12:59.465 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:12:59.466 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:12:59.466 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:13:04.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:04.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:04.467 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:13:04.528 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:13:04.528 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:13:04.607 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:13:04.607 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:13:04.607 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:13:04.607 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:13:04.607 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:13:04.607 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:13:04.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:04.608 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:13:04.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:04.608 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:13:04.608 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:04.608 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:04.608 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:13:04.608 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:04.608 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:04.608 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:04.608 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:13:04.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:04.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:04.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:04.609 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:13:04.609 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:04.609 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:04.609 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:13:04.609 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:04.609 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:04.609 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:04.609 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:13:04.609 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:04.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:04.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:04.610 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:13:04.610 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:04.610 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:04.610 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:13:04.610 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:04.610 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:04.610 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:04.610 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:13:04.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:04.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:04.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:04.611 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:13:04.611 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:04.611 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:04.611 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:13:04.611 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:04.611 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:04.611 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:04.611 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:13:04.611 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:04.611 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:04.611 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:13:04.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:04.612 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:13:04.612 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:13:04.612 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:13:04.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:04.612 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:13:04Z" 2024-04-20T00:13:04.612 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:13:04.613 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:13:04Z" } 2024-04-20T00:13:04.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:13:04.613 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:13:04.613 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:13:04.613 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:13:09.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:09.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:09.615 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:13:09.677 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:13:09.677 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:13:09.757 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:13:09.757 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:13:09.758 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:13:09.758 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:13:09.758 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:13:09.758 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:13:09.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:09.758 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:13:09.758 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:09.758 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:13:09.758 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:09.758 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:09.758 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:13:09.758 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:09.759 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:09.759 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:09.759 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:13:09.759 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:09.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:09.759 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:09.759 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:13:09.759 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:09.759 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:09.759 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:13:09.759 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:09.759 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:09.760 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:13:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:13:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:13:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:13:09.761 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:13:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:13:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:13:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:13:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:09.762 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:13:09Z" 2024-04-20T00:13:09.762 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:13:09.763 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:13:09Z" } 2024-04-20T00:13:09.763 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:13:09.763 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:13:09.763 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:13:09.763 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:13:14.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:14.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:14.767 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:13:14.829 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:13:14.829 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:13:14.901 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:13:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:13:14.901 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:13:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:13:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:13:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:13:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:13:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:13:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:13:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:14.902 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:13:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:13:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:13:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:14.903 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:13:14.904 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:14.905 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:13:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:13:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:13:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:13:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:14.906 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:13:14Z" 2024-04-20T00:13:14.906 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:13:14.907 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:13:14Z" } 2024-04-20T00:13:14.907 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:13:14.907 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:13:14.907 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:13:14.907 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:13:19.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:19.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:19.909 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:13:19.971 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:13:19.971 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:13:20.045 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:13:20.045 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:13:20.045 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:13:20.046 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:13:20.046 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:13:20.046 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:13:20.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:20.046 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:13:20.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:20.046 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:13:20.046 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:20.046 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:20.046 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:13:20.046 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:20.047 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:20.047 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:20.047 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:13:20.047 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:20.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:20.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:20.047 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:13:20.047 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:20.047 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:20.047 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:20.048 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:20.049 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:20.049 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:13:20.049 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:20.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:20.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:20.049 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:13:20.049 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:20.049 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:20.049 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:13:20.049 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:20.049 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:20.049 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:20.050 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:13:20.050 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:20.050 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:20.050 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:13:20.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:20.050 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:13:20.050 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:13:20.050 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:13:20.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:20.050 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:13:20Z" 2024-04-20T00:13:20.050 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:13:20.051 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:13:20Z" } 2024-04-20T00:13:20.051 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:13:20.052 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:13:20.052 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:13:20.052 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:13:25.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:25.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:25.053 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:13:25.115 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:13:25.115 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:13:25.179 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:13:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:13:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:13:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:13:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:13:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:13:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:13:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:13:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:13:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:13:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:13:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:13:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:13:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:13:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:13:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:25.183 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:25.183 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:25.183 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:13:25.183 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:25.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:25.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:25.183 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:13:25.183 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:25.183 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:25.183 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:25.184 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:13:25Z" 2024-04-20T00:13:25.185 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:13:25.185 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:13:25Z" } 2024-04-20T00:13:25.186 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:13:25.186 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:13:25.186 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:13:25.186 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:13:30.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:30.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:30.188 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:13:30.251 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:13:30.251 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:13:30.346 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:13:30.347 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:13:30.347 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:13:30.347 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:13:30.347 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:13:30.347 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:13:30.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:30.347 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:13:30.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:30.347 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:13:30.347 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:30.348 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:30.348 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:13:30.348 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:30.348 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:30.348 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:30.348 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:13:30.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:30.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:30.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:30.348 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:13:30.348 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:30.348 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:13:30.349 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:30.350 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:30.350 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:30.350 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:13:30.350 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:30.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:30.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:30.350 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:13:30.350 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:30.350 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:30.350 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:13:30.351 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:30.351 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:30.351 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:30.351 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:13:30.351 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:30.351 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:30.351 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:13:30.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:30.351 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:13:30.351 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:13:30.351 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:13:30.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:30.352 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:13:30Z" 2024-04-20T00:13:30.352 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:13:30.352 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:13:30Z" } 2024-04-20T00:13:30.353 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:13:30.353 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:13:30.353 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:13:30.353 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:13:35.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:35.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:35.354 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:13:35.416 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:13:35.416 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:13:35.486 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:13:35.487 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:13:35.487 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:13:35.487 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:13:35.487 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:13:35.487 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:13:35.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:35.487 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:13:35.487 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:35.487 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:13:35.487 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:35.488 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:35.488 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:13:35.488 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:35.488 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:35.488 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:35.488 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:13:35.488 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:35.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:35.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:35.488 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:13:35.488 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:35.488 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:35.489 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:13:35.489 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:35.489 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:35.489 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:35.489 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:13:35.489 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:35.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:35.489 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:35.489 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:13:35.489 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:35.489 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:35.490 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:13:35.490 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:35.490 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:35.490 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:35.490 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:13:35.490 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:35.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:35.490 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:35.490 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:13:35.490 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:35.490 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:13:35.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:35.492 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:13:35Z" 2024-04-20T00:13:35.492 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:13:35.492 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:13:35Z" } 2024-04-20T00:13:35.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:13:35.493 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:13:35.493 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:13:35.493 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:13:40.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:40.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:40.494 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:13:40.556 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:13:40.556 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:13:40.628 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:13:40.628 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:13:40.628 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:13:40.629 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:13:40.629 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:13:40.629 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:13:40.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:40.629 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:13:40.629 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:40.629 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:13:40.629 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:40.629 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:40.629 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:13:40.629 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:40.630 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:40.630 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:40.630 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:13:40.630 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:40.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:40.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:40.630 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:13:40.630 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:40.630 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:40.630 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:13:40.630 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:40.631 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:40.631 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:40.631 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:13:40.631 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:40.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:40.631 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:40.631 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:13:40.631 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:40.631 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:40.631 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:13:40.631 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:40.632 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:40.632 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:40.632 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:13:40.632 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:40.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:40.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:40.632 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:13:40.632 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:40.632 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:40.632 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:13:40.633 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:40.633 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:40.633 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:40.633 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:13:40.633 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:40.633 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:40.633 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:13:40.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:40.633 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:13:40.633 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:13:40.633 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:13:40.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:40.634 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:13:40Z" 2024-04-20T00:13:40.634 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:13:40.635 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:13:40Z" } 2024-04-20T00:13:40.635 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:13:40.635 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:13:40.635 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:13:40.635 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:13:45.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:45.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:45.637 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:13:45.699 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:13:45.699 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:13:45.769 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:13:45.769 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:13:45.769 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:13:45.769 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:13:45.769 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:13:45.769 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:13:45.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:45.770 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:13:45.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:45.770 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:13:45.770 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:45.770 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:45.770 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:13:45.770 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:45.770 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:45.770 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:45.770 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:13:45.770 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:45.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:45.771 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:45.771 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:13:45.771 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:45.771 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:45.771 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:13:45.771 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:45.771 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:45.771 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:45.771 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:13:45.771 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:13:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:13:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:13:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:45.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:13:45.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:45.774 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:13:45.774 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:13:45.774 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:13:45.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:45.774 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:13:45Z" 2024-04-20T00:13:45.774 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:13:45.775 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:13:45Z" } 2024-04-20T00:13:45.775 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:13:45.775 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:13:45.775 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:13:45.775 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:13:50.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:50.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:50.777 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:13:50.840 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:13:50.840 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:13:50.913 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:13:50.913 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:13:50.913 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:13:50.913 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:13:50.913 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:13:50.913 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:13:50.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:50.913 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:13:50.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:50.913 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:13:50.913 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:50.914 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:50.914 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:13:50.914 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:50.914 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:50.914 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:50.914 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:13:50.914 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:50.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:50.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:50.914 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:13:50.914 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:50.914 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:13:50.915 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:50.916 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:50.916 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:50.916 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:13:50.916 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:50.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:50.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:50.916 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:13:50.916 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:50.916 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:50.916 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:13:50.916 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:50.916 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:50.917 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:50.917 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:13:50.917 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:50.917 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:50.917 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:13:50.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:50.917 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:13:50.917 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:13:50.917 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:13:50.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:50.917 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:13:50Z" 2024-04-20T00:13:50.917 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:13:50.918 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:13:50Z" } 2024-04-20T00:13:50.918 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:13:50.919 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:13:50.919 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:13:50.919 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:13:55.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:55.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:13:55.920 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:13:55.982 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:13:55.982 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:13:56.049 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:13:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:13:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:13:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:13:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:13:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:13:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:13:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:56.050 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:13:56.050 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:56.050 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:56.050 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:13:56.050 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:56.050 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:56.050 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:56.050 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:13:56.050 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:56.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:56.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:56.050 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:13:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:13:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:13:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:13:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:56.052 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:56.052 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:13:56.052 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:56.052 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:56.052 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:56.052 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:13:56.052 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:56.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:56.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:13:56.052 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:13:56.052 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:13:56.053 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:13:56.053 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:13:56.053 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:13:56.053 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:13:56.053 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:13:56.053 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:13:56.053 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:56.053 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:13:56.054 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:13:56.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:56.054 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:13:56.054 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:13:56.054 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:13:56.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:13:56.054 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:13:56Z" 2024-04-20T00:13:56.054 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:13:56.055 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:13:56Z" } 2024-04-20T00:13:56.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:13:56.055 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:13:56.056 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:13:56.056 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:14:01.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:01.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:01.056 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:14:01.117 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:14:01.117 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:14:01.192 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:14:01.192 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:14:01.192 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:14:01.192 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:14:01.192 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:14:01.192 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:14:01.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:01.192 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:14:01.192 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:01.193 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:14:01.193 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:01.193 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:01.193 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:14:01.193 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:01.193 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:01.193 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:01.193 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:14:01.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:01.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:01.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:01.194 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:14:01.194 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:01.194 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:01.194 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:14:01.194 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:01.194 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:01.194 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:01.194 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:01.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:01.196 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:14:01.196 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:01.196 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:01.196 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:14:01.196 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:01.196 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:01.196 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:01.196 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:14:01.196 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:01.197 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:01.197 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:14:01.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:01.197 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:14:01.197 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:14:01.197 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:14:01.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:01.197 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:14:01Z" 2024-04-20T00:14:01.197 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:14:01.198 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:14:01Z" } 2024-04-20T00:14:01.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:14:01.199 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:14:01.199 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:14:01.199 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:14:06.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:06.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:06.201 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:14:06.262 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:14:06.262 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:14:06.332 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:14:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:14:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:14:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:14:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:14:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:14:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:14:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:14:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:14:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:14:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:14:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:14:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:14:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:14:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:14:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:14:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:14:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:14:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:14:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:14:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:14:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:14:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:14:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:14:06Z" 2024-04-20T00:14:06.338 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:14:06.339 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:14:06Z" } 2024-04-20T00:14:06.340 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:14:06.340 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:14:06.340 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:14:06.340 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:14:11.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:11.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:11.341 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:14:11.403 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:14:11.403 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:14:11.466 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:14:11.466 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:14:11.467 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:14:11.467 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:14:11.467 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:14:11.467 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:14:11.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:11.467 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:14:11.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:11.467 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:14:11.467 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:11.467 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:11.467 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:14:11.467 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:11.468 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:14:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:14:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:14:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:11.469 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:14:11.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:11.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:11.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:11.470 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:14:11.470 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:11.470 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:11.470 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:14:11.470 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:11.470 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:11.470 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:11.470 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:14:11.471 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:11.471 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:11.471 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:14:11.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:11.471 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:14:11.471 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:14:11.471 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:14:11.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:11.471 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:14:11Z" 2024-04-20T00:14:11.471 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:14:11.472 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:14:11Z" } 2024-04-20T00:14:11.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:14:11.473 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:14:11.473 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:14:11.473 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:14:16.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:16.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:16.474 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:14:16.535 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:14:16.535 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:14:16.599 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:14:16.599 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:14:16.599 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:14:16.600 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:14:16.600 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:14:16.600 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:14:16.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:16.600 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:14:16.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:16.600 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:14:16.601 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:16.601 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:16.601 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:14:16.601 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:16.601 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:16.601 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:16.601 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:14:16.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:16.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:16.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:16.602 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:14:16.602 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:16.602 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:16.602 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:14:16.602 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:16.602 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:16.602 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:16.602 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:14:16.602 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:16.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:16.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:16.603 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:14:16.603 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:16.603 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:16.603 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:14:16.603 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:16.603 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:16.603 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:16.603 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:14:16.603 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:16.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:16.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:16.604 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:14:16.604 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:16.604 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:16.604 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:14:16.604 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:16.604 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:16.604 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:16.604 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:14:16.604 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:16.604 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:16.605 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:14:16.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:16.605 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:14:16.605 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:14:16.605 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:14:16.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:16.605 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:14:16Z" 2024-04-20T00:14:16.605 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:14:16.606 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:14:16Z" } 2024-04-20T00:14:16.606 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:14:16.606 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:14:16.606 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:14:16.607 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:14:21.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:21.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:21.608 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:14:21.670 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:14:21.670 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:14:21.738 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:14:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:14:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:14:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:14:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:14:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:14:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:14:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:14:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:14:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:14:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:14:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:14:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:14:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:14:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:14:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:14:21Z" 2024-04-20T00:14:21.742 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:14:21.743 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:14:21Z" } 2024-04-20T00:14:21.743 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:14:21.744 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:14:21.744 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:14:21.744 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:14:26.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:26.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:26.746 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:14:26.806 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:14:26.806 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:14:26.889 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:14:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:14:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:14:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:14:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:14:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:14:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:14:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:14:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:14:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:14:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:14:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:14:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:14:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:14:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:14:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:14:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:14:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:14:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:14:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:14:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:14:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:14:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:14:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:26.894 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:14:26Z" 2024-04-20T00:14:26.894 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:14:26.895 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:14:26Z" } 2024-04-20T00:14:26.895 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:14:26.895 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:14:26.895 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:14:26.895 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:14:31.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:31.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:31.897 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:14:31.960 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:14:31.960 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:14:33.027 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:14:33.028 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:14:33.028 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:14:33.028 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:14:33.028 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:14:33.028 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:14:33.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:33.028 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:14:33.028 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:33.028 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:14:33.028 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:33.028 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:33.029 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:14:33.029 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:33.029 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:33.029 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:33.029 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:14:33.029 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:33.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:33.029 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:33.029 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:14:33.029 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:33.029 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:33.029 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:14:33.030 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:33.030 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:33.030 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:33.030 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:14:33.030 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:33.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:33.030 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:33.030 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:14:33.030 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:33.030 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:33.030 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:14:33.030 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:33.031 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:33.031 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:33.031 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:14:33.031 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:33.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:33.031 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:33.031 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:14:33.031 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:33.031 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:33.031 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:14:33.031 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:33.031 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:33.032 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:33.032 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:14:33.032 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:33.032 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:33.032 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:14:33.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:33.032 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:14:33.032 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:14:33.032 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:14:33.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:33.032 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:14:33Z" 2024-04-20T00:14:33.032 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:14:33.033 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:14:33Z" } 2024-04-20T00:14:33.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:14:33.034 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:14:33.034 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:14:33.034 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:14:38.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:38.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:38.036 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:14:38.098 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:14:38.098 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:14:38.176 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:14:38.176 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:14:38.176 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:14:38.176 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:14:38.176 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:14:38.176 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:14:38.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:38.176 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:14:38.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:38.176 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:14:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:14:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:14:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:14:38.177 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:38.178 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:38.178 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:14:38.178 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:38.178 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:38.178 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:38.178 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:14:38.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:38.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:38.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:38.178 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:14:38.178 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:38.178 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:14:38.179 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:14:38Z" 2024-04-20T00:14:38.180 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:14:38.181 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:14:38Z" } 2024-04-20T00:14:38.181 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:14:38.182 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:14:38.182 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:14:38.182 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:14:43.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:43.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:43.183 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:14:43.245 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:14:43.245 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:14:43.322 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:14:43.322 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:14:43.322 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:14:43.322 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:14:43.323 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:14:43.323 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:14:43.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:43.323 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:14:43.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:43.323 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:14:43.323 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:43.323 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:43.323 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:14:43.323 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:43.324 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:43.324 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:43.324 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:14:43.324 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:43.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:43.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:43.324 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:14:43.324 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:43.324 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:43.324 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:14:43.324 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:43.325 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:43.325 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:43.325 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:14:43.325 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:43.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:43.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:43.325 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:14:43.325 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:43.325 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:43.325 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:14:43.325 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:14:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:14:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:43.326 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:14:43.327 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:43.327 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:43.327 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:43.327 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:14:43.327 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:43.327 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:43.327 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:14:43.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:43.327 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:14:43.327 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:14:43.327 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:14:43.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:43.328 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:14:43Z" 2024-04-20T00:14:43.328 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:14:43.329 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:14:43Z" } 2024-04-20T00:14:43.329 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:14:43.329 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:14:43.329 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:14:43.329 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:14:48.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:48.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:48.331 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:14:48.392 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:14:48.392 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:14:48.459 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:14:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:14:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:14:48.459 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:14:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:14:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:14:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:14:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:14:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:14:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:48.460 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:14:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:14:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:14:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:14:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:14:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:14:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:14:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:14:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:14:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:14:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:14:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:14:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:14:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:14:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:14:48Z" 2024-04-20T00:14:48.464 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:14:48.465 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:14:48Z" } 2024-04-20T00:14:48.465 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:14:48.466 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:14:48.466 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:14:48.466 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:14:53.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:53.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:53.469 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:14:53.529 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:14:53.530 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:14:53.608 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:14:53.609 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:14:53.609 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:14:53.609 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:14:53.609 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:14:53.609 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:14:53.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:53.609 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:14:53.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:53.609 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:14:53.609 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:53.609 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:53.610 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:14:53.610 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:53.610 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:53.611 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:53.613 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:14:53.614 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:53.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:53.614 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:53.614 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:14:53.614 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:53.614 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:53.614 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:14:53.614 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:53.614 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:53.614 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:53.614 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:14:53.615 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:53.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:53.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:53.615 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:14:53.615 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:53.615 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:53.615 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:14:53.615 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:53.615 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:53.615 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:53.615 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:14:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:14:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:14:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:14:53.616 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:14:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:14:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:14:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:14:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:53.617 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:14:53Z" 2024-04-20T00:14:53.617 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:14:53.618 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:14:53Z" } 2024-04-20T00:14:53.618 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:14:53.619 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:14:53.619 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:14:53.619 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:14:58.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:58.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:14:58.621 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:14:58.683 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:14:58.683 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:14:58.760 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:14:58.761 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:14:58.761 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:14:58.761 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:14:58.761 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:14:58.761 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:14:58.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:58.761 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:14:58.761 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:58.762 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:14:58.762 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:58.762 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:58.762 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:14:58.762 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:58.762 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:58.762 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:58.762 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:14:58.762 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:58.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:58.763 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:58.763 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:14:58.763 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:58.763 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:58.763 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:14:58.763 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:58.763 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:58.763 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:58.763 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:14:58.763 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:58.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:58.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:58.764 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:14:58.764 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:58.764 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:58.764 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:14:58.764 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:58.764 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:58.764 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:58.764 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:14:58.764 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:58.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:58.765 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:14:58.765 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:14:58.765 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:14:58.765 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:14:58.765 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:14:58.765 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:14:58.765 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:14:58.765 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:14:58.765 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:14:58.765 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:58.765 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:14:58.766 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:14:58.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:58.766 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:14:58.766 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:14:58.766 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:14:58.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:14:58.766 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:14:58Z" 2024-04-20T00:14:58.766 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:14:58.767 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:14:58Z" } 2024-04-20T00:14:58.767 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:14:58.767 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:14:58.768 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:14:58.768 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:15:03.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:15:03.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:15:03.770 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:15:03.831 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:15:03.832 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:15:03.893 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:15:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:15:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:15:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:15:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:15:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:15:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:15:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:15:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:15:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:15:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:15:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:15:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:15:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:15:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:15:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:15:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:15:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:15:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:15:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:15:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:15:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:15:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:15:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:15:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:15:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:15:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:15:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:15:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:15:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:15:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:15:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:15:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:15:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:15:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:15:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:15:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:15:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:15:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:15:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:15:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:15:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:15:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:15:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:15:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:15:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:15:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:15:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:15:03Z" 2024-04-20T00:15:03.898 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:15:03.899 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:15:03Z" } 2024-04-20T00:15:03.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:15:03.900 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:15:03.900 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:15:03.900 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:15:08.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:15:08.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:15:08.901 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:15:08.963 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:15:08.963 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:15:09.049 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:15:09.050 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:15:09.050 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:15:09.050 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:15:09.050 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:15:09.050 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 6925600971813139478 2024-04-20T00:15:09.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:09.050 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:15:09.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:15:09.050 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:15:09.050 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:15:09.051 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:15:09.051 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T00:15:09.051 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:15:09.051 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:15:09.051 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:15:09.051 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T00:15:09.051 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:09.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:09.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:15:09.051 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:15:09.051 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:15:09.052 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:15:09.052 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T00:15:09.052 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:15:09.052 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:15:09.052 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:15:09.052 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T00:15:09.052 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:09.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:09.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:15:09.052 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:15:09.052 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:15:09.052 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:15:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T00:15:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:15:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:15:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:15:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T00:15:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:15:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:15:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:15:09.053 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T00:15:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T00:15:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:15:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:15:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:15:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T00:15:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:15:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:15:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 0, 2024-04-20T00:15:09.055 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 0 2024-04-20T00:15:09.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:09.055 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:15:09Z" 2024-04-20T00:15:09.055 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:15:09.056 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6925600971813139478 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003896", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:33:20.691358Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007562", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:54:37.707798Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008408", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:36:38.704801Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002914", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:26:44.680751Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T00:15:09Z" } 2024-04-20T00:15:09.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T00:15:09.056 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T00:15:09.056 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T00:15:09.056 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T00:15:14.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... FAIL 2024-04-20T00:15:14.067 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-58 2024-04-20T00:15:14.068 DEBUG:boto:path=/aktlji-58/ 2024-04-20T00:15:14.068 DEBUG:boto:auth_path=/aktlji-58/ 2024-04-20T00:15:14.068 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.068 DEBUG:boto:Path: /aktlji-58/ 2024-04-20T00:15:14.068 DEBUG:boto:Data: 2024-04-20T00:15:14.068 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.068 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:15:14.068 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.069 DEBUG:boto:Params: {} 2024-04-20T00:15:14.069 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:15:14.069 DEBUG:boto:Token: None 2024-04-20T00:15:14.069 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:15:14 GMT /aktlji-58/ 2024-04-20T00:15:14.070 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Uw1N46Oi7wLiRVxDI2tTX4XvDIY= 2024-04-20T00:15:14.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Uw1N46Oi7wLiRVxDI2tTX4XvDIY=', 'Content-Length': '0'} 2024-04-20T00:15:14.133 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000507b64efe8535686-0066230912-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.134 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-59 2024-04-20T00:15:14.134 DEBUG:boto:path=/aktlji-59/ 2024-04-20T00:15:14.134 DEBUG:boto:auth_path=/aktlji-59/ 2024-04-20T00:15:14.134 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.134 DEBUG:boto:Path: /aktlji-59/ 2024-04-20T00:15:14.134 DEBUG:boto:Data: 2024-04-20T00:15:14.134 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.135 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:15:14.135 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.135 DEBUG:boto:Params: {} 2024-04-20T00:15:14.135 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:15:14.135 DEBUG:boto:Token: None 2024-04-20T00:15:14.135 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:15:14 GMT /aktlji-59/ 2024-04-20T00:15:14.135 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4/lrIrhRBl6B3Dj5e7mUWPuBnGA= 2024-04-20T00:15:14.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4/lrIrhRBl6B3Dj5e7mUWPuBnGA=', 'Content-Length': '0'} 2024-04-20T00:15:14.212 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a34385aebf0907c-0066230912-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.212 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-60 2024-04-20T00:15:14.212 DEBUG:boto:path=/aktlji-60/ 2024-04-20T00:15:14.212 DEBUG:boto:auth_path=/aktlji-60/ 2024-04-20T00:15:14.212 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.212 DEBUG:boto:Path: /aktlji-60/ 2024-04-20T00:15:14.213 DEBUG:boto:Data: 2024-04-20T00:15:14.213 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.213 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:15:14.213 DEBUG:boto:Port: 8001 2024-04-20T00:15:14.213 DEBUG:boto:Params: {} 2024-04-20T00:15:14.213 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T00:15:14.213 DEBUG:boto:Token: None 2024-04-20T00:15:14.213 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:15:14 GMT /aktlji-60/ 2024-04-20T00:15:14.214 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m893jTxAdDaNyLk47Ic1QlhqQOQ= 2024-04-20T00:15:14.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m893jTxAdDaNyLk47Ic1QlhqQOQ=', 'Content-Length': '0'} 2024-04-20T00:15:14.295 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008e735612b264b681-0066230912-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.296 DEBUG:boto:path=/aktlji-58/ 2024-04-20T00:15:14.296 DEBUG:boto:auth_path=/aktlji-58/ 2024-04-20T00:15:14.296 DEBUG:boto:Method: HEAD 2024-04-20T00:15:14.296 DEBUG:boto:Path: /aktlji-58/ 2024-04-20T00:15:14.296 DEBUG:boto:Data: 2024-04-20T00:15:14.296 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.296 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:15:14.296 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.296 DEBUG:boto:Params: {} 2024-04-20T00:15:14.296 DEBUG:boto:Token: None 2024-04-20T00:15:14.297 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:15:14 GMT /aktlji-58/ 2024-04-20T00:15:14.297 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU= 2024-04-20T00:15:14.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU=', 'Content-Length': '0'} 2024-04-20T00:15:14.301 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069d279ab4b1b6137-0066230912-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.301 DEBUG:boto:path=/aktlji-58/obj1 2024-04-20T00:15:14.301 DEBUG:boto:auth_path=/aktlji-58/obj1 2024-04-20T00:15:14.301 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.301 DEBUG:boto:Path: /aktlji-58/obj1 2024-04-20T00:15:14.301 DEBUG:boto:Data: 2024-04-20T00:15:14.301 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:15:14.302 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:15:14.302 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.302 DEBUG:boto:Params: {} 2024-04-20T00:15:14.302 DEBUG:boto:Token: None 2024-04-20T00:15:14.302 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:15:14 GMT /aktlji-58/obj1 2024-04-20T00:15:14.302 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sJ3b+TWcUcWSkJyQx4jDQjJNMJw= 2024-04-20T00:15:14.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sJ3b+TWcUcWSkJyQx4jDQjJNMJw='} 2024-04-20T00:15:14.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af7857bfcdb35ead-0066230912-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.310 DEBUG:boto:path=/aktlji-58/ 2024-04-20T00:15:14.310 DEBUG:boto:auth_path=/aktlji-58/ 2024-04-20T00:15:14.310 DEBUG:boto:Method: HEAD 2024-04-20T00:15:14.310 DEBUG:boto:Path: /aktlji-58/ 2024-04-20T00:15:14.310 DEBUG:boto:Data: 2024-04-20T00:15:14.310 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.311 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:15:14.311 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.311 DEBUG:boto:Params: {} 2024-04-20T00:15:14.311 DEBUG:boto:Token: None 2024-04-20T00:15:14.311 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:15:14 GMT /aktlji-58/ 2024-04-20T00:15:14.311 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU= 2024-04-20T00:15:14.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU=', 'Content-Length': '0'} 2024-04-20T00:15:14.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092e18a1c5ee0e3f1-0066230912-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.314 DEBUG:boto:path=/aktlji-58/obj2 2024-04-20T00:15:14.314 DEBUG:boto:auth_path=/aktlji-58/obj2 2024-04-20T00:15:14.314 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.314 DEBUG:boto:Path: /aktlji-58/obj2 2024-04-20T00:15:14.314 DEBUG:boto:Data: 2024-04-20T00:15:14.314 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:15:14.314 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:15:14.314 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.314 DEBUG:boto:Params: {} 2024-04-20T00:15:14.314 DEBUG:boto:Token: None 2024-04-20T00:15:14.315 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:15:14 GMT /aktlji-58/obj2 2024-04-20T00:15:14.315 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FgfpwYYvfUFkYrXj7q9NKOEpJB0= 2024-04-20T00:15:14.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FgfpwYYvfUFkYrXj7q9NKOEpJB0='} 2024-04-20T00:15:14.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000570c3b0cebc218a4-0066230912-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.321 DEBUG:boto:path=/aktlji-58/ 2024-04-20T00:15:14.321 DEBUG:boto:auth_path=/aktlji-58/ 2024-04-20T00:15:14.321 DEBUG:boto:Method: HEAD 2024-04-20T00:15:14.321 DEBUG:boto:Path: /aktlji-58/ 2024-04-20T00:15:14.321 DEBUG:boto:Data: 2024-04-20T00:15:14.321 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.321 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:15:14.321 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.322 DEBUG:boto:Params: {} 2024-04-20T00:15:14.322 DEBUG:boto:Token: None 2024-04-20T00:15:14.322 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:15:14 GMT /aktlji-58/ 2024-04-20T00:15:14.322 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU= 2024-04-20T00:15:14.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU=', 'Content-Length': '0'} 2024-04-20T00:15:14.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ec78a372168cc2d-0066230912-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.324 DEBUG:boto:path=/aktlji-58/obj3 2024-04-20T00:15:14.324 DEBUG:boto:auth_path=/aktlji-58/obj3 2024-04-20T00:15:14.324 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.324 DEBUG:boto:Path: /aktlji-58/obj3 2024-04-20T00:15:14.325 DEBUG:boto:Data: 2024-04-20T00:15:14.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:15:14.325 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:15:14.325 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.325 DEBUG:boto:Params: {} 2024-04-20T00:15:14.325 DEBUG:boto:Token: None 2024-04-20T00:15:14.325 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:15:14 GMT /aktlji-58/obj3 2024-04-20T00:15:14.325 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kbB12d9lIrLPU8TWMs2RNHzo1eo= 2024-04-20T00:15:14.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kbB12d9lIrLPU8TWMs2RNHzo1eo='} 2024-04-20T00:15:14.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f480254e4028d4e-0066230912-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.371 DEBUG:boto:path=/aktlji-58/ 2024-04-20T00:15:14.371 DEBUG:boto:auth_path=/aktlji-58/ 2024-04-20T00:15:14.371 DEBUG:boto:Method: HEAD 2024-04-20T00:15:14.371 DEBUG:boto:Path: /aktlji-58/ 2024-04-20T00:15:14.371 DEBUG:boto:Data: 2024-04-20T00:15:14.371 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.371 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:15:14.372 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.372 DEBUG:boto:Params: {} 2024-04-20T00:15:14.372 DEBUG:boto:Token: None 2024-04-20T00:15:14.372 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:15:14 GMT /aktlji-58/ 2024-04-20T00:15:14.372 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU= 2024-04-20T00:15:14.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU=', 'Content-Length': '0'} 2024-04-20T00:15:14.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000188e84a987157055-0066230912-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.375 DEBUG:boto:path=/aktlji-58/obj4 2024-04-20T00:15:14.375 DEBUG:boto:auth_path=/aktlji-58/obj4 2024-04-20T00:15:14.375 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.375 DEBUG:boto:Path: /aktlji-58/obj4 2024-04-20T00:15:14.375 DEBUG:boto:Data: 2024-04-20T00:15:14.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:15:14.376 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:15:14.376 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.376 DEBUG:boto:Params: {} 2024-04-20T00:15:14.376 DEBUG:boto:Token: None 2024-04-20T00:15:14.376 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:15:14 GMT /aktlji-58/obj4 2024-04-20T00:15:14.376 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8l4azwgngBPXs5a+64kq1SIe90E= 2024-04-20T00:15:14.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8l4azwgngBPXs5a+64kq1SIe90E='} 2024-04-20T00:15:14.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0a517c18ae3a4d0-0066230912-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.382 DEBUG:boto:path=/aktlji-59/ 2024-04-20T00:15:14.383 DEBUG:boto:auth_path=/aktlji-59/ 2024-04-20T00:15:14.383 DEBUG:boto:Method: HEAD 2024-04-20T00:15:14.383 DEBUG:boto:Path: /aktlji-59/ 2024-04-20T00:15:14.383 DEBUG:boto:Data: 2024-04-20T00:15:14.383 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.383 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:15:14.383 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.383 DEBUG:boto:Params: {} 2024-04-20T00:15:14.383 DEBUG:boto:Token: None 2024-04-20T00:15:14.383 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:15:14 GMT /aktlji-59/ 2024-04-20T00:15:14.384 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0= 2024-04-20T00:15:14.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0=', 'Content-Length': '0'} 2024-04-20T00:15:14.386 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ebd2cf450b1d602-0066230912-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.387 DEBUG:boto:path=/aktlji-59/obj1 2024-04-20T00:15:14.387 DEBUG:boto:auth_path=/aktlji-59/obj1 2024-04-20T00:15:14.387 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.387 DEBUG:boto:Path: /aktlji-59/obj1 2024-04-20T00:15:14.387 DEBUG:boto:Data: 2024-04-20T00:15:14.387 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:15:14.387 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:15:14.387 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.387 DEBUG:boto:Params: {} 2024-04-20T00:15:14.388 DEBUG:boto:Token: None 2024-04-20T00:15:14.388 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:15:14 GMT /aktlji-59/obj1 2024-04-20T00:15:14.388 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Q21Hu9dIGS8vlo9n1Hx4g4zHI28= 2024-04-20T00:15:14.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q21Hu9dIGS8vlo9n1Hx4g4zHI28='} 2024-04-20T00:15:14.397 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091d8749dec40f0d0-0066230912-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.397 DEBUG:boto:path=/aktlji-59/ 2024-04-20T00:15:14.397 DEBUG:boto:auth_path=/aktlji-59/ 2024-04-20T00:15:14.397 DEBUG:boto:Method: HEAD 2024-04-20T00:15:14.397 DEBUG:boto:Path: /aktlji-59/ 2024-04-20T00:15:14.397 DEBUG:boto:Data: 2024-04-20T00:15:14.397 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.398 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:15:14.398 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.398 DEBUG:boto:Params: {} 2024-04-20T00:15:14.398 DEBUG:boto:Token: None 2024-04-20T00:15:14.398 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:15:14 GMT /aktlji-59/ 2024-04-20T00:15:14.398 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0= 2024-04-20T00:15:14.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0=', 'Content-Length': '0'} 2024-04-20T00:15:14.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a838b0be22d70aa9-0066230912-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.400 DEBUG:boto:path=/aktlji-59/obj2 2024-04-20T00:15:14.401 DEBUG:boto:auth_path=/aktlji-59/obj2 2024-04-20T00:15:14.401 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.401 DEBUG:boto:Path: /aktlji-59/obj2 2024-04-20T00:15:14.401 DEBUG:boto:Data: 2024-04-20T00:15:14.401 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:15:14.401 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:15:14.401 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.401 DEBUG:boto:Params: {} 2024-04-20T00:15:14.401 DEBUG:boto:Token: None 2024-04-20T00:15:14.401 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:15:14 GMT /aktlji-59/obj2 2024-04-20T00:15:14.402 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DZpeP8HndGLAoEcoKg21G8e8XIY= 2024-04-20T00:15:14.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DZpeP8HndGLAoEcoKg21G8e8XIY='} 2024-04-20T00:15:14.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f9f7efc2217011c-0066230912-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.410 DEBUG:boto:path=/aktlji-59/ 2024-04-20T00:15:14.410 DEBUG:boto:auth_path=/aktlji-59/ 2024-04-20T00:15:14.410 DEBUG:boto:Method: HEAD 2024-04-20T00:15:14.410 DEBUG:boto:Path: /aktlji-59/ 2024-04-20T00:15:14.410 DEBUG:boto:Data: 2024-04-20T00:15:14.410 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.410 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:15:14.410 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.410 DEBUG:boto:Params: {} 2024-04-20T00:15:14.411 DEBUG:boto:Token: None 2024-04-20T00:15:14.411 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:15:14 GMT /aktlji-59/ 2024-04-20T00:15:14.411 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0= 2024-04-20T00:15:14.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0=', 'Content-Length': '0'} 2024-04-20T00:15:14.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e0fa88933c781eb-0066230912-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.413 DEBUG:boto:path=/aktlji-59/obj3 2024-04-20T00:15:14.413 DEBUG:boto:auth_path=/aktlji-59/obj3 2024-04-20T00:15:14.414 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.414 DEBUG:boto:Path: /aktlji-59/obj3 2024-04-20T00:15:14.414 DEBUG:boto:Data: 2024-04-20T00:15:14.414 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:15:14.414 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:15:14.414 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.414 DEBUG:boto:Params: {} 2024-04-20T00:15:14.414 DEBUG:boto:Token: None 2024-04-20T00:15:14.414 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:15:14 GMT /aktlji-59/obj3 2024-04-20T00:15:14.414 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ULEgHV1bOeMrE52BEz529L17xsw= 2024-04-20T00:15:14.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ULEgHV1bOeMrE52BEz529L17xsw='} 2024-04-20T00:15:14.459 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009da038568d2a0d8-0066230912-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.459 DEBUG:boto:path=/aktlji-59/ 2024-04-20T00:15:14.459 DEBUG:boto:auth_path=/aktlji-59/ 2024-04-20T00:15:14.459 DEBUG:boto:Method: HEAD 2024-04-20T00:15:14.459 DEBUG:boto:Path: /aktlji-59/ 2024-04-20T00:15:14.459 DEBUG:boto:Data: 2024-04-20T00:15:14.459 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.459 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:15:14.459 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.460 DEBUG:boto:Params: {} 2024-04-20T00:15:14.460 DEBUG:boto:Token: None 2024-04-20T00:15:14.460 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:15:14 GMT /aktlji-59/ 2024-04-20T00:15:14.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0= 2024-04-20T00:15:14.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0=', 'Content-Length': '0'} 2024-04-20T00:15:14.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029c879e4aa53d7b4-0066230912-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.463 DEBUG:boto:path=/aktlji-59/obj4 2024-04-20T00:15:14.463 DEBUG:boto:auth_path=/aktlji-59/obj4 2024-04-20T00:15:14.463 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.463 DEBUG:boto:Path: /aktlji-59/obj4 2024-04-20T00:15:14.463 DEBUG:boto:Data: 2024-04-20T00:15:14.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:15:14.463 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:15:14.463 DEBUG:boto:Port: 8000 2024-04-20T00:15:14.463 DEBUG:boto:Params: {} 2024-04-20T00:15:14.464 DEBUG:boto:Token: None 2024-04-20T00:15:14.464 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:15:14 GMT /aktlji-59/obj4 2024-04-20T00:15:14.464 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:r+JfoDoW6G1xwR4BjxSftuM7MS8= 2024-04-20T00:15:14.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r+JfoDoW6G1xwR4BjxSftuM7MS8='} 2024-04-20T00:15:14.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008af5bd50eaeb867e-0066230912-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.472 DEBUG:boto:path=/aktlji-60/ 2024-04-20T00:15:14.472 DEBUG:boto:auth_path=/aktlji-60/ 2024-04-20T00:15:14.472 DEBUG:boto:Method: HEAD 2024-04-20T00:15:14.472 DEBUG:boto:Path: /aktlji-60/ 2024-04-20T00:15:14.472 DEBUG:boto:Data: 2024-04-20T00:15:14.472 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.473 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:15:14.473 DEBUG:boto:Port: 8001 2024-04-20T00:15:14.473 DEBUG:boto:Params: {} 2024-04-20T00:15:14.473 DEBUG:boto:Token: None 2024-04-20T00:15:14.473 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:15:14 GMT /aktlji-60/ 2024-04-20T00:15:14.473 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA= 2024-04-20T00:15:14.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA=', 'Content-Length': '0'} 2024-04-20T00:15:14.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000105d76e60edc62eb-0066230912-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.476 DEBUG:boto:path=/aktlji-60/obj1 2024-04-20T00:15:14.476 DEBUG:boto:auth_path=/aktlji-60/obj1 2024-04-20T00:15:14.476 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.476 DEBUG:boto:Path: /aktlji-60/obj1 2024-04-20T00:15:14.476 DEBUG:boto:Data: 2024-04-20T00:15:14.476 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:15:14.476 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:15:14.476 DEBUG:boto:Port: 8001 2024-04-20T00:15:14.476 DEBUG:boto:Params: {} 2024-04-20T00:15:14.476 DEBUG:boto:Token: None 2024-04-20T00:15:14.477 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:15:14 GMT /aktlji-60/obj1 2024-04-20T00:15:14.477 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:C34MfmQDDYUoFQfbL91D1v217oE= 2024-04-20T00:15:14.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:C34MfmQDDYUoFQfbL91D1v217oE='} 2024-04-20T00:15:14.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001524945254796eaf-0066230912-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.486 DEBUG:boto:path=/aktlji-60/ 2024-04-20T00:15:14.486 DEBUG:boto:auth_path=/aktlji-60/ 2024-04-20T00:15:14.486 DEBUG:boto:Method: HEAD 2024-04-20T00:15:14.486 DEBUG:boto:Path: /aktlji-60/ 2024-04-20T00:15:14.486 DEBUG:boto:Data: 2024-04-20T00:15:14.486 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.486 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:15:14.486 DEBUG:boto:Port: 8001 2024-04-20T00:15:14.486 DEBUG:boto:Params: {} 2024-04-20T00:15:14.487 DEBUG:boto:Token: None 2024-04-20T00:15:14.487 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:15:14 GMT /aktlji-60/ 2024-04-20T00:15:14.487 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA= 2024-04-20T00:15:14.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA=', 'Content-Length': '0'} 2024-04-20T00:15:14.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000077bc338feace2cd-0066230912-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.490 DEBUG:boto:path=/aktlji-60/obj2 2024-04-20T00:15:14.490 DEBUG:boto:auth_path=/aktlji-60/obj2 2024-04-20T00:15:14.490 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.490 DEBUG:boto:Path: /aktlji-60/obj2 2024-04-20T00:15:14.490 DEBUG:boto:Data: 2024-04-20T00:15:14.490 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:15:14.490 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:15:14.490 DEBUG:boto:Port: 8001 2024-04-20T00:15:14.490 DEBUG:boto:Params: {} 2024-04-20T00:15:14.490 DEBUG:boto:Token: None 2024-04-20T00:15:14.491 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:15:14 GMT /aktlji-60/obj2 2024-04-20T00:15:14.491 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lCkXsPpkxknOP6hOIrCpzfeRN+Q= 2024-04-20T00:15:14.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lCkXsPpkxknOP6hOIrCpzfeRN+Q='} 2024-04-20T00:15:14.499 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c23ad40f3f88510d-0066230912-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.499 DEBUG:boto:path=/aktlji-60/ 2024-04-20T00:15:14.499 DEBUG:boto:auth_path=/aktlji-60/ 2024-04-20T00:15:14.499 DEBUG:boto:Method: HEAD 2024-04-20T00:15:14.499 DEBUG:boto:Path: /aktlji-60/ 2024-04-20T00:15:14.499 DEBUG:boto:Data: 2024-04-20T00:15:14.500 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.500 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:15:14.500 DEBUG:boto:Port: 8001 2024-04-20T00:15:14.500 DEBUG:boto:Params: {} 2024-04-20T00:15:14.500 DEBUG:boto:Token: None 2024-04-20T00:15:14.500 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:15:14 GMT /aktlji-60/ 2024-04-20T00:15:14.500 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA= 2024-04-20T00:15:14.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA=', 'Content-Length': '0'} 2024-04-20T00:15:14.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007e47ce5b3f5d495-0066230912-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.503 DEBUG:boto:path=/aktlji-60/obj3 2024-04-20T00:15:14.503 DEBUG:boto:auth_path=/aktlji-60/obj3 2024-04-20T00:15:14.503 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.503 DEBUG:boto:Path: /aktlji-60/obj3 2024-04-20T00:15:14.504 DEBUG:boto:Data: 2024-04-20T00:15:14.504 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:15:14.504 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:15:14.504 DEBUG:boto:Port: 8001 2024-04-20T00:15:14.504 DEBUG:boto:Params: {} 2024-04-20T00:15:14.504 DEBUG:boto:Token: None 2024-04-20T00:15:14.504 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:15:14 GMT /aktlji-60/obj3 2024-04-20T00:15:14.504 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rGAljMQNIAOotE46qWJxJWhlj80= 2024-04-20T00:15:14.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rGAljMQNIAOotE46qWJxJWhlj80='} 2024-04-20T00:15:14.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6caf0178f892c29-0066230912-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.546 DEBUG:boto:path=/aktlji-60/ 2024-04-20T00:15:14.547 DEBUG:boto:auth_path=/aktlji-60/ 2024-04-20T00:15:14.547 DEBUG:boto:Method: HEAD 2024-04-20T00:15:14.547 DEBUG:boto:Path: /aktlji-60/ 2024-04-20T00:15:14.547 DEBUG:boto:Data: 2024-04-20T00:15:14.547 DEBUG:boto:Headers: {} 2024-04-20T00:15:14.547 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:15:14.547 DEBUG:boto:Port: 8001 2024-04-20T00:15:14.547 DEBUG:boto:Params: {} 2024-04-20T00:15:14.547 DEBUG:boto:Token: None 2024-04-20T00:15:14.547 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:15:14 GMT /aktlji-60/ 2024-04-20T00:15:14.548 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA= 2024-04-20T00:15:14.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA=', 'Content-Length': '0'} 2024-04-20T00:15:14.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f197682550131549-0066230912-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.551 DEBUG:boto:path=/aktlji-60/obj4 2024-04-20T00:15:14.551 DEBUG:boto:auth_path=/aktlji-60/obj4 2024-04-20T00:15:14.551 DEBUG:boto:Method: PUT 2024-04-20T00:15:14.551 DEBUG:boto:Path: /aktlji-60/obj4 2024-04-20T00:15:14.551 DEBUG:boto:Data: 2024-04-20T00:15:14.551 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:15:14.551 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:15:14.551 DEBUG:boto:Port: 8001 2024-04-20T00:15:14.551 DEBUG:boto:Params: {} 2024-04-20T00:15:14.551 DEBUG:boto:Token: None 2024-04-20T00:15:14.552 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:15:14 GMT /aktlji-60/obj4 2024-04-20T00:15:14.552 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UxTm6skByjgL2sOKXu8AM48jQl0= 2024-04-20T00:15:14.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UxTm6skByjgL2sOKXu8AM48jQl0='} 2024-04-20T00:15:14.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a8f3aa5986a534a-0066230912-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:15:14.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:15:14.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:15:14.559 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:15:14.617 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:15:14.617 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:15:14.654 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:15:14.669 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:15:14.669 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:15:14.669 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:15:14.669 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:15:14.669 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:15:14.670 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:15:14.670 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:15:14.670 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:15:14.670 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:15:14.670 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:15:14.670 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713572114.281893_688.1", 2024-04-20T00:15:14.670 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:15:14.281893Z" 2024-04-20T00:15:14.670 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:15:14.670 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:15:14.670 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:15:14.670 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:15:14.671 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:15:14.671 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:15:14.672 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572114.281893_688.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:15:14.672 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:15:14.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:15:14.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:15:14.672 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:15:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:15:14.731 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:15:14.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:15:14.791+0000 7f40909a9780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:15:14.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:15:14.792+0000 7f40909a9780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:15:14.819 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:15:14.819 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:15:14.820 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:15:14.820 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:15:14.820 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:15:14.820 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:15:14.820 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:15:14.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:14.820 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:15:14.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:15:14.820 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:15:14.820 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:15:14.821 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:15:14.821 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:15:14.821 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:15:14.821 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:15:14.821 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:15:14.821 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:15:14.821 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:15:14.821 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:14.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:14.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:15:14.821 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:15:14.821 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:15:14.822 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:15:14.822 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:15:14.822 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:15:14.822 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:15:14.822 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:15:14.822 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:15:14.822 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:15:14.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:14.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:14.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:15:14.823 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:15:14.823 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:15:14.823 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:15:14.823 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713572114.281893_688.1", 2024-04-20T00:15:14.823 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:15:14.823 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:15:14.823 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:15:14.823 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:15:14.281893Z", 2024-04-20T00:15:14.823 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:15:14.823 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:14.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:14.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:15:14.824 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:15:14.824 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:15:14.824 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:15:14.824 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:15:14.824 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:15:14.824 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:15:14.824 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:15:14.824 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:15:14.824 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:15:14.824 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:14.825 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:15:14.825 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:15:14.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:14.825 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:15:14.825 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:15:14.825 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:15:14.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:15:14.825 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:15:14Z" 2024-04-20T00:15:14.825 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:15:14.826 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713572114.281893_688.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:15:14.281893Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:15:14Z" } 2024-04-20T00:15:14.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572114.281893_688.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:15:14.281893Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:15:14.826 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572114.281893_688.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:15:14.827 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572114.281893_688.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:15:14.827 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:15:14.827 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:15:14.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:15:14.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:15:14.827 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:15:14.886 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:15:14.886 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:15:14.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:14.918+0000 7f0e9b20c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:15:14.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:14.919+0000 7f0e9b20c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:15:14.936 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:15:14.936 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:15:14.937 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:15:14.937 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:15:14.937 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:15:14.937 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:15:14.937 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:15:14.937 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:15:14.937 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:15:14.937 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:15:14.937 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:15:14.937 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:15:14.938 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:15:14.938 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:15:14.938 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:15:14.938 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:15:14.938 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:15:14.938 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:15:14.938 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:15:14.938 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:15:14.938 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:15:14.938 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:15:14.939 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:15:14.939 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:15:14.939 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:15:14.939 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:15:14.939 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:15:14.939 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:15:14.939 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:15:14.939 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:15:14.939 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:15:14.939 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:15:14.939 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:15:14.940 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:15:14.940 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:15:14.940 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:15:14.940 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:15:14.940 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713572114.281893_688.1", 2024-04-20T00:15:14.940 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:15:14.940 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:15:14.940 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:15:14.940 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:15:14.281893Z", 2024-04-20T00:15:14.940 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:15:14.941 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:15:14.941 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:15:14.941 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:15:14.941 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:15:14.941 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:15:14.941 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:15:14.941 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:15:14.941 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:15:14.941 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:15:14.941 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:15:14.941 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:15:14.942 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:15:14.942 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:15:14.942 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:15:14.942 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:15:14.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:15:14.942 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:15:14.942 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:15:14.942 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:15:14.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:15:14.942 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:15:14Z" 2024-04-20T00:15:14.942 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:15:14.943 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713572114.281893_688.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:15:14.281893Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:15:14Z" } 2024-04-20T00:15:14.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572114.281893_688.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:15:14.281893Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:15:14.944 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572114.281893_688.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:15:14.944 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572114.281893_688.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:15:14.944 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:15:14.944 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-58 2024-04-20T00:15:14.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:15:14.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:15:14.944 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-58 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T00:15:15.011 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:15:15.011 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:15:15.067 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-58 2024-04-20T00:15:15.085 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-58 2024-04-20T00:15:15.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:15:15.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:15:15.086 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-58 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:15:15.153 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:15:15.153 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:15:15.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:15.191+0000 7fcd9462e780 1 waiting to reach incremental sync.. 2024-04-20T00:15:20.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:20.192+0000 7fcd9462e780 1 waiting to reach incremental sync.. 2024-04-20T00:15:25.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:25.194+0000 7fcd9462e780 1 bucket sync caught up with source: 2024-04-20T00:15:25.197 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [00000000001.25.6, , 00000000001.33.6, , 00000000001.35.6, , , 00000000001.50.6, , , ] 2024-04-20T00:15:25.198 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [00000000001.25.6, , 00000000001.33.6, , 00000000001.35.6, , , 00000000001.50.6, , , ] 2024-04-20T00:15:25.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:25.194+0000 7fcd9462e780 0 bucket checkpoint complete 2024-04-20T00:15:25.215 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-58 2024-04-20T00:15:25.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-58', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:15:25.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-58', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:15:25.215 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-58 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:15:25.281 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:15:25.281 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:15:25.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:25.321+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:15:30.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:30.324+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:15:35.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:35.325+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:15:40.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:40.328+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:15:45.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:45.328+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:15:50.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:50.329+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:15:55.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:15:55.330+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:16:00.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:16:00.330+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:16:05.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:16:05.331+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:16:10.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:16:10.332+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:16:15.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:16:15.332+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:16:20.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:16:20.333+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:16:25.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:16:25.334+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:16:30.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:16:30.336+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:16:35.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:16:35.336+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:16:40.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:16:40.336+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:16:45.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:16:45.337+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:16:50.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:16:50.337+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:16:55.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:16:55.338+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:17:00.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:17:00.340+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:17:05.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:17:05.341+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:17:10.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:17:10.341+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:17:15.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:17:15.342+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:17:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:17:20.342+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:17:25.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:17:25.343+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:17:30.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:17:30.345+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:17:35.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:17:35.345+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:17:40.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:17:40.345+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:17:45.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:17:45.346+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:17:50.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:17:50.347+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:17:55.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:17:55.348+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:18:00.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:18:00.349+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:18:05.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:18:05.349+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:18:10.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:18:10.350+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:18:15.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:18:15.351+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:18:20.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:18:20.351+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:18:25.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:18:25.352+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:18:30.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:18:30.352+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:18:35.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:18:35.352+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:18:40.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:18:40.353+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:18:45.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:18:45.354+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:18:50.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:18:50.356+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:18:55.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:18:55.357+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:19:00.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:19:00.357+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:19:05.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:19:05.358+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:19:10.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:19:10.358+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:19:15.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:19:15.359+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:19:20.360+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:19:25.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:19:25.361+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:19:30.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:19:30.361+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:19:35.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:19:35.362+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:19:40.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:19:40.363+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:19:45.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:19:45.364+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:19:50.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:19:50.365+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:19:55.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:19:55.365+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:20:00.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:00.366+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:20:05.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:05.366+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:20:10.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:10.367+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:20:15.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:15.366+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T00:20:20.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:20.367+0000 7fa062b01780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T00:20:20.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:20.367+0000 7fa062b01780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:20:20.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:20.367+0000 7fa062b01780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:20:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T00:20:20.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2024-04-20T00:20:20.391 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-61 2024-04-20T00:20:20.391 DEBUG:boto:path=/aktlji-61/ 2024-04-20T00:20:20.392 DEBUG:boto:auth_path=/aktlji-61/ 2024-04-20T00:20:20.392 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.392 DEBUG:boto:Path: /aktlji-61/ 2024-04-20T00:20:20.392 DEBUG:boto:Data: 2024-04-20T00:20:20.392 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.392 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:20:20.392 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.392 DEBUG:boto:Params: {} 2024-04-20T00:20:20.393 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:20:20.393 DEBUG:boto:Token: None 2024-04-20T00:20:20.393 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:20:20 GMT /aktlji-61/ 2024-04-20T00:20:20.393 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zELhLL8pFAhZyy2zQmsdh4kFWbM= 2024-04-20T00:20:20.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zELhLL8pFAhZyy2zQmsdh4kFWbM=', 'Content-Length': '0'} 2024-04-20T00:20:20.415 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007280f33b15bf3dbf-0066230a44-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.416 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-62 2024-04-20T00:20:20.416 DEBUG:boto:path=/aktlji-62/ 2024-04-20T00:20:20.416 DEBUG:boto:auth_path=/aktlji-62/ 2024-04-20T00:20:20.416 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.416 DEBUG:boto:Path: /aktlji-62/ 2024-04-20T00:20:20.416 DEBUG:boto:Data: 2024-04-20T00:20:20.416 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.416 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:20:20.416 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.416 DEBUG:boto:Params: {} 2024-04-20T00:20:20.417 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:20:20.417 DEBUG:boto:Token: None 2024-04-20T00:20:20.417 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:20:20 GMT /aktlji-62/ 2024-04-20T00:20:20.417 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bJu5FKskFP+s98QairhbeSVBcyk= 2024-04-20T00:20:20.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bJu5FKskFP+s98QairhbeSVBcyk=', 'Content-Length': '0'} 2024-04-20T00:20:20.458 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006553eba53bd93cdf-0066230a44-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.459 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-63 2024-04-20T00:20:20.459 DEBUG:boto:path=/aktlji-63/ 2024-04-20T00:20:20.459 DEBUG:boto:auth_path=/aktlji-63/ 2024-04-20T00:20:20.459 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.459 DEBUG:boto:Path: /aktlji-63/ 2024-04-20T00:20:20.459 DEBUG:boto:Data: 2024-04-20T00:20:20.459 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.459 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:20:20.459 DEBUG:boto:Port: 8001 2024-04-20T00:20:20.459 DEBUG:boto:Params: {} 2024-04-20T00:20:20.460 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T00:20:20.460 DEBUG:boto:Token: None 2024-04-20T00:20:20.460 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:20:20 GMT /aktlji-63/ 2024-04-20T00:20:20.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sy5sRecQvBss291kAJ87TF8JQsw= 2024-04-20T00:20:20.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sy5sRecQvBss291kAJ87TF8JQsw=', 'Content-Length': '0'} 2024-04-20T00:20:20.499 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a29df66556c19c2f-0066230a44-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.500 DEBUG:boto:path=/aktlji-61/ 2024-04-20T00:20:20.500 DEBUG:boto:auth_path=/aktlji-61/ 2024-04-20T00:20:20.500 DEBUG:boto:Method: HEAD 2024-04-20T00:20:20.500 DEBUG:boto:Path: /aktlji-61/ 2024-04-20T00:20:20.500 DEBUG:boto:Data: 2024-04-20T00:20:20.500 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.500 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:20:20.500 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.500 DEBUG:boto:Params: {} 2024-04-20T00:20:20.501 DEBUG:boto:Token: None 2024-04-20T00:20:20.501 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:20:20 GMT /aktlji-61/ 2024-04-20T00:20:20.501 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c= 2024-04-20T00:20:20.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c=', 'Content-Length': '0'} 2024-04-20T00:20:20.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025bbf59016fe55a0-0066230a44-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.504 DEBUG:boto:path=/aktlji-61/obj1 2024-04-20T00:20:20.504 DEBUG:boto:auth_path=/aktlji-61/obj1 2024-04-20T00:20:20.504 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.504 DEBUG:boto:Path: /aktlji-61/obj1 2024-04-20T00:20:20.504 DEBUG:boto:Data: 2024-04-20T00:20:20.504 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:20:20.505 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:20:20.505 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.505 DEBUG:boto:Params: {} 2024-04-20T00:20:20.505 DEBUG:boto:Token: None 2024-04-20T00:20:20.505 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:20:20 GMT /aktlji-61/obj1 2024-04-20T00:20:20.505 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:B8g5HTe3ec0Cg+5yBwW3HwIrNb8= 2024-04-20T00:20:20.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B8g5HTe3ec0Cg+5yBwW3HwIrNb8='} 2024-04-20T00:20:20.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000913bea91a45ca312-0066230a44-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.512 DEBUG:boto:path=/aktlji-61/ 2024-04-20T00:20:20.512 DEBUG:boto:auth_path=/aktlji-61/ 2024-04-20T00:20:20.513 DEBUG:boto:Method: HEAD 2024-04-20T00:20:20.513 DEBUG:boto:Path: /aktlji-61/ 2024-04-20T00:20:20.513 DEBUG:boto:Data: 2024-04-20T00:20:20.513 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.513 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:20:20.513 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.513 DEBUG:boto:Params: {} 2024-04-20T00:20:20.513 DEBUG:boto:Token: None 2024-04-20T00:20:20.513 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:20:20 GMT /aktlji-61/ 2024-04-20T00:20:20.513 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c= 2024-04-20T00:20:20.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c=', 'Content-Length': '0'} 2024-04-20T00:20:20.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000769f872b6c4a16fe-0066230a44-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.516 DEBUG:boto:path=/aktlji-61/obj2 2024-04-20T00:20:20.516 DEBUG:boto:auth_path=/aktlji-61/obj2 2024-04-20T00:20:20.516 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.516 DEBUG:boto:Path: /aktlji-61/obj2 2024-04-20T00:20:20.516 DEBUG:boto:Data: 2024-04-20T00:20:20.516 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:20:20.517 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:20:20.517 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.517 DEBUG:boto:Params: {} 2024-04-20T00:20:20.517 DEBUG:boto:Token: None 2024-04-20T00:20:20.517 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:20:20 GMT /aktlji-61/obj2 2024-04-20T00:20:20.517 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qnDAvb9cyiqAVAfIS/q9QTsDJAw= 2024-04-20T00:20:20.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qnDAvb9cyiqAVAfIS/q9QTsDJAw='} 2024-04-20T00:20:20.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009ed0cd32c15f9d2-0066230a44-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.563 DEBUG:boto:path=/aktlji-61/ 2024-04-20T00:20:20.563 DEBUG:boto:auth_path=/aktlji-61/ 2024-04-20T00:20:20.563 DEBUG:boto:Method: HEAD 2024-04-20T00:20:20.563 DEBUG:boto:Path: /aktlji-61/ 2024-04-20T00:20:20.563 DEBUG:boto:Data: 2024-04-20T00:20:20.563 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.563 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:20:20.563 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.563 DEBUG:boto:Params: {} 2024-04-20T00:20:20.563 DEBUG:boto:Token: None 2024-04-20T00:20:20.564 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:20:20 GMT /aktlji-61/ 2024-04-20T00:20:20.564 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c= 2024-04-20T00:20:20.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c=', 'Content-Length': '0'} 2024-04-20T00:20:20.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050ea4574c5444c6a-0066230a44-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.567 DEBUG:boto:path=/aktlji-61/obj3 2024-04-20T00:20:20.567 DEBUG:boto:auth_path=/aktlji-61/obj3 2024-04-20T00:20:20.567 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.567 DEBUG:boto:Path: /aktlji-61/obj3 2024-04-20T00:20:20.567 DEBUG:boto:Data: 2024-04-20T00:20:20.567 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:20:20.567 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:20:20.567 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.567 DEBUG:boto:Params: {} 2024-04-20T00:20:20.567 DEBUG:boto:Token: None 2024-04-20T00:20:20.568 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:20:20 GMT /aktlji-61/obj3 2024-04-20T00:20:20.568 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ujMVbv0LQR3hOXa7EXeA20YRKvs= 2024-04-20T00:20:20.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ujMVbv0LQR3hOXa7EXeA20YRKvs='} 2024-04-20T00:20:20.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da8edbf4c7dd647e-0066230a44-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.574 DEBUG:boto:path=/aktlji-61/ 2024-04-20T00:20:20.574 DEBUG:boto:auth_path=/aktlji-61/ 2024-04-20T00:20:20.575 DEBUG:boto:Method: HEAD 2024-04-20T00:20:20.575 DEBUG:boto:Path: /aktlji-61/ 2024-04-20T00:20:20.575 DEBUG:boto:Data: 2024-04-20T00:20:20.575 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.575 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:20:20.575 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.575 DEBUG:boto:Params: {} 2024-04-20T00:20:20.575 DEBUG:boto:Token: None 2024-04-20T00:20:20.575 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:20:20 GMT /aktlji-61/ 2024-04-20T00:20:20.575 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c= 2024-04-20T00:20:20.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c=', 'Content-Length': '0'} 2024-04-20T00:20:20.578 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008137e8ecf65ca9f-0066230a44-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.578 DEBUG:boto:path=/aktlji-61/obj4 2024-04-20T00:20:20.578 DEBUG:boto:auth_path=/aktlji-61/obj4 2024-04-20T00:20:20.578 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.578 DEBUG:boto:Path: /aktlji-61/obj4 2024-04-20T00:20:20.578 DEBUG:boto:Data: 2024-04-20T00:20:20.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:20:20.579 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:20:20.579 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.579 DEBUG:boto:Params: {} 2024-04-20T00:20:20.579 DEBUG:boto:Token: None 2024-04-20T00:20:20.579 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:20:20 GMT /aktlji-61/obj4 2024-04-20T00:20:20.579 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zMlrHofW2ZFCv0OJMaOkdGL6LvA= 2024-04-20T00:20:20.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zMlrHofW2ZFCv0OJMaOkdGL6LvA='} 2024-04-20T00:20:20.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f33c0af208f8cb95-0066230a44-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.623 DEBUG:boto:path=/aktlji-62/ 2024-04-20T00:20:20.623 DEBUG:boto:auth_path=/aktlji-62/ 2024-04-20T00:20:20.623 DEBUG:boto:Method: HEAD 2024-04-20T00:20:20.623 DEBUG:boto:Path: /aktlji-62/ 2024-04-20T00:20:20.623 DEBUG:boto:Data: 2024-04-20T00:20:20.623 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.623 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:20:20.623 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.623 DEBUG:boto:Params: {} 2024-04-20T00:20:20.623 DEBUG:boto:Token: None 2024-04-20T00:20:20.624 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:20:20 GMT /aktlji-62/ 2024-04-20T00:20:20.624 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4= 2024-04-20T00:20:20.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4=', 'Content-Length': '0'} 2024-04-20T00:20:20.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af7de60da7202999-0066230a44-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.627 DEBUG:boto:path=/aktlji-62/obj1 2024-04-20T00:20:20.627 DEBUG:boto:auth_path=/aktlji-62/obj1 2024-04-20T00:20:20.627 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.627 DEBUG:boto:Path: /aktlji-62/obj1 2024-04-20T00:20:20.627 DEBUG:boto:Data: 2024-04-20T00:20:20.627 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:20:20.627 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:20:20.627 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.627 DEBUG:boto:Params: {} 2024-04-20T00:20:20.628 DEBUG:boto:Token: None 2024-04-20T00:20:20.628 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:20:20 GMT /aktlji-62/obj1 2024-04-20T00:20:20.628 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hdG8QdSKDvfX3To0iOzswI9zdck= 2024-04-20T00:20:20.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hdG8QdSKDvfX3To0iOzswI9zdck='} 2024-04-20T00:20:20.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000055bf84b1ca46dd3-0066230a44-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.647 DEBUG:boto:path=/aktlji-62/ 2024-04-20T00:20:20.648 DEBUG:boto:auth_path=/aktlji-62/ 2024-04-20T00:20:20.648 DEBUG:boto:Method: HEAD 2024-04-20T00:20:20.648 DEBUG:boto:Path: /aktlji-62/ 2024-04-20T00:20:20.648 DEBUG:boto:Data: 2024-04-20T00:20:20.648 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.648 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:20:20.648 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.648 DEBUG:boto:Params: {} 2024-04-20T00:20:20.648 DEBUG:boto:Token: None 2024-04-20T00:20:20.648 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:20:20 GMT /aktlji-62/ 2024-04-20T00:20:20.649 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4= 2024-04-20T00:20:20.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4=', 'Content-Length': '0'} 2024-04-20T00:20:20.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2747f0692b0c5cc-0066230a44-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.651 DEBUG:boto:path=/aktlji-62/obj2 2024-04-20T00:20:20.651 DEBUG:boto:auth_path=/aktlji-62/obj2 2024-04-20T00:20:20.651 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.651 DEBUG:boto:Path: /aktlji-62/obj2 2024-04-20T00:20:20.651 DEBUG:boto:Data: 2024-04-20T00:20:20.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:20:20.652 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:20:20.652 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.652 DEBUG:boto:Params: {} 2024-04-20T00:20:20.652 DEBUG:boto:Token: None 2024-04-20T00:20:20.652 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:20:20 GMT /aktlji-62/obj2 2024-04-20T00:20:20.652 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8i0Tzo9GXSavfamHyrB2UNTkveM= 2024-04-20T00:20:20.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8i0Tzo9GXSavfamHyrB2UNTkveM='} 2024-04-20T00:20:20.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d08169187e5672c-0066230a44-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.695 DEBUG:boto:path=/aktlji-62/ 2024-04-20T00:20:20.695 DEBUG:boto:auth_path=/aktlji-62/ 2024-04-20T00:20:20.695 DEBUG:boto:Method: HEAD 2024-04-20T00:20:20.695 DEBUG:boto:Path: /aktlji-62/ 2024-04-20T00:20:20.695 DEBUG:boto:Data: 2024-04-20T00:20:20.695 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.695 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:20:20.695 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.695 DEBUG:boto:Params: {} 2024-04-20T00:20:20.696 DEBUG:boto:Token: None 2024-04-20T00:20:20.696 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:20:20 GMT /aktlji-62/ 2024-04-20T00:20:20.696 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4= 2024-04-20T00:20:20.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4=', 'Content-Length': '0'} 2024-04-20T00:20:20.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b288a45c879d9c8e-0066230a44-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.698 DEBUG:boto:path=/aktlji-62/obj3 2024-04-20T00:20:20.698 DEBUG:boto:auth_path=/aktlji-62/obj3 2024-04-20T00:20:20.699 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.699 DEBUG:boto:Path: /aktlji-62/obj3 2024-04-20T00:20:20.699 DEBUG:boto:Data: 2024-04-20T00:20:20.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:20:20.699 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:20:20.699 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.699 DEBUG:boto:Params: {} 2024-04-20T00:20:20.699 DEBUG:boto:Token: None 2024-04-20T00:20:20.699 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:20:20 GMT /aktlji-62/obj3 2024-04-20T00:20:20.699 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DCf5raunn+HVOZUoJ34efX1+qps= 2024-04-20T00:20:20.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DCf5raunn+HVOZUoJ34efX1+qps='} 2024-04-20T00:20:20.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f56ba6ad076f550-0066230a44-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.708 DEBUG:boto:path=/aktlji-62/ 2024-04-20T00:20:20.708 DEBUG:boto:auth_path=/aktlji-62/ 2024-04-20T00:20:20.708 DEBUG:boto:Method: HEAD 2024-04-20T00:20:20.708 DEBUG:boto:Path: /aktlji-62/ 2024-04-20T00:20:20.709 DEBUG:boto:Data: 2024-04-20T00:20:20.709 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.709 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:20:20.709 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.709 DEBUG:boto:Params: {} 2024-04-20T00:20:20.709 DEBUG:boto:Token: None 2024-04-20T00:20:20.710 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:20:20 GMT /aktlji-62/ 2024-04-20T00:20:20.710 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4= 2024-04-20T00:20:20.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4=', 'Content-Length': '0'} 2024-04-20T00:20:20.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032c0ac9ac405ed29-0066230a44-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.712 DEBUG:boto:path=/aktlji-62/obj4 2024-04-20T00:20:20.713 DEBUG:boto:auth_path=/aktlji-62/obj4 2024-04-20T00:20:20.713 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.713 DEBUG:boto:Path: /aktlji-62/obj4 2024-04-20T00:20:20.713 DEBUG:boto:Data: 2024-04-20T00:20:20.713 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:20:20.714 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:20:20.714 DEBUG:boto:Port: 8000 2024-04-20T00:20:20.714 DEBUG:boto:Params: {} 2024-04-20T00:20:20.714 DEBUG:boto:Token: None 2024-04-20T00:20:20.714 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:20:20 GMT /aktlji-62/obj4 2024-04-20T00:20:20.714 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hLcjJ8Trc4EhT93SvuWoYzmCBoY= 2024-04-20T00:20:20.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hLcjJ8Trc4EhT93SvuWoYzmCBoY='} 2024-04-20T00:20:20.721 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d9b3a56fdb9c22a-0066230a44-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.721 DEBUG:boto:path=/aktlji-63/ 2024-04-20T00:20:20.721 DEBUG:boto:auth_path=/aktlji-63/ 2024-04-20T00:20:20.721 DEBUG:boto:Method: HEAD 2024-04-20T00:20:20.722 DEBUG:boto:Path: /aktlji-63/ 2024-04-20T00:20:20.722 DEBUG:boto:Data: 2024-04-20T00:20:20.722 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.722 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:20:20.722 DEBUG:boto:Port: 8001 2024-04-20T00:20:20.722 DEBUG:boto:Params: {} 2024-04-20T00:20:20.722 DEBUG:boto:Token: None 2024-04-20T00:20:20.723 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:20:20 GMT /aktlji-63/ 2024-04-20T00:20:20.723 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs= 2024-04-20T00:20:20.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs=', 'Content-Length': '0'} 2024-04-20T00:20:20.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2d52d2fa7c11271-0066230a44-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.725 DEBUG:boto:path=/aktlji-63/obj1 2024-04-20T00:20:20.726 DEBUG:boto:auth_path=/aktlji-63/obj1 2024-04-20T00:20:20.726 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.726 DEBUG:boto:Path: /aktlji-63/obj1 2024-04-20T00:20:20.726 DEBUG:boto:Data: 2024-04-20T00:20:20.726 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:20:20.726 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:20:20.726 DEBUG:boto:Port: 8001 2024-04-20T00:20:20.727 DEBUG:boto:Params: {} 2024-04-20T00:20:20.727 DEBUG:boto:Token: None 2024-04-20T00:20:20.727 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:20:20 GMT /aktlji-63/obj1 2024-04-20T00:20:20.727 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:inhQ3CBVkAbuxBlccsbFkC3sPWo= 2024-04-20T00:20:20.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:inhQ3CBVkAbuxBlccsbFkC3sPWo='} 2024-04-20T00:20:20.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9deee732e798844-0066230a44-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.736 DEBUG:boto:path=/aktlji-63/ 2024-04-20T00:20:20.736 DEBUG:boto:auth_path=/aktlji-63/ 2024-04-20T00:20:20.736 DEBUG:boto:Method: HEAD 2024-04-20T00:20:20.736 DEBUG:boto:Path: /aktlji-63/ 2024-04-20T00:20:20.736 DEBUG:boto:Data: 2024-04-20T00:20:20.736 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.736 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:20:20.737 DEBUG:boto:Port: 8001 2024-04-20T00:20:20.737 DEBUG:boto:Params: {} 2024-04-20T00:20:20.737 DEBUG:boto:Token: None 2024-04-20T00:20:20.737 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:20:20 GMT /aktlji-63/ 2024-04-20T00:20:20.737 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs= 2024-04-20T00:20:20.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs=', 'Content-Length': '0'} 2024-04-20T00:20:20.740 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ada6f1c8a021e1e-0066230a44-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.740 DEBUG:boto:path=/aktlji-63/obj2 2024-04-20T00:20:20.741 DEBUG:boto:auth_path=/aktlji-63/obj2 2024-04-20T00:20:20.741 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.741 DEBUG:boto:Path: /aktlji-63/obj2 2024-04-20T00:20:20.741 DEBUG:boto:Data: 2024-04-20T00:20:20.741 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:20:20.741 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:20:20.741 DEBUG:boto:Port: 8001 2024-04-20T00:20:20.742 DEBUG:boto:Params: {} 2024-04-20T00:20:20.742 DEBUG:boto:Token: None 2024-04-20T00:20:20.742 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:20:20 GMT /aktlji-63/obj2 2024-04-20T00:20:20.742 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eNd/t8fOGs4X9kg06udtRv00+jI= 2024-04-20T00:20:20.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eNd/t8fOGs4X9kg06udtRv00+jI='} 2024-04-20T00:20:20.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059809a182e0876fc-0066230a44-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.750 DEBUG:boto:path=/aktlji-63/ 2024-04-20T00:20:20.750 DEBUG:boto:auth_path=/aktlji-63/ 2024-04-20T00:20:20.750 DEBUG:boto:Method: HEAD 2024-04-20T00:20:20.751 DEBUG:boto:Path: /aktlji-63/ 2024-04-20T00:20:20.751 DEBUG:boto:Data: 2024-04-20T00:20:20.751 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.751 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:20:20.751 DEBUG:boto:Port: 8001 2024-04-20T00:20:20.751 DEBUG:boto:Params: {} 2024-04-20T00:20:20.751 DEBUG:boto:Token: None 2024-04-20T00:20:20.751 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:20:20 GMT /aktlji-63/ 2024-04-20T00:20:20.751 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs= 2024-04-20T00:20:20.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs=', 'Content-Length': '0'} 2024-04-20T00:20:20.754 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb2c723e79526275-0066230a44-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.755 DEBUG:boto:path=/aktlji-63/obj3 2024-04-20T00:20:20.755 DEBUG:boto:auth_path=/aktlji-63/obj3 2024-04-20T00:20:20.755 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.755 DEBUG:boto:Path: /aktlji-63/obj3 2024-04-20T00:20:20.755 DEBUG:boto:Data: 2024-04-20T00:20:20.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:20:20.755 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:20:20.755 DEBUG:boto:Port: 8001 2024-04-20T00:20:20.756 DEBUG:boto:Params: {} 2024-04-20T00:20:20.756 DEBUG:boto:Token: None 2024-04-20T00:20:20.756 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:20:20 GMT /aktlji-63/obj3 2024-04-20T00:20:20.756 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:LsPEGFxoe+Hd3KIHJbrSgAIfX0U= 2024-04-20T00:20:20.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LsPEGFxoe+Hd3KIHJbrSgAIfX0U='} 2024-04-20T00:20:20.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036185da51af73b7a-0066230a44-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.799 DEBUG:boto:path=/aktlji-63/ 2024-04-20T00:20:20.799 DEBUG:boto:auth_path=/aktlji-63/ 2024-04-20T00:20:20.799 DEBUG:boto:Method: HEAD 2024-04-20T00:20:20.799 DEBUG:boto:Path: /aktlji-63/ 2024-04-20T00:20:20.799 DEBUG:boto:Data: 2024-04-20T00:20:20.799 DEBUG:boto:Headers: {} 2024-04-20T00:20:20.799 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:20:20.799 DEBUG:boto:Port: 8001 2024-04-20T00:20:20.799 DEBUG:boto:Params: {} 2024-04-20T00:20:20.800 DEBUG:boto:Token: None 2024-04-20T00:20:20.800 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:20:20 GMT /aktlji-63/ 2024-04-20T00:20:20.800 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs= 2024-04-20T00:20:20.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs=', 'Content-Length': '0'} 2024-04-20T00:20:20.802 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb42a48cdb7b18bd-0066230a44-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.803 DEBUG:boto:path=/aktlji-63/obj4 2024-04-20T00:20:20.803 DEBUG:boto:auth_path=/aktlji-63/obj4 2024-04-20T00:20:20.803 DEBUG:boto:Method: PUT 2024-04-20T00:20:20.803 DEBUG:boto:Path: /aktlji-63/obj4 2024-04-20T00:20:20.803 DEBUG:boto:Data: 2024-04-20T00:20:20.803 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T00:20:20.803 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:20:20.804 DEBUG:boto:Port: 8001 2024-04-20T00:20:20.804 DEBUG:boto:Params: {} 2024-04-20T00:20:20.804 DEBUG:boto:Token: None 2024-04-20T00:20:20.804 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 00:20:20 GMT /aktlji-63/obj4 2024-04-20T00:20:20.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yu0Lf8/p6zOFSWzWWDDxi7sISCo= 2024-04-20T00:20:20.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yu0Lf8/p6zOFSWzWWDDxi7sISCo='} 2024-04-20T00:20:20.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041cbd7e5d46822ec-0066230a44-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:20:20.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:20:20.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:20:20.811 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:20:20.871 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:20:20.871 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:20:20.908 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:20:20.923 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:20:20.923 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:20:20.923 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:20:20.924 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:20:20.924 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:20.924 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:20:20.924 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:20:20.924 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:20:20.924 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:20.924 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:20:20.925 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713572420.485008_738.1", 2024-04-20T00:20:20.925 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:20:20.485008Z" 2024-04-20T00:20:20.925 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:20.925 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:20:20.925 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:20:20.926 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:20:20.926 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:20:20.926 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:20:20.927 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.485008_738.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:20:20.927 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:20:20.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:20:20.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:20:20.929 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:20:20.989 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:20:20.989 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:20:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:20:21.040+0000 7f173d1d2780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:20:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:20:21.041+0000 7f173d1d2780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:20:21.068 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:20:21.068 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:20:21.068 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:20:21.069 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:20:21.069 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:20:21.069 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:20:21.069 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:20:21.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:20:21.070 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:20:21.070 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:20:21.070 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:20:21.070 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:20:21.070 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:20:21.070 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:20:21.070 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:20:21.070 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:20:21.070 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:20:21.071 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:20:21.071 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:20:21.071 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:20:21.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:20:21.071 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:20:21.071 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:20:21.071 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:20:21.071 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:20:21.072 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:20:21.072 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:20:21.072 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:20:21.072 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:20:21.072 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:20:21.072 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:20:21.072 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:20:21.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:20:21.072 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:20:21.073 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:20:21.073 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:20:21.073 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:20:21.073 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713572420.485008_738.1", 2024-04-20T00:20:21.073 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:20:21.073 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:20:21.073 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:20:21.073 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:20:20.485008Z", 2024-04-20T00:20:21.073 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:20:21.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:20:21.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:20:21.074 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:20:21.074 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:20:21.074 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:20:21.074 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:20:21.074 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:20:21.074 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:20:21.074 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:20:21.074 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:20:21.074 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:20:21.074 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:20:21.074 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:20:21.075 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:20:21.075 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:20:21.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:20:21.075 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:20:21.075 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:20:21.075 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:20:21.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:20:21.075 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:20:21Z" 2024-04-20T00:20:21.075 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:20:21.076 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713572420.485008_738.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:20:20.485008Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:20:21Z" } 2024-04-20T00:20:21.076 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572420.485008_738.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:20:20.485008Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:20:21.077 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.485008_738.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:20:21.077 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.485008_738.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:20:21.077 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:20:21.077 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:20:21.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:20:21.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:20:21.077 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:20:21.136 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:20:21.136 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:20:21.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:21.168+0000 7f1a3819e780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:20:21.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:21.169+0000 7f1a3819e780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:20:21.186 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:20:21.186 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:20:21.187 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:20:21.187 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:20:21.187 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:20:21.187 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:20:21.187 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:20:21.187 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:21.187 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:20:21.187 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:20:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:20:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:20:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:20:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:20:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:20:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:20:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:20:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:20:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:20:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:20:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:21.188 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:20:21.189 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:20:21.189 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:20:21.189 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:20:21.189 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:20:21.189 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:20:21.189 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:20:21.189 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:20:21.189 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:20:21.189 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:20:21.189 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:20:21.189 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:21.190 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:20:21.190 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:20:21.190 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:20:21.190 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:20:21.190 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713572420.432178_731.1", 2024-04-20T00:20:21.190 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:20:21.190 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:20:21.190 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:20:21.190 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:20:20.432178Z", 2024-04-20T00:20:21.190 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:20:21.190 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:20:21.191 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:21.191 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:20:21.191 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:20:21.191 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:20:21.191 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:20:21.191 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:20:21.191 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:20:21.191 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:20:21.191 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:20:21.191 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:20:21.191 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:20:21.191 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:20:21.192 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:20:21.192 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:20:21.192 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:21.192 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:20:21.192 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:20:21.192 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:20:21.192 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:21.192 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:20:21Z" 2024-04-20T00:20:21.192 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:20:21.193 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713572420.432178_731.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:20:20.432178Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:20:21Z" } 2024-04-20T00:20:21.193 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572420.432178_731.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:20:20.432178Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:20:21.194 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.485008_738.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:20:21.194 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.432178_731.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:20:21.194 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1713572420.485008_738.1 target=1_1713572420.432178_731.1 2024-04-20T00:20:26.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:20:26.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:20:26.196 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:20:26.256 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:20:26.256 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:20:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:26.289+0000 7fe88f99f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:20:26.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:26.290+0000 7fe88f99f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:20:26.308 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:20:26.309 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:20:26.309 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:20:26.309 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:20:26.309 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:20:26.309 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:20:26.309 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:20:26.309 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:26.309 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:20:26.309 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:20:26.310 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:20:26.310 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:20:26.310 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:20:26.310 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:20:26.310 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:20:26.310 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:20:26.310 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:20:26.310 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:20:26.310 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:20:26.310 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:20:26.310 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:26.311 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:20:26.311 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:20:26.311 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:20:26.311 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:20:26.311 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:20:26.311 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:20:26.311 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:20:26.311 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:20:26.311 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:20:26.311 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:20:26.311 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:20:26.311 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:26.312 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:20:26.312 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:20:26.312 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:20:26.312 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:20:26.312 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713572420.485008_738.1", 2024-04-20T00:20:26.312 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:20:26.312 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:20:26.312 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:20:26.312 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:20:20.485008Z", 2024-04-20T00:20:26.312 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:20:26.312 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:20:26.313 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:26.313 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:20:26.313 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:20:26.313 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:20:26.313 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:20:26.313 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:20:26.313 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:20:26.313 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:20:26.313 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:20:26.313 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:20:26.313 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:20:26.314 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:20:26.314 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:20:26.314 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:20:26.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:26.314 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:20:26.314 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:20:26.314 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:20:26.314 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:20:26.314 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:20:26Z" 2024-04-20T00:20:26.314 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:20:26.315 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713572420.485008_738.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:20:20.485008Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:20:26Z" } 2024-04-20T00:20:26.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572420.485008_738.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:20:20.485008Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:20:26.316 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.485008_738.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:20:26.316 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.485008_738.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:20:26.316 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:20:26.316 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-61 2024-04-20T00:20:26.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:20:26.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:20:26.316 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T00:20:26.382 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:20:26.382 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:20:26.438 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-61 2024-04-20T00:20:26.456 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-61 2024-04-20T00:20:26.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:20:26.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:20:26.457 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:20:26.526 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:20:26.526 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:20:26.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:26.565+0000 7f6d504a2780 1 bucket sync caught up with source: 2024-04-20T00:20:26.568 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [00000000001.48.6, , 00000000001.27.6, , 00000000001.159.6, , , 00000000001.39.6, , , ] 2024-04-20T00:20:26.568 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [00000000001.48.6, , 00000000001.27.6, , 00000000001.159.6, , , 00000000001.39.6, , , ] 2024-04-20T00:20:26.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:26.565+0000 7f6d504a2780 0 bucket checkpoint complete 2024-04-20T00:20:26.583 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-61 2024-04-20T00:20:26.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:20:26.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:20:26.584 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:20:26.646 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:20:26.647 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:20:26.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:26.685+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:20:31.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:31.686+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:20:36.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:36.686+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:20:41.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:41.687+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:20:46.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:46.687+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:20:51.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:51.687+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:20:56.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:20:56.688+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:21:01.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:21:01.689+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:21:06.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:21:06.689+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:21:11.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:21:11.689+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:21:16.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:21:16.690+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:21:21.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:21:21.691+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:21:26.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:21:26.692+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:21:31.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:21:31.693+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:21:36.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:21:36.693+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:21:41.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:21:41.694+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:21:46.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:21:46.694+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:21:51.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:21:51.695+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:21:56.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:21:56.695+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:22:01.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:22:01.696+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:22:06.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:22:06.696+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:22:11.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:22:11.696+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:22:16.699 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:22:16.697+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:22:21.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:22:21.697+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:22:26.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:22:26.698+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:22:31.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:22:31.698+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:22:36.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:22:36.699+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:22:41.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:22:41.700+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:22:46.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:22:46.701+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:22:51.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:22:51.701+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:22:56.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:22:56.702+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:23:01.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:23:01.702+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:23:06.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:23:06.703+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:23:11.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:23:11.704+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:23:16.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:23:16.704+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:23:21.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:23:21.705+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:23:26.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:23:26.706+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:23:31.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:23:31.706+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:23:36.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:23:36.706+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:23:41.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:23:41.707+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:23:46.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:23:46.707+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:23:51.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:23:51.707+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:23:56.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:23:56.707+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:24:01.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:24:01.708+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:24:06.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:24:06.709+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:24:11.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:24:11.709+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:24:16.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:24:16.710+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:24:21.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:24:21.710+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:24:26.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:24:26.711+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:24:31.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:24:31.711+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:24:36.712+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:24:41.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:24:41.712+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:24:46.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:24:46.713+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:24:51.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:24:51.714+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:24:56.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:24:56.714+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:25:01.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:01.715+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:25:06.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:06.716+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:25:11.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:11.717+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:25:16.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:16.717+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T00:25:21.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:21.718+0000 7f658ad94780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T00:25:21.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:21.718+0000 7f658ad94780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:25:21.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:21.718+0000 7f658ad94780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:25:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T00:25:21.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2024-04-20T00:25:21.741 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-64 2024-04-20T00:25:21.741 DEBUG:boto:path=/aktlji-64/ 2024-04-20T00:25:21.741 DEBUG:boto:auth_path=/aktlji-64/ 2024-04-20T00:25:21.741 DEBUG:boto:Method: PUT 2024-04-20T00:25:21.741 DEBUG:boto:Path: /aktlji-64/ 2024-04-20T00:25:21.741 DEBUG:boto:Data: 2024-04-20T00:25:21.741 DEBUG:boto:Headers: {} 2024-04-20T00:25:21.742 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:25:21.742 DEBUG:boto:Port: 8000 2024-04-20T00:25:21.742 DEBUG:boto:Params: {} 2024-04-20T00:25:21.742 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:25:21.742 DEBUG:boto:Token: None 2024-04-20T00:25:21.743 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:25:21 GMT /aktlji-64/ 2024-04-20T00:25:21.743 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pSkwUN5rt1FWM6NayF4QxoVnL/4= 2024-04-20T00:25:21.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:25:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pSkwUN5rt1FWM6NayF4QxoVnL/4=', 'Content-Length': '0'} 2024-04-20T00:25:21.766 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c7890efb504c22b6-0066230b71-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:25:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:25:21.766 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-65 2024-04-20T00:25:21.766 DEBUG:boto:path=/aktlji-65/ 2024-04-20T00:25:21.766 DEBUG:boto:auth_path=/aktlji-65/ 2024-04-20T00:25:21.766 DEBUG:boto:Method: PUT 2024-04-20T00:25:21.766 DEBUG:boto:Path: /aktlji-65/ 2024-04-20T00:25:21.767 DEBUG:boto:Data: 2024-04-20T00:25:21.767 DEBUG:boto:Headers: {} 2024-04-20T00:25:21.767 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:25:21.767 DEBUG:boto:Port: 8000 2024-04-20T00:25:21.767 DEBUG:boto:Params: {} 2024-04-20T00:25:21.767 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:25:21.767 DEBUG:boto:Token: None 2024-04-20T00:25:21.767 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:25:21 GMT /aktlji-65/ 2024-04-20T00:25:21.767 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:J4QsIqWb6zxtHBxtns+nKDT4J6o= 2024-04-20T00:25:21.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:25:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:J4QsIqWb6zxtHBxtns+nKDT4J6o=', 'Content-Length': '0'} 2024-04-20T00:25:21.817 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000078c3ca9e19492fc9-0066230b71-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:25:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:25:21.817 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=aktlji-66 2024-04-20T00:25:21.817 DEBUG:boto:path=/aktlji-66/ 2024-04-20T00:25:21.817 DEBUG:boto:auth_path=/aktlji-66/ 2024-04-20T00:25:21.817 DEBUG:boto:Method: PUT 2024-04-20T00:25:21.817 DEBUG:boto:Path: /aktlji-66/ 2024-04-20T00:25:21.817 DEBUG:boto:Data: 2024-04-20T00:25:21.817 DEBUG:boto:Headers: {} 2024-04-20T00:25:21.817 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:25:21.818 DEBUG:boto:Port: 8001 2024-04-20T00:25:21.818 DEBUG:boto:Params: {} 2024-04-20T00:25:21.818 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T00:25:21.818 DEBUG:boto:Token: None 2024-04-20T00:25:21.818 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:25:21 GMT /aktlji-66/ 2024-04-20T00:25:21.818 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:s+RUmNN45rI9DSMZub+k9zDrKnw= 2024-04-20T00:25:21.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:25:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:s+RUmNN45rI9DSMZub+k9zDrKnw=', 'Content-Length': '0'} 2024-04-20T00:25:21.856 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009f017945377e0d3b-0066230b71-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:25:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:25:21.856 DEBUG:boto:path=/aktlji-64/MULTIPART 2024-04-20T00:25:21.857 DEBUG:boto:auth_path=/aktlji-64/MULTIPART 2024-04-20T00:25:21.857 DEBUG:boto:path=/aktlji-64/MULTIPART?uploads 2024-04-20T00:25:21.857 DEBUG:boto:auth_path=/aktlji-64/MULTIPART?uploads 2024-04-20T00:25:21.857 DEBUG:boto:Method: POST 2024-04-20T00:25:21.857 DEBUG:boto:Path: /aktlji-64/MULTIPART?uploads 2024-04-20T00:25:21.857 DEBUG:boto:Data: 2024-04-20T00:25:21.857 DEBUG:boto:Headers: {} 2024-04-20T00:25:21.857 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:25:21.857 DEBUG:boto:Port: 8000 2024-04-20T00:25:21.857 DEBUG:boto:Params: {} 2024-04-20T00:25:21.857 DEBUG:boto:Token: None 2024-04-20T00:25:21.858 DEBUG:boto:StringToSign: POST Sat, 20 Apr 2024 00:25:21 GMT /aktlji-64/MULTIPART?uploads 2024-04-20T00:25:21.858 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1yrjRGj5hCmARrbew9bIftY+MJU= 2024-04-20T00:25:21.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:25:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1yrjRGj5hCmARrbew9bIftY+MJU=', 'Content-Length': '0'} 2024-04-20T00:25:23.004 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000056bbca11aab0fec4-0066230b71-15b6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:25:23.004 DEBUG:boto:b'aktlji-64MULTIPART2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5' 2024-04-20T00:25:23.042 DEBUG:boto:path=/aktlji-64/MULTIPART 2024-04-20T00:25:23.043 DEBUG:boto:auth_path=/aktlji-64/MULTIPART 2024-04-20T00:25:23.043 DEBUG:boto:path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=1 2024-04-20T00:25:23.043 DEBUG:boto:auth_path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=1 2024-04-20T00:25:23.043 DEBUG:boto:Method: PUT 2024-04-20T00:25:23.043 DEBUG:boto:Path: /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=1 2024-04-20T00:25:23.043 DEBUG:boto:Data: 2024-04-20T00:25:23.043 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-20T00:25:23.043 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:25:23.044 DEBUG:boto:Port: 8000 2024-04-20T00:25:23.044 DEBUG:boto:Params: {} 2024-04-20T00:25:23.044 DEBUG:boto:Token: None 2024-04-20T00:25:23.044 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Sat, 20 Apr 2024 00:25:23 GMT /aktlji-64/MULTIPART?partNumber=1&uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T00:25:23.044 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dTIbx+16/TUnJGVJG2HSCOjL7h0= 2024-04-20T00:25:23.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:25:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dTIbx+16/TUnJGVJG2HSCOjL7h0='} 2024-04-20T00:25:23.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000389fb27a967f0449-0066230b73-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:25:23.111 DEBUG:boto:path=/aktlji-64/MULTIPART 2024-04-20T00:25:23.112 DEBUG:boto:auth_path=/aktlji-64/MULTIPART 2024-04-20T00:25:23.112 DEBUG:boto:path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=2 2024-04-20T00:25:23.112 DEBUG:boto:auth_path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=2 2024-04-20T00:25:23.112 DEBUG:boto:Method: PUT 2024-04-20T00:25:23.112 DEBUG:boto:Path: /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=2 2024-04-20T00:25:23.112 DEBUG:boto:Data: 2024-04-20T00:25:23.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-20T00:25:23.112 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:25:23.113 DEBUG:boto:Port: 8000 2024-04-20T00:25:23.113 DEBUG:boto:Params: {} 2024-04-20T00:25:23.113 DEBUG:boto:Token: None 2024-04-20T00:25:23.113 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Sat, 20 Apr 2024 00:25:23 GMT /aktlji-64/MULTIPART?partNumber=2&uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T00:25:23.113 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:9W0y84ZR5gqH1rNoqgK+zZRCkdU= 2024-04-20T00:25:23.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:25:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9W0y84ZR5gqH1rNoqgK+zZRCkdU='} 2024-04-20T00:25:23.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e7fc214f140a311-0066230b73-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:25:23.166 DEBUG:boto:path=/aktlji-64/MULTIPART 2024-04-20T00:25:23.167 DEBUG:boto:auth_path=/aktlji-64/MULTIPART 2024-04-20T00:25:23.167 DEBUG:boto:path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=3 2024-04-20T00:25:23.167 DEBUG:boto:auth_path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=3 2024-04-20T00:25:23.167 DEBUG:boto:Method: PUT 2024-04-20T00:25:23.167 DEBUG:boto:Path: /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=3 2024-04-20T00:25:23.167 DEBUG:boto:Data: 2024-04-20T00:25:23.167 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-20T00:25:23.167 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:25:23.167 DEBUG:boto:Port: 8000 2024-04-20T00:25:23.167 DEBUG:boto:Params: {} 2024-04-20T00:25:23.168 DEBUG:boto:Token: None 2024-04-20T00:25:23.168 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Sat, 20 Apr 2024 00:25:23 GMT /aktlji-64/MULTIPART?partNumber=3&uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T00:25:23.168 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zZs0UaatU5T1LQuTlO1CcDYhzxk= 2024-04-20T00:25:23.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:25:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zZs0UaatU5T1LQuTlO1CcDYhzxk='} 2024-04-20T00:25:23.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f638f9ae544046a3-0066230b73-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:25:23.219 DEBUG:boto:path=/aktlji-64/MULTIPART 2024-04-20T00:25:23.220 DEBUG:boto:auth_path=/aktlji-64/MULTIPART 2024-04-20T00:25:23.220 DEBUG:boto:path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=4 2024-04-20T00:25:23.220 DEBUG:boto:auth_path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=4 2024-04-20T00:25:23.220 DEBUG:boto:Method: PUT 2024-04-20T00:25:23.220 DEBUG:boto:Path: /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=4 2024-04-20T00:25:23.220 DEBUG:boto:Data: 2024-04-20T00:25:23.220 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-20T00:25:23.220 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:25:23.220 DEBUG:boto:Port: 8000 2024-04-20T00:25:23.220 DEBUG:boto:Params: {} 2024-04-20T00:25:23.220 DEBUG:boto:Token: None 2024-04-20T00:25:23.221 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Sat, 20 Apr 2024 00:25:23 GMT /aktlji-64/MULTIPART?partNumber=4&uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T00:25:23.221 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:c/Ga37AzTgI/Y1dL8JNc+GPgqnw= 2024-04-20T00:25:23.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:25:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c/Ga37AzTgI/Y1dL8JNc+GPgqnw='} 2024-04-20T00:25:23.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb3a99e11bb2b384-0066230b73-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:25:23.254 DEBUG:boto:path=/aktlji-64/MULTIPART 2024-04-20T00:25:23.254 DEBUG:boto:auth_path=/aktlji-64/MULTIPART 2024-04-20T00:25:23.254 DEBUG:boto:path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T00:25:23.254 DEBUG:boto:auth_path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T00:25:23.255 DEBUG:boto:Method: GET 2024-04-20T00:25:23.255 DEBUG:boto:Path: /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T00:25:23.255 DEBUG:boto:Data: 2024-04-20T00:25:23.255 DEBUG:boto:Headers: {} 2024-04-20T00:25:23.255 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:25:23.255 DEBUG:boto:Port: 8000 2024-04-20T00:25:23.255 DEBUG:boto:Params: {} 2024-04-20T00:25:23.255 DEBUG:boto:Token: None 2024-04-20T00:25:23.255 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 00:25:23 GMT /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T00:25:23.255 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:I144DsXgjw4tpR0TUgEA5/LR5UI= 2024-04-20T00:25:23.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:25:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:I144DsXgjw4tpR0TUgEA5/LR5UI=', 'Content-Length': '0'} 2024-04-20T00:25:23.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000094d361532fff48-0066230b73-15b6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:25:23.303 DEBUG:boto:path=/aktlji-64/MULTIPART 2024-04-20T00:25:23.303 DEBUG:boto:auth_path=/aktlji-64/MULTIPART 2024-04-20T00:25:23.303 DEBUG:boto:path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T00:25:23.303 DEBUG:boto:auth_path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T00:25:23.303 DEBUG:boto:Method: POST 2024-04-20T00:25:23.303 DEBUG:boto:Path: /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T00:25:23.303 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2024-04-20T00:25:23.303 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2024-04-20T00:25:23.303 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:25:23.304 DEBUG:boto:Port: 8000 2024-04-20T00:25:23.304 DEBUG:boto:Params: {} 2024-04-20T00:25:23.304 DEBUG:boto:Token: None 2024-04-20T00:25:23.304 DEBUG:boto:StringToSign: POST text/xml Sat, 20 Apr 2024 00:25:23 GMT /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T00:25:23.304 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IsnPSZrFKEiMAzfnOViJXUeI0nk= 2024-04-20T00:25:23.304 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:25:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IsnPSZrFKEiMAzfnOViJXUeI0nk=', 'Content-Length': '460'} 2024-04-20T00:25:23.317 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a153a317eea4ddf5-0066230b73-15b6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '325'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:25:23.317 DEBUG:boto:http://smithi019.front.sepia.ceph.com:8000/aktlji-64/MULTIPARTaktlji-64MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-04-20T00:25:23.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:25:23.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:25:23.318 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:25:23.376 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:25:23.376 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:25:23.414 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:25:23.429 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:25:23.429 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:25:23.429 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:25:23.429 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:25:23.429 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:25:23.429 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:25:23.429 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:25:23.429 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:25:23.429 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:25:23.429 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:25:23.430 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713572721.843198_786.1", 2024-04-20T00:25:23.430 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:25:21.843198Z" 2024-04-20T00:25:23.430 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:25:23.430 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:25:23.430 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:25:23.430 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:25:23.430 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:25:23.430 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:25:23.431 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572721.843198_786.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:25:23.431 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:25:23.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:25:23.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:25:23.432 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:25:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:25:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:25:23.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:25:23.553+0000 7fbeca55c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:25:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:25:23.554+0000 7fbeca55c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:25:23.583 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:25:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:25:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:25:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:25:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:25:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:25:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:25:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:25:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:25:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:25:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:25:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:25:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:25:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:25:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:25:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:25:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:25:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:25:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:25:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:25:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:25:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:25:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:25:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:25:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:25:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:25:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:25:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:25:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:25:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:25:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:25:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:25:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:25:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:25:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:25:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:25:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:25:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713572721.843198_786.1", 2024-04-20T00:25:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:25:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:25:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:25:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:25:21.843198Z", 2024-04-20T00:25:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:25:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:25:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:25:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:25:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:25:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:25:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:25:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:25:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:25:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:25:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:25:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:25:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:25:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:25:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:25:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:25:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:25:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:25:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:25:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:25:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:25:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:25:23Z" 2024-04-20T00:25:23.588 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:25:23.589 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713572721.843198_786.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:25:21.843198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:25:23Z" } 2024-04-20T00:25:23.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572721.843198_786.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:25:21.843198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:25:23.590 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572721.843198_786.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:25:23.590 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572721.843198_786.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:25:23.590 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:25:23.590 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:25:23.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:25:23.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:25:23.590 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:25:23.649 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:25:23.649 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:25:23.684 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:23.683+0000 7f206f41b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:25:23.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:23.684+0000 7f206f41b780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:25:23.700 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:25:23.700 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:25:23.700 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:25:23.700 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:25:23.700 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:25:23.700 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:25:23.700 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:25:23.700 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:25:23.700 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:25:23.700 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:25:23.700 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:25:23.701 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:25:23.701 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:25:23.701 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:25:23.701 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:25:23.701 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:25:23.701 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:25:23.701 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:25:23.701 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:25:23.701 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:25:23.701 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:25:23.701 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:25:23.701 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:25:23.702 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:25:23.702 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:25:23.702 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:25:23.702 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:25:23.702 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:25:23.702 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:25:23.702 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:25:23.702 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:25:23.702 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:25:23.702 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:25:23.702 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:25:23.703 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:25:23.703 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:25:23.703 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:25:23.703 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713572721.843198_786.1", 2024-04-20T00:25:23.703 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:25:23.703 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:25:23.703 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:25:23.703 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:25:21.843198Z", 2024-04-20T00:25:23.703 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:25:23.703 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:25:23.703 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:25:23.703 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:25:23.704 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:25:23.704 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:25:23.704 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:25:23.704 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:25:23.704 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:25:23.704 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:25:23.704 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:25:23.704 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:25:23.704 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:25:23.704 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:25:23.704 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:25:23.704 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:25:23.705 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:25:23.705 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:25:23.705 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:25:23.705 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:25:23.705 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:25:23.705 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:25:23Z" 2024-04-20T00:25:23.705 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:25:23.706 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713572721.843198_786.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:25:21.843198Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:25:23Z" } 2024-04-20T00:25:23.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572721.843198_786.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:25:21.843198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:25:23.706 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572721.843198_786.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:25:23.706 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572721.843198_786.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:25:23.706 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:25:23.707 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-64 2024-04-20T00:25:23.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:25:23.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:25:23.707 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T00:25:23.774 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:25:23.775 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:25:23.840 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-64 2024-04-20T00:25:23.859 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-64 2024-04-20T00:25:23.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:25:23.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:25:23.859 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:25:23.926 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:25:23.927 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:25:24.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:24.973+0000 7fec6c4b7780 1 waiting to reach incremental sync.. 2024-04-20T00:25:29.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:29.974+0000 7fec6c4b7780 1 waiting to reach incremental sync.. 2024-04-20T00:25:34.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:34.974+0000 7fec6c4b7780 1 waiting to reach incremental sync.. 2024-04-20T00:25:39.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:39.974+0000 7fec6c4b7780 1 waiting to reach incremental sync.. 2024-04-20T00:25:44.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:44.976+0000 7fec6c4b7780 1 bucket sync caught up with source: 2024-04-20T00:25:44.979 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000011.61.6, , , , , ] 2024-04-20T00:25:44.979 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000011.61.6, , , , , ] 2024-04-20T00:25:44.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:44.976+0000 7fec6c4b7780 0 bucket checkpoint complete 2024-04-20T00:25:44.996 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-64 2024-04-20T00:25:44.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:25:44.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:25:44.997 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:25:45.062 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:25:45.063 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:25:45.103 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:45.101+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:25:50.104 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:50.101+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:25:55.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:25:55.102+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:26:00.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:26:00.103+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:26:05.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:26:05.103+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:26:10.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:26:10.104+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:26:15.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:26:15.104+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:26:20.108 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:26:20.105+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:26:25.108 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:26:25.106+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:26:30.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:26:30.106+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:26:35.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:26:35.107+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:26:40.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:26:40.108+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:26:45.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:26:45.109+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:26:50.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:26:50.109+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:26:55.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:26:55.110+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:27:00.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:27:00.111+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:27:05.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:27:05.111+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:27:10.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:27:10.112+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:27:15.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:27:15.113+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:27:20.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:27:20.113+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:27:25.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:27:25.114+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:27:30.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:27:30.113+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:27:35.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:27:35.114+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:27:40.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:27:40.114+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:27:45.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:27:45.115+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:27:50.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:27:50.116+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:27:55.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:27:55.116+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:28:00.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:28:00.117+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:28:05.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:28:05.118+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:28:10.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:28:10.118+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:28:15.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:28:15.119+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:28:20.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:28:20.119+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:28:25.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:28:25.120+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:28:30.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:28:30.121+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:28:35.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:28:35.121+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:28:40.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:28:40.122+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:28:45.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:28:45.123+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:28:50.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:28:50.123+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:28:55.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:28:55.124+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:29:00.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:29:00.124+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:29:05.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:29:05.125+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:29:10.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:29:10.126+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:29:15.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:29:15.126+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:29:20.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:29:20.125+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:29:25.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:29:25.126+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:29:30.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:29:30.127+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:29:35.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:29:35.127+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:29:40.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:29:40.128+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:29:45.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:29:45.128+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:29:50.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:29:50.129+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:29:55.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:29:55.129+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:30:00.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:00.130+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:30:05.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:05.131+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:30:10.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:10.132+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:30:15.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:15.132+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:30:20.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:20.133+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:30:25.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:25.133+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:30:30.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:30.134+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:30:35.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:35.135+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T00:30:40.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:40.136+0000 7fd83969d780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T00:30:40.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:40.136+0000 7fd83969d780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:30:40.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:40.136+0000 7fd83969d780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:30:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T00:30:40.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... FAIL 2024-04-20T00:30:40.161 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-67 2024-04-20T00:30:40.162 DEBUG:boto:path=/aktlji-67/ 2024-04-20T00:30:40.162 DEBUG:boto:auth_path=/aktlji-67/ 2024-04-20T00:30:40.162 DEBUG:boto:Method: PUT 2024-04-20T00:30:40.162 DEBUG:boto:Path: /aktlji-67/ 2024-04-20T00:30:40.162 DEBUG:boto:Data: 2024-04-20T00:30:40.163 DEBUG:boto:Headers: {} 2024-04-20T00:30:40.163 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:30:40.163 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.163 DEBUG:boto:Params: {} 2024-04-20T00:30:40.164 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:30:40.164 DEBUG:boto:Token: None 2024-04-20T00:30:40.164 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:30:40 GMT /aktlji-67/ 2024-04-20T00:30:40.165 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Q0f2EBTSb8yD5E83s5WCCGgxV+M= 2024-04-20T00:30:40.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q0f2EBTSb8yD5E83s5WCCGgxV+M=', 'Content-Length': '0'} 2024-04-20T00:30:40.188 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000777525ac7a25ec23-0066230cb0-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.189 DEBUG:boto:path=/aktlji-67/testobj-sse-c 2024-04-20T00:30:40.189 DEBUG:boto:auth_path=/aktlji-67/testobj-sse-c 2024-04-20T00:30:40.189 DEBUG:boto:Method: PUT 2024-04-20T00:30:40.189 DEBUG:boto:Path: /aktlji-67/testobj-sse-c 2024-04-20T00:30:40.189 DEBUG:boto:Data: 2024-04-20T00:30:40.190 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-20T00:30:40.190 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:30:40.190 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.190 DEBUG:boto:Params: {} 2024-04-20T00:30:40.190 DEBUG:boto:Token: None 2024-04-20T00:30:40.190 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Sat, 20 Apr 2024 00:30:40 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /aktlji-67/testobj-sse-c 2024-04-20T00:30:40.190 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:w1vJCP+ahsdL3zAFW/RmmgtpOp4= 2024-04-20T00:30:40.191 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:w1vJCP+ahsdL3zAFW/RmmgtpOp4='} 2024-04-20T00:30:40.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000637b91c77d52e874-0066230cb0-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.200 DEBUG:boto:path=/aktlji-67/testobj-sse-kms 2024-04-20T00:30:40.200 DEBUG:boto:auth_path=/aktlji-67/testobj-sse-kms 2024-04-20T00:30:40.200 DEBUG:boto:Method: PUT 2024-04-20T00:30:40.200 DEBUG:boto:Path: /aktlji-67/testobj-sse-kms 2024-04-20T00:30:40.200 DEBUG:boto:Data: 2024-04-20T00:30:40.200 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-20T00:30:40.200 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:30:40.200 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.201 DEBUG:boto:Params: {} 2024-04-20T00:30:40.201 DEBUG:boto:Token: None 2024-04-20T00:30:40.201 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Sat, 20 Apr 2024 00:30:40 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /aktlji-67/testobj-sse-kms 2024-04-20T00:30:40.201 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mP01sqkX0HyD9vIhfkJRuSbKiYs= 2024-04-20T00:30:40.201 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mP01sqkX0HyD9vIhfkJRuSbKiYs='} 2024-04-20T00:30:40.208 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6YWt0bGppLTY3L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000bbe0f69eed3d74dd-0066230cb0-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:40.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:40.209 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:30:40.267 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:30:40.267 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:30:40.304 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:30:40.319 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:30:40.319 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:40.319 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:30:40.319 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:30:40.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:40.320 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:40.320 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:30:40.320 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:30:40.320 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:40.320 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:40.320 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713573040.185141_826.1", 2024-04-20T00:30:40.320 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:30:40.185141Z" 2024-04-20T00:30:40.321 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:40.321 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:40.321 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:30:40.321 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:30:40.321 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:30:40.321 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:30:40.322 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.185141_826.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:30:40.323 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:30:40.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:40.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:40.323 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:30:40.389 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:30:40.389 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:30:40.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:30:40.454+0000 7fa7da69c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:30:40.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:30:40.455+0000 7fa7da69c780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:30:40.476 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:30:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:30:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:30:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:30:40.476 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:30:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:30:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:30:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:30:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:30:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:30:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:30:40.477 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:30:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:30:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:30:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:30:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:30:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:30:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:30:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:30:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:30:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:30:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:30:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:30:40.479 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:30:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:30:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:30:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:30:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:30:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:30:40.480 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713573040.185141_826.1", 2024-04-20T00:30:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:30:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:30:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:30:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:30:40.185141Z", 2024-04-20T00:30:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:40.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:30:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:30:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:30:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:30:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:30:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:30:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:30:40.482 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:30:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:30:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:30:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:30:40.483 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:30:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:30:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:40.484 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:30:40Z" 2024-04-20T00:30:40.484 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:30:40.485 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573040.185141_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:30:40.185141Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:30:40Z" } 2024-04-20T00:30:40.485 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573040.185141_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:30:40.185141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:30:40.485 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.185141_826.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:30:40.485 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.185141_826.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:30:40.486 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:30:40.486 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:30:40.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:40.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:40.486 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:30:40.546 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:30:40.546 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:30:40.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:40.579+0000 7f1ef9178780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:30:40.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:40.580+0000 7f1ef9178780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:30:40.597 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:30:40.597 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:30:40.597 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:30:40.598 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:30:40.598 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:30:40.598 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:30:40.598 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:30:40.598 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:40.598 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:30:40.598 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:40.598 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:30:40.599 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:30:40.599 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:30:40.599 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:30:40.599 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:30:40.599 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:30:40.599 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:30:40.599 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:30:40.599 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:30:40.599 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:30:40.600 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:40.600 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:40.600 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:30:40.600 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:30:40.600 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:30:40.600 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:30:40.600 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:30:40.600 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:30:40.600 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:30:40.600 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:30:40.601 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:30:40.601 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:30:40.601 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:40.601 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:40.601 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:30:40.601 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:30:40.601 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:30:40.601 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713573040.185141_826.1", 2024-04-20T00:30:40.601 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:30:40.602 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:30:40.602 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:30:40.602 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:30:40.185141Z", 2024-04-20T00:30:40.602 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:30:40.602 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:30:40.602 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:40.602 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:40.602 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:30:40.603 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:30:40.603 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:30:40.603 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:30:40.603 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:30:40.603 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:30:40.603 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:30:40.603 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:30:40.603 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:30:40.603 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:30:40.603 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:30:40.604 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:30:40.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:40.604 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:30:40.604 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:30:40.604 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:30:40.604 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:40.604 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:30:40Z" 2024-04-20T00:30:40.604 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:30:40.605 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573040.185141_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:30:40.185141Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:30:40Z" } 2024-04-20T00:30:40.605 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573040.185141_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:30:40.185141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:30:40.606 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.185141_826.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:30:40.606 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.185141_826.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:30:40.606 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:30:40.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:30:40.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:30:40.607 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-67 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T00:30:40.673 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:30:40.674 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:30:40.718 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-67 2024-04-20T00:30:40.736 DEBUG:boto:path=/aktlji-67/ 2024-04-20T00:30:40.736 DEBUG:boto:auth_path=/aktlji-67/ 2024-04-20T00:30:40.737 DEBUG:boto:Method: HEAD 2024-04-20T00:30:40.737 DEBUG:boto:Path: /aktlji-67/ 2024-04-20T00:30:40.737 DEBUG:boto:Data: 2024-04-20T00:30:40.737 DEBUG:boto:Headers: {} 2024-04-20T00:30:40.737 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:30:40.737 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.737 DEBUG:boto:Params: {} 2024-04-20T00:30:40.737 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:30:40.738 DEBUG:boto:Token: None 2024-04-20T00:30:40.738 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:30:40 GMT /aktlji-67/ 2024-04-20T00:30:40.738 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:V+gO8br0FghZeQiTWWU2Mf5B5+E= 2024-04-20T00:30:40.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:V+gO8br0FghZeQiTWWU2Mf5B5+E=', 'Content-Length': '0'} 2024-04-20T00:30:40.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000050d707ad67179bd-0066230cb0-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.743 DEBUG:boto:path=/aktlji-67/testobj-sse-c 2024-04-20T00:30:40.743 DEBUG:boto:auth_path=/aktlji-67/testobj-sse-c 2024-04-20T00:30:40.743 DEBUG:boto:Method: HEAD 2024-04-20T00:30:40.744 DEBUG:boto:Path: /aktlji-67/testobj-sse-c 2024-04-20T00:30:40.744 DEBUG:boto:Data: 2024-04-20T00:30:40.744 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2024-04-20T00:30:40.744 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:30:40.744 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.744 DEBUG:boto:Params: {} 2024-04-20T00:30:40.744 DEBUG:boto:Token: None 2024-04-20T00:30:40.744 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:30:40 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /aktlji-67/testobj-sse-c 2024-04-20T00:30:40.745 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:j3X9fr3JDMo+4Arf1Du67igZqYM= 2024-04-20T00:30:40.745 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:j3X9fr3JDMo+4Arf1Du67igZqYM=', 'Content-Length': '0'} 2024-04-20T00:30:40.747 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000599ea9f098d4938a-0066230cb0-1bfc-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2024-04-20T00:30:40.748 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-68 2024-04-20T00:30:40.748 DEBUG:boto:path=/aktlji-68/ 2024-04-20T00:30:40.748 DEBUG:boto:auth_path=/aktlji-68/ 2024-04-20T00:30:40.748 DEBUG:boto:Method: PUT 2024-04-20T00:30:40.748 DEBUG:boto:Path: /aktlji-68/ 2024-04-20T00:30:40.749 DEBUG:boto:Data: 2024-04-20T00:30:40.749 DEBUG:boto:Headers: {} 2024-04-20T00:30:40.749 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:30:40.749 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.749 DEBUG:boto:Params: {} 2024-04-20T00:30:40.749 DEBUG:boto:Token: None 2024-04-20T00:30:40.749 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:30:40 GMT /aktlji-68/ 2024-04-20T00:30:40.749 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iskKeeAKonkc+Cgmz0utnMli6dA= 2024-04-20T00:30:40.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iskKeeAKonkc+Cgmz0utnMli6dA=', 'Content-Length': '0'} 2024-04-20T00:30:40.770 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000061455e1a9d366f2d-0066230cb0-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.770 DEBUG:boto:path=/aktlji-68/ 2024-04-20T00:30:40.771 DEBUG:boto:auth_path=/aktlji-68/ 2024-04-20T00:30:40.771 DEBUG:boto:Method: HEAD 2024-04-20T00:30:40.771 DEBUG:boto:Path: /aktlji-68/ 2024-04-20T00:30:40.771 DEBUG:boto:Data: 2024-04-20T00:30:40.771 DEBUG:boto:Headers: {} 2024-04-20T00:30:40.771 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:30:40.771 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.771 DEBUG:boto:Params: {} 2024-04-20T00:30:40.771 DEBUG:boto:Token: None 2024-04-20T00:30:40.772 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:30:40 GMT /aktlji-68/ 2024-04-20T00:30:40.772 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM= 2024-04-20T00:30:40.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM=', 'Content-Length': '0'} 2024-04-20T00:30:40.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e3be95405b92798-0066230cb0-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.775 DEBUG:boto:path=/aktlji-68/a 2024-04-20T00:30:40.775 DEBUG:boto:auth_path=/aktlji-68/a 2024-04-20T00:30:40.775 DEBUG:boto:Method: PUT 2024-04-20T00:30:40.775 DEBUG:boto:Path: /aktlji-68/a 2024-04-20T00:30:40.775 DEBUG:boto:Data: 2024-04-20T00:30:40.775 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:30:40.776 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:30:40.776 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.776 DEBUG:boto:Params: {} 2024-04-20T00:30:40.776 DEBUG:boto:Token: None 2024-04-20T00:30:40.776 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:30:40 GMT /aktlji-68/a 2024-04-20T00:30:40.776 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aUOd+DlhZn6zM5gahjG3E/pPbko= 2024-04-20T00:30:40.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aUOd+DlhZn6zM5gahjG3E/pPbko='} 2024-04-20T00:30:40.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000871e83ddac4eccaa-0066230cb0-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.819 DEBUG:boto:path=/aktlji-68/ 2024-04-20T00:30:40.819 DEBUG:boto:auth_path=/aktlji-68/ 2024-04-20T00:30:40.819 DEBUG:boto:Method: HEAD 2024-04-20T00:30:40.819 DEBUG:boto:Path: /aktlji-68/ 2024-04-20T00:30:40.819 DEBUG:boto:Data: 2024-04-20T00:30:40.819 DEBUG:boto:Headers: {} 2024-04-20T00:30:40.819 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:30:40.819 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.820 DEBUG:boto:Params: {} 2024-04-20T00:30:40.820 DEBUG:boto:Token: None 2024-04-20T00:30:40.820 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:30:40 GMT /aktlji-68/ 2024-04-20T00:30:40.820 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM= 2024-04-20T00:30:40.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM=', 'Content-Length': '0'} 2024-04-20T00:30:40.823 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d4b3d96f8447258-0066230cb0-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.823 DEBUG:boto:path=/aktlji-68/b 2024-04-20T00:30:40.823 DEBUG:boto:auth_path=/aktlji-68/b 2024-04-20T00:30:40.823 DEBUG:boto:Method: PUT 2024-04-20T00:30:40.823 DEBUG:boto:Path: /aktlji-68/b 2024-04-20T00:30:40.824 DEBUG:boto:Data: 2024-04-20T00:30:40.824 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:30:40.824 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:30:40.824 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.824 DEBUG:boto:Params: {} 2024-04-20T00:30:40.824 DEBUG:boto:Token: None 2024-04-20T00:30:40.824 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:30:40 GMT /aktlji-68/b 2024-04-20T00:30:40.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:98Nn0a8dfWOiZ6kq8KqNdvtrmc8= 2024-04-20T00:30:40.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:98Nn0a8dfWOiZ6kq8KqNdvtrmc8='} 2024-04-20T00:30:40.831 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057c9bef41d096f2c-0066230cb0-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.832 DEBUG:boto:path=/aktlji-68/ 2024-04-20T00:30:40.832 DEBUG:boto:auth_path=/aktlji-68/ 2024-04-20T00:30:40.832 DEBUG:boto:Method: HEAD 2024-04-20T00:30:40.832 DEBUG:boto:Path: /aktlji-68/ 2024-04-20T00:30:40.832 DEBUG:boto:Data: 2024-04-20T00:30:40.832 DEBUG:boto:Headers: {} 2024-04-20T00:30:40.832 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:30:40.832 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.832 DEBUG:boto:Params: {} 2024-04-20T00:30:40.833 DEBUG:boto:Token: None 2024-04-20T00:30:40.833 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:30:40 GMT /aktlji-68/ 2024-04-20T00:30:40.833 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM= 2024-04-20T00:30:40.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM=', 'Content-Length': '0'} 2024-04-20T00:30:40.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4d129e40ce932b0-0066230cb0-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.836 DEBUG:boto:path=/aktlji-68/c 2024-04-20T00:30:40.836 DEBUG:boto:auth_path=/aktlji-68/c 2024-04-20T00:30:40.836 DEBUG:boto:Method: PUT 2024-04-20T00:30:40.836 DEBUG:boto:Path: /aktlji-68/c 2024-04-20T00:30:40.836 DEBUG:boto:Data: 2024-04-20T00:30:40.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:30:40.837 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:30:40.837 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.837 DEBUG:boto:Params: {} 2024-04-20T00:30:40.837 DEBUG:boto:Token: None 2024-04-20T00:30:40.837 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:30:40 GMT /aktlji-68/c 2024-04-20T00:30:40.837 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mDgtTHX2+XpbcUgMHS0xpH+ocD8= 2024-04-20T00:30:40.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mDgtTHX2+XpbcUgMHS0xpH+ocD8='} 2024-04-20T00:30:40.883 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e57c405a6adca730-0066230cb0-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.883 DEBUG:boto:path=/aktlji-68/ 2024-04-20T00:30:40.883 DEBUG:boto:auth_path=/aktlji-68/ 2024-04-20T00:30:40.883 DEBUG:boto:Method: HEAD 2024-04-20T00:30:40.883 DEBUG:boto:Path: /aktlji-68/ 2024-04-20T00:30:40.883 DEBUG:boto:Data: 2024-04-20T00:30:40.883 DEBUG:boto:Headers: {} 2024-04-20T00:30:40.884 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:30:40.884 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.884 DEBUG:boto:Params: {} 2024-04-20T00:30:40.884 DEBUG:boto:Token: None 2024-04-20T00:30:40.884 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:30:40 GMT /aktlji-68/ 2024-04-20T00:30:40.884 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM= 2024-04-20T00:30:40.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM=', 'Content-Length': '0'} 2024-04-20T00:30:40.888 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000472872a5445ce247-0066230cb0-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.888 DEBUG:boto:path=/aktlji-68/d 2024-04-20T00:30:40.888 DEBUG:boto:auth_path=/aktlji-68/d 2024-04-20T00:30:40.888 DEBUG:boto:Method: PUT 2024-04-20T00:30:40.888 DEBUG:boto:Path: /aktlji-68/d 2024-04-20T00:30:40.888 DEBUG:boto:Data: 2024-04-20T00:30:40.888 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:30:40.889 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:30:40.889 DEBUG:boto:Port: 8000 2024-04-20T00:30:40.889 DEBUG:boto:Params: {} 2024-04-20T00:30:40.889 DEBUG:boto:Token: None 2024-04-20T00:30:40.889 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:30:40 GMT /aktlji-68/d 2024-04-20T00:30:40.889 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R43ssYu0Kjjdlqb0gJHub5g+esw= 2024-04-20T00:30:40.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R43ssYu0Kjjdlqb0gJHub5g+esw='} 2024-04-20T00:30:40.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d26458c82a96cd9-0066230cb0-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:30:40.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:40.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:40.897 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:30:40.956 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:30:40.956 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:30:40.998 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:30:41.017 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:30:41.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:41.018 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:30:41.018 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:30:41.018 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:41.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:41.018 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:30:41.018 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:30:41.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:41.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:41.019 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713573040.767133_830.1", 2024-04-20T00:30:41.019 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:30:40.767133Z" 2024-04-20T00:30:41.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:41.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:41.019 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:30:41.019 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:30:41.019 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:30:41.020 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:30:41.021 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.767133_830.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:30:41.021 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:30:41.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:41.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:41.022 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:30:41.080 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:30:41.080 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:30:41.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:30:41.134+0000 7ff6ac576780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:30:41.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:30:41.136+0000 7ff6ac576780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:30:41.157 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:30:41.157 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:30:41.157 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:30:41.157 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:30:41.158 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:30:41.158 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:30:41.158 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:41.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:41.158 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:30:41.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:30:41.158 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:30:41.158 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:30:41.159 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:30:41.159 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:30:41.159 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:30:41.159 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:30:41.159 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:30:41.159 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:30:41.159 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:41.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:41.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:41.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:30:41.160 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:30:41.160 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:30:41.160 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:30:41.160 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:30:41.160 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:30:41.160 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:30:41.161 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:30:41.161 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:30:41.161 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:41.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:41.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:41.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:30:41.161 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:30:41.161 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:30:41.161 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:30:41.162 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713573040.758847_827.1", 2024-04-20T00:30:41.162 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:30:41.162 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:30:41.162 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:30:41.162 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:30:40.758847Z", 2024-04-20T00:30:41.162 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:41.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:41.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:41.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:30:41.163 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:30:41.163 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:30:41.163 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:30:41.163 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:30:41.163 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:30:41.163 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:30:41.164 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:30:41.164 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:30:41.164 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:41.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:41.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:41.164 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:30:41.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:41.164 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:30:41.165 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:30:41.165 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:30:41.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:41.165 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:30:41Z" 2024-04-20T00:30:41.165 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:30:41.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573040.758847_827.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:30:40.758847Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:30:41Z" } 2024-04-20T00:30:41.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573040.758847_827.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:30:40.758847Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:30:41.167 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.767133_830.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:30:41.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.758847_827.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:30:41.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713573040.767133_830.1 target=1_1713573040.758847_827.1 2024-04-20T00:30:46.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:46.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:46.168 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:30:46.229 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:30:46.230 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:30:46.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:30:46.284+0000 7fd90b7a0780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:30:46.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:30:46.285+0000 7fd90b7a0780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:30:46.310 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:30:46.310 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:30:46.311 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:30:46.311 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:30:46.311 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:30:46.311 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:30:46.311 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:46.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:46.311 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:30:46.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:30:46.311 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:30:46.311 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:30:46.312 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:30:46.312 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:30:46.312 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:30:46.312 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:30:46.312 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:30:46.312 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:30:46.312 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:46.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:46.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:46.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:30:46.313 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:30:46.313 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:30:46.313 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:30:46.313 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:30:46.313 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:30:46.313 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:30:46.313 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:30:46.313 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:30:46.313 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:46.313 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:46.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:46.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:30:46.314 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:30:46.314 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:30:46.314 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:30:46.314 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713573040.767133_830.1", 2024-04-20T00:30:46.314 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:30:46.314 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:30:46.314 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:30:46.315 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:30:40.767133Z", 2024-04-20T00:30:46.315 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:46.315 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:46.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:46.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:30:46.315 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:30:46.315 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:30:46.315 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:30:46.315 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:30:46.315 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:30:46.315 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:30:46.316 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:30:46.316 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:30:46.316 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:30:46.316 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:46.316 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:30:46.316 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:30:46.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:46.316 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:30:46.316 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:30:46.316 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:30:46.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:30:46.317 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:30:46Z" 2024-04-20T00:30:46.317 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:30:46.318 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573040.767133_830.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:30:40.767133Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:30:46Z" } 2024-04-20T00:30:46.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573040.767133_830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:30:40.767133Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:30:46.318 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.767133_830.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:30:46.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.767133_830.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:30:46.318 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:30:46.318 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:30:46.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:46.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:30:46.319 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:30:46.378 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:30:46.378 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:30:46.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:46.410+0000 7f60f74b3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:30:46.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:46.412+0000 7f60f74b3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:30:46.427 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:30:46.427 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:30:46.428 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:30:46.428 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:30:46.428 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:30:46.428 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:30:46.428 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:30:46.428 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:46.428 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:30:46.428 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:46.428 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:30:46.428 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:30:46.429 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:30:46.429 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:30:46.429 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:30:46.429 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:30:46.429 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:30:46.429 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:30:46.429 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:30:46.429 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:30:46.429 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:46.430 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:46.430 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:30:46.430 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:30:46.430 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:30:46.430 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:30:46.430 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:30:46.430 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:30:46.430 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:30:46.430 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:30:46.431 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:30:46.431 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:30:46.431 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:46.431 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:46.431 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:30:46.431 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:30:46.431 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:30:46.431 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713573040.767133_830.1", 2024-04-20T00:30:46.431 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:30:46.431 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:30:46.432 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:30:46.432 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:30:40.767133Z", 2024-04-20T00:30:46.432 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:30:46.432 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:30:46.432 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:46.432 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:30:46.432 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:30:46.432 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:30:46.432 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:30:46.432 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:30:46.433 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:30:46.433 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:30:46.433 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:30:46.433 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:30:46.433 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:30:46.433 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:30:46.433 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:30:46.433 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:30:46.433 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:46.434 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:30:46.434 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:30:46.434 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:30:46.434 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:30:46.434 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:30:46Z" 2024-04-20T00:30:46.434 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:30:46.435 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573040.767133_830.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:30:40.767133Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:30:46Z" } 2024-04-20T00:30:46.435 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573040.767133_830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:30:40.767133Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:30:46.436 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.767133_830.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:30:46.436 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.767133_830.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:30:46.436 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:30:46.436 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-68 2024-04-20T00:30:46.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:30:46.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:30:46.436 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T00:30:46.503 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:30:46.503 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:30:46.553 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-68 2024-04-20T00:30:46.573 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-68 2024-04-20T00:30:46.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:30:46.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:30:46.574 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:30:46.640 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:30:46.641 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:30:46.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:46.683+0000 7f964666e780 1 bucket sync caught up with source: 2024-04-20T00:30:46.685 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.37.6, , , , 00000000001.25.6, , , 00000000001.59.6, , 00000000001.77.6] 2024-04-20T00:30:46.685 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000001.37.6, , , , 00000000001.25.6, , , 00000000001.59.6, , 00000000001.77.6] 2024-04-20T00:30:46.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:46.683+0000 7f964666e780 0 bucket checkpoint complete 2024-04-20T00:30:46.700 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-68 2024-04-20T00:30:46.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-68', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:30:46.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-68', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:30:46.700 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-68 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:30:46.762 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:30:46.762 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:30:46.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:46.800+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:30:51.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:51.800+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:30:56.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:30:56.801+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:31:01.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:31:01.801+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:31:06.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:31:06.801+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:31:11.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:31:11.801+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:31:16.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:31:16.802+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:31:21.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:31:21.802+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:31:26.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:31:26.803+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:31:31.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:31:31.804+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:31:36.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:31:36.805+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:31:41.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:31:41.805+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:31:46.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:31:46.806+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:31:51.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:31:51.806+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:31:56.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:31:56.807+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:32:01.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:32:01.807+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:32:06.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:32:06.808+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:32:11.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:32:11.808+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:32:16.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:32:16.809+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:32:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:32:21.810+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:32:26.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:32:26.811+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:32:31.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:32:31.812+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:32:36.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:32:36.813+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:32:41.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:32:41.813+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:32:46.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:32:46.813+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:32:51.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:32:51.813+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:32:56.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:32:56.814+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:33:01.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:33:01.814+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:33:06.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:33:06.815+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:33:11.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:33:11.815+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:33:16.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:33:16.816+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:33:21.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:33:21.817+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:33:26.820 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:33:26.817+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:33:31.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:33:31.818+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:33:36.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:33:36.819+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:33:41.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:33:41.819+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:33:46.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:33:46.820+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:33:51.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:33:51.820+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:33:56.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:33:56.821+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:34:01.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:34:01.822+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:34:06.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:34:06.823+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:34:11.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:34:11.824+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:34:16.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:34:16.824+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:34:21.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:34:21.825+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:34:26.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:34:26.825+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:34:31.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:34:31.826+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:34:36.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:34:36.826+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:34:41.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:34:41.826+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:34:46.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:34:46.826+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:34:51.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:34:51.827+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:34:56.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:34:56.828+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:35:01.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:01.828+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:35:06.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:06.829+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:35:11.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:11.829+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:35:16.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:16.830+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:35:21.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:21.831+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:35:26.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:26.831+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:35:31.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:31.832+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:35:36.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:36.833+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T00:35:41.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:41.833+0000 7f8eda369780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T00:35:41.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:41.833+0000 7f8eda369780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:35:41.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:41.833+0000 7f8eda369780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:35:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T00:35:41.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2024-04-20T00:35:41.856 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-69 2024-04-20T00:35:41.857 DEBUG:boto:path=/aktlji-69/ 2024-04-20T00:35:41.857 DEBUG:boto:auth_path=/aktlji-69/ 2024-04-20T00:35:41.857 DEBUG:boto:Method: PUT 2024-04-20T00:35:41.857 DEBUG:boto:Path: /aktlji-69/ 2024-04-20T00:35:41.857 DEBUG:boto:Data: 2024-04-20T00:35:41.857 DEBUG:boto:Headers: {} 2024-04-20T00:35:41.857 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:35:41.857 DEBUG:boto:Port: 8000 2024-04-20T00:35:41.857 DEBUG:boto:Params: {} 2024-04-20T00:35:41.858 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:35:41.858 DEBUG:boto:Token: None 2024-04-20T00:35:41.858 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:35:41 GMT /aktlji-69/ 2024-04-20T00:35:41.858 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RsZ9hlvpQ2/5qdu6H0K7ePZ5sb0= 2024-04-20T00:35:41.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RsZ9hlvpQ2/5qdu6H0K7ePZ5sb0=', 'Content-Length': '0'} 2024-04-20T00:35:41.881 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000819e2b0ae7c12107-0066230ddd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:35:41.881 DEBUG:boto:path=/aktlji-69/ 2024-04-20T00:35:41.881 DEBUG:boto:auth_path=/aktlji-69/ 2024-04-20T00:35:41.881 DEBUG:boto:Method: HEAD 2024-04-20T00:35:41.882 DEBUG:boto:Path: /aktlji-69/ 2024-04-20T00:35:41.882 DEBUG:boto:Data: 2024-04-20T00:35:41.882 DEBUG:boto:Headers: {} 2024-04-20T00:35:41.882 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:35:41.882 DEBUG:boto:Port: 8000 2024-04-20T00:35:41.882 DEBUG:boto:Params: {} 2024-04-20T00:35:41.882 DEBUG:boto:Token: None 2024-04-20T00:35:41.882 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:35:41 GMT /aktlji-69/ 2024-04-20T00:35:41.882 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA= 2024-04-20T00:35:41.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA=', 'Content-Length': '0'} 2024-04-20T00:35:41.885 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000831ed5edae64b130-0066230ddd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:35:41.885 DEBUG:boto:path=/aktlji-69/a 2024-04-20T00:35:41.885 DEBUG:boto:auth_path=/aktlji-69/a 2024-04-20T00:35:41.885 DEBUG:boto:Method: PUT 2024-04-20T00:35:41.885 DEBUG:boto:Path: /aktlji-69/a 2024-04-20T00:35:41.886 DEBUG:boto:Data: 2024-04-20T00:35:41.886 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:35:41.886 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:35:41.886 DEBUG:boto:Port: 8000 2024-04-20T00:35:41.886 DEBUG:boto:Params: {} 2024-04-20T00:35:41.886 DEBUG:boto:Token: None 2024-04-20T00:35:41.886 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:35:41 GMT /aktlji-69/a 2024-04-20T00:35:41.886 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JfHFy/OhA/rSbOjMPxM3p1D/A1I= 2024-04-20T00:35:41.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JfHFy/OhA/rSbOjMPxM3p1D/A1I='} 2024-04-20T00:35:41.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066b9056edf44cf0c-0066230ddd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:35:41.894 DEBUG:boto:path=/aktlji-69/ 2024-04-20T00:35:41.894 DEBUG:boto:auth_path=/aktlji-69/ 2024-04-20T00:35:41.895 DEBUG:boto:Method: HEAD 2024-04-20T00:35:41.895 DEBUG:boto:Path: /aktlji-69/ 2024-04-20T00:35:41.895 DEBUG:boto:Data: 2024-04-20T00:35:41.895 DEBUG:boto:Headers: {} 2024-04-20T00:35:41.895 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:35:41.895 DEBUG:boto:Port: 8000 2024-04-20T00:35:41.895 DEBUG:boto:Params: {} 2024-04-20T00:35:41.895 DEBUG:boto:Token: None 2024-04-20T00:35:41.895 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:35:41 GMT /aktlji-69/ 2024-04-20T00:35:41.895 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA= 2024-04-20T00:35:41.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA=', 'Content-Length': '0'} 2024-04-20T00:35:41.897 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f59c59a60128626-0066230ddd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:35:41.898 DEBUG:boto:path=/aktlji-69/b 2024-04-20T00:35:41.898 DEBUG:boto:auth_path=/aktlji-69/b 2024-04-20T00:35:41.898 DEBUG:boto:Method: PUT 2024-04-20T00:35:41.898 DEBUG:boto:Path: /aktlji-69/b 2024-04-20T00:35:41.898 DEBUG:boto:Data: 2024-04-20T00:35:41.898 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:35:41.898 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:35:41.898 DEBUG:boto:Port: 8000 2024-04-20T00:35:41.898 DEBUG:boto:Params: {} 2024-04-20T00:35:41.899 DEBUG:boto:Token: None 2024-04-20T00:35:41.899 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:35:41 GMT /aktlji-69/b 2024-04-20T00:35:41.899 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:E+AVTVhq93OAQMTTXYbu02nub7s= 2024-04-20T00:35:41.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E+AVTVhq93OAQMTTXYbu02nub7s='} 2024-04-20T00:35:41.905 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1e525b38e8b7a3b-0066230ddd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:35:41.905 DEBUG:boto:path=/aktlji-69/ 2024-04-20T00:35:41.905 DEBUG:boto:auth_path=/aktlji-69/ 2024-04-20T00:35:41.905 DEBUG:boto:Method: HEAD 2024-04-20T00:35:41.905 DEBUG:boto:Path: /aktlji-69/ 2024-04-20T00:35:41.905 DEBUG:boto:Data: 2024-04-20T00:35:41.906 DEBUG:boto:Headers: {} 2024-04-20T00:35:41.906 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:35:41.906 DEBUG:boto:Port: 8000 2024-04-20T00:35:41.906 DEBUG:boto:Params: {} 2024-04-20T00:35:41.906 DEBUG:boto:Token: None 2024-04-20T00:35:41.906 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:35:41 GMT /aktlji-69/ 2024-04-20T00:35:41.906 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA= 2024-04-20T00:35:41.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA=', 'Content-Length': '0'} 2024-04-20T00:35:41.908 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5edb605fadf1ba8-0066230ddd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:35:41.909 DEBUG:boto:path=/aktlji-69/c 2024-04-20T00:35:41.909 DEBUG:boto:auth_path=/aktlji-69/c 2024-04-20T00:35:41.909 DEBUG:boto:Method: PUT 2024-04-20T00:35:41.909 DEBUG:boto:Path: /aktlji-69/c 2024-04-20T00:35:41.909 DEBUG:boto:Data: 2024-04-20T00:35:41.909 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:35:41.909 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:35:41.909 DEBUG:boto:Port: 8000 2024-04-20T00:35:41.909 DEBUG:boto:Params: {} 2024-04-20T00:35:41.910 DEBUG:boto:Token: None 2024-04-20T00:35:41.910 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:35:41 GMT /aktlji-69/c 2024-04-20T00:35:41.910 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:slCGFJ6lAshGu+0IH8RW6I5iSQ4= 2024-04-20T00:35:41.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:slCGFJ6lAshGu+0IH8RW6I5iSQ4='} 2024-04-20T00:35:41.955 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a249b1bc9f58d27-0066230ddd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:35:41.955 DEBUG:boto:path=/aktlji-69/ 2024-04-20T00:35:41.955 DEBUG:boto:auth_path=/aktlji-69/ 2024-04-20T00:35:41.955 DEBUG:boto:Method: HEAD 2024-04-20T00:35:41.955 DEBUG:boto:Path: /aktlji-69/ 2024-04-20T00:35:41.955 DEBUG:boto:Data: 2024-04-20T00:35:41.955 DEBUG:boto:Headers: {} 2024-04-20T00:35:41.955 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:35:41.955 DEBUG:boto:Port: 8000 2024-04-20T00:35:41.955 DEBUG:boto:Params: {} 2024-04-20T00:35:41.955 DEBUG:boto:Token: None 2024-04-20T00:35:41.956 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:35:41 GMT /aktlji-69/ 2024-04-20T00:35:41.956 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA= 2024-04-20T00:35:41.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA=', 'Content-Length': '0'} 2024-04-20T00:35:41.958 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000919f02caf6671097-0066230ddd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:35:41.959 DEBUG:boto:path=/aktlji-69/d 2024-04-20T00:35:41.959 DEBUG:boto:auth_path=/aktlji-69/d 2024-04-20T00:35:41.959 DEBUG:boto:Method: PUT 2024-04-20T00:35:41.959 DEBUG:boto:Path: /aktlji-69/d 2024-04-20T00:35:41.959 DEBUG:boto:Data: 2024-04-20T00:35:41.959 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:35:41.959 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:35:41.959 DEBUG:boto:Port: 8000 2024-04-20T00:35:41.959 DEBUG:boto:Params: {} 2024-04-20T00:35:41.959 DEBUG:boto:Token: None 2024-04-20T00:35:41.959 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:35:41 GMT /aktlji-69/d 2024-04-20T00:35:41.960 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kCIiwyTcW6hugRqgjWlksA1CO3w= 2024-04-20T00:35:41.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kCIiwyTcW6hugRqgjWlksA1CO3w='} 2024-04-20T00:35:41.967 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000255cfcdb6245d1ec-0066230ddd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:35:41.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:35:41.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:35:41.968 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:35:42.025 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:35:42.025 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:35:42.069 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:35:42.083 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:35:42.083 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:35:42.084 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:35:42.084 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:35:42.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:35:42.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:35:42.084 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:35:42.084 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:35:42.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:35:42.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:35:42.084 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713573341.878158_866.1", 2024-04-20T00:35:42.084 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:35:41.878158Z" 2024-04-20T00:35:42.084 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:35:42.084 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:35:42.085 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:35:42.085 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:35:42.085 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:35:42.085 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:35:42.086 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573341.878158_866.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:35:42.086 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:35:42.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:35:42.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:35:42.086 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:35:42.147 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:35:42.147 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:35:42.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:35:42.198+0000 7fed1cee1780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:35:42.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:35:42.199+0000 7fed1cee1780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:35:42.223 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:35:42.223 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:35:42.223 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:35:42.223 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:35:42.223 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:35:42.223 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:35:42.223 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:35:42.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:35:42.223 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:35:42.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:35:42.224 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:35:42.224 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:35:42.224 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:35:42.224 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:35:42.224 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:35:42.224 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:35:42.224 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:35:42.224 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:35:42.224 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:35:42.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:35:42.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:35:42.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:35:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:35:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:35:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:35:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:35:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:35:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:35:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:35:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:35:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:35:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:35:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:35:42.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:35:42.226 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:35:42.226 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:35:42.226 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:35:42.226 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713573341.878158_866.1", 2024-04-20T00:35:42.226 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:35:42.226 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:35:42.226 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:35:42.226 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:35:41.878158Z", 2024-04-20T00:35:42.226 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:35:42.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:35:42.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:35:42.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:35:42.227 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:35:42.227 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:35:42.227 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:35:42.227 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:35:42.227 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:35:42.227 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:35:42.227 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:35:42.227 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:35:42.227 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:35:42.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:35:42.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:35:42.227 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:35:42.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:35:42.228 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:35:42.228 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:35:42.228 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:35:42.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:35:42.228 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:35:42Z" 2024-04-20T00:35:42.228 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:35:42.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573341.878158_866.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:35:41.878158Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:35:42Z" } 2024-04-20T00:35:42.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573341.878158_866.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:35:41.878158Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:35:42.229 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573341.878158_866.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:35:42.229 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573341.878158_866.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:35:42.229 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:35:42.230 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:35:42.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:35:42.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:35:42.230 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:35:42.289 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:35:42.289 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:35:42.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:42.322+0000 7f06a6fdb780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:35:42.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:42.323+0000 7f06a6fdb780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:35:42.340 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:35:42.340 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:35:42.340 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:35:42.340 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:35:42.340 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:35:42.340 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:35:42.340 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:35:42.340 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:35:42.340 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:35:42.340 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:35:42.340 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:35:42.341 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:35:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:35:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:35:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:35:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:35:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:35:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:35:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:35:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:35:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:35:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:35:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:35:42.342 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713573341.878158_866.1", 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:35:41.878158Z", 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:35:42.343 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:35:42.344 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:35:42.344 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:35:42.344 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:35:42.344 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:35:42.344 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:35:42.344 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:35:42.344 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:35:42.344 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:35:42.344 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:35:42.344 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:35:42.344 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:35:42.345 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:35:42.345 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:35:42.345 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:35:42.345 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:35:42Z" 2024-04-20T00:35:42.345 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:35:42.346 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573341.878158_866.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:35:41.878158Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:35:42Z" } 2024-04-20T00:35:42.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573341.878158_866.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:35:41.878158Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:35:42.346 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573341.878158_866.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:35:42.346 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573341.878158_866.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:35:42.346 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:35:42.346 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-69 2024-04-20T00:35:42.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:35:42.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:35:42.347 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T00:35:42.413 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:35:42.413 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:35:42.461 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-69 2024-04-20T00:35:42.481 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-69 2024-04-20T00:35:42.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:35:42.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:35:42.481 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:35:42.563 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:35:42.563 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:35:42.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:42.636+0000 7f7389d5d780 1 waiting to reach incremental sync.. 2024-04-20T00:35:47.640 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:47.638+0000 7f7389d5d780 1 bucket sync caught up with source: 2024-04-20T00:35:47.640 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.53.6, , , , 00000000001.51.6, , , 00000000001.41.6, , 00000000001.31.6] 2024-04-20T00:35:47.641 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000001.53.6, , , , 00000000001.51.6, , , 00000000001.41.6, , 00000000001.31.6] 2024-04-20T00:35:47.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:47.638+0000 7f7389d5d780 0 bucket checkpoint complete 2024-04-20T00:35:47.658 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-69 2024-04-20T00:35:47.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:35:47.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:35:47.658 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:35:47.724 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:35:47.724 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:35:47.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:47.763+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:35:52.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:52.764+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:35:57.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:35:57.765+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:36:02.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:36:02.766+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:36:07.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:36:07.767+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:36:12.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:36:12.767+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:36:17.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:36:17.768+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:36:22.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:36:22.768+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:36:27.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:36:27.768+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:36:32.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:36:32.768+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:36:37.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:36:37.769+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:36:42.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:36:42.769+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:36:47.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:36:47.770+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:36:52.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:36:52.770+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:36:57.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:36:57.771+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:37:02.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:37:02.772+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:37:07.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:37:07.772+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:37:12.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:37:12.773+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:37:17.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:37:17.773+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:37:22.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:37:22.774+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:37:27.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:37:27.774+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:37:32.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:37:32.775+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:37:37.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:37:37.776+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:37:42.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:37:42.777+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:37:47.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:37:47.778+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:37:52.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:37:52.778+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:37:57.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:37:57.779+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:38:02.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:38:02.779+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:38:07.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:38:07.780+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:38:12.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:38:12.781+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:38:17.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:38:17.780+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:38:22.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:38:22.781+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:38:27.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:38:27.782+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:38:32.784 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:38:32.782+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:38:37.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:38:37.782+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:38:42.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:38:42.783+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:38:47.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:38:47.784+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:38:52.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:38:52.785+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:38:57.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:38:57.786+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:39:02.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:39:02.787+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:39:07.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:39:07.787+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:39:12.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:39:12.788+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:39:17.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:39:17.788+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:39:22.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:39:22.789+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:39:27.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:39:27.790+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:39:32.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:39:32.791+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:39:37.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:39:37.792+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:39:42.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:39:42.792+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:39:47.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:39:47.793+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:39:52.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:39:52.793+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:39:57.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:39:57.794+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:40:02.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:02.794+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:40:07.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:07.794+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:40:12.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:12.795+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:40:17.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:17.795+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:40:22.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:22.796+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:40:27.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:27.796+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:40:32.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:32.797+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:40:37.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:37.798+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T00:40:42.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:42.798+0000 7f5e6af92780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T00:40:42.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:42.798+0000 7f5e6af92780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:40:42.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:42.798+0000 7f5e6af92780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:40:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T00:40:42.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2024-04-20T00:40:42.823 DEBUG:boto:path=/aktlji-70/ 2024-04-20T00:40:42.823 DEBUG:boto:auth_path=/aktlji-70/ 2024-04-20T00:40:42.823 DEBUG:boto:Method: PUT 2024-04-20T00:40:42.823 DEBUG:boto:Path: /aktlji-70/ 2024-04-20T00:40:42.823 DEBUG:boto:Data: 2024-04-20T00:40:42.823 DEBUG:boto:Headers: {} 2024-04-20T00:40:42.824 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:40:42.824 DEBUG:boto:Port: 8000 2024-04-20T00:40:42.824 DEBUG:boto:Params: {} 2024-04-20T00:40:42.824 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:40:42.824 DEBUG:boto:Token: None 2024-04-20T00:40:42.825 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:40:42 GMT /aktlji-70/ 2024-04-20T00:40:42.825 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q7m1pHNOKOcDvN8GwuzIzYFobO4= 2024-04-20T00:40:42.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:40:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q7m1pHNOKOcDvN8GwuzIzYFobO4=', 'Content-Length': '0'} 2024-04-20T00:40:42.849 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000022e7b86caa5b7bf6-0066230f0a-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:40:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:40:42.849 DEBUG:rgw_multi.tests:created bucket=aktlji-70 2024-04-20T00:40:42.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:40:42.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:40:42.850 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:40:42.909 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:40:42.909 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:40:42.950 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:40:42.965 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:40:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:40:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:40:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:40:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:40:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:40:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:40:42.966 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:40:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:40:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:40:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713573642.846158_911.1", 2024-04-20T00:40:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:40:42.846158Z" 2024-04-20T00:40:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:40:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:40:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:40:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:40:42.967 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:40:42.968 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:40:42.969 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573642.846158_911.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:40:42.969 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:40:42.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:40:42.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:40:42.969 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:40:43.030 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:40:43.030 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:40:43.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:40:43.081+0000 7f4f75025780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:40:43.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:40:43.082+0000 7f4f75025780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:40:43.102 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:40:43.103 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:40:43.103 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:40:43.103 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:40:43.103 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:40:43.103 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:40:43.103 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:40:43.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:40:43.103 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:40:43.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:40:43.103 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:40:43.104 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:40:43.104 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:40:43.104 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:40:43.104 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:40:43.104 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:40:43.104 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:40:43.104 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:40:43.104 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:40:43.104 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:40:43.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:40:43.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:40:43.105 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:40:43.105 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:40:43.105 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:40:43.105 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:40:43.105 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:40:43.105 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:40:43.105 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:40:43.105 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:40:43.106 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:40:43.106 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:40:43.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:40:43.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:40:43.106 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:40:43.106 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:40:43.106 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:40:43.106 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713573642.846158_911.1", 2024-04-20T00:40:43.106 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:40:43.107 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:40:43.107 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:40:43.107 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:40:42.846158Z", 2024-04-20T00:40:43.107 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:40:43.107 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:40:43.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:40:43.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:40:43.107 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:40:43.107 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:40:43.107 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:40:43.108 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:40:43.108 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:40:43.108 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:40:43.108 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:40:43.108 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:40:43.108 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:40:43.108 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:40:43.108 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:40:43.108 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:40:43.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:40:43.109 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:40:43.109 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:40:43.109 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:40:43.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:40:43.109 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:40:43Z" 2024-04-20T00:40:43.109 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:40:43.110 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573642.846158_911.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:40:42.846158Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:40:43Z" } 2024-04-20T00:40:43.110 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573642.846158_911.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:40:42.846158Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:40:43.110 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573642.846158_911.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:40:43.111 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573642.846158_911.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:40:43.111 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:40:43.111 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:40:43.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:40:43.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:40:43.111 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:40:43.171 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:40:43.171 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:40:43.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:43.204+0000 7f713f8ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:40:43.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:43.206+0000 7f713f8ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:40:43.222 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:40:43.222 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:40:43.223 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:40:43.223 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:40:43.223 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:40:43.223 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:40:43.223 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:40:43.223 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:40:43.223 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:40:43.223 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:40:43.223 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:40:43.224 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:40:43.224 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:40:43.224 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:40:43.224 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:40:43.224 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:40:43.224 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:40:43.224 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:40:43.224 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:40:43.224 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:40:43.224 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:40:43.224 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:40:43.225 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:40:43.225 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:40:43.225 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:40:43.225 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:40:43.225 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:40:43.225 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:40:43.225 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:40:43.225 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:40:43.225 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:40:43.225 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:40:43.226 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:40:43.226 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:40:43.226 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:40:43.226 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:40:43.226 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:40:43.226 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713573642.846158_911.1", 2024-04-20T00:40:43.226 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:40:43.226 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:40:43.226 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:40:43.226 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:40:42.846158Z", 2024-04-20T00:40:43.227 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:40:43.227 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:40:43.227 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:40:43.227 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:40:43.227 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:40:43.227 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:40:43.227 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:40:43.227 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:40:43.227 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:40:43.227 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:40:43.228 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:40:43.228 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:40:43.228 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:40:43.228 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:40:43.228 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:40:43.228 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:40:43.228 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:40:43.228 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:40:43.228 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:40:43.228 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:40:43.229 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:40:43.229 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:40:43Z" 2024-04-20T00:40:43.229 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:40:43.230 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573642.846158_911.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:40:42.846158Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:40:43Z" } 2024-04-20T00:40:43.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573642.846158_911.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:40:42.846158Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:40:43.230 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573642.846158_911.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:40:43.230 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573642.846158_911.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:40:43.230 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:40:43.231 DEBUG:boto:path=/aktlji-70/ 2024-04-20T00:40:43.231 DEBUG:boto:auth_path=/aktlji-70/ 2024-04-20T00:40:43.231 DEBUG:boto:Method: HEAD 2024-04-20T00:40:43.231 DEBUG:boto:Path: /aktlji-70/ 2024-04-20T00:40:43.231 DEBUG:boto:Data: 2024-04-20T00:40:43.231 DEBUG:boto:Headers: {} 2024-04-20T00:40:43.231 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:40:43.231 DEBUG:boto:Port: 8000 2024-04-20T00:40:43.231 DEBUG:boto:Params: {} 2024-04-20T00:40:43.232 DEBUG:boto:Token: None 2024-04-20T00:40:43.232 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:40:43 GMT /aktlji-70/ 2024-04-20T00:40:43.232 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU= 2024-04-20T00:40:43.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU=', 'Content-Length': '0'} 2024-04-20T00:40:43.234 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096de8cc27e5e9e04-0066230f0b-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:40:43.235 DEBUG:boto:path=/aktlji-70/a 2024-04-20T00:40:43.235 DEBUG:boto:auth_path=/aktlji-70/a 2024-04-20T00:40:43.235 DEBUG:boto:Method: PUT 2024-04-20T00:40:43.235 DEBUG:boto:Path: /aktlji-70/a 2024-04-20T00:40:43.235 DEBUG:boto:Data: 2024-04-20T00:40:43.235 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:40:43.236 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:40:43.236 DEBUG:boto:Port: 8000 2024-04-20T00:40:43.236 DEBUG:boto:Params: {} 2024-04-20T00:40:43.236 DEBUG:boto:Token: None 2024-04-20T00:40:43.236 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:40:43 GMT /aktlji-70/a 2024-04-20T00:40:43.236 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Y3RPz0A+Gk4dVFzdB2zg99+mACA= 2024-04-20T00:40:43.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y3RPz0A+Gk4dVFzdB2zg99+mACA='} 2024-04-20T00:40:43.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fccb80b3b5b32578-0066230f0b-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:40:43.244 DEBUG:boto:path=/aktlji-70/ 2024-04-20T00:40:43.244 DEBUG:boto:auth_path=/aktlji-70/ 2024-04-20T00:40:43.244 DEBUG:boto:Method: HEAD 2024-04-20T00:40:43.244 DEBUG:boto:Path: /aktlji-70/ 2024-04-20T00:40:43.244 DEBUG:boto:Data: 2024-04-20T00:40:43.245 DEBUG:boto:Headers: {} 2024-04-20T00:40:43.245 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:40:43.245 DEBUG:boto:Port: 8000 2024-04-20T00:40:43.245 DEBUG:boto:Params: {} 2024-04-20T00:40:43.245 DEBUG:boto:Token: None 2024-04-20T00:40:43.245 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:40:43 GMT /aktlji-70/ 2024-04-20T00:40:43.245 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU= 2024-04-20T00:40:43.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU=', 'Content-Length': '0'} 2024-04-20T00:40:43.248 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b62e71e86dccea6-0066230f0b-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:40:43.248 DEBUG:boto:path=/aktlji-70/b 2024-04-20T00:40:43.248 DEBUG:boto:auth_path=/aktlji-70/b 2024-04-20T00:40:43.248 DEBUG:boto:Method: PUT 2024-04-20T00:40:43.248 DEBUG:boto:Path: /aktlji-70/b 2024-04-20T00:40:43.248 DEBUG:boto:Data: 2024-04-20T00:40:43.249 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:40:43.249 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:40:43.249 DEBUG:boto:Port: 8000 2024-04-20T00:40:43.249 DEBUG:boto:Params: {} 2024-04-20T00:40:43.249 DEBUG:boto:Token: None 2024-04-20T00:40:43.249 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:40:43 GMT /aktlji-70/b 2024-04-20T00:40:43.249 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gkNcH1aGAP0gK0gK1hWBassHdfI= 2024-04-20T00:40:43.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gkNcH1aGAP0gK0gK1hWBassHdfI='} 2024-04-20T00:40:43.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014e3297547e1738a-0066230f0b-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:40:43.295 DEBUG:boto:path=/aktlji-70/ 2024-04-20T00:40:43.295 DEBUG:boto:auth_path=/aktlji-70/ 2024-04-20T00:40:43.295 DEBUG:boto:Method: HEAD 2024-04-20T00:40:43.295 DEBUG:boto:Path: /aktlji-70/ 2024-04-20T00:40:43.295 DEBUG:boto:Data: 2024-04-20T00:40:43.295 DEBUG:boto:Headers: {} 2024-04-20T00:40:43.295 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:40:43.295 DEBUG:boto:Port: 8000 2024-04-20T00:40:43.296 DEBUG:boto:Params: {} 2024-04-20T00:40:43.296 DEBUG:boto:Token: None 2024-04-20T00:40:43.296 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:40:43 GMT /aktlji-70/ 2024-04-20T00:40:43.296 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU= 2024-04-20T00:40:43.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU=', 'Content-Length': '0'} 2024-04-20T00:40:43.298 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039c4eca17803c9bc-0066230f0b-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:40:43.299 DEBUG:boto:path=/aktlji-70/c 2024-04-20T00:40:43.299 DEBUG:boto:auth_path=/aktlji-70/c 2024-04-20T00:40:43.299 DEBUG:boto:Method: PUT 2024-04-20T00:40:43.299 DEBUG:boto:Path: /aktlji-70/c 2024-04-20T00:40:43.299 DEBUG:boto:Data: 2024-04-20T00:40:43.299 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:40:43.299 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:40:43.299 DEBUG:boto:Port: 8000 2024-04-20T00:40:43.300 DEBUG:boto:Params: {} 2024-04-20T00:40:43.300 DEBUG:boto:Token: None 2024-04-20T00:40:43.300 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:40:43 GMT /aktlji-70/c 2024-04-20T00:40:43.300 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cEKCPFA8Cnw60001l3KY1AzfWb0= 2024-04-20T00:40:43.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cEKCPFA8Cnw60001l3KY1AzfWb0='} 2024-04-20T00:40:43.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e6aba41482bb90f-0066230f0b-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:40:43.307 DEBUG:boto:path=/aktlji-70/ 2024-04-20T00:40:43.307 DEBUG:boto:auth_path=/aktlji-70/ 2024-04-20T00:40:43.307 DEBUG:boto:Method: HEAD 2024-04-20T00:40:43.307 DEBUG:boto:Path: /aktlji-70/ 2024-04-20T00:40:43.307 DEBUG:boto:Data: 2024-04-20T00:40:43.307 DEBUG:boto:Headers: {} 2024-04-20T00:40:43.307 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:40:43.307 DEBUG:boto:Port: 8000 2024-04-20T00:40:43.307 DEBUG:boto:Params: {} 2024-04-20T00:40:43.308 DEBUG:boto:Token: None 2024-04-20T00:40:43.308 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:40:43 GMT /aktlji-70/ 2024-04-20T00:40:43.308 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU= 2024-04-20T00:40:43.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU=', 'Content-Length': '0'} 2024-04-20T00:40:43.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d562664088c7fb82-0066230f0b-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:40:43.311 DEBUG:boto:path=/aktlji-70/d 2024-04-20T00:40:43.311 DEBUG:boto:auth_path=/aktlji-70/d 2024-04-20T00:40:43.311 DEBUG:boto:Method: PUT 2024-04-20T00:40:43.311 DEBUG:boto:Path: /aktlji-70/d 2024-04-20T00:40:43.311 DEBUG:boto:Data: 2024-04-20T00:40:43.311 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:40:43.312 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:40:43.312 DEBUG:boto:Port: 8000 2024-04-20T00:40:43.312 DEBUG:boto:Params: {} 2024-04-20T00:40:43.312 DEBUG:boto:Token: None 2024-04-20T00:40:43.312 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:40:43 GMT /aktlji-70/d 2024-04-20T00:40:43.312 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eIkW0cm0lW6PGdheRPdFQhadDEw= 2024-04-20T00:40:43.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eIkW0cm0lW6PGdheRPdFQhadDEw='} 2024-04-20T00:40:43.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b80b6d2d0c43672-0066230f0b-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:40:43.355 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-70 2024-04-20T00:40:43.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:40:43.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:40:43.355 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T00:40:43.423 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:40:43.423 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:40:43.467 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-70 2024-04-20T00:40:43.495 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-70 2024-04-20T00:40:43.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:40:43.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:40:43.496 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:40:43.565 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:40:43.565 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:40:43.607 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:43.605+0000 7f912d553780 1 waiting to reach incremental sync.. 2024-04-20T00:40:48.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:48.608+0000 7f912d553780 1 bucket sync caught up with source: 2024-04-20T00:40:48.610 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, 00000000001.36.6, , , , 00000000001.54.6, , , 00000000001.47.6, , 00000000001.62.6] 2024-04-20T00:40:48.610 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, 00000000001.36.6, , , , 00000000001.54.6, , , 00000000001.47.6, , 00000000001.62.6] 2024-04-20T00:40:48.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:48.608+0000 7f912d553780 0 bucket checkpoint complete 2024-04-20T00:40:48.628 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-70 2024-04-20T00:40:48.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:40:48.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:40:48.628 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:40:48.693 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:40:48.693 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:40:48.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:48.733+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:40:53.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:53.734+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:40:58.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:40:58.734+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:41:03.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:41:03.735+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:41:08.738 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:41:08.736+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:41:13.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:41:13.736+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:41:18.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:41:18.737+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:41:23.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:41:23.738+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:41:28.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:41:28.739+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:41:33.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:41:33.740+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:41:38.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:41:38.740+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:41:43.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:41:43.741+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:41:48.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:41:48.741+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:41:53.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:41:53.743+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:41:58.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:41:58.743+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:42:03.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:42:03.744+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:42:08.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:42:08.744+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:42:13.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:42:13.745+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:42:18.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:42:18.745+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:42:23.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:42:23.746+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:42:28.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:42:28.746+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:42:33.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:42:33.747+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:42:38.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:42:38.747+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:42:43.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:42:43.748+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:42:48.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:42:48.749+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:42:53.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:42:53.752+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:42:58.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:42:58.753+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:43:03.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:43:03.753+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:43:08.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:43:08.755+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:43:13.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:43:13.756+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:43:18.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:43:18.756+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:43:23.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:43:23.757+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:43:28.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:43:28.758+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:43:33.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:43:33.758+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:43:38.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:43:38.759+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:43:43.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:43:43.759+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:43:48.761 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:43:48.759+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:43:53.762 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:43:53.760+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:43:58.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:43:58.760+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:44:03.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:44:03.761+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:44:08.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:44:08.762+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:44:13.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:44:13.762+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:44:18.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:44:18.762+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:44:23.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:44:23.763+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:44:28.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:44:28.764+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:44:33.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:44:33.764+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:44:38.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:44:38.765+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:44:43.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:44:43.766+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:44:48.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:44:48.767+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:44:53.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:44:53.767+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:44:58.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:44:58.768+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:45:03.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:03.769+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:45:08.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:08.769+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:45:13.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:13.770+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:45:18.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:18.770+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:45:23.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:23.771+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:45:28.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:28.771+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:45:33.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:33.771+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:45:38.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:38.772+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T00:45:43.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:43.772+0000 7f166948a780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T00:45:43.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:43.772+0000 7f166948a780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:45:43.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:43.772+0000 7f166948a780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:45:43.792 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T00:45:43.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... FAIL 2024-04-20T00:45:43.796 DEBUG:boto:path=/aktlji-71/ 2024-04-20T00:45:43.797 DEBUG:boto:auth_path=/aktlji-71/ 2024-04-20T00:45:43.797 DEBUG:boto:Method: PUT 2024-04-20T00:45:43.797 DEBUG:boto:Path: /aktlji-71/ 2024-04-20T00:45:43.797 DEBUG:boto:Data: 2024-04-20T00:45:43.797 DEBUG:boto:Headers: {} 2024-04-20T00:45:43.797 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:45:43.797 DEBUG:boto:Port: 8000 2024-04-20T00:45:43.797 DEBUG:boto:Params: {} 2024-04-20T00:45:43.798 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:45:43.798 DEBUG:boto:Token: None 2024-04-20T00:45:43.798 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:45:43 GMT /aktlji-71/ 2024-04-20T00:45:43.798 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WId1r/HRr3QRORISmp0vCa5QP7I= 2024-04-20T00:45:43.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:45:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WId1r/HRr3QRORISmp0vCa5QP7I=', 'Content-Length': '0'} 2024-04-20T00:45:43.823 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b79229535b68761f-0066231037-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:45:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:45:43.823 DEBUG:rgw_multi.tests:created bucket=aktlji-71 2024-04-20T00:45:43.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:45:43.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:45:43.824 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:45:43.883 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:45:43.883 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:45:43.926 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:45:43.941 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:45:43.941 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:45:43.941 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:45:43.941 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:45:43.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:45:43.942 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:45:43.942 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:45:43.942 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:45:43.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:45:43.942 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:45:43.942 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713573943.819700_950.1", 2024-04-20T00:45:43.942 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:45:43.819700Z" 2024-04-20T00:45:43.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:45:43.943 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:45:43.943 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:45:43.943 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:45:43.943 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:45:43.943 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:45:43.944 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.819700_950.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:45:43.944 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:45:43.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:45:43.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:45:43.944 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:45:44.003 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:45:44.003 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:45:44.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:45:44.050+0000 7fd8b71ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:45:44.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:45:44.052+0000 7fd8b71ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:45:44.074 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:45:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:45:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:45:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:45:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:45:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:45:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:45:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:45:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:45:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:45:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:45:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:45:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:45:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:45:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:45:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:45:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:45:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:45:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:45:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:45:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:45:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:45:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:45:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:45:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:45:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:45:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:45:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:45:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:45:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:45:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:45:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:45:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:45:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:45:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:45:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:45:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:45:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713573943.811171_947.1", 2024-04-20T00:45:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:45:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:45:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:45:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:45:43.811171Z", 2024-04-20T00:45:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:45:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:45:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:45:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:45:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:45:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:45:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:45:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:45:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:45:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:45:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:45:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:45:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:45:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:45:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:45:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:45:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:45:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:45:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:45:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:45:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:45:44.080 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:45:44Z" 2024-04-20T00:45:44.080 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:45:44.081 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573943.811171_947.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:45:43.811171Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:45:44Z" } 2024-04-20T00:45:44.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573943.811171_947.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:45:43.811171Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:45:44.082 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.819700_950.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:45:44.082 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.811171_947.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:45:44.082 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713573943.819700_950.1 target=1_1713573943.811171_947.1 2024-04-20T00:45:49.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:45:49.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:45:49.084 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:45:49.143 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:45:49.143 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:45:49.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:45:49.186+0000 7fc99df5f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:45:49.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:45:49.188+0000 7fc99df5f780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:45:49.209 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:45:49.210 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:45:49.210 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:45:49.210 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:45:49.210 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:45:49.210 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:45:49.210 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:45:49.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:45:49.210 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:45:49.210 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:45:49.210 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:45:49.210 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:45:49.211 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:45:49.211 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:45:49.211 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:45:49.211 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:45:49.211 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:45:49.211 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:45:49.211 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:45:49.211 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:45:49.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:45:49.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:45:49.211 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:45:49.211 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:45:49.212 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713573943.819700_950.1", 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:45:43.819700Z", 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:45:49.213 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:45:49.214 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:45:49.214 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:45:49.214 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:45:49.214 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:45:49.214 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:45:49.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:45:49.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:45:49.214 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:45:49.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:45:49.214 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:45:49.214 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:45:49.214 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:45:49.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:45:49.215 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:45:49Z" 2024-04-20T00:45:49.215 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:45:49.215 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573943.819700_950.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:45:43.819700Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:45:49Z" } 2024-04-20T00:45:49.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573943.819700_950.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:45:43.819700Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:45:49.216 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.819700_950.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:45:49.216 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.819700_950.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:45:49.216 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:45:49.216 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:45:49.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:45:49.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:45:49.217 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:45:49.276 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:45:49.276 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:45:49.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:49.310+0000 7fddcba88780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:45:49.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:45:49.311+0000 7fddcba88780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:45:49.328 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:45:49.328 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:45:49.328 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:45:49.328 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:45:49.328 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:45:49.328 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:45:49.328 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:45:49.328 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:45:49.328 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:45:49.328 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:45:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:45:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:45:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:45:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:45:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:45:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:45:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:45:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:45:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:45:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:45:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:45:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:45:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:45:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:45:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:45:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:45:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:45:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:45:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:45:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:45:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:45:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:45:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:45:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:45:49.331 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:45:49.331 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:45:49.331 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:45:49.331 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713573943.819700_950.1", 2024-04-20T00:45:49.331 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:45:49.331 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:45:49.331 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:45:49.331 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:45:43.819700Z", 2024-04-20T00:45:49.331 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:45:49.331 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:45:49.331 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:45:49.331 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:45:49.332 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:45:49.332 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:45:49.332 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:45:49.332 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:45:49.332 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:45:49.332 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:45:49.332 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:45:49.332 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:45:49.332 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:45:49.332 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:45:49.332 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:45:49.332 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:45:49.333 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:45:49.333 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:45:49.333 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:45:49.333 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:45:49.333 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:45:49.333 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:45:49Z" 2024-04-20T00:45:49.333 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:45:49.334 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573943.819700_950.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:45:43.819700Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:45:49Z" } 2024-04-20T00:45:49.334 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573943.819700_950.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:45:43.819700Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:45:49.334 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.819700_950.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:45:49.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.819700_950.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:45:49.337 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:45:49.337 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-20T00:45:49.337 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T00:45:49.338 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-20T00:45:49.337+0000 7fe8dd09a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 48535) UID: 0 2024-04-20T00:45:49.338 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-20T00:45:49.337+0000 7fe8e226f7c0 -1 shutting down 2024-04-20T00:45:53.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-20T00:45:55.440 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-20T00:45:55.441 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-20T00:45:55.441 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T00:45:55.442 INFO:tasks.rgw.c1.client.1.smithi019.stdout:2024-04-20T00:45:55.440+0000 7f37d525f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 49475) UID: 0 2024-04-20T00:45:55.442 INFO:tasks.rgw.c1.client.1.smithi019.stdout:2024-04-20T00:45:55.440+0000 7f37da4347c0 -1 shutting down 2024-04-20T00:45:59.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-20T00:45:59.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-20T00:46:01.542 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-20T00:46:01.543 DEBUG:boto:path=/aktlji-71/ 2024-04-20T00:46:01.543 DEBUG:boto:auth_path=/aktlji-71/ 2024-04-20T00:46:01.543 DEBUG:boto:Method: HEAD 2024-04-20T00:46:01.543 DEBUG:boto:Path: /aktlji-71/ 2024-04-20T00:46:01.543 DEBUG:boto:Data: 2024-04-20T00:46:01.543 DEBUG:boto:Headers: {} 2024-04-20T00:46:01.544 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.544 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.544 DEBUG:boto:Params: {} 2024-04-20T00:46:01.544 DEBUG:boto:Token: None 2024-04-20T00:46:01.544 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/ 2024-04-20T00:46:01.544 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T00:46:01.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T00:46:01.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009eb5c3b0e3b09280-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.548 DEBUG:boto:path=/aktlji-71/a 2024-04-20T00:46:01.548 DEBUG:boto:auth_path=/aktlji-71/a 2024-04-20T00:46:01.548 DEBUG:boto:Method: PUT 2024-04-20T00:46:01.548 DEBUG:boto:Path: /aktlji-71/a 2024-04-20T00:46:01.548 DEBUG:boto:Data: 2024-04-20T00:46:01.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:46:01.549 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.549 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.549 DEBUG:boto:Params: {} 2024-04-20T00:46:01.549 DEBUG:boto:Token: None 2024-04-20T00:46:01.550 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/a 2024-04-20T00:46:01.550 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:H6YcSYwXBr0vfVTRUJPVO6R4tJ8= 2024-04-20T00:46:01.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:H6YcSYwXBr0vfVTRUJPVO6R4tJ8='} 2024-04-20T00:46:01.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ec3b5e92aaa8a35-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.559 DEBUG:boto:path=/aktlji-71/ 2024-04-20T00:46:01.559 DEBUG:boto:auth_path=/aktlji-71/ 2024-04-20T00:46:01.559 DEBUG:boto:Method: HEAD 2024-04-20T00:46:01.559 DEBUG:boto:Path: /aktlji-71/ 2024-04-20T00:46:01.559 DEBUG:boto:Data: 2024-04-20T00:46:01.559 DEBUG:boto:Headers: {} 2024-04-20T00:46:01.559 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.559 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.560 DEBUG:boto:Params: {} 2024-04-20T00:46:01.560 DEBUG:boto:Token: None 2024-04-20T00:46:01.560 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/ 2024-04-20T00:46:01.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T00:46:01.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T00:46:01.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af281c41a8107049-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.563 DEBUG:boto:path=/aktlji-71/b 2024-04-20T00:46:01.563 DEBUG:boto:auth_path=/aktlji-71/b 2024-04-20T00:46:01.563 DEBUG:boto:Method: PUT 2024-04-20T00:46:01.563 DEBUG:boto:Path: /aktlji-71/b 2024-04-20T00:46:01.563 DEBUG:boto:Data: 2024-04-20T00:46:01.563 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:46:01.563 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.564 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.564 DEBUG:boto:Params: {} 2024-04-20T00:46:01.564 DEBUG:boto:Token: None 2024-04-20T00:46:01.564 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/b 2024-04-20T00:46:01.564 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kvk+rS/pHLXR0LjLY77Za3O/Ezc= 2024-04-20T00:46:01.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kvk+rS/pHLXR0LjLY77Za3O/Ezc='} 2024-04-20T00:46:01.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098474d4a012b9d21-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.607 DEBUG:boto:path=/aktlji-71/ 2024-04-20T00:46:01.607 DEBUG:boto:auth_path=/aktlji-71/ 2024-04-20T00:46:01.607 DEBUG:boto:Method: HEAD 2024-04-20T00:46:01.607 DEBUG:boto:Path: /aktlji-71/ 2024-04-20T00:46:01.607 DEBUG:boto:Data: 2024-04-20T00:46:01.607 DEBUG:boto:Headers: {} 2024-04-20T00:46:01.607 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.607 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.608 DEBUG:boto:Params: {} 2024-04-20T00:46:01.608 DEBUG:boto:Token: None 2024-04-20T00:46:01.608 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/ 2024-04-20T00:46:01.608 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T00:46:01.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T00:46:01.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001691372bd535cf84-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.611 DEBUG:boto:path=/aktlji-71/c 2024-04-20T00:46:01.611 DEBUG:boto:auth_path=/aktlji-71/c 2024-04-20T00:46:01.611 DEBUG:boto:Method: PUT 2024-04-20T00:46:01.611 DEBUG:boto:Path: /aktlji-71/c 2024-04-20T00:46:01.611 DEBUG:boto:Data: 2024-04-20T00:46:01.611 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:46:01.612 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.612 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.612 DEBUG:boto:Params: {} 2024-04-20T00:46:01.612 DEBUG:boto:Token: None 2024-04-20T00:46:01.612 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/c 2024-04-20T00:46:01.612 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:L9Dly0WeXsCgOV6bKUPfSidhDKs= 2024-04-20T00:46:01.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L9Dly0WeXsCgOV6bKUPfSidhDKs='} 2024-04-20T00:46:01.619 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b3c74b42f332bcc-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.619 DEBUG:boto:path=/aktlji-71/ 2024-04-20T00:46:01.619 DEBUG:boto:auth_path=/aktlji-71/ 2024-04-20T00:46:01.619 DEBUG:boto:Method: HEAD 2024-04-20T00:46:01.619 DEBUG:boto:Path: /aktlji-71/ 2024-04-20T00:46:01.619 DEBUG:boto:Data: 2024-04-20T00:46:01.620 DEBUG:boto:Headers: {} 2024-04-20T00:46:01.620 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.620 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.620 DEBUG:boto:Params: {} 2024-04-20T00:46:01.620 DEBUG:boto:Token: None 2024-04-20T00:46:01.620 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/ 2024-04-20T00:46:01.620 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T00:46:01.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T00:46:01.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f8c7f8ea3e4855a-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.623 DEBUG:boto:path=/aktlji-71/d 2024-04-20T00:46:01.623 DEBUG:boto:auth_path=/aktlji-71/d 2024-04-20T00:46:01.623 DEBUG:boto:Method: PUT 2024-04-20T00:46:01.623 DEBUG:boto:Path: /aktlji-71/d 2024-04-20T00:46:01.623 DEBUG:boto:Data: 2024-04-20T00:46:01.623 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:46:01.623 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.624 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.624 DEBUG:boto:Params: {} 2024-04-20T00:46:01.624 DEBUG:boto:Token: None 2024-04-20T00:46:01.624 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/d 2024-04-20T00:46:01.624 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1AozUUsFcJDkcwZ2WMZQs5bHRns= 2024-04-20T00:46:01.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1AozUUsFcJDkcwZ2WMZQs5bHRns='} 2024-04-20T00:46:01.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ed055b1a4134d1f-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'aktlji-71', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:46:01.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'aktlji-71', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T00:46:01.668 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket aktlji-71 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T00:46:01.727 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:46:01.727 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:46:01.775 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T00:46:01.776 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-71 2024-04-20T00:46:01.780 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 4 2024-04-20T00:46:01.827 DEBUG:boto:path=/aktlji-71/ 2024-04-20T00:46:01.827 DEBUG:boto:auth_path=/aktlji-71/ 2024-04-20T00:46:01.828 DEBUG:boto:Method: HEAD 2024-04-20T00:46:01.828 DEBUG:boto:Path: /aktlji-71/ 2024-04-20T00:46:01.828 DEBUG:boto:Data: 2024-04-20T00:46:01.828 DEBUG:boto:Headers: {} 2024-04-20T00:46:01.828 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.828 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.828 DEBUG:boto:Params: {} 2024-04-20T00:46:01.828 DEBUG:boto:Token: None 2024-04-20T00:46:01.828 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/ 2024-04-20T00:46:01.829 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T00:46:01.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T00:46:01.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b8368ff32b9836a-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.832 DEBUG:boto:path=/aktlji-71/e 2024-04-20T00:46:01.832 DEBUG:boto:auth_path=/aktlji-71/e 2024-04-20T00:46:01.832 DEBUG:boto:Method: PUT 2024-04-20T00:46:01.832 DEBUG:boto:Path: /aktlji-71/e 2024-04-20T00:46:01.832 DEBUG:boto:Data: 2024-04-20T00:46:01.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:46:01.832 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.833 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.833 DEBUG:boto:Params: {} 2024-04-20T00:46:01.833 DEBUG:boto:Token: None 2024-04-20T00:46:01.833 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/e 2024-04-20T00:46:01.833 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/wu48I6TFzwV5fJTrx9A4IsS7Vw= 2024-04-20T00:46:01.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/wu48I6TFzwV5fJTrx9A4IsS7Vw='} 2024-04-20T00:46:01.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ec9205171adbf58-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.840 DEBUG:boto:path=/aktlji-71/ 2024-04-20T00:46:01.840 DEBUG:boto:auth_path=/aktlji-71/ 2024-04-20T00:46:01.840 DEBUG:boto:Method: HEAD 2024-04-20T00:46:01.840 DEBUG:boto:Path: /aktlji-71/ 2024-04-20T00:46:01.840 DEBUG:boto:Data: 2024-04-20T00:46:01.840 DEBUG:boto:Headers: {} 2024-04-20T00:46:01.840 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.840 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.840 DEBUG:boto:Params: {} 2024-04-20T00:46:01.841 DEBUG:boto:Token: None 2024-04-20T00:46:01.841 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/ 2024-04-20T00:46:01.841 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T00:46:01.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T00:46:01.843 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8be2ac9f6e12b33-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.843 DEBUG:boto:path=/aktlji-71/f 2024-04-20T00:46:01.843 DEBUG:boto:auth_path=/aktlji-71/f 2024-04-20T00:46:01.843 DEBUG:boto:Method: PUT 2024-04-20T00:46:01.843 DEBUG:boto:Path: /aktlji-71/f 2024-04-20T00:46:01.843 DEBUG:boto:Data: 2024-04-20T00:46:01.843 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:46:01.843 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.843 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.844 DEBUG:boto:Params: {} 2024-04-20T00:46:01.844 DEBUG:boto:Token: None 2024-04-20T00:46:01.844 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/f 2024-04-20T00:46:01.844 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:S9F20SUOIOASKEXZe/gztz7i4g0= 2024-04-20T00:46:01.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:S9F20SUOIOASKEXZe/gztz7i4g0='} 2024-04-20T00:46:01.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a85ca78514843554-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.887 DEBUG:boto:path=/aktlji-71/ 2024-04-20T00:46:01.887 DEBUG:boto:auth_path=/aktlji-71/ 2024-04-20T00:46:01.887 DEBUG:boto:Method: HEAD 2024-04-20T00:46:01.887 DEBUG:boto:Path: /aktlji-71/ 2024-04-20T00:46:01.887 DEBUG:boto:Data: 2024-04-20T00:46:01.887 DEBUG:boto:Headers: {} 2024-04-20T00:46:01.887 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.887 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.887 DEBUG:boto:Params: {} 2024-04-20T00:46:01.887 DEBUG:boto:Token: None 2024-04-20T00:46:01.888 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/ 2024-04-20T00:46:01.888 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T00:46:01.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T00:46:01.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098bd757019deaa96-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.890 DEBUG:boto:path=/aktlji-71/g 2024-04-20T00:46:01.890 DEBUG:boto:auth_path=/aktlji-71/g 2024-04-20T00:46:01.890 DEBUG:boto:Method: PUT 2024-04-20T00:46:01.890 DEBUG:boto:Path: /aktlji-71/g 2024-04-20T00:46:01.890 DEBUG:boto:Data: 2024-04-20T00:46:01.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:46:01.890 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.890 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.890 DEBUG:boto:Params: {} 2024-04-20T00:46:01.891 DEBUG:boto:Token: None 2024-04-20T00:46:01.891 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/g 2024-04-20T00:46:01.891 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rAq85E+DxbpGO8CbzcSboFk/JKY= 2024-04-20T00:46:01.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rAq85E+DxbpGO8CbzcSboFk/JKY='} 2024-04-20T00:46:01.897 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000980174ec2f433953-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.897 DEBUG:boto:path=/aktlji-71/ 2024-04-20T00:46:01.897 DEBUG:boto:auth_path=/aktlji-71/ 2024-04-20T00:46:01.897 DEBUG:boto:Method: HEAD 2024-04-20T00:46:01.897 DEBUG:boto:Path: /aktlji-71/ 2024-04-20T00:46:01.898 DEBUG:boto:Data: 2024-04-20T00:46:01.898 DEBUG:boto:Headers: {} 2024-04-20T00:46:01.898 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.898 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.898 DEBUG:boto:Params: {} 2024-04-20T00:46:01.898 DEBUG:boto:Token: None 2024-04-20T00:46:01.898 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/ 2024-04-20T00:46:01.898 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T00:46:01.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T00:46:01.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bea185cfc2458d1-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.900 DEBUG:boto:path=/aktlji-71/h 2024-04-20T00:46:01.901 DEBUG:boto:auth_path=/aktlji-71/h 2024-04-20T00:46:01.901 DEBUG:boto:Method: PUT 2024-04-20T00:46:01.901 DEBUG:boto:Path: /aktlji-71/h 2024-04-20T00:46:01.901 DEBUG:boto:Data: 2024-04-20T00:46:01.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:46:01.901 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:46:01.901 DEBUG:boto:Port: 8000 2024-04-20T00:46:01.901 DEBUG:boto:Params: {} 2024-04-20T00:46:01.901 DEBUG:boto:Token: None 2024-04-20T00:46:01.901 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:46:01 GMT /aktlji-71/h 2024-04-20T00:46:01.901 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6elNbVD7uedbOGiSIPunpMqtDFo= 2024-04-20T00:46:01.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6elNbVD7uedbOGiSIPunpMqtDFo='} 2024-04-20T00:46:01.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ce0bf043ab89f33-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:46:01.906 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-20T00:46:01.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-20T00:46:01.909 INFO:tasks.rgw.c2.client.0:Started 2024-04-20T00:46:01.909 DEBUG:teuthology.orchestra.run.smithi031:> curl http://smithi031.front.sepia.ceph.com:8000 2024-04-20T00:46:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-20T00:46:01.936 INFO:teuthology.orchestra.run.smithi031.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T00:46:01.937 INFO:teuthology.orchestra.run.smithi031.stderr: Dload Upload Total Spent Left Speed 2024-04-20T00:46:01.937 INFO:teuthology.orchestra.run.smithi031.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi031.front.sepia.ceph.com port 8000: Connection refused 2024-04-20T00:46:02.938 DEBUG:teuthology.orchestra.run.smithi031:> curl http://smithi031.front.sepia.ceph.com:8000 2024-04-20T00:46:02.968 INFO:teuthology.orchestra.run.smithi031.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T00:46:02.968 INFO:teuthology.orchestra.run.smithi031.stderr: Dload Upload Total Spent Left Speed 2024-04-20T00:46:02.968 INFO:teuthology.orchestra.run.smithi031.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-20T00:46:02.969 INFO:teuthology.orchestra.run.smithi031.stdout:anonymous 2024-04-20T00:46:02.969 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-20T00:46:02.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-20T00:46:02.971 INFO:tasks.rgw.c1.client.1:Started 2024-04-20T00:46:02.971 DEBUG:teuthology.orchestra.run.smithi019:> curl http://smithi019.front.sepia.ceph.com:8001 2024-04-20T00:46:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-20T00:46:03.005 INFO:teuthology.orchestra.run.smithi019.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T00:46:03.005 INFO:teuthology.orchestra.run.smithi019.stderr: Dload Upload Total Spent Left Speed 2024-04-20T00:46:03.005 INFO:teuthology.orchestra.run.smithi019.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi019.front.sepia.ceph.com port 8001: Connection refused 2024-04-20T00:46:04.006 DEBUG:teuthology.orchestra.run.smithi019:> curl http://smithi019.front.sepia.ceph.com:8001 2024-04-20T00:46:04.035 INFO:teuthology.orchestra.run.smithi019.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T00:46:04.035 INFO:teuthology.orchestra.run.smithi019.stderr: Dload Upload Total Spent Left Speed 2024-04-20T00:46:04.036 INFO:teuthology.orchestra.run.smithi019.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-20T00:46:04.038 INFO:teuthology.orchestra.run.smithi019.stdout:anonymous 2024-04-20T00:46:04.038 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-71 2024-04-20T00:46:04.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:46:04.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:46:04.039 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T00:46:04.106 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:46:04.106 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:46:04.149 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-71 2024-04-20T00:46:04.175 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-71 2024-04-20T00:46:04.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:46:04.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:46:04.175 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:46:04.245 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:46:04.245 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:46:04.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:46:04.285+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:46:05.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-20T00:46:09.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:46:09.286+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:46:14.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:46:14.287+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:46:19.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:46:19.287+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:46:24.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:46:24.288+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:46:29.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:46:29.289+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:46:34.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:46:34.289+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:46:39.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:46:39.290+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:46:44.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:46:44.291+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:46:49.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:46:49.291+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:46:54.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:46:54.292+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:46:59.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:46:59.292+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:47:04.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:47:04.293+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:47:09.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:47:09.294+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:47:14.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:47:14.293+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:47:19.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:47:19.294+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:47:24.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:47:24.294+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:47:29.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:47:29.295+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:47:34.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:47:34.299+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:47:39.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:47:39.299+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:47:44.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:47:44.300+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:47:49.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:47:49.300+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:47:54.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:47:54.301+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:47:59.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:47:59.302+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:48:04.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:04.303+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T00:48:09.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:09.304+0000 7f162f19a780 1 bucket sync caught up with source: 2024-04-20T00:48:09.306 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [00000000003.99.6, 00000000002.51.6, 00000000002.97.6] 2024-04-20T00:48:09.306 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [00000000003.99.6, 00000000002.51.6, 00000000002.97.6] 2024-04-20T00:48:09.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:09.304+0000 7f162f19a780 0 bucket checkpoint complete 2024-04-20T00:48:09.324 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-71 2024-04-20T00:48:09.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:48:09.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:48:09.325 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-71 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:48:09.392 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:48:09.392 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:48:09.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:09.433+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:48:14.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:14.434+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:48:19.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:19.435+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:48:24.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:24.436+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:48:29.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:29.436+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:48:34.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:34.437+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:48:39.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:39.437+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:48:44.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:44.438+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:48:49.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:49.439+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:48:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:54.439+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:48:59.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:48:59.439+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:49:04.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:49:04.439+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:49:09.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:49:09.440+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:49:14.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:49:14.441+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:49:19.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:49:19.441+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:49:24.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:49:24.442+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:49:29.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:49:29.443+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:49:34.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:49:34.443+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:49:39.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:49:39.444+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:49:44.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:49:44.444+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:49:49.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:49:49.445+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:49:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:49:54.445+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:49:59.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:49:59.446+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:50:04.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:50:04.446+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:50:09.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:50:09.448+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:50:14.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:50:14.449+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:50:19.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:50:19.449+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:50:24.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:50:24.450+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:50:29.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:50:29.451+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:50:34.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:50:34.451+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:50:39.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:50:39.452+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:50:44.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:50:44.453+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:50:49.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:50:49.452+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:50:54.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:50:54.453+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:50:59.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:50:59.453+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:51:04.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:51:04.454+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:51:09.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:51:09.455+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:51:14.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:51:14.456+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:51:19.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:51:19.456+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:51:24.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:51:24.457+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:51:29.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:51:29.457+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:51:34.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:51:34.458+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:51:39.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:51:39.458+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:51:44.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:51:44.459+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:51:49.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:51:49.460+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:51:54.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:51:54.460+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:51:59.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:51:59.462+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:52:04.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:52:04.463+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:52:09.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:52:09.464+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:52:14.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:52:14.464+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:52:19.465 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:52:19.465+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:52:24.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:52:24.465+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:52:29.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:52:29.466+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:52:34.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:52:34.466+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:52:39.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:52:39.466+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:52:44.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:52:44.467+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:52:49.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:52:49.468+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:52:54.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:52:54.468+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:52:59.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:52:59.469+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T00:53:04.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:04.470+0000 7fed53e78780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T00:53:04.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:04.470+0000 7fed53e78780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:53:04.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:04.470+0000 7fed53e78780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:53:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T00:53:04.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... FAIL 2024-04-20T00:53:04.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:04.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:04.493 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:53:04.555 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:53:04.555 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:53:04.592 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:53:04.606 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:53:04.607 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:04.607 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:53:04.607 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:53:04.607 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:04.607 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:04.607 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:53:04.607 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:53:04.607 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:04.607 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:04.607 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713573961.785127_956.1", 2024-04-20T00:53:04.608 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:46:01.785127Z" 2024-04-20T00:53:04.608 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:04.608 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:04.608 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:53:04.608 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:53:04.608 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:53:04.608 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:53:04.609 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573961.785127_956.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:53:04.609 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:53:04.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:04.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:04.610 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:53:04.816 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:53:04.816 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:53:04.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:53:04.872+0000 7f38ebbd3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:53:04.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:53:04.874+0000 7f38ebbd3780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:53:04.894 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:53:04.895 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:53:04.895 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:53:04.895 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:53:04.895 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:53:04.895 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:53:04.895 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:53:04.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:53:04.895 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:53:04.895 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:53:04.895 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:53:04.896 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:53:04.896 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:53:04.896 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:53:04.896 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:53:04.896 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:53:04.896 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:53:04.896 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:53:04.896 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:53:04.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:53:04.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:53:04.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:53:04.897 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:53:04.897 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:53:04.897 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:53:04.897 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:53:04.897 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:53:04.897 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:53:04.897 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:53:04.897 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:53:04.897 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:53:04.897 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:53:04.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:53:04.898 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:53:04.898 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:53:04.898 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:53:04.898 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:53:04.898 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713573961.785127_956.1", 2024-04-20T00:53:04.898 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:53:04.898 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:53:04.898 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:53:04.898 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:46:01.785127Z", 2024-04-20T00:53:04.898 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:53:04.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:53:04.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:53:04.899 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:53:04.899 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:53:04.899 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:53:04.899 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:53:04.899 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:53:04.899 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:53:04.899 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:53:04.899 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:53:04.899 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:53:04.899 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:53:04.900 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:53:04.900 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:53:04.900 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:53:04.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:53:04.900 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:53:04.900 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:53:04.900 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:53:04.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:53:04.900 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:53:04Z" 2024-04-20T00:53:04.900 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:53:04.901 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573961.785127_956.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:46:01.785127Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:53:04Z" } 2024-04-20T00:53:04.901 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573961.785127_956.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:46:01.785127Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:53:04.901 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573961.785127_956.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:53:04.902 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573961.785127_956.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:53:04.902 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:53:04.902 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:53:04.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:04.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:04.902 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:53:04.962 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:53:04.962 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:53:04.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:04.997+0000 7f23348b6780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:53:05.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:04.999+0000 7f23348b6780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:53:05.015 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:53:05.015 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:53:05.015 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:53:05.015 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:53:05.015 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:53:05.015 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:53:05.015 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:53:05.015 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.016 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:53:05.016 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:05.016 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:53:05.016 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:53:05.016 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:53:05.016 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:53:05.016 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:53:05.016 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:53:05.016 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:53:05.016 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:53:05.016 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:53:05.016 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:53:05.017 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.017 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:05.017 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:53:05.017 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:53:05.017 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:53:05.017 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:53:05.017 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:53:05.017 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:53:05.017 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:53:05.017 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:53:05.017 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:53:05.017 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:53:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:53:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:53:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:53:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713573961.785127_956.1", 2024-04-20T00:53:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:53:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:53:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:53:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:46:01.785127Z", 2024-04-20T00:53:05.018 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:53:05.019 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:53:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:53:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:53:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:53:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:53:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:53:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.020 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:53:05Z" 2024-04-20T00:53:05.020 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:53:05.021 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713573961.785127_956.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:46:01.785127Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:53:05Z" } 2024-04-20T00:53:05.021 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573961.785127_956.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:46:01.785127Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:53:05.021 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573961.785127_956.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:53:05.021 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573961.785127_956.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:53:05.022 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:53:05.022 DEBUG:boto:path=/ 2024-04-20T00:53:05.022 DEBUG:boto:auth_path=/ 2024-04-20T00:53:05.022 DEBUG:boto:Method: GET 2024-04-20T00:53:05.022 DEBUG:boto:Path: / 2024-04-20T00:53:05.022 DEBUG:boto:Data: 2024-04-20T00:53:05.022 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.023 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.023 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.023 DEBUG:boto:Params: {} 2024-04-20T00:53:05.023 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:53:05.023 DEBUG:boto:Token: None 2024-04-20T00:53:05.023 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 00:53:05 GMT / 2024-04-20T00:53:05.024 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:316/E35kVCjMNrbHZoHKxSq6Wbk= 2024-04-20T00:53:05.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:316/E35kVCjMNrbHZoHKxSq6Wbk=', 'Content-Length': '0'} 2024-04-20T00:53:05.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072b12f8d528ee54a-00662311f1-15b6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.032 DEBUG:boto:path=/ 2024-04-20T00:53:05.032 DEBUG:boto:auth_path=/ 2024-04-20T00:53:05.032 DEBUG:boto:Method: GET 2024-04-20T00:53:05.032 DEBUG:boto:Path: / 2024-04-20T00:53:05.032 DEBUG:boto:Data: 2024-04-20T00:53:05.032 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.032 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T00:53:05.032 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.032 DEBUG:boto:Params: {} 2024-04-20T00:53:05.033 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:53:05.033 DEBUG:boto:Token: None 2024-04-20T00:53:05.033 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 00:53:05 GMT / 2024-04-20T00:53:05.033 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:316/E35kVCjMNrbHZoHKxSq6Wbk= 2024-04-20T00:53:05.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:316/E35kVCjMNrbHZoHKxSq6Wbk=', 'Content-Length': '0'} 2024-04-20T00:53:05.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb30f51bc9974d5b-00662311f1-1fb0-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.042 DEBUG:boto:path=/ 2024-04-20T00:53:05.042 DEBUG:boto:auth_path=/ 2024-04-20T00:53:05.042 DEBUG:boto:Method: GET 2024-04-20T00:53:05.042 DEBUG:boto:Path: / 2024-04-20T00:53:05.043 DEBUG:boto:Data: 2024-04-20T00:53:05.043 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.043 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T00:53:05.043 DEBUG:boto:Port: 8001 2024-04-20T00:53:05.043 DEBUG:boto:Params: {} 2024-04-20T00:53:05.043 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T00:53:05.043 DEBUG:boto:Token: None 2024-04-20T00:53:05.043 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 00:53:05 GMT / 2024-04-20T00:53:05.043 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:316/E35kVCjMNrbHZoHKxSq6Wbk= 2024-04-20T00:53:05.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:316/E35kVCjMNrbHZoHKxSq6Wbk=', 'Content-Length': '0'} 2024-04-20T00:53:05.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099b30daa8dc29bfb-00662311f1-16f1-test-zone3'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2024-04-20T00:53:05.054 DEBUG:boto:path=/aktlji-72/ 2024-04-20T00:53:05.055 DEBUG:boto:auth_path=/aktlji-72/ 2024-04-20T00:53:05.055 DEBUG:boto:Method: PUT 2024-04-20T00:53:05.055 DEBUG:boto:Path: /aktlji-72/ 2024-04-20T00:53:05.055 DEBUG:boto:Data: 2024-04-20T00:53:05.055 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.055 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.055 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.055 DEBUG:boto:Params: {} 2024-04-20T00:53:05.055 DEBUG:boto:Token: None 2024-04-20T00:53:05.055 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/ 2024-04-20T00:53:05.056 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UE9wsSVQjJMIjxzvVehNz2XM+ds= 2024-04-20T00:53:05.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UE9wsSVQjJMIjxzvVehNz2XM+ds=', 'Content-Length': '0'} 2024-04-20T00:53:05.074 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bd3e8b13b5423f16-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.074 DEBUG:rgw_multi.tests:created bucket=aktlji-72 2024-04-20T00:53:05.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:05.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:05.075 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:53:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:53:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:53:05.172 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:53:05.186 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:53:05.187 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:05.187 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:53:05.187 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:53:05.187 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.187 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:05.187 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:53:05.187 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:53:05.187 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.187 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:05.187 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713574385.072559_1017.1", 2024-04-20T00:53:05.187 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:53:05.072559Z" 2024-04-20T00:53:05.188 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.188 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:05.188 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:53:05.188 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:53:05.188 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:53:05.188 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:53:05.189 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574385.072559_1017.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:53:05.189 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:53:05.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:05.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:05.190 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:53:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:53:05.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:53:05.307+0000 7f580ca83780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:53:05.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:53:05.308+0000 7f580ca83780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:53:05.337 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:53:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:53:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:53:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:53:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:53:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:53:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:53:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:53:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:53:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:53:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:53:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:53:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:53:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:53:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:53:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:53:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:53:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:53:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:53:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:53:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:53:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:53:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:53:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:53:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:53:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713574385.072559_1017.1", 2024-04-20T00:53:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:53:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:53:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:53:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:53:05.072559Z", 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:53:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:53:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:53:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:53:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:53:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:53:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:53:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:53:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:53:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:53:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:53:05Z" 2024-04-20T00:53:05.342 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:53:05.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713574385.072559_1017.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:53:05.072559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:53:05Z" } 2024-04-20T00:53:05.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713574385.072559_1017.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:53:05.072559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:53:05.343 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574385.072559_1017.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:53:05.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574385.072559_1017.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:53:05.344 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:53:05.344 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:53:05.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:05.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:05.344 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:53:05.403 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:53:05.404 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:53:05.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:05.438+0000 7f0267f51780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:53:05.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:05.439+0000 7f0267f51780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:53:05.455 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:53:05.455 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:53:05.455 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:53:05.455 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:53:05.455 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:53:05.456 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:53:05.456 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:53:05.456 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.456 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:53:05.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:05.456 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:53:05.456 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:53:05.456 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:53:05.456 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:53:05.456 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:53:05.456 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:53:05.456 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:53:05.457 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:53:05.458 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:53:05.458 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:53:05.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.458 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:05.458 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:53:05.458 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:53:05.458 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:53:05.458 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713574385.072559_1017.1", 2024-04-20T00:53:05.458 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:53:05.458 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:53:05.458 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:53:05.458 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:53:05.072559Z", 2024-04-20T00:53:05.459 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:53:05.459 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:53:05.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.459 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:53:05.459 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:53:05.459 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:53:05.459 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:53:05.459 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:53:05.459 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:53:05.459 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:53:05.459 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:53:05.459 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:53:05.460 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:53:05.460 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:53:05.460 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:53:05.460 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:53:05.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.460 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:53:05.460 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:53:05.460 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:53:05.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:53:05.460 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:53:05Z" 2024-04-20T00:53:05.460 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:53:05.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713574385.072559_1017.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:53:05.072559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:53:05Z" } 2024-04-20T00:53:05.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713574385.072559_1017.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:53:05.072559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:53:05.461 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574385.072559_1017.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:53:05.461 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574385.072559_1017.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:53:05.462 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:53:05.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:05.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:53:05.462 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:53:05.519 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:53:05.519 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:53:05.568 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:53:05.568 INFO:teuthology.orchestra.run.smithi019.stdout: "objs": [ 2024-04-20T00:53:05.568 INFO:teuthology.orchestra.run.smithi019.stdout: "obj00000000000000000001", 2024-04-20T00:53:05.568 INFO:teuthology.orchestra.run.smithi019.stdout: "obj00000000000000000005", 2024-04-20T00:53:05.568 INFO:teuthology.orchestra.run.smithi019.stdout: "obj00000000000000000012", 2024-04-20T00:53:05.568 INFO:teuthology.orchestra.run.smithi019.stdout: "obj00000000000000000004", 2024-04-20T00:53:05.568 INFO:teuthology.orchestra.run.smithi019.stdout: "obj00000000000000000011", 2024-04-20T00:53:05.568 INFO:teuthology.orchestra.run.smithi019.stdout: "obj00000000000000000000", 2024-04-20T00:53:05.568 INFO:teuthology.orchestra.run.smithi019.stdout: "obj00000000000000000010", 2024-04-20T00:53:05.568 INFO:teuthology.orchestra.run.smithi019.stdout: "obj00000000000000000003", 2024-04-20T00:53:05.569 INFO:teuthology.orchestra.run.smithi019.stdout: "obj00000000000000000013", 2024-04-20T00:53:05.569 INFO:teuthology.orchestra.run.smithi019.stdout: "obj00000000000000000002", 2024-04-20T00:53:05.569 INFO:teuthology.orchestra.run.smithi019.stdout: "obj00000000000000000006" 2024-04-20T00:53:05.569 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:53:05.569 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:53:05.570 DEBUG:boto:path=/aktlji-72/ 2024-04-20T00:53:05.570 DEBUG:boto:auth_path=/aktlji-72/ 2024-04-20T00:53:05.570 DEBUG:boto:Method: HEAD 2024-04-20T00:53:05.570 DEBUG:boto:Path: /aktlji-72/ 2024-04-20T00:53:05.570 DEBUG:boto:Data: 2024-04-20T00:53:05.570 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.571 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.571 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.571 DEBUG:boto:Params: {} 2024-04-20T00:53:05.571 DEBUG:boto:Token: None 2024-04-20T00:53:05.571 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/ 2024-04-20T00:53:05.571 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T00:53:05.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T00:53:05.574 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094ecf7d64f54cbce-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.574 DEBUG:boto:path=/aktlji-72/obj00000000000000000002 2024-04-20T00:53:05.574 DEBUG:boto:auth_path=/aktlji-72/obj00000000000000000002 2024-04-20T00:53:05.574 DEBUG:boto:Method: PUT 2024-04-20T00:53:05.574 DEBUG:boto:Path: /aktlji-72/obj00000000000000000002 2024-04-20T00:53:05.574 DEBUG:boto:Data: 2024-04-20T00:53:05.575 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:53:05.575 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.575 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.575 DEBUG:boto:Params: {} 2024-04-20T00:53:05.575 DEBUG:boto:Token: None 2024-04-20T00:53:05.575 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/obj00000000000000000002 2024-04-20T00:53:05.575 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dH6Y8sH8w6WtD9T4iVZrSt3oddA= 2024-04-20T00:53:05.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dH6Y8sH8w6WtD9T4iVZrSt3oddA='} 2024-04-20T00:53:05.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0277dbba0c40c93-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.583 DEBUG:boto:path=/aktlji-72/ 2024-04-20T00:53:05.583 DEBUG:boto:auth_path=/aktlji-72/ 2024-04-20T00:53:05.583 DEBUG:boto:Method: HEAD 2024-04-20T00:53:05.583 DEBUG:boto:Path: /aktlji-72/ 2024-04-20T00:53:05.583 DEBUG:boto:Data: 2024-04-20T00:53:05.583 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.583 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.583 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.583 DEBUG:boto:Params: {} 2024-04-20T00:53:05.583 DEBUG:boto:Token: None 2024-04-20T00:53:05.584 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/ 2024-04-20T00:53:05.584 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T00:53:05.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T00:53:05.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a2273292ee5f024-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.587 DEBUG:boto:path=/aktlji-72/obj00000000000000000006 2024-04-20T00:53:05.587 DEBUG:boto:auth_path=/aktlji-72/obj00000000000000000006 2024-04-20T00:53:05.587 DEBUG:boto:Method: PUT 2024-04-20T00:53:05.587 DEBUG:boto:Path: /aktlji-72/obj00000000000000000006 2024-04-20T00:53:05.587 DEBUG:boto:Data: 2024-04-20T00:53:05.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:53:05.587 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.587 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.587 DEBUG:boto:Params: {} 2024-04-20T00:53:05.588 DEBUG:boto:Token: None 2024-04-20T00:53:05.588 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/obj00000000000000000006 2024-04-20T00:53:05.588 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:80gAY/d4TySH/6kHDtojiYeIvXg= 2024-04-20T00:53:05.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:80gAY/d4TySH/6kHDtojiYeIvXg='} 2024-04-20T00:53:05.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4e23ad06d816eae-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.630 DEBUG:boto:path=/aktlji-72/ 2024-04-20T00:53:05.631 DEBUG:boto:auth_path=/aktlji-72/ 2024-04-20T00:53:05.631 DEBUG:boto:Method: HEAD 2024-04-20T00:53:05.631 DEBUG:boto:Path: /aktlji-72/ 2024-04-20T00:53:05.631 DEBUG:boto:Data: 2024-04-20T00:53:05.631 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.631 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.631 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.631 DEBUG:boto:Params: {} 2024-04-20T00:53:05.631 DEBUG:boto:Token: None 2024-04-20T00:53:05.631 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/ 2024-04-20T00:53:05.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T00:53:05.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T00:53:05.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afcd7bb477b5e131-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.634 DEBUG:boto:path=/aktlji-72/obj00000000000000000011 2024-04-20T00:53:05.635 DEBUG:boto:auth_path=/aktlji-72/obj00000000000000000011 2024-04-20T00:53:05.635 DEBUG:boto:Method: PUT 2024-04-20T00:53:05.635 DEBUG:boto:Path: /aktlji-72/obj00000000000000000011 2024-04-20T00:53:05.635 DEBUG:boto:Data: 2024-04-20T00:53:05.635 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:53:05.635 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.635 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.635 DEBUG:boto:Params: {} 2024-04-20T00:53:05.635 DEBUG:boto:Token: None 2024-04-20T00:53:05.636 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/obj00000000000000000011 2024-04-20T00:53:05.636 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OLu+qWqckjHhfExSIVZveY3cWu4= 2024-04-20T00:53:05.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OLu+qWqckjHhfExSIVZveY3cWu4='} 2024-04-20T00:53:05.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019a59cd629217576-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.642 DEBUG:boto:path=/aktlji-72/ 2024-04-20T00:53:05.642 DEBUG:boto:auth_path=/aktlji-72/ 2024-04-20T00:53:05.642 DEBUG:boto:Method: HEAD 2024-04-20T00:53:05.642 DEBUG:boto:Path: /aktlji-72/ 2024-04-20T00:53:05.642 DEBUG:boto:Data: 2024-04-20T00:53:05.642 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.642 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.642 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.642 DEBUG:boto:Params: {} 2024-04-20T00:53:05.643 DEBUG:boto:Token: None 2024-04-20T00:53:05.643 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/ 2024-04-20T00:53:05.643 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T00:53:05.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T00:53:05.645 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f2a2cd577238e03-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.646 DEBUG:boto:path=/aktlji-72/obj00000000000000000000 2024-04-20T00:53:05.646 DEBUG:boto:auth_path=/aktlji-72/obj00000000000000000000 2024-04-20T00:53:05.646 DEBUG:boto:Method: PUT 2024-04-20T00:53:05.646 DEBUG:boto:Path: /aktlji-72/obj00000000000000000000 2024-04-20T00:53:05.646 DEBUG:boto:Data: 2024-04-20T00:53:05.646 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:53:05.646 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.646 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.646 DEBUG:boto:Params: {} 2024-04-20T00:53:05.646 DEBUG:boto:Token: None 2024-04-20T00:53:05.647 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/obj00000000000000000000 2024-04-20T00:53:05.647 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:63a60/G+QTrdnveC0q0lg1EDk+Y= 2024-04-20T00:53:05.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:63a60/G+QTrdnveC0q0lg1EDk+Y='} 2024-04-20T00:53:05.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000613885d886d6260c-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.691 DEBUG:boto:path=/aktlji-72/ 2024-04-20T00:53:05.691 DEBUG:boto:auth_path=/aktlji-72/ 2024-04-20T00:53:05.691 DEBUG:boto:Method: HEAD 2024-04-20T00:53:05.691 DEBUG:boto:Path: /aktlji-72/ 2024-04-20T00:53:05.691 DEBUG:boto:Data: 2024-04-20T00:53:05.691 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.691 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.691 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.691 DEBUG:boto:Params: {} 2024-04-20T00:53:05.691 DEBUG:boto:Token: None 2024-04-20T00:53:05.692 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/ 2024-04-20T00:53:05.692 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T00:53:05.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T00:53:05.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000558c2bfff50a1658-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.695 DEBUG:boto:path=/aktlji-72/obj00000000000000000012 2024-04-20T00:53:05.695 DEBUG:boto:auth_path=/aktlji-72/obj00000000000000000012 2024-04-20T00:53:05.695 DEBUG:boto:Method: PUT 2024-04-20T00:53:05.695 DEBUG:boto:Path: /aktlji-72/obj00000000000000000012 2024-04-20T00:53:05.695 DEBUG:boto:Data: 2024-04-20T00:53:05.695 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:53:05.695 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.695 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.695 DEBUG:boto:Params: {} 2024-04-20T00:53:05.695 DEBUG:boto:Token: None 2024-04-20T00:53:05.696 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/obj00000000000000000012 2024-04-20T00:53:05.696 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6Ni3nCSz5x7HyYp5l2SQ+6+dCQQ= 2024-04-20T00:53:05.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6Ni3nCSz5x7HyYp5l2SQ+6+dCQQ='} 2024-04-20T00:53:05.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d19205184cd46a09-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.703 DEBUG:boto:path=/aktlji-72/ 2024-04-20T00:53:05.703 DEBUG:boto:auth_path=/aktlji-72/ 2024-04-20T00:53:05.703 DEBUG:boto:Method: HEAD 2024-04-20T00:53:05.703 DEBUG:boto:Path: /aktlji-72/ 2024-04-20T00:53:05.703 DEBUG:boto:Data: 2024-04-20T00:53:05.703 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.703 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.703 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.704 DEBUG:boto:Params: {} 2024-04-20T00:53:05.704 DEBUG:boto:Token: None 2024-04-20T00:53:05.704 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/ 2024-04-20T00:53:05.704 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T00:53:05.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T00:53:05.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000733a79903b20643-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.707 DEBUG:boto:path=/aktlji-72/obj00000000000000000010 2024-04-20T00:53:05.707 DEBUG:boto:auth_path=/aktlji-72/obj00000000000000000010 2024-04-20T00:53:05.707 DEBUG:boto:Method: PUT 2024-04-20T00:53:05.707 DEBUG:boto:Path: /aktlji-72/obj00000000000000000010 2024-04-20T00:53:05.707 DEBUG:boto:Data: 2024-04-20T00:53:05.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:53:05.708 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.708 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.708 DEBUG:boto:Params: {} 2024-04-20T00:53:05.708 DEBUG:boto:Token: None 2024-04-20T00:53:05.708 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/obj00000000000000000010 2024-04-20T00:53:05.708 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:d8+ZPb+QbkVQbpGGqrBnD3VhZuA= 2024-04-20T00:53:05.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d8+ZPb+QbkVQbpGGqrBnD3VhZuA='} 2024-04-20T00:53:05.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007738205c2eca72c5-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.751 DEBUG:boto:path=/aktlji-72/ 2024-04-20T00:53:05.751 DEBUG:boto:auth_path=/aktlji-72/ 2024-04-20T00:53:05.751 DEBUG:boto:Method: HEAD 2024-04-20T00:53:05.751 DEBUG:boto:Path: /aktlji-72/ 2024-04-20T00:53:05.751 DEBUG:boto:Data: 2024-04-20T00:53:05.751 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.751 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.751 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.751 DEBUG:boto:Params: {} 2024-04-20T00:53:05.751 DEBUG:boto:Token: None 2024-04-20T00:53:05.751 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/ 2024-04-20T00:53:05.752 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T00:53:05.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T00:53:05.754 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002125699db9dcc552-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.755 DEBUG:boto:path=/aktlji-72/obj00000000000000000001 2024-04-20T00:53:05.755 DEBUG:boto:auth_path=/aktlji-72/obj00000000000000000001 2024-04-20T00:53:05.755 DEBUG:boto:Method: PUT 2024-04-20T00:53:05.755 DEBUG:boto:Path: /aktlji-72/obj00000000000000000001 2024-04-20T00:53:05.755 DEBUG:boto:Data: 2024-04-20T00:53:05.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:53:05.755 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.755 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.755 DEBUG:boto:Params: {} 2024-04-20T00:53:05.755 DEBUG:boto:Token: None 2024-04-20T00:53:05.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/obj00000000000000000001 2024-04-20T00:53:05.756 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cNXt7wwxMa9t4aC63R0/YeTdO18= 2024-04-20T00:53:05.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cNXt7wwxMa9t4aC63R0/YeTdO18='} 2024-04-20T00:53:05.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af56a54b05a3e9b8-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.762 DEBUG:boto:path=/aktlji-72/ 2024-04-20T00:53:05.763 DEBUG:boto:auth_path=/aktlji-72/ 2024-04-20T00:53:05.763 DEBUG:boto:Method: HEAD 2024-04-20T00:53:05.763 DEBUG:boto:Path: /aktlji-72/ 2024-04-20T00:53:05.763 DEBUG:boto:Data: 2024-04-20T00:53:05.763 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.763 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.763 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.763 DEBUG:boto:Params: {} 2024-04-20T00:53:05.763 DEBUG:boto:Token: None 2024-04-20T00:53:05.763 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/ 2024-04-20T00:53:05.763 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T00:53:05.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T00:53:05.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f796983fa18d727-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.766 DEBUG:boto:path=/aktlji-72/obj00000000000000000004 2024-04-20T00:53:05.766 DEBUG:boto:auth_path=/aktlji-72/obj00000000000000000004 2024-04-20T00:53:05.767 DEBUG:boto:Method: PUT 2024-04-20T00:53:05.767 DEBUG:boto:Path: /aktlji-72/obj00000000000000000004 2024-04-20T00:53:05.767 DEBUG:boto:Data: 2024-04-20T00:53:05.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:53:05.767 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.767 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.767 DEBUG:boto:Params: {} 2024-04-20T00:53:05.767 DEBUG:boto:Token: None 2024-04-20T00:53:05.767 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/obj00000000000000000004 2024-04-20T00:53:05.767 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tZAR6oSHJwadb9/Oq+igxLm49LI= 2024-04-20T00:53:05.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tZAR6oSHJwadb9/Oq+igxLm49LI='} 2024-04-20T00:53:05.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ad50978b56daefca-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.811 DEBUG:boto:path=/aktlji-72/ 2024-04-20T00:53:05.811 DEBUG:boto:auth_path=/aktlji-72/ 2024-04-20T00:53:05.811 DEBUG:boto:Method: HEAD 2024-04-20T00:53:05.811 DEBUG:boto:Path: /aktlji-72/ 2024-04-20T00:53:05.811 DEBUG:boto:Data: 2024-04-20T00:53:05.811 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.811 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.811 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.811 DEBUG:boto:Params: {} 2024-04-20T00:53:05.811 DEBUG:boto:Token: None 2024-04-20T00:53:05.812 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/ 2024-04-20T00:53:05.812 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T00:53:05.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T00:53:05.814 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075ceafba51acfc7e-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.815 DEBUG:boto:path=/aktlji-72/obj00000000000000000013 2024-04-20T00:53:05.815 DEBUG:boto:auth_path=/aktlji-72/obj00000000000000000013 2024-04-20T00:53:05.815 DEBUG:boto:Method: PUT 2024-04-20T00:53:05.815 DEBUG:boto:Path: /aktlji-72/obj00000000000000000013 2024-04-20T00:53:05.815 DEBUG:boto:Data: 2024-04-20T00:53:05.815 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:53:05.815 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.815 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.815 DEBUG:boto:Params: {} 2024-04-20T00:53:05.815 DEBUG:boto:Token: None 2024-04-20T00:53:05.816 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/obj00000000000000000013 2024-04-20T00:53:05.816 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:s2wvGBF6qkJrWk43B88+XYd5hy0= 2024-04-20T00:53:05.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:s2wvGBF6qkJrWk43B88+XYd5hy0='} 2024-04-20T00:53:05.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f38f318e2dc3c902-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.822 DEBUG:boto:path=/aktlji-72/ 2024-04-20T00:53:05.823 DEBUG:boto:auth_path=/aktlji-72/ 2024-04-20T00:53:05.823 DEBUG:boto:Method: HEAD 2024-04-20T00:53:05.823 DEBUG:boto:Path: /aktlji-72/ 2024-04-20T00:53:05.823 DEBUG:boto:Data: 2024-04-20T00:53:05.823 DEBUG:boto:Headers: {} 2024-04-20T00:53:05.823 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.823 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.823 DEBUG:boto:Params: {} 2024-04-20T00:53:05.823 DEBUG:boto:Token: None 2024-04-20T00:53:05.823 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/ 2024-04-20T00:53:05.824 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T00:53:05.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T00:53:05.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ccbee340dd8cc66-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.827 DEBUG:boto:path=/aktlji-72/obj00000000000000000003 2024-04-20T00:53:05.827 DEBUG:boto:auth_path=/aktlji-72/obj00000000000000000003 2024-04-20T00:53:05.827 DEBUG:boto:Method: PUT 2024-04-20T00:53:05.827 DEBUG:boto:Path: /aktlji-72/obj00000000000000000003 2024-04-20T00:53:05.827 DEBUG:boto:Data: 2024-04-20T00:53:05.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:53:05.827 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:53:05.827 DEBUG:boto:Port: 8000 2024-04-20T00:53:05.827 DEBUG:boto:Params: {} 2024-04-20T00:53:05.827 DEBUG:boto:Token: None 2024-04-20T00:53:05.827 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:53:05 GMT /aktlji-72/obj00000000000000000003 2024-04-20T00:53:05.828 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xlFIQhx//WARG+nNsGXh5v6YTDs= 2024-04-20T00:53:05.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xlFIQhx//WARG+nNsGXh5v6YTDs='} 2024-04-20T00:53:05.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005feede02f6a91edd-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:53:05.871 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-72 2024-04-20T00:53:05.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:53:05.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:53:05.871 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T00:53:05.937 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:53:05.937 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:53:05.980 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-72 2024-04-20T00:53:06.002 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-72 2024-04-20T00:53:06.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:53:06.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:53:06.003 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:53:06.070 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:53:06.070 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:53:06.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:06.112+0000 7f242a0fd780 1 waiting to reach incremental sync.. 2024-04-20T00:53:11.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:11.112+0000 7f242a0fd780 1 waiting to reach incremental sync.. 2024-04-20T00:53:16.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:16.114+0000 7f242a0fd780 1 bucket sync caught up with source: 2024-04-20T00:53:16.116 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [00000000001.41.6, , 00000000001.41.6, 00000000001.84.6, 00000000001.21.6, 00000000001.39.6, 00000000001.65.6, 00000000001.45.6, 00000000001.67.6, 00000000001.59.6, 00000000001.37.6] 2024-04-20T00:53:16.116 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [00000000001.41.6, , 00000000001.41.6, 00000000001.84.6, 00000000001.21.6, 00000000001.39.6, 00000000001.65.6, 00000000001.45.6, 00000000001.67.6, 00000000001.59.6, 00000000001.37.6] 2024-04-20T00:53:16.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:16.114+0000 7f242a0fd780 0 bucket checkpoint complete 2024-04-20T00:53:16.133 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-72 2024-04-20T00:53:16.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:53:16.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:53:16.134 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:53:16.199 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:53:16.200 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:53:16.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:16.241+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:53:21.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:21.242+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:53:26.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:26.243+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:53:31.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:31.243+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:53:36.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:36.244+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:53:41.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:41.244+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:53:46.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:46.245+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:53:51.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:51.246+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:53:56.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:53:56.247+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:54:01.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:54:01.248+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:54:06.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:54:06.248+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:54:11.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:54:11.249+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:54:16.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:54:16.249+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:54:21.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:54:21.249+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:54:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:54:26.250+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:54:31.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:54:31.250+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:54:36.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:54:36.251+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:54:41.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:54:41.252+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:54:46.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:54:46.252+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:54:51.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:54:51.252+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:54:56.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:54:56.253+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:55:01.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:55:01.254+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:55:06.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:55:06.254+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:55:11.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:55:11.255+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:55:16.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:55:16.255+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:55:21.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:55:21.256+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:55:26.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:55:26.256+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:55:31.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:55:31.257+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:55:36.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:55:36.259+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:55:41.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:55:41.259+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:55:46.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:55:46.260+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:55:51.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:55:51.261+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:55:56.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:55:56.261+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:56:01.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:56:01.262+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:56:06.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:56:06.263+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:56:11.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:56:11.262+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:56:16.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:56:16.263+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:56:21.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:56:21.263+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:56:26.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:56:26.264+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:56:31.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:56:31.265+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:56:36.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:56:36.265+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:56:41.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:56:41.266+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:56:46.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:56:46.266+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:56:51.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:56:51.267+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:56:56.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:56:56.268+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:57:01.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:57:01.269+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:57:06.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:57:06.269+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:57:11.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:57:11.270+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:57:16.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:57:16.270+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:57:21.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:57:21.271+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:57:26.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:57:26.272+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:57:31.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:57:31.273+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:57:36.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:57:36.274+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:57:41.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:57:41.274+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:57:46.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:57:46.275+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:57:51.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:57:51.275+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:57:56.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:57:56.275+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:58:01.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:01.275+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:58:06.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:06.276+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T00:58:11.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:11.277+0000 7fa4f484a780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T00:58:11.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:11.277+0000 7fa4f484a780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:58:11.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:11.277+0000 7fa4f484a780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T00:58:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T00:58:11.299 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... FAIL 2024-04-20T00:58:11.300 DEBUG:boto:path=/aktlji-73/ 2024-04-20T00:58:11.300 DEBUG:boto:auth_path=/aktlji-73/ 2024-04-20T00:58:11.300 DEBUG:boto:Method: PUT 2024-04-20T00:58:11.300 DEBUG:boto:Path: /aktlji-73/ 2024-04-20T00:58:11.300 DEBUG:boto:Data: 2024-04-20T00:58:11.300 DEBUG:boto:Headers: {} 2024-04-20T00:58:11.300 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:58:11.300 DEBUG:boto:Port: 8000 2024-04-20T00:58:11.300 DEBUG:boto:Params: {} 2024-04-20T00:58:11.301 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T00:58:11.301 DEBUG:boto:Token: None 2024-04-20T00:58:11.301 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 00:58:11 GMT /aktlji-73/ 2024-04-20T00:58:11.302 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sFSXiTUhXSSuUu4NuPNnOv2HgqQ= 2024-04-20T00:58:11.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sFSXiTUhXSSuUu4NuPNnOv2HgqQ=', 'Content-Length': '0'} 2024-04-20T00:58:11.326 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000012f45b2f854bff52-0066231323-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:58:11.326 DEBUG:rgw_multi.tests:created bucket=aktlji-73 2024-04-20T00:58:11.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:58:11.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:58:11.327 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:58:11.384 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:58:11.384 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:58:11.420 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T00:58:11.435 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T00:58:11.435 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:58:11.435 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:58:11.435 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T00:58:11.435 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:58:11.435 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:58:11.436 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:58:11.436 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T00:58:11.436 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:58:11.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:58:11.436 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713574691.323977_1063.1", 2024-04-20T00:58:11.436 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:58:11.323977Z" 2024-04-20T00:58:11.436 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:58:11.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:58:11.436 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:58:11.436 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T00:58:11.436 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:58:11.436 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T00:58:11.437 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:58:11.437 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T00:58:11.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:58:11.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:58:11.438 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:58:11.557 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:58:11.557 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:58:11.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:58:11.610+0000 7fc20ab82780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:58:11.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T00:58:11.611+0000 7fc20ab82780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:58:11.632 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T00:58:11.632 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T00:58:11.632 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T00:58:11.632 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T00:58:11.632 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T00:58:11.632 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:58:11.633 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:58:11.634 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:58:11.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:58:11.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:58:11.634 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T00:58:11.634 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:58:11.634 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:58:11.634 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T00:58:11.634 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:58:11.634 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:58:11.634 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:58:11.634 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T00:58:11.634 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:58:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:58:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:58:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:58:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T00:58:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:58:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:58:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713574691.323977_1063.1", 2024-04-20T00:58:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:58:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:58:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:58:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:58:11.323977Z", 2024-04-20T00:58:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T00:58:11.636 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:58:11.637 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T00:58:11.637 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T00:58:11.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:58:11.637 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T00:58:11.637 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T00:58:11.637 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T00:58:11.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T00:58:11.637 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T00:58:11Z" 2024-04-20T00:58:11.637 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T00:58:11.638 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713574691.323977_1063.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:58:11.323977Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:58:11Z" } 2024-04-20T00:58:11.638 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713574691.323977_1063.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:58:11.323977Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:58:11.638 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:58:11.638 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:58:11.638 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T00:58:11.639 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T00:58:11.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:58:11.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T00:58:11.639 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T00:58:11.700 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:58:11.700 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:58:11.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:11.735+0000 7f34c3423780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:58:11.737 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:11.736+0000 7f34c3423780 20 RGW-SYNC:meta: read sync status 2024-04-20T00:58:11.751 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T00:58:11.751 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T00:58:11.752 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T00:58:11.752 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T00:58:11.752 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T00:58:11.752 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T00:58:11.752 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:58:11.752 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:58:11.752 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T00:58:11.752 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:58:11.752 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T00:58:11.752 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:58:11.752 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:58:11.752 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T00:58:11.753 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:58:11.754 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:58:11.754 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:58:11.754 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T00:58:11.754 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:58:11.754 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:58:11.754 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:58:11.754 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:58:11.754 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T00:58:11.754 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:58:11.754 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:58:11.754 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713574691.323977_1063.1", 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:58:11.323977Z", 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568988.397150_80.1", 2024-04-20T00:58:11.755 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T00:58:11.756 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T00:58:11.756 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T00:58:11.756 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T00:58:11.756 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T00:58:11.756 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:58:11.756 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T00:58:11.756 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T00:58:11.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:58:11.756 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T00:58:11.756 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T00:58:11.756 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T00:58:11.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T00:58:11.757 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T00:58:11Z" 2024-04-20T00:58:11.757 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T00:58:11.757 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713574691.323977_1063.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:58:11.323977Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713568988.397150_80.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-19T23:23:08.397150Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T00:58:11Z" } 2024-04-20T00:58:11.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713574691.323977_1063.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:58:11.323977Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T00:58:11.758 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:58:11.758 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T00:58:11.758 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T00:58:11.758 DEBUG:boto:path=/aktlji-73/ 2024-04-20T00:58:11.758 DEBUG:boto:auth_path=/aktlji-73/ 2024-04-20T00:58:11.758 DEBUG:boto:Method: HEAD 2024-04-20T00:58:11.758 DEBUG:boto:Path: /aktlji-73/ 2024-04-20T00:58:11.758 DEBUG:boto:Data: 2024-04-20T00:58:11.759 DEBUG:boto:Headers: {} 2024-04-20T00:58:11.759 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:58:11.759 DEBUG:boto:Port: 8000 2024-04-20T00:58:11.759 DEBUG:boto:Params: {} 2024-04-20T00:58:11.759 DEBUG:boto:Token: None 2024-04-20T00:58:11.759 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:58:11 GMT /aktlji-73/ 2024-04-20T00:58:11.759 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY= 2024-04-20T00:58:11.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY=', 'Content-Length': '0'} 2024-04-20T00:58:11.761 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f91c7c5995e2135-0066231323-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:58:11.762 DEBUG:boto:path=/aktlji-73/obj11 2024-04-20T00:58:11.762 DEBUG:boto:auth_path=/aktlji-73/obj11 2024-04-20T00:58:11.762 DEBUG:boto:Method: PUT 2024-04-20T00:58:11.762 DEBUG:boto:Path: /aktlji-73/obj11 2024-04-20T00:58:11.762 DEBUG:boto:Data: 2024-04-20T00:58:11.762 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:58:11.762 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:58:11.762 DEBUG:boto:Port: 8000 2024-04-20T00:58:11.763 DEBUG:boto:Params: {} 2024-04-20T00:58:11.763 DEBUG:boto:Token: None 2024-04-20T00:58:11.763 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:58:11 GMT /aktlji-73/obj11 2024-04-20T00:58:11.763 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:uAuX9VCnFp3QMTYx963x5kCUI3I= 2024-04-20T00:58:11.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uAuX9VCnFp3QMTYx963x5kCUI3I='} 2024-04-20T00:58:11.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca89e6169c9d663e-0066231323-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:58:11.770 DEBUG:boto:path=/aktlji-73/ 2024-04-20T00:58:11.770 DEBUG:boto:auth_path=/aktlji-73/ 2024-04-20T00:58:11.771 DEBUG:boto:Method: HEAD 2024-04-20T00:58:11.771 DEBUG:boto:Path: /aktlji-73/ 2024-04-20T00:58:11.771 DEBUG:boto:Data: 2024-04-20T00:58:11.771 DEBUG:boto:Headers: {} 2024-04-20T00:58:11.771 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:58:11.771 DEBUG:boto:Port: 8000 2024-04-20T00:58:11.771 DEBUG:boto:Params: {} 2024-04-20T00:58:11.771 DEBUG:boto:Token: None 2024-04-20T00:58:11.771 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:58:11 GMT /aktlji-73/ 2024-04-20T00:58:11.771 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY= 2024-04-20T00:58:11.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY=', 'Content-Length': '0'} 2024-04-20T00:58:11.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069ece72e58c953d6-0066231323-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:58:11.774 DEBUG:boto:path=/aktlji-73/obj22 2024-04-20T00:58:11.774 DEBUG:boto:auth_path=/aktlji-73/obj22 2024-04-20T00:58:11.774 DEBUG:boto:Method: PUT 2024-04-20T00:58:11.774 DEBUG:boto:Path: /aktlji-73/obj22 2024-04-20T00:58:11.774 DEBUG:boto:Data: 2024-04-20T00:58:11.774 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:58:11.774 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:58:11.774 DEBUG:boto:Port: 8000 2024-04-20T00:58:11.775 DEBUG:boto:Params: {} 2024-04-20T00:58:11.775 DEBUG:boto:Token: None 2024-04-20T00:58:11.775 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:58:11 GMT /aktlji-73/obj22 2024-04-20T00:58:11.775 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QUK90/CMs8q3alpCf9hT0aWhjSE= 2024-04-20T00:58:11.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QUK90/CMs8q3alpCf9hT0aWhjSE='} 2024-04-20T00:58:11.779 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000566a42085a82cf31-0066231323-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:58:11.779 DEBUG:boto:path=/aktlji-73/ 2024-04-20T00:58:11.779 DEBUG:boto:auth_path=/aktlji-73/ 2024-04-20T00:58:11.779 DEBUG:boto:Method: HEAD 2024-04-20T00:58:11.779 DEBUG:boto:Path: /aktlji-73/ 2024-04-20T00:58:11.779 DEBUG:boto:Data: 2024-04-20T00:58:11.779 DEBUG:boto:Headers: {} 2024-04-20T00:58:11.779 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:58:11.780 DEBUG:boto:Port: 8000 2024-04-20T00:58:11.780 DEBUG:boto:Params: {} 2024-04-20T00:58:11.780 DEBUG:boto:Token: None 2024-04-20T00:58:11.780 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:58:11 GMT /aktlji-73/ 2024-04-20T00:58:11.780 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY= 2024-04-20T00:58:11.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY=', 'Content-Length': '0'} 2024-04-20T00:58:11.782 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eacc4384e59c566c-0066231323-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:58:11.782 DEBUG:boto:path=/aktlji-73/obj33 2024-04-20T00:58:11.782 DEBUG:boto:auth_path=/aktlji-73/obj33 2024-04-20T00:58:11.783 DEBUG:boto:Method: PUT 2024-04-20T00:58:11.783 DEBUG:boto:Path: /aktlji-73/obj33 2024-04-20T00:58:11.783 DEBUG:boto:Data: 2024-04-20T00:58:11.783 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:58:11.783 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:58:11.783 DEBUG:boto:Port: 8000 2024-04-20T00:58:11.783 DEBUG:boto:Params: {} 2024-04-20T00:58:11.783 DEBUG:boto:Token: None 2024-04-20T00:58:11.783 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:58:11 GMT /aktlji-73/obj33 2024-04-20T00:58:11.783 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zApvcus8FnxB1IJnVBJg7QCUrUg= 2024-04-20T00:58:11.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zApvcus8FnxB1IJnVBJg7QCUrUg='} 2024-04-20T00:58:11.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000592ecdfabaf05da1-0066231323-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:58:11.826 DEBUG:boto:path=/aktlji-73/ 2024-04-20T00:58:11.827 DEBUG:boto:auth_path=/aktlji-73/ 2024-04-20T00:58:11.827 DEBUG:boto:Method: HEAD 2024-04-20T00:58:11.827 DEBUG:boto:Path: /aktlji-73/ 2024-04-20T00:58:11.827 DEBUG:boto:Data: 2024-04-20T00:58:11.827 DEBUG:boto:Headers: {} 2024-04-20T00:58:11.827 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:58:11.827 DEBUG:boto:Port: 8000 2024-04-20T00:58:11.827 DEBUG:boto:Params: {} 2024-04-20T00:58:11.827 DEBUG:boto:Token: None 2024-04-20T00:58:11.827 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:58:11 GMT /aktlji-73/ 2024-04-20T00:58:11.827 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY= 2024-04-20T00:58:11.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY=', 'Content-Length': '0'} 2024-04-20T00:58:11.830 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008161af0b62d18a6e-0066231323-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:58:11.830 DEBUG:boto:path=/aktlji-73/obj44 2024-04-20T00:58:11.830 DEBUG:boto:auth_path=/aktlji-73/obj44 2024-04-20T00:58:11.830 DEBUG:boto:Method: PUT 2024-04-20T00:58:11.830 DEBUG:boto:Path: /aktlji-73/obj44 2024-04-20T00:58:11.831 DEBUG:boto:Data: 2024-04-20T00:58:11.831 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:58:11.831 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:58:11.831 DEBUG:boto:Port: 8000 2024-04-20T00:58:11.831 DEBUG:boto:Params: {} 2024-04-20T00:58:11.831 DEBUG:boto:Token: None 2024-04-20T00:58:11.831 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:58:11 GMT /aktlji-73/obj44 2024-04-20T00:58:11.831 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:faRgBH7uYpjXRW5wVhZjkPYq9wE= 2024-04-20T00:58:11.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:faRgBH7uYpjXRW5wVhZjkPYq9wE='} 2024-04-20T00:58:11.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c5ded47a39488b0-0066231323-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:58:11.838 DEBUG:boto:path=/aktlji-73/ 2024-04-20T00:58:11.838 DEBUG:boto:auth_path=/aktlji-73/ 2024-04-20T00:58:11.838 DEBUG:boto:Method: HEAD 2024-04-20T00:58:11.838 DEBUG:boto:Path: /aktlji-73/ 2024-04-20T00:58:11.838 DEBUG:boto:Data: 2024-04-20T00:58:11.838 DEBUG:boto:Headers: {} 2024-04-20T00:58:11.838 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:58:11.838 DEBUG:boto:Port: 8000 2024-04-20T00:58:11.838 DEBUG:boto:Params: {} 2024-04-20T00:58:11.838 DEBUG:boto:Token: None 2024-04-20T00:58:11.838 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 00:58:11 GMT /aktlji-73/ 2024-04-20T00:58:11.839 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY= 2024-04-20T00:58:11.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY=', 'Content-Length': '0'} 2024-04-20T00:58:11.841 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000456ea47d3028d0f2-0066231323-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:58:11.841 DEBUG:boto:path=/aktlji-73/obj55 2024-04-20T00:58:11.841 DEBUG:boto:auth_path=/aktlji-73/obj55 2024-04-20T00:58:11.841 DEBUG:boto:Method: PUT 2024-04-20T00:58:11.842 DEBUG:boto:Path: /aktlji-73/obj55 2024-04-20T00:58:11.842 DEBUG:boto:Data: 2024-04-20T00:58:11.842 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T00:58:11.842 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T00:58:11.842 DEBUG:boto:Port: 8000 2024-04-20T00:58:11.842 DEBUG:boto:Params: {} 2024-04-20T00:58:11.842 DEBUG:boto:Token: None 2024-04-20T00:58:11.842 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 00:58:11 GMT /aktlji-73/obj55 2024-04-20T00:58:11.842 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3ms3zpQEpOL0dr2tFPoMu1AjFD0= 2024-04-20T00:58:11.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3ms3zpQEpOL0dr2tFPoMu1AjFD0='} 2024-04-20T00:58:11.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000766102c2dc765709-0066231323-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T00:58:11.847 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-73 2024-04-20T00:58:11.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:58:11.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:58:11.847 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T00:58:11.914 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:58:11.914 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:58:11.967 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-73 2024-04-20T00:58:11.989 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-73 2024-04-20T00:58:11.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:58:11.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:58:11.989 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:58:12.056 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:58:12.056 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:58:12.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:12.097+0000 7f869acb9780 1 waiting to reach incremental sync.. 2024-04-20T00:58:17.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:17.097+0000 7f869acb9780 1 waiting to reach incremental sync.. 2024-04-20T00:58:22.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:22.098+0000 7f869acb9780 1 waiting to reach incremental sync.. 2024-04-20T00:58:27.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:27.100+0000 7f869acb9780 1 bucket sync caught up with source: 2024-04-20T00:58:27.102 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.24.6, 00000000002.89.6, 00000000002.46.6] 2024-04-20T00:58:27.102 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.24.6, 00000000002.89.6, 00000000002.46.6] 2024-04-20T00:58:27.103 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:27.100+0000 7f869acb9780 0 bucket checkpoint complete 2024-04-20T00:58:27.119 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-73 2024-04-20T00:58:27.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:58:27.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T00:58:27.120 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-73 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T00:58:27.186 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T00:58:27.186 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T00:58:27.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:27.226+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:58:32.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:32.226+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:58:37.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:37.227+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:58:42.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:42.228+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:58:47.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:47.228+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:58:52.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:52.229+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:58:57.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:58:57.230+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:59:02.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:59:02.230+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:59:07.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:59:07.231+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:59:12.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:59:12.232+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:59:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:59:17.233+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:59:22.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:59:22.233+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:59:27.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:59:27.234+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:59:32.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:59:32.234+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:59:37.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:59:37.235+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:59:42.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:59:42.236+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:59:47.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:59:47.235+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:59:52.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:59:52.236+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T00:59:57.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T00:59:57.236+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:00:02.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:00:02.237+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:00:07.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:00:07.238+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:00:12.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:00:12.238+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:00:17.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:00:17.239+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:00:22.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:00:22.239+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:00:27.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:00:27.240+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:00:32.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:00:32.240+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:00:37.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:00:37.241+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:00:42.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:00:42.241+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:00:47.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:00:47.241+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:00:52.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:00:52.242+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:00:57.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:00:57.243+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:01:02.244 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:01:02.244+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:01:07.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:01:07.244+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:01:12.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:01:12.245+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:01:17.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:01:17.246+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:01:22.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:01:22.246+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:01:27.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:01:27.247+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:01:32.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:01:32.246+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:01:37.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:01:37.247+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:01:42.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:01:42.247+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:01:47.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:01:47.248+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:01:52.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:01:52.248+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:01:57.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:01:57.249+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:02:02.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:02:02.250+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:02:07.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:02:07.250+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:02:12.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:02:12.251+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:02:17.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:02:17.252+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:02:22.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:02:22.252+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:02:27.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:02:27.253+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:02:32.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:02:32.254+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:02:37.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:02:37.254+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:02:42.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:02:42.255+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:02:47.256 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:02:47.256+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:02:52.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:02:52.257+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:02:57.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:02:57.257+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:03:02.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:02.258+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:03:07.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:07.259+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:03:12.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:12.259+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:03:17.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:17.260+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:03:22.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:22.259+0000 7fca338b5780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:03:22.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:22.259+0000 7fca338b5780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:03:22.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:22.260+0000 7fca338b5780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:03:22.279 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T01:03:22.281 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... FAIL 2024-04-20T01:03:22.283 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-aktlji-1 2024-04-20T01:03:22.283 DEBUG:boto:Method: POST 2024-04-20T01:03:22.283 DEBUG:boto:Path: / 2024-04-20T01:03:22.283 DEBUG:boto:Data: 2024-04-20T01:03:22.284 DEBUG:boto:Headers: {} 2024-04-20T01:03:22.284 DEBUG:boto:Host: smithi019.front.sepia.ceph.com 2024-04-20T01:03:22.284 DEBUG:boto:Port: 8000 2024-04-20T01:03:22.284 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-04-20T01:03:22.284 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:03:22.285 DEBUG:boto:Token: None 2024-04-20T01:03:22.285 DEBUG:boto:CanonicalRequest: POST / host:smithi019.front.sepia.ceph.com:8000 x-amz-date:20240420T010322Z host;x-amz-date a04830f302cb126c0d85e28932979780eb1d61e6abf92b3ee0fa8929d9b4fd34 2024-04-20T01:03:22.285 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010322Z 20240420/front/smithi019/aws4_request 8520e6088bad1d3c6b3ed03bac15772b790459e719b0a401b575d36fd39a4908 2024-04-20T01:03:22.286 DEBUG:boto:Signature: a3dc6319d8c76d8d4e9d5ffad13b85b836e6eb02b8f0299290e417eb7fd23740 2024-04-20T01:03:22.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010322Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi019/aws4_request,SignedHeaders=host;x-amz-date,Signature=a3dc6319d8c76d8d4e9d5ffad13b85b836e6eb02b8f0299290e417eb7fd23740', 'Host': 'smithi019.front.sepia.ceph.com:8000'} 2024-04-20T01:03:22.302 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c64d40df5a8a395a-006623145a-15b6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Sat, 20 Apr 2024 01:03:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:22.302 DEBUG:boto:b'f9810da8-9c41-4e92-8e12-6d47b7c5927croles-aktlji-1/arn:aws:iam:::role/roles-aktlji-12024-04-20T01:03:22.293Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000c64d40df5a8a395a-006623145a-15b6-test-zone1' 2024-04-20T01:03:22.304 INFO:rgw_multi.tests:create role zone=test-zone2 name=roles-aktlji-2 2024-04-20T01:03:22.304 DEBUG:boto:Method: POST 2024-04-20T01:03:22.304 DEBUG:boto:Path: / 2024-04-20T01:03:22.304 DEBUG:boto:Data: 2024-04-20T01:03:22.304 DEBUG:boto:Headers: {} 2024-04-20T01:03:22.304 DEBUG:boto:Host: smithi031.front.sepia.ceph.com 2024-04-20T01:03:22.304 DEBUG:boto:Port: 8000 2024-04-20T01:03:22.305 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-04-20T01:03:22.305 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:03:22.305 DEBUG:boto:Token: None 2024-04-20T01:03:22.305 DEBUG:boto:CanonicalRequest: POST / host:smithi031.front.sepia.ceph.com:8000 x-amz-date:20240420T010322Z host;x-amz-date 31e4df1ff0fda903c2528903ba6ab306ff0b36e7c0fa2ef966d5e389b4bf9a10 2024-04-20T01:03:22.305 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010322Z 20240420/front/smithi031/aws4_request 6de0f79bb74b7ac0f72bf631600b3db968922ea10d7477ce81e6dac8c1181299 2024-04-20T01:03:22.306 DEBUG:boto:Signature: f7f2b2e9ec3d6fa67e7cba55018c07be2890abe062f9cb2863058f8ef0e715b9 2024-04-20T01:03:22.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010322Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi031/aws4_request,SignedHeaders=host;x-amz-date,Signature=f7f2b2e9ec3d6fa67e7cba55018c07be2890abe062f9cb2863058f8ef0e715b9', 'Host': 'smithi031.front.sepia.ceph.com:8000'} 2024-04-20T01:03:23.332 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000d3d68e0406b3ffa-006623145a-1fb0-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Sat, 20 Apr 2024 01:03:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:23.333 DEBUG:boto:b'ebc48c27-2bea-46ed-bfbc-98578910e519roles-aktlji-2/arn:aws:iam:::role/roles-aktlji-22024-04-20T01:03:23.324Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000000d3d68e0406b3ffa-006623145a-1fb0-test-zone2' 2024-04-20T01:03:23.333 INFO:rgw_multi.tests:create role zone=test-zone3 name=roles-aktlji-3 2024-04-20T01:03:23.333 DEBUG:boto:Method: POST 2024-04-20T01:03:23.334 DEBUG:boto:Path: / 2024-04-20T01:03:23.334 DEBUG:boto:Data: 2024-04-20T01:03:23.334 DEBUG:boto:Headers: {} 2024-04-20T01:03:23.334 DEBUG:boto:Host: smithi019.front.sepia.ceph.com 2024-04-20T01:03:23.334 DEBUG:boto:Port: 8001 2024-04-20T01:03:23.334 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-3', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-04-20T01:03:23.334 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T01:03:23.334 DEBUG:boto:Token: None 2024-04-20T01:03:23.334 DEBUG:boto:CanonicalRequest: POST / host:smithi019.front.sepia.ceph.com:8001 x-amz-date:20240420T010323Z host;x-amz-date 2fd5158aa70b488bf528deebbb36f40db096e4d6b7a7c0f774b05d32ee6203a3 2024-04-20T01:03:23.335 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010323Z 20240420/front/smithi019/aws4_request 8a0bcd5e2906e0ed1b35cd228fe259c078d2a9c2c19738a99e9440bc0ab408cb 2024-04-20T01:03:23.335 DEBUG:boto:Signature: fe73835cb4b610ac05963f3a7da1b2b520748703e0818f791660e92ec7b0124a 2024-04-20T01:03:23.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010323Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi019/aws4_request,SignedHeaders=host;x-amz-date,Signature=fe73835cb4b610ac05963f3a7da1b2b520748703e0818f791660e92ec7b0124a', 'Host': 'smithi019.front.sepia.ceph.com:8001'} 2024-04-20T01:03:23.356 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005a830dd439f7205d-006623145b-16f1-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '731'), ('Date', 'Sat, 20 Apr 2024 01:03:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:23.356 DEBUG:boto:b'616874f1-54c3-40c0-9ce8-f1292e1579d3roles-aktlji-3/arn:aws:iam:::role/roles-aktlji-32024-04-20T01:03:23.350Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000005a830dd439f7205d-006623145b-16f1-test-zone3' 2024-04-20T01:03:23.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:23.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:23.357 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:03:23.416 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:03:23.416 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:03:23.454 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:03:23.470 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:03:23.470 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:23.470 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:03:23.470 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:03:23.470 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:23.470 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:23.470 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:03:23.470 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:03:23.470 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:23.471 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:23.471 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713574691.323977_1063.1", 2024-04-20T01:03:23.471 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T00:58:11.323977Z" 2024-04-20T01:03:23.471 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:23.471 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:23.471 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:03:23.471 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:03:23.471 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:23.471 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:03:23.472 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:23.472 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:03:23.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:23.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:23.473 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:03:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:03:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:03:23.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:03:23.573+0000 7f3cd22e6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:03:23.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:03:23.575+0000 7f3cd22e6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:03:23.593 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:03:23.593 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:03:23.593 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:03:23.593 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:03:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:03:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:03:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:03:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:03:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:03:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:03:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:03:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:03:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:03:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:03:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:03:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:03:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:03:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:03:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:03:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:03:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:03:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:03:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:03:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:03:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:03:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:03:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:03:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:03:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:03:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:03:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:03:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:03:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:03:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:03:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:03:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:03:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:03:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713574691.323977_1063.1", 2024-04-20T01:03:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:03:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:03:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:03:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T00:58:11.323977Z", 2024-04-20T01:03:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:03:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:03:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:03:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:03:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:03:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:03:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:03:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:03:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:03:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:03:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:03:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:03:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:03:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:03:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:03:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:03:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:03:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:03:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:03:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:03:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:03:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:03:23Z" 2024-04-20T01:03:23.599 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:03:23.599 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713574691.323977_1063.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:58:11.323977Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:03:23Z" } 2024-04-20T01:03:23.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713574691.323977_1063.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:58:11.323977Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:03:23.600 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:23.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:23.600 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:03:23.600 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:03:23.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:23.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:23.600 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:03:23.660 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:03:23.660 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:03:23.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:23.694+0000 7f719b114780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:03:23.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:23.695+0000 7f719b114780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:03:23.711 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:03:23.711 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:03:23.711 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:03:23.711 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:03:23.711 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:03:23.711 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:03:23.711 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:23.711 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:23.711 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:03:23.711 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:23.711 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:03:23.712 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:23.712 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:23.712 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:03:23.712 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:23.712 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:03:23.712 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:23.712 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:03:23.712 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:23.712 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:23.712 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:23.712 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:23.712 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:03:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713574691.323977_1063.1", 2024-04-20T01:03:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:03:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:58:11.323977Z", 2024-04-20T01:03:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:03:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.314268_1442.1", 2024-04-20T01:03:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:03:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.314268Z", 2024-04-20T01:03:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:03:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:23.716 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:03:23.716 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:03:23.716 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:03:23.716 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:23.716 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:03:23Z" 2024-04-20T01:03:23.716 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:03:23.717 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713574691.323977_1063.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:58:11.323977Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.314268_1442.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.314268Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:03:23Z" } 2024-04-20T01:03:23.717 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713574691.323977_1063.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:58:11.323977Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.314268_1442.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.314268Z', 'realm_epoch': 2}}] 2024-04-20T01:03:23.717 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:23.717 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713575003.314268_1442.1'} 2024-04-20T01:03:23.717 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1713575003.344070_1445.1 target=1_1713575003.314268_1442.1 2024-04-20T01:03:28.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:28.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:28.719 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:03:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:03:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:03:28.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:28.815+0000 7fd898417780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:03:28.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:28.816+0000 7fd898417780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:03:28.832 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:03:28.832 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:03:28.832 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:03:28.832 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:03:28.832 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:03:28.832 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:03:28.832 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:28.832 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:28.832 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:03:28.833 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:28.833 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:03:28.833 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:28.833 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:28.833 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:03:28.833 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:28.833 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:03:28.833 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:28.833 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:03:28.833 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:28.833 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:28.833 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:28.834 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:28.834 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:03:28.834 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:28.834 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:28.834 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:03:28.834 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:28.834 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:03:28.834 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:28.834 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:03:28.834 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:28.834 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713574691.323977_1063.1", 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T00:58:11.323977Z", 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:28.835 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:28.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:28.836 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:03:28.836 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:28.836 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:28.836 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:03:28.836 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:28.836 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:03:28.836 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:28.836 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:03:28.836 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:28.836 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:28.837 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:28.837 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:03:28.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:28.837 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:03:28.837 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:03:28.837 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:03:28.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:28.837 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:03:28Z" 2024-04-20T01:03:28.837 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:03:28.838 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713574691.323977_1063.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T00:58:11.323977Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:03:28Z" } 2024-04-20T01:03:28.838 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713574691.323977_1063.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:58:11.323977Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:03:28.838 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:28.839 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:28.839 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:03:28.839 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-aktlji-1 zones={test-zone2, test-zone1} 2024-04-20T01:03:28.839 DEBUG:boto:Method: POST 2024-04-20T01:03:28.839 DEBUG:boto:Path: / 2024-04-20T01:03:28.839 DEBUG:boto:Data: 2024-04-20T01:03:28.839 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.839 DEBUG:boto:Host: smithi031.front.sepia.ceph.com 2024-04-20T01:03:28.839 DEBUG:boto:Port: 8000 2024-04-20T01:03:28.839 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-20T01:03:28.840 DEBUG:boto:Token: None 2024-04-20T01:03:28.840 DEBUG:boto:CanonicalRequest: POST / host:smithi031.front.sepia.ceph.com:8000 x-amz-date:20240420T010328Z host;x-amz-date ebaa2306653e1bcead2d9ea1126695792e86050b87014ccc24565e95a70ef038 2024-04-20T01:03:28.840 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010328Z 20240420/front/smithi031/aws4_request 1e33ad09b43cd77cd37fcdf46d8c811abe0cb3279a1f2c3622ec08a0ca171e42 2024-04-20T01:03:28.840 DEBUG:boto:Signature: c80fa2c44f9b9cb6e82f151c8e8d28edbf3085182a2bfb0b23232a90887efd4b 2024-04-20T01:03:28.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi031/aws4_request,SignedHeaders=host;x-amz-date,Signature=c80fa2c44f9b9cb6e82f151c8e8d28edbf3085182a2bfb0b23232a90887efd4b', 'Host': 'smithi031.front.sepia.ceph.com:8000'} 2024-04-20T01:03:28.842 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ada51637ac1991d8-0066231460-1fb0-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.843 DEBUG:boto:b'tx00000ada51637ac1991d8-0066231460-1fb0-test-zone2f9810da8-9c41-4e92-8e12-6d47b7c5927croles-aktlji-1/arn:aws:iam:::role/roles-aktlji-12024-04-20T01:03:22.487Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-20T01:03:28.843 DEBUG:boto:Method: POST 2024-04-20T01:03:28.843 DEBUG:boto:Path: / 2024-04-20T01:03:28.844 DEBUG:boto:Data: 2024-04-20T01:03:28.844 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.844 DEBUG:boto:Host: smithi019.front.sepia.ceph.com 2024-04-20T01:03:28.844 DEBUG:boto:Port: 8000 2024-04-20T01:03:28.844 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-20T01:03:28.844 DEBUG:boto:Token: None 2024-04-20T01:03:28.844 DEBUG:boto:CanonicalRequest: POST / host:smithi019.front.sepia.ceph.com:8000 x-amz-date:20240420T010328Z host;x-amz-date ebaa2306653e1bcead2d9ea1126695792e86050b87014ccc24565e95a70ef038 2024-04-20T01:03:28.844 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010328Z 20240420/front/smithi019/aws4_request 63e9c461db7c592f16e99371944d205021c24ab60e6fcbdfc6c348795c1ef957 2024-04-20T01:03:28.844 DEBUG:boto:Signature: fee783c2593b763afd55f374aa83d3410cb13570764525999b73e0b2fe7f5f7f 2024-04-20T01:03:28.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi019/aws4_request,SignedHeaders=host;x-amz-date,Signature=fee783c2593b763afd55f374aa83d3410cb13570764525999b73e0b2fe7f5f7f', 'Host': 'smithi019.front.sepia.ceph.com:8000'} 2024-04-20T01:03:28.846 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ea1829340830345-0066231460-15b6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.846 DEBUG:boto:b'tx000007ea1829340830345-0066231460-15b6-test-zone1f9810da8-9c41-4e92-8e12-6d47b7c5927croles-aktlji-1/arn:aws:iam:::role/roles-aktlji-12024-04-20T01:03:22.293Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-20T01:03:28.847 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-aktlji-1 2024-04-20T01:03:28.847 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-aktlji-1 zones={test-zone2, test-zone1} 2024-04-20T01:03:28.847 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-aktlji-1 zones={test-zone3, test-zone1} 2024-04-20T01:03:28.847 DEBUG:boto:Method: POST 2024-04-20T01:03:28.847 DEBUG:boto:Path: / 2024-04-20T01:03:28.847 DEBUG:boto:Data: 2024-04-20T01:03:28.847 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.847 DEBUG:boto:Host: smithi019.front.sepia.ceph.com 2024-04-20T01:03:28.847 DEBUG:boto:Port: 8001 2024-04-20T01:03:28.847 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-20T01:03:28.847 DEBUG:boto:Token: None 2024-04-20T01:03:28.848 DEBUG:boto:CanonicalRequest: POST / host:smithi019.front.sepia.ceph.com:8001 x-amz-date:20240420T010328Z host;x-amz-date ebaa2306653e1bcead2d9ea1126695792e86050b87014ccc24565e95a70ef038 2024-04-20T01:03:28.848 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010328Z 20240420/front/smithi019/aws4_request 1f7bc997e45d56e2d8610df25614c25c1e03e8390cf9cada20161a1313037926 2024-04-20T01:03:28.848 DEBUG:boto:Signature: 143b9c14da09541de5032048a88c292af4e33833cf27fc0791e860389bdb5663 2024-04-20T01:03:28.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi019/aws4_request,SignedHeaders=host;x-amz-date,Signature=143b9c14da09541de5032048a88c292af4e33833cf27fc0791e860389bdb5663', 'Host': 'smithi019.front.sepia.ceph.com:8001'} 2024-04-20T01:03:28.850 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000080517e2b4b20c897-0066231460-16f1-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.850 DEBUG:boto:b'tx0000080517e2b4b20c897-0066231460-16f1-test-zone3f9810da8-9c41-4e92-8e12-6d47b7c5927croles-aktlji-1/arn:aws:iam:::role/roles-aktlji-12024-04-20T01:03:22.486Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-20T01:03:28.851 DEBUG:boto:Method: POST 2024-04-20T01:03:28.851 DEBUG:boto:Path: / 2024-04-20T01:03:28.851 DEBUG:boto:Data: 2024-04-20T01:03:28.851 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.851 DEBUG:boto:Host: smithi019.front.sepia.ceph.com 2024-04-20T01:03:28.851 DEBUG:boto:Port: 8000 2024-04-20T01:03:28.851 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-20T01:03:28.851 DEBUG:boto:Token: None 2024-04-20T01:03:28.851 DEBUG:boto:CanonicalRequest: POST / host:smithi019.front.sepia.ceph.com:8000 x-amz-date:20240420T010328Z host;x-amz-date ebaa2306653e1bcead2d9ea1126695792e86050b87014ccc24565e95a70ef038 2024-04-20T01:03:28.852 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010328Z 20240420/front/smithi019/aws4_request 63e9c461db7c592f16e99371944d205021c24ab60e6fcbdfc6c348795c1ef957 2024-04-20T01:03:28.852 DEBUG:boto:Signature: fee783c2593b763afd55f374aa83d3410cb13570764525999b73e0b2fe7f5f7f 2024-04-20T01:03:28.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi019/aws4_request,SignedHeaders=host;x-amz-date,Signature=fee783c2593b763afd55f374aa83d3410cb13570764525999b73e0b2fe7f5f7f', 'Host': 'smithi019.front.sepia.ceph.com:8000'} 2024-04-20T01:03:28.853 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c1c193061033e8f7-0066231460-15b6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.853 DEBUG:boto:b'tx00000c1c193061033e8f7-0066231460-15b6-test-zone1f9810da8-9c41-4e92-8e12-6d47b7c5927croles-aktlji-1/arn:aws:iam:::role/roles-aktlji-12024-04-20T01:03:22.293Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-20T01:03:28.854 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-aktlji-1 2024-04-20T01:03:28.854 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-aktlji-1 zones={test-zone3, test-zone1} 2024-04-20T01:03:28.854 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-aktlji-2 zones={test-zone1, test-zone2} 2024-04-20T01:03:28.854 DEBUG:boto:Method: POST 2024-04-20T01:03:28.854 DEBUG:boto:Path: / 2024-04-20T01:03:28.854 DEBUG:boto:Data: 2024-04-20T01:03:28.854 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.854 DEBUG:boto:Host: smithi019.front.sepia.ceph.com 2024-04-20T01:03:28.854 DEBUG:boto:Port: 8000 2024-04-20T01:03:28.854 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-20T01:03:28.854 DEBUG:boto:Token: None 2024-04-20T01:03:28.855 DEBUG:boto:CanonicalRequest: POST / host:smithi019.front.sepia.ceph.com:8000 x-amz-date:20240420T010328Z host;x-amz-date 4243ecdcd2bab9b9c6c4c0f61f61fbddff999e014ef265d43e7ed0891117b0e9 2024-04-20T01:03:28.855 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010328Z 20240420/front/smithi019/aws4_request c28c2a53ed845ab26eeaa380719dc9ee1906efcdded6a39e77089863c2af57c3 2024-04-20T01:03:28.855 DEBUG:boto:Signature: 0bd122c496036c9f9b129de27e1aa0f5e2dcd6d4aa4d29c33e9b01d9c9810839 2024-04-20T01:03:28.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi019/aws4_request,SignedHeaders=host;x-amz-date,Signature=0bd122c496036c9f9b129de27e1aa0f5e2dcd6d4aa4d29c33e9b01d9c9810839', 'Host': 'smithi019.front.sepia.ceph.com:8000'} 2024-04-20T01:03:28.856 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b2dbcbf2ace8d03f-0066231460-15b6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.857 DEBUG:boto:b'tx00000b2dbcbf2ace8d03f-0066231460-15b6-test-zone1ebc48c27-2bea-46ed-bfbc-98578910e519roles-aktlji-2/arn:aws:iam:::role/roles-aktlji-22024-04-20T01:03:23.314Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-20T01:03:28.857 DEBUG:boto:Method: POST 2024-04-20T01:03:28.857 DEBUG:boto:Path: / 2024-04-20T01:03:28.857 DEBUG:boto:Data: 2024-04-20T01:03:28.857 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.857 DEBUG:boto:Host: smithi031.front.sepia.ceph.com 2024-04-20T01:03:28.858 DEBUG:boto:Port: 8000 2024-04-20T01:03:28.858 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-20T01:03:28.858 DEBUG:boto:Token: None 2024-04-20T01:03:28.858 DEBUG:boto:CanonicalRequest: POST / host:smithi031.front.sepia.ceph.com:8000 x-amz-date:20240420T010328Z host;x-amz-date 4243ecdcd2bab9b9c6c4c0f61f61fbddff999e014ef265d43e7ed0891117b0e9 2024-04-20T01:03:28.858 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010328Z 20240420/front/smithi031/aws4_request 29e704dd9118f7dff1fd3f41cf91b3f8320abd3cae7238334b46b471267439f0 2024-04-20T01:03:28.858 DEBUG:boto:Signature: 07eb6ee152a2da4cf92ca44f643289b50044902524ec859144e4b57c0bb3320e 2024-04-20T01:03:28.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi031/aws4_request,SignedHeaders=host;x-amz-date,Signature=07eb6ee152a2da4cf92ca44f643289b50044902524ec859144e4b57c0bb3320e', 'Host': 'smithi031.front.sepia.ceph.com:8000'} 2024-04-20T01:03:28.861 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008134b5ee5257b0de-0066231460-1fb0-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.861 DEBUG:boto:b'tx000008134b5ee5257b0de-0066231460-1fb0-test-zone2ebc48c27-2bea-46ed-bfbc-98578910e519roles-aktlji-2/arn:aws:iam:::role/roles-aktlji-22024-04-20T01:03:23.314Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-20T01:03:28.861 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-aktlji-2 2024-04-20T01:03:28.861 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-aktlji-2 zones={test-zone1, test-zone2} 2024-04-20T01:03:28.861 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-aktlji-2 zones={test-zone3, test-zone2} 2024-04-20T01:03:28.862 DEBUG:boto:Method: POST 2024-04-20T01:03:28.862 DEBUG:boto:Path: / 2024-04-20T01:03:28.862 DEBUG:boto:Data: 2024-04-20T01:03:28.862 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.862 DEBUG:boto:Host: smithi019.front.sepia.ceph.com 2024-04-20T01:03:28.862 DEBUG:boto:Port: 8001 2024-04-20T01:03:28.862 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-20T01:03:28.862 DEBUG:boto:Token: None 2024-04-20T01:03:28.862 DEBUG:boto:CanonicalRequest: POST / host:smithi019.front.sepia.ceph.com:8001 x-amz-date:20240420T010328Z host;x-amz-date 4243ecdcd2bab9b9c6c4c0f61f61fbddff999e014ef265d43e7ed0891117b0e9 2024-04-20T01:03:28.862 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010328Z 20240420/front/smithi019/aws4_request 1b32d890355dec6615fb098cf6837ed3ac921c1498d5ee669cdb44239930d931 2024-04-20T01:03:28.863 DEBUG:boto:Signature: e7315ca2a6d0e16955746bf5f5807d5268fd08f68a0dcb01fb774cc284bb24ff 2024-04-20T01:03:28.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi019/aws4_request,SignedHeaders=host;x-amz-date,Signature=e7315ca2a6d0e16955746bf5f5807d5268fd08f68a0dcb01fb774cc284bb24ff', 'Host': 'smithi019.front.sepia.ceph.com:8001'} 2024-04-20T01:03:28.864 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000257bed18c026bc53-0066231460-16f1-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.865 DEBUG:boto:b'tx00000257bed18c026bc53-0066231460-16f1-test-zone3ebc48c27-2bea-46ed-bfbc-98578910e519roles-aktlji-2/arn:aws:iam:::role/roles-aktlji-22024-04-20T01:03:23.314Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-20T01:03:28.865 DEBUG:boto:Method: POST 2024-04-20T01:03:28.865 DEBUG:boto:Path: / 2024-04-20T01:03:28.865 DEBUG:boto:Data: 2024-04-20T01:03:28.865 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.866 DEBUG:boto:Host: smithi031.front.sepia.ceph.com 2024-04-20T01:03:28.866 DEBUG:boto:Port: 8000 2024-04-20T01:03:28.866 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-20T01:03:28.866 DEBUG:boto:Token: None 2024-04-20T01:03:28.866 DEBUG:boto:CanonicalRequest: POST / host:smithi031.front.sepia.ceph.com:8000 x-amz-date:20240420T010328Z host;x-amz-date 4243ecdcd2bab9b9c6c4c0f61f61fbddff999e014ef265d43e7ed0891117b0e9 2024-04-20T01:03:28.866 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010328Z 20240420/front/smithi031/aws4_request 29e704dd9118f7dff1fd3f41cf91b3f8320abd3cae7238334b46b471267439f0 2024-04-20T01:03:28.866 DEBUG:boto:Signature: 07eb6ee152a2da4cf92ca44f643289b50044902524ec859144e4b57c0bb3320e 2024-04-20T01:03:28.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi031/aws4_request,SignedHeaders=host;x-amz-date,Signature=07eb6ee152a2da4cf92ca44f643289b50044902524ec859144e4b57c0bb3320e', 'Host': 'smithi031.front.sepia.ceph.com:8000'} 2024-04-20T01:03:28.868 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a99de4bad3b24894-0066231460-1fb0-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.868 DEBUG:boto:b'tx00000a99de4bad3b24894-0066231460-1fb0-test-zone2ebc48c27-2bea-46ed-bfbc-98578910e519roles-aktlji-2/arn:aws:iam:::role/roles-aktlji-22024-04-20T01:03:23.314Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-20T01:03:28.868 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-aktlji-2 2024-04-20T01:03:28.868 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-aktlji-2 zones={test-zone3, test-zone2} 2024-04-20T01:03:28.868 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-aktlji-3 zones={test-zone1, test-zone3} 2024-04-20T01:03:28.868 DEBUG:boto:Method: POST 2024-04-20T01:03:28.869 DEBUG:boto:Path: / 2024-04-20T01:03:28.869 DEBUG:boto:Data: 2024-04-20T01:03:28.869 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.869 DEBUG:boto:Host: smithi019.front.sepia.ceph.com 2024-04-20T01:03:28.869 DEBUG:boto:Port: 8000 2024-04-20T01:03:28.869 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-20T01:03:28.869 DEBUG:boto:Token: None 2024-04-20T01:03:28.869 DEBUG:boto:CanonicalRequest: POST / host:smithi019.front.sepia.ceph.com:8000 x-amz-date:20240420T010328Z host;x-amz-date a76447f275d1b0fc432e67b7cc54f8d0664e9f910831215ea5a402d6925069fe 2024-04-20T01:03:28.869 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010328Z 20240420/front/smithi019/aws4_request ebd2885d569eed2ebcdbca9928899c996054fd26ac2c61d97be55b1b4c80291d 2024-04-20T01:03:28.870 DEBUG:boto:Signature: 18152432b3aad6d9537ec0a1849117a4781735cde70d77bd959e8d6b5690adb7 2024-04-20T01:03:28.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi019/aws4_request,SignedHeaders=host;x-amz-date,Signature=18152432b3aad6d9537ec0a1849117a4781735cde70d77bd959e8d6b5690adb7', 'Host': 'smithi019.front.sepia.ceph.com:8000'} 2024-04-20T01:03:28.871 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c0014afa29c0efb3-0066231460-15b6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.871 DEBUG:boto:b'tx00000c0014afa29c0efb3-0066231460-15b6-test-zone1616874f1-54c3-40c0-9ce8-f1292e1579d3roles-aktlji-3/arn:aws:iam:::role/roles-aktlji-32024-04-20T01:03:23.340Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-20T01:03:28.871 DEBUG:boto:Method: POST 2024-04-20T01:03:28.871 DEBUG:boto:Path: / 2024-04-20T01:03:28.872 DEBUG:boto:Data: 2024-04-20T01:03:28.872 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.872 DEBUG:boto:Host: smithi019.front.sepia.ceph.com 2024-04-20T01:03:28.872 DEBUG:boto:Port: 8001 2024-04-20T01:03:28.872 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-20T01:03:28.872 DEBUG:boto:Token: None 2024-04-20T01:03:28.872 DEBUG:boto:CanonicalRequest: POST / host:smithi019.front.sepia.ceph.com:8001 x-amz-date:20240420T010328Z host;x-amz-date a76447f275d1b0fc432e67b7cc54f8d0664e9f910831215ea5a402d6925069fe 2024-04-20T01:03:28.872 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010328Z 20240420/front/smithi019/aws4_request d26f97c2ed8ac8e8c8e0d2f65fcd38e90f067a0007049c4540d921edd29266b7 2024-04-20T01:03:28.872 DEBUG:boto:Signature: 86f2e3ea712c27577859ebc2221f41aee7b90596f37cdd5ed5c787303af4566c 2024-04-20T01:03:28.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi019/aws4_request,SignedHeaders=host;x-amz-date,Signature=86f2e3ea712c27577859ebc2221f41aee7b90596f37cdd5ed5c787303af4566c', 'Host': 'smithi019.front.sepia.ceph.com:8001'} 2024-04-20T01:03:28.874 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ec2b6f54fe8a71f4-0066231460-16f1-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.874 DEBUG:boto:b'tx00000ec2b6f54fe8a71f4-0066231460-16f1-test-zone3616874f1-54c3-40c0-9ce8-f1292e1579d3roles-aktlji-3/arn:aws:iam:::role/roles-aktlji-32024-04-20T01:03:23.340Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-20T01:03:28.875 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-aktlji-3 2024-04-20T01:03:28.875 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-aktlji-3 zones={test-zone1, test-zone3} 2024-04-20T01:03:28.875 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-aktlji-3 zones={test-zone2, test-zone3} 2024-04-20T01:03:28.875 DEBUG:boto:Method: POST 2024-04-20T01:03:28.875 DEBUG:boto:Path: / 2024-04-20T01:03:28.875 DEBUG:boto:Data: 2024-04-20T01:03:28.875 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.875 DEBUG:boto:Host: smithi031.front.sepia.ceph.com 2024-04-20T01:03:28.876 DEBUG:boto:Port: 8000 2024-04-20T01:03:28.876 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-20T01:03:28.876 DEBUG:boto:Token: None 2024-04-20T01:03:28.876 DEBUG:boto:CanonicalRequest: POST / host:smithi031.front.sepia.ceph.com:8000 x-amz-date:20240420T010328Z host;x-amz-date a76447f275d1b0fc432e67b7cc54f8d0664e9f910831215ea5a402d6925069fe 2024-04-20T01:03:28.876 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010328Z 20240420/front/smithi031/aws4_request 4f618360a6f809984dfb9aff22b8c574dfe5af7b87551ec44517e510cae7edfd 2024-04-20T01:03:28.876 DEBUG:boto:Signature: 71878f6fae651b89568e8f64c69bc21a0ff29c0cbf2ed88514b1ab93ed035cb9 2024-04-20T01:03:28.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi031/aws4_request,SignedHeaders=host;x-amz-date,Signature=71878f6fae651b89568e8f64c69bc21a0ff29c0cbf2ed88514b1ab93ed035cb9', 'Host': 'smithi031.front.sepia.ceph.com:8000'} 2024-04-20T01:03:28.878 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b6a95ad1f6707f41-0066231460-1fb0-test-zone2'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.878 DEBUG:boto:b'tx00000b6a95ad1f6707f41-0066231460-1fb0-test-zone2616874f1-54c3-40c0-9ce8-f1292e1579d3roles-aktlji-3/arn:aws:iam:::role/roles-aktlji-32024-04-20T01:03:23.379Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-20T01:03:28.879 DEBUG:boto:Method: POST 2024-04-20T01:03:28.879 DEBUG:boto:Path: / 2024-04-20T01:03:28.879 DEBUG:boto:Data: 2024-04-20T01:03:28.879 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.879 DEBUG:boto:Host: smithi019.front.sepia.ceph.com 2024-04-20T01:03:28.879 DEBUG:boto:Port: 8001 2024-04-20T01:03:28.879 DEBUG:boto:Params: {'RoleName': 'roles-aktlji-3', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-20T01:03:28.880 DEBUG:boto:Token: None 2024-04-20T01:03:28.880 DEBUG:boto:CanonicalRequest: POST / host:smithi019.front.sepia.ceph.com:8001 x-amz-date:20240420T010328Z host;x-amz-date a76447f275d1b0fc432e67b7cc54f8d0664e9f910831215ea5a402d6925069fe 2024-04-20T01:03:28.880 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240420T010328Z 20240420/front/smithi019/aws4_request d26f97c2ed8ac8e8c8e0d2f65fcd38e90f067a0007049c4540d921edd29266b7 2024-04-20T01:03:28.880 DEBUG:boto:Signature: 86f2e3ea712c27577859ebc2221f41aee7b90596f37cdd5ed5c787303af4566c 2024-04-20T01:03:28.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240420T010328Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=T4PF8H1DJQT67HZSKX5F/20240420/front/smithi019/aws4_request,SignedHeaders=host;x-amz-date,Signature=86f2e3ea712c27577859ebc2221f41aee7b90596f37cdd5ed5c787303af4566c', 'Host': 'smithi019.front.sepia.ceph.com:8001'} 2024-04-20T01:03:28.881 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000027d235aa143a2d48-0066231460-16f1-test-zone3'), ('Content-Type', 'application/xml'), ('Content-Length', '719'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.882 DEBUG:boto:b'tx0000027d235aa143a2d48-0066231460-16f1-test-zone3616874f1-54c3-40c0-9ce8-f1292e1579d3roles-aktlji-3/arn:aws:iam:::role/roles-aktlji-32024-04-20T01:03:23.340Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-20T01:03:28.882 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-aktlji-3 2024-04-20T01:03:28.882 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-aktlji-3 zones={test-zone2, test-zone3} 2024-04-20T01:03:28.882 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2024-04-20T01:03:28.883 DEBUG:boto:path=/aktlji-74/ 2024-04-20T01:03:28.883 DEBUG:boto:auth_path=/aktlji-74/ 2024-04-20T01:03:28.883 DEBUG:boto:Method: PUT 2024-04-20T01:03:28.883 DEBUG:boto:Path: /aktlji-74/ 2024-04-20T01:03:28.883 DEBUG:boto:Data: 2024-04-20T01:03:28.883 DEBUG:boto:Headers: {} 2024-04-20T01:03:28.884 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:03:28.884 DEBUG:boto:Port: 8000 2024-04-20T01:03:28.884 DEBUG:boto:Params: {} 2024-04-20T01:03:28.884 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:03:28.884 DEBUG:boto:Token: None 2024-04-20T01:03:28.884 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:03:28 GMT /aktlji-74/ 2024-04-20T01:03:28.884 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gre5QuW480JFWxNer0oVfpqYdAY= 2024-04-20T01:03:28.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:03:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gre5QuW480JFWxNer0oVfpqYdAY=', 'Content-Length': '0'} 2024-04-20T01:03:28.906 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc0802ff73dd7102-0066231460-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:28.906 DEBUG:rgw_multi.tests:created bucket=aktlji-74 2024-04-20T01:03:28.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:28.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:28.907 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:03:28.962 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:03:28.962 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:03:28.998 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:03:29.013 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:03:29.013 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:29.014 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:03:29.014 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:03:29.014 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:29.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:29.014 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:03:29.014 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:03:29.014 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:29.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:29.014 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575008.904609_1102.1", 2024-04-20T01:03:29.014 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:28.904609Z" 2024-04-20T01:03:29.015 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:29.015 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:29.015 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:03:29.015 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:03:29.015 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:29.015 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:03:29.016 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.904609_1102.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:29.016 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:03:29.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:29.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:29.016 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:03:29.076 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:03:29.077 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:03:29.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:03:29.143+0000 7f79e2d06780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:03:29.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:03:29.144+0000 7f79e2d06780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:03:29.170 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:03:29.170 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:03:29.170 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:03:29.170 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:03:29.170 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:03:29.170 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:03:29.170 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:03:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:03:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:03:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:03:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:03:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:03:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:03:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:03:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:03:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:03:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:03:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:03:29.171 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:03:29.172 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:03:29.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:03:29.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:03:29.172 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:03:29.172 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:03:29.172 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:03:29.172 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:03:29.172 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:03:29.172 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:03:29.172 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:03:29.172 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:03:29.172 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:03:29.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:03:29.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:03:29.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:03:29.173 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:03:29.173 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:03:29.173 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:03:29.173 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575008.904609_1102.1", 2024-04-20T01:03:29.173 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:03:29.173 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:03:29.173 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:03:29.173 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:28.904609Z", 2024-04-20T01:03:29.173 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:03:29.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:03:29.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:03:29.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:03:29.174 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:03:29.174 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:03:29.174 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:03:29.174 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:03:29.174 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:03:29.174 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:03:29.174 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:03:29.174 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:03:29.175 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:03:29.175 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:03:29.175 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:03:29.175 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:03:29.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:03:29.175 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:03:29.175 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:03:29.175 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:03:29.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:03:29.175 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:03:29Z" 2024-04-20T01:03:29.175 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:03:29.176 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713575008.904609_1102.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:28.904609Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:03:29Z" } 2024-04-20T01:03:29.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575008.904609_1102.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:28.904609Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:03:29.177 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.904609_1102.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:29.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.904609_1102.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:29.177 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:03:29.177 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:03:29.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:29.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:29.177 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:03:29.237 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:03:29.237 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:03:29.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:29.272+0000 7fe1f8c4f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:03:29.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:29.273+0000 7fe1f8c4f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:03:29.290 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:03:29.290 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:03:29.290 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:03:29.290 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:03:29.290 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:03:29.290 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:03:29.290 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:29.290 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:29.290 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:29.291 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:29.292 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:03:29.292 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:29.292 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:29.292 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:03:29.292 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:29.292 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:03:29.292 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:29.292 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:03:29.292 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:29.292 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:29.292 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:29.292 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:29.293 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:03:29.293 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:29.293 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:29.293 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575008.896558_1099.1", 2024-04-20T01:03:29.293 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:29.293 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:03:29.293 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:29.293 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:28.896558Z", 2024-04-20T01:03:29.293 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:29.293 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:29.293 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:29.293 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:29.294 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:03:29.294 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:29.294 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:29.294 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:03:29.294 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:29.294 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:03:29.294 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:29.294 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:03:29.294 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:29.294 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:29.294 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:29.295 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:03:29.295 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:29.295 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:03:29.295 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:03:29.295 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:03:29.295 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:29.295 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:03:29Z" 2024-04-20T01:03:29.295 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:03:29.296 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713575008.896558_1099.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:28.896558Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:03:29Z" } 2024-04-20T01:03:29.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575008.896558_1099.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:28.896558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:03:29.296 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.904609_1102.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:29.296 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.896558_1099.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:29.296 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1713575008.904609_1102.1 target=1_1713575008.896558_1099.1 2024-04-20T01:03:34.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:34.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:03:34.299 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:03:34.359 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:03:34.359 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:03:34.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:34.394+0000 7f3834791780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:03:34.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:34.395+0000 7f3834791780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:03:34.411 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:03:34.411 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:03:34.411 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:03:34.411 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:03:34.411 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:03:34.411 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:03:34.411 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:34.411 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:34.412 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:03:34.412 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:34.412 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:03:34.412 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:34.412 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:34.412 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:03:34.412 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:34.412 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:03:34.413 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:34.413 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:03:34.413 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:34.413 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:34.413 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:34.413 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:34.413 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:03:34.414 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:34.414 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:34.414 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:03:34.414 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:34.414 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:03:34.414 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:34.414 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:03:34.414 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:34.415 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:34.415 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:34.415 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:34.415 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:03:34.415 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:34.415 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:34.415 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575008.904609_1102.1", 2024-04-20T01:03:34.415 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:34.416 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:03:34.416 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:34.416 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:28.904609Z", 2024-04-20T01:03:34.416 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:34.416 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:34.416 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:34.416 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:03:34.416 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:03:34.416 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:03:34.417 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:03:34.417 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:03:34.417 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:03:34.417 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:03:34.417 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:03:34.417 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:03:34.417 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:03:34.417 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:34.417 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:03:34.418 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:03:34.418 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:34.418 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:03:34.418 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:03:34.418 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:03:34.418 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:03:34.418 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:03:34Z" 2024-04-20T01:03:34.418 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:03:34.419 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713575008.904609_1102.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:28.904609Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:03:34Z" } 2024-04-20T01:03:34.420 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575008.904609_1102.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:28.904609Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:03:34.420 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.904609_1102.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:34.420 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.904609_1102.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:03:34.420 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:03:34.421 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-20T01:03:34.421 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:03:34.422 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-20T01:03:34.421+0000 7faf8cbd8700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 100462) UID: 0 2024-04-20T01:03:34.422 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-20T01:03:34.421+0000 7faf91dad7c0 -1 shutting down 2024-04-20T01:03:40.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-20T01:03:40.523 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-20T01:03:40.523 DEBUG:boto:path=/aktlji-74/ 2024-04-20T01:03:40.523 DEBUG:boto:auth_path=/aktlji-74/ 2024-04-20T01:03:40.524 DEBUG:boto:Method: HEAD 2024-04-20T01:03:40.524 DEBUG:boto:Path: /aktlji-74/ 2024-04-20T01:03:40.524 DEBUG:boto:Data: 2024-04-20T01:03:40.524 DEBUG:boto:Headers: {} 2024-04-20T01:03:40.524 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:03:40.524 DEBUG:boto:Port: 8000 2024-04-20T01:03:40.524 DEBUG:boto:Params: {} 2024-04-20T01:03:40.525 DEBUG:boto:Token: None 2024-04-20T01:03:40.525 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:03:40 GMT /aktlji-74/ 2024-04-20T01:03:40.525 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4= 2024-04-20T01:03:40.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4=', 'Content-Length': '0'} 2024-04-20T01:03:40.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b82c76c0fda1e0e4-006623146c-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:40.529 DEBUG:boto:path=/aktlji-74/obj11 2024-04-20T01:03:40.529 DEBUG:boto:auth_path=/aktlji-74/obj11 2024-04-20T01:03:40.529 DEBUG:boto:Method: PUT 2024-04-20T01:03:40.529 DEBUG:boto:Path: /aktlji-74/obj11 2024-04-20T01:03:40.530 DEBUG:boto:Data: 2024-04-20T01:03:40.530 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:03:40.530 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:03:40.530 DEBUG:boto:Port: 8000 2024-04-20T01:03:40.530 DEBUG:boto:Params: {} 2024-04-20T01:03:40.530 DEBUG:boto:Token: None 2024-04-20T01:03:40.530 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:03:40 GMT /aktlji-74/obj11 2024-04-20T01:03:40.531 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KLLYFcx31SY8zsPwNo9txJNSW6Q= 2024-04-20T01:03:40.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KLLYFcx31SY8zsPwNo9txJNSW6Q='} 2024-04-20T01:03:40.539 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bff0ff75ce660c2e-006623146c-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:40.540 DEBUG:boto:path=/aktlji-74/ 2024-04-20T01:03:40.540 DEBUG:boto:auth_path=/aktlji-74/ 2024-04-20T01:03:40.540 DEBUG:boto:Method: HEAD 2024-04-20T01:03:40.540 DEBUG:boto:Path: /aktlji-74/ 2024-04-20T01:03:40.540 DEBUG:boto:Data: 2024-04-20T01:03:40.540 DEBUG:boto:Headers: {} 2024-04-20T01:03:40.540 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:03:40.540 DEBUG:boto:Port: 8000 2024-04-20T01:03:40.541 DEBUG:boto:Params: {} 2024-04-20T01:03:40.541 DEBUG:boto:Token: None 2024-04-20T01:03:40.541 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:03:40 GMT /aktlji-74/ 2024-04-20T01:03:40.541 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4= 2024-04-20T01:03:40.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4=', 'Content-Length': '0'} 2024-04-20T01:03:40.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d246e66b906b05c0-006623146c-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:40.547 DEBUG:boto:path=/aktlji-74/obj22 2024-04-20T01:03:40.547 DEBUG:boto:auth_path=/aktlji-74/obj22 2024-04-20T01:03:40.547 DEBUG:boto:Method: PUT 2024-04-20T01:03:40.547 DEBUG:boto:Path: /aktlji-74/obj22 2024-04-20T01:03:40.547 DEBUG:boto:Data: 2024-04-20T01:03:40.547 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:03:40.547 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:03:40.547 DEBUG:boto:Port: 8000 2024-04-20T01:03:40.547 DEBUG:boto:Params: {} 2024-04-20T01:03:40.548 DEBUG:boto:Token: None 2024-04-20T01:03:40.548 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:03:40 GMT /aktlji-74/obj22 2024-04-20T01:03:40.548 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:t6Vhg/2Pa3d1dh7qUIDt+77jmjI= 2024-04-20T01:03:40.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:t6Vhg/2Pa3d1dh7qUIDt+77jmjI='} 2024-04-20T01:03:40.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfa0f255808b4316-006623146c-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:40.591 DEBUG:boto:path=/aktlji-74/ 2024-04-20T01:03:40.591 DEBUG:boto:auth_path=/aktlji-74/ 2024-04-20T01:03:40.591 DEBUG:boto:Method: HEAD 2024-04-20T01:03:40.591 DEBUG:boto:Path: /aktlji-74/ 2024-04-20T01:03:40.591 DEBUG:boto:Data: 2024-04-20T01:03:40.591 DEBUG:boto:Headers: {} 2024-04-20T01:03:40.591 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:03:40.592 DEBUG:boto:Port: 8000 2024-04-20T01:03:40.592 DEBUG:boto:Params: {} 2024-04-20T01:03:40.592 DEBUG:boto:Token: None 2024-04-20T01:03:40.592 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:03:40 GMT /aktlji-74/ 2024-04-20T01:03:40.592 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4= 2024-04-20T01:03:40.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4=', 'Content-Length': '0'} 2024-04-20T01:03:40.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e20c1de30acf869e-006623146c-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:40.595 DEBUG:boto:path=/aktlji-74/obj33 2024-04-20T01:03:40.595 DEBUG:boto:auth_path=/aktlji-74/obj33 2024-04-20T01:03:40.595 DEBUG:boto:Method: PUT 2024-04-20T01:03:40.595 DEBUG:boto:Path: /aktlji-74/obj33 2024-04-20T01:03:40.596 DEBUG:boto:Data: 2024-04-20T01:03:40.596 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:03:40.596 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:03:40.596 DEBUG:boto:Port: 8000 2024-04-20T01:03:40.596 DEBUG:boto:Params: {} 2024-04-20T01:03:40.596 DEBUG:boto:Token: None 2024-04-20T01:03:40.596 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:03:40 GMT /aktlji-74/obj33 2024-04-20T01:03:40.596 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:feHLlioOVlgAb1sKs2YoxrLIJCU= 2024-04-20T01:03:40.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:feHLlioOVlgAb1sKs2YoxrLIJCU='} 2024-04-20T01:03:40.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094ee7789141c0ce8-006623146c-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:40.604 DEBUG:boto:path=/aktlji-74/ 2024-04-20T01:03:40.604 DEBUG:boto:auth_path=/aktlji-74/ 2024-04-20T01:03:40.604 DEBUG:boto:Method: HEAD 2024-04-20T01:03:40.604 DEBUG:boto:Path: /aktlji-74/ 2024-04-20T01:03:40.604 DEBUG:boto:Data: 2024-04-20T01:03:40.604 DEBUG:boto:Headers: {} 2024-04-20T01:03:40.604 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:03:40.605 DEBUG:boto:Port: 8000 2024-04-20T01:03:40.605 DEBUG:boto:Params: {} 2024-04-20T01:03:40.605 DEBUG:boto:Token: None 2024-04-20T01:03:40.605 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:03:40 GMT /aktlji-74/ 2024-04-20T01:03:40.605 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4= 2024-04-20T01:03:40.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4=', 'Content-Length': '0'} 2024-04-20T01:03:40.608 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b360b6d8578b3b6-006623146c-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:40.608 DEBUG:boto:path=/aktlji-74/obj44 2024-04-20T01:03:40.608 DEBUG:boto:auth_path=/aktlji-74/obj44 2024-04-20T01:03:40.609 DEBUG:boto:Method: PUT 2024-04-20T01:03:40.609 DEBUG:boto:Path: /aktlji-74/obj44 2024-04-20T01:03:40.609 DEBUG:boto:Data: 2024-04-20T01:03:40.609 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:03:40.609 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:03:40.609 DEBUG:boto:Port: 8000 2024-04-20T01:03:40.609 DEBUG:boto:Params: {} 2024-04-20T01:03:40.609 DEBUG:boto:Token: None 2024-04-20T01:03:40.609 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:03:40 GMT /aktlji-74/obj44 2024-04-20T01:03:40.610 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2vkRpRUOOq+s/XUHCrJHn4MA7VI= 2024-04-20T01:03:40.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2vkRpRUOOq+s/XUHCrJHn4MA7VI='} 2024-04-20T01:03:40.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e621aef8424939e0-006623146c-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:40.655 DEBUG:boto:path=/aktlji-74/ 2024-04-20T01:03:40.655 DEBUG:boto:auth_path=/aktlji-74/ 2024-04-20T01:03:40.655 DEBUG:boto:Method: HEAD 2024-04-20T01:03:40.655 DEBUG:boto:Path: /aktlji-74/ 2024-04-20T01:03:40.655 DEBUG:boto:Data: 2024-04-20T01:03:40.655 DEBUG:boto:Headers: {} 2024-04-20T01:03:40.655 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:03:40.656 DEBUG:boto:Port: 8000 2024-04-20T01:03:40.656 DEBUG:boto:Params: {} 2024-04-20T01:03:40.656 DEBUG:boto:Token: None 2024-04-20T01:03:40.656 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:03:40 GMT /aktlji-74/ 2024-04-20T01:03:40.656 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4= 2024-04-20T01:03:40.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4=', 'Content-Length': '0'} 2024-04-20T01:03:40.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cf768cd7a25d343-006623146c-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:40.659 DEBUG:boto:path=/aktlji-74/obj55 2024-04-20T01:03:40.659 DEBUG:boto:auth_path=/aktlji-74/obj55 2024-04-20T01:03:40.659 DEBUG:boto:Method: PUT 2024-04-20T01:03:40.659 DEBUG:boto:Path: /aktlji-74/obj55 2024-04-20T01:03:40.660 DEBUG:boto:Data: 2024-04-20T01:03:40.660 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:03:40.660 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:03:40.660 DEBUG:boto:Port: 8000 2024-04-20T01:03:40.660 DEBUG:boto:Params: {} 2024-04-20T01:03:40.660 DEBUG:boto:Token: None 2024-04-20T01:03:40.660 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:03:40 GMT /aktlji-74/obj55 2024-04-20T01:03:40.660 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:lF9REuFNEudtFC9RA55Q8kP2KxA= 2024-04-20T01:03:40.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lF9REuFNEudtFC9RA55Q8kP2KxA='} 2024-04-20T01:03:40.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048c0a2ab1aedd7a6-006623146c-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:03:40.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:03:40.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:03:40.666 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-20T01:03:40.727 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:03:40.727 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:03:41.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:03:41.778+0000 7fd370546780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-20T01:03:41.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:03:41.780+0000 7fd370546780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-20T01:03:41.814 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-20T01:03:41.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-20T01:03:41.818 INFO:tasks.rgw.c2.client.0:Started 2024-04-20T01:03:41.818 DEBUG:teuthology.orchestra.run.smithi031:> curl http://smithi031.front.sepia.ceph.com:8000 2024-04-20T01:03:41.844 INFO:teuthology.orchestra.run.smithi031.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T01:03:41.844 INFO:teuthology.orchestra.run.smithi031.stderr: Dload Upload Total Spent Left Speed 2024-04-20T01:03:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-20T01:03:41.846 INFO:teuthology.orchestra.run.smithi031.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi031.front.sepia.ceph.com port 8000: Connection refused 2024-04-20T01:03:42.847 DEBUG:teuthology.orchestra.run.smithi031:> curl http://smithi031.front.sepia.ceph.com:8000 2024-04-20T01:03:42.873 INFO:teuthology.orchestra.run.smithi031.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T01:03:42.875 INFO:teuthology.orchestra.run.smithi031.stderr: Dload Upload Total Spent Left Speed 2024-04-20T01:03:42.875 INFO:teuthology.orchestra.run.smithi031.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-20T01:03:42.876 INFO:teuthology.orchestra.run.smithi031.stdout:anonymous 2024-04-20T01:03:42.876 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-74 2024-04-20T01:03:42.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:03:42.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:03:42.877 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:03:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:03:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:03:43.052 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-74 2024-04-20T01:03:43.088 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-74 2024-04-20T01:03:43.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:03:43.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:03:43.088 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:03:43.155 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:03:43.155 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:03:43.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:43.197+0000 7fbdb4f20780 1 waiting to reach incremental sync.. 2024-04-20T01:03:46.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-20T01:03:48.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:48.199+0000 7fbdb4f20780 1 bucket sync caught up with source: 2024-04-20T01:03:48.200 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.43.6, 00000000002.42.6, 00000000002.64.6] 2024-04-20T01:03:48.201 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.43.6, 00000000002.42.6, 00000000002.64.6] 2024-04-20T01:03:48.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:48.199+0000 7fbdb4f20780 0 bucket checkpoint complete 2024-04-20T01:03:48.218 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-74 2024-04-20T01:03:48.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:03:48.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:03:48.219 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:03:48.283 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:03:48.284 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:03:48.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:48.325+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:03:53.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:53.326+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:03:58.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:03:58.326+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:04:03.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:04:03.327+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:04:08.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:04:08.328+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:04:13.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:04:13.328+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:04:18.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:04:18.329+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:04:23.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:04:23.329+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:04:28.331 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:04:28.330+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:04:33.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:04:33.331+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:04:38.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:04:38.332+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:04:43.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:04:43.332+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:04:48.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:04:48.332+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:04:53.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:04:53.333+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:04:58.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:04:58.333+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:05:03.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:05:03.334+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:05:08.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:05:08.334+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:05:13.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:05:13.334+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:05:18.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:05:18.335+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:05:23.336 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:05:23.335+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:05:28.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:05:28.336+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:05:33.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:05:33.337+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:05:38.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:05:38.337+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:05:43.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:05:43.338+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:05:48.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:05:48.338+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:05:53.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:05:53.339+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:05:58.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:05:58.340+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:06:03.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:06:03.340+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:06:08.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:06:08.341+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:06:13.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:06:13.342+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:06:18.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:06:18.343+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:06:23.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:06:23.344+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:06:28.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:06:28.344+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:06:33.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:06:33.345+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:06:38.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:06:38.346+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:06:43.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:06:43.346+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:06:48.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:06:48.347+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:06:53.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:06:53.346+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:06:58.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:06:58.347+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:07:03.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:07:03.347+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:07:08.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:07:08.348+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:07:13.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:07:13.349+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:07:18.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:07:18.349+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:07:23.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:07:23.350+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:07:28.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:07:28.351+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:07:33.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:07:33.351+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:07:38.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:07:38.352+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:07:43.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:07:43.353+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:07:48.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:07:48.354+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:07:53.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:07:53.354+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:07:58.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:07:58.355+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:08:03.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:03.356+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:08:08.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:08.357+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:08:13.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:13.357+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:08:18.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:18.358+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:08:23.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:23.359+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:08:28.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:28.359+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:08:33.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:33.360+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:08:38.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:38.361+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:08:43.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:43.360+0000 7f0e26770780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:08:43.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:43.360+0000 7f0e26770780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:08:43.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:43.360+0000 7f0e26770780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:08:43.380 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T01:08:43.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... FAIL 2024-04-20T01:08:43.384 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:08:43.384 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:08:43.385 DEBUG:boto:Method: PUT 2024-04-20T01:08:43.385 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:08:43.385 DEBUG:boto:Data: 2024-04-20T01:08:43.385 DEBUG:boto:Headers: {} 2024-04-20T01:08:43.385 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:08:43.385 DEBUG:boto:Port: 8000 2024-04-20T01:08:43.385 DEBUG:boto:Params: {} 2024-04-20T01:08:43.386 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:08:43.386 DEBUG:boto:Token: None 2024-04-20T01:08:43.387 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:08:43 GMT /aktlji-75/ 2024-04-20T01:08:43.387 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EVL4Q1Jwt8qEdV5UXxclgOFVQdI= 2024-04-20T01:08:43.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:08:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EVL4Q1Jwt8qEdV5UXxclgOFVQdI=', 'Content-Length': '0'} 2024-04-20T01:08:43.411 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000099b191ea4b029d1-006623159b-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:08:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:08:43.411 DEBUG:rgw_multi.tests:created bucket=aktlji-75 2024-04-20T01:08:43.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:08:43.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:08:43.412 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:08:43.474 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:08:43.474 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:08:43.511 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:08:43.528 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:08:43.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:08:43.528 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:08:43.528 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:08:43.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:08:43.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:08:43.528 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:08:43.528 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:08:43.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:08:43.528 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:08:43.528 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575323.409191_1143.1", 2024-04-20T01:08:43.529 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:08:43.409191Z" 2024-04-20T01:08:43.529 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:08:43.529 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:08:43.529 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:08:43.529 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:08:43.529 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:08:43.529 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:08:43.530 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:08:43.530 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:08:43.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:08:43.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:08:43.530 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:08:43.595 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:08:43.595 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:08:43.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:08:43.648+0000 7f5e97a47780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:08:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:08:43.649+0000 7f5e97a47780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:08:43.668 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:08:43.668 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:08:43.668 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:08:43.668 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:08:43.668 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:08:43.668 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:08:43.668 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:43.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:43.669 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:08:43.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:08:43.669 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:08:43.669 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:08:43.669 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:08:43.669 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:08:43.669 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:08:43.669 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:08:43.669 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:08:43.669 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:08:43.669 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:43.670 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:43.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:43.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:08:43.670 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:08:43.670 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:08:43.670 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:08:43.670 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:08:43.670 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:08:43.670 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:08:43.670 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:08:43.670 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:08:43.670 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:43.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:43.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:43.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:08:43.671 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:08:43.671 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:08:43.671 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:08:43.671 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575323.401070_1140.1", 2024-04-20T01:08:43.671 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:08:43.671 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:08:43.671 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:08:43.671 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:08:43.401070Z", 2024-04-20T01:08:43.671 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:43.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:08:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:08:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:08:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:08:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:43.673 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:08:43Z" 2024-04-20T01:08:43.673 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:08:43.674 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713575323.401070_1140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:08:43.401070Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:08:43Z" } 2024-04-20T01:08:43.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575323.401070_1140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:08:43.401070Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:08:43.674 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:08:43.674 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.401070_1140.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:08:43.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713575323.409191_1143.1 target=1_1713575323.401070_1140.1 2024-04-20T01:08:48.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:08:48.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:08:48.677 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:08:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:08:48.738 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:08:48.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:08:48.788+0000 7f4ebb339780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:08:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:08:48.789+0000 7f4ebb339780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:08:48.811 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:08:48.811 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:08:48.811 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:08:48.811 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:08:48.811 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:08:48.811 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:08:48.811 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:48.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:48.812 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:08:48.812 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:08:48.812 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:08:48.812 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:08:48.812 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:08:48.812 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:08:48.813 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:08:48.813 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:08:48.813 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:08:48.813 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:08:48.813 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:48.813 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:48.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:48.813 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:08:48.813 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:08:48.814 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:08:48.814 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:08:48.814 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:08:48.814 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:08:48.814 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:08:48.814 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:08:48.814 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:08:48.814 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:48.815 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:48.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:48.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:08:48.815 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:08:48.815 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:08:48.815 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:08:48.815 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575323.401070_1140.1", 2024-04-20T01:08:48.815 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:08:48.816 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:08:48.816 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:08:48.816 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:08:43.401070Z", 2024-04-20T01:08:48.816 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:48.816 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:48.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:48.816 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:08:48.816 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:08:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:08:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:08:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:08:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:08:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:08:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:08:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:08:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:48.818 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:48.818 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:48.818 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:08:48.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:48.818 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:08:48.818 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:08:48.818 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:08:48.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:48.819 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:08:48Z" 2024-04-20T01:08:48.819 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:08:48.820 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713575323.401070_1140.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:08:43.401070Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:08:48Z" } 2024-04-20T01:08:48.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575323.401070_1140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:08:43.401070Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:08:48.821 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:08:48.821 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.401070_1140.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:08:48.821 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713575323.409191_1143.1 target=1_1713575323.401070_1140.1 2024-04-20T01:08:53.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:08:53.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:08:53.822 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:08:53.882 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:08:53.882 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:08:53.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:08:53.930+0000 7f2e9d8ee780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:08:53.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:08:53.931+0000 7f2e9d8ee780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:08:53.951 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:08:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:08:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:08:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:08:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:08:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:08:53.951 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:08:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:08:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:08:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:08:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:08:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:08:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:08:53.952 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:08:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:08:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:08:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:08:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:08:53.953 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:08:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:08:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:08:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:08:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:08:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:08:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:08:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:53.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:08:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:08:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:08:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:08:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575323.409191_1143.1", 2024-04-20T01:08:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:08:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:08:53.955 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:08:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:08:43.409191Z", 2024-04-20T01:08:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:08:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:08:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:08:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:08:53.956 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:08:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:08:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:08:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:08:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:08:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:08:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:08:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:08:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:08:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:08:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:08:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:08:53.958 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:08:53Z" 2024-04-20T01:08:53.958 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:08:53.959 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713575323.409191_1143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:08:43.409191Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:08:53Z" } 2024-04-20T01:08:53.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575323.409191_1143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:08:43.409191Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:08:53.960 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:08:53.960 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:08:53.961 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:08:53.961 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:08:53.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:08:53.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:08:53.961 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:08:54.020 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:08:54.020 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:08:54.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:54.055+0000 7f1fdd42c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:08:54.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:08:54.057+0000 7f1fdd42c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:08:54.072 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:08:54.072 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:08:54.072 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:08:54.072 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:08:54.073 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:08:54.073 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:08:54.073 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:08:54.073 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:08:54.073 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:08:54.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:08:54.073 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:08:54.073 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:08:54.074 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:08:54.074 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:08:54.074 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:08:54.074 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:08:54.074 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:08:54.074 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:08:54.074 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:08:54.074 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:08:54.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:08:54.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:08:54.075 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:08:54.075 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:08:54.075 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:08:54.075 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:08:54.075 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:08:54.075 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:08:54.076 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:08:54.076 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:08:54.076 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:08:54.076 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:08:54.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:08:54.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:08:54.076 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:08:54.076 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:08:54.077 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:08:54.077 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575323.409191_1143.1", 2024-04-20T01:08:54.077 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:08:54.077 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:08:54.077 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:08:54.077 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:08:43.409191Z", 2024-04-20T01:08:54.077 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:08:54.077 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:08:54.077 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:08:54.078 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:08:54.078 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:08:54.078 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:08:54.078 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:08:54.078 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:08:54.078 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:08:54.078 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:08:54.078 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:08:54.079 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:08:54.079 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:08:54.079 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:08:54.079 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:08:54.079 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:08:54.079 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:08:54.079 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:08:54.079 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:08:54.080 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:08:54.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:08:54.080 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:08:54Z" 2024-04-20T01:08:54.080 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:08:54.081 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713575323.409191_1143.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:08:43.409191Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:08:54Z" } 2024-04-20T01:08:54.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575323.409191_1143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:08:43.409191Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:08:54.082 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:08:54.082 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:08:54.082 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:08:54.082 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-20T01:08:54.083 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:08:54.083 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-20T01:08:54.083+0000 7f1203990700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 102262) UID: 0 2024-04-20T01:08:54.084 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-20T01:08:54.083+0000 7f1208b657c0 -1 shutting down 2024-04-20T01:08:59.864 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-20T01:09:00.185 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-20T01:09:00.185 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.185 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.185 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.186 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.186 DEBUG:boto:Data: 2024-04-20T01:09:00.186 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.186 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.186 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.186 DEBUG:boto:Params: {} 2024-04-20T01:09:00.186 DEBUG:boto:Token: None 2024-04-20T01:09:00.186 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.187 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048a6f22aee2f528c-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.190 DEBUG:boto:path=/aktlji-75/obj11 2024-04-20T01:09:00.190 DEBUG:boto:auth_path=/aktlji-75/obj11 2024-04-20T01:09:00.190 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.190 DEBUG:boto:Path: /aktlji-75/obj11 2024-04-20T01:09:00.190 DEBUG:boto:Data: 2024-04-20T01:09:00.190 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.190 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.190 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.190 DEBUG:boto:Params: {} 2024-04-20T01:09:00.191 DEBUG:boto:Token: None 2024-04-20T01:09:00.191 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj11 2024-04-20T01:09:00.191 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WV1BymXys4gH3jySusyNnRgIf/E= 2024-04-20T01:09:00.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WV1BymXys4gH3jySusyNnRgIf/E='} 2024-04-20T01:09:00.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d595c1055f2c6ad-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.199 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.199 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.199 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.199 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.199 DEBUG:boto:Data: 2024-04-20T01:09:00.199 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.199 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.199 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.200 DEBUG:boto:Params: {} 2024-04-20T01:09:00.200 DEBUG:boto:Token: None 2024-04-20T01:09:00.200 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.200 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bb5a7bcd758814b-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.203 DEBUG:boto:path=/aktlji-75/obj22 2024-04-20T01:09:00.203 DEBUG:boto:auth_path=/aktlji-75/obj22 2024-04-20T01:09:00.203 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.203 DEBUG:boto:Path: /aktlji-75/obj22 2024-04-20T01:09:00.203 DEBUG:boto:Data: 2024-04-20T01:09:00.203 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.203 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.203 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.204 DEBUG:boto:Params: {} 2024-04-20T01:09:00.204 DEBUG:boto:Token: None 2024-04-20T01:09:00.204 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj22 2024-04-20T01:09:00.204 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WEK5K+wr/s1DRi/jdrBD0T36gsk= 2024-04-20T01:09:00.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WEK5K+wr/s1DRi/jdrBD0T36gsk='} 2024-04-20T01:09:00.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019ac8039b17b1ac0-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.209 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.209 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.209 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.209 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.209 DEBUG:boto:Data: 2024-04-20T01:09:00.209 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.209 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.209 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.209 DEBUG:boto:Params: {} 2024-04-20T01:09:00.210 DEBUG:boto:Token: None 2024-04-20T01:09:00.210 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.210 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c07b1e72a4458d3-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.212 DEBUG:boto:path=/aktlji-75/obj33 2024-04-20T01:09:00.212 DEBUG:boto:auth_path=/aktlji-75/obj33 2024-04-20T01:09:00.212 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.213 DEBUG:boto:Path: /aktlji-75/obj33 2024-04-20T01:09:00.213 DEBUG:boto:Data: 2024-04-20T01:09:00.213 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.213 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.213 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.213 DEBUG:boto:Params: {} 2024-04-20T01:09:00.213 DEBUG:boto:Token: None 2024-04-20T01:09:00.213 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj33 2024-04-20T01:09:00.213 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PVl+CFd41SJDje+CUqOTygr4QmE= 2024-04-20T01:09:00.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PVl+CFd41SJDje+CUqOTygr4QmE='} 2024-04-20T01:09:00.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e8c04067a89e4f4-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.259 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.259 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.259 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.259 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.259 DEBUG:boto:Data: 2024-04-20T01:09:00.259 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.259 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.259 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.259 DEBUG:boto:Params: {} 2024-04-20T01:09:00.259 DEBUG:boto:Token: None 2024-04-20T01:09:00.260 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.260 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.262 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2b6198fd7bad820-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.263 DEBUG:boto:path=/aktlji-75/obj44 2024-04-20T01:09:00.263 DEBUG:boto:auth_path=/aktlji-75/obj44 2024-04-20T01:09:00.263 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.263 DEBUG:boto:Path: /aktlji-75/obj44 2024-04-20T01:09:00.263 DEBUG:boto:Data: 2024-04-20T01:09:00.263 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.263 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.263 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.263 DEBUG:boto:Params: {} 2024-04-20T01:09:00.264 DEBUG:boto:Token: None 2024-04-20T01:09:00.264 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj44 2024-04-20T01:09:00.264 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:999mwR8QWdn51Bpep/BbK7MNl6U= 2024-04-20T01:09:00.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:999mwR8QWdn51Bpep/BbK7MNl6U='} 2024-04-20T01:09:00.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002efe01e910677bd9-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.270 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.270 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.270 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.270 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.270 DEBUG:boto:Data: 2024-04-20T01:09:00.271 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.271 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.271 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.271 DEBUG:boto:Params: {} 2024-04-20T01:09:00.271 DEBUG:boto:Token: None 2024-04-20T01:09:00.271 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.271 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031cea4ef2422b078-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.273 DEBUG:boto:path=/aktlji-75/obj55 2024-04-20T01:09:00.274 DEBUG:boto:auth_path=/aktlji-75/obj55 2024-04-20T01:09:00.274 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.274 DEBUG:boto:Path: /aktlji-75/obj55 2024-04-20T01:09:00.274 DEBUG:boto:Data: 2024-04-20T01:09:00.274 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.274 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.274 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.274 DEBUG:boto:Params: {} 2024-04-20T01:09:00.274 DEBUG:boto:Token: None 2024-04-20T01:09:00.274 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj55 2024-04-20T01:09:00.275 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/lIqI/fALGpkKGzn3xvboh+duL0= 2024-04-20T01:09:00.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/lIqI/fALGpkKGzn3xvboh+duL0='} 2024-04-20T01:09:00.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066911843e55d3f75-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:00.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:00.319 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-75 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:09:00.382 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:09:00.382 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:09:00.434 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T01:09:00.434 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-75 2024-04-20T01:09:00.439 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 5 2024-04-20T01:09:00.486 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.486 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.486 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.486 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.487 DEBUG:boto:Data: 2024-04-20T01:09:00.487 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.487 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.487 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.487 DEBUG:boto:Params: {} 2024-04-20T01:09:00.487 DEBUG:boto:Token: None 2024-04-20T01:09:00.487 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.487 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009fb81f61634beff-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.490 DEBUG:boto:path=/aktlji-75/obj17 2024-04-20T01:09:00.491 DEBUG:boto:auth_path=/aktlji-75/obj17 2024-04-20T01:09:00.491 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.491 DEBUG:boto:Path: /aktlji-75/obj17 2024-04-20T01:09:00.491 DEBUG:boto:Data: 2024-04-20T01:09:00.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.491 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.491 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.491 DEBUG:boto:Params: {} 2024-04-20T01:09:00.491 DEBUG:boto:Token: None 2024-04-20T01:09:00.491 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj17 2024-04-20T01:09:00.492 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xbDQZg76Np/yYNqSn1lsRooNno8= 2024-04-20T01:09:00.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xbDQZg76Np/yYNqSn1lsRooNno8='} 2024-04-20T01:09:00.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000329cbbf6557e2ff7-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.498 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.498 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.498 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.498 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.498 DEBUG:boto:Data: 2024-04-20T01:09:00.498 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.498 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.499 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.499 DEBUG:boto:Params: {} 2024-04-20T01:09:00.499 DEBUG:boto:Token: None 2024-04-20T01:09:00.499 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.499 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.502 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001017aab0432ffaad-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.502 DEBUG:boto:path=/aktlji-75/obj34 2024-04-20T01:09:00.502 DEBUG:boto:auth_path=/aktlji-75/obj34 2024-04-20T01:09:00.502 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.503 DEBUG:boto:Path: /aktlji-75/obj34 2024-04-20T01:09:00.503 DEBUG:boto:Data: 2024-04-20T01:09:00.503 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.503 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.503 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.503 DEBUG:boto:Params: {} 2024-04-20T01:09:00.503 DEBUG:boto:Token: None 2024-04-20T01:09:00.503 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj34 2024-04-20T01:09:00.503 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a7uKIATiH145ODg/4rnFX+1HoCU= 2024-04-20T01:09:00.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a7uKIATiH145ODg/4rnFX+1HoCU='} 2024-04-20T01:09:00.509 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e49d47cc1137e19a-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.509 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.509 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.509 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.510 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.510 DEBUG:boto:Data: 2024-04-20T01:09:00.510 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.510 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.510 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.510 DEBUG:boto:Params: {} 2024-04-20T01:09:00.510 DEBUG:boto:Token: None 2024-04-20T01:09:00.510 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.510 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073b879c059a27216-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.513 DEBUG:boto:path=/aktlji-75/obj51 2024-04-20T01:09:00.513 DEBUG:boto:auth_path=/aktlji-75/obj51 2024-04-20T01:09:00.513 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.513 DEBUG:boto:Path: /aktlji-75/obj51 2024-04-20T01:09:00.513 DEBUG:boto:Data: 2024-04-20T01:09:00.514 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.514 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.514 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.514 DEBUG:boto:Params: {} 2024-04-20T01:09:00.514 DEBUG:boto:Token: None 2024-04-20T01:09:00.514 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj51 2024-04-20T01:09:00.514 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gKXAVTODbSpVj0GNWC45VHZ1134= 2024-04-20T01:09:00.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gKXAVTODbSpVj0GNWC45VHZ1134='} 2024-04-20T01:09:00.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df95ed9cd9f17c36-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.559 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.559 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.559 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.559 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.559 DEBUG:boto:Data: 2024-04-20T01:09:00.559 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.559 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.559 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.559 DEBUG:boto:Params: {} 2024-04-20T01:09:00.559 DEBUG:boto:Token: None 2024-04-20T01:09:00.559 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.560 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.562 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025d74bfe9c1fdd2d-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.562 DEBUG:boto:path=/aktlji-75/obj68 2024-04-20T01:09:00.563 DEBUG:boto:auth_path=/aktlji-75/obj68 2024-04-20T01:09:00.563 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.563 DEBUG:boto:Path: /aktlji-75/obj68 2024-04-20T01:09:00.563 DEBUG:boto:Data: 2024-04-20T01:09:00.563 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.563 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.563 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.563 DEBUG:boto:Params: {} 2024-04-20T01:09:00.563 DEBUG:boto:Token: None 2024-04-20T01:09:00.563 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj68 2024-04-20T01:09:00.564 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:whdlUghEvMJC16mi2NXeSkRhSu4= 2024-04-20T01:09:00.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:whdlUghEvMJC16mi2NXeSkRhSu4='} 2024-04-20T01:09:00.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc69b8698ff05b18-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.570 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.570 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.570 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.570 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.570 DEBUG:boto:Data: 2024-04-20T01:09:00.570 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.570 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.570 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.570 DEBUG:boto:Params: {} 2024-04-20T01:09:00.570 DEBUG:boto:Token: None 2024-04-20T01:09:00.571 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.571 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ab19876b85b64d7-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.574 DEBUG:boto:path=/aktlji-75/obj85 2024-04-20T01:09:00.574 DEBUG:boto:auth_path=/aktlji-75/obj85 2024-04-20T01:09:00.574 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.574 DEBUG:boto:Path: /aktlji-75/obj85 2024-04-20T01:09:00.574 DEBUG:boto:Data: 2024-04-20T01:09:00.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.574 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.574 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.574 DEBUG:boto:Params: {} 2024-04-20T01:09:00.574 DEBUG:boto:Token: None 2024-04-20T01:09:00.574 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj85 2024-04-20T01:09:00.575 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Uu4qftHaZwCs99H9OIiKSTbUC+8= 2024-04-20T01:09:00.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Uu4qftHaZwCs99H9OIiKSTbUC+8='} 2024-04-20T01:09:00.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e460cb838c287f2-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:00.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:00.619 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-75 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:09:00.681 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:09:00.681 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:09:00.734 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T01:09:00.734 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-75 2024-04-20T01:09:00.742 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 10 2024-04-20T01:09:00.797 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.797 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.797 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.797 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.797 DEBUG:boto:Data: 2024-04-20T01:09:00.797 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.797 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.797 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.797 DEBUG:boto:Params: {} 2024-04-20T01:09:00.797 DEBUG:boto:Token: None 2024-04-20T01:09:00.798 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.798 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.801 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007664418d86881d15-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.801 DEBUG:boto:path=/aktlji-75/obj19 2024-04-20T01:09:00.801 DEBUG:boto:auth_path=/aktlji-75/obj19 2024-04-20T01:09:00.801 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.802 DEBUG:boto:Path: /aktlji-75/obj19 2024-04-20T01:09:00.802 DEBUG:boto:Data: 2024-04-20T01:09:00.802 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.802 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.802 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.802 DEBUG:boto:Params: {} 2024-04-20T01:09:00.802 DEBUG:boto:Token: None 2024-04-20T01:09:00.802 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj19 2024-04-20T01:09:00.802 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6uFT/n0xpqL0LaY8AtJSvhPKCBI= 2024-04-20T01:09:00.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6uFT/n0xpqL0LaY8AtJSvhPKCBI='} 2024-04-20T01:09:00.808 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efbbb305d8e199c6-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.808 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.808 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.808 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.809 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.809 DEBUG:boto:Data: 2024-04-20T01:09:00.809 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.809 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.809 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.809 DEBUG:boto:Params: {} 2024-04-20T01:09:00.809 DEBUG:boto:Token: None 2024-04-20T01:09:00.809 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.809 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089f2ab260dbc4406-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.812 DEBUG:boto:path=/aktlji-75/obj38 2024-04-20T01:09:00.813 DEBUG:boto:auth_path=/aktlji-75/obj38 2024-04-20T01:09:00.813 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.813 DEBUG:boto:Path: /aktlji-75/obj38 2024-04-20T01:09:00.813 DEBUG:boto:Data: 2024-04-20T01:09:00.813 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.813 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.813 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.813 DEBUG:boto:Params: {} 2024-04-20T01:09:00.813 DEBUG:boto:Token: None 2024-04-20T01:09:00.813 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj38 2024-04-20T01:09:00.813 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:yhenCCdq2XqZDnL2IL+J5S8wleU= 2024-04-20T01:09:00.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yhenCCdq2XqZDnL2IL+J5S8wleU='} 2024-04-20T01:09:00.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c43602ffdb28f61a-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.818 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.818 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.818 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.818 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.818 DEBUG:boto:Data: 2024-04-20T01:09:00.818 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.818 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.818 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.818 DEBUG:boto:Params: {} 2024-04-20T01:09:00.818 DEBUG:boto:Token: None 2024-04-20T01:09:00.818 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.819 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f236d88c94ae48b2-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.821 DEBUG:boto:path=/aktlji-75/obj57 2024-04-20T01:09:00.822 DEBUG:boto:auth_path=/aktlji-75/obj57 2024-04-20T01:09:00.822 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.822 DEBUG:boto:Path: /aktlji-75/obj57 2024-04-20T01:09:00.822 DEBUG:boto:Data: 2024-04-20T01:09:00.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.822 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.822 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.822 DEBUG:boto:Params: {} 2024-04-20T01:09:00.822 DEBUG:boto:Token: None 2024-04-20T01:09:00.822 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj57 2024-04-20T01:09:00.823 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CzUv7i/o/wczp+Hel/XILDmmRw0= 2024-04-20T01:09:00.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CzUv7i/o/wczp+Hel/XILDmmRw0='} 2024-04-20T01:09:00.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7bc4f4d79076526-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.867 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.867 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.867 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.867 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.867 DEBUG:boto:Data: 2024-04-20T01:09:00.867 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.867 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.867 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.867 DEBUG:boto:Params: {} 2024-04-20T01:09:00.867 DEBUG:boto:Token: None 2024-04-20T01:09:00.868 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.868 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.870 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c364915ef18c78a-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.871 DEBUG:boto:path=/aktlji-75/obj76 2024-04-20T01:09:00.871 DEBUG:boto:auth_path=/aktlji-75/obj76 2024-04-20T01:09:00.871 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.871 DEBUG:boto:Path: /aktlji-75/obj76 2024-04-20T01:09:00.871 DEBUG:boto:Data: 2024-04-20T01:09:00.871 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.871 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.871 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.871 DEBUG:boto:Params: {} 2024-04-20T01:09:00.871 DEBUG:boto:Token: None 2024-04-20T01:09:00.872 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj76 2024-04-20T01:09:00.872 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GktW9uRGWuuCa9+BvcpKabYqKbY= 2024-04-20T01:09:00.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GktW9uRGWuuCa9+BvcpKabYqKbY='} 2024-04-20T01:09:00.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000757a555d0f0d04a0-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.878 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:00.878 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:00.878 DEBUG:boto:Method: HEAD 2024-04-20T01:09:00.878 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:00.879 DEBUG:boto:Data: 2024-04-20T01:09:00.879 DEBUG:boto:Headers: {} 2024-04-20T01:09:00.879 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.879 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.879 DEBUG:boto:Params: {} 2024-04-20T01:09:00.879 DEBUG:boto:Token: None 2024-04-20T01:09:00.879 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/ 2024-04-20T01:09:00.879 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:09:00.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:09:00.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c57ea330e4320a5-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.882 DEBUG:boto:path=/aktlji-75/obj95 2024-04-20T01:09:00.883 DEBUG:boto:auth_path=/aktlji-75/obj95 2024-04-20T01:09:00.883 DEBUG:boto:Method: PUT 2024-04-20T01:09:00.883 DEBUG:boto:Path: /aktlji-75/obj95 2024-04-20T01:09:00.883 DEBUG:boto:Data: 2024-04-20T01:09:00.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:00.883 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:00.883 DEBUG:boto:Port: 8000 2024-04-20T01:09:00.883 DEBUG:boto:Params: {} 2024-04-20T01:09:00.883 DEBUG:boto:Token: None 2024-04-20T01:09:00.883 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:00 GMT /aktlji-75/obj95 2024-04-20T01:09:00.884 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5IsZ4qjDDDYXAnIBihP3vJEKO9c= 2024-04-20T01:09:00.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5IsZ4qjDDDYXAnIBihP3vJEKO9c='} 2024-04-20T01:09:00.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dac7a58be76fef5e-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:00.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:00.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:00.927 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-75 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:09:00.991 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:09:00.991 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:09:01.046 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T01:09:01.046 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-75 2024-04-20T01:09:01.054 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 15 2024-04-20T01:09:01.113 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.114 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.114 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.114 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.114 DEBUG:boto:Data: 2024-04-20T01:09:01.114 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.114 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.114 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.114 DEBUG:boto:Params: {} 2024-04-20T01:09:01.114 DEBUG:boto:Token: None 2024-04-20T01:09:01.115 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.115 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.118 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009591f2a76b5e3fe1-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.119 DEBUG:boto:path=/aktlji-75/obj23 2024-04-20T01:09:01.119 DEBUG:boto:auth_path=/aktlji-75/obj23 2024-04-20T01:09:01.119 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.119 DEBUG:boto:Path: /aktlji-75/obj23 2024-04-20T01:09:01.119 DEBUG:boto:Data: 2024-04-20T01:09:01.119 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.119 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.119 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.119 DEBUG:boto:Params: {} 2024-04-20T01:09:01.120 DEBUG:boto:Token: None 2024-04-20T01:09:01.120 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj23 2024-04-20T01:09:01.120 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:U+fOwGptBGIrbz7uqrdRADLdpFg= 2024-04-20T01:09:01.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:U+fOwGptBGIrbz7uqrdRADLdpFg='} 2024-04-20T01:09:01.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001de860ac0ea497b1-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.126 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.126 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.126 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.126 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.127 DEBUG:boto:Data: 2024-04-20T01:09:01.127 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.127 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.127 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.127 DEBUG:boto:Params: {} 2024-04-20T01:09:01.127 DEBUG:boto:Token: None 2024-04-20T01:09:01.127 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.127 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001d40b3e890c7f17-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.131 DEBUG:boto:path=/aktlji-75/obj46 2024-04-20T01:09:01.131 DEBUG:boto:auth_path=/aktlji-75/obj46 2024-04-20T01:09:01.131 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.131 DEBUG:boto:Path: /aktlji-75/obj46 2024-04-20T01:09:01.131 DEBUG:boto:Data: 2024-04-20T01:09:01.131 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.131 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.131 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.132 DEBUG:boto:Params: {} 2024-04-20T01:09:01.132 DEBUG:boto:Token: None 2024-04-20T01:09:01.132 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj46 2024-04-20T01:09:01.132 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ny2fjq+NelNW0gyJrl6VB0mDR00= 2024-04-20T01:09:01.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ny2fjq+NelNW0gyJrl6VB0mDR00='} 2024-04-20T01:09:01.175 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003cfae39382fb04fb-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.175 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.175 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.175 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.175 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.175 DEBUG:boto:Data: 2024-04-20T01:09:01.175 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.175 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.176 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.176 DEBUG:boto:Params: {} 2024-04-20T01:09:01.176 DEBUG:boto:Token: None 2024-04-20T01:09:01.176 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.176 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000864cddb5a7ae4c43-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.180 DEBUG:boto:path=/aktlji-75/obj69 2024-04-20T01:09:01.180 DEBUG:boto:auth_path=/aktlji-75/obj69 2024-04-20T01:09:01.180 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.180 DEBUG:boto:Path: /aktlji-75/obj69 2024-04-20T01:09:01.180 DEBUG:boto:Data: 2024-04-20T01:09:01.180 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.180 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.180 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.180 DEBUG:boto:Params: {} 2024-04-20T01:09:01.181 DEBUG:boto:Token: None 2024-04-20T01:09:01.181 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj69 2024-04-20T01:09:01.181 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QeVwxr5KRg0/WmLykzU7uddMwz4= 2024-04-20T01:09:01.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QeVwxr5KRg0/WmLykzU7uddMwz4='} 2024-04-20T01:09:01.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a92295291655d387-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.188 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.188 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.188 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.188 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.188 DEBUG:boto:Data: 2024-04-20T01:09:01.188 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.189 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.189 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.189 DEBUG:boto:Params: {} 2024-04-20T01:09:01.189 DEBUG:boto:Token: None 2024-04-20T01:09:01.189 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.189 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.192 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086ea014fbe526568-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.192 DEBUG:boto:path=/aktlji-75/obj92 2024-04-20T01:09:01.193 DEBUG:boto:auth_path=/aktlji-75/obj92 2024-04-20T01:09:01.193 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.193 DEBUG:boto:Path: /aktlji-75/obj92 2024-04-20T01:09:01.193 DEBUG:boto:Data: 2024-04-20T01:09:01.193 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.193 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.193 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.193 DEBUG:boto:Params: {} 2024-04-20T01:09:01.194 DEBUG:boto:Token: None 2024-04-20T01:09:01.194 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj92 2024-04-20T01:09:01.194 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:5zMtUnpe0syEnLX5ZILWwieZg6w= 2024-04-20T01:09:01.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5zMtUnpe0syEnLX5ZILWwieZg6w='} 2024-04-20T01:09:01.239 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008bada654bc51acbd-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.239 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.239 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.239 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.239 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.239 DEBUG:boto:Data: 2024-04-20T01:09:01.239 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.240 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.240 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.240 DEBUG:boto:Params: {} 2024-04-20T01:09:01.240 DEBUG:boto:Token: None 2024-04-20T01:09:01.240 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.240 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.244 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000427e2511c56ff65b-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.244 DEBUG:boto:path=/aktlji-75/obj115 2024-04-20T01:09:01.244 DEBUG:boto:auth_path=/aktlji-75/obj115 2024-04-20T01:09:01.245 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.245 DEBUG:boto:Path: /aktlji-75/obj115 2024-04-20T01:09:01.245 DEBUG:boto:Data: 2024-04-20T01:09:01.245 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.245 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.245 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.245 DEBUG:boto:Params: {} 2024-04-20T01:09:01.245 DEBUG:boto:Token: None 2024-04-20T01:09:01.246 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj115 2024-04-20T01:09:01.246 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PRJ8J2IfWSiu9tPEt07LQX7JUdU= 2024-04-20T01:09:01.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PRJ8J2IfWSiu9tPEt07LQX7JUdU='} 2024-04-20T01:09:01.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007783402eb5ca5d6b-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:01.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:01.254 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-75 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:09:01.317 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:09:01.317 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:09:01.372 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T01:09:01.372 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-75 2024-04-20T01:09:01.383 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 20 2024-04-20T01:09:01.447 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.447 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.448 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.448 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.448 DEBUG:boto:Data: 2024-04-20T01:09:01.448 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.448 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.448 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.448 DEBUG:boto:Params: {} 2024-04-20T01:09:01.448 DEBUG:boto:Token: None 2024-04-20T01:09:01.449 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.449 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.453 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e9b3fc4ca83a595-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.453 DEBUG:boto:path=/aktlji-75/obj29 2024-04-20T01:09:01.454 DEBUG:boto:auth_path=/aktlji-75/obj29 2024-04-20T01:09:01.454 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.454 DEBUG:boto:Path: /aktlji-75/obj29 2024-04-20T01:09:01.454 DEBUG:boto:Data: 2024-04-20T01:09:01.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.454 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.454 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.454 DEBUG:boto:Params: {} 2024-04-20T01:09:01.455 DEBUG:boto:Token: None 2024-04-20T01:09:01.455 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj29 2024-04-20T01:09:01.455 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1LfbGEiCzmYEO+tzx7ajy+jrj+Q= 2024-04-20T01:09:01.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1LfbGEiCzmYEO+tzx7ajy+jrj+Q='} 2024-04-20T01:09:01.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a50808eed0e756e8-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.499 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.499 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.499 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.499 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.499 DEBUG:boto:Data: 2024-04-20T01:09:01.499 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.499 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.500 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.500 DEBUG:boto:Params: {} 2024-04-20T01:09:01.500 DEBUG:boto:Token: None 2024-04-20T01:09:01.500 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.500 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df6a114c5b869b44-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.504 DEBUG:boto:path=/aktlji-75/obj58 2024-04-20T01:09:01.504 DEBUG:boto:auth_path=/aktlji-75/obj58 2024-04-20T01:09:01.504 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.505 DEBUG:boto:Path: /aktlji-75/obj58 2024-04-20T01:09:01.505 DEBUG:boto:Data: 2024-04-20T01:09:01.505 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.505 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.505 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.505 DEBUG:boto:Params: {} 2024-04-20T01:09:01.505 DEBUG:boto:Token: None 2024-04-20T01:09:01.505 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj58 2024-04-20T01:09:01.506 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bb+beoF78n9O0bZp4P4mLHtR7cY= 2024-04-20T01:09:01.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bb+beoF78n9O0bZp4P4mLHtR7cY='} 2024-04-20T01:09:01.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017e29029d743844a-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.513 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.513 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.513 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.513 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.513 DEBUG:boto:Data: 2024-04-20T01:09:01.513 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.513 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.513 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.514 DEBUG:boto:Params: {} 2024-04-20T01:09:01.514 DEBUG:boto:Token: None 2024-04-20T01:09:01.514 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.514 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f45db7fd184172ec-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.518 DEBUG:boto:path=/aktlji-75/obj87 2024-04-20T01:09:01.518 DEBUG:boto:auth_path=/aktlji-75/obj87 2024-04-20T01:09:01.518 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.518 DEBUG:boto:Path: /aktlji-75/obj87 2024-04-20T01:09:01.518 DEBUG:boto:Data: 2024-04-20T01:09:01.519 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.519 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.519 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.519 DEBUG:boto:Params: {} 2024-04-20T01:09:01.519 DEBUG:boto:Token: None 2024-04-20T01:09:01.519 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj87 2024-04-20T01:09:01.519 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:e4I5hXIhSK79sKBNV779ReXqtWc= 2024-04-20T01:09:01.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:e4I5hXIhSK79sKBNV779ReXqtWc='} 2024-04-20T01:09:01.563 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053bba21bb8328c94-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.563 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.563 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.563 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.563 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.563 DEBUG:boto:Data: 2024-04-20T01:09:01.564 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.564 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.564 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.564 DEBUG:boto:Params: {} 2024-04-20T01:09:01.564 DEBUG:boto:Token: None 2024-04-20T01:09:01.564 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.564 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.569 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef4cf2e38c8c67db-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.569 DEBUG:boto:path=/aktlji-75/obj116 2024-04-20T01:09:01.569 DEBUG:boto:auth_path=/aktlji-75/obj116 2024-04-20T01:09:01.569 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.569 DEBUG:boto:Path: /aktlji-75/obj116 2024-04-20T01:09:01.569 DEBUG:boto:Data: 2024-04-20T01:09:01.570 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.570 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.570 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.570 DEBUG:boto:Params: {} 2024-04-20T01:09:01.570 DEBUG:boto:Token: None 2024-04-20T01:09:01.570 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj116 2024-04-20T01:09:01.570 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:104rpxWBODYh6YyInDcK5k6jQzA= 2024-04-20T01:09:01.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:104rpxWBODYh6YyInDcK5k6jQzA='} 2024-04-20T01:09:01.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d15704370a52c41d-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.579 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.579 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.579 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.579 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.579 DEBUG:boto:Data: 2024-04-20T01:09:01.579 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.579 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.580 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.580 DEBUG:boto:Params: {} 2024-04-20T01:09:01.580 DEBUG:boto:Token: None 2024-04-20T01:09:01.580 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.580 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df75eea371381b76-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.585 DEBUG:boto:path=/aktlji-75/obj145 2024-04-20T01:09:01.585 DEBUG:boto:auth_path=/aktlji-75/obj145 2024-04-20T01:09:01.585 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.585 DEBUG:boto:Path: /aktlji-75/obj145 2024-04-20T01:09:01.585 DEBUG:boto:Data: 2024-04-20T01:09:01.585 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.585 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.585 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.586 DEBUG:boto:Params: {} 2024-04-20T01:09:01.586 DEBUG:boto:Token: None 2024-04-20T01:09:01.586 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj145 2024-04-20T01:09:01.586 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nBHNpF+gWIQ5RnzpTZ7evscXCJ0= 2024-04-20T01:09:01.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nBHNpF+gWIQ5RnzpTZ7evscXCJ0='} 2024-04-20T01:09:01.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ea8244a072e1523-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:01.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:01.631 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-75 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:09:01.697 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:09:01.698 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:09:01.755 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T01:09:01.755 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-75 2024-04-20T01:09:01.768 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 25 2024-04-20T01:09:01.839 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.840 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.840 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.840 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.840 DEBUG:boto:Data: 2024-04-20T01:09:01.840 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.840 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.840 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.840 DEBUG:boto:Params: {} 2024-04-20T01:09:01.841 DEBUG:boto:Token: None 2024-04-20T01:09:01.841 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.841 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000354c5342f6ac1455-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.845 DEBUG:boto:path=/aktlji-75/obj31 2024-04-20T01:09:01.845 DEBUG:boto:auth_path=/aktlji-75/obj31 2024-04-20T01:09:01.845 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.845 DEBUG:boto:Path: /aktlji-75/obj31 2024-04-20T01:09:01.845 DEBUG:boto:Data: 2024-04-20T01:09:01.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.846 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.846 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.846 DEBUG:boto:Params: {} 2024-04-20T01:09:01.846 DEBUG:boto:Token: None 2024-04-20T01:09:01.846 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj31 2024-04-20T01:09:01.846 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+C6X35vKwb/d6PoRgEqVDz9Aw3E= 2024-04-20T01:09:01.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+C6X35vKwb/d6PoRgEqVDz9Aw3E='} 2024-04-20T01:09:01.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a4d9a04b0412c34-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.853 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.853 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.853 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.853 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.853 DEBUG:boto:Data: 2024-04-20T01:09:01.853 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.853 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.854 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.854 DEBUG:boto:Params: {} 2024-04-20T01:09:01.854 DEBUG:boto:Token: None 2024-04-20T01:09:01.854 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.854 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000639cc67704e1f1d1-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.858 DEBUG:boto:path=/aktlji-75/obj62 2024-04-20T01:09:01.859 DEBUG:boto:auth_path=/aktlji-75/obj62 2024-04-20T01:09:01.859 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.859 DEBUG:boto:Path: /aktlji-75/obj62 2024-04-20T01:09:01.859 DEBUG:boto:Data: 2024-04-20T01:09:01.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.859 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.859 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.859 DEBUG:boto:Params: {} 2024-04-20T01:09:01.860 DEBUG:boto:Token: None 2024-04-20T01:09:01.860 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj62 2024-04-20T01:09:01.860 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8kWgJT0flyuD/R3qMnilAP488tk= 2024-04-20T01:09:01.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8kWgJT0flyuD/R3qMnilAP488tk='} 2024-04-20T01:09:01.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddae6ed9d7290ce4-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.866 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.866 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.866 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.867 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.867 DEBUG:boto:Data: 2024-04-20T01:09:01.867 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.867 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.867 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.867 DEBUG:boto:Params: {} 2024-04-20T01:09:01.867 DEBUG:boto:Token: None 2024-04-20T01:09:01.868 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.868 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2b227adb1cff4b9-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.873 DEBUG:boto:path=/aktlji-75/obj93 2024-04-20T01:09:01.873 DEBUG:boto:auth_path=/aktlji-75/obj93 2024-04-20T01:09:01.873 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.873 DEBUG:boto:Path: /aktlji-75/obj93 2024-04-20T01:09:01.873 DEBUG:boto:Data: 2024-04-20T01:09:01.873 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.873 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.873 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.874 DEBUG:boto:Params: {} 2024-04-20T01:09:01.874 DEBUG:boto:Token: None 2024-04-20T01:09:01.874 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj93 2024-04-20T01:09:01.874 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kfmnLOJcxx5EwJVcd+dCAJrWo0c= 2024-04-20T01:09:01.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kfmnLOJcxx5EwJVcd+dCAJrWo0c='} 2024-04-20T01:09:01.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce92c16a4f68ccaa-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.919 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.919 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.919 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.919 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.919 DEBUG:boto:Data: 2024-04-20T01:09:01.919 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.920 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.920 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.920 DEBUG:boto:Params: {} 2024-04-20T01:09:01.920 DEBUG:boto:Token: None 2024-04-20T01:09:01.920 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.920 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.924 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000385650e4745a54e-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.924 DEBUG:boto:path=/aktlji-75/obj124 2024-04-20T01:09:01.924 DEBUG:boto:auth_path=/aktlji-75/obj124 2024-04-20T01:09:01.924 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.924 DEBUG:boto:Path: /aktlji-75/obj124 2024-04-20T01:09:01.924 DEBUG:boto:Data: 2024-04-20T01:09:01.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.925 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.925 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.925 DEBUG:boto:Params: {} 2024-04-20T01:09:01.925 DEBUG:boto:Token: None 2024-04-20T01:09:01.925 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj124 2024-04-20T01:09:01.925 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8z6sQGEwY/DKpnd692K39GHxqGQ= 2024-04-20T01:09:01.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8z6sQGEwY/DKpnd692K39GHxqGQ='} 2024-04-20T01:09:01.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1fbd9df934dc0fb-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.932 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:01.933 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:01.933 DEBUG:boto:Method: HEAD 2024-04-20T01:09:01.933 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:01.933 DEBUG:boto:Data: 2024-04-20T01:09:01.933 DEBUG:boto:Headers: {} 2024-04-20T01:09:01.933 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.933 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.933 DEBUG:boto:Params: {} 2024-04-20T01:09:01.933 DEBUG:boto:Token: None 2024-04-20T01:09:01.934 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/ 2024-04-20T01:09:01.934 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:09:01.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:09:01.937 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aef16d5f9cc43cbd-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.938 DEBUG:boto:path=/aktlji-75/obj155 2024-04-20T01:09:01.938 DEBUG:boto:auth_path=/aktlji-75/obj155 2024-04-20T01:09:01.938 DEBUG:boto:Method: PUT 2024-04-20T01:09:01.938 DEBUG:boto:Path: /aktlji-75/obj155 2024-04-20T01:09:01.938 DEBUG:boto:Data: 2024-04-20T01:09:01.938 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:01.938 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:01.938 DEBUG:boto:Port: 8000 2024-04-20T01:09:01.939 DEBUG:boto:Params: {} 2024-04-20T01:09:01.939 DEBUG:boto:Token: None 2024-04-20T01:09:01.939 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:01 GMT /aktlji-75/obj155 2024-04-20T01:09:01.939 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:akifC3vRJ++nWyrP0yYp7Y6CeoE= 2024-04-20T01:09:01.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:akifC3vRJ++nWyrP0yYp7Y6CeoE='} 2024-04-20T01:09:01.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b71853ca5ff28d8-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:01.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:01.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:01.983 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-75 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:09:02.046 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:09:02.046 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:09:02.104 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T01:09:02.105 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-75 2024-04-20T01:09:02.119 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 30 2024-04-20T01:09:02.198 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:02.198 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:02.198 DEBUG:boto:Method: HEAD 2024-04-20T01:09:02.198 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:02.198 DEBUG:boto:Data: 2024-04-20T01:09:02.199 DEBUG:boto:Headers: {} 2024-04-20T01:09:02.199 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:02.199 DEBUG:boto:Port: 8000 2024-04-20T01:09:02.199 DEBUG:boto:Params: {} 2024-04-20T01:09:02.199 DEBUG:boto:Token: None 2024-04-20T01:09:02.199 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:02 GMT /aktlji-75/ 2024-04-20T01:09:02.199 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI= 2024-04-20T01:09:02.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI=', 'Content-Length': '0'} 2024-04-20T01:09:02.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2add120f8461130-00662315ae-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:02.204 DEBUG:boto:path=/aktlji-75/obj37 2024-04-20T01:09:02.204 DEBUG:boto:auth_path=/aktlji-75/obj37 2024-04-20T01:09:02.205 DEBUG:boto:Method: PUT 2024-04-20T01:09:02.205 DEBUG:boto:Path: /aktlji-75/obj37 2024-04-20T01:09:02.205 DEBUG:boto:Data: 2024-04-20T01:09:02.205 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:02.205 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:02.205 DEBUG:boto:Port: 8000 2024-04-20T01:09:02.205 DEBUG:boto:Params: {} 2024-04-20T01:09:02.205 DEBUG:boto:Token: None 2024-04-20T01:09:02.206 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:02 GMT /aktlji-75/obj37 2024-04-20T01:09:02.206 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cRgsQzUWjbdbqbpRsnZAQLBJ75g= 2024-04-20T01:09:02.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cRgsQzUWjbdbqbpRsnZAQLBJ75g='} 2024-04-20T01:09:02.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8c1649ebe67f232-00662315ae-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:02.212 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:02.212 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:02.213 DEBUG:boto:Method: HEAD 2024-04-20T01:09:02.213 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:02.213 DEBUG:boto:Data: 2024-04-20T01:09:02.213 DEBUG:boto:Headers: {} 2024-04-20T01:09:02.213 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:02.213 DEBUG:boto:Port: 8000 2024-04-20T01:09:02.213 DEBUG:boto:Params: {} 2024-04-20T01:09:02.213 DEBUG:boto:Token: None 2024-04-20T01:09:02.214 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:02 GMT /aktlji-75/ 2024-04-20T01:09:02.214 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI= 2024-04-20T01:09:02.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI=', 'Content-Length': '0'} 2024-04-20T01:09:02.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3d8d8e833b1167f-00662315ae-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:02.218 DEBUG:boto:path=/aktlji-75/obj74 2024-04-20T01:09:02.219 DEBUG:boto:auth_path=/aktlji-75/obj74 2024-04-20T01:09:02.219 DEBUG:boto:Method: PUT 2024-04-20T01:09:02.219 DEBUG:boto:Path: /aktlji-75/obj74 2024-04-20T01:09:02.219 DEBUG:boto:Data: 2024-04-20T01:09:02.219 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:02.219 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:02.219 DEBUG:boto:Port: 8000 2024-04-20T01:09:02.219 DEBUG:boto:Params: {} 2024-04-20T01:09:02.220 DEBUG:boto:Token: None 2024-04-20T01:09:02.220 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:02 GMT /aktlji-75/obj74 2024-04-20T01:09:02.220 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:anNSQSPZMuvxPB/kJA+vXlyTkAw= 2024-04-20T01:09:02.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:anNSQSPZMuvxPB/kJA+vXlyTkAw='} 2024-04-20T01:09:02.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edf2f2641a47031d-00662315ae-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:02.226 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:02.227 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:02.227 DEBUG:boto:Method: HEAD 2024-04-20T01:09:02.227 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:02.227 DEBUG:boto:Data: 2024-04-20T01:09:02.227 DEBUG:boto:Headers: {} 2024-04-20T01:09:02.227 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:02.227 DEBUG:boto:Port: 8000 2024-04-20T01:09:02.227 DEBUG:boto:Params: {} 2024-04-20T01:09:02.227 DEBUG:boto:Token: None 2024-04-20T01:09:02.228 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:02 GMT /aktlji-75/ 2024-04-20T01:09:02.228 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI= 2024-04-20T01:09:02.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI=', 'Content-Length': '0'} 2024-04-20T01:09:02.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f6e9661ecec62b0-00662315ae-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:02.233 DEBUG:boto:path=/aktlji-75/obj111 2024-04-20T01:09:02.233 DEBUG:boto:auth_path=/aktlji-75/obj111 2024-04-20T01:09:02.233 DEBUG:boto:Method: PUT 2024-04-20T01:09:02.233 DEBUG:boto:Path: /aktlji-75/obj111 2024-04-20T01:09:02.233 DEBUG:boto:Data: 2024-04-20T01:09:02.233 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:02.233 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:02.233 DEBUG:boto:Port: 8000 2024-04-20T01:09:02.234 DEBUG:boto:Params: {} 2024-04-20T01:09:02.234 DEBUG:boto:Token: None 2024-04-20T01:09:02.234 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:02 GMT /aktlji-75/obj111 2024-04-20T01:09:02.234 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SzsUBaVw9k+uN2W4d8CM8oM9aXg= 2024-04-20T01:09:02.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SzsUBaVw9k+uN2W4d8CM8oM9aXg='} 2024-04-20T01:09:02.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fcc0d277bb85c5d-00662315ae-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:02.279 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:02.279 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:02.279 DEBUG:boto:Method: HEAD 2024-04-20T01:09:02.279 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:02.279 DEBUG:boto:Data: 2024-04-20T01:09:02.279 DEBUG:boto:Headers: {} 2024-04-20T01:09:02.280 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:02.280 DEBUG:boto:Port: 8000 2024-04-20T01:09:02.280 DEBUG:boto:Params: {} 2024-04-20T01:09:02.280 DEBUG:boto:Token: None 2024-04-20T01:09:02.280 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:02 GMT /aktlji-75/ 2024-04-20T01:09:02.280 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI= 2024-04-20T01:09:02.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI=', 'Content-Length': '0'} 2024-04-20T01:09:02.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019c87e5b0c9d7366-00662315ae-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:02.285 DEBUG:boto:path=/aktlji-75/obj148 2024-04-20T01:09:02.285 DEBUG:boto:auth_path=/aktlji-75/obj148 2024-04-20T01:09:02.285 DEBUG:boto:Method: PUT 2024-04-20T01:09:02.285 DEBUG:boto:Path: /aktlji-75/obj148 2024-04-20T01:09:02.285 DEBUG:boto:Data: 2024-04-20T01:09:02.285 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:02.286 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:02.286 DEBUG:boto:Port: 8000 2024-04-20T01:09:02.286 DEBUG:boto:Params: {} 2024-04-20T01:09:02.286 DEBUG:boto:Token: None 2024-04-20T01:09:02.286 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:02 GMT /aktlji-75/obj148 2024-04-20T01:09:02.286 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:TZaJteZAeZlaItxrkaaLy0mtspA= 2024-04-20T01:09:02.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TZaJteZAeZlaItxrkaaLy0mtspA='} 2024-04-20T01:09:02.293 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c336fb6a908f8503-00662315ae-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:02.293 DEBUG:boto:path=/aktlji-75/ 2024-04-20T01:09:02.294 DEBUG:boto:auth_path=/aktlji-75/ 2024-04-20T01:09:02.294 DEBUG:boto:Method: HEAD 2024-04-20T01:09:02.294 DEBUG:boto:Path: /aktlji-75/ 2024-04-20T01:09:02.294 DEBUG:boto:Data: 2024-04-20T01:09:02.294 DEBUG:boto:Headers: {} 2024-04-20T01:09:02.294 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:02.294 DEBUG:boto:Port: 8000 2024-04-20T01:09:02.294 DEBUG:boto:Params: {} 2024-04-20T01:09:02.295 DEBUG:boto:Token: None 2024-04-20T01:09:02.295 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:09:02 GMT /aktlji-75/ 2024-04-20T01:09:02.295 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI= 2024-04-20T01:09:02.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI=', 'Content-Length': '0'} 2024-04-20T01:09:02.299 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c13850dd2babcd90-00662315ae-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:02.300 DEBUG:boto:path=/aktlji-75/obj185 2024-04-20T01:09:02.300 DEBUG:boto:auth_path=/aktlji-75/obj185 2024-04-20T01:09:02.300 DEBUG:boto:Method: PUT 2024-04-20T01:09:02.300 DEBUG:boto:Path: /aktlji-75/obj185 2024-04-20T01:09:02.300 DEBUG:boto:Data: 2024-04-20T01:09:02.300 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:09:02.300 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:09:02.300 DEBUG:boto:Port: 8000 2024-04-20T01:09:02.301 DEBUG:boto:Params: {} 2024-04-20T01:09:02.301 DEBUG:boto:Token: None 2024-04-20T01:09:02.301 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:09:02 GMT /aktlji-75/obj185 2024-04-20T01:09:02.301 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:KdZdjTqwgqdH8eKertDSZw0+2Uc= 2024-04-20T01:09:02.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KdZdjTqwgqdH8eKertDSZw0+2Uc='} 2024-04-20T01:09:02.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000934c0543faadb449-00662315ae-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:09:02.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:02.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:09:02.348 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-20T01:09:02.408 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:09:02.409 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:09:02.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:09:02.473+0000 7f0c06bee780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-20T01:09:02.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:09:02.475+0000 7f0c06bee780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-20T01:09:02.510 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-20T01:09:02.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-20T01:09:02.513 INFO:tasks.rgw.c2.client.0:Started 2024-04-20T01:09:02.514 DEBUG:teuthology.orchestra.run.smithi031:> curl http://smithi031.front.sepia.ceph.com:8000 2024-04-20T01:09:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-20T01:09:02.555 INFO:teuthology.orchestra.run.smithi031.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T01:09:02.556 INFO:teuthology.orchestra.run.smithi031.stderr: Dload Upload Total Spent Left Speed 2024-04-20T01:09:02.556 INFO:teuthology.orchestra.run.smithi031.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi031.front.sepia.ceph.com port 8000: Connection refused 2024-04-20T01:09:03.557 DEBUG:teuthology.orchestra.run.smithi031:> curl http://smithi031.front.sepia.ceph.com:8000 2024-04-20T01:09:03.583 INFO:teuthology.orchestra.run.smithi031.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T01:09:03.583 INFO:teuthology.orchestra.run.smithi031.stderr: Dload Upload Total Spent Left Speed 2024-04-20T01:09:03.584 INFO:teuthology.orchestra.run.smithi031.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-20T01:09:03.585 INFO:teuthology.orchestra.run.smithi031.stdout:anonymous 2024-04-20T01:09:03.585 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-75 2024-04-20T01:09:03.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:09:03.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:09:03.586 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:09:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:09:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:09:03.753 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-75 2024-04-20T01:09:03.772 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-75 2024-04-20T01:09:03.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:09:03.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:09:03.773 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:09:03.839 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:09:03.839 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:09:03.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:09:03.882+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:09:05.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-20T01:09:08.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:09:08.883+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:09:13.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:09:13.884+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:09:18.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:09:18.885+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:09:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:09:23.886+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:09:28.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:09:28.891+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:09:33.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:09:33.892+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:09:38.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:09:38.893+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:09:43.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:09:43.893+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:09:48.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:09:48.897+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:09:53.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:09:53.898+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:09:58.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:09:58.899+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:10:03.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:10:03.900+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:10:08.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:10:08.902+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:10:13.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:10:13.903+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:10:18.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:10:18.903+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:10:23.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:10:23.904+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:10:28.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:10:28.905+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:10:33.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:10:33.904+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:10:38.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:10:38.905+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:10:43.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:10:43.906+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:10:48.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:10:48.906+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:10:53.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:10:53.907+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:10:58.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:10:58.908+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:11:03.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:03.909+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:11:08.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:08.909+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:11:13.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:13.910+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:11:18.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:18.911+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:11:23.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:23.912+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:11:28.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:28.912+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:11:33.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:33.913+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:11:38.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:38.914+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:11:43.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:43.915+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:11:48.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:48.917+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:11:53.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:53.921+0000 7f95a6056780 1 bucket sync caught up with source: 2024-04-20T01:11:53.921 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , 00000000002.77.6, , , , , , 00000000002.52.6, , , , , 00000000002.64.6, , , , , , , , , , , , , , , 00000000001.81.6, , , , 00000000001.53.6] 2024-04-20T01:11:53.921 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , 00000000002.77.6, , , , , , 00000000002.52.6, , , , , 00000000002.64.6, , , , , , , , , , , , , , , 00000000001.81.6, , , , 00000000001.53.6] 2024-04-20T01:11:53.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:53.921+0000 7f95a6056780 0 bucket checkpoint complete 2024-04-20T01:11:53.939 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-75 2024-04-20T01:11:53.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:11:53.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:11:53.940 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:11:54.005 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:11:54.005 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:11:54.047 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:54.048+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:11:59.048 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:11:59.049+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:12:04.049 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:12:04.049+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:12:09.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:12:09.050+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:12:14.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:12:14.050+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:12:19.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:12:19.051+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:12:24.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:12:24.051+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:12:29.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:12:29.051+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:12:34.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:12:34.052+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:12:39.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:12:39.052+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:12:44.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:12:44.052+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:12:49.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:12:49.053+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:12:54.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:12:54.053+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:12:59.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:12:59.054+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:13:04.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:13:04.055+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:13:09.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:13:09.055+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:13:14.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:13:14.056+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:13:19.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:13:19.057+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:13:24.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:13:24.057+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:13:29.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:13:29.058+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:13:34.059 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:13:34.059+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:13:39.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:13:39.060+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:13:44.060 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:13:44.061+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:13:49.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:13:49.061+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:13:54.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:13:54.062+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:13:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:13:59.062+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:14:04.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:14:04.063+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:14:09.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:14:09.063+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:14:14.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:14:14.063+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:14:19.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:14:19.064+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:14:24.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:14:24.064+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:14:29.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:14:29.065+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:14:34.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:14:34.065+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:14:39.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:14:39.066+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:14:44.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:14:44.066+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:14:49.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:14:49.067+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:14:54.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:14:54.067+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:14:59.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:14:59.068+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:15:04.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:15:04.068+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:15:09.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:15:09.069+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:15:14.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:15:14.069+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:15:19.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:15:19.070+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:15:24.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:15:24.071+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:15:29.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:15:29.072+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:15:34.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:15:34.072+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:15:39.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:15:39.072+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:15:44.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:15:44.073+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:15:49.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:15:49.073+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:15:54.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:15:54.074+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:15:59.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:15:59.074+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:16:04.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:04.074+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:16:09.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:09.074+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:16:14.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:14.075+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:16:19.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:19.076+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:16:24.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:24.076+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:16:29.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:29.077+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:16:34.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:34.077+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:16:39.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:39.078+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:16:44.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:44.078+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:16:49.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:49.079+0000 7f2f97a4a780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:16:49.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:49.079+0000 7f2f97a4a780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:16:49.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:49.079+0000 7f2f97a4a780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:16:49.097 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T01:16:49.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... FAIL 2024-04-20T01:16:49.100 DEBUG:boto:path=/aktlji-76/ 2024-04-20T01:16:49.100 DEBUG:boto:auth_path=/aktlji-76/ 2024-04-20T01:16:49.101 DEBUG:boto:Method: PUT 2024-04-20T01:16:49.101 DEBUG:boto:Path: /aktlji-76/ 2024-04-20T01:16:49.101 DEBUG:boto:Data: 2024-04-20T01:16:49.101 DEBUG:boto:Headers: {} 2024-04-20T01:16:49.101 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:16:49.101 DEBUG:boto:Port: 8000 2024-04-20T01:16:49.101 DEBUG:boto:Params: {} 2024-04-20T01:16:49.102 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:16:49.102 DEBUG:boto:Token: None 2024-04-20T01:16:49.102 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:16:49 GMT /aktlji-76/ 2024-04-20T01:16:49.102 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rh7KF+lmFeIVwr1Gg0fFCQwmMzM= 2024-04-20T01:16:49.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:16:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rh7KF+lmFeIVwr1Gg0fFCQwmMzM=', 'Content-Length': '0'} 2024-04-20T01:16:49.127 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dcf1db16a1098cc5-0066231781-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:16:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:16:49.127 DEBUG:rgw_multi.tests:created bucket=aktlji-76 2024-04-20T01:16:49.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:16:49.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:16:49.128 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:16:49.184 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:16:49.184 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:16:49.235 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:16:49.255 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:16:49.255 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:16:49.255 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:16:49.255 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:16:49.255 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:16:49.256 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:16:49.256 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:16:49.256 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:16:49.256 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:16:49.256 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:16:49.256 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575809.125650_1219.1", 2024-04-20T01:16:49.256 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:16:49.125650Z" 2024-04-20T01:16:49.256 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:16:49.256 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:16:49.256 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:16:49.256 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:16:49.256 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:16:49.257 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:16:49.257 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575809.125650_1219.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:16:49.258 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:16:49.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:16:49.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:16:49.258 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:16:49.320 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:16:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:16:49.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:16:49.372+0000 7f51df366780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:16:49.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:16:49.373+0000 7f51df366780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:16:49.396 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:16:49.396 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:16:49.396 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:16:49.396 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:16:49.396 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:16:49.396 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:16:49.397 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:16:49.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:16:49.397 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:16:49.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:16:49.397 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:16:49.397 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:16:49.397 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:16:49.397 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:16:49.397 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:16:49.397 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:16:49.397 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:16:49.397 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:16:49.398 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:16:49.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:16:49.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:16:49.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:16:49.398 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:16:49.398 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:16:49.398 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:16:49.398 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:16:49.398 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:16:49.398 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:16:49.398 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:16:49.399 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:16:49.399 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:16:49.399 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:16:49.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:16:49.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:16:49.399 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:16:49.399 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:16:49.399 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:16:49.399 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575809.125650_1219.1", 2024-04-20T01:16:49.399 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:16:49.399 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:16:49.400 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:16:49.400 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:16:49.125650Z", 2024-04-20T01:16:49.400 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:16:49.400 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:16:49.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:16:49.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:16:49.400 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:16:49.400 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:16:49.400 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:16:49.400 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:16:49.400 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:16:49.400 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:16:49Z" 2024-04-20T01:16:49.401 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:16:49.402 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713575809.125650_1219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:16:49.125650Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:16:49Z" } 2024-04-20T01:16:49.402 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575809.125650_1219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:16:49.125650Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:16:49.403 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575809.125650_1219.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:16:49.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575809.125650_1219.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:16:49.403 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:16:49.403 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:16:49.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:16:49.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:16:49.403 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:16:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:16:49.463 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:16:49.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:49.507+0000 7f32f3689780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:16:49.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:49.508+0000 7f32f3689780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:16:49.523 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:16:49.523 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:16:49.523 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:16:49.523 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:16:49.523 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:16:49.524 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:16:49.524 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:16:49.524 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:16:49.524 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:16:49.524 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:16:49.524 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:16:49.524 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:16:49.524 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:16:49.524 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:16:49.524 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:16:49.524 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:16:49.524 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:16:49.525 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:16:49.525 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:16:49.525 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:16:49.525 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:16:49.525 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:16:49.525 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:16:49.525 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:16:49.525 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:16:49.525 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:16:49.525 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:16:49.525 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:16:49.525 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:16:49.526 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:16:49.526 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:16:49.526 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:16:49.526 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:16:49.526 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:16:49.526 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:16:49.526 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:16:49.526 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:16:49.526 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575809.125650_1219.1", 2024-04-20T01:16:49.526 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:16:49.527 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:16:49.527 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:16:49.527 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:16:49.125650Z", 2024-04-20T01:16:49.527 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:16:49.527 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:16:49.527 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:16:49.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:16:49.527 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:16:49.527 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:16:49.527 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:16:49.527 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:16:49.527 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:16:49.528 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:16:49.528 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:16:49.528 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:16:49.528 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:16:49.528 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:16:49.528 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:16:49.528 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:16:49.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:16:49.528 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:16:49.528 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:16:49.528 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:16:49.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:16:49.529 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:16:49Z" 2024-04-20T01:16:49.529 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:16:49.529 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713575809.125650_1219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:16:49.125650Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:16:49Z" } 2024-04-20T01:16:49.530 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575809.125650_1219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:16:49.125650Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:16:49.530 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575809.125650_1219.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:16:49.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575809.125650_1219.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:16:49.530 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:16:49.530 DEBUG:boto:path=/aktlji-76/ 2024-04-20T01:16:49.530 DEBUG:boto:auth_path=/aktlji-76/ 2024-04-20T01:16:49.530 DEBUG:boto:Method: HEAD 2024-04-20T01:16:49.530 DEBUG:boto:Path: /aktlji-76/ 2024-04-20T01:16:49.531 DEBUG:boto:Data: 2024-04-20T01:16:49.531 DEBUG:boto:Headers: {} 2024-04-20T01:16:49.531 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:16:49.531 DEBUG:boto:Port: 8000 2024-04-20T01:16:49.531 DEBUG:boto:Params: {} 2024-04-20T01:16:49.531 DEBUG:boto:Token: None 2024-04-20T01:16:49.531 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:16:49 GMT /aktlji-76/ 2024-04-20T01:16:49.531 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ny9xGTiw1HqwFFVcElPpQi77kX8= 2024-04-20T01:16:49.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:16:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ny9xGTiw1HqwFFVcElPpQi77kX8=', 'Content-Length': '0'} 2024-04-20T01:16:49.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041495a2ee85b2680-0066231781-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:16:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:16:49.534 DEBUG:boto:path=/aktlji-76/dummy 2024-04-20T01:16:49.534 DEBUG:boto:auth_path=/aktlji-76/dummy 2024-04-20T01:16:49.534 DEBUG:boto:Method: PUT 2024-04-20T01:16:49.534 DEBUG:boto:Path: /aktlji-76/dummy 2024-04-20T01:16:49.534 DEBUG:boto:Data: 2024-04-20T01:16:49.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:16:49.535 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:16:49.535 DEBUG:boto:Port: 8000 2024-04-20T01:16:49.535 DEBUG:boto:Params: {} 2024-04-20T01:16:49.535 DEBUG:boto:Token: None 2024-04-20T01:16:49.535 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:16:49 GMT /aktlji-76/dummy 2024-04-20T01:16:49.535 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CXEax3Mtb5vqsqbNgyUQctMqzPY= 2024-04-20T01:16:49.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:16:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CXEax3Mtb5vqsqbNgyUQctMqzPY='} 2024-04-20T01:16:49.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000941bccd289db4fcf-0066231781-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:16:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:16:49.542 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-76 2024-04-20T01:16:49.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:16:49.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:16:49.543 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-76 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:16:49.609 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:16:49.609 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:16:49.669 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-76 2024-04-20T01:16:49.693 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-76 2024-04-20T01:16:49.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:16:49.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:16:49.694 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-76 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:16:49.761 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:16:49.761 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:16:49.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:49.804+0000 7f0928a5d780 1 waiting to reach incremental sync.. 2024-04-20T01:16:54.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:54.804+0000 7f0928a5d780 1 waiting to reach incremental sync.. 2024-04-20T01:16:59.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:16:59.805+0000 7f0928a5d780 1 waiting to reach incremental sync.. 2024-04-20T01:17:04.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:04.806+0000 7f0928a5d780 1 waiting to reach incremental sync.. 2024-04-20T01:17:09.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:09.807+0000 7f0928a5d780 1 bucket sync caught up with source: 2024-04-20T01:17:09.809 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , 00000000001.50.6, , , , , ] 2024-04-20T01:17:09.809 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , 00000000001.50.6, , , , , ] 2024-04-20T01:17:09.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:09.807+0000 7f0928a5d780 0 bucket checkpoint complete 2024-04-20T01:17:09.827 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-76 2024-04-20T01:17:09.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:17:09.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:17:09.828 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-76 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:17:09.894 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:17:09.894 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:17:09.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:09.950+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:17:14.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:14.951+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:17:19.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:19.952+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:17:24.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:24.953+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:17:29.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:29.953+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:17:34.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:34.954+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:17:39.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:39.954+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:17:44.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:44.955+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:17:49.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:49.955+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:17:54.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:54.955+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:17:59.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:17:59.956+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:18:04.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:18:04.956+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:18:09.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:18:09.957+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:18:14.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:18:14.957+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:18:19.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:18:19.958+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:18:24.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:18:24.958+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:18:29.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:18:29.959+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:18:34.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:18:34.960+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:18:39.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:18:39.961+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:18:44.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:18:44.961+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:18:49.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:18:49.962+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:18:54.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:18:54.962+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:18:59.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:18:59.963+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:19:04.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:19:04.964+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:19:09.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:19:09.965+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:19:14.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:19:14.966+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:19:19.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:19:19.966+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:19:24.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:19:24.967+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:19:29.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:19:29.967+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:19:34.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:19:34.968+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:19:39.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:19:39.969+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:19:44.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:19:44.968+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:19:49.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:19:49.969+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:19:54.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:19:54.970+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:19:59.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:19:59.970+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:20:04.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:20:04.971+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:20:09.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:20:09.971+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:20:14.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:20:14.972+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:20:19.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:20:19.972+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:20:24.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:20:24.973+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:20:29.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:20:29.974+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:20:34.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:20:34.975+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:20:39.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:20:39.975+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:20:44.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:20:44.976+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:20:49.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:20:49.977+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:20:54.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:20:54.977+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:20:59.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:20:59.978+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:21:04.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:21:04.979+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:21:09.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:21:09.980+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:21:14.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:21:14.980+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:21:19.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:21:19.981+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:21:24.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:21:24.981+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:21:29.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:21:29.982+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:21:34.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:21:34.982+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:21:39.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:21:39.982+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:21:44.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:21:44.983+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:21:49.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:21:49.983+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:21:54.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:21:54.984+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:21:59.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:21:59.985+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:22:04.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:04.985+0000 7fa9579d5780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:22:04.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:04.985+0000 7fa9579d5780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:22:04.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:04.985+0000 7fa9579d5780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:22:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T01:22:05.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... FAIL 2024-04-20T01:22:05.006 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:05.006 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:05.006 DEBUG:boto:Method: PUT 2024-04-20T01:22:05.006 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:05.006 DEBUG:boto:Data: 2024-04-20T01:22:05.006 DEBUG:boto:Headers: {} 2024-04-20T01:22:05.006 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.007 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.007 DEBUG:boto:Params: {} 2024-04-20T01:22:05.007 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:22:05.007 DEBUG:boto:Token: None 2024-04-20T01:22:05.008 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/ 2024-04-20T01:22:05.008 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:W5z/SgdUCd8HfLoin/gHpxcstJw= 2024-04-20T01:22:05.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W5z/SgdUCd8HfLoin/gHpxcstJw=', 'Content-Length': '0'} 2024-04-20T01:22:05.032 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000053bffdba745723e9-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.032 DEBUG:rgw_multi.tests:created bucket=aktlji-77 2024-04-20T01:22:05.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:22:05.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:22:05.033 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:22:05.092 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:22:05.092 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:22:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:22:05.146 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:22:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:22:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:22:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:22:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:22:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:22:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:22:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:22:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:22:05.147 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:22:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576125.034276_1247.1", 2024-04-20T01:22:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:22:05.034276Z" 2024-04-20T01:22:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:22:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:22:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:22:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:22:05.148 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:22:05.148 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:22:05.149 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576125.034276_1247.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:22:05.149 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:22:05.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:22:05.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:22:05.150 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:22:05.241 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:22:05.241 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:22:05.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:22:05.300+0000 7f45e5c28780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:22:05.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:22:05.301+0000 7f45e5c28780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:22:05.319 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:22:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:22:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:22:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:22:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:22:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:22:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:22:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:22:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:22:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:22:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:22:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:22:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:22:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:22:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:22:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:22:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:22:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:22:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:22:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:22:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:22:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:22:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:22:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:22:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:22:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:22:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:22:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:22:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:22:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:22:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:22:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:22:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:22:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:22:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:22:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:22:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:22:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576125.034276_1247.1", 2024-04-20T01:22:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:22:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:22:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:22:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:22:05.034276Z", 2024-04-20T01:22:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:22:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:22:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:22:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:22:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:22:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:22:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:22:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:22:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:22:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:22:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:22:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:22:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:22:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:22:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:22:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:22:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:22:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:22:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:22:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:22:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:22:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:22:05Z" 2024-04-20T01:22:05.326 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:22:05.327 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576125.034276_1247.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:22:05.034276Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:22:05Z" } 2024-04-20T01:22:05.327 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576125.034276_1247.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:22:05.034276Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:22:05.327 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576125.034276_1247.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:22:05.327 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576125.034276_1247.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:22:05.327 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:22:05.327 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:22:05.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:22:05.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:22:05.328 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:22:05.387 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:22:05.387 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:22:05.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:05.425+0000 7f5d37391780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:22:05.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:05.426+0000 7f5d37391780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:22:05.438 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:22:05.438 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:22:05.438 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:22:05.438 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:22:05.439 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:22:05.439 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:22:05.439 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:22:05.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:22:05.439 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:22:05.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:22:05.439 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:22:05.439 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:22:05.439 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:22:05.439 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:22:05.439 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:22:05.440 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:22:05.440 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:22:05.440 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:22:05.440 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:22:05.440 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:22:05.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:22:05.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:22:05.440 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:22:05.440 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:22:05.440 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:22:05.440 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:22:05.441 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:22:05.441 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:22:05.441 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:22:05.441 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:22:05.441 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:22:05.441 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:22:05.441 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:22:05.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:22:05.441 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:22:05.441 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:22:05.441 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:22:05.441 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576125.034276_1247.1", 2024-04-20T01:22:05.442 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:22:05.442 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:22:05.442 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:22:05.442 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:22:05.034276Z", 2024-04-20T01:22:05.442 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:22:05.442 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:22:05.442 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:22:05.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:22:05.442 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:22:05.442 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:22:05.442 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:22:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:22:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:22:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:22:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:22:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:22:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:22:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:22:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:22:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:22:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:22:05.444 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:22:05.444 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:22:05.444 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:22:05.444 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:22:05.444 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:22:05Z" 2024-04-20T01:22:05.444 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:22:05.445 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576125.034276_1247.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:22:05.034276Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:22:05Z" } 2024-04-20T01:22:05.445 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576125.034276_1247.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:22:05.034276Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:22:05.445 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576125.034276_1247.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:22:05.445 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576125.034276_1247.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:22:05.445 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:22:05.446 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:05.446 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:05.446 DEBUG:boto:Method: HEAD 2024-04-20T01:22:05.446 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:05.446 DEBUG:boto:Data: 2024-04-20T01:22:05.446 DEBUG:boto:Headers: {} 2024-04-20T01:22:05.446 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.446 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.446 DEBUG:boto:Params: {} 2024-04-20T01:22:05.447 DEBUG:boto:Token: None 2024-04-20T01:22:05.447 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/ 2024-04-20T01:22:05.447 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:22:05.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:22:05.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007498be42dd850835-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.450 DEBUG:boto:path=/aktlji-77/obj11 2024-04-20T01:22:05.450 DEBUG:boto:auth_path=/aktlji-77/obj11 2024-04-20T01:22:05.450 DEBUG:boto:Method: PUT 2024-04-20T01:22:05.450 DEBUG:boto:Path: /aktlji-77/obj11 2024-04-20T01:22:05.450 DEBUG:boto:Data: 2024-04-20T01:22:05.450 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:05.450 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.451 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.451 DEBUG:boto:Params: {} 2024-04-20T01:22:05.451 DEBUG:boto:Token: None 2024-04-20T01:22:05.451 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/obj11 2024-04-20T01:22:05.451 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ouNMp4NlmUgSOpH6pyAV1mAIueg= 2024-04-20T01:22:05.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ouNMp4NlmUgSOpH6pyAV1mAIueg='} 2024-04-20T01:22:05.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b39e294fdaf223d-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.458 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:05.458 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:05.459 DEBUG:boto:Method: HEAD 2024-04-20T01:22:05.459 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:05.459 DEBUG:boto:Data: 2024-04-20T01:22:05.459 DEBUG:boto:Headers: {} 2024-04-20T01:22:05.459 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.459 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.459 DEBUG:boto:Params: {} 2024-04-20T01:22:05.459 DEBUG:boto:Token: None 2024-04-20T01:22:05.459 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/ 2024-04-20T01:22:05.460 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:22:05.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:22:05.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047acfdb13dd43847-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.462 DEBUG:boto:path=/aktlji-77/obj22 2024-04-20T01:22:05.462 DEBUG:boto:auth_path=/aktlji-77/obj22 2024-04-20T01:22:05.462 DEBUG:boto:Method: PUT 2024-04-20T01:22:05.462 DEBUG:boto:Path: /aktlji-77/obj22 2024-04-20T01:22:05.462 DEBUG:boto:Data: 2024-04-20T01:22:05.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:05.463 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.463 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.463 DEBUG:boto:Params: {} 2024-04-20T01:22:05.463 DEBUG:boto:Token: None 2024-04-20T01:22:05.463 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/obj22 2024-04-20T01:22:05.463 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iTY3mIfQN80DYEoFgKiGYodhGRc= 2024-04-20T01:22:05.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iTY3mIfQN80DYEoFgKiGYodhGRc='} 2024-04-20T01:22:05.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008157ac1bc06c22a9-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.468 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:05.468 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:05.468 DEBUG:boto:Method: HEAD 2024-04-20T01:22:05.468 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:05.468 DEBUG:boto:Data: 2024-04-20T01:22:05.468 DEBUG:boto:Headers: {} 2024-04-20T01:22:05.468 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.468 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.468 DEBUG:boto:Params: {} 2024-04-20T01:22:05.469 DEBUG:boto:Token: None 2024-04-20T01:22:05.469 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/ 2024-04-20T01:22:05.469 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:22:05.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:22:05.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fec0975f3125778d-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.471 DEBUG:boto:path=/aktlji-77/obj33 2024-04-20T01:22:05.471 DEBUG:boto:auth_path=/aktlji-77/obj33 2024-04-20T01:22:05.472 DEBUG:boto:Method: PUT 2024-04-20T01:22:05.472 DEBUG:boto:Path: /aktlji-77/obj33 2024-04-20T01:22:05.472 DEBUG:boto:Data: 2024-04-20T01:22:05.472 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:05.472 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.472 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.472 DEBUG:boto:Params: {} 2024-04-20T01:22:05.472 DEBUG:boto:Token: None 2024-04-20T01:22:05.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/obj33 2024-04-20T01:22:05.472 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ii6UHlN0w6hs323xg5ixAe81K+o= 2024-04-20T01:22:05.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ii6UHlN0w6hs323xg5ixAe81K+o='} 2024-04-20T01:22:05.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fcceb188d8b2917-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.514 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:05.515 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:05.515 DEBUG:boto:Method: HEAD 2024-04-20T01:22:05.515 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:05.515 DEBUG:boto:Data: 2024-04-20T01:22:05.515 DEBUG:boto:Headers: {} 2024-04-20T01:22:05.515 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.515 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.515 DEBUG:boto:Params: {} 2024-04-20T01:22:05.515 DEBUG:boto:Token: None 2024-04-20T01:22:05.515 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/ 2024-04-20T01:22:05.516 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:22:05.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:22:05.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c689bb725ebee116-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.518 DEBUG:boto:path=/aktlji-77/obj44 2024-04-20T01:22:05.518 DEBUG:boto:auth_path=/aktlji-77/obj44 2024-04-20T01:22:05.518 DEBUG:boto:Method: PUT 2024-04-20T01:22:05.518 DEBUG:boto:Path: /aktlji-77/obj44 2024-04-20T01:22:05.519 DEBUG:boto:Data: 2024-04-20T01:22:05.519 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:05.519 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.519 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.519 DEBUG:boto:Params: {} 2024-04-20T01:22:05.519 DEBUG:boto:Token: None 2024-04-20T01:22:05.519 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/obj44 2024-04-20T01:22:05.519 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:T7wR2ecN303BsFjok9mtuWd/1rY= 2024-04-20T01:22:05.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T7wR2ecN303BsFjok9mtuWd/1rY='} 2024-04-20T01:22:05.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001036aee4ab4dcdd4-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.526 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:05.526 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:05.526 DEBUG:boto:Method: HEAD 2024-04-20T01:22:05.526 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:05.526 DEBUG:boto:Data: 2024-04-20T01:22:05.526 DEBUG:boto:Headers: {} 2024-04-20T01:22:05.527 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.527 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.527 DEBUG:boto:Params: {} 2024-04-20T01:22:05.527 DEBUG:boto:Token: None 2024-04-20T01:22:05.527 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/ 2024-04-20T01:22:05.527 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:22:05.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:22:05.529 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af711cf498863d52-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.530 DEBUG:boto:path=/aktlji-77/obj55 2024-04-20T01:22:05.530 DEBUG:boto:auth_path=/aktlji-77/obj55 2024-04-20T01:22:05.530 DEBUG:boto:Method: PUT 2024-04-20T01:22:05.530 DEBUG:boto:Path: /aktlji-77/obj55 2024-04-20T01:22:05.530 DEBUG:boto:Data: 2024-04-20T01:22:05.530 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:05.530 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.531 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.531 DEBUG:boto:Params: {} 2024-04-20T01:22:05.531 DEBUG:boto:Token: None 2024-04-20T01:22:05.531 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/obj55 2024-04-20T01:22:05.531 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:R+ujjuo+5K5FfUbfpPIl0FIwGmM= 2024-04-20T01:22:05.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R+ujjuo+5K5FfUbfpPIl0FIwGmM='} 2024-04-20T01:22:05.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5acb03d421de65a-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:22:05.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:22:05.536 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-77 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:22:05.597 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:22:05.598 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:22:05.659 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T01:22:05.660 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-77 2024-04-20T01:22:05.665 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 5 2024-04-20T01:22:05.716 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:05.716 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:05.717 DEBUG:boto:Method: HEAD 2024-04-20T01:22:05.717 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:05.717 DEBUG:boto:Data: 2024-04-20T01:22:05.717 DEBUG:boto:Headers: {} 2024-04-20T01:22:05.717 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.717 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.717 DEBUG:boto:Params: {} 2024-04-20T01:22:05.717 DEBUG:boto:Token: None 2024-04-20T01:22:05.717 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/ 2024-04-20T01:22:05.718 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:22:05.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:22:05.721 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aafcde9f53067389-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.721 DEBUG:boto:path=/aktlji-77/obj17 2024-04-20T01:22:05.721 DEBUG:boto:auth_path=/aktlji-77/obj17 2024-04-20T01:22:05.721 DEBUG:boto:Method: PUT 2024-04-20T01:22:05.721 DEBUG:boto:Path: /aktlji-77/obj17 2024-04-20T01:22:05.721 DEBUG:boto:Data: 2024-04-20T01:22:05.721 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:05.722 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.722 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.722 DEBUG:boto:Params: {} 2024-04-20T01:22:05.722 DEBUG:boto:Token: None 2024-04-20T01:22:05.722 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/obj17 2024-04-20T01:22:05.722 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:n+4ekqLfX12nxL+Y2AmfFP/9gRU= 2024-04-20T01:22:05.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:n+4ekqLfX12nxL+Y2AmfFP/9gRU='} 2024-04-20T01:22:05.729 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f1d24b7a8378939-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.729 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:05.729 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:05.729 DEBUG:boto:Method: HEAD 2024-04-20T01:22:05.729 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:05.729 DEBUG:boto:Data: 2024-04-20T01:22:05.729 DEBUG:boto:Headers: {} 2024-04-20T01:22:05.729 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.729 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.729 DEBUG:boto:Params: {} 2024-04-20T01:22:05.730 DEBUG:boto:Token: None 2024-04-20T01:22:05.730 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/ 2024-04-20T01:22:05.730 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:22:05.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:22:05.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005d48b13729b75c4-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.733 DEBUG:boto:path=/aktlji-77/obj34 2024-04-20T01:22:05.733 DEBUG:boto:auth_path=/aktlji-77/obj34 2024-04-20T01:22:05.733 DEBUG:boto:Method: PUT 2024-04-20T01:22:05.733 DEBUG:boto:Path: /aktlji-77/obj34 2024-04-20T01:22:05.733 DEBUG:boto:Data: 2024-04-20T01:22:05.733 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:05.734 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.734 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.734 DEBUG:boto:Params: {} 2024-04-20T01:22:05.734 DEBUG:boto:Token: None 2024-04-20T01:22:05.734 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/obj34 2024-04-20T01:22:05.734 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:pFImoluXp0un0ham5wLxZZHa65U= 2024-04-20T01:22:05.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pFImoluXp0un0ham5wLxZZHa65U='} 2024-04-20T01:22:05.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000932ed2420122ef97-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.779 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:05.779 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:05.779 DEBUG:boto:Method: HEAD 2024-04-20T01:22:05.779 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:05.779 DEBUG:boto:Data: 2024-04-20T01:22:05.779 DEBUG:boto:Headers: {} 2024-04-20T01:22:05.779 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.779 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.780 DEBUG:boto:Params: {} 2024-04-20T01:22:05.780 DEBUG:boto:Token: None 2024-04-20T01:22:05.780 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/ 2024-04-20T01:22:05.780 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:22:05.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:22:05.783 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d39ee21979ba20d-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.783 DEBUG:boto:path=/aktlji-77/obj51 2024-04-20T01:22:05.783 DEBUG:boto:auth_path=/aktlji-77/obj51 2024-04-20T01:22:05.784 DEBUG:boto:Method: PUT 2024-04-20T01:22:05.784 DEBUG:boto:Path: /aktlji-77/obj51 2024-04-20T01:22:05.784 DEBUG:boto:Data: 2024-04-20T01:22:05.784 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:05.784 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.784 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.784 DEBUG:boto:Params: {} 2024-04-20T01:22:05.784 DEBUG:boto:Token: None 2024-04-20T01:22:05.784 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/obj51 2024-04-20T01:22:05.784 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2RsQ0TCVn69+zBqMKm+kgA46FDI= 2024-04-20T01:22:05.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2RsQ0TCVn69+zBqMKm+kgA46FDI='} 2024-04-20T01:22:05.791 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a7c6c6a5b71ed59f-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.791 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:05.791 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:05.792 DEBUG:boto:Method: HEAD 2024-04-20T01:22:05.792 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:05.792 DEBUG:boto:Data: 2024-04-20T01:22:05.792 DEBUG:boto:Headers: {} 2024-04-20T01:22:05.792 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.792 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.792 DEBUG:boto:Params: {} 2024-04-20T01:22:05.792 DEBUG:boto:Token: None 2024-04-20T01:22:05.792 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/ 2024-04-20T01:22:05.792 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:22:05.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:22:05.795 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d91d52aec3d61e1-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.795 DEBUG:boto:path=/aktlji-77/obj68 2024-04-20T01:22:05.795 DEBUG:boto:auth_path=/aktlji-77/obj68 2024-04-20T01:22:05.795 DEBUG:boto:Method: PUT 2024-04-20T01:22:05.796 DEBUG:boto:Path: /aktlji-77/obj68 2024-04-20T01:22:05.796 DEBUG:boto:Data: 2024-04-20T01:22:05.796 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:05.796 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.796 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.796 DEBUG:boto:Params: {} 2024-04-20T01:22:05.796 DEBUG:boto:Token: None 2024-04-20T01:22:05.796 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/obj68 2024-04-20T01:22:05.796 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Q9bGpwtlvzHDQLOJfuev9Se2iTw= 2024-04-20T01:22:05.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q9bGpwtlvzHDQLOJfuev9Se2iTw='} 2024-04-20T01:22:05.839 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a1beab8d52ee0e8-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.839 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:05.839 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:05.839 DEBUG:boto:Method: HEAD 2024-04-20T01:22:05.840 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:05.840 DEBUG:boto:Data: 2024-04-20T01:22:05.840 DEBUG:boto:Headers: {} 2024-04-20T01:22:05.840 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.840 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.840 DEBUG:boto:Params: {} 2024-04-20T01:22:05.841 DEBUG:boto:Token: None 2024-04-20T01:22:05.841 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/ 2024-04-20T01:22:05.841 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:22:05.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:22:05.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dedfb2996511a1f8-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.844 DEBUG:boto:path=/aktlji-77/obj85 2024-04-20T01:22:05.844 DEBUG:boto:auth_path=/aktlji-77/obj85 2024-04-20T01:22:05.845 DEBUG:boto:Method: PUT 2024-04-20T01:22:05.845 DEBUG:boto:Path: /aktlji-77/obj85 2024-04-20T01:22:05.845 DEBUG:boto:Data: 2024-04-20T01:22:05.845 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:05.845 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:05.845 DEBUG:boto:Port: 8000 2024-04-20T01:22:05.845 DEBUG:boto:Params: {} 2024-04-20T01:22:05.845 DEBUG:boto:Token: None 2024-04-20T01:22:05.846 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:05 GMT /aktlji-77/obj85 2024-04-20T01:22:05.846 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CkE9SgEa5K2ATNh6mgr+pWhDBNY= 2024-04-20T01:22:05.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CkE9SgEa5K2ATNh6mgr+pWhDBNY='} 2024-04-20T01:22:05.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc68576d5ae91c99-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:05.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:22:05.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:22:05.853 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-77 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:22:05.916 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:22:05.916 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:22:05.974 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T01:22:05.974 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-77 2024-04-20T01:22:05.982 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 10 2024-04-20T01:22:06.057 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.058 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.058 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.058 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.058 DEBUG:boto:Data: 2024-04-20T01:22:06.058 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.058 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.058 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.058 DEBUG:boto:Params: {} 2024-04-20T01:22:06.059 DEBUG:boto:Token: None 2024-04-20T01:22:06.059 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.059 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0d10758f3311b39-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.062 DEBUG:boto:path=/aktlji-77/obj19 2024-04-20T01:22:06.062 DEBUG:boto:auth_path=/aktlji-77/obj19 2024-04-20T01:22:06.062 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.063 DEBUG:boto:Path: /aktlji-77/obj19 2024-04-20T01:22:06.063 DEBUG:boto:Data: 2024-04-20T01:22:06.063 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.063 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.063 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.063 DEBUG:boto:Params: {} 2024-04-20T01:22:06.063 DEBUG:boto:Token: None 2024-04-20T01:22:06.063 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj19 2024-04-20T01:22:06.063 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sYJWYO+YpEssECkjJyXyE6ELfRY= 2024-04-20T01:22:06.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sYJWYO+YpEssECkjJyXyE6ELfRY='} 2024-04-20T01:22:06.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009774f7f4a2b472b8-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.070 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.070 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.070 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.070 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.070 DEBUG:boto:Data: 2024-04-20T01:22:06.070 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.070 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.070 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.071 DEBUG:boto:Params: {} 2024-04-20T01:22:06.071 DEBUG:boto:Token: None 2024-04-20T01:22:06.071 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.071 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.076 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a8945892a453c2e-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.076 DEBUG:boto:path=/aktlji-77/obj38 2024-04-20T01:22:06.076 DEBUG:boto:auth_path=/aktlji-77/obj38 2024-04-20T01:22:06.076 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.076 DEBUG:boto:Path: /aktlji-77/obj38 2024-04-20T01:22:06.076 DEBUG:boto:Data: 2024-04-20T01:22:06.077 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.077 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.077 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.077 DEBUG:boto:Params: {} 2024-04-20T01:22:06.077 DEBUG:boto:Token: None 2024-04-20T01:22:06.077 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj38 2024-04-20T01:22:06.077 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Bk5bcJrPgArD639If8FAWafZqHc= 2024-04-20T01:22:06.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bk5bcJrPgArD639If8FAWafZqHc='} 2024-04-20T01:22:06.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d61c87957da41dd3-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.123 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.123 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.123 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.123 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.123 DEBUG:boto:Data: 2024-04-20T01:22:06.123 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.123 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.123 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.123 DEBUG:boto:Params: {} 2024-04-20T01:22:06.123 DEBUG:boto:Token: None 2024-04-20T01:22:06.124 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.124 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e9445b5f6890c18-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.127 DEBUG:boto:path=/aktlji-77/obj57 2024-04-20T01:22:06.127 DEBUG:boto:auth_path=/aktlji-77/obj57 2024-04-20T01:22:06.127 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.127 DEBUG:boto:Path: /aktlji-77/obj57 2024-04-20T01:22:06.127 DEBUG:boto:Data: 2024-04-20T01:22:06.127 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.127 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.127 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.127 DEBUG:boto:Params: {} 2024-04-20T01:22:06.127 DEBUG:boto:Token: None 2024-04-20T01:22:06.128 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj57 2024-04-20T01:22:06.128 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8+5gxWt5sWXqDbODXZkbuPLrqMQ= 2024-04-20T01:22:06.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8+5gxWt5sWXqDbODXZkbuPLrqMQ='} 2024-04-20T01:22:06.132 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2404ddf6e73abba-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.132 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.132 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.132 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.132 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.132 DEBUG:boto:Data: 2024-04-20T01:22:06.133 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.133 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.133 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.133 DEBUG:boto:Params: {} 2024-04-20T01:22:06.133 DEBUG:boto:Token: None 2024-04-20T01:22:06.133 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.133 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.136 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000648858bad4d2c4c5-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.136 DEBUG:boto:path=/aktlji-77/obj76 2024-04-20T01:22:06.136 DEBUG:boto:auth_path=/aktlji-77/obj76 2024-04-20T01:22:06.136 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.136 DEBUG:boto:Path: /aktlji-77/obj76 2024-04-20T01:22:06.136 DEBUG:boto:Data: 2024-04-20T01:22:06.136 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.137 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.137 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.137 DEBUG:boto:Params: {} 2024-04-20T01:22:06.137 DEBUG:boto:Token: None 2024-04-20T01:22:06.137 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj76 2024-04-20T01:22:06.137 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:x3+7SrPn5/b+ozDyxy7i1aj7nUg= 2024-04-20T01:22:06.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x3+7SrPn5/b+ozDyxy7i1aj7nUg='} 2024-04-20T01:22:06.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000106106536248e116-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.183 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.183 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.183 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.183 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.183 DEBUG:boto:Data: 2024-04-20T01:22:06.183 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.183 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.183 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.184 DEBUG:boto:Params: {} 2024-04-20T01:22:06.184 DEBUG:boto:Token: None 2024-04-20T01:22:06.184 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.184 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006781492c13d25d21-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.187 DEBUG:boto:path=/aktlji-77/obj95 2024-04-20T01:22:06.187 DEBUG:boto:auth_path=/aktlji-77/obj95 2024-04-20T01:22:06.188 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.188 DEBUG:boto:Path: /aktlji-77/obj95 2024-04-20T01:22:06.188 DEBUG:boto:Data: 2024-04-20T01:22:06.188 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.188 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.188 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.188 DEBUG:boto:Params: {} 2024-04-20T01:22:06.188 DEBUG:boto:Token: None 2024-04-20T01:22:06.188 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj95 2024-04-20T01:22:06.188 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3NIucZo6xpHJzK5SySKbO1EIWEA= 2024-04-20T01:22:06.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3NIucZo6xpHJzK5SySKbO1EIWEA='} 2024-04-20T01:22:06.193 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2eefe48653813b8-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:22:06.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:22:06.194 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-77 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:22:06.256 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:22:06.256 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:22:06.312 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T01:22:06.312 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-77 2024-04-20T01:22:06.320 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 15 2024-04-20T01:22:06.388 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.388 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.388 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.388 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.388 DEBUG:boto:Data: 2024-04-20T01:22:06.388 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.388 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.388 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.388 DEBUG:boto:Params: {} 2024-04-20T01:22:06.389 DEBUG:boto:Token: None 2024-04-20T01:22:06.389 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.389 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046c4a1fe13bd1c9b-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.393 DEBUG:boto:path=/aktlji-77/obj23 2024-04-20T01:22:06.393 DEBUG:boto:auth_path=/aktlji-77/obj23 2024-04-20T01:22:06.393 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.393 DEBUG:boto:Path: /aktlji-77/obj23 2024-04-20T01:22:06.393 DEBUG:boto:Data: 2024-04-20T01:22:06.393 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.393 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.393 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.393 DEBUG:boto:Params: {} 2024-04-20T01:22:06.394 DEBUG:boto:Token: None 2024-04-20T01:22:06.394 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj23 2024-04-20T01:22:06.394 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JoY9ZlIk0pAnC/Duzbr2tBLb6Mw= 2024-04-20T01:22:06.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JoY9ZlIk0pAnC/Duzbr2tBLb6Mw='} 2024-04-20T01:22:06.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd2a1b745a3bf01c-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.439 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.439 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.439 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.439 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.439 DEBUG:boto:Data: 2024-04-20T01:22:06.439 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.439 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.439 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.439 DEBUG:boto:Params: {} 2024-04-20T01:22:06.440 DEBUG:boto:Token: None 2024-04-20T01:22:06.440 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.440 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.443 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba7fe11ed92cbc73-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.443 DEBUG:boto:path=/aktlji-77/obj46 2024-04-20T01:22:06.444 DEBUG:boto:auth_path=/aktlji-77/obj46 2024-04-20T01:22:06.444 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.444 DEBUG:boto:Path: /aktlji-77/obj46 2024-04-20T01:22:06.444 DEBUG:boto:Data: 2024-04-20T01:22:06.444 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.444 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.444 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.444 DEBUG:boto:Params: {} 2024-04-20T01:22:06.444 DEBUG:boto:Token: None 2024-04-20T01:22:06.444 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj46 2024-04-20T01:22:06.445 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:68ni2RjKPeecMy//tE4PJHdvp/Q= 2024-04-20T01:22:06.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:68ni2RjKPeecMy//tE4PJHdvp/Q='} 2024-04-20T01:22:06.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008032d50011b2e10a-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.451 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.451 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.451 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.451 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.452 DEBUG:boto:Data: 2024-04-20T01:22:06.452 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.452 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.452 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.452 DEBUG:boto:Params: {} 2024-04-20T01:22:06.452 DEBUG:boto:Token: None 2024-04-20T01:22:06.452 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.452 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.455 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aae973c35dbb3d83-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.456 DEBUG:boto:path=/aktlji-77/obj69 2024-04-20T01:22:06.456 DEBUG:boto:auth_path=/aktlji-77/obj69 2024-04-20T01:22:06.456 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.456 DEBUG:boto:Path: /aktlji-77/obj69 2024-04-20T01:22:06.456 DEBUG:boto:Data: 2024-04-20T01:22:06.456 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.456 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.456 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.456 DEBUG:boto:Params: {} 2024-04-20T01:22:06.456 DEBUG:boto:Token: None 2024-04-20T01:22:06.457 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj69 2024-04-20T01:22:06.457 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eXXWr8Wus+ucvtXw2kw71fBFuCE= 2024-04-20T01:22:06.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eXXWr8Wus+ucvtXw2kw71fBFuCE='} 2024-04-20T01:22:06.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003577b5d2b6a64222-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.499 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.499 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.499 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.499 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.499 DEBUG:boto:Data: 2024-04-20T01:22:06.499 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.499 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.499 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.499 DEBUG:boto:Params: {} 2024-04-20T01:22:06.500 DEBUG:boto:Token: None 2024-04-20T01:22:06.500 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.500 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7d8a7360233c160-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.504 DEBUG:boto:path=/aktlji-77/obj92 2024-04-20T01:22:06.504 DEBUG:boto:auth_path=/aktlji-77/obj92 2024-04-20T01:22:06.504 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.504 DEBUG:boto:Path: /aktlji-77/obj92 2024-04-20T01:22:06.504 DEBUG:boto:Data: 2024-04-20T01:22:06.504 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.504 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.504 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.504 DEBUG:boto:Params: {} 2024-04-20T01:22:06.504 DEBUG:boto:Token: None 2024-04-20T01:22:06.505 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj92 2024-04-20T01:22:06.505 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:egv4Y6GyyEPafTkalrYW3HdfzfQ= 2024-04-20T01:22:06.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:egv4Y6GyyEPafTkalrYW3HdfzfQ='} 2024-04-20T01:22:06.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b4cf12743a756c2-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.512 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.512 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.512 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.512 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.512 DEBUG:boto:Data: 2024-04-20T01:22:06.512 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.512 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.512 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.512 DEBUG:boto:Params: {} 2024-04-20T01:22:06.513 DEBUG:boto:Token: None 2024-04-20T01:22:06.513 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.513 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1a584cd8cd04f4f-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.517 DEBUG:boto:path=/aktlji-77/obj115 2024-04-20T01:22:06.517 DEBUG:boto:auth_path=/aktlji-77/obj115 2024-04-20T01:22:06.517 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.517 DEBUG:boto:Path: /aktlji-77/obj115 2024-04-20T01:22:06.517 DEBUG:boto:Data: 2024-04-20T01:22:06.517 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.517 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.518 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.518 DEBUG:boto:Params: {} 2024-04-20T01:22:06.518 DEBUG:boto:Token: None 2024-04-20T01:22:06.518 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj115 2024-04-20T01:22:06.518 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:dm5QwWzqkVo/Ku/3bInNeQZLoqk= 2024-04-20T01:22:06.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dm5QwWzqkVo/Ku/3bInNeQZLoqk='} 2024-04-20T01:22:06.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cdbf9a508564a01-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:22:06.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:22:06.563 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-77 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:22:06.631 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:22:06.631 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:22:06.693 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T01:22:06.694 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-77 2024-04-20T01:22:06.704 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 20 2024-04-20T01:22:06.789 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.789 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.789 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.789 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.789 DEBUG:boto:Data: 2024-04-20T01:22:06.789 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.789 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.789 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.789 DEBUG:boto:Params: {} 2024-04-20T01:22:06.789 DEBUG:boto:Token: None 2024-04-20T01:22:06.790 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.790 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000633aa38930a40ea0-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.794 DEBUG:boto:path=/aktlji-77/obj29 2024-04-20T01:22:06.795 DEBUG:boto:auth_path=/aktlji-77/obj29 2024-04-20T01:22:06.795 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.795 DEBUG:boto:Path: /aktlji-77/obj29 2024-04-20T01:22:06.795 DEBUG:boto:Data: 2024-04-20T01:22:06.795 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.795 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.795 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.795 DEBUG:boto:Params: {} 2024-04-20T01:22:06.795 DEBUG:boto:Token: None 2024-04-20T01:22:06.796 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj29 2024-04-20T01:22:06.796 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:me/ELblCV3EVtEGcTU/F0kLOkMY= 2024-04-20T01:22:06.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:me/ELblCV3EVtEGcTU/F0kLOkMY='} 2024-04-20T01:22:06.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a19700d2fb2bfd4-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.803 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.803 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.803 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.804 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.804 DEBUG:boto:Data: 2024-04-20T01:22:06.804 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.804 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.804 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.804 DEBUG:boto:Params: {} 2024-04-20T01:22:06.804 DEBUG:boto:Token: None 2024-04-20T01:22:06.804 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.804 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070570624814e2f86-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.809 DEBUG:boto:path=/aktlji-77/obj58 2024-04-20T01:22:06.809 DEBUG:boto:auth_path=/aktlji-77/obj58 2024-04-20T01:22:06.809 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.809 DEBUG:boto:Path: /aktlji-77/obj58 2024-04-20T01:22:06.809 DEBUG:boto:Data: 2024-04-20T01:22:06.809 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.809 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.809 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.809 DEBUG:boto:Params: {} 2024-04-20T01:22:06.809 DEBUG:boto:Token: None 2024-04-20T01:22:06.810 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj58 2024-04-20T01:22:06.810 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:273MvvHPtzZJ+cKSpM2IWi9gxL8= 2024-04-20T01:22:06.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:273MvvHPtzZJ+cKSpM2IWi9gxL8='} 2024-04-20T01:22:06.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009fddde84bf028f5-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.855 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.855 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.855 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.855 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.855 DEBUG:boto:Data: 2024-04-20T01:22:06.855 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.855 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.855 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.856 DEBUG:boto:Params: {} 2024-04-20T01:22:06.856 DEBUG:boto:Token: None 2024-04-20T01:22:06.856 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.856 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ad52cc5f788f277-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.860 DEBUG:boto:path=/aktlji-77/obj87 2024-04-20T01:22:06.860 DEBUG:boto:auth_path=/aktlji-77/obj87 2024-04-20T01:22:06.860 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.860 DEBUG:boto:Path: /aktlji-77/obj87 2024-04-20T01:22:06.860 DEBUG:boto:Data: 2024-04-20T01:22:06.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.861 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.861 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.861 DEBUG:boto:Params: {} 2024-04-20T01:22:06.861 DEBUG:boto:Token: None 2024-04-20T01:22:06.861 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj87 2024-04-20T01:22:06.861 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z2pLyUu1yzS4b38Q90IIX6Wk0hY= 2024-04-20T01:22:06.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z2pLyUu1yzS4b38Q90IIX6Wk0hY='} 2024-04-20T01:22:06.869 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a44d740a20c75e9c-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.869 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.869 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.869 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.869 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.869 DEBUG:boto:Data: 2024-04-20T01:22:06.869 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.869 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.869 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.869 DEBUG:boto:Params: {} 2024-04-20T01:22:06.870 DEBUG:boto:Token: None 2024-04-20T01:22:06.870 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.870 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.873 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cda15536ff443e1-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.873 DEBUG:boto:path=/aktlji-77/obj116 2024-04-20T01:22:06.874 DEBUG:boto:auth_path=/aktlji-77/obj116 2024-04-20T01:22:06.874 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.874 DEBUG:boto:Path: /aktlji-77/obj116 2024-04-20T01:22:06.874 DEBUG:boto:Data: 2024-04-20T01:22:06.874 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.874 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.874 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.874 DEBUG:boto:Params: {} 2024-04-20T01:22:06.874 DEBUG:boto:Token: None 2024-04-20T01:22:06.875 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj116 2024-04-20T01:22:06.875 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cev/2EYsoAificmW9hLNBhLIZAs= 2024-04-20T01:22:06.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cev/2EYsoAificmW9hLNBhLIZAs='} 2024-04-20T01:22:06.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000656eda1f0baa9a3f-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.919 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:06.919 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:06.919 DEBUG:boto:Method: HEAD 2024-04-20T01:22:06.919 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:06.919 DEBUG:boto:Data: 2024-04-20T01:22:06.919 DEBUG:boto:Headers: {} 2024-04-20T01:22:06.919 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.919 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.919 DEBUG:boto:Params: {} 2024-04-20T01:22:06.920 DEBUG:boto:Token: None 2024-04-20T01:22:06.920 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/ 2024-04-20T01:22:06.920 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:22:06.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:22:06.924 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf22fb66a6c53205-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.924 DEBUG:boto:path=/aktlji-77/obj145 2024-04-20T01:22:06.924 DEBUG:boto:auth_path=/aktlji-77/obj145 2024-04-20T01:22:06.924 DEBUG:boto:Method: PUT 2024-04-20T01:22:06.924 DEBUG:boto:Path: /aktlji-77/obj145 2024-04-20T01:22:06.924 DEBUG:boto:Data: 2024-04-20T01:22:06.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:06.925 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:06.925 DEBUG:boto:Port: 8000 2024-04-20T01:22:06.925 DEBUG:boto:Params: {} 2024-04-20T01:22:06.925 DEBUG:boto:Token: None 2024-04-20T01:22:06.925 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:06 GMT /aktlji-77/obj145 2024-04-20T01:22:06.925 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:athWNQtFoCHcCZctAGgoYy+/c3A= 2024-04-20T01:22:06.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:athWNQtFoCHcCZctAGgoYy+/c3A='} 2024-04-20T01:22:06.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c62dba12af93b14-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:06.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:22:06.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:22:06.932 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-77 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:22:06.990 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:22:06.990 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:22:07.071 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T01:22:07.071 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-77 2024-04-20T01:22:07.086 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 25 2024-04-20T01:22:07.174 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:07.174 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:07.174 DEBUG:boto:Method: HEAD 2024-04-20T01:22:07.174 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:07.174 DEBUG:boto:Data: 2024-04-20T01:22:07.174 DEBUG:boto:Headers: {} 2024-04-20T01:22:07.174 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.174 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.175 DEBUG:boto:Params: {} 2024-04-20T01:22:07.175 DEBUG:boto:Token: None 2024-04-20T01:22:07.175 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/ 2024-04-20T01:22:07.175 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:22:07.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:22:07.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050d7828d261b3d46-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.179 DEBUG:boto:path=/aktlji-77/obj31 2024-04-20T01:22:07.179 DEBUG:boto:auth_path=/aktlji-77/obj31 2024-04-20T01:22:07.179 DEBUG:boto:Method: PUT 2024-04-20T01:22:07.179 DEBUG:boto:Path: /aktlji-77/obj31 2024-04-20T01:22:07.180 DEBUG:boto:Data: 2024-04-20T01:22:07.180 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:07.180 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.180 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.180 DEBUG:boto:Params: {} 2024-04-20T01:22:07.180 DEBUG:boto:Token: None 2024-04-20T01:22:07.180 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/obj31 2024-04-20T01:22:07.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:nGMmORNhMjkneyLFVsvyapQj+FA= 2024-04-20T01:22:07.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGMmORNhMjkneyLFVsvyapQj+FA='} 2024-04-20T01:22:07.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002755e69c53d8b2a-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.227 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:07.227 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:07.227 DEBUG:boto:Method: HEAD 2024-04-20T01:22:07.227 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:07.227 DEBUG:boto:Data: 2024-04-20T01:22:07.227 DEBUG:boto:Headers: {} 2024-04-20T01:22:07.227 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.227 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.227 DEBUG:boto:Params: {} 2024-04-20T01:22:07.228 DEBUG:boto:Token: None 2024-04-20T01:22:07.228 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/ 2024-04-20T01:22:07.228 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:22:07.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:22:07.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000749a33231cbf70f6-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.232 DEBUG:boto:path=/aktlji-77/obj62 2024-04-20T01:22:07.232 DEBUG:boto:auth_path=/aktlji-77/obj62 2024-04-20T01:22:07.232 DEBUG:boto:Method: PUT 2024-04-20T01:22:07.232 DEBUG:boto:Path: /aktlji-77/obj62 2024-04-20T01:22:07.232 DEBUG:boto:Data: 2024-04-20T01:22:07.233 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:07.233 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.233 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.233 DEBUG:boto:Params: {} 2024-04-20T01:22:07.233 DEBUG:boto:Token: None 2024-04-20T01:22:07.233 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/obj62 2024-04-20T01:22:07.233 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:0Csf2+CcYtSamZ0s9Fhe8KYPCe4= 2024-04-20T01:22:07.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0Csf2+CcYtSamZ0s9Fhe8KYPCe4='} 2024-04-20T01:22:07.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4341f1e43006059-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.240 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:07.240 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:07.240 DEBUG:boto:Method: HEAD 2024-04-20T01:22:07.241 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:07.241 DEBUG:boto:Data: 2024-04-20T01:22:07.241 DEBUG:boto:Headers: {} 2024-04-20T01:22:07.241 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.241 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.241 DEBUG:boto:Params: {} 2024-04-20T01:22:07.241 DEBUG:boto:Token: None 2024-04-20T01:22:07.241 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/ 2024-04-20T01:22:07.241 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:22:07.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:22:07.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d432265f003dbcd-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.245 DEBUG:boto:path=/aktlji-77/obj93 2024-04-20T01:22:07.245 DEBUG:boto:auth_path=/aktlji-77/obj93 2024-04-20T01:22:07.245 DEBUG:boto:Method: PUT 2024-04-20T01:22:07.245 DEBUG:boto:Path: /aktlji-77/obj93 2024-04-20T01:22:07.246 DEBUG:boto:Data: 2024-04-20T01:22:07.246 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:07.246 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.246 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.246 DEBUG:boto:Params: {} 2024-04-20T01:22:07.246 DEBUG:boto:Token: None 2024-04-20T01:22:07.246 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/obj93 2024-04-20T01:22:07.246 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6ICjO7vvH418IxxsJArrV2pIewY= 2024-04-20T01:22:07.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ICjO7vvH418IxxsJArrV2pIewY='} 2024-04-20T01:22:07.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbeb1c11e156c545-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.291 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:07.291 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:07.291 DEBUG:boto:Method: HEAD 2024-04-20T01:22:07.291 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:07.291 DEBUG:boto:Data: 2024-04-20T01:22:07.291 DEBUG:boto:Headers: {} 2024-04-20T01:22:07.291 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.291 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.291 DEBUG:boto:Params: {} 2024-04-20T01:22:07.291 DEBUG:boto:Token: None 2024-04-20T01:22:07.292 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/ 2024-04-20T01:22:07.292 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:22:07.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:22:07.296 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068968a0d442a4247-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.296 DEBUG:boto:path=/aktlji-77/obj124 2024-04-20T01:22:07.296 DEBUG:boto:auth_path=/aktlji-77/obj124 2024-04-20T01:22:07.296 DEBUG:boto:Method: PUT 2024-04-20T01:22:07.296 DEBUG:boto:Path: /aktlji-77/obj124 2024-04-20T01:22:07.296 DEBUG:boto:Data: 2024-04-20T01:22:07.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:07.297 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.297 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.297 DEBUG:boto:Params: {} 2024-04-20T01:22:07.297 DEBUG:boto:Token: None 2024-04-20T01:22:07.297 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/obj124 2024-04-20T01:22:07.297 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WLPZeRBlQD9/p8WT6WLNV93FHNY= 2024-04-20T01:22:07.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WLPZeRBlQD9/p8WT6WLNV93FHNY='} 2024-04-20T01:22:07.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f504494284be022e-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.304 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:07.304 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:07.304 DEBUG:boto:Method: HEAD 2024-04-20T01:22:07.304 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:07.304 DEBUG:boto:Data: 2024-04-20T01:22:07.304 DEBUG:boto:Headers: {} 2024-04-20T01:22:07.304 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.304 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.304 DEBUG:boto:Params: {} 2024-04-20T01:22:07.304 DEBUG:boto:Token: None 2024-04-20T01:22:07.305 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/ 2024-04-20T01:22:07.305 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:22:07.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:22:07.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c0bc78182814b20-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.308 DEBUG:boto:path=/aktlji-77/obj155 2024-04-20T01:22:07.309 DEBUG:boto:auth_path=/aktlji-77/obj155 2024-04-20T01:22:07.309 DEBUG:boto:Method: PUT 2024-04-20T01:22:07.309 DEBUG:boto:Path: /aktlji-77/obj155 2024-04-20T01:22:07.309 DEBUG:boto:Data: 2024-04-20T01:22:07.309 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:07.309 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.309 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.309 DEBUG:boto:Params: {} 2024-04-20T01:22:07.310 DEBUG:boto:Token: None 2024-04-20T01:22:07.310 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/obj155 2024-04-20T01:22:07.310 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:z83ovdzPz95akeex35wVIFpa2hQ= 2024-04-20T01:22:07.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z83ovdzPz95akeex35wVIFpa2hQ='} 2024-04-20T01:22:07.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd1cd3568a7dbd62-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:22:07.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:22:07.359 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-77 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:22:07.422 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:22:07.422 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:22:07.490 INFO:teuthology.orchestra.run.smithi019.stdout:tenant: 2024-04-20T01:22:07.491 INFO:teuthology.orchestra.run.smithi019.stdout:bucket name: aktlji-77 2024-04-20T01:22:07.504 INFO:teuthology.orchestra.run.smithi019.stdout:total entries: 30 2024-04-20T01:22:07.606 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:07.606 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:07.606 DEBUG:boto:Method: HEAD 2024-04-20T01:22:07.606 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:07.606 DEBUG:boto:Data: 2024-04-20T01:22:07.606 DEBUG:boto:Headers: {} 2024-04-20T01:22:07.607 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.607 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.607 DEBUG:boto:Params: {} 2024-04-20T01:22:07.607 DEBUG:boto:Token: None 2024-04-20T01:22:07.607 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/ 2024-04-20T01:22:07.607 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:22:07.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:22:07.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff9f4f2ae993bbc4-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.611 DEBUG:boto:path=/aktlji-77/obj37 2024-04-20T01:22:07.612 DEBUG:boto:auth_path=/aktlji-77/obj37 2024-04-20T01:22:07.612 DEBUG:boto:Method: PUT 2024-04-20T01:22:07.612 DEBUG:boto:Path: /aktlji-77/obj37 2024-04-20T01:22:07.612 DEBUG:boto:Data: 2024-04-20T01:22:07.612 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:07.612 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.612 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.612 DEBUG:boto:Params: {} 2024-04-20T01:22:07.612 DEBUG:boto:Token: None 2024-04-20T01:22:07.613 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/obj37 2024-04-20T01:22:07.613 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mpyUizAZHDnGRlllniIK8tLIaLU= 2024-04-20T01:22:07.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mpyUizAZHDnGRlllniIK8tLIaLU='} 2024-04-20T01:22:07.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000766c155b0a2d7f04-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.620 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:07.620 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:07.620 DEBUG:boto:Method: HEAD 2024-04-20T01:22:07.620 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:07.620 DEBUG:boto:Data: 2024-04-20T01:22:07.621 DEBUG:boto:Headers: {} 2024-04-20T01:22:07.621 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.621 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.621 DEBUG:boto:Params: {} 2024-04-20T01:22:07.621 DEBUG:boto:Token: None 2024-04-20T01:22:07.621 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/ 2024-04-20T01:22:07.621 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:22:07.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:22:07.625 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b594fe27bb352db-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.626 DEBUG:boto:path=/aktlji-77/obj74 2024-04-20T01:22:07.626 DEBUG:boto:auth_path=/aktlji-77/obj74 2024-04-20T01:22:07.626 DEBUG:boto:Method: PUT 2024-04-20T01:22:07.626 DEBUG:boto:Path: /aktlji-77/obj74 2024-04-20T01:22:07.626 DEBUG:boto:Data: 2024-04-20T01:22:07.626 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:07.626 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.627 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.627 DEBUG:boto:Params: {} 2024-04-20T01:22:07.627 DEBUG:boto:Token: None 2024-04-20T01:22:07.627 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/obj74 2024-04-20T01:22:07.627 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WahL01DHylbkLPNECsYb7lhsP0U= 2024-04-20T01:22:07.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WahL01DHylbkLPNECsYb7lhsP0U='} 2024-04-20T01:22:07.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4fa4b01c4bf2ff0-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.671 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:07.671 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:07.671 DEBUG:boto:Method: HEAD 2024-04-20T01:22:07.671 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:07.671 DEBUG:boto:Data: 2024-04-20T01:22:07.671 DEBUG:boto:Headers: {} 2024-04-20T01:22:07.671 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.671 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.672 DEBUG:boto:Params: {} 2024-04-20T01:22:07.672 DEBUG:boto:Token: None 2024-04-20T01:22:07.672 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/ 2024-04-20T01:22:07.672 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:22:07.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:22:07.677 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fbcc92f07874180-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.677 DEBUG:boto:path=/aktlji-77/obj111 2024-04-20T01:22:07.677 DEBUG:boto:auth_path=/aktlji-77/obj111 2024-04-20T01:22:07.678 DEBUG:boto:Method: PUT 2024-04-20T01:22:07.678 DEBUG:boto:Path: /aktlji-77/obj111 2024-04-20T01:22:07.678 DEBUG:boto:Data: 2024-04-20T01:22:07.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:07.678 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.678 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.678 DEBUG:boto:Params: {} 2024-04-20T01:22:07.678 DEBUG:boto:Token: None 2024-04-20T01:22:07.678 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/obj111 2024-04-20T01:22:07.679 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XFVLOkyXaeNCWt7c3iyza0UoqvI= 2024-04-20T01:22:07.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XFVLOkyXaeNCWt7c3iyza0UoqvI='} 2024-04-20T01:22:07.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0eca5fe5d5c4bcc-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.690 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:07.690 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:07.690 DEBUG:boto:Method: HEAD 2024-04-20T01:22:07.690 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:07.691 DEBUG:boto:Data: 2024-04-20T01:22:07.691 DEBUG:boto:Headers: {} 2024-04-20T01:22:07.691 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.691 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.691 DEBUG:boto:Params: {} 2024-04-20T01:22:07.691 DEBUG:boto:Token: None 2024-04-20T01:22:07.691 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/ 2024-04-20T01:22:07.691 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:22:07.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:22:07.695 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ec46cef8886beff-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.696 DEBUG:boto:path=/aktlji-77/obj148 2024-04-20T01:22:07.696 DEBUG:boto:auth_path=/aktlji-77/obj148 2024-04-20T01:22:07.696 DEBUG:boto:Method: PUT 2024-04-20T01:22:07.696 DEBUG:boto:Path: /aktlji-77/obj148 2024-04-20T01:22:07.696 DEBUG:boto:Data: 2024-04-20T01:22:07.696 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:07.696 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.696 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.696 DEBUG:boto:Params: {} 2024-04-20T01:22:07.696 DEBUG:boto:Token: None 2024-04-20T01:22:07.697 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/obj148 2024-04-20T01:22:07.697 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IzUU5PEnHUi12I0/I+fjsNO0cok= 2024-04-20T01:22:07.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IzUU5PEnHUi12I0/I+fjsNO0cok='} 2024-04-20T01:22:07.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e379fdfb2e53dfd-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.743 DEBUG:boto:path=/aktlji-77/ 2024-04-20T01:22:07.743 DEBUG:boto:auth_path=/aktlji-77/ 2024-04-20T01:22:07.743 DEBUG:boto:Method: HEAD 2024-04-20T01:22:07.743 DEBUG:boto:Path: /aktlji-77/ 2024-04-20T01:22:07.743 DEBUG:boto:Data: 2024-04-20T01:22:07.743 DEBUG:boto:Headers: {} 2024-04-20T01:22:07.743 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.743 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.743 DEBUG:boto:Params: {} 2024-04-20T01:22:07.743 DEBUG:boto:Token: None 2024-04-20T01:22:07.744 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/ 2024-04-20T01:22:07.744 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:22:07.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:22:07.748 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085ea693e299f7ad3-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.748 DEBUG:boto:path=/aktlji-77/obj185 2024-04-20T01:22:07.748 DEBUG:boto:auth_path=/aktlji-77/obj185 2024-04-20T01:22:07.748 DEBUG:boto:Method: PUT 2024-04-20T01:22:07.748 DEBUG:boto:Path: /aktlji-77/obj185 2024-04-20T01:22:07.748 DEBUG:boto:Data: 2024-04-20T01:22:07.748 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:22:07.748 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:22:07.749 DEBUG:boto:Port: 8000 2024-04-20T01:22:07.749 DEBUG:boto:Params: {} 2024-04-20T01:22:07.749 DEBUG:boto:Token: None 2024-04-20T01:22:07.749 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:22:07 GMT /aktlji-77/obj185 2024-04-20T01:22:07.749 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ROmZMhld595jsC1yZgbhWyv0Uzc= 2024-04-20T01:22:07.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ROmZMhld595jsC1yZgbhWyv0Uzc='} 2024-04-20T01:22:07.755 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000fa00ce4eadd3adf-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:22:07.756 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-77 2024-04-20T01:22:07.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:22:07.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:22:07.757 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-77 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:22:07.824 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:22:07.824 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:22:07.867 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-77 2024-04-20T01:22:07.887 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-77 2024-04-20T01:22:07.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:22:07.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:22:07.888 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-77 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:22:07.963 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:22:07.964 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:22:08.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:08.054+0000 7f6b17288780 1 waiting to reach incremental sync.. 2024-04-20T01:22:13.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:13.058+0000 7f6b17288780 1 bucket sync caught up with source: 2024-04-20T01:22:13.055 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , 00000000002.71.6, , , , , , 00000000002.84.6, , , , , 00000000002.123.6, , , , , , , , , , , , , , , 00000000001.68.6, , , , 00000000001.114.6] 2024-04-20T01:22:13.055 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , 00000000002.71.6, , , , , , 00000000002.84.6, , , , , 00000000002.123.6, , , , , , , , , , , , , , , 00000000001.68.6, , , , 00000000001.114.6] 2024-04-20T01:22:13.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:13.058+0000 7f6b17288780 0 bucket checkpoint complete 2024-04-20T01:22:13.078 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-77 2024-04-20T01:22:13.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-77', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:22:13.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-77', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:22:13.079 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-77 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:22:13.146 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:22:13.147 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:22:13.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:13.207+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:22:18.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:18.208+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:22:23.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:23.209+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:22:28.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:28.209+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:22:33.207 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:33.210+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:22:38.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:38.211+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:22:43.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:43.211+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:22:48.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:48.212+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:22:53.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:53.214+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:22:58.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:22:58.214+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:23:03.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:23:03.215+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:23:08.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:23:08.216+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:23:13.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:23:13.216+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:23:18.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:23:18.217+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:23:23.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:23:23.217+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:23:28.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:23:28.217+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:23:33.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:23:33.217+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:23:38.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:23:38.218+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:23:43.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:23:43.219+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:23:48.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:23:48.219+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:23:53.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:23:53.220+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:23:58.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:23:58.220+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:24:03.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:24:03.221+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:24:08.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:24:08.221+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:24:13.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:24:13.222+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:24:18.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:24:18.225+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:24:23.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:24:23.226+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:24:28.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:24:28.226+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:24:33.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:24:33.227+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:24:38.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:24:38.227+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:24:43.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:24:43.229+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:24:48.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:24:48.229+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:24:53.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:24:53.230+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:24:58.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:24:58.231+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:25:03.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:25:03.231+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:25:08.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:25:08.232+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:25:13.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:25:13.232+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:25:18.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:25:18.232+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:25:23.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:25:23.232+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:25:28.227 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:25:28.233+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:25:33.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:25:33.234+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:25:38.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:25:38.234+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:25:43.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:25:43.235+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:25:48.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:25:48.235+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:25:53.230 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:25:53.236+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:25:58.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:25:58.237+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:26:03.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:26:03.237+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:26:08.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:26:08.238+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:26:13.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:26:13.239+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:26:18.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:26:18.239+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:26:23.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:26:23.240+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:26:28.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:26:28.240+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:26:33.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:26:33.242+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:26:38.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:26:38.242+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:26:43.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:26:43.243+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:26:48.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:26:48.244+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:26:53.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:26:53.244+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:26:58.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:26:58.245+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:27:03.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:03.245+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:27:08.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:08.245+0000 7f24e241b780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:27:08.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:08.245+0000 7f24e241b780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:27:08.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:08.245+0000 7f24e241b780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:27:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T01:27:08.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2024-04-20T01:27:08.260 DEBUG:boto:path=/aktlji-78/ 2024-04-20T01:27:08.261 DEBUG:boto:auth_path=/aktlji-78/ 2024-04-20T01:27:08.261 DEBUG:boto:Method: PUT 2024-04-20T01:27:08.261 DEBUG:boto:Path: /aktlji-78/ 2024-04-20T01:27:08.261 DEBUG:boto:Data: 2024-04-20T01:27:08.261 DEBUG:boto:Headers: {} 2024-04-20T01:27:08.261 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:27:08.261 DEBUG:boto:Port: 8000 2024-04-20T01:27:08.261 DEBUG:boto:Params: {} 2024-04-20T01:27:08.262 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:27:08.262 DEBUG:boto:Token: None 2024-04-20T01:27:08.262 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:27:08 GMT /aktlji-78/ 2024-04-20T01:27:08.262 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qx7VaSXgTN+wdroLMo6uWANwbuc= 2024-04-20T01:27:08.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:27:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qx7VaSXgTN+wdroLMo6uWANwbuc=', 'Content-Length': '0'} 2024-04-20T01:27:08.287 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000471c281676a8698a-00662319ec-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:27:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:27:08.287 DEBUG:rgw_multi.tests:created bucket=aktlji-78 2024-04-20T01:27:08.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:27:08.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:27:08.288 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:27:08.346 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:27:08.346 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:27:08.391 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:27:08.407 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:27:08.407 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:27:08.407 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:27:08.407 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:27:08.407 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:08.407 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:27:08.407 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:27:08.407 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:27:08.407 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:08.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:27:08.408 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:27:08.408 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:27:08.292227Z" 2024-04-20T01:27:08.408 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:08.408 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:27:08.408 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:27:08.408 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:27:08.408 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:27:08.408 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:27:08.409 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:27:08.409 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:27:08.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:27:08.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:27:08.410 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:27:08.472 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:27:08.472 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:27:08.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:27:08.539+0000 7fbebd8b0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:27:08.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:27:08.540+0000 7fbebd8b0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:27:08.552 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:27:08.552 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:27:08.552 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:27:08.552 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:27:08.552 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:27:08.552 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:27:08.552 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:27:08.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:27:08.553 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:27:08.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:27:08.553 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:27:08.553 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:27:08.553 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:27:08.553 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:27:08.553 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:27:08.553 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:27:08.553 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:27:08.553 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:27:08.554 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:27:08.554 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:27:08.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:27:08.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:27:08.554 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:27:08.554 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:27:08.554 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:27:08.554 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:27:08.554 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:27:08.554 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:27:08.554 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:27:08.555 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:27:08.555 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:27:08.555 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:27:08.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:27:08.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:27:08.555 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:27:08.555 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:27:08.555 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:27:08.555 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:27:08.555 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:27:08.555 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:27:08.556 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:27:08.556 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:27:08.556 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:27:08.556 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:27:08.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:27:08.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:27:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:27:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:27:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:27:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:27:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:27:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:27:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:27:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:27:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:27:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:27:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:27:08.557 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:27:08.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:27:08.558 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:27:08.558 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:27:08.558 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:27:08.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:27:08.558 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:27:08Z" 2024-04-20T01:27:08.558 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:27:08.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:27:08Z" } 2024-04-20T01:27:08.559 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:27:08.559 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:27:08.559 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:27:08.560 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:27:08.560 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:27:08.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:27:08.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:27:08.560 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:27:08.621 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:27:08.621 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:27:08.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:08.665+0000 7ff4b7b73780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:27:08.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:08.666+0000 7ff4b7b73780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:27:08.674 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:27:08.674 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:27:08.675 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:27:08.675 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:27:08.675 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:27:08.675 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:27:08.675 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:27:08.675 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:08.675 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:27:08.675 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:27:08.675 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:27:08.675 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:27:08.675 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:27:08.676 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:27:08.676 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:27:08.676 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:27:08.676 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:27:08.676 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:27:08.676 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:27:08.676 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:27:08.676 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:08.676 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:27:08.676 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:27:08.676 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:27:08.677 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:27:08.677 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:27:08.677 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:27:08.677 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:27:08.677 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:27:08.677 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:27:08.677 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:27:08.677 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:27:08.677 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:08.677 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:27:08.677 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:27:08.677 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:27:08.678 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:27:08.678 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576127.522556_1271.1", 2024-04-20T01:27:08.678 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:27:08.678 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:27:08.678 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:27:08.678 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:22:07.522556Z", 2024-04-20T01:27:08.678 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:27:08.678 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:27:08.678 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:08.679 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:27:08.679 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:27:08.679 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:27:08.679 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:27:08.679 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:27:08.679 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:27:08.679 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:27:08.679 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:27:08.679 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:27:08.679 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:27:08.679 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:27:08.680 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:27:08.680 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:27:08.680 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:08.680 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:27:08.680 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:27:08.680 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:27:08.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:08.681 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:27:08Z" 2024-04-20T01:27:08.681 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:27:08.682 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576127.522556_1271.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:22:07.522556Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:27:08Z" } 2024-04-20T01:27:08.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576127.522556_1271.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:22:07.522556Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:27:08.682 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:27:08.682 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576127.522556_1271.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:27:08.682 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1713576428.292227_1338.1 target=1_1713576127.522556_1271.1 2024-04-20T01:27:13.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:27:13.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:27:13.687 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:27:13.747 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:27:13.747 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:27:13.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:13.799+0000 7f8009551780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:27:13.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:13.800+0000 7f8009551780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:27:13.809 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:27:13.809 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:27:13.809 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:27:13.809 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:27:13.809 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:27:13.810 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:27:13.810 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:27:13.810 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:13.810 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:27:13.810 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:27:13.810 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:27:13.810 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:27:13.810 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:27:13.810 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:27:13.810 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:27:13.811 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:27:13.811 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:27:13.811 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:27:13.811 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:27:13.811 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:27:13.811 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:13.811 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:27:13.811 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:27:13.811 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:27:13.811 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:27:13.811 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:27:13.812 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:27:13.812 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:27:13.812 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:27:13.812 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:27:13.812 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:27:13.812 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:27:13.812 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:13.812 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:27:13.812 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:27:13.812 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:27:13.812 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:27:13.812 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:27:13.813 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:27:13.813 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:27:13.813 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:27:13.813 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:27:13.813 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:27:13.813 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:27:13.813 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:13.813 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:27:13.813 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:27:13.813 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:27:13.813 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:27:13.814 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:27:13.814 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:27:13.814 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:27:13.814 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:27:13.814 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:27:13.814 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:27:13.814 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:27:13.814 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:27:13.814 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:27:13.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:13.815 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:27:13.815 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:27:13.815 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:27:13.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:27:13.815 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:27:13Z" 2024-04-20T01:27:13.815 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:27:13.816 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:27:13Z" } 2024-04-20T01:27:13.816 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:27:13.816 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:27:13.817 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:27:13.817 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:27:13.817 DEBUG:boto:path=/aktlji-78/ 2024-04-20T01:27:13.817 DEBUG:boto:auth_path=/aktlji-78/ 2024-04-20T01:27:13.817 DEBUG:boto:Method: HEAD 2024-04-20T01:27:13.817 DEBUG:boto:Path: /aktlji-78/ 2024-04-20T01:27:13.817 DEBUG:boto:Data: 2024-04-20T01:27:13.817 DEBUG:boto:Headers: {} 2024-04-20T01:27:13.817 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:27:13.817 DEBUG:boto:Port: 8000 2024-04-20T01:27:13.818 DEBUG:boto:Params: {} 2024-04-20T01:27:13.818 DEBUG:boto:Token: None 2024-04-20T01:27:13.818 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:27:13 GMT /aktlji-78/ 2024-04-20T01:27:13.819 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w= 2024-04-20T01:27:13.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w=', 'Content-Length': '0'} 2024-04-20T01:27:13.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079be9952398f4e78-00662319f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:27:13.822 DEBUG:boto:path=/aktlji-78/obj11 2024-04-20T01:27:13.822 DEBUG:boto:auth_path=/aktlji-78/obj11 2024-04-20T01:27:13.822 DEBUG:boto:Method: PUT 2024-04-20T01:27:13.822 DEBUG:boto:Path: /aktlji-78/obj11 2024-04-20T01:27:13.822 DEBUG:boto:Data: 2024-04-20T01:27:13.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:27:13.822 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:27:13.822 DEBUG:boto:Port: 8000 2024-04-20T01:27:13.822 DEBUG:boto:Params: {} 2024-04-20T01:27:13.823 DEBUG:boto:Token: None 2024-04-20T01:27:13.823 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:27:13 GMT /aktlji-78/obj11 2024-04-20T01:27:13.823 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:MlmUsjsDdWXFO2oTQi4+/3Kw5VM= 2024-04-20T01:27:13.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MlmUsjsDdWXFO2oTQi4+/3Kw5VM='} 2024-04-20T01:27:13.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af8877fd92dc2904-00662319f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:27:13.830 DEBUG:boto:path=/aktlji-78/ 2024-04-20T01:27:13.830 DEBUG:boto:auth_path=/aktlji-78/ 2024-04-20T01:27:13.831 DEBUG:boto:Method: HEAD 2024-04-20T01:27:13.831 DEBUG:boto:Path: /aktlji-78/ 2024-04-20T01:27:13.831 DEBUG:boto:Data: 2024-04-20T01:27:13.831 DEBUG:boto:Headers: {} 2024-04-20T01:27:13.831 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:27:13.831 DEBUG:boto:Port: 8000 2024-04-20T01:27:13.831 DEBUG:boto:Params: {} 2024-04-20T01:27:13.831 DEBUG:boto:Token: None 2024-04-20T01:27:13.832 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:27:13 GMT /aktlji-78/ 2024-04-20T01:27:13.832 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w= 2024-04-20T01:27:13.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w=', 'Content-Length': '0'} 2024-04-20T01:27:13.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7b32aff48cdf5f9-00662319f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:27:13.834 DEBUG:boto:path=/aktlji-78/obj22 2024-04-20T01:27:13.835 DEBUG:boto:auth_path=/aktlji-78/obj22 2024-04-20T01:27:13.835 DEBUG:boto:Method: PUT 2024-04-20T01:27:13.835 DEBUG:boto:Path: /aktlji-78/obj22 2024-04-20T01:27:13.835 DEBUG:boto:Data: 2024-04-20T01:27:13.835 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:27:13.835 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:27:13.835 DEBUG:boto:Port: 8000 2024-04-20T01:27:13.835 DEBUG:boto:Params: {} 2024-04-20T01:27:13.835 DEBUG:boto:Token: None 2024-04-20T01:27:13.835 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:27:13 GMT /aktlji-78/obj22 2024-04-20T01:27:13.836 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AVFijVeEeD8RI7+K2TYHYlEIUiA= 2024-04-20T01:27:13.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AVFijVeEeD8RI7+K2TYHYlEIUiA='} 2024-04-20T01:27:13.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000123cabd8d17a0c15-00662319f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:27:13.841 DEBUG:boto:path=/aktlji-78/ 2024-04-20T01:27:13.841 DEBUG:boto:auth_path=/aktlji-78/ 2024-04-20T01:27:13.841 DEBUG:boto:Method: HEAD 2024-04-20T01:27:13.841 DEBUG:boto:Path: /aktlji-78/ 2024-04-20T01:27:13.841 DEBUG:boto:Data: 2024-04-20T01:27:13.841 DEBUG:boto:Headers: {} 2024-04-20T01:27:13.841 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:27:13.841 DEBUG:boto:Port: 8000 2024-04-20T01:27:13.841 DEBUG:boto:Params: {} 2024-04-20T01:27:13.841 DEBUG:boto:Token: None 2024-04-20T01:27:13.842 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:27:13 GMT /aktlji-78/ 2024-04-20T01:27:13.842 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w= 2024-04-20T01:27:13.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w=', 'Content-Length': '0'} 2024-04-20T01:27:13.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c1c64c127c58983-00662319f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:27:13.844 DEBUG:boto:path=/aktlji-78/obj33 2024-04-20T01:27:13.844 DEBUG:boto:auth_path=/aktlji-78/obj33 2024-04-20T01:27:13.844 DEBUG:boto:Method: PUT 2024-04-20T01:27:13.845 DEBUG:boto:Path: /aktlji-78/obj33 2024-04-20T01:27:13.845 DEBUG:boto:Data: 2024-04-20T01:27:13.845 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:27:13.845 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:27:13.845 DEBUG:boto:Port: 8000 2024-04-20T01:27:13.845 DEBUG:boto:Params: {} 2024-04-20T01:27:13.845 DEBUG:boto:Token: None 2024-04-20T01:27:13.845 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:27:13 GMT /aktlji-78/obj33 2024-04-20T01:27:13.845 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2oD0iX6ZTBKJhmLBfbgiHIH6apc= 2024-04-20T01:27:13.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2oD0iX6ZTBKJhmLBfbgiHIH6apc='} 2024-04-20T01:27:13.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068ba0ab5f943ee46-00662319f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:27:13.894 DEBUG:boto:path=/aktlji-78/ 2024-04-20T01:27:13.895 DEBUG:boto:auth_path=/aktlji-78/ 2024-04-20T01:27:13.895 DEBUG:boto:Method: HEAD 2024-04-20T01:27:13.895 DEBUG:boto:Path: /aktlji-78/ 2024-04-20T01:27:13.895 DEBUG:boto:Data: 2024-04-20T01:27:13.896 DEBUG:boto:Headers: {} 2024-04-20T01:27:13.896 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:27:13.896 DEBUG:boto:Port: 8000 2024-04-20T01:27:13.896 DEBUG:boto:Params: {} 2024-04-20T01:27:13.896 DEBUG:boto:Token: None 2024-04-20T01:27:13.896 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:27:13 GMT /aktlji-78/ 2024-04-20T01:27:13.896 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w= 2024-04-20T01:27:13.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w=', 'Content-Length': '0'} 2024-04-20T01:27:13.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e96a1c3b45221b9c-00662319f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:27:13.899 DEBUG:boto:path=/aktlji-78/obj44 2024-04-20T01:27:13.899 DEBUG:boto:auth_path=/aktlji-78/obj44 2024-04-20T01:27:13.899 DEBUG:boto:Method: PUT 2024-04-20T01:27:13.899 DEBUG:boto:Path: /aktlji-78/obj44 2024-04-20T01:27:13.899 DEBUG:boto:Data: 2024-04-20T01:27:13.900 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:27:13.900 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:27:13.900 DEBUG:boto:Port: 8000 2024-04-20T01:27:13.900 DEBUG:boto:Params: {} 2024-04-20T01:27:13.900 DEBUG:boto:Token: None 2024-04-20T01:27:13.901 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:27:13 GMT /aktlji-78/obj44 2024-04-20T01:27:13.901 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Pcz/KceXnp9ZkbzhogxskMt+9sk= 2024-04-20T01:27:13.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Pcz/KceXnp9ZkbzhogxskMt+9sk='} 2024-04-20T01:27:13.907 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c35bf85808ae2a50-00662319f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:27:13.908 DEBUG:boto:path=/aktlji-78/ 2024-04-20T01:27:13.908 DEBUG:boto:auth_path=/aktlji-78/ 2024-04-20T01:27:13.908 DEBUG:boto:Method: HEAD 2024-04-20T01:27:13.908 DEBUG:boto:Path: /aktlji-78/ 2024-04-20T01:27:13.908 DEBUG:boto:Data: 2024-04-20T01:27:13.908 DEBUG:boto:Headers: {} 2024-04-20T01:27:13.908 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:27:13.908 DEBUG:boto:Port: 8000 2024-04-20T01:27:13.908 DEBUG:boto:Params: {} 2024-04-20T01:27:13.908 DEBUG:boto:Token: None 2024-04-20T01:27:13.909 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:27:13 GMT /aktlji-78/ 2024-04-20T01:27:13.909 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w= 2024-04-20T01:27:13.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w=', 'Content-Length': '0'} 2024-04-20T01:27:13.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f737f7f19a121006-00662319f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:27:13.912 DEBUG:boto:path=/aktlji-78/obj55 2024-04-20T01:27:13.912 DEBUG:boto:auth_path=/aktlji-78/obj55 2024-04-20T01:27:13.912 DEBUG:boto:Method: PUT 2024-04-20T01:27:13.912 DEBUG:boto:Path: /aktlji-78/obj55 2024-04-20T01:27:13.912 DEBUG:boto:Data: 2024-04-20T01:27:13.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:27:13.912 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:27:13.912 DEBUG:boto:Port: 8000 2024-04-20T01:27:13.912 DEBUG:boto:Params: {} 2024-04-20T01:27:13.913 DEBUG:boto:Token: None 2024-04-20T01:27:13.913 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 20 Apr 2024 01:27:13 GMT /aktlji-78/obj55 2024-04-20T01:27:13.913 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:JVhb7AQ6x/S4mq4iDRhpvKRHieQ= 2024-04-20T01:27:13.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JVhb7AQ6x/S4mq4iDRhpvKRHieQ='} 2024-04-20T01:27:13.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ed9a86c8c1c40de-00662319f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:27:13.955 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-78 2024-04-20T01:27:13.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:27:13.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:27:13.955 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:27:14.022 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:27:14.022 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:27:14.068 INFO:teuthology.orchestra.run.smithi031.stdout:Sync is disabled for bucket aktlji-78 2024-04-20T01:27:14.089 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-78 2024-04-20T01:27:14.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:27:14.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:27:14.089 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:27:14.156 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:27:14.156 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:27:14.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:14.205+0000 7f35a78ef780 1 waiting to reach incremental sync.. 2024-04-20T01:27:19.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:19.205+0000 7f35a78ef780 1 waiting to reach incremental sync.. 2024-04-20T01:27:24.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:24.206+0000 7f35a78ef780 1 waiting to reach incremental sync.. 2024-04-20T01:27:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:29.206+0000 7f35a78ef780 1 waiting to reach incremental sync.. 2024-04-20T01:27:34.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:34.208+0000 7f35a78ef780 1 bucket sync caught up with source: 2024-04-20T01:27:34.202 INFO:teuthology.orchestra.run.smithi019.stderr: local status: [, , , , , , , , 00000000001.77.6, 00000000002.53.6, 00000000002.110.6] 2024-04-20T01:27:34.202 INFO:teuthology.orchestra.run.smithi019.stderr: remote markers: [, , , , , , , , 00000000001.77.6, 00000000002.53.6, 00000000002.110.6] 2024-04-20T01:27:34.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:34.208+0000 7f35a78ef780 0 bucket checkpoint complete 2024-04-20T01:27:34.224 DEBUG:rgw_multi.tests:bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-78 2024-04-20T01:27:34.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:27:34.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:27:34.224 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:27:34.290 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:27:34.291 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:27:34.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:34.345+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:27:39.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:39.346+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:27:44.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:44.347+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:27:49.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:49.347+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:27:54.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:54.348+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:27:59.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:27:59.349+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:28:04.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:28:04.349+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:28:09.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:28:09.350+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:28:14.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:28:14.350+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:28:19.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:28:19.351+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:28:24.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:28:24.352+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:28:29.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:28:29.353+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:28:34.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:28:34.354+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:28:39.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:28:39.354+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:28:44.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:28:44.355+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:28:49.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:28:49.355+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:28:54.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:28:54.356+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:28:59.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:28:59.356+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:29:04.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:29:04.356+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:29:09.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:29:09.357+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:29:14.350 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:29:14.357+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:29:19.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:29:19.358+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:29:24.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:29:24.359+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:29:29.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:29:29.359+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:29:34.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:29:34.360+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:29:39.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:29:39.365+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:29:44.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:29:44.365+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:29:49.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:29:49.366+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:29:54.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:29:54.367+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:29:59.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:29:59.367+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:30:04.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:30:04.368+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:30:09.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:30:09.369+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:30:14.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:30:14.370+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:30:19.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:30:19.370+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:30:24.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:30:24.371+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:30:29.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:30:29.372+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:30:34.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:30:34.372+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:30:39.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:30:39.375+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:30:44.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:30:44.374+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:30:49.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:30:49.375+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:30:54.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:30:54.376+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:30:59.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:30:59.376+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:31:04.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:31:04.377+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:31:09.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:31:09.377+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:31:14.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:31:14.378+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:31:19.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:31:19.378+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:31:24.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:31:24.379+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:31:29.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:31:29.379+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:31:34.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:31:34.380+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:31:39.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:31:39.381+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:31:44.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:31:44.382+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:31:49.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:31:49.383+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:31:54.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:31:54.383+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:31:59.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:31:59.385+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:32:04.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:32:04.386+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:32:09.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:32:09.387+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:32:14.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:32:14.387+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:32:19.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:32:19.388+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:32:24.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:32:24.388+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:32:29.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:32:29.389+0000 7f9301bd2780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:32:29.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:32:29.389+0000 7f9301bd2780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:32:29.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:32:29.389+0000 7f9301bd2780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:32:29.399 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-20T01:32:29.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2024-04-20T01:32:29.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:29.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:29.403 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:32:29.461 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:29.462 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:29.506 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:32:29.525 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:32:29.525 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:29.525 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:32:29.526 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:32:29.526 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:29.526 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:29.526 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:32:29.526 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:32:29.526 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:29.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:29.527 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:32:29.527 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:27:08.292227Z" 2024-04-20T01:32:29.527 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:29.527 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:29.527 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:32:29.528 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:32:29.528 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:29.528 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:32:29.530 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:29.531 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:32:29.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:29.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:29.532 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:32:29.592 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:29.592 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:29.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:29.647+0000 7f4befc9c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:29.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:29.648+0000 7f4befc9c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:29.660 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:32:29.661 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:32:29.661 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:32:29.661 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:32:29.661 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:32:29.661 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:32:29.661 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:29.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:29.661 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:32:29.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:29.662 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:32:29.662 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:29.662 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:29.662 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:32:29.662 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:29.662 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:32:29.662 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:29.662 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:32:29.662 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:29.662 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:29.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:29.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:29.663 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:32:29.663 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:29.663 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:29.663 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:32:29.663 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:29.663 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:29.663 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:29.663 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:32:29.663 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:29.663 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:29.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:29.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:29.664 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:32:29.664 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:29.664 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:29.664 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:32:29.664 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:29.664 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:29.664 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:29.664 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:32:29.664 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:29.665 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:29.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:29.665 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:29.665 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:32:29.665 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:29.665 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:29.665 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:32:29.665 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:29.665 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:29.665 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:29.665 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:32:29.666 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:29.666 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:29.666 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:29.666 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:32:29.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:29.666 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:32:29.666 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:32:29.666 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:32:29.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:29.666 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:32:29Z" 2024-04-20T01:32:29.666 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:32:29.667 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:32:29Z" } 2024-04-20T01:32:29.668 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:32:29.668 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:29.668 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:29.668 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:32:29.668 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:32:29.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:29.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:29.669 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:32:29.730 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:29.730 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:29.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:32:29.784+0000 7fbfe7d46780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:29.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:32:29.785+0000 7fbfe7d46780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:29.790 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:32:29.790 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:32:29.790 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:32:29.790 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:32:29.790 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:32:29.791 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:32:29.791 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:32:29.791 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:29.791 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:32:29.791 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:29.791 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:32:29.791 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:32:29.791 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:32:29.791 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:32:29.791 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:32:29.792 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:32:29.792 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:32:29.792 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:32:29.792 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:32:29.792 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:29.792 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:29.792 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:29.792 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:32:29.792 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:32:29.792 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:32:29.792 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:32:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:32:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:32:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:32:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:32:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:32:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:32:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:32:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:32:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:32:29.794 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:32:29.794 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:32:29.794 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:32:29.794 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:32:29.794 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:32:29.794 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:29.794 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:29.794 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:29.794 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:32:29.794 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:32:29.795 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:32:29.795 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:32:29.795 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:32:29.795 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:32:29.795 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:32:29.795 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:32:29.795 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:32:29.795 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:29.795 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:29.795 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:32:29.795 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:29.796 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:32:29.796 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:32:29.796 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:32:29.796 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:29.796 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:32:29Z" 2024-04-20T01:32:29.796 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:32:29.797 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:32:29Z" } 2024-04-20T01:32:29.797 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:32:29.797 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:29.797 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:29.797 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:32:29.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:29.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:29.798 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:29.850 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:29.850 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:29.909 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[]} 2024-04-20T01:32:29.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:29.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:29.910 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:29.966 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:29.966 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:30.031 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-20T01:32:30.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.032 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:30.087 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:30.087 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:30.143 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-20T01:32:30.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.144 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:30.196 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:30.196 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:30.264 INFO:teuthology.orchestra.run.smithi019.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-20T01:32:30.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.265 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:30.316 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:30.316 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:30.383 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-20T01:32:30.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.384 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:30.444 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:30.444 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:30.515 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-20T01:32:30.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.516 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:30.576 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:30.576 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:30.641 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-04-20T01:32:30.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.642 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:30.700 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:30.700 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:30.767 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:32:30.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.767 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:30.822 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:30.823 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:30.888 INFO:teuthology.orchestra.run.smithi019.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2024-04-20T01:32:30.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:30.889 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:30.943 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:30.943 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:31.005 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:32:31.005 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:32:31.005 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 9, 2024-04-20T01:32:31.005 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:32:31.005 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T01:32:31.005 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T01:32:31.005 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:32:31.006 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T01:32:31.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:31.006 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T01:32:31.006 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T01:32:31.006 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T01:32:31.006 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T01:32:31.006 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:32:31.006 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:32:31.006 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:32:31.006 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T01:32:31.006 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T01:32:31.007 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:32:31.007 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:32:31.007 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:31.007 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:32:31.007 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T01:32:31.007 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:32:31.007 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:32:31.007 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:32:31.007 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:32:31.007 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:32:31.007 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:32:31.008 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:32:31.008 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:32:31.008 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T01:32:31.008 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:32:31.008 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:32:31.008 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:32:31.008 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:32:31.008 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:32:31.008 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:32:31.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.008 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:31.009 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:32:31.009 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T01:32:31.009 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:32:31.009 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:32:31.009 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:32:31.009 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:32:31.009 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:32:31.009 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:32:31.009 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:32:31.009 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:32:31.009 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:32:31.009 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:32:31.010 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:32:31.010 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:32:31.010 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:32:31.010 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:32:31.010 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:32:31.010 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.010 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:31.010 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:32:31.010 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T01:32:31.010 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:32:31.010 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:32:31.011 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:32:31.011 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:32:31.011 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:32:31.011 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:32:31.012 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:32:31.012 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:32:31.012 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:32:31.012 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:32:31.012 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:32:31.012 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:32:31.012 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:32:31.012 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:32:31.013 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:32:31.013 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:31.013 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:32:31.013 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T01:32:31.013 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:31.013 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T01:32:31.013 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T01:32:31.013 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T01:32:31.013 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T01:32:31.013 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:32:31.013 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:31.013 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:32:31.014 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T01:32:31.014 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:32:31.014 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T01:32:31.014 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [ 2024-04-20T01:32:31.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:31.014 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-group", 2024-04-20T01:32:31.014 INFO:teuthology.orchestra.run.smithi019.stdout: "data_flow": { 2024-04-20T01:32:31.014 INFO:teuthology.orchestra.run.smithi019.stdout: "symmetrical": [ 2024-04-20T01:32:31.014 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:31.014 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow1", 2024-04-20T01:32:31.015 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:32:31.015 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:32:31.023 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:32:31.023 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:32:31.023 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:31.023 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:32:31.023 INFO:teuthology.orchestra.run.smithi019.stdout: "directional": [ 2024-04-20T01:32:31.023 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:31.024 INFO:teuthology.orchestra.run.smithi019.stdout: "source_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:32:31.024 INFO:teuthology.orchestra.run.smithi019.stdout: "dest_zone": "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:32:31.024 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:31.024 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:32:31.024 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.024 INFO:teuthology.orchestra.run.smithi019.stdout: "pipes": [ 2024-04-20T01:32:31.024 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:31.024 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe", 2024-04-20T01:32:31.024 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:32:31.024 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "*", 2024-04-20T01:32:31.024 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:32:31.025 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:32:31.025 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:32:31.025 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:32:31.025 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.025 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:32:31.025 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "*", 2024-04-20T01:32:31.025 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:32:31.026 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:32:31.026 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:32:31.026 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:32:31.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.026 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:32:31.026 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:32:31.026 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:32:31.026 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:32:31.026 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:31.026 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.027 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:32:31.027 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:32:31.027 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:32:31.027 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:32:31.027 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:31.027 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:31.027 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:32:31.027 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "enabled" 2024-04-20T01:32:31.027 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:31.027 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:32:31.027 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.027 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T01:32:31.028 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:32:31.028 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:32:31.029 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:31.029 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:32:31.029 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T01:32:31.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:31.029 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:32:31.029 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T01:32:31.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.029 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:31.029 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:32:31.029 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T01:32:31.029 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.030 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:31.030 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:32:31.030 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T01:32:31.030 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:31.030 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:32:31.030 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.030 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T01:32:31.030 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:32:31.030 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T01:32:31.030 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T01:32:31.030 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:32:31.031 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:32:31.031 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:32:31.031 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:32:31.031 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:32:31.031 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.031 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T01:32:31.031 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:32:31.031 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:32:31.031 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:32:31.031 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:32:31.031 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:32:31.032 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.032 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T01:32:31.032 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:32:31.032 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:32:31.032 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:32:31.032 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:32:31.032 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:32:31.032 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.032 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T01:32:31.032 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:32:31.032 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:32:31.033 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:32:31.033 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:32:31.033 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:32:31.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.033 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T01:32:31.033 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:32:31.033 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:32:31.033 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:32:31.033 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:32:31.033 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:32:31.033 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:31.033 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:31.034 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:32:31.034 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T01:32:31.034 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:32:31.034 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:32:31.035 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-79 2024-04-20T01:32:31.035 DEBUG:boto:path=/aktlji-79/ 2024-04-20T01:32:31.035 DEBUG:boto:auth_path=/aktlji-79/ 2024-04-20T01:32:31.036 DEBUG:boto:Method: PUT 2024-04-20T01:32:31.036 DEBUG:boto:Path: /aktlji-79/ 2024-04-20T01:32:31.036 DEBUG:boto:Data: 2024-04-20T01:32:31.036 DEBUG:boto:Headers: {} 2024-04-20T01:32:31.036 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:32:31.036 DEBUG:boto:Port: 8000 2024-04-20T01:32:31.036 DEBUG:boto:Params: {} 2024-04-20T01:32:31.037 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:32:31.037 DEBUG:boto:Token: None 2024-04-20T01:32:31.037 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:32:31 GMT /aktlji-79/ 2024-04-20T01:32:31.037 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/IuzX7NnforScbucoP/JEJkpxr4= 2024-04-20T01:32:31.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:32:31 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/IuzX7NnforScbucoP/JEJkpxr4=', 'Content-Length': '0'} 2024-04-20T01:32:31.527 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b279d85e70f3ab13-0066231b2f-18eb-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:32:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:32:31.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:31.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:31.527 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:31.622 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:31.623 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:31.842 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-20T01:32:31.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:31.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:31.843 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:31.930 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:31.931 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:32.071 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-20T01:32:32.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:32.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:32.072 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:32.138 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:32.138 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:32.232 INFO:teuthology.orchestra.run.smithi019.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-20T01:32:32.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:32.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:32.233 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:32.290 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:32.291 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:32.365 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-20T01:32:32.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:32.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:32.366 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:32.430 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:32.430 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:32.493 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-20T01:32:32.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:32.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:32.493 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --bucket aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:32.555 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:32.556 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:32.622 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-04-20T01:32:32.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:32.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:32.622 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:32.684 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:32.684 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-20T01:32:32.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:32.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:32:32.754 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:32:32.809 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:32.810 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:32.870 INFO:teuthology.orchestra.run.smithi019.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2024-04-20T01:32:32.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:32.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:32.871 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:32:32.925 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:32.925 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:32.962 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:32:32.978 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:32:32.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:32.978 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:32:32.978 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:32:32.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:32.978 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:32.978 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:32:32.978 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:32:32.978 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:32.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:32.979 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576752.746185_1427.1", 2024-04-20T01:32:32.979 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:32:32.746185Z" 2024-04-20T01:32:32.979 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:32:32.979 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:32:32.979 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:32:32.979 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:32:32.979 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:32:32.979 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:32:32.980 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:32.980 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:32:32.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:32.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:32.981 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:32:33.041 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:33.041 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:33.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:33.099+0000 7fc87e696780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:33.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:33.100+0000 7fc87e696780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:33.118 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:32:33.119 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:32:33.119 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:32:33.119 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:32:33.119 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:32:33.119 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:32:33.119 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:33.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:33.119 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:32:33.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:33.120 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:32:33.120 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:33.120 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:33.120 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:32:33.120 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:33.120 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:32:33.120 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:33.120 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:32:33.120 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:33.120 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:33.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:33.121 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:33.121 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:32:33.121 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:33.121 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:33.121 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:32:33.121 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:33.121 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:33.121 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:33.121 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:32:33.122 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:33.122 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:33.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:33.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:33.122 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:32:33.122 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:33.122 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:33.122 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:32:33.122 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:33.122 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:33.123 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:33.123 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:32:33.123 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:33.123 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:33.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:33.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:33.123 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:32:33.124 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:33.124 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:33.124 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:32:33.124 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:33.124 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:33.124 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:33.124 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:32:33.124 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:33.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:33.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:33.125 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:32:33.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:33.125 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:32:33.125 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:32:33.125 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:32:33.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:33.125 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:32:33Z" 2024-04-20T01:32:33.125 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:32:33.126 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:32:33Z" } 2024-04-20T01:32:33.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:32:33.127 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:33.127 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:33.127 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:32:36.042 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-20T01:32:36.051+0000 7f0a26de0700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-20T01:32:38.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:38.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:38.129 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:32:38.193 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:38.193 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:38.255+0000 7fc665cbb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:38.256+0000 7fc665cbb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:38.267 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:32:38.267 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:32:38.268 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:32:38.268 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:32:38.268 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:32:38.268 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:32:38.268 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:38.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:38.268 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:32:38.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:38.268 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:32:38.268 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:38.268 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:38.269 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:32:38.269 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:38.269 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:32:38.269 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:38.269 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:32:38.269 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:38.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:38.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:38.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:38.269 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:32:38.269 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:38.269 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:38.270 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:32:38.270 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:38.270 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:38.270 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:38.270 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:32:38.270 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:38.270 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:38.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:38.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:38.270 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:32:38.270 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:38.270 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:38.271 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:32:38.271 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:38.271 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:38.271 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:38.271 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:32:38.271 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:38.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:38.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:38.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:38.271 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:32:38.271 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:38.272 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:38.272 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:32:38.272 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:38.272 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:38.272 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:38.272 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:32:38.272 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:38.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:38.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:38.272 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:32:38.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:38.272 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:32:38.273 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:32:38.273 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:32:38.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:38.273 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:32:38Z" 2024-04-20T01:32:38.273 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:32:38.274 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:32:38Z" } 2024-04-20T01:32:38.275 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:32:38.275 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:38.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:38.275 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:32:43.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:43.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:43.300 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:32:43.360 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:43.360 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:43.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:43.410+0000 7fcddedaa780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:43.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:43.412+0000 7fcddedaa780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:43.421 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:32:43.421 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:32:43.421 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:32:43.421 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:32:43.421 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:32:43.421 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:32:43.421 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:43.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:43.422 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:32:43.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:43.422 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:32:43.422 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:43.422 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:43.422 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:32:43.422 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:43.422 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:32:43.422 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:43.422 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:32:43.423 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:43.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:43.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:43.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:43.423 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:32:43.423 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:43.423 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:43.423 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:32:43.423 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:43.423 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:43.423 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:43.424 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:32:43.424 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:43.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:43.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:43.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:43.424 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:32:43.424 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:43.425 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:43.425 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:32:43.425 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:43.425 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:43.425 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:43.425 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:32:43.425 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:43.425 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:43.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:43.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:43.425 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:32:43.426 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:43.426 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:43.426 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:32:43.426 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:43.426 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:43.426 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:43.426 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:32:43.426 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:43.426 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:43.426 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:43.427 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:32:43.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:43.427 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:32:43.427 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:32:43.427 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:32:43.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:43.427 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:32:43Z" 2024-04-20T01:32:43.427 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:32:43.428 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:32:43Z" } 2024-04-20T01:32:43.428 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:32:43.428 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:43.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:43.429 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:32:48.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:48.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:48.430 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:32:48.491 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:48.491 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:48.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:48.551+0000 7fa2e33cc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:48.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:48.553+0000 7fa2e33cc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:48.569 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:32:48.569 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:32:48.570 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:32:48.570 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:32:48.570 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:32:48.570 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:32:48.570 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:48.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:48.570 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:32:48.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:48.570 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:32:48.570 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:48.571 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:48.571 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:32:48.571 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:48.571 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:32:48.571 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:48.571 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:32:48.571 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:48.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:48.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:48.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:48.571 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:32:48.572 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:48.572 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:48.572 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:32:48.572 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:48.572 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:48.572 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:48.572 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:32:48.572 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:48.572 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:48.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:48.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:48.573 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:32:48.573 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:48.573 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:48.573 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:32:48.573 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:48.573 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:48.573 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:48.573 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:32:48.573 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:48.573 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:48.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:48.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:48.574 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:32:48.574 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:48.574 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:48.574 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:32:48.574 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:48.574 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:48.574 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:48.574 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:32:48.574 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:48.574 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:48.575 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:48.575 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:32:48.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:48.575 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:32:48.575 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:32:48.575 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:32:48.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:48.575 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:32:48Z" 2024-04-20T01:32:48.575 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:32:48.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:32:48Z" } 2024-04-20T01:32:48.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:32:48.577 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:48.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:48.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:32:53.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:53.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:53.579 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:32:53.638 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:53.639 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:53.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:53.694+0000 7f9275898780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:53.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:53.695+0000 7f9275898780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:53.703 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:32:53.703 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:32:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:32:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:32:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:32:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:32:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:32:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:32:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:32:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:32:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:32:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:32:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:32:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:32:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:32:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:32:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:32:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:32:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:32:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:32:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:32:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:32:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:32:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:32:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:32:53Z" 2024-04-20T01:32:53.709 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:32:53.710 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:32:53Z" } 2024-04-20T01:32:53.711 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:32:53.711 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:53.711 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:53.711 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:32:58.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:58.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:32:58.713 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:32:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:32:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:32:58.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:58.834+0000 7fc3e02d2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:58.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:32:58.836+0000 7fc3e02d2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:32:58.850 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:32:58.850 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:32:58.851 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:32:58.851 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:32:58.851 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:32:58.851 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:32:58.851 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:58.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:58.851 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:32:58.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:58.851 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:32:58.851 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:58.851 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:58.852 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:32:58.852 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:58.852 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:32:58.852 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:58.852 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:32:58.852 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:58.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:58.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:58.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:58.852 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:32:58.852 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:58.853 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:58.853 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:32:58.853 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:58.853 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:58.853 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:58.853 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:32:58.853 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:58.853 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:58.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:58.853 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:58.854 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:32:58.854 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:58.854 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:58.854 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:32:58.854 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:58.854 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:58.854 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:58.854 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:32:58.854 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:58.854 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:58.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:58.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:32:58.858 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:32:58.858 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:32:58.858 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:32:58.858 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:32:58.859 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:32:58.859 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:32:58.859 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:32:58.859 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:32:58.859 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:32:58.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:58.860 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:32:58.860 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:32:58.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:58.860 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:32:58.860 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:32:58.860 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:32:58.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:32:58.861 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:32:58Z" 2024-04-20T01:32:58.861 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:32:58.862 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:32:58Z" } 2024-04-20T01:32:58.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:32:58.863 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:58.863 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:32:58.863 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:33:03.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:03.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:03.864 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:33:03.924 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:33:03.924 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:33:03.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:03.981+0000 7fab8f725780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:03.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:03.982+0000 7fab8f725780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:03.991 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:33:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:33:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:33:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:33:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:33:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:33:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:33:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:33:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:33:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:33:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:33:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:33:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:33:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:33:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:33:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:33:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:33:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:33:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:33:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:33:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:33:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:33:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:33:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:33:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:33:03Z" 2024-04-20T01:33:03.997 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:33:03.998 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:33:03Z" } 2024-04-20T01:33:03.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:33:03.998 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:03.998 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:03.998 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:33:08.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:08.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:08.999 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:33:09.071 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:33:09.072 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:33:09.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:09.132+0000 7f7b72078780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:09.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:09.133+0000 7f7b72078780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:09.143 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:33:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:33:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:33:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:33:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:33:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:33:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:09.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:33:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:33:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:33:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:33:09.144 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:33:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:33:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:33:09.145 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:09.146 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:33:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:33:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:09.147 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:33:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:33:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:33:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:33:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:33:09.149 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:33:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:09.150 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:33:09Z" 2024-04-20T01:33:09.150 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:33:09.151 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:33:09Z" } 2024-04-20T01:33:09.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:33:09.151 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:09.151 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:09.151 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:33:14.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:14.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:14.153 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:33:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:33:14.212 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:33:14.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:14.276+0000 7f5c6a558780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:14.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:14.277+0000 7f5c6a558780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:14.286 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:33:14.287 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:33:14.287 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:33:14.287 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:33:14.287 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:33:14.287 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:33:14.287 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:14.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:14.287 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:33:14.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:14.287 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:33:14.288 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:14.288 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:14.288 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:33:14.288 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:14.288 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:33:14.288 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:14.288 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:33:14.288 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:14.288 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:14.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:14.288 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:14.289 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:33:14.289 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:14.289 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:14.289 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:33:14.289 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:14.289 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:14.289 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:14.289 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:33:14.289 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:14.289 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:14.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:14.290 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:14.290 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:33:14.290 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:14.290 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:14.290 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:33:14.290 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:14.290 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:14.290 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:14.290 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:33:14.290 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:14.290 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:14.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:14.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:14.291 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:33:14.291 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:14.291 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:14.291 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:33:14.291 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:14.291 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:14.291 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:14.291 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:33:14.291 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:14.291 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:14.292 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:14.292 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:33:14.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:14.292 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:33:14.292 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:33:14.292 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:33:14.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:14.292 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:33:14Z" 2024-04-20T01:33:14.292 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:33:14.293 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:33:14Z" } 2024-04-20T01:33:14.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:33:14.294 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:14.294 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:14.294 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:33:19.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:19.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:19.295 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:33:19.354 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:33:19.354 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:33:19.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:19.421+0000 7f9ef9d1b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:19.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:19.422+0000 7f9ef9d1b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:19.433 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:33:19.434 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:33:19.434 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:33:19.434 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:33:19.434 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:33:19.434 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:33:19.435 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:19.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:19.435 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:33:19.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:19.435 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:33:19.435 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:19.435 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:19.435 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:33:19.435 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:19.435 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:33:19.435 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:19.436 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:33:19.436 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:19.436 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:19.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:19.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:19.436 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:33:19.436 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:19.436 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:19.436 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:33:19.436 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:19.436 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:19.436 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:19.437 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:33:19.437 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:19.437 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:19.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:19.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:19.437 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:33:19.438 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:19.438 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:19.438 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:33:19.438 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:19.438 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:19.438 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:19.438 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:33:19.438 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:19.438 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:19.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:19.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:19.439 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:33:19.439 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:19.439 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:19.439 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:33:19.439 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:19.440 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:19.440 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:19.440 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:33:19.440 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:19.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:19.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:19.441 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:33:19.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:19.441 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:33:19.441 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:33:19.441 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:33:19.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:19.441 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:33:19Z" 2024-04-20T01:33:19.441 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:33:19.442 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:33:19Z" } 2024-04-20T01:33:19.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:33:19.443 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:19.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:19.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:33:24.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:24.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:24.445 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:33:24.506 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:33:24.506 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:33:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:24.565+0000 7f29e81d7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:24.566+0000 7f29e81d7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:24.575 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:33:24.575 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:33:24.575 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:33:24.575 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:33:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:33:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:33:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:33:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:33:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:33:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:33:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:33:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:33:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:33:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:33:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:33:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:33:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:33:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:33:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:33:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:33:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:33:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:33:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:33:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:33:24.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:24.581 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:33:24Z" 2024-04-20T01:33:24.581 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:33:24.582 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:33:24Z" } 2024-04-20T01:33:24.582 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:33:24.582 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:24.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:24.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:33:29.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:29.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:29.585 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:33:29.645 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:33:29.645 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:33:29.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:29.706+0000 7f99bd4e0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:29.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:29.707+0000 7f99bd4e0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:29.722 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:33:29.722 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:33:29.722 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:33:29.722 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:33:29.722 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:33:29.722 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:33:29.722 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:29.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:29.723 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:33:29.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:29.723 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:33:29.723 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:29.723 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:29.723 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:33:29.723 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:29.723 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:33:29.723 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:29.723 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:33:29.723 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:29.724 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:29.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:29.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:29.724 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:33:29.724 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:29.724 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:29.724 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:33:29.724 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:29.724 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:29.724 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:29.724 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:33:29.725 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:29.725 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:29.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:29.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:29.725 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:33:29.725 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:29.725 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:29.725 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:33:29.725 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:29.725 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:29.725 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:29.726 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:33:29.726 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:29.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:29.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:29.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:29.726 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:33:29.726 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:29.726 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:29.726 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:33:29.726 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:29.726 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:29.727 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:29.727 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:33:29.727 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:29.727 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:29.727 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:29.727 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:33:29.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:29.728 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:33:29.728 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:33:29.728 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:33:29.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:29.728 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:33:29Z" 2024-04-20T01:33:29.728 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:33:29.729 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:33:29Z" } 2024-04-20T01:33:29.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:33:29.730 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:29.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:29.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:33:34.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:34.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:34.732 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:33:34.794 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:33:34.794 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:33:34.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:34.852+0000 7fb0bf6ea780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:34.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:34.853+0000 7fb0bf6ea780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:34.866 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:33:34.867 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:33:34.867 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:33:34.867 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:33:34.867 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:33:34.867 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:33:34.867 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:34.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:34.867 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:33:34.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:34.868 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:33:34.868 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:34.868 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:34.868 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:33:34.868 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:34.868 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:33:34.868 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:34.868 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:33:34.868 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:34.868 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:34.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:34.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:34.869 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:33:34.869 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:34.869 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:34.869 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:33:34.869 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:34.869 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:34.869 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:34.869 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:33:34.869 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:34.869 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:34.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:34.870 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:34.870 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:33:34.870 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:34.870 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:34.870 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:33:34.870 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:34.870 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:34.870 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:34.870 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:33:34.870 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:34.871 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:34.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:34.871 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:34.871 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:33:34.871 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:34.871 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:34.871 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:33:34.871 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:34.871 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:34.871 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:34.871 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:33:34.871 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:34.872 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:34.872 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:34.872 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:33:34.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:34.872 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:33:34.872 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:33:34.872 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:33:34.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:34.872 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:33:34Z" 2024-04-20T01:33:34.872 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:33:34.873 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:33:34Z" } 2024-04-20T01:33:34.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:33:34.874 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:34.874 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:34.874 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:33:39.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:39.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:39.876 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:33:39.935 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:33:39.936 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:33:39.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:39.988+0000 7fc6bc5fc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:39.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:39.989+0000 7fc6bc5fc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:39.997 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:33:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:33:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:33:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:33:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:33:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:33:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:33:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:33:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:33:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:33:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:33:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:33:40.000 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:40.000 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:40.000 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:33:40.000 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:40.000 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:40.000 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:40.000 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:33:40.000 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:40.000 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:40.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:40.000 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:33:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:33:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:33:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:40.001 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:40.002 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:33:40.002 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:40.002 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:40.002 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:33:40.002 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:40.002 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:40.002 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:40.002 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:33:40.002 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:40.002 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:40.002 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:40.003 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:33:40.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:40.003 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:33:40.003 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:33:40.003 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:33:40.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:40.003 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:33:39Z" 2024-04-20T01:33:40.003 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:33:40.004 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:33:39Z" } 2024-04-20T01:33:40.004 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:33:40.005 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:40.005 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:40.005 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:33:45.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:45.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:45.006 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:33:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:33:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:33:45.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:45.133+0000 7f942fe85780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:45.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:45.134+0000 7f942fe85780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:45.146 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:33:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:33:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:33:45.146 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:33:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:33:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:33:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:33:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:33:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:33:45.147 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:33:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:33:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:33:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:45.148 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:33:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:33:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:33:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:45.149 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:33:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:33:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:45.150 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:33:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:33:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:45.151 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:33:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:33:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:33:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:33:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:33:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:45.152 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:33:45Z" 2024-04-20T01:33:45.153 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:33:45.153 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:33:45Z" } 2024-04-20T01:33:45.154 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:33:45.154 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:45.154 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:45.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:33:50.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:50.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:50.156 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:33:50.215 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:33:50.216 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:33:50.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:50.277+0000 7f3fdef72780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:50.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:50.278+0000 7f3fdef72780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:50.290 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:33:50.290 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:33:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:33:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:33:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:33:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:33:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:33:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:33:50.291 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:33:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:33:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:33:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:50.292 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:33:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:33:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:33:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:50.293 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:33:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:33:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:33:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:50.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:33:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:33:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:50.295 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:33:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:33:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:33:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:33:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:33:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:50.296 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:33:50Z" 2024-04-20T01:33:50.296 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:33:50.298 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:33:50Z" } 2024-04-20T01:33:50.298 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:33:50.298 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:50.298 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:50.298 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:33:55.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:55.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:33:55.299 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:33:55.359 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:33:55.359 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:33:55.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:55.411+0000 7fd3d4cac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:55.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:33:55.413+0000 7fd3d4cac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:33:55.421 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:33:55.421 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:33:55.421 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:33:55.422 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:33:55.422 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:33:55.422 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:33:55.422 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:55.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:55.422 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:33:55.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:55.422 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:33:55.422 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:55.422 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:55.422 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:33:55.423 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:55.423 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:33:55.423 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:55.423 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:33:55.423 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:55.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:55.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:55.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:55.423 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:33:55.423 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:55.423 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:55.424 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:33:55.424 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:55.424 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:55.424 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:55.424 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:33:55.424 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:55.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:55.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:55.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:55.424 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:33:55.424 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:55.425 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:55.425 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:33:55.425 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:55.425 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:55.425 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:55.425 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:33:55.425 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:55.425 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:55.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:55.425 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:33:55.425 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:33:55.425 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:33:55.426 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:33:55.426 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:33:55.426 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:33:55.426 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:33:55.426 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:33:55.426 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:33:55.426 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:33:55.426 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:55.427 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:33:55.427 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:33:55.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:55.427 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:33:55.427 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:33:55.427 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:33:55.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:33:55.427 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:33:55Z" 2024-04-20T01:33:55.427 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:33:55.428 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:33:55Z" } 2024-04-20T01:33:55.428 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:33:55.429 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:55.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:33:55.429 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:34:00.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:00.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:00.430 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:00.490 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:00.490 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:00.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:00.541+0000 7fc5ff8ac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:00.543+0000 7fc5ff8ac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:00.553 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:34:00.553 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:34:00.554 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:34:00.554 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:34:00.554 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:34:00.554 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:00.554 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:00.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:00.554 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:34:00.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:00.554 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:34:00.554 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:00.555 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:00.555 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:00.555 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:00.555 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:34:00.555 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:00.555 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:00.555 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:00.555 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:00.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:00.555 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:00.555 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:34:00.555 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:00.556 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:00.556 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:34:00.556 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:00.556 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:00.556 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:00.556 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:34:00.556 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:00.556 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:00.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:00.556 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:00.556 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:34:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:34:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:34:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:00.557 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:34:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:00.558 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:34:00.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:00.559 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:34:00.559 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:34:00.559 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:34:00.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:00.559 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:34:00Z" 2024-04-20T01:34:00.559 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:34:00.560 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:00Z" } 2024-04-20T01:34:00.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:00.561 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:00.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:00.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:34:05.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:05.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:05.564 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:05.624 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:05.624 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:05.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:05.687+0000 7f4269797780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:05.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:05.688+0000 7f4269797780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:05.703 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:34:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:34:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:34:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:34:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:34:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:34:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:34:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:34:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:34:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:34:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:34:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:34:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:34:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:34:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:34:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:34:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:34:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:34:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:34:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:34:05Z" 2024-04-20T01:34:05.711 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:34:05.713 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:05Z" } 2024-04-20T01:34:05.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:05.713 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:05.713 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:05.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:34:10.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:10.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:10.715 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:10.775 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:10.775 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:10.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:10.825+0000 7f2a5d18c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:10.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:10.827+0000 7f2a5d18c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:10.836 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:34:10.836 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:34:10.836 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:34:10.836 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:34:10.837 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:34:10.837 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:10.837 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:10.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:10.837 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:34:10.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:10.837 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:34:10.837 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:10.837 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:10.837 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:10.837 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:10.838 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:34:10.838 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:10.838 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:10.838 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:10.838 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:10.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:10.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:10.838 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:34:10.838 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:10.838 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:10.839 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:34:10.839 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:10.839 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:10.839 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:10.839 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:34:10.839 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:10.839 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:10.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:10.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:10.839 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:34:10.839 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:10.839 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:10.840 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:34:10.840 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:10.840 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:10.840 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:10.840 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:34:10.840 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:10.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:10.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:10.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:10.840 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:34:10.840 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:10.841 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:10.841 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:10.841 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:10.841 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:10.841 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:10.841 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:10.841 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:10.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:10.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:10.841 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:34:10.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:10.841 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:34:10.842 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:34:10.842 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:34:10.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:10.842 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:34:10Z" 2024-04-20T01:34:10.842 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:34:10.843 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:10Z" } 2024-04-20T01:34:10.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:10.843 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:10.843 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:10.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:34:15.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:15.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:15.845 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:15.904 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:15.905 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:15.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:15.968+0000 7f2b3633d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:15.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:15.969+0000 7f2b3633d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:15.981 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:34:15.981 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:34:15.981 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:34:15.981 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:34:15.981 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:34:15.981 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:15.982 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:15.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:15.982 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:34:15.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:15.982 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:34:15.982 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:15.982 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:15.982 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:15.982 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:15.982 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:34:15.982 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:15.983 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:15.983 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:15.983 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:15.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:15.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:15.983 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:34:15.983 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:15.983 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:15.983 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:34:15.983 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:15.983 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:15.984 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:15.984 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:34:15.984 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:15.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:15.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:15.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:15.984 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:34:15.984 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:15.984 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:15.985 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:34:15.985 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:15.985 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:15.985 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:15.985 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:34:15.985 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:15.985 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:15.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:15.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:15.985 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:34:15.986 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:15.986 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:15.986 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:15.986 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:15.986 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:15.986 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:15.986 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:15.986 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:15.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:15.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:15.986 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:34:15.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:15.987 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:34:15.987 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:34:15.987 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:34:15.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:15.987 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:34:15Z" 2024-04-20T01:34:15.987 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:34:15.991 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:15Z" } 2024-04-20T01:34:15.991 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:15.991 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:15.991 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:15.991 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:34:20.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:20.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:20.993 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:21.052 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:21.052 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:21.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:21.115+0000 7f49bee03780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:21.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:21.117+0000 7f49bee03780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:21.126 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:34:21.126 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:34:21.127 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:34:21.127 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:34:21.127 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:34:21.127 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:21.127 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:21.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:21.127 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:34:21.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:21.127 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:34:21.127 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:34:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:21.128 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:34:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:34:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:34:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:21.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:21.130 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:34:21.130 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:21.130 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:21.130 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:34:21.130 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:21.130 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:21.130 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:21.130 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:34:21.130 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:21.130 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:21.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:21.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:21.131 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:34:21.131 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:21.131 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:21.131 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:21.131 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:21.131 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:21.131 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:21.131 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:21.131 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:21.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:21.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:21.132 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:34:21.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:21.132 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:34:21.132 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:34:21.132 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:34:21.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:21.133 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:34:21Z" 2024-04-20T01:34:21.133 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:34:21.134 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:21Z" } 2024-04-20T01:34:21.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:21.134 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:21.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:21.135 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:34:26.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:26.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:26.136 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:26.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:26.257+0000 7f3e2dc32780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:26.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:26.258+0000 7f3e2dc32780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:26.266 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:34:26.266 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:34:26.266 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:34:26.266 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:34:26.267 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:34:26.267 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:26.267 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:26.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:26.267 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:34:26.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:26.267 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:34:26.267 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:26.267 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:26.267 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:26.267 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:26.268 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:34:26.268 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:26.268 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:26.268 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:26.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:26.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:26.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:26.268 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:34:26.268 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:26.268 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:26.268 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:34:26.269 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:26.269 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:26.269 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:26.269 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:34:26.269 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:26.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:26.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:26.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:26.269 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:34:26.269 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:26.269 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:26.270 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576752.746185_1427.1", 2024-04-20T01:34:26.270 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:26.270 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:26.270 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:26.270 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:32:32.746185Z", 2024-04-20T01:34:26.270 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:26.270 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:26.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:26.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:26.271 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:34:26.271 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:26.271 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:26.271 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:26.271 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:26.271 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:26.271 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:26.271 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:26.272 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:26.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:26.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:26.272 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:34:26.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:26.272 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:34:26.272 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:34:26.272 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:34:26.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:26.272 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:34:26Z" 2024-04-20T01:34:26.272 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:34:26.273 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576752.746185_1427.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:32:32.746185Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:26Z" } 2024-04-20T01:34:26.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576752.746185_1427.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:32.746185Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:26.274 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:26.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:26.274 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:34:26.274 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:34:26.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:26.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:26.275 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:26.334 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:26.334 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:26.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:26.380+0000 7f98dfbc9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:26.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:26.381+0000 7f98dfbc9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:26.386 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:34:26.386 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:34:26.386 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:34:26.386 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:34:26.386 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:34:26.387 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:26.387 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:26.387 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:26.387 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:34:26.387 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:26.387 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:34:26.387 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:26.387 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:26.387 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:26.387 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:26.387 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:34:26.387 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:26.388 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:26.388 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:26.388 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:26.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:26.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:26.388 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:34:26.388 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:26.388 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:26.388 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:34:26.388 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:26.388 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:26.388 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:26.389 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:34:26.389 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:26.389 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:26.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:26.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:26.389 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:34:26.389 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:26.389 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:26.389 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:34:26.389 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:26.389 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:26.390 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:26.390 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:34:26.390 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:26.390 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:26.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:26.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:26.390 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:34:26.390 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:26.390 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:26.390 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:26.390 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:26.390 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:26.391 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:26.391 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:26.391 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:26.391 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:26.391 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:26.391 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:26.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:26.391 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:34:26.391 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:34:26.391 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:34:26.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:26.392 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:34:26Z" 2024-04-20T01:34:26.392 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:34:26.393 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:26Z" } 2024-04-20T01:34:26.393 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:26.393 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:26.393 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:26.393 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:34:31.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:31.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:31.394 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:31.455 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:31.455 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:31.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:31.501+0000 7f048db18780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:31.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:31.503+0000 7f048db18780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:31.515 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:34:31.515 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:34:31.515 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:34:31.515 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:34:31.515 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:34:31.516 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:31.516 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:31.516 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:31.516 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:34:31.516 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:31.516 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:34:31.516 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:31.516 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:31.516 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:31.516 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:31.516 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:34:31.516 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:31.517 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:31.517 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:31.517 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:31.517 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:31.517 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:31.517 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:34:31.517 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:31.517 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:31.517 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:34:31.517 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:31.517 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:31.517 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:31.518 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:34:31.518 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:31.518 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:31.518 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:31.518 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:31.518 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:34:31.518 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:31.518 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:31.518 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:34:31.519 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:31.519 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:31.519 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:31.519 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:34:31.519 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:31.519 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:31.519 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:31.519 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:31.519 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:34:31.519 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:31.519 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:31.520 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:31.520 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:31.520 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:31.520 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:31.520 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:31.520 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:31.520 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:31.520 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:31.520 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:31.520 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:31.520 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:34:31.520 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:34:31.521 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:34:31.521 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:31.521 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:34:31Z" 2024-04-20T01:34:31.521 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:34:31.522 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576428.292227_1338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:08.292227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:31Z" } 2024-04-20T01:34:31.522 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:31.522 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:31.522 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:31.522 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576428.292227_1338.1 2024-04-20T01:34:36.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:36.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:36.524 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:36.584 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:36.584 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:36.630+0000 7f7cc0e56780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:36.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:36.631+0000 7f7cc0e56780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:36.640 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:34:36.640 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:34:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:34:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:34:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:34:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:34:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:34:36.641 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:34:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:36.642 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:34:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:34:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:34:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:36.643 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:34:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576752.613859_1425.1", 2024-04-20T01:34:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:36.644 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:32:32.613859Z", 2024-04-20T01:34:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:34:36.645 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:36.647 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:34:36.647 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:34:36.647 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:34:36.647 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:36.647 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:34:36Z" 2024-04-20T01:34:36.647 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:34:36.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576752.613859_1425.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:32:32.613859Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:36Z" } 2024-04-20T01:34:36.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576752.613859_1425.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:32.613859Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:36.649 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:36.649 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.613859_1425.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:36.650 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1713576752.746185_1427.1 target=1_1713576752.613859_1425.1 2024-04-20T01:34:41.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:41.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:41.651 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:41.711 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:41.711 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:41.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:41.768+0000 7f18d9cc7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:41.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:41.769+0000 7f18d9cc7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:41.777 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:34:41.778 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:34:41.778 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:34:41.778 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:34:41.778 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:34:41.778 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:41.778 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:41.778 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:41.778 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:34:41.778 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:41.779 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:34:41.779 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:41.779 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:41.779 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:41.779 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:41.779 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:34:41.779 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:41.779 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:41.779 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:41.779 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:41.779 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:41.779 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:41.780 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:34:41.780 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:41.780 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:41.780 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:34:41.780 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:41.780 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:41.780 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:41.780 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:34:41.780 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:41.780 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:41.780 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:41.781 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:41.781 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:34:41.781 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:41.781 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:41.781 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576752.746185_1427.1", 2024-04-20T01:34:41.781 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:41.781 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:41.781 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:41.781 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:32:32.746185Z", 2024-04-20T01:34:41.781 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:41.781 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:41.782 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:41.782 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:41.782 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:34:41.782 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:41.782 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:41.782 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:41.782 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:41.782 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:41.782 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:41.782 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:41.782 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:41.783 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:41.783 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:41.783 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:41.783 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:41.783 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:34:41.783 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:34:41.783 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:34:41.783 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:41.783 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:34:41Z" 2024-04-20T01:34:41.783 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:34:41.784 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576752.746185_1427.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:32:32.746185Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:41Z" } 2024-04-20T01:34:41.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576752.746185_1427.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:32.746185Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:41.785 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:41.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:41.785 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:34:41.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:41.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:41.785 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:34:41.843 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:41.844 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:41.904 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-04-20T01:34:41.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:41.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:41.905 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --bucket aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:34:41.966 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:41.966 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:42.044 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-20T01:34:42.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:42.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:42.044 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:34:42.106 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:42.106 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:42.168 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-20T01:34:42.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:42.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:42.169 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:34:42.224 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:42.224 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:42.294 INFO:teuthology.orchestra.run.smithi019.stdout:{"sync_policy":{"groups":[]}} 2024-04-20T01:34:42.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:42.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'aktlji-79', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:42.295 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket aktlji-79 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:34:42.348 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:42.348 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:42.414 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[]} 2024-04-20T01:34:42.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:42.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:42.415 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:42.469 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:42.469 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:42.506 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:34:42.524 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:34:42.524 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:42.525 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:42.525 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:34:42.525 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:42.525 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:42.525 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:34:42.525 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:34:42.525 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:42.525 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:42.525 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:34:42.525 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:34:42.281271Z" 2024-04-20T01:34:42.525 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:34:42.526 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:42.526 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:34:42.527 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:42.527 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:34:42.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:42.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:42.527 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:42.587 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:42.588 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:42.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:42.646+0000 7fc706a53780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:42.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:42.647+0000 7fc706a53780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:42.656 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:34:42.656 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:34:42.656 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:34:42.657 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:34:42.657 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:34:42.657 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:42.657 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:42.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:42.657 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:34:42.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:42.657 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:34:42.657 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:42.657 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:42.657 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:42.658 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:42.658 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:34:42.658 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:42.658 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:42.658 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:42.658 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:42.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:42.658 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:42.658 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:34:42.658 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:42.658 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:42.659 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:34:42.659 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:42.659 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:42.659 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:42.659 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:34:42.659 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:42.659 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:42.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:42.659 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:42.659 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:34:42.659 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:42.660 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:42.660 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576881.894760_1453.1", 2024-04-20T01:34:42.660 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:42.660 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:42.660 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:42.660 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:41.894760Z", 2024-04-20T01:34:42.660 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:42.660 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:42.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:42.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:42.660 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:34:42.660 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:42.661 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:42.661 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:42.661 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:42.661 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:42.661 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:42.661 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:42.661 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:42.661 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:42.662 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:42.662 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:34:42.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:42.662 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:34:42.662 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:34:42.662 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:34:42.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:42.662 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:34:42Z" 2024-04-20T01:34:42.663 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:34:42.663 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576881.894760_1453.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:41.894760Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:42Z" } 2024-04-20T01:34:42.664 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576881.894760_1453.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:41.894760Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:42.664 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:42.664 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576881.894760_1453.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:42.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576882.281271_1460.1 target=1_1713576881.894760_1453.1 2024-04-20T01:34:47.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:47.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:47.666 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:47.725 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:47.726 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:47.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:47.790+0000 7f163367a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:47.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:47.791+0000 7f163367a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:47.806 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:34:47.806 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:34:47.806 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:34:47.806 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:34:47.807 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:34:47.807 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:47.807 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:47.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:47.807 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:34:47.807 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:47.807 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:34:47.807 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:47.807 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:47.807 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:47.807 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:47.808 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:34:47.808 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:47.808 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:47.808 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:47.808 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:47.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:47.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:47.808 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:34:47.808 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:47.808 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:47.808 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:34:47.808 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:47.809 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:47.809 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:47.809 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:34:47.809 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:47.809 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:47.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:47.809 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:47.809 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:34:47.809 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:47.809 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:47.809 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:34:47.810 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:47.810 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:47.810 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:47.810 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:34:47.810 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:47.810 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:47.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:47.810 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:47.810 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:34:47.810 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:47.810 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:47.811 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:47.811 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:47.811 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:47.811 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:47.811 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:47.811 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:47.811 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:47.811 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:47.811 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:34:47.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:47.811 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:34:47.811 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:34:47.812 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:34:47.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:47.812 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:34:47Z" 2024-04-20T01:34:47.812 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:34:47.813 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:47Z" } 2024-04-20T01:34:47.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:47.813 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:47.813 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:47.813 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:34:47.814 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:34:47.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:47.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:47.814 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:47.873 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:47.873 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:47.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:47.925+0000 7f6291b80780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:47.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:47.926+0000 7f6291b80780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:47.933 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:34:47.933 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:34:47.933 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:34:47.933 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:34:47.933 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:34:47.933 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:47.933 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:47.933 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:47.933 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:34:47.934 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:47.934 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:34:47.934 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:47.934 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:47.934 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:47.934 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:47.935 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:34:47.935 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:47.935 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:47.935 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:47.935 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:47.935 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:47.935 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:47.935 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:34:47.935 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:47.935 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:47.935 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:34:47.935 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:47.936 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:47.936 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:47.936 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:34:47.936 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:47.936 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:47.936 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:47.936 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:47.936 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:34:47.936 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:47.936 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:47.936 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576752.746185_1427.1", 2024-04-20T01:34:47.937 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:47.937 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:47.937 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:47.937 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:32:32.746185Z", 2024-04-20T01:34:47.937 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:47.937 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:47.937 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:47.937 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:47.937 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:34:47.938 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:47.938 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:47.938 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:47.938 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:47.938 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:47.938 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:47.938 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:47.938 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:47.938 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:47.938 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:47.938 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:47.939 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:47.939 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:34:47.939 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:34:47.939 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:34:47.939 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:47.939 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:34:47Z" 2024-04-20T01:34:47.939 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:34:47.940 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576752.746185_1427.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:32:32.746185Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:47Z" } 2024-04-20T01:34:47.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576752.746185_1427.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:32.746185Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:47.941 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:47.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:47.941 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1713576882.281271_1460.1 target=1_1713576752.746185_1427.1 2024-04-20T01:34:52.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:52.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:52.942 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:53.001 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:53.002 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:53.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:53.050+0000 7fa376dae780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:53.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:53.051+0000 7fa376dae780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:53.056 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:34:53.056 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:34:53.056 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:34:53.056 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:34:53.056 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:34:53.056 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:53.057 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:53.057 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:53.057 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:34:53.057 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:53.057 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:34:53.057 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:53.057 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:53.057 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:53.057 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:53.057 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:34:53.057 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:53.058 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:53.058 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:53.058 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:53.058 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:53.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:53.058 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:34:53.058 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:53.058 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:53.058 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:34:53.058 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:53.058 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:53.059 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:53.059 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:34:53.059 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:53.059 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:53.059 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:53.059 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:53.059 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:34:53.060 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:53.060 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:53.060 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576752.746185_1427.1", 2024-04-20T01:34:53.060 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:53.060 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:53.060 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:53.060 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:32:32.746185Z", 2024-04-20T01:34:53.060 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:53.060 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:53.060 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:53.060 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:53.061 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:34:53.061 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:53.061 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:53.061 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:53.061 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:53.061 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:53.061 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:53.061 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:53.061 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:53.061 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:53.061 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:53.061 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:53.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:53.062 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:34:53.062 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:34:53.062 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:34:53.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:53.062 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:34:53Z" 2024-04-20T01:34:53.062 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:34:53.063 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576752.746185_1427.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:32:32.746185Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:53Z" } 2024-04-20T01:34:53.063 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576752.746185_1427.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:32.746185Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:53.063 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:53.064 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576752.746185_1427.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:53.064 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1713576882.281271_1460.1 target=1_1713576752.746185_1427.1 2024-04-20T01:34:58.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:58.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:58.066 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:58.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:58.170+0000 7f8c63348780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:58.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:58.171+0000 7f8c63348780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:58.176 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:34:58.176 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:34:58.176 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:34:58.176 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:34:58.176 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:34:58.176 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:58.176 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:58.176 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.176 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:34:58.177 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:58.177 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:34:58.177 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:58.177 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:58.177 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:58.177 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:58.177 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:34:58.177 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:58.177 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:58.177 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:58.177 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:58.177 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.178 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:58.178 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:34:58.178 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:58.178 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:58.178 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:34:58.178 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:58.178 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:58.178 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:58.178 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:34:58.178 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:58.178 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:58.179 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.179 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:58.179 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:34:58.179 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:58.179 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:58.179 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:34:58.179 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:58.179 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:58.179 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:58.179 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:34:58.179 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:58.180 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:58.180 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.180 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:58.180 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:34:58.180 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:58.180 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:58.180 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:58.180 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:58.180 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:58.180 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:58.181 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:58.181 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:58.181 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:58.181 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:58.181 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:58.181 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.181 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:34:58.181 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:34:58.181 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:34:58.181 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.181 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:34:58Z" 2024-04-20T01:34:58.182 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:34:58.182 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:58Z" } 2024-04-20T01:34:58.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:58.183 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:58.183 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:58.183 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:34:58.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:58.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:58.184 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:34:58.240 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:58.240 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:58.298 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-04-20T01:34:58.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:58.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:58.299 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:34:58.360 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:58.360 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:58.427 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-20T01:34:58.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:58.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:58.428 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:34:58.486 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:58.486 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:58.557 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-20T01:34:58.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:58.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:58.557 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:34:58.610 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:58.610 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:58.672 INFO:teuthology.orchestra.run.smithi019.stdout:{"sync_policy":{"groups":[]}} 2024-04-20T01:34:58.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:58.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:58.672 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:34:58.722 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:58.722 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:58.777 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[]} 2024-04-20T01:34:58.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:58.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:58.778 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:34:58.831 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:58.831 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:58.895 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:34:58.895 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:58.895 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 10, 2024-04-20T01:34:58.895 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:34:58.895 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T01:34:58.895 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T01:34:58.895 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:58.895 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T01:34:58.896 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:58.896 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T01:34:58.896 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T01:34:58.896 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T01:34:58.896 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T01:34:58.896 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:34:58.896 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:34:58.896 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:34:58.896 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T01:34:58.896 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T01:34:58.896 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:34:58.896 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:34:58.897 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:58.897 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:34:58.897 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T01:34:58.897 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:34:58.897 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:34:58.897 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:34:58.897 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:34:58.897 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:34:58.897 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:34:58.897 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:34:58.897 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:34:58.898 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T01:34:58.898 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:34:58.898 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:34:58.898 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:34:58.898 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:34:58.898 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:34:58.898 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:58.898 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.898 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:58.898 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:34:58.898 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T01:34:58.899 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:34:58.899 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:34:58.899 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:34:58.899 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:34:58.899 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:34:58.899 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:34:58.899 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:34:58.899 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:34:58.899 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:34:58.899 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:34:58.899 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:34:58.900 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:34:58.900 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:34:58.900 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:34:58.900 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:58.900 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.900 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:58.900 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:34:58.900 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T01:34:58.900 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:34:58.900 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:34:58.900 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:34:58.901 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:34:58.901 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:34:58.901 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:34:58.901 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:34:58.901 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:34:58.901 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:34:58.901 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:34:58.901 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:34:58.901 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:34:58.901 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:34:58.902 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:34:58.902 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:58.902 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:58.902 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:34:58.902 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T01:34:58.902 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:58.902 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T01:34:58.902 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T01:34:58.902 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T01:34:58.902 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T01:34:58.903 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:58.903 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:58.903 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:34:58.903 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T01:34:58.903 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:34:58.903 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T01:34:58.903 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [] 2024-04-20T01:34:58.903 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.903 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T01:34:58.903 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:34:58.903 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:58.903 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:58.904 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:34:58.904 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T01:34:58.904 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:58.904 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:34:58.904 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T01:34:58.904 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.904 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:58.904 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:34:58.904 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T01:34:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:34:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T01:34:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.916 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T01:34:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:34:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T01:34:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T01:34:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:34:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:34:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:34:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:34:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:34:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.917 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T01:34:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:34:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:34:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:34:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:34:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:34:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T01:34:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:34:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:34:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:34:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:34:58.918 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:34:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T01:34:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:34:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:34:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:34:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:34:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:34:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T01:34:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:34:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:34:58.919 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:34:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:34:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:34:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:34:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T01:34:58.920 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:58.920 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:34:58.921 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2024-04-20T01:34:58.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:58.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:58.923 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:58.988 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:58.988 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:59.054 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:34:59.081 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:34:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:59.081 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:34:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:34:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:34:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:34:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:34:42.281271Z" 2024-04-20T01:34:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:59.082 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:34:59.083 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:59.083 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:34:59.084 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:59.084 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:34:59.084 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:59.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:59.085 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:59.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:59.257+0000 7ff71a661780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:59.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:34:59.260+0000 7ff71a661780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:59.301 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:34:59.301 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:34:59.301 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:34:59.302 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:34:59.302 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:34:59.302 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:59.302 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:59.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:59.302 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:34:59.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:59.302 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:34:59.302 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:59.303 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:59.303 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:59.303 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:59.303 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:34:59.303 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:59.303 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:59.303 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:59.303 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:59.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:59.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:59.303 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:34:59.304 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:59.304 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:59.304 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:34:59.304 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:59.304 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:59.304 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:59.304 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:34:59.304 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:59.304 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:59.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:59.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:59.305 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:34:59.305 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:59.305 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:59.305 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:34:59.305 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:59.305 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:59.305 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:59.305 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:34:59.305 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:59.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:59.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:59.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:34:59.306 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:34:59.306 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:34:59.306 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:34:59.306 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:59.306 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:34:59.306 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:34:59.306 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:34:59.306 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:59.306 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:34:59.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:59.307 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:34:59.307 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:34:59.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:59.307 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:34:59.307 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:34:59.307 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:34:59.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:34:59.307 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:34:59Z" 2024-04-20T01:34:59.307 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:34:59.308 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:59Z" } 2024-04-20T01:34:59.308 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:59.309 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:59.309 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:59.309 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:34:59.309 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:34:59.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:59.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:34:59.309 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:34:59.415 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:59.415 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:59.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:59.639+0000 7f892875d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:59.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:34:59.643+0000 7f892875d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:34:59.692 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:34:59.693 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:34:59.693 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:34:59.693 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:34:59.693 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:34:59.693 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:34:59.693 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:59.693 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:59.693 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:34:59.693 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:59.693 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:34:59.693 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:59.694 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:59.694 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:34:59.694 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:59.694 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:34:59.694 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:59.694 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:34:59.694 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:59.694 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:59.694 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:59.694 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:59.695 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:34:59.695 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:59.695 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:59.695 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:34:59.695 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:59.695 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:59.695 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:59.695 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:34:59.695 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:59.695 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:59.695 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:59.695 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:59.696 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:34:59.696 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:59.696 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:59.696 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:34:59.696 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:59.696 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:59.696 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:59.696 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:34:59.696 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:59.696 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:59.696 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:59.697 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:34:59.697 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:34:59.697 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:34:59.697 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:34:59.697 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:34:59.697 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:34:59.697 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:34:59.697 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:34:59.697 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:34:59.697 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:34:59.697 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:59.697 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:34:59.698 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:34:59.698 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:59.698 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:34:59.698 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:34:59.698 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:34:59.698 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:34:59.698 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:34:59Z" 2024-04-20T01:34:59.698 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:34:59.698 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:34:59Z" } 2024-04-20T01:34:59.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:34:59.699 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:59.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:34:59.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:34:59.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:59.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:59.700 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:34:59.780 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:34:59.780 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:34:59.935 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[]} 2024-04-20T01:34:59.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:59.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:34:59.936 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:35:00.009 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:00.010 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:00.116 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-20T01:35:00.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:35:00.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:35:00.117 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:35:00.192 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:00.192 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:00.297 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-20T01:35:00.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:35:00.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:35:00.298 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:35:00.362 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:00.362 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:00.420 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:35:00.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:35:00.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:35:00.421 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:35:00.476 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:00.476 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:00.542 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:35:00.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:35:00.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:35:00.543 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:35:00.597 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:00.597 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:00.663 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:35:00.663 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:00.664 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 11, 2024-04-20T01:35:00.664 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:35:00.664 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T01:35:00.664 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T01:35:00.664 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:00.664 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T01:35:00.664 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:00.664 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T01:35:00.664 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T01:35:00.664 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T01:35:00.664 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T01:35:00.665 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:35:00.665 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:35:00.665 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:35:00.665 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T01:35:00.665 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T01:35:00.665 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:35:00.665 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:35:00.665 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:00.665 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:35:00.665 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T01:35:00.665 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:35:00.666 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:35:00.666 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:35:00.666 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:35:00.666 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:35:00.666 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:35:00.666 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:35:00.666 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:35:00.666 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T01:35:00.666 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:35:00.666 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:35:00.666 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:35:00.667 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:35:00.667 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:35:00.667 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:35:00.667 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.667 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:00.667 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:35:00.667 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T01:35:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:35:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:35:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:35:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:35:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:35:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:35:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:35:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:35:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:35:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:35:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:35:00.668 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:35:00.669 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:35:00.669 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:35:00.669 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:35:00.669 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.669 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:00.669 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:35:00.669 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T01:35:00.669 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:35:00.669 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:35:00.669 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:35:00.669 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:35:00.669 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:35:00.670 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:35:00.670 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:35:00.670 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:35:00.670 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:35:00.670 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:35:00.670 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:35:00.670 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:35:00.670 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:35:00.670 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:35:00.670 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:35:00.670 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:35:00.671 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:35:00.671 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T01:35:00.671 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:00.671 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T01:35:00.671 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T01:35:00.671 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T01:35:00.672 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T01:35:00.672 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:35:00.672 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:35:00.672 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:35:00.672 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T01:35:00.672 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:35:00.672 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T01:35:00.672 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [ 2024-04-20T01:35:00.672 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:00.673 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-group", 2024-04-20T01:35:00.673 INFO:teuthology.orchestra.run.smithi019.stdout: "data_flow": { 2024-04-20T01:35:00.673 INFO:teuthology.orchestra.run.smithi019.stdout: "symmetrical": [ 2024-04-20T01:35:00.673 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:00.673 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow1", 2024-04-20T01:35:00.673 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:35:00.673 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:35:00.679 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:35:00.680 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:35:00.680 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:35:00.680 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:35:00.680 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.680 INFO:teuthology.orchestra.run.smithi019.stdout: "pipes": [ 2024-04-20T01:35:00.680 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:00.680 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe", 2024-04-20T01:35:00.680 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:35:00.680 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "*", 2024-04-20T01:35:00.680 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:35:00.681 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:35:00.681 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:35:00.681 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:35:00.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.681 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:35:00.681 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "*", 2024-04-20T01:35:00.681 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:35:00.681 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:35:00.681 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:35:00.681 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:35:00.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.682 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:35:00.682 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:35:00.682 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:35:00.682 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:35:00.682 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:35:00.682 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.682 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:35:00.682 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:35:00.682 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:35:00.682 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:35:00.683 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:35:00.683 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:35:00.683 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:35:00.683 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "enabled" 2024-04-20T01:35:00.683 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:35:00.683 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:35:00.683 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.683 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T01:35:00.683 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:35:00.683 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:35:00.683 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:35:00.684 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:35:00.684 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T01:35:00.684 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:00.684 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:35:00.684 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T01:35:00.684 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.684 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:00.684 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:35:00.684 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T01:35:00.684 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.684 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:00.685 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:35:00.685 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T01:35:00.685 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:35:00.685 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:35:00.685 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.685 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T01:35:00.685 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:35:00.685 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T01:35:00.685 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T01:35:00.685 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:35:00.685 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:35:00.686 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:35:00.686 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:35:00.686 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:35:00.686 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.686 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T01:35:00.686 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:35:00.686 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:35:00.686 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:35:00.686 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:35:00.686 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:35:00.686 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.687 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T01:35:00.687 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:35:00.687 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:35:00.687 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:35:00.687 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:35:00.687 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:35:00.687 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.687 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T01:35:00.687 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:35:00.687 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:35:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:35:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:35:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:35:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T01:35:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:35:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:35:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:35:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:35:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:35:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:35:00.689 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:00.689 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:35:00.689 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T01:35:00.689 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:35:00.689 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:35:00.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:35:00.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:35:00.690 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:35:00.746 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:00.746 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:00.836 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:35:00.836 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-80 2024-04-20T01:35:00.837 DEBUG:boto:path=/aktlji-80/ 2024-04-20T01:35:00.837 DEBUG:boto:auth_path=/aktlji-80/ 2024-04-20T01:35:00.837 DEBUG:boto:Method: PUT 2024-04-20T01:35:00.837 DEBUG:boto:Path: /aktlji-80/ 2024-04-20T01:35:00.837 DEBUG:boto:Data: 2024-04-20T01:35:00.837 DEBUG:boto:Headers: {} 2024-04-20T01:35:00.837 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:35:00.837 DEBUG:boto:Port: 8000 2024-04-20T01:35:00.838 DEBUG:boto:Params: {} 2024-04-20T01:35:00.838 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:35:00.838 DEBUG:boto:Token: None 2024-04-20T01:35:00.839 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:35:00 GMT /aktlji-80/ 2024-04-20T01:35:00.839 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zHY8sgPC+DI1hitUnsdcb57Lo1w= 2024-04-20T01:35:00.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zHY8sgPC+DI1hitUnsdcb57Lo1w=', 'Content-Length': '0'} 2024-04-20T01:35:00.913 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f59b5f2b80a09323-0066231bc4-1a14-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:35:00.913 DEBUG:boto:path=/aktlji-80/ 2024-04-20T01:35:00.914 DEBUG:boto:auth_path=/aktlji-80/ 2024-04-20T01:35:00.914 DEBUG:boto:Method: HEAD 2024-04-20T01:35:00.914 DEBUG:boto:Path: /aktlji-80/ 2024-04-20T01:35:00.914 DEBUG:boto:Data: 2024-04-20T01:35:00.914 DEBUG:boto:Headers: {} 2024-04-20T01:35:00.914 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:35:00.914 DEBUG:boto:Port: 8000 2024-04-20T01:35:00.914 DEBUG:boto:Params: {} 2024-04-20T01:35:00.914 DEBUG:boto:Token: None 2024-04-20T01:35:00.914 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:35:00 GMT /aktlji-80/ 2024-04-20T01:35:00.915 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fAFCTFM4J34pdIUZpc84TECl3iw= 2024-04-20T01:35:00.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fAFCTFM4J34pdIUZpc84TECl3iw=', 'Content-Length': '0'} 2024-04-20T01:35:00.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000485dba4c616e5070-0066231bc4-1a14-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:35:00.919 DEBUG:boto:path=/aktlji-80/obj1 2024-04-20T01:35:00.919 DEBUG:boto:auth_path=/aktlji-80/obj1 2024-04-20T01:35:00.919 DEBUG:boto:Method: PUT 2024-04-20T01:35:00.919 DEBUG:boto:Path: /aktlji-80/obj1 2024-04-20T01:35:00.919 DEBUG:boto:Data: 2024-04-20T01:35:00.919 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:35:00.919 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:35:00.919 DEBUG:boto:Port: 8000 2024-04-20T01:35:00.919 DEBUG:boto:Params: {} 2024-04-20T01:35:00.920 DEBUG:boto:Token: None 2024-04-20T01:35:00.920 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:35:00 GMT /aktlji-80/obj1 2024-04-20T01:35:00.920 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a6277L6S+qIjU7QVC4K1REkSw2A= 2024-04-20T01:35:00.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a6277L6S+qIjU7QVC4K1REkSw2A='} 2024-04-20T01:35:00.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c554f60784784f91-0066231bc4-1a14-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:35:00.942 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-81 2024-04-20T01:35:00.943 DEBUG:boto:path=/aktlji-81/ 2024-04-20T01:35:00.943 DEBUG:boto:auth_path=/aktlji-81/ 2024-04-20T01:35:00.943 DEBUG:boto:Method: PUT 2024-04-20T01:35:00.943 DEBUG:boto:Path: /aktlji-81/ 2024-04-20T01:35:00.943 DEBUG:boto:Data: 2024-04-20T01:35:00.943 DEBUG:boto:Headers: {} 2024-04-20T01:35:00.943 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:35:00.943 DEBUG:boto:Port: 8000 2024-04-20T01:35:00.943 DEBUG:boto:Params: {} 2024-04-20T01:35:00.944 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:35:00.944 DEBUG:boto:Token: None 2024-04-20T01:35:00.944 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:35:00 GMT /aktlji-81/ 2024-04-20T01:35:00.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:IRVvqCy6B104OvfM8472GTwhsSc= 2024-04-20T01:35:00.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IRVvqCy6B104OvfM8472GTwhsSc=', 'Content-Length': '0'} 2024-04-20T01:35:01.021 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000f010139d9eb59ce-0066231bc4-214b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:35:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:35:01.021 DEBUG:boto:path=/aktlji-81/ 2024-04-20T01:35:01.022 DEBUG:boto:auth_path=/aktlji-81/ 2024-04-20T01:35:01.022 DEBUG:boto:Method: HEAD 2024-04-20T01:35:01.022 DEBUG:boto:Path: /aktlji-81/ 2024-04-20T01:35:01.022 DEBUG:boto:Data: 2024-04-20T01:35:01.022 DEBUG:boto:Headers: {} 2024-04-20T01:35:01.022 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:35:01.022 DEBUG:boto:Port: 8000 2024-04-20T01:35:01.022 DEBUG:boto:Params: {} 2024-04-20T01:35:01.022 DEBUG:boto:Token: None 2024-04-20T01:35:01.023 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:35:01 GMT /aktlji-81/ 2024-04-20T01:35:01.023 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fdpx45D434IPtURLvtz2oxM3lqs= 2024-04-20T01:35:01.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:35:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fdpx45D434IPtURLvtz2oxM3lqs=', 'Content-Length': '0'} 2024-04-20T01:35:01.026 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084b2fd1df345d854-0066231bc5-214b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:35:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:35:01.027 DEBUG:boto:path=/aktlji-81/obj2 2024-04-20T01:35:01.027 DEBUG:boto:auth_path=/aktlji-81/obj2 2024-04-20T01:35:01.027 DEBUG:boto:Method: PUT 2024-04-20T01:35:01.027 DEBUG:boto:Path: /aktlji-81/obj2 2024-04-20T01:35:01.027 DEBUG:boto:Data: 2024-04-20T01:35:01.027 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:35:01.027 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:35:01.027 DEBUG:boto:Port: 8000 2024-04-20T01:35:01.027 DEBUG:boto:Params: {} 2024-04-20T01:35:01.027 DEBUG:boto:Token: None 2024-04-20T01:35:01.028 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:35:01 GMT /aktlji-81/obj2 2024-04-20T01:35:01.028 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o6lT7M6/Il+i86f+6ub2qoh4a1U= 2024-04-20T01:35:01.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:35:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o6lT7M6/Il+i86f+6ub2qoh4a1U='} 2024-04-20T01:35:01.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000677ee53ed9fef1f7-0066231bc5-214b-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:35:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:35:01.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:01.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:01.053 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:01.142 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:01.142 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:01.245 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:35:01.287 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:35:01.287 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:01.288 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:01.288 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:35:01.288 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:01.288 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:01.288 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:35:01.288 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:35:01.288 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:01.288 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:01.288 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:35:01.288 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:35:01.005366Z" 2024-04-20T01:35:01.289 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:35:01.289 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:35:01.289 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:01.289 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:35:01.289 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:35:01.289 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:35:01.290 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:01.290 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:35:01.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:01.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:01.291 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:01.350 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:01.351 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:01.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:01.431+0000 7fef99dbf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:01.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:01.432+0000 7fef99dbf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:01.449 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:35:01.450 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:35:01.450 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:35:01.450 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:35:01.450 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:35:01.450 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:01.450 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:01.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:01.450 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:35:01.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:01.451 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:35:01.451 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:01.451 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:01.451 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:01.451 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:01.451 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:35:01.451 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:01.451 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:35:01.451 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:01.451 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:01.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:01.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:01.452 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:35:01.452 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:01.452 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:01.452 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:35:01.452 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:01.452 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:01.452 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:01.452 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:35:01.452 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:01.452 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:01.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:01.454 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:01.454 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:35:01.454 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:01.454 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:01.454 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:01.454 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:01.454 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:01.454 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:01.454 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:35:01.455 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:01.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:01.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:01.455 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:35:01.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:01.455 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:35:01.455 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:35:01.455 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:35:01.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:01.455 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:35:01Z" 2024-04-20T01:35:01.455 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:35:01.456 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:35:01Z" } 2024-04-20T01:35:01.456 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:35:01.457 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:01.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:01.457 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:35:06.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:06.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:06.459 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:06.522 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:06.522 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:06.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:06.588+0000 7f10575ac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:06.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:06.589+0000 7f10575ac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:06.605 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:35:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:35:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:35:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:35:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:35:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:35:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:35:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:35:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:35:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:35:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:35:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:35:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:35:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:35:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:35:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:35:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:35:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:35:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:35:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:35:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:35:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:35:06Z" 2024-04-20T01:35:06.611 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:35:06.612 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:35:06Z" } 2024-04-20T01:35:06.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:35:06.613 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:06.613 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:06.613 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:35:11.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:11.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:11.614 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:11.673 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:11.725 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:11.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:11.734+0000 7f7853221780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:11.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:11.735+0000 7f7853221780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:11.745 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:35:11.745 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:35:11.745 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:35:11.745 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:35:11.745 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:35:11.745 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:11.745 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:11.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:11.746 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:35:11.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:11.746 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:35:11.746 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:11.746 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:11.746 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:11.746 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:11.746 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:35:11.746 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:11.746 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:35:11.747 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:11.747 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:11.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:11.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:11.747 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:35:11.747 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:11.747 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:11.747 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:35:11.747 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:11.747 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:11.747 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:11.748 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:35:11.748 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:11.748 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:11.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:11.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:11.748 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:35:11.748 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:11.748 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:11.748 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:35:11.748 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:11.748 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:11.749 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:11.749 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:35:11.749 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:11.749 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:11.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:11.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:11.749 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:35:11.749 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:11.749 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:11.749 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:11.750 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:11.750 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:11.750 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:11.750 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:35:11.750 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:11.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:11.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:11.750 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:35:11.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:11.750 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:35:11.750 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:35:11.751 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:35:11.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:11.751 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:35:11Z" 2024-04-20T01:35:11.751 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:35:11.752 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:35:11Z" } 2024-04-20T01:35:11.752 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:35:11.752 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:11.752 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:11.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:35:16.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:16.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:16.754 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:16.816 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:16.816 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:16.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:16.874+0000 7f2b88432780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:16.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:16.875+0000 7f2b88432780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:16.885 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:35:16.885 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:35:16.885 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:35:16.885 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:35:16.885 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:35:16.885 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:16.885 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:16.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:16.886 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:35:16.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:16.886 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:35:16.886 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:16.886 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:16.886 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:16.886 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:16.886 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:35:16.886 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:16.886 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:35:16.886 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:16.887 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:16.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:16.887 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:16.887 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:35:16.887 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:16.887 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:16.887 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:35:16.887 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:16.887 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:16.887 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:16.887 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:35:16.888 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:16.888 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:16.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:16.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:16.888 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:35:16.888 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:16.888 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:16.888 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:35:16.888 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:16.888 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:16.888 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:16.888 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:35:16.889 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:16.889 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:16.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:16.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:16.889 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:35:16.889 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:16.889 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:16.889 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:16.889 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:16.889 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:16.890 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:16.890 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:35:16.890 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:16.890 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:16.890 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:16.890 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:35:16.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:16.890 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:35:16.890 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:35:16.890 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:35:16.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:16.891 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:35:16Z" 2024-04-20T01:35:16.891 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:35:16.892 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:35:16Z" } 2024-04-20T01:35:16.892 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:35:16.892 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:16.892 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:16.892 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:35:21.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:21.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:21.894 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:21.954 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:21.954 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:22.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:22.015+0000 7f8f0446e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:22.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:22.016+0000 7f8f0446e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:22.029 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:35:22.030 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:35:22.030 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:35:22.030 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:35:22.030 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:35:22.030 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:22.030 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:22.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:22.030 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:35:22.030 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:22.031 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:35:22.031 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:22.031 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:22.031 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:22.031 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:22.031 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:35:22.031 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:22.031 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:35:22.031 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:22.031 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:22.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:22.032 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:22.032 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:35:22.032 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:22.032 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:22.032 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:35:22.032 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:22.032 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:22.032 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:22.032 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:35:22.032 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:22.033 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:22.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:22.033 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:22.033 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:35:22.033 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:22.033 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:22.033 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:35:22.033 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:22.033 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:22.034 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:22.034 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:35:22.034 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:22.034 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:22.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:22.034 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:22.034 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:35:22.034 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:22.034 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:22.034 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:22.034 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:22.035 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:22.035 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:22.035 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:35:22.035 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:22.035 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:22.035 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:22.035 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:35:22.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:22.035 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:35:22.035 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:35:22.035 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:35:22.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:22.036 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:35:22Z" 2024-04-20T01:35:22.036 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:35:22.037 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:35:22Z" } 2024-04-20T01:35:22.037 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:35:22.038 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:22.038 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:22.038 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:35:27.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:27.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:27.040 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:27.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:27.152+0000 7f0d7c734780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:27.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:27.154+0000 7f0d7c734780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:27.161 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:35:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:35:27.161 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:35:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:35:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:35:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:35:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:35:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:27.162 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:35:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:35:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:35:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:27.163 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:35:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:35:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:27.164 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:35:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:35:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:35:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:27.165 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:35:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:35:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:27.166 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:35:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:35:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:35:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:35:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:27.167 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:35:27Z" 2024-04-20T01:35:27.167 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:35:27.169 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:35:27Z" } 2024-04-20T01:35:27.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:35:27.170 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:27.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:27.170 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:35:32.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:32.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:32.172 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:32.231 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:32.231 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:32.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:32.292+0000 7fbd8f54f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:32.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:32.293+0000 7fbd8f54f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:32.303 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:35:32.303 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:35:32.303 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:35:32.304 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:35:32.304 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:35:32.304 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:32.304 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:32.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:32.304 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:35:32.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:32.304 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:35:32.304 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:32.304 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:32.304 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:35:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:35:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:35:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:32.305 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:35:32.306 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:32.306 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:32.306 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:32.306 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:35:32.306 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:32.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:32.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:32.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:32.306 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:35:32.306 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:32.306 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:32.307 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:35:32.307 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:32.307 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:32.307 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:32.307 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:35:32.307 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:32.307 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:32.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:32.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:32.307 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:35:32.307 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:32.307 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:35:32.308 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:35:32.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:32.309 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:35:32Z" 2024-04-20T01:35:32.309 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:35:32.310 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:35:32Z" } 2024-04-20T01:35:32.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:35:32.310 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:32.310 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:32.310 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:35:37.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:37.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:37.312 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:37.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:37.432+0000 7f0524492780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:37.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:37.434+0000 7f0524492780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:37.448 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:35:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:35:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:35:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:35:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:35:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:35:37.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:35:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:35:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:35:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:37.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:35:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:35:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:35:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:37.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:37.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:35:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:37.453 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:35:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:35:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:35:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:35:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:35:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:37.454 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:35:37Z" 2024-04-20T01:35:37.455 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:35:37.455 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:35:37Z" } 2024-04-20T01:35:37.456 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:35:37.456 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:37.456 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:37.456 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:35:42.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:42.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:42.457 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:42.516 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:42.516 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:42.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:42.568+0000 7fae5901b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:42.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:42.569+0000 7fae5901b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:42.576 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:35:42.576 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:35:42.576 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:35:42.576 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:35:42.576 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:35:42.577 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:42.577 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:42.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:42.577 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:35:42.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:42.577 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:35:42.577 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:42.577 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:42.577 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:42.577 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:42.577 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:35:42.577 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:42.578 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:35:42.578 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:42.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:42.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:42.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:42.578 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:35:42.578 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:42.578 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:42.578 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:35:42.578 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:42.578 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:42.579 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:42.579 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:35:42.579 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:42.579 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:42.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:42.579 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:42.579 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:35:42.579 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:42.579 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:42.579 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:35:42.579 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:42.579 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:42.580 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:42.580 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:35:42.580 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:42.580 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:42.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:42.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:42.580 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:35:42.580 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:42.580 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:42.580 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:42.580 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:42.581 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:42.581 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:42.581 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:35:42.581 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:42.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:42.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:42.581 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:35:42.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:42.581 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:35:42.581 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:35:42.581 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:35:42.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:42.582 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:35:42Z" 2024-04-20T01:35:42.582 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:35:42.582 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:35:42Z" } 2024-04-20T01:35:42.583 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:35:42.583 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:42.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:42.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:35:47.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:47.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:47.584 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:47.643 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:47.644 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:47.716+0000 7f221ed32780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:47.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:47.717+0000 7f221ed32780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:47.724 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:35:47.724 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:35:47.724 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:35:47.725 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:35:47.725 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:35:47.725 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:47.725 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:47.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:47.725 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:35:47.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:47.725 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:35:47.725 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:47.725 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:47.725 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:47.726 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:47.726 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:35:47.726 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:47.726 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:35:47.726 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:47.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:47.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:47.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:47.726 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:35:47.726 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:47.727 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:47.727 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:35:47.727 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:47.727 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:47.727 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:47.727 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:35:47.727 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:47.727 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:47.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:47.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:47.728 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:35:47.728 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:47.728 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:47.728 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:35:47.728 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:47.728 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:47.728 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:47.728 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:35:47.728 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:47.728 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:47.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:47.730 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:35:47.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:47.730 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:35:47.730 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:35:47.730 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:35:47.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:47.730 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:35:47Z" 2024-04-20T01:35:47.730 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:35:47.731 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:35:47Z" } 2024-04-20T01:35:47.731 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:35:47.731 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:47.732 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:47.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:35:52.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:52.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:52.733 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:52.793 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:52.793 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:52.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:52.851+0000 7f94a7271780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:52.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:52.852+0000 7f94a7271780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:52.859 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:35:52.859 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:35:52.859 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:35:52.859 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:35:52.860 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:35:52.860 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:52.860 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:52.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:52.860 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:35:52.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:52.860 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:35:52.860 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:52.860 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:52.860 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:52.861 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:52.861 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:35:52.861 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:52.861 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:35:52.861 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:52.861 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:52.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:52.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:52.861 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:35:52.861 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:52.861 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:52.861 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:35:52.862 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:52.862 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:52.862 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:52.862 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:35:52.862 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:52.862 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:52.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:52.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:52.862 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:35:52.863 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:52.863 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:52.863 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:35:52.863 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:52.863 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:52.863 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:52.863 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:35:52.863 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:52.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:52.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:52.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:52.863 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:35:52.864 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:52.864 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:52.864 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:52.864 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:52.864 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:52.864 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:52.864 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:35:52.864 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:52.864 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:52.864 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:52.864 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:35:52.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:52.865 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:35:52.865 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:35:52.865 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:35:52.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:52.865 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:35:52Z" 2024-04-20T01:35:52.865 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:35:52.866 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:35:52Z" } 2024-04-20T01:35:52.866 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:35:52.866 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:52.866 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:52.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:35:57.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:57.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:35:57.868 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:35:57.928 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:35:57.928 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:35:57.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:57.991+0000 7f0825f98780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:57.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:35:57.992+0000 7f0825f98780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:35:58.009 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:35:58.009 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:35:58.009 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:35:58.009 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:35:58.009 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:35:58.009 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:35:58.010 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:58.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:58.010 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:35:58.010 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:58.010 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:35:58.010 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:58.010 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:58.010 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:35:58.010 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:58.010 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:35:58.010 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:58.011 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:35:58.011 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:58.011 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:58.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:58.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:58.011 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:35:58.011 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:58.011 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:58.011 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:35:58.011 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:58.012 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:58.012 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:58.012 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:35:58.012 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:58.012 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:58.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:58.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:58.012 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:35:58.012 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:58.012 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:58.012 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:35:58.013 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:58.013 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:58.013 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:58.013 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:35:58.013 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:58.013 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:58.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:58.013 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:35:58.013 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:35:58.013 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:35:58.013 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:35:58.013 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:35:58.014 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:35:58.014 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:35:58.014 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:35:58.014 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:35:58.014 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:35:58.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:58.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:35:58.014 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:35:58.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:58.014 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:35:58.014 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:35:58.015 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:35:58.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:35:58.015 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:35:57Z" 2024-04-20T01:35:58.015 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:35:58.016 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:35:57Z" } 2024-04-20T01:35:58.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:35:58.016 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:58.016 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:35:58.016 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:36:03.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:03.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:03.018 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:03.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:03.133+0000 7f799a9fa780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:03.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:03.135+0000 7f799a9fa780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:03.143 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:36:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:36:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:36:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:36:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:36:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:36:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:36:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:36:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:36:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:36:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:36:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:36:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:36:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:36:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:36:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:36:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:36:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:36:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:36:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:36:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:36:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:36:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:36:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:36:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:36:03Z" 2024-04-20T01:36:03.150 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:36:03.150 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:36:03Z" } 2024-04-20T01:36:03.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:36:03.151 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:03.151 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:03.151 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:36:08.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:08.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:08.152 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:08.211 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:08.212 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:08.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:08.272+0000 7f2c78fac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:08.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:08.274+0000 7f2c78fac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:08.282 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:36:08.282 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:36:08.282 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:36:08.283 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:36:08.283 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:36:08.283 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:36:08.283 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:08.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:08.283 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:36:08.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:08.283 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:36:08.283 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:08.283 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:08.284 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:36:08.284 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:08.284 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:36:08.284 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:08.284 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:36:08.284 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:08.284 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:08.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:08.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:08.285 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:36:08.285 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:08.285 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:08.285 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:36:08.285 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:08.285 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:08.285 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:08.285 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:36:08.285 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:08.285 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:08.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:08.286 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:08.286 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:36:08.286 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:08.286 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:08.286 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:36:08.286 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:08.286 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:08.286 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:08.286 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:36:08.287 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:08.287 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:08.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:08.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:08.287 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:36:08.287 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:08.287 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:08.287 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:36:08.287 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:08.287 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:08.288 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:08.288 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:36:08.288 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:08.288 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:08.288 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:08.288 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:36:08.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:08.288 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:36:08.288 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:36:08.288 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:36:08.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:08.289 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:36:08Z" 2024-04-20T01:36:08.289 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:36:08.290 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:36:08Z" } 2024-04-20T01:36:08.290 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:36:08.290 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:08.290 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:08.290 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:36:13.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:13.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:13.292 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:13.352 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:13.352 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:13.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:13.410+0000 7f7abf470780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:13.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:13.411+0000 7f7abf470780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:13.418 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:36:13.418 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:36:13.418 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:36:13.418 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:36:13.419 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:36:13.419 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:36:13.419 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:13.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:13.419 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:36:13.419 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:13.419 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:36:13.419 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:13.419 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:13.419 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:36:13.419 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:13.420 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:36:13.420 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:13.420 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:36:13.420 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:13.420 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:13.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:13.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:13.420 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:36:13.420 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:13.420 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:13.420 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:36:13.421 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:13.421 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:13.421 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:13.421 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:36:13.421 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:13.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:13.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:13.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:13.421 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:36:13.421 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:13.422 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:13.422 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:36:13.422 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:13.422 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:13.422 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:13.422 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:36:13.422 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:13.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:13.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:13.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:13.422 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:36:13.423 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:13.423 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:13.423 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:36:13.423 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:13.423 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:13.423 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:13.423 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:36:13.423 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:13.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:13.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:13.423 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:36:13.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:13.424 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:36:13.424 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:36:13.424 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:36:13.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:13.424 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:36:13Z" 2024-04-20T01:36:13.424 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:36:13.425 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:36:13Z" } 2024-04-20T01:36:13.425 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:36:13.425 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:13.425 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:13.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:36:18.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:18.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:18.427 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:18.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:18.562+0000 7f5759bf9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:18.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:18.563+0000 7f5759bf9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:18.574 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:36:18.574 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:36:18.574 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:36:18.574 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:36:18.574 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:36:18.574 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:36:18.575 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:18.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:18.575 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:36:18.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:18.575 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:36:18.575 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:18.575 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:18.575 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:36:18.575 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:18.575 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:36:18.575 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:18.575 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:36:18.576 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:18.577 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:18.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:18.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:18.577 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:36:18.577 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:18.577 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:18.577 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:36:18.577 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:18.577 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:18.577 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:18.577 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:36:18.577 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:18.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:18.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:18.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:18.578 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:36:18.578 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:18.578 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:18.578 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:36:18.578 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:18.578 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:18.578 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:18.578 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:36:18.579 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:18.579 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:18.579 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:18.579 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:36:18.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:18.579 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:36:18.579 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:36:18.579 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:36:18.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:18.579 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:36:18Z" 2024-04-20T01:36:18.579 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:36:18.580 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:36:18Z" } 2024-04-20T01:36:18.580 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:36:18.581 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:18.581 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:18.581 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:36:23.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:23.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:23.582 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:23.641 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:23.641 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:23.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:23.696+0000 7f925dd53780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:23.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:23.697+0000 7f925dd53780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:23.704 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:36:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:36:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:36:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:36:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:36:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:36:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:36:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:36:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:36:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:36:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:36:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:36:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:36:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:36:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:36:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:36:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:36:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:36:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:36:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:36:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:36:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:36:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:36:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:36:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:36:23Z" 2024-04-20T01:36:23.710 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:36:23.711 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:36:23Z" } 2024-04-20T01:36:23.711 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:36:23.711 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:23.711 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:23.712 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:36:28.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:28.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:28.713 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:28.772 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:28.772 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:28.831+0000 7ff907d2d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:28.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:28.832+0000 7ff907d2d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:28.843 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:36:28.843 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:36:28.843 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:36:28.843 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:36:28.843 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:36:28.843 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:36:28.844 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:28.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:28.844 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:36:28.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:28.844 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:36:28.844 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:28.844 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:28.844 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:36:28.844 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:28.844 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:36:28.845 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:28.845 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:36:28.845 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:28.845 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:28.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:28.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:28.845 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:36:28.845 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:28.845 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:28.845 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:36:28.845 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:28.845 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:28.846 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:28.846 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:36:28.846 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:28.846 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:28.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:28.846 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:28.846 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:36:28.846 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:28.846 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:28.846 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:36:28.846 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:28.847 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:28.847 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:28.847 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:36:28.847 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:28.847 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:28.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:28.847 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:28.847 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:36:28.847 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:28.847 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:28.847 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:36:28.848 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:28.848 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:28.848 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:28.848 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:36:28.848 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:28.848 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:28.848 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:28.848 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:36:28.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:28.848 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:36:28.848 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:36:28.849 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:36:28.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:28.849 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:36:28Z" 2024-04-20T01:36:28.849 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:36:28.850 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:36:28Z" } 2024-04-20T01:36:28.850 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:36:28.850 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:28.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:28.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:36:33.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:33.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:33.852 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:33.911 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:33.911 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:33.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:33.971+0000 7f3ec43d8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:33.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:33.972+0000 7f3ec43d8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:33.979 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:36:33.979 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:36:33.979 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:36:33.979 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:36:33.979 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:36:33.979 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:36:33.979 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:33.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:33.980 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:36:33.980 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:33.980 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:36:33.980 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:33.980 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:33.980 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:36:33.980 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:33.980 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:36:33.980 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:33.980 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:36:33.980 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:33.981 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:33.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:33.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:33.981 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:36:33.981 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:33.981 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:33.981 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:36:33.981 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:33.981 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:33.981 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:33.981 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:36:33.982 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:33.982 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:33.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:33.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:33.982 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:36:33.982 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:33.982 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:33.982 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:36:33.982 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:33.982 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:33.982 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:33.983 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:36:33.983 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:33.983 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:33.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:33.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:33.983 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:36:33.983 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:33.983 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:33.983 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:36:33.983 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:33.983 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:33.983 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:33.984 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:36:33.984 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:33.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:33.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:33.984 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:36:33.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:33.984 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:36:33.984 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:36:33.984 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:36:33.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:33.984 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:36:33Z" 2024-04-20T01:36:33.985 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:36:33.985 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:36:33Z" } 2024-04-20T01:36:33.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:36:33.986 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:33.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:33.986 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:36:38.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:38.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:38.988 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:39.049 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:39.049 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:39.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:39.111+0000 7f9c5d388780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:39.113+0000 7f9c5d388780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:39.127 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:36:39.127 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:36:39.127 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:36:39.127 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:36:39.127 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:36:39.128 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:36:39.128 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:39.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:39.128 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:36:39.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:39.128 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:36:39.128 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:39.128 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:39.128 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:36:39.129 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:39.129 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:36:39.129 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:39.129 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:36:39.129 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:39.129 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:39.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:39.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:39.129 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:36:39.129 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:39.129 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:39.130 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:36:39.130 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:39.130 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:39.130 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:39.130 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:36:39.130 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:39.130 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:39.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:39.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:39.130 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:36:39.130 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:39.131 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:39.131 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:36:39.131 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:39.131 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:39.131 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:39.131 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:36:39.131 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:39.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:39.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:39.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:39.131 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:36:39.132 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:39.132 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:39.132 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:36:39.132 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:39.132 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:39.132 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:39.132 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:36:39.132 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:39.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:39.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:39.132 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:36:39.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:39.133 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:36:39.133 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:36:39.133 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:36:39.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:39.133 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:36:39Z" 2024-04-20T01:36:39.133 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:36:39.134 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:36:39Z" } 2024-04-20T01:36:39.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:36:39.134 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:39.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:39.135 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:36:44.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:44.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:44.136 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:44.193 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:44.193 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:44.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:44.254+0000 7f369a4b6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:44.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:44.255+0000 7f369a4b6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:44.267 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:36:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:36:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:36:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:36:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:36:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:36:44.267 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:36:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:36:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:36:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:36:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:36:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:36:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:36:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:44.269 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:36:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:36:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:36:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:44.270 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:36:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:36:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:36:44.271 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:36:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:36:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:36:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:36:44.272 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:36:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:44.273 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:36:44Z" 2024-04-20T01:36:44.273 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:36:44.274 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:36:44Z" } 2024-04-20T01:36:44.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:36:44.274 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:44.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:44.274 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:36:49.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:49.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:49.276 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:49.335 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:49.335 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:49.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:49.405+0000 7faa9e96d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:49.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:49.406+0000 7faa9e96d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:49.413 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:36:49.413 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:36:49.413 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:36:49.414 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:36:49.414 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:36:49.414 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:36:49.414 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:49.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:49.414 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:36:49.414 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:49.414 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:36:49.414 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:49.414 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:49.415 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:36:49.416 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:49.416 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:49.416 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:49.416 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:36:49.416 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:49.416 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:49.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:49.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:49.416 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:36:49.416 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:49.416 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:49.416 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:36:49.417 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:49.417 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:49.417 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:49.417 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:36:49.417 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:49.417 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:49.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:49.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:49.417 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:36:49.417 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:49.417 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:49.417 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:36:49.418 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:49.418 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:49.418 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:49.418 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:36:49.418 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:49.418 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:49.418 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:49.418 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:36:49.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:49.418 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:36:49.418 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:36:49.419 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:36:49.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:49.419 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:36:49Z" 2024-04-20T01:36:49.419 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:36:49.420 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576882.281271_1460.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:34:42.281271Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:36:49Z" } 2024-04-20T01:36:49.420 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:36:49.420 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:49.420 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:49.420 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:36:54.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:54.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:54.422 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:54.481 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:54.482 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:54.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:54.555+0000 7fcdafd30780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:54.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:36:54.556+0000 7fcdafd30780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:54.569 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:36:54.570 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:36:54.570 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:36:54.570 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:36:54.570 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:36:54.570 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:36:54.570 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:54.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:54.570 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:36:54.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:54.571 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:36:54.571 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:54.571 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:54.571 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:36:54.571 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:54.571 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:36:54.571 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:54.571 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:36:54.571 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:54.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:54.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:54.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:54.572 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:36:54.572 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:54.572 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:54.572 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:36:54.572 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:54.572 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:54.572 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:54.572 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:36:54.572 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:54.572 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:54.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:54.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:54.573 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:36:54.573 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:54.573 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:54.573 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:36:54.573 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:54.573 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:54.573 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:54.573 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:36:54.573 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:54.573 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:54.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:54.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:54.574 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:36:54.574 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:54.574 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:36:54.574 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:36:54.574 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:54.574 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:54.574 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:54.574 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:36:54.574 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:36:54.575 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:54.575 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:54.575 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:36:54.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:54.575 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:36:54.575 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:36:54.575 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:36:54.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:54.575 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:36:54Z" 2024-04-20T01:36:54.575 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:36:54.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:36:54Z" } 2024-04-20T01:36:54.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:36:54.577 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:54.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:54.577 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:36:54.577 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:36:54.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:54.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:54.577 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:54.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:36:54.691+0000 7f37d73b4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:54.681 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:36:54.692+0000 7f37d73b4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:36:54.698 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:36:54.699 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:36:54.699 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:36:54.699 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:36:54.699 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:36:54.699 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:36:54.699 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:36:54.699 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:36:54.699 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:36:54.699 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:36:54.699 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:36:54.699 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:36:54.699 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:36:54.700 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:36:54.700 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:36:54.700 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:36:54.700 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:36:54.700 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:36:54.700 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:36:54.700 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:36:54.700 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:36:54.700 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:36:54.700 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:36:54.700 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:36:54.700 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:36:54.701 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:36:54.701 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:36:54.701 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:36:54.701 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:36:54.701 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:36:54.701 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:36:54.701 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:36:54.701 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:36:54.701 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:36:54.701 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:36:54.701 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:36:54.701 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:36:54.702 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:36:54.702 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:36:54.702 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:36:54.702 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:36:54.702 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:36:54.702 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:36:54.702 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:36:54.702 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:36:54.702 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:36:54.702 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:36:54.702 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:36:54.702 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:36:54.703 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:36:54.704 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:36:54.704 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:36:54Z" 2024-04-20T01:36:54.704 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:36:54.704 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:36:54Z" } 2024-04-20T01:36:54.705 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:36:54.705 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:54.705 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:36:54.705 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:36:54.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:54.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:54.705 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:54.758 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:54.759 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:54.836 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:36:54.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:36:54.836 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:36:54.836 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:36:54.836 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:36:54.836 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:36:54.836 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000026070", 2024-04-20T01:36:54.837 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:35:00.951408Z" 2024-04-20T01:36:54.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:36:54.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:36:54.837 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:36:54.837 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:36:54.837 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:36:54.837 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:36:54.837 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:36:54.837 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:36:54.837 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:36:54.837 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:36:54.838 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:36:54.838 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:36:54.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:54.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:54.839 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:36:54.904 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:36:54.904 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:36:54.990 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:36:54.990 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:36:54.990 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:36:54.990 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:36:54.990 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:36:54.991 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:36:54.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:54.991 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:36:54.991 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:54.991 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:36:54.991 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:54.991 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:36:54.991 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:36:54.991 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:54.991 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:36:54.992 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:54.992 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:36:54.992 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:54.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:54.992 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:54.992 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:36:54.993 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:54.993 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:36:54.993 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000025929", 2024-04-20T01:36:54.993 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:54.993 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:36:54.993 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:54.993 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:13.914481Z" 2024-04-20T01:36:54.993 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:54.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:54.993 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:54.993 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:36:54.993 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:54.994 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:36:54.994 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:36:54.994 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:54.994 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:54.994 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:54.994 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:36:54.994 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:54.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:54.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:36:54.995 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:36:54.995 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:36:54.995 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:36:54.995 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:36:54.995 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:36:54.995 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:36:54.995 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:36:54.995 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:36:54.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:54.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:36:54.995 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:36:54.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:54.996 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:36:54.996 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:36:54.996 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:36:54.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:36:54.996 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:36:54Z" 2024-04-20T01:36:54.996 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:36:54.997 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014922", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:22:09.862093Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025929", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:27:13.914481Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020176", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.871854Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:36:54Z" } 2024-04-20T01:36:54.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014922', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:22:09.862093Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025929', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:27:13.914481Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.871854Z'}}] 2024-04-20T01:36:54.997 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:36:54.998 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000025929', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:36:54.998 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026070 target=00000000000000000000:00000000000000025929 2024-04-20T01:36:59.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:59.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:36:59.999 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:00.060 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:00.060 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:00.140 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:00.140 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:00.141 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:00.141 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:00.141 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:00.141 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:37:00.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:00.141 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:00.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:00.141 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:00.141 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:00.141 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:37:00.141 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:37:00.141 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:00.142 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:37:00.142 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:00.142 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:37:00.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:00.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:00.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:00.142 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:00.142 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:00.142 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:37:00.142 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000025929", 2024-04-20T01:37:00.142 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:37:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:13.914481Z" 2024-04-20T01:37:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:37:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:37:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:37:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:37:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:00.146 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:00Z" 2024-04-20T01:37:00.146 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:00.146 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014922", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:22:09.862093Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025929", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:27:13.914481Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020176", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.871854Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:37:00Z" } 2024-04-20T01:37:00.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014922', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:22:09.862093Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025929', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:27:13.914481Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.871854Z'}}] 2024-04-20T01:37:00.147 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:37:00.147 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000025929', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:37:00.147 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026070 target=00000000000000000000:00000000000000025929 2024-04-20T01:37:05.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:05.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:05.149 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:05.210 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:05.210 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:05.291 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:37:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:37:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:37:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:37:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:37:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026070", 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:00.951408Z" 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:37:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:37:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:37:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:37:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:37:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:37:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:37:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:37:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:05Z" 2024-04-20T01:37:05.296 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:05.297 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014922", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:22:09.862093Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026070", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:35:00.951408Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020176", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.871854Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:37:05Z" } 2024-04-20T01:37:05.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014922', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:22:09.862093Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026070', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:35:00.951408Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.871854Z'}}] 2024-04-20T01:37:05.297 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:37:05.297 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:37:05.297 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:37:05.298 DEBUG:boto:path=/aktlji-80/ 2024-04-20T01:37:05.298 DEBUG:boto:auth_path=/aktlji-80/ 2024-04-20T01:37:05.298 DEBUG:boto:Method: HEAD 2024-04-20T01:37:05.298 DEBUG:boto:Path: /aktlji-80/ 2024-04-20T01:37:05.298 DEBUG:boto:Data: 2024-04-20T01:37:05.298 DEBUG:boto:Headers: {} 2024-04-20T01:37:05.298 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:37:05.298 DEBUG:boto:Port: 8000 2024-04-20T01:37:05.298 DEBUG:boto:Params: {} 2024-04-20T01:37:05.299 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:37:05.299 DEBUG:boto:Token: None 2024-04-20T01:37:05.299 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:37:05 GMT /aktlji-80/ 2024-04-20T01:37:05.300 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:oFQfFoM/QoG6tZRnSKSucYeFfVk= 2024-04-20T01:37:05.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oFQfFoM/QoG6tZRnSKSucYeFfVk=', 'Content-Length': '0'} 2024-04-20T01:37:05.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052c52808507b4774-0066231c41-214b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:05.305 DEBUG:boto:path=/aktlji-80/obj1 2024-04-20T01:37:05.305 DEBUG:boto:auth_path=/aktlji-80/obj1 2024-04-20T01:37:05.305 DEBUG:boto:Method: HEAD 2024-04-20T01:37:05.305 DEBUG:boto:Path: /aktlji-80/obj1 2024-04-20T01:37:05.305 DEBUG:boto:Data: 2024-04-20T01:37:05.305 DEBUG:boto:Headers: {} 2024-04-20T01:37:05.305 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:37:05.306 DEBUG:boto:Port: 8000 2024-04-20T01:37:05.306 DEBUG:boto:Params: {} 2024-04-20T01:37:05.306 DEBUG:boto:Token: None 2024-04-20T01:37:05.306 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:37:05 GMT /aktlji-80/obj1 2024-04-20T01:37:05.306 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:W66jrqJ3cFuGMlA6GELsnxV77wY= 2024-04-20T01:37:05.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W66jrqJ3cFuGMlA6GELsnxV77wY=', 'Content-Length': '0'} 2024-04-20T01:37:05.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:35:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-80:73f46f02-c7ab-4426-a267-2273af911da5.6678.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000798d6acc008907e4-0066231c41-214b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:37:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:05.309 DEBUG:boto:path=/aktlji-80/obj1 2024-04-20T01:37:05.309 DEBUG:boto:auth_path=/aktlji-80/obj1 2024-04-20T01:37:05.309 DEBUG:boto:Method: GET 2024-04-20T01:37:05.309 DEBUG:boto:Path: /aktlji-80/obj1 2024-04-20T01:37:05.309 DEBUG:boto:Data: 2024-04-20T01:37:05.309 DEBUG:boto:Headers: {} 2024-04-20T01:37:05.309 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:37:05.309 DEBUG:boto:Port: 8000 2024-04-20T01:37:05.309 DEBUG:boto:Params: {} 2024-04-20T01:37:05.309 DEBUG:boto:Token: None 2024-04-20T01:37:05.310 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:37:05 GMT /aktlji-80/obj1 2024-04-20T01:37:05.310 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:f0kgrI8agX9jhDEhF0cls0mHl+Q= 2024-04-20T01:37:05.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:f0kgrI8agX9jhDEhF0cls0mHl+Q=', 'Content-Length': '0'} 2024-04-20T01:37:05.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:35:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-80:73f46f02-c7ab-4426-a267-2273af911da5.6678.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bf59d6e04909330-0066231c41-214b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:37:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:05.312 DEBUG:boto:path=/aktlji-81/ 2024-04-20T01:37:05.312 DEBUG:boto:auth_path=/aktlji-81/ 2024-04-20T01:37:05.312 DEBUG:boto:Method: HEAD 2024-04-20T01:37:05.312 DEBUG:boto:Path: /aktlji-81/ 2024-04-20T01:37:05.312 DEBUG:boto:Data: 2024-04-20T01:37:05.312 DEBUG:boto:Headers: {} 2024-04-20T01:37:05.312 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:37:05.312 DEBUG:boto:Port: 8000 2024-04-20T01:37:05.313 DEBUG:boto:Params: {} 2024-04-20T01:37:05.313 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:37:05.313 DEBUG:boto:Token: None 2024-04-20T01:37:05.313 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:37:05 GMT /aktlji-81/ 2024-04-20T01:37:05.313 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cfSw4saMrlsfyUzZIWzITsBdp2g= 2024-04-20T01:37:05.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cfSw4saMrlsfyUzZIWzITsBdp2g=', 'Content-Length': '0'} 2024-04-20T01:37:05.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035832d1b9800d6a2-0066231c41-1a14-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:05.318 DEBUG:boto:path=/aktlji-81/obj2 2024-04-20T01:37:05.318 DEBUG:boto:auth_path=/aktlji-81/obj2 2024-04-20T01:37:05.318 DEBUG:boto:Method: HEAD 2024-04-20T01:37:05.318 DEBUG:boto:Path: /aktlji-81/obj2 2024-04-20T01:37:05.318 DEBUG:boto:Data: 2024-04-20T01:37:05.319 DEBUG:boto:Headers: {} 2024-04-20T01:37:05.319 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:37:05.319 DEBUG:boto:Port: 8000 2024-04-20T01:37:05.319 DEBUG:boto:Params: {} 2024-04-20T01:37:05.319 DEBUG:boto:Token: None 2024-04-20T01:37:05.319 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:37:05 GMT /aktlji-81/obj2 2024-04-20T01:37:05.319 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:M2KWg98T3TQa8SKdr+K5trydlOk= 2024-04-20T01:37:05.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:M2KWg98T3TQa8SKdr+K5trydlOk=', 'Content-Length': '0'} 2024-04-20T01:37:05.321 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d82829a508eee6e7-0066231c41-1a14-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:37:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:05.323 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2024-04-20T01:37:05.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:05.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:05.324 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:05.382 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:05.383 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:05.426 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:37:05.442 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:37:05.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:37:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:37:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:37:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:05.443 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:05.444 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:35:01.005366Z" 2024-04-20T01:37:05.444 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:05.444 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:05.444 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:05.444 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:37:05.444 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:05.444 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:37:05.445 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:05.445 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:37:05.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:05.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:05.445 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:05.506 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:05.506 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:05.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:05.567+0000 7f55d1841780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:05.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:05.568+0000 7f55d1841780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:05.579 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:37:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:37:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:37:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:05Z" 2024-04-20T01:37:05.584 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:05.585 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:05Z" } 2024-04-20T01:37:05.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:05.586 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:05.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:05.586 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:37:05.586 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:37:05.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:05.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:05.586 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:05.646 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:05.646 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:05.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:37:05.698+0000 7f8499774780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:05.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:37:05.699+0000 7f8499774780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:05.707 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:37:05.707 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:37:05.707 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:37:05.707 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:37:05.708 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:37:05.708 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:05.708 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:37:05.708 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:05.708 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:37:05.708 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:05.708 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:37:05.708 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:37:05.708 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:37:05.708 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:05.709 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:37:05.709 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:37:05.709 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:37:05.709 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:05.709 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:37:05.709 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:05.709 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:05.709 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:05.709 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:37:05.709 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:37:05.710 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:37:05.710 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:37:05.710 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:37:05.710 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:37:05.710 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:37:05.710 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:37:05.710 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:37:05.710 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:05.710 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:05.710 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:05.710 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:37:05.711 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:37:05.712 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:37:05.713 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:05.713 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:37:05Z" 2024-04-20T01:37:05.713 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:37:05.714 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:05Z" } 2024-04-20T01:37:05.714 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:05.714 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:05.714 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:05.714 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:37:05.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:05.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:05.715 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:37:05.765 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:05.765 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:05.833 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:37:05.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:05.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:05.834 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:37:05.889 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:05.889 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:05.946 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:37:05.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:05.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:05.947 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:37:06.005 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:06.006 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:06.073 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:37:06.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:06.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:06.074 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:37:06.132 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:06.133 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:06.200 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:37:06.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:06.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:06.201 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:37:06.256 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:06.256 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:06.314 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:37:06.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:06.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:06.315 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:37:06.368 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:06.369 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:06.434 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:37:06.434 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:06.435 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 12, 2024-04-20T01:37:06.435 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:37:06.435 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T01:37:06.435 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T01:37:06.435 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:06.435 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T01:37:06.435 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:06.435 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T01:37:06.435 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T01:37:06.435 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T01:37:06.435 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T01:37:06.436 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:37:06.436 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:37:06.436 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:37:06.436 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T01:37:06.436 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T01:37:06.436 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:37:06.436 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:37:06.436 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:06.436 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:37:06.436 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T01:37:06.436 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:37:06.437 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:37:06.437 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:37:06.437 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:37:06.437 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:37:06.437 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:37:06.437 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:37:06.437 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:37:06.437 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T01:37:06.437 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:37:06.437 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:37:06.437 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:37:06.437 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:37:06.438 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:37:06.438 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:06.438 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.438 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:06.438 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:37:06.438 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T01:37:06.438 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:37:06.438 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:37:06.438 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:37:06.438 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:37:06.439 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:37:06.439 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:37:06.439 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:37:06.439 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:37:06.439 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:37:06.439 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:37:06.439 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:37:06.439 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:37:06.439 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:37:06.439 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:37:06.439 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:06.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:06.440 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:37:06.440 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T01:37:06.440 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:37:06.440 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:37:06.440 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:37:06.440 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:37:06.440 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:37:06.440 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:37:06.440 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:37:06.441 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:37:06.441 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:37:06.441 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:37:06.441 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:37:06.441 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:37:06.441 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:37:06.441 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:37:06.441 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:06.441 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:06.441 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:37:06.441 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T01:37:06.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:06.442 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T01:37:06.442 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T01:37:06.442 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T01:37:06.442 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T01:37:06.442 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:06.442 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:06.442 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:37:06.442 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T01:37:06.442 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:37:06.442 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T01:37:06.442 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [ 2024-04-20T01:37:06.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:06.443 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-group", 2024-04-20T01:37:06.443 INFO:teuthology.orchestra.run.smithi019.stdout: "data_flow": { 2024-04-20T01:37:06.443 INFO:teuthology.orchestra.run.smithi019.stdout: "symmetrical": [ 2024-04-20T01:37:06.443 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:06.443 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow1", 2024-04-20T01:37:06.443 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:37:06.443 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:37:06.451 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:37:06.451 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:06.451 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.451 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:06.451 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow", 2024-04-20T01:37:06.451 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:37:06.452 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:37:06.452 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:37:06.452 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:06.452 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:06.452 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:06.452 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.452 INFO:teuthology.orchestra.run.smithi019.stdout: "pipes": [ 2024-04-20T01:37:06.452 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:06.452 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe", 2024-04-20T01:37:06.452 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:37:06.453 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "*", 2024-04-20T01:37:06.453 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:37:06.453 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:37:06.453 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:37:06.453 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:06.453 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.453 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:37:06.453 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "*", 2024-04-20T01:37:06.453 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:37:06.453 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:37:06.453 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:37:06.453 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:06.454 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.454 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:37:06.454 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:37:06.454 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:37:06.454 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:37:06.454 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:06.454 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.454 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:37:06.454 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:37:06.454 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:37:06.454 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:37:06.455 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:06.455 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:06.455 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:37:06.455 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "enabled" 2024-04-20T01:37:06.455 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:06.455 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:06.455 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.455 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T01:37:06.455 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:37:06.455 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:06.455 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:06.456 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:37:06.456 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T01:37:06.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:06.456 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:37:06.456 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T01:37:06.456 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:06.456 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:37:06.456 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T01:37:06.456 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.456 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:06.457 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:37:06.457 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T01:37:06.457 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:06.457 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:37:06.457 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.457 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T01:37:06.457 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:37:06.457 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T01:37:06.457 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T01:37:06.457 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:37:06.457 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:37:06.458 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:37:06.458 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:37:06.458 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:37:06.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.458 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T01:37:06.458 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:37:06.458 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:37:06.458 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:37:06.458 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:37:06.458 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:37:06.458 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.458 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T01:37:06.459 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:37:06.459 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:37:06.459 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:37:06.459 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:37:06.459 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:37:06.459 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.459 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T01:37:06.459 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:37:06.459 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:37:06.459 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:37:06.459 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:37:06.460 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:37:06.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.460 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T01:37:06.460 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:37:06.460 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:37:06.460 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:37:06.460 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:37:06.460 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:37:06.460 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:06.460 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:06.460 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:37:06.461 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T01:37:06.461 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:37:06.461 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:37:06.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:06.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:37:06.462 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:37:06.522 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:06.522 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:06.631 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:37:06.631 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-82 2024-04-20T01:37:06.631 DEBUG:boto:path=/aktlji-82/ 2024-04-20T01:37:06.631 DEBUG:boto:auth_path=/aktlji-82/ 2024-04-20T01:37:06.631 DEBUG:boto:Method: PUT 2024-04-20T01:37:06.631 DEBUG:boto:Path: /aktlji-82/ 2024-04-20T01:37:06.632 DEBUG:boto:Data: 2024-04-20T01:37:06.632 DEBUG:boto:Headers: {} 2024-04-20T01:37:06.632 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:37:06.632 DEBUG:boto:Port: 8000 2024-04-20T01:37:06.632 DEBUG:boto:Params: {} 2024-04-20T01:37:06.632 DEBUG:boto:Token: None 2024-04-20T01:37:06.632 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:37:06 GMT /aktlji-82/ 2024-04-20T01:37:06.632 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PvV6lZimwM3UDDQJ5+vQ41zKrcw= 2024-04-20T01:37:06.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PvV6lZimwM3UDDQJ5+vQ41zKrcw=', 'Content-Length': '0'} 2024-04-20T01:37:06.824 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000eea9ec4df1104e5-0066231c42-1a85-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:06.824 DEBUG:boto:path=/aktlji-82/ 2024-04-20T01:37:06.824 DEBUG:boto:auth_path=/aktlji-82/ 2024-04-20T01:37:06.824 DEBUG:boto:Method: HEAD 2024-04-20T01:37:06.824 DEBUG:boto:Path: /aktlji-82/ 2024-04-20T01:37:06.824 DEBUG:boto:Data: 2024-04-20T01:37:06.825 DEBUG:boto:Headers: {} 2024-04-20T01:37:06.825 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:37:06.825 DEBUG:boto:Port: 8000 2024-04-20T01:37:06.825 DEBUG:boto:Params: {} 2024-04-20T01:37:06.825 DEBUG:boto:Token: None 2024-04-20T01:37:06.825 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:37:06 GMT /aktlji-82/ 2024-04-20T01:37:06.825 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WyLsm31kewzKEDbXYC63OakHPuE= 2024-04-20T01:37:06.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WyLsm31kewzKEDbXYC63OakHPuE=', 'Content-Length': '0'} 2024-04-20T01:37:06.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f398f607b044242-0066231c42-1a85-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:06.833 DEBUG:boto:path=/aktlji-82/obj1 2024-04-20T01:37:06.833 DEBUG:boto:auth_path=/aktlji-82/obj1 2024-04-20T01:37:06.833 DEBUG:boto:Method: PUT 2024-04-20T01:37:06.833 DEBUG:boto:Path: /aktlji-82/obj1 2024-04-20T01:37:06.833 DEBUG:boto:Data: 2024-04-20T01:37:06.833 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:37:06.833 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:37:06.833 DEBUG:boto:Port: 8000 2024-04-20T01:37:06.833 DEBUG:boto:Params: {} 2024-04-20T01:37:06.833 DEBUG:boto:Token: None 2024-04-20T01:37:06.834 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:37:06 GMT /aktlji-82/obj1 2024-04-20T01:37:06.834 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EhNkmXd/AhCe96ROE0B2NdByLog= 2024-04-20T01:37:06.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EhNkmXd/AhCe96ROE0B2NdByLog='} 2024-04-20T01:37:06.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096cba567cc846e48-0066231c42-1a85-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:06.864 DEBUG:boto:path=/aktlji-82/ 2024-04-20T01:37:06.864 DEBUG:boto:auth_path=/aktlji-82/ 2024-04-20T01:37:06.865 DEBUG:boto:Method: HEAD 2024-04-20T01:37:06.865 DEBUG:boto:Path: /aktlji-82/ 2024-04-20T01:37:06.865 DEBUG:boto:Data: 2024-04-20T01:37:06.865 DEBUG:boto:Headers: {} 2024-04-20T01:37:06.865 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:37:06.865 DEBUG:boto:Port: 8000 2024-04-20T01:37:06.865 DEBUG:boto:Params: {} 2024-04-20T01:37:06.865 DEBUG:boto:Token: None 2024-04-20T01:37:06.865 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:37:06 GMT /aktlji-82/ 2024-04-20T01:37:06.866 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WyLsm31kewzKEDbXYC63OakHPuE= 2024-04-20T01:37:06.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WyLsm31kewzKEDbXYC63OakHPuE=', 'Content-Length': '0'} 2024-04-20T01:37:06.870 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008214e85accf3caa1-0066231c42-1a85-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:06.871 DEBUG:boto:path=/aktlji-82/obj2 2024-04-20T01:37:06.871 DEBUG:boto:auth_path=/aktlji-82/obj2 2024-04-20T01:37:06.871 DEBUG:boto:Method: PUT 2024-04-20T01:37:06.871 DEBUG:boto:Path: /aktlji-82/obj2 2024-04-20T01:37:06.871 DEBUG:boto:Data: 2024-04-20T01:37:06.871 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:37:06.871 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:37:06.871 DEBUG:boto:Port: 8000 2024-04-20T01:37:06.872 DEBUG:boto:Params: {} 2024-04-20T01:37:06.872 DEBUG:boto:Token: None 2024-04-20T01:37:06.872 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:37:06 GMT /aktlji-82/obj2 2024-04-20T01:37:06.872 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/sp3wPG7jXdhf1rR5YyhvW2Q/II= 2024-04-20T01:37:06.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/sp3wPG7jXdhf1rR5YyhvW2Q/II='} 2024-04-20T01:37:06.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7edf268edd79dcc-0066231c42-1a85-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:06.915 DEBUG:boto:path=/aktlji-82/ 2024-04-20T01:37:06.915 DEBUG:boto:auth_path=/aktlji-82/ 2024-04-20T01:37:06.915 DEBUG:boto:Method: HEAD 2024-04-20T01:37:06.915 DEBUG:boto:Path: /aktlji-82/ 2024-04-20T01:37:06.915 DEBUG:boto:Data: 2024-04-20T01:37:06.915 DEBUG:boto:Headers: {} 2024-04-20T01:37:06.915 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:37:06.915 DEBUG:boto:Port: 8000 2024-04-20T01:37:06.915 DEBUG:boto:Params: {} 2024-04-20T01:37:06.916 DEBUG:boto:Token: None 2024-04-20T01:37:06.916 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:37:06 GMT /aktlji-82/ 2024-04-20T01:37:06.916 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WyLsm31kewzKEDbXYC63OakHPuE= 2024-04-20T01:37:06.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WyLsm31kewzKEDbXYC63OakHPuE=', 'Content-Length': '0'} 2024-04-20T01:37:06.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009554b0e201012c0b-0066231c42-1a85-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:06.923 DEBUG:boto:path=/aktlji-82/obj3 2024-04-20T01:37:06.923 DEBUG:boto:auth_path=/aktlji-82/obj3 2024-04-20T01:37:06.923 DEBUG:boto:Method: PUT 2024-04-20T01:37:06.923 DEBUG:boto:Path: /aktlji-82/obj3 2024-04-20T01:37:06.923 DEBUG:boto:Data: 2024-04-20T01:37:06.923 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:37:06.924 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:37:06.924 DEBUG:boto:Port: 8000 2024-04-20T01:37:06.924 DEBUG:boto:Params: {} 2024-04-20T01:37:06.924 DEBUG:boto:Token: None 2024-04-20T01:37:06.924 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:37:06 GMT /aktlji-82/obj3 2024-04-20T01:37:06.924 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BlXPxbgEjI8rQ9s0OAHLSUiu1GQ= 2024-04-20T01:37:06.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BlXPxbgEjI8rQ9s0OAHLSUiu1GQ='} 2024-04-20T01:37:06.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfc93fe32be1584c-0066231c42-1a85-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:06.952 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-83 2024-04-20T01:37:06.953 DEBUG:boto:path=/aktlji-83/ 2024-04-20T01:37:06.953 DEBUG:boto:auth_path=/aktlji-83/ 2024-04-20T01:37:06.953 DEBUG:boto:Method: PUT 2024-04-20T01:37:06.953 DEBUG:boto:Path: /aktlji-83/ 2024-04-20T01:37:06.953 DEBUG:boto:Data: 2024-04-20T01:37:06.953 DEBUG:boto:Headers: {} 2024-04-20T01:37:06.953 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:37:06.953 DEBUG:boto:Port: 8000 2024-04-20T01:37:06.953 DEBUG:boto:Params: {} 2024-04-20T01:37:06.953 DEBUG:boto:Token: None 2024-04-20T01:37:06.954 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:37:06 GMT /aktlji-83/ 2024-04-20T01:37:06.954 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ifjdD3+ka7ykU8NHMnrFep1TMAI= 2024-04-20T01:37:06.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ifjdD3+ka7ykU8NHMnrFep1TMAI=', 'Content-Length': '0'} 2024-04-20T01:37:07.079 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004d3a7f76c223d190-0066231c42-2227-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:07.080 DEBUG:boto:path=/aktlji-83/ 2024-04-20T01:37:07.080 DEBUG:boto:auth_path=/aktlji-83/ 2024-04-20T01:37:07.080 DEBUG:boto:Method: HEAD 2024-04-20T01:37:07.080 DEBUG:boto:Path: /aktlji-83/ 2024-04-20T01:37:07.080 DEBUG:boto:Data: 2024-04-20T01:37:07.080 DEBUG:boto:Headers: {} 2024-04-20T01:37:07.080 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:37:07.081 DEBUG:boto:Port: 8000 2024-04-20T01:37:07.081 DEBUG:boto:Params: {} 2024-04-20T01:37:07.081 DEBUG:boto:Token: None 2024-04-20T01:37:07.081 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:37:07 GMT /aktlji-83/ 2024-04-20T01:37:07.081 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mTNBRxj2ItidhuAaSB6piL04svs= 2024-04-20T01:37:07.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mTNBRxj2ItidhuAaSB6piL04svs=', 'Content-Length': '0'} 2024-04-20T01:37:07.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003500419773a435f4-0066231c43-2227-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:07.084 DEBUG:boto:path=/aktlji-83/obj4 2024-04-20T01:37:07.084 DEBUG:boto:auth_path=/aktlji-83/obj4 2024-04-20T01:37:07.084 DEBUG:boto:Method: PUT 2024-04-20T01:37:07.085 DEBUG:boto:Path: /aktlji-83/obj4 2024-04-20T01:37:07.085 DEBUG:boto:Data: 2024-04-20T01:37:07.085 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:37:07.085 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:37:07.085 DEBUG:boto:Port: 8000 2024-04-20T01:37:07.085 DEBUG:boto:Params: {} 2024-04-20T01:37:07.085 DEBUG:boto:Token: None 2024-04-20T01:37:07.085 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:37:07 GMT /aktlji-83/obj4 2024-04-20T01:37:07.085 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:8FYp9nEIhRT8/6bg4kpvDoixZMg= 2024-04-20T01:37:07.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:37:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8FYp9nEIhRT8/6bg4kpvDoixZMg='} 2024-04-20T01:37:07.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c01d2380719a188b-0066231c43-2227-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:07.109 DEBUG:boto:path=/aktlji-83/ 2024-04-20T01:37:07.109 DEBUG:boto:auth_path=/aktlji-83/ 2024-04-20T01:37:07.109 DEBUG:boto:Method: HEAD 2024-04-20T01:37:07.109 DEBUG:boto:Path: /aktlji-83/ 2024-04-20T01:37:07.110 DEBUG:boto:Data: 2024-04-20T01:37:07.110 DEBUG:boto:Headers: {} 2024-04-20T01:37:07.110 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:37:07.110 DEBUG:boto:Port: 8000 2024-04-20T01:37:07.110 DEBUG:boto:Params: {} 2024-04-20T01:37:07.110 DEBUG:boto:Token: None 2024-04-20T01:37:07.110 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:37:07 GMT /aktlji-83/ 2024-04-20T01:37:07.110 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mTNBRxj2ItidhuAaSB6piL04svs= 2024-04-20T01:37:07.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mTNBRxj2ItidhuAaSB6piL04svs=', 'Content-Length': '0'} 2024-04-20T01:37:07.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068106d727cc2ed95-0066231c43-2227-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:07.114 DEBUG:boto:path=/aktlji-83/obj5 2024-04-20T01:37:07.114 DEBUG:boto:auth_path=/aktlji-83/obj5 2024-04-20T01:37:07.114 DEBUG:boto:Method: PUT 2024-04-20T01:37:07.114 DEBUG:boto:Path: /aktlji-83/obj5 2024-04-20T01:37:07.114 DEBUG:boto:Data: 2024-04-20T01:37:07.114 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:37:07.114 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:37:07.114 DEBUG:boto:Port: 8000 2024-04-20T01:37:07.115 DEBUG:boto:Params: {} 2024-04-20T01:37:07.115 DEBUG:boto:Token: None 2024-04-20T01:37:07.115 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:37:07 GMT /aktlji-83/obj5 2024-04-20T01:37:07.115 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BAtkieSUp9xADodimn09ZYBlZcM= 2024-04-20T01:37:07.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:37:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BAtkieSUp9xADodimn09ZYBlZcM='} 2024-04-20T01:37:07.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea0435e9ccd1d42b-0066231c43-2227-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:07.159 DEBUG:boto:path=/aktlji-83/ 2024-04-20T01:37:07.159 DEBUG:boto:auth_path=/aktlji-83/ 2024-04-20T01:37:07.159 DEBUG:boto:Method: HEAD 2024-04-20T01:37:07.159 DEBUG:boto:Path: /aktlji-83/ 2024-04-20T01:37:07.159 DEBUG:boto:Data: 2024-04-20T01:37:07.159 DEBUG:boto:Headers: {} 2024-04-20T01:37:07.159 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:37:07.159 DEBUG:boto:Port: 8000 2024-04-20T01:37:07.159 DEBUG:boto:Params: {} 2024-04-20T01:37:07.160 DEBUG:boto:Token: None 2024-04-20T01:37:07.160 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:37:07 GMT /aktlji-83/ 2024-04-20T01:37:07.160 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mTNBRxj2ItidhuAaSB6piL04svs= 2024-04-20T01:37:07.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mTNBRxj2ItidhuAaSB6piL04svs=', 'Content-Length': '0'} 2024-04-20T01:37:07.162 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab86712260c4189a-0066231c43-2227-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:07.163 DEBUG:boto:path=/aktlji-83/obj6 2024-04-20T01:37:07.163 DEBUG:boto:auth_path=/aktlji-83/obj6 2024-04-20T01:37:07.163 DEBUG:boto:Method: PUT 2024-04-20T01:37:07.163 DEBUG:boto:Path: /aktlji-83/obj6 2024-04-20T01:37:07.163 DEBUG:boto:Data: 2024-04-20T01:37:07.163 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:37:07.163 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:37:07.163 DEBUG:boto:Port: 8000 2024-04-20T01:37:07.163 DEBUG:boto:Params: {} 2024-04-20T01:37:07.163 DEBUG:boto:Token: None 2024-04-20T01:37:07.164 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:37:07 GMT /aktlji-83/obj6 2024-04-20T01:37:07.164 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SuaB4VyEfPVj9zzzGzUwSiYhpnc= 2024-04-20T01:37:07.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:37:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SuaB4VyEfPVj9zzzGzUwSiYhpnc='} 2024-04-20T01:37:07.185 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cba9e03ecce6a55-0066231c43-2227-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:37:07.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:07.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:07.186 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:07.268 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:07.268 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:07.364 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:37:07.397 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:37:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:37:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:37:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:37:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:07.397 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:37:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:37:07.057559Z" 2024-04-20T01:37:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:37:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:37:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:37:07.398 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:37:07.398 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:37:07.399 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:07.399 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:37:07.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:07.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:07.400 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:07.461 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:07.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:07.513+0000 7f5a99788780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:07.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:07.514+0000 7f5a99788780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:07.522 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:07.522 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:07.522 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:07.522 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:07.522 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:07.522 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:07.522 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:07.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:07.523 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:07.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:07.523 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:07.523 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:07.523 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:07.523 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:07.523 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:07.523 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:37:07.523 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:07.523 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:07.523 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:07.523 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:07.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:07.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:07.524 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:07.524 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:07.524 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:07.524 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:37:07.524 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:07.524 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:07.524 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:07.524 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:37:07.524 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:07.525 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:07.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:07.525 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:07.525 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:07.525 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:07.525 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:07.525 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:07.525 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:07.525 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:07.526 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:07.526 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:07.526 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:07.526 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:07.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:07.526 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:07.526 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:07.526 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:07.526 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:07.526 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:07.527 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:07.527 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:07.527 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:07.528 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:07.528 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:07.528 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:07.528 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:07.528 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:07.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:07.528 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:07.528 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:37:07.528 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:37:07.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:07.529 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:07Z" 2024-04-20T01:37:07.529 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:07.529 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:07Z" } 2024-04-20T01:37:07.530 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:07.530 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:07.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:07.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:37:12.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:12.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:12.531 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:12.591 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:12.591 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:12.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:12.664+0000 7fba0a682780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:12.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:12.665+0000 7fba0a682780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:12.681 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:12.681 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:12.682 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:37:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:12.683 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:37:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:37:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:12.684 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:12.685 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:12.686 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:37:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:37:12.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:12.688 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:12Z" 2024-04-20T01:37:12.688 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:12.689 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:12Z" } 2024-04-20T01:37:12.689 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:12.689 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:12.689 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:12.689 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:37:17.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:17.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:17.691 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:17.751 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:17.751 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:17.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:17.806+0000 7f18ceb8d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:17.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:17.807+0000 7f18ceb8d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:17.813 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:17.814 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:17.815 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:17.815 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:17.815 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:17.817 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:37:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:17.818 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:37:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:37:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:17.819 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:17.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:17.821 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:17.821 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:17.821 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:17.821 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:17.821 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:17.821 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:17.821 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:17.821 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:17.821 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:17.821 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:17.821 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:17.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:17.822 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:17.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:17.822 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:17.822 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:37:17.822 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:37:17.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:17.822 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:17Z" 2024-04-20T01:37:17.822 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:17.823 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:17Z" } 2024-04-20T01:37:17.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:17.824 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:17.824 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:17.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:37:22.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:22.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:22.826 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:22.886 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:22.887 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:22.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:22.949+0000 7f969a957780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:22.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:22.950+0000 7f969a957780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:22.961 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:37:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:37:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:37:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:37:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:37:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:22Z" 2024-04-20T01:37:22.966 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:22.967 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:22Z" } 2024-04-20T01:37:22.967 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:22.967 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:22.967 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:22.967 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:37:27.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:27.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:27.969 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:28.028 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:28.028 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:28.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:28.088+0000 7f3cc0634780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:28.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:28.089+0000 7f3cc0634780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:28.098 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:28.098 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:28.098 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:28.100 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:37:28.100 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:28.100 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:28.100 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:28.100 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:28.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:28.100 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:28.100 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:28.100 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:28.100 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:28.100 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:28.101 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:28.102 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:28.102 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:28.102 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:28.102 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:28.102 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:28.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:28.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:28.102 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:28.102 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:28.102 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:28.103 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:28.103 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:28.103 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:28.103 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:28.103 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:28.103 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:28.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:28.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:28.103 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:28.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:28.104 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:28.104 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:37:28.104 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:37:28.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:28.104 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:28Z" 2024-04-20T01:37:28.104 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:28.105 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:28Z" } 2024-04-20T01:37:28.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:28.105 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:28.105 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:28.105 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:37:33.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:33.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:33.107 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:33.170 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:33.170 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:33.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:33.222+0000 7f0c9b749780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:33.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:33.224+0000 7f0c9b749780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:33.232 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:33.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:37:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:33.233 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:37:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:33.234 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:37:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:33.235 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:33.236 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:37:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:37:33.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:33.238 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:33Z" 2024-04-20T01:37:33.238 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:33.238 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:33Z" } 2024-04-20T01:37:33.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:33.239 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:33.239 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:33.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:37:38.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:38.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:38.240 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:38.300 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:38.300 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:38.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:38.368+0000 7f66dce6e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:38.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:38.369+0000 7f66dce6e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:38.380 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:38.380 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:37:38.381 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:37:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:38.382 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:37:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:38.383 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:38.384 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:37:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:37:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:38.385 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:38Z" 2024-04-20T01:37:38.385 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:38.386 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:38Z" } 2024-04-20T01:37:38.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:38.387 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:38.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:38.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:37:43.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:43.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:43.388 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:43.448 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:43.448 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:43.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:43.509+0000 7fe131f3d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:43.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:43.510+0000 7fe131f3d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:43.517 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:43.517 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:43.517 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:43.517 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:43.517 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:43.517 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:43.517 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:43.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:43.518 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:43.518 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:43.518 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:43.518 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:43.518 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:43.518 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:43.518 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:43.518 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:37:43.518 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:43.519 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:43.519 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:43.519 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:43.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:43.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:43.519 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:43.519 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:43.519 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:43.519 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:37:43.519 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:43.519 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:43.519 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:43.520 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:37:43.520 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:43.520 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:43.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:43.520 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:43.520 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:43.520 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:43.520 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:43.520 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:43.521 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:43.521 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:43.521 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:43.521 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:43.521 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:43.521 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:43.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:43.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:43.521 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:43.521 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:43.521 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:43.522 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:43.522 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:43.522 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:43.522 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:43.522 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:43.522 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:43.522 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:43.522 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:43.522 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:43.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:43.523 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:43.523 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:37:43.523 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:37:43.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:43.523 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:43Z" 2024-04-20T01:37:43.523 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:43.524 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:43Z" } 2024-04-20T01:37:43.524 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:43.524 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:43.524 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:43.525 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:37:48.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:48.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:48.526 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:48.584 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:48.585 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:48.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:48.643+0000 7fe44ae04780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:48.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:48.644+0000 7fe44ae04780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:48.659 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:48.659 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:48.659 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:48.659 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:48.659 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:48.659 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:48.659 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:48.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:48.660 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:48.660 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:48.660 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:48.660 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:48.660 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:48.660 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:48.660 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:48.660 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:37:48.660 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:48.660 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:48.660 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:48.660 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:48.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:48.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:48.661 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:48.661 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:48.661 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:48.661 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:37:48.661 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:48.661 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:48.661 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:48.661 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:37:48.661 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:48.661 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:48.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:48.662 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:48.662 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:48.662 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:48.662 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:48.662 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:48.662 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:48.662 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:48.662 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:48.663 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:48.663 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:48.663 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:48.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:48.663 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:48.663 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:48.663 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:48.663 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:48.663 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:48.663 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:48.663 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:48.663 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:48.664 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:48.664 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:48.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:48.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:48.664 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:48.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:48.664 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:48.664 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:37:48.664 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:37:48.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:48.664 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:48Z" 2024-04-20T01:37:48.664 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:48.665 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:48Z" } 2024-04-20T01:37:48.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:48.666 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:48.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:48.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:37:53.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:53.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:53.667 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:53.726 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:53.726 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:53.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:53.802+0000 7fc436448780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:53.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:53.803+0000 7fc436448780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:53.814 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:53.814 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:53.814 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:53.814 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:53.814 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:53.814 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:53.814 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:53.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:37:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:53.815 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:53.817 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:53.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:53.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:53.817 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:53.817 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:53.817 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:53.817 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:37:53.817 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:53.817 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:53.818 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:53.818 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:37:53.818 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:53.818 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:53.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:53.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:53.818 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:53.818 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:53.818 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:53.819 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:53.819 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:53.819 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:53.819 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:53.819 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:53.819 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:53.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:53.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:53.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:53.819 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:53.819 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:53.819 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:37:53.820 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:37:53.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:53.821 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:53Z" 2024-04-20T01:37:53.821 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:53.822 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:53Z" } 2024-04-20T01:37:53.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:53.822 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:53.823 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:53.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:37:58.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:58.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:37:58.825 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:37:58.885 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:37:58.885 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:37:58.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:58.946+0000 7f27db4ec780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:58.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:37:58.948+0000 7f27db4ec780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:37:58.954 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:37:58.954 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:37:58.954 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:37:58.954 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:37:58.954 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:37:58.955 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:37:58.955 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:58.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:58.955 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:37:58.955 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:58.955 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:37:58.955 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:58.955 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:58.955 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:37:58.955 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:58.955 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:37:58.956 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:58.956 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:37:58.956 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:58.956 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:58.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:58.956 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:58.956 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:37:58.956 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:58.956 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:58.957 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:37:58.957 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:58.957 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:58.957 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:58.957 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:37:58.957 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:58.957 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:58.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:58.957 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:58.957 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:37:58.957 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:58.957 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:58.958 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:37:58.958 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:58.958 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:58.958 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:58.958 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:37:58.958 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:58.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:58.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:58.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:37:58.958 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:37:58.959 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:37:58.959 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:37:58.959 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:37:58.959 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:37:58.959 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:37:58.959 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:37:58.959 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:37:58.959 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:37:58.959 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:58.959 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:37:58.959 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:37:58.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:58.960 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:37:58.960 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:37:58.960 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:37:58.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:37:58.960 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:37:58Z" 2024-04-20T01:37:58.960 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:37:58.961 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:37:58Z" } 2024-04-20T01:37:58.961 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:37:58.961 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:58.962 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:37:58.962 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:38:03.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:03.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:03.963 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:38:04.022 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:38:04.023 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:38:04.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:04.077+0000 7f4f1d356780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:04.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:04.078+0000 7f4f1d356780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:04.093 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:38:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:38:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:38:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:38:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:38:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:38:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:38:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:38:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:38:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:38:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:38:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:38:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:38:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:38:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:38:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:38:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:38:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:38:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:38:04Z" 2024-04-20T01:38:04.099 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:38:04.099 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:38:04Z" } 2024-04-20T01:38:04.100 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:38:04.100 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:04.100 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:04.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:38:09.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:09.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:09.101 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:38:09.160 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:38:09.160 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:38:09.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:09.226+0000 7fba4fe30780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:09.226+0000 7fba4fe30780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:09.234 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:38:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:38:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:38:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:38:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:38:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:38:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:09.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:38:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:38:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:38:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:38:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:38:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:09.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:38:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:38:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:38:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:09.236 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:38:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:38:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:38:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:09.237 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:38:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:38:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:38:09.238 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:09.239 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:09.239 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:09.239 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:38:09.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:09.239 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:38:09.239 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:38:09.239 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:38:09.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:09.239 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:38:09Z" 2024-04-20T01:38:09.239 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:38:09.240 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:38:09Z" } 2024-04-20T01:38:09.240 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:38:09.241 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:09.241 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:09.241 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:38:14.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:14.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:14.242 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:38:14.300 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:38:14.300 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:38:14.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:14.353+0000 7f44fd931780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:14.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:14.355+0000 7f44fd931780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:14.363 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:38:14.364 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:38:14.364 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:38:14.364 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:38:14.364 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:38:14.364 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:38:14.364 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:14.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:14.364 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:38:14.364 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:14.364 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:38:14.365 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:14.365 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:14.365 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:38:14.365 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:14.365 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:38:14.365 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:14.365 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:38:14.365 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:14.365 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:14.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:14.366 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:14.366 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:38:14.366 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:14.366 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:14.366 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:38:14.366 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:14.366 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:14.366 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:14.366 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:38:14.366 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:14.367 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:14.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:14.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:14.367 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:38:14.367 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:14.367 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:14.367 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:38:14.367 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:14.367 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:14.367 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:14.368 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:38:14.368 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:14.368 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:14.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:14.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:14.368 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:38:14.368 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:14.368 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:14.368 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:38:14.368 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:14.369 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:14.369 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:14.369 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:38:14.369 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:14.369 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:14.369 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:14.369 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:38:14.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:14.369 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:38:14.369 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:38:14.369 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:38:14.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:14.370 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:38:14Z" 2024-04-20T01:38:14.370 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:38:14.371 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:38:14Z" } 2024-04-20T01:38:14.371 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:38:14.371 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:14.371 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:14.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:38:19.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:19.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:19.373 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:38:19.432 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:38:19.432 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:38:19.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:19.499+0000 7fe60cd3a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:19.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:19.500+0000 7fe60cd3a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:19.509 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:38:19.509 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:38:19.509 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:38:19.509 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:38:19.509 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:38:19.509 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:38:19.509 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:19.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:19.510 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:38:19.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:19.510 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:38:19.510 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:19.510 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:19.510 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:38:19.510 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:19.510 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:38:19.510 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:19.510 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:38:19.510 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:19.511 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:19.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:19.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:19.511 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:38:19.511 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:19.511 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:19.511 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:38:19.511 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:19.511 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:19.511 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:19.511 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:38:19.512 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:19.512 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:19.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:19.512 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:19.512 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:38:19.512 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:19.512 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:19.512 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:38:19.512 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:19.512 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:19.512 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:19.513 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:38:19.513 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:19.513 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:19.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:19.513 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:19.513 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:38:19.513 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:19.513 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:19.513 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:38:19.513 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:19.513 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:19.513 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:19.514 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:38:19.514 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:19.514 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:19.514 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:19.514 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:38:19.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:19.514 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:38:19.514 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:38:19.514 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:38:19.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:19.514 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:38:19Z" 2024-04-20T01:38:19.515 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:38:19.515 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:38:19Z" } 2024-04-20T01:38:19.516 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:38:19.516 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:19.516 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:19.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:38:24.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:24.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:24.517 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:38:24.577 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:38:24.577 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:38:24.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:24.629+0000 7f7dfb766780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:24.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:24.630+0000 7f7dfb766780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:24.636 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:38:24.636 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:38:24.636 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:38:24.636 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:38:24.636 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:38:24.636 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:38:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:38:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:38:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:38:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:38:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:38:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:38:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:38:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:38:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:38:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:38:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:38:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:38:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:38:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:38:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:38:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:38:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:38:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:38:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:38:24Z" 2024-04-20T01:38:24.642 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:38:24.643 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:38:24Z" } 2024-04-20T01:38:24.643 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:38:24.643 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:24.643 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:24.643 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:38:29.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:29.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:29.645 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:38:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:38:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:38:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:29.757+0000 7f9117763780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:29.758+0000 7f9117763780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:29.766 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:38:29.766 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:38:29.767 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:38:29.767 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:38:29.767 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:38:29.767 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:38:29.767 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:29.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:29.767 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:38:29.767 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:29.767 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:38:29.767 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:29.767 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:29.768 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:38:29.768 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:29.768 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:38:29.768 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:29.768 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:38:29.768 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:29.768 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:29.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:29.768 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:29.768 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:38:29.768 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:29.768 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:29.769 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:38:29.769 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:29.769 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:29.769 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:29.769 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:38:29.769 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:29.769 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:29.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:29.769 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:29.769 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:38:29.769 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:29.769 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:29.770 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:38:29.770 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:29.770 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:29.770 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:29.770 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:38:29.770 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:29.770 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:29.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:29.770 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:29.770 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:38:29.770 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:29.770 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:38:29.771 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:38:29.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:29.772 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:38:29Z" 2024-04-20T01:38:29.772 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:38:29.772 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:38:29Z" } 2024-04-20T01:38:29.773 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:38:29.773 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:29.773 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:29.773 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:38:34.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:34.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:34.774 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:38:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:38:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:38:34.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:34.897+0000 7f37474a0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:34.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:34.898+0000 7f37474a0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:34.908 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:38:34.908 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:38:34.909 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:38:34.909 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:38:34.909 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:38:34.909 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:38:34.910 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:34.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:34.910 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:38:34.910 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:34.910 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:38:34.911 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:34.911 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:34.911 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:38:34.911 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:34.911 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:38:34.912 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:34.912 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:38:34.912 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:34.912 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:34.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:34.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:34.913 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:38:34.913 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:34.913 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:34.914 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:38:34.914 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:34.914 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:34.914 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:34.915 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:38:34.915 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:34.915 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:34.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:34.915 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:34.916 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:38:34.916 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:34.916 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:34.916 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:38:34.916 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:34.917 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:34.917 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:34.917 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:38:34.917 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:34.917 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:34.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:34.918 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:34.918 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:38:34.918 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:34.921 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:34.922 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:38:34.922 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:34.922 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:34.922 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:34.923 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:38:34.923 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:34.923 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:34.923 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:34.923 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:38:34.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:34.924 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:38:34.924 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:38:34.924 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:38:34.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:34.925 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:38:34Z" 2024-04-20T01:38:34.925 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:38:34.926 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:38:34Z" } 2024-04-20T01:38:34.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:38:34.927 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:34.927 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:34.928 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:38:39.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:39.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:39.929 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:38:39.988 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:38:39.988 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:38:40.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:40.041+0000 7f3d55bc1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:40.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:40.042+0000 7f3d55bc1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:40.049 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:38:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:38:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:38:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:38:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:38:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:38:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:40.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:40.050 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:38:40.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:40.050 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:38:40.050 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:40.050 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:40.050 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:38:40.050 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:40.050 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:38:40.050 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:40.050 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:38:40.050 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:40.051 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:40.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:40.051 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:40.051 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:38:40.051 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:40.051 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:40.051 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:38:40.051 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:40.051 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:40.051 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:40.051 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:38:40.051 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:40.052 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:40.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:40.052 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:40.052 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:38:40.052 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:40.052 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:40.052 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:38:40.052 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:40.052 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:40.052 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:40.052 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:38:40.053 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:40.053 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:40.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:40.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:40.053 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:38:40.053 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:40.053 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:40.053 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:38:40.053 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:40.053 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:40.053 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:40.053 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:38:40.054 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:40.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:40.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:40.054 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:38:40.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:40.054 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:38:40.054 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:38:40.054 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:38:40.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:40.054 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:38:40Z" 2024-04-20T01:38:40.054 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:38:40.055 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:38:40Z" } 2024-04-20T01:38:40.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:38:40.056 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:40.056 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:40.056 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:38:45.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:45.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:45.057 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:38:45.116 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:38:45.117 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:38:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:45.190+0000 7f1acd257780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:45.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:45.191+0000 7f1acd257780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:45.204 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:38:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:38:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:38:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:38:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:38:45.204 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:38:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:38:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:38:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:38:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:38:45.205 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:38:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:38:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:38:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:45.206 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:38:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:38:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:38:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:45.207 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:38:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:38:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:38:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:45.208 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:38:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:38:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:38:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:38:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:38:45.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:45.210 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:38:45Z" 2024-04-20T01:38:45.210 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:38:45.211 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:38:45Z" } 2024-04-20T01:38:45.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:38:45.211 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:45.211 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:45.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:38:50.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:50.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:50.213 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:38:50.273 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:38:50.274 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:38:50.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:50.328+0000 7fc39b76c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:50.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:50.329+0000 7fc39b76c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:50.342 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:38:50.342 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:38:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:38:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:38:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:38:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:38:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:38:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:38:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:50.343 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:38:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:38:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:38:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:38:50.344 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:38:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:38:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:50.345 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:38:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:38:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:38:50.346 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:38:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:38:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:38:50.347 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:38:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:38:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:38:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:38:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:50.348 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:38:50Z" 2024-04-20T01:38:50.348 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:38:50.349 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713576901.005366_1501.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:35:01.005366Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:38:50Z" } 2024-04-20T01:38:50.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:38:50.350 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:50.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:50.350 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:38:55.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:55.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:38:55.352 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:38:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:38:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:38:55.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:55.466+0000 7f4b64680780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:55.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:38:55.467+0000 7f4b64680780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:38:55.473 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:38:55.473 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:38:55.473 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:38:55.473 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:38:55.473 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:38:55.473 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:38:55.473 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:55.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:55.473 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:38:55.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:55.474 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:38:55.474 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:55.474 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:55.474 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:38:55.474 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:55.474 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:38:55.474 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:55.474 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:38:55.475 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:55.476 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:55.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:55.476 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:55.476 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:38:55.476 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:55.476 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:55.476 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577026.786661_1526.1", 2024-04-20T01:38:55.476 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:55.476 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:55.476 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:55.476 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:06.786661Z", 2024-04-20T01:38:55.477 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:55.477 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:55.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:55.477 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:38:55.477 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:38:55.477 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:38:55.477 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:38:55.477 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:38:55.477 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:38:55.478 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:38:55.478 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:38:55.478 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:38:55.478 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:38:55.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:55.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:38:55.478 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:38:55.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:55.478 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:38:55.479 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:38:55.479 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:38:55.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:38:55.479 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:38:55Z" 2024-04-20T01:38:55.479 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:38:55.480 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577026.786661_1526.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:06.786661Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:38:55Z" } 2024-04-20T01:38:55.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577026.786661_1526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:06.786661Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:38:55.480 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:55.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577026.786661_1526.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:38:55.481 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713577026.786661_1526.1 2024-04-20T01:39:00.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:00.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:00.482 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:00.541 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:00.541 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:00.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:00.595+0000 7f22e0f1b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:00.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:00.596+0000 7f22e0f1b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:00.605 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:39:00.605 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:39:00.605 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:39:00.605 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:39:00.605 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:39:00.605 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:39:00.606 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:00.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:00.606 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:39:00.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:00.606 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:39:00.606 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:00.606 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:00.606 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:39:00.606 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:00.606 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:39:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:39:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:39:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:00.607 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:39:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:39:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:39:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:00.608 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:39:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:39:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:39:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:39:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:00.610 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:39:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:39:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:39:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:39:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:39:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:00.611 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:39:00Z" 2024-04-20T01:39:00.611 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:39:00.612 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:39:00Z" } 2024-04-20T01:39:00.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:39:00.613 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:00.613 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:00.613 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:39:00.613 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:39:00.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:00.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:00.614 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:00.674 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:00.675 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:00.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:39:00.722+0000 7fe27e7f8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:00.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:39:00.723+0000 7fe27e7f8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:00.727 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:39:00.728 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:39:00.728 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:39:00.728 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:39:00.728 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:39:00.728 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:39:00.728 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:39:00.728 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:00.728 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:39:00.728 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:00.728 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:39:00.729 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:00.729 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:39:00.729 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:39:00.729 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:00.729 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:39:00.729 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:00.729 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:39:00.729 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:39:00.729 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:00.729 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:00.729 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:00.730 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:39:00.730 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:00.730 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:39:00.730 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:39:00.730 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:00.730 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:00.730 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:00.730 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:39:00.730 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:39:00.730 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:00.731 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:00.731 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:00.731 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:39:00.731 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:00.731 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:39:00.731 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:39:00.731 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:00.731 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:00.731 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:00.731 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:39:00.731 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:39:00.731 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:00.732 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:00.732 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:00.732 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:39:00.732 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:00.732 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:39:00.732 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:39:00.732 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:00.732 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:00.732 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:00.732 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:39:00.732 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:39:00.733 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:00.733 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:00.733 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:00.733 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:00.733 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:39:00.733 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:39:00.733 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:39:00.733 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:00.733 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:39:00Z" 2024-04-20T01:39:00.733 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:39:00.734 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:39:00Z" } 2024-04-20T01:39:00.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:39:00.735 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:00.735 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:00.735 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:39:00.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:00.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:00.735 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:00.789 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:00.789 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:00.866 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:39:00.867 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:00.867 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:39:00.867 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:39:00.867 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:00.867 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:00.867 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:39:00.867 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:39:00.867 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:00.867 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:00.868 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:39:00.868 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:39:00.868 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:00.868 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:00.868 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000020458", 2024-04-20T01:39:00.868 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:38:37.839807Z" 2024-04-20T01:39:00.868 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:00.868 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:39:00.868 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020458'} 2024-04-20T01:39:00.869 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:39:00.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:00.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:00.869 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:00.931 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:00.931 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:01.020 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:39:01.020 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:39:01.020 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:39:01.020 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:39:01.020 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:39:01.020 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:39:01.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:01.020 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:39:01.021 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:01.021 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:39:01.021 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:01.021 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:39:01.021 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:39:01.021 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:01.021 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:39:01.021 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:01.021 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:39:01.021 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:01.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:01.022 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:01.022 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:39:01.022 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:01.022 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:39:01.022 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026070", 2024-04-20T01:39:01.022 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:01.022 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:39:01.022 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:01.022 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:00.951408Z" 2024-04-20T01:39:01.023 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:01.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:01.023 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:01.023 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:39:01.023 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:01.023 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:39:01.023 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:39:01.023 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:01.023 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:01.023 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:01.023 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:39:01.024 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:01.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:01.024 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:01.024 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:39:01.024 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:01.024 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:39:01.024 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:39:01.024 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:01.024 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:01.025 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:01.025 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:39:01.025 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:01.025 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:01.025 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:39:01.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:01.025 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:39:01.025 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:39:01.025 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:39:01.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:01.025 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:39:01Z" 2024-04-20T01:39:01.026 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:39:01.026 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014922", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:22:09.862093Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026070", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:35:00.951408Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020176", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.871854Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:39:01Z" } 2024-04-20T01:39:01.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014922', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:22:09.862093Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026070', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:35:00.951408Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.871854Z'}}] 2024-04-20T01:39:01.027 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020458'} 2024-04-20T01:39:01.027 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:39:01.027 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015063 target=00000000000000000000:00000000000000014922, shard=1 master=00000000000000000000:00000000000000026490 target=00000000000000000000:00000000000000026070, shard=3 master=00000000000000000000:00000000000000020458 target=00000000000000000000:00000000000000020176 2024-04-20T01:39:06.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:06.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:06.029 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:06.090 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:06.091 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:06.159 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:39:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:39:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:39:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:39:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:39:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:39:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:39:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:39:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:39:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:39:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:39:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:39:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:39:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:39:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026070", 2024-04-20T01:39:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:39:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:35:00.951408Z" 2024-04-20T01:39:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:39:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:39:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:39:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:39:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:39:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:39:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:39:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:39:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:39:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:39:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:39:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:39:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:39:06Z" 2024-04-20T01:39:06.165 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:39:06.166 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014922", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:22:09.862093Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026070", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:35:00.951408Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020176", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.871854Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:39:06Z" } 2024-04-20T01:39:06.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014922', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:22:09.862093Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026070', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:35:00.951408Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.871854Z'}}] 2024-04-20T01:39:06.166 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020458'} 2024-04-20T01:39:06.166 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:39:06.166 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015063 target=00000000000000000000:00000000000000014922, shard=1 master=00000000000000000000:00000000000000026490 target=00000000000000000000:00000000000000026070, shard=3 master=00000000000000000000:00000000000000020458 target=00000000000000000000:00000000000000020176 2024-04-20T01:39:11.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:11.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:11.168 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:11.230 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:11.230 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:11.322 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:39:11.323 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:39:11.323 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:39:11.323 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:39:11.323 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:39:11.323 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:39:11.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:11.323 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:39:11.323 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:11.323 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:39:11.323 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:11.324 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:39:11.324 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:39:11.324 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:11.324 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:39:11.324 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:11.324 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:39:11.324 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:11.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:11.324 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:11.324 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:39:11.324 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:11.325 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:39:11.325 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:39:11.325 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:11.325 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:39:11.325 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:11.325 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:39:11.325 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:11.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:11.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:11.325 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:39:11.325 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:11.326 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:39:11.326 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:39:11.326 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:11.326 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:11.326 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:11.326 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:39:11.326 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:11.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:11.327 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:11.327 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:39:11.327 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:11.327 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:39:11.327 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020458", 2024-04-20T01:39:11.327 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:11.327 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:11.327 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:11.327 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.839807Z" 2024-04-20T01:39:11.328 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:11.328 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:11.328 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:39:11.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:11.328 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:39:11.328 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:39:11.328 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:39:11.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:11.328 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:39:11Z" 2024-04-20T01:39:11.328 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:39:11.329 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015063", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:38:37.851769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026490", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:38:37.842528Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020458", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:38:37.839807Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:39:11Z" } 2024-04-20T01:39:11.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015063', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.851769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026490', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.842528Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020458', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.839807Z'}}] 2024-04-20T01:39:11.330 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020458'} 2024-04-20T01:39:11.330 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020458'} 2024-04-20T01:39:11.330 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:39:11.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:11.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:11.331 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:11.386 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:11.473 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-04-20T01:39:11.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:11.473 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:39:11.473 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:39:11.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:11.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:11.473 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000012909", 2024-04-20T01:39:11.473 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-20T01:39:08.004155Z" 2024-04-20T01:39:11.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:11.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:11.474 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:39:11.474 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:39:11.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:11.474 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:11.474 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000008844", 2024-04-20T01:39:11.474 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-20T01:39:08.044239Z" 2024-04-20T01:39:11.474 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:11.474 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-04-20T01:39:11.475 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008844'} 2024-04-20T01:39:11.475 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-20T01:39:11.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:11.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:11.476 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:11.539 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:11.539 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:11.610 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:39:11.610 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:39:11.610 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:39:11.610 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:39:11.610 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:39:11.610 INFO:teuthology.orchestra.run.smithi019.stdout: "instance_id": 11239922052735367740 2024-04-20T01:39:11.610 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:11.610 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:39:11.610 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:11.611 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:39:11.611 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:11.611 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:11.611 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:39:11.611 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:11.611 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:11.611 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:11.611 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:39:11.611 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:11.611 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:11.611 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:11.611 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:39:11.612 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:11.612 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:11.612 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:39:11.612 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:11.612 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:11.612 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:11.612 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:37:07.140694Z" 2024-04-20T01:39:11.612 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:11.612 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:11.612 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:11.612 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:39:11.612 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:11.613 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:11.613 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:39:11.613 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:11.613 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:11.613 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:11.613 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:39:11.613 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:11.613 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:11.613 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:11.613 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:39:11.613 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:11.614 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:11.614 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000008704", 2024-04-20T01:39:11.614 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:11.614 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:11.614 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:11.614 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:37:07.190175Z" 2024-04-20T01:39:11.614 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:11.614 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:11.614 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:11.614 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:11.614 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:39:11.614 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 0, 2024-04-20T01:39:11.615 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 0 2024-04-20T01:39:11.615 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:11.615 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:39:11Z" 2024-04-20T01:39:11.615 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:39:11.616 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11239922052735367740 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007274", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.116484Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012769", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.140694Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012769", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:32:53.230970Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008704", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.190175Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T01:39:11Z" } 2024-04-20T01:39:11.616 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.116484Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012769', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.140694Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012769', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:53.230970Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008704', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.190175Z'}}] 2024-04-20T01:39:11.616 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008844'} 2024-04-20T01:39:11.616 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012769', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008704'} 2024-04-20T01:39:11.616 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=1 master=00000000000000000000:00000000000000012909 target=00000000000000000000:00000000000000012769, shard=3 master=00000000000000000000:00000000000000008844 target=00000000000000000000:00000000000000008704 2024-04-20T01:39:16.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:16.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:16.617 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:16.681 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:16.681 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:16.740 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:39:16.741 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:39:16.741 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:39:16.741 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:39:16.741 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:39:16.741 INFO:teuthology.orchestra.run.smithi019.stdout: "instance_id": 11239922052735367740 2024-04-20T01:39:16.741 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:16.741 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:39:16.741 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:16.741 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:39:16.741 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:16.741 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:16.742 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:39:16.742 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:16.742 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:16.742 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:16.742 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:39:16.742 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:16.742 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:16.742 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:16.742 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:39:16.742 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:16.742 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:16.743 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:39:16.743 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:16.743 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:16.743 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:16.743 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:37:07.140694Z" 2024-04-20T01:39:16.743 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:16.743 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:16.743 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:16.743 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:39:16.743 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:16.743 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:16.744 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:39:16.744 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:16.744 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:16.744 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:16.744 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:39:16.744 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:16.744 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:16.744 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:16.744 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:39:16.744 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:16.744 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:16.744 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000008704", 2024-04-20T01:39:16.745 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:16.745 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:16.745 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:16.745 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:37:07.190175Z" 2024-04-20T01:39:16.745 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:16.745 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:16.745 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:16.745 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:16.745 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:39:16.745 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 0, 2024-04-20T01:39:16.745 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 0 2024-04-20T01:39:16.746 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:16.746 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:39:16Z" 2024-04-20T01:39:16.746 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:39:16.747 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11239922052735367740 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007274", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.116484Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012769", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.140694Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012769", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:32:53.230970Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008704", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.190175Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T01:39:16Z" } 2024-04-20T01:39:16.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.116484Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012769', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.140694Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012769', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:53.230970Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008704', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.190175Z'}}] 2024-04-20T01:39:16.747 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008844'} 2024-04-20T01:39:16.747 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012769', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008704'} 2024-04-20T01:39:16.747 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=1 master=00000000000000000000:00000000000000012909 target=00000000000000000000:00000000000000012769, shard=3 master=00000000000000000000:00000000000000008844 target=00000000000000000000:00000000000000008704 2024-04-20T01:39:21.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:21.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:21.748 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:22.870 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:39:22.870 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:39:22.871 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:39:22.871 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:39:22.871 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:39:22.871 INFO:teuthology.orchestra.run.smithi019.stdout: "instance_id": 11239922052735367740 2024-04-20T01:39:22.871 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:22.871 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:39:22.871 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:22.871 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:39:22.871 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:22.871 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:22.872 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:39:22.872 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:22.872 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:22.872 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:22.872 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:39:22.872 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:22.872 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:22.872 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:22.872 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:39:22.872 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:22.872 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:22.872 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000012909", 2024-04-20T01:39:22.873 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:22.873 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:22.873 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:22.873 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:39:08.004155Z" 2024-04-20T01:39:22.873 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:22.873 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:22.873 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:22.873 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:39:22.873 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:22.873 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:22.873 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:39:22.874 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:22.874 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:22.874 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:22.874 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:39:22.874 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:22.874 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:22.874 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:22.874 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:39:22.874 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:22.874 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:22.874 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000008704", 2024-04-20T01:39:22.875 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:22.875 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:22.875 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:22.875 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:37:07.190175Z" 2024-04-20T01:39:22.875 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:22.875 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:22.875 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:22.875 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:22.875 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:39:22.875 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 0, 2024-04-20T01:39:22.875 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 0 2024-04-20T01:39:22.876 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:22.876 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:39:22Z" 2024-04-20T01:39:22.876 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:39:22.877 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11239922052735367740 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007274", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.116484Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:39:08.004155Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012769", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:32:53.230970Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008704", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.190175Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T01:39:22Z" } 2024-04-20T01:39:22.877 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.116484Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:08.004155Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012769', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:53.230970Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008704', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.190175Z'}}] 2024-04-20T01:39:22.877 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008844'} 2024-04-20T01:39:22.877 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008704'} 2024-04-20T01:39:22.877 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=3 master=00000000000000000000:00000000000000008844 target=00000000000000000000:00000000000000008704 2024-04-20T01:39:27.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:27.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:27.878 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:27.945 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:27.946 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:28.003 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:39:28.003 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:39:28.003 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:39:28.004 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:39:28.004 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:39:28.004 INFO:teuthology.orchestra.run.smithi019.stdout: "instance_id": 11239922052735367740 2024-04-20T01:39:28.004 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.004 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:39:28.004 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:28.004 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:39:28.004 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:28.004 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:28.004 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:39:28.004 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:39:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:39:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000012909", 2024-04-20T01:39:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:39:08.004155Z" 2024-04-20T01:39:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:39:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:39:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:39:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:39:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:39:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000008844", 2024-04-20T01:39:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:28.008 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:39:08.044239Z" 2024-04-20T01:39:28.008 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:28.008 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:28.008 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:28.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.008 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:39:28.008 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 0, 2024-04-20T01:39:28.008 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 0 2024-04-20T01:39:28.008 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.008 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:39:28Z" 2024-04-20T01:39:28.008 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:39:28.009 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11239922052735367740 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007274", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.116484Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012909", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:39:08.004155Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012769", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:32:53.230970Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008844", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:39:08.044239Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T01:39:28Z" } 2024-04-20T01:39:28.009 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.116484Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:08.004155Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012769', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:53.230970Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008844', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:08.044239Z'}}] 2024-04-20T01:39:28.010 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008844'} 2024-04-20T01:39:28.010 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008844'} 2024-04-20T01:39:28.010 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-20T01:39:28.010 DEBUG:boto:path=/aktlji-82/ 2024-04-20T01:39:28.010 DEBUG:boto:auth_path=/aktlji-82/ 2024-04-20T01:39:28.011 DEBUG:boto:Method: HEAD 2024-04-20T01:39:28.011 DEBUG:boto:Path: /aktlji-82/ 2024-04-20T01:39:28.011 DEBUG:boto:Data: 2024-04-20T01:39:28.011 DEBUG:boto:Headers: {} 2024-04-20T01:39:28.011 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:28.011 DEBUG:boto:Port: 8000 2024-04-20T01:39:28.011 DEBUG:boto:Params: {} 2024-04-20T01:39:28.011 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:39:28.012 DEBUG:boto:Token: None 2024-04-20T01:39:28.012 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:39:28 GMT /aktlji-82/ 2024-04-20T01:39:28.012 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:eovWTalBBgr6i/AWulGbvpwwm18= 2024-04-20T01:39:28.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eovWTalBBgr6i/AWulGbvpwwm18=', 'Content-Length': '0'} 2024-04-20T01:39:28.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c5d3a7447dc8fdc-0066231cd0-2227-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:28.017 DEBUG:boto:path=/aktlji-82/obj1 2024-04-20T01:39:28.017 DEBUG:boto:auth_path=/aktlji-82/obj1 2024-04-20T01:39:28.017 DEBUG:boto:Method: HEAD 2024-04-20T01:39:28.017 DEBUG:boto:Path: /aktlji-82/obj1 2024-04-20T01:39:28.017 DEBUG:boto:Data: 2024-04-20T01:39:28.017 DEBUG:boto:Headers: {} 2024-04-20T01:39:28.017 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:28.018 DEBUG:boto:Port: 8000 2024-04-20T01:39:28.018 DEBUG:boto:Params: {} 2024-04-20T01:39:28.018 DEBUG:boto:Token: None 2024-04-20T01:39:28.018 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:39:28 GMT /aktlji-82/obj1 2024-04-20T01:39:28.018 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wLd8AqGjDbuTv+1f9M2IBcGJYrg= 2024-04-20T01:39:28.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wLd8AqGjDbuTv+1f9M2IBcGJYrg=', 'Content-Length': '0'} 2024-04-20T01:39:28.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-82:73f46f02-c7ab-4426-a267-2273af911da5.6793.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098d87b99f5d5f0d6-0066231cd0-2227-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:28.020 DEBUG:boto:path=/aktlji-82/obj1 2024-04-20T01:39:28.020 DEBUG:boto:auth_path=/aktlji-82/obj1 2024-04-20T01:39:28.020 DEBUG:boto:Method: GET 2024-04-20T01:39:28.021 DEBUG:boto:Path: /aktlji-82/obj1 2024-04-20T01:39:28.021 DEBUG:boto:Data: 2024-04-20T01:39:28.021 DEBUG:boto:Headers: {} 2024-04-20T01:39:28.021 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:28.021 DEBUG:boto:Port: 8000 2024-04-20T01:39:28.021 DEBUG:boto:Params: {} 2024-04-20T01:39:28.021 DEBUG:boto:Token: None 2024-04-20T01:39:28.021 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:39:28 GMT /aktlji-82/obj1 2024-04-20T01:39:28.021 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:1tO/c646dPtzLxbcMWXh17HNUTI= 2024-04-20T01:39:28.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1tO/c646dPtzLxbcMWXh17HNUTI=', 'Content-Length': '0'} 2024-04-20T01:39:28.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-82:73f46f02-c7ab-4426-a267-2273af911da5.6793.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007625753151cdf63d-0066231cd0-2227-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:28.023 DEBUG:boto:path=/aktlji-82/obj2 2024-04-20T01:39:28.024 DEBUG:boto:auth_path=/aktlji-82/obj2 2024-04-20T01:39:28.024 DEBUG:boto:Method: HEAD 2024-04-20T01:39:28.024 DEBUG:boto:Path: /aktlji-82/obj2 2024-04-20T01:39:28.024 DEBUG:boto:Data: 2024-04-20T01:39:28.024 DEBUG:boto:Headers: {} 2024-04-20T01:39:28.024 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:28.024 DEBUG:boto:Port: 8000 2024-04-20T01:39:28.024 DEBUG:boto:Params: {} 2024-04-20T01:39:28.024 DEBUG:boto:Token: None 2024-04-20T01:39:28.024 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:39:28 GMT /aktlji-82/obj2 2024-04-20T01:39:28.025 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/hX3pqwLf/yjrwXKCwBer7NhSvk= 2024-04-20T01:39:28.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/hX3pqwLf/yjrwXKCwBer7NhSvk=', 'Content-Length': '0'} 2024-04-20T01:39:28.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-82:73f46f02-c7ab-4426-a267-2273af911da5.6793.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d1900ceb4a58eed-0066231cd0-2227-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:28.026 DEBUG:boto:path=/aktlji-82/obj2 2024-04-20T01:39:28.027 DEBUG:boto:auth_path=/aktlji-82/obj2 2024-04-20T01:39:28.027 DEBUG:boto:Method: GET 2024-04-20T01:39:28.027 DEBUG:boto:Path: /aktlji-82/obj2 2024-04-20T01:39:28.027 DEBUG:boto:Data: 2024-04-20T01:39:28.027 DEBUG:boto:Headers: {} 2024-04-20T01:39:28.027 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:28.027 DEBUG:boto:Port: 8000 2024-04-20T01:39:28.027 DEBUG:boto:Params: {} 2024-04-20T01:39:28.027 DEBUG:boto:Token: None 2024-04-20T01:39:28.027 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:39:28 GMT /aktlji-82/obj2 2024-04-20T01:39:28.028 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:N7vF+RL++FlekOi9yOlLDrOrs7M= 2024-04-20T01:39:28.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N7vF+RL++FlekOi9yOlLDrOrs7M=', 'Content-Length': '0'} 2024-04-20T01:39:28.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-82:73f46f02-c7ab-4426-a267-2273af911da5.6793.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef26ed010c8bccbb-0066231cd0-2227-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:28.029 DEBUG:boto:path=/aktlji-82/obj3 2024-04-20T01:39:28.030 DEBUG:boto:auth_path=/aktlji-82/obj3 2024-04-20T01:39:28.030 DEBUG:boto:Method: HEAD 2024-04-20T01:39:28.030 DEBUG:boto:Path: /aktlji-82/obj3 2024-04-20T01:39:28.030 DEBUG:boto:Data: 2024-04-20T01:39:28.030 DEBUG:boto:Headers: {} 2024-04-20T01:39:28.030 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:28.030 DEBUG:boto:Port: 8000 2024-04-20T01:39:28.030 DEBUG:boto:Params: {} 2024-04-20T01:39:28.030 DEBUG:boto:Token: None 2024-04-20T01:39:28.031 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:39:28 GMT /aktlji-82/obj3 2024-04-20T01:39:28.031 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6hQMOL8+4wP3cXbr9AHvOb3qdrQ= 2024-04-20T01:39:28.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6hQMOL8+4wP3cXbr9AHvOb3qdrQ=', 'Content-Length': '0'} 2024-04-20T01:39:28.032 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000015d3e789bebfb61-0066231cd0-2227-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:28.035 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... FAIL 2024-04-20T01:39:28.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:28.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:28.036 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:28.092 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:28.092 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:28.132 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:39:28.148 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:39:28.148 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:28.149 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:39:28.149 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:39:28.149 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.149 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:28.149 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:39:28.149 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:39:28.149 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.149 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:28.149 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:39:28.149 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:37:07.057559Z" 2024-04-20T01:39:28.149 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.150 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:28.150 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:39:28.150 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:39:28.150 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:28.150 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:39:28.151 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:28.151 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:39:28.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:28.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:28.152 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:28.211 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:28.211 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:28.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:28.272+0000 7f14416e0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:28.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:28.273+0000 7f14416e0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:28.281 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:39:28.281 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:39:28.281 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:39:28.281 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:39:28.282 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:39:28.282 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:39:28.282 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:28.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:28.282 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:39:28.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:28.282 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:39:28.282 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:28.282 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:28.282 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:39:28.282 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:28.283 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:39:28.283 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:28.283 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:39:28.283 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:28.283 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:28.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:28.283 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:28.283 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:39:28.283 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:28.283 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:28.284 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:39:28.284 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:28.284 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:28.284 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:28.284 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:39:28.284 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:28.284 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:28.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:28.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:28.284 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:39:28.284 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:28.285 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:28.285 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:39:28.285 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:28.285 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:28.285 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:28.285 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:39:28.285 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:28.285 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:28.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:28.285 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:28.285 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:39:28.286 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:28.286 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:28.286 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:39:28.286 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:28.286 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:28.286 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:28.286 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:39:28.286 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:28.286 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:28.286 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:28.286 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:39:28.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:28.287 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:39:28.287 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:39:28.287 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:39:28.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:28.287 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:39:28Z" 2024-04-20T01:39:28.287 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:39:28.288 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:39:28Z" } 2024-04-20T01:39:28.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:39:28.288 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:28.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:28.288 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:39:28.289 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:39:28.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:28.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:28.289 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:28.349 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:28.349 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:28.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:39:28.403+0000 7fba96bfd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:28.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:39:28.404+0000 7fba96bfd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:28.408 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:39:28.408 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:39:28.408 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:39:28.408 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:39:28.408 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:39:28.408 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:39:28.409 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:39:28.409 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.409 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:39:28.409 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:28.409 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:39:28.409 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:28.409 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:39:28.409 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:39:28.409 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:28.409 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:39:28.409 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:28.410 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:39:28.410 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:39:28.410 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:28.410 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.410 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:28.410 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:39:28.410 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:28.410 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:39:28.410 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:39:28.410 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:28.411 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:28.411 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:28.411 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:39:28.411 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:39:28.411 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:28.411 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.411 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:28.411 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:39:28.411 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:28.411 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:39:28.412 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:39:28.412 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:28.412 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:28.412 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:28.412 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:39:28.412 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:39:28.412 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:28.412 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.412 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:28.412 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:39:28.412 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:39:28.413 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:39:28.413 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:39:28.413 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:39:28.413 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:39:28.413 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:39:28.413 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:39:28.413 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:39:28.413 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:28.413 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:28.413 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:28.413 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.414 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:39:28.414 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:39:28.414 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:39:28.414 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:28.414 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:39:28Z" 2024-04-20T01:39:28.414 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:39:28.415 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:39:28Z" } 2024-04-20T01:39:28.415 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:39:28.415 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:28.415 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:28.415 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:39:28.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:28.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:28.416 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:39:28.466 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:28.467 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:28.529 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:39:28.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:28.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:28.530 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:39:28.584 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:28.584 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:28.643 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:39:28.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:28.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:28.644 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:39:28.703 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:28.703 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:28.762 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:39:28.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:28.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:28.762 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:39:28.819 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:28.819 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:28.885 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:39:28.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:28.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:28.885 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:39:28.940 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:28.941 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:29.008 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:39:29.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:29.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:29.008 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:39:29.065 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:29.065 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:29.173 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:39:29.173 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:39:29.174 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 13, 2024-04-20T01:39:29.174 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:39:29.174 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T01:39:29.174 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T01:39:29.174 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:39:29.174 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T01:39:29.174 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.174 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T01:39:29.174 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T01:39:29.174 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T01:39:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T01:39:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:39:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:39:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:39:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T01:39:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T01:39:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:39:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:39:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:39:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T01:39:29.176 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:39:29.176 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:39:29.176 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:39:29.176 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:39:29.176 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:39:29.176 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:39:29.176 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:39:29.176 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:39:29.176 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T01:39:29.176 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:39:29.176 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:39:29.176 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:39:29.177 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:39:29.177 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:39:29.177 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:29.177 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.177 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.177 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:39:29.177 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T01:39:29.177 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:39:29.177 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:39:29.177 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:39:29.178 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:39:29.178 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:39:29.178 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:39:29.178 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:39:29.178 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:39:29.178 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:39:29.178 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:39:29.178 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:39:29.178 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:39:29.179 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:39:29.179 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:39:29.179 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:29.179 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.179 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.179 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:39:29.179 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T01:39:29.179 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:39:29.179 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:39:29.179 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:39:29.179 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:39:29.180 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:39:29.180 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:39:29.180 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:39:29.180 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:39:29.180 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:39:29.180 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:39:29.180 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:39:29.180 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:39:29.180 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:39:29.180 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:39:29.180 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:29.181 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:29.181 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:39:29.181 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T01:39:29.181 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.181 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T01:39:29.181 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T01:39:29.181 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T01:39:29.181 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T01:39:29.181 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:29.181 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:29.181 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:39:29.182 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T01:39:29.182 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:39:29.182 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T01:39:29.182 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [ 2024-04-20T01:39:29.182 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.182 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-group", 2024-04-20T01:39:29.182 INFO:teuthology.orchestra.run.smithi019.stdout: "data_flow": { 2024-04-20T01:39:29.183 INFO:teuthology.orchestra.run.smithi019.stdout: "symmetrical": [ 2024-04-20T01:39:29.183 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.183 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow1", 2024-04-20T01:39:29.183 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:39:29.183 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:39:29.183 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:39:29.183 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:29.183 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.183 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.183 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow", 2024-04-20T01:39:29.184 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:39:29.184 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:39:29.184 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:39:29.184 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:29.184 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:29.184 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:39:29.184 INFO:teuthology.orchestra.run.smithi019.stdout: "directional": [ 2024-04-20T01:39:29.184 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.184 INFO:teuthology.orchestra.run.smithi019.stdout: "source_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:39:29.184 INFO:teuthology.orchestra.run.smithi019.stdout: "dest_zone": "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:39:29.185 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:29.185 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:29.185 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.185 INFO:teuthology.orchestra.run.smithi019.stdout: "pipes": [ 2024-04-20T01:39:29.185 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.185 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe", 2024-04-20T01:39:29.185 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:39:29.185 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "*", 2024-04-20T01:39:29.185 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:39:29.185 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:39:29.185 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:39:29.185 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:39:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "*", 2024-04-20T01:39:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:39:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:39:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:39:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:39:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:39:29.186 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:39:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:39:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:39:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:39:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:39:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:39:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:39:29.187 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "enabled" 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T01:39:29.188 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:39:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T01:39:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:39:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T01:39:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:39:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.189 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T01:39:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:39:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T01:39:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T01:39:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:39:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:39:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:39:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:39:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:39:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T01:39:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:39:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:39:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:39:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:39:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:39:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T01:39:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:39:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:39:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:39:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:39:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:39:29.191 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T01:39:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:39:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:39:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:39:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:39:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:39:29.192 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.201 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T01:39:29.201 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:39:29.201 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:39:29.201 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:39:29.201 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:39:29.201 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:39:29.202 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:29.202 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:29.202 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:39:29.202 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T01:39:29.202 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:39:29.202 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:39:29.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:29.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:39:29.204 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:39:29.261 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:29.261 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:29.332 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:39:29.332 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-84 2024-04-20T01:39:29.332 DEBUG:boto:path=/aktlji-84/ 2024-04-20T01:39:29.333 DEBUG:boto:auth_path=/aktlji-84/ 2024-04-20T01:39:29.333 DEBUG:boto:Method: PUT 2024-04-20T01:39:29.333 DEBUG:boto:Path: /aktlji-84/ 2024-04-20T01:39:29.333 DEBUG:boto:Data: 2024-04-20T01:39:29.333 DEBUG:boto:Headers: {} 2024-04-20T01:39:29.333 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:39:29.333 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.333 DEBUG:boto:Params: {} 2024-04-20T01:39:29.333 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:39:29.334 DEBUG:boto:Token: None 2024-04-20T01:39:29.334 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:39:29 GMT /aktlji-84/ 2024-04-20T01:39:29.334 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:raIT75XfteKyAnPqirknLmovY/I= 2024-04-20T01:39:29.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:raIT75XfteKyAnPqirknLmovY/I=', 'Content-Length': '0'} 2024-04-20T01:39:29.456 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c1a1ba9a3b40159e-0066231cd1-1afa-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.456 DEBUG:boto:path=/aktlji-84/ 2024-04-20T01:39:29.456 DEBUG:boto:auth_path=/aktlji-84/ 2024-04-20T01:39:29.456 DEBUG:boto:Method: HEAD 2024-04-20T01:39:29.456 DEBUG:boto:Path: /aktlji-84/ 2024-04-20T01:39:29.457 DEBUG:boto:Data: 2024-04-20T01:39:29.457 DEBUG:boto:Headers: {} 2024-04-20T01:39:29.457 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:39:29.457 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.457 DEBUG:boto:Params: {} 2024-04-20T01:39:29.457 DEBUG:boto:Token: None 2024-04-20T01:39:29.457 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:39:29 GMT /aktlji-84/ 2024-04-20T01:39:29.457 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PTXe08CBU/rEonEezwlIT/DoYF0= 2024-04-20T01:39:29.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PTXe08CBU/rEonEezwlIT/DoYF0=', 'Content-Length': '0'} 2024-04-20T01:39:29.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff18b47de1847e38-0066231cd1-1afa-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.462 DEBUG:boto:path=/aktlji-84/obj1 2024-04-20T01:39:29.462 DEBUG:boto:auth_path=/aktlji-84/obj1 2024-04-20T01:39:29.463 DEBUG:boto:Method: PUT 2024-04-20T01:39:29.463 DEBUG:boto:Path: /aktlji-84/obj1 2024-04-20T01:39:29.463 DEBUG:boto:Data: 2024-04-20T01:39:29.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:39:29.463 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:39:29.463 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.463 DEBUG:boto:Params: {} 2024-04-20T01:39:29.463 DEBUG:boto:Token: None 2024-04-20T01:39:29.463 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:39:29 GMT /aktlji-84/obj1 2024-04-20T01:39:29.464 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bN9WfzkIjsC3fzqVU7pZrdMKzwA= 2024-04-20T01:39:29.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bN9WfzkIjsC3fzqVU7pZrdMKzwA='} 2024-04-20T01:39:29.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099b6cd1479cb2ea3-0066231cd1-1afa-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.482 DEBUG:boto:path=/aktlji-84/ 2024-04-20T01:39:29.482 DEBUG:boto:auth_path=/aktlji-84/ 2024-04-20T01:39:29.482 DEBUG:boto:Method: HEAD 2024-04-20T01:39:29.482 DEBUG:boto:Path: /aktlji-84/ 2024-04-20T01:39:29.482 DEBUG:boto:Data: 2024-04-20T01:39:29.482 DEBUG:boto:Headers: {} 2024-04-20T01:39:29.483 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:39:29.483 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.483 DEBUG:boto:Params: {} 2024-04-20T01:39:29.483 DEBUG:boto:Token: None 2024-04-20T01:39:29.483 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:39:29 GMT /aktlji-84/ 2024-04-20T01:39:29.483 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PTXe08CBU/rEonEezwlIT/DoYF0= 2024-04-20T01:39:29.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PTXe08CBU/rEonEezwlIT/DoYF0=', 'Content-Length': '0'} 2024-04-20T01:39:29.487 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f602b1c77dc448a5-0066231cd1-1afa-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.487 DEBUG:boto:path=/aktlji-84/obj2 2024-04-20T01:39:29.487 DEBUG:boto:auth_path=/aktlji-84/obj2 2024-04-20T01:39:29.487 DEBUG:boto:Method: PUT 2024-04-20T01:39:29.487 DEBUG:boto:Path: /aktlji-84/obj2 2024-04-20T01:39:29.487 DEBUG:boto:Data: 2024-04-20T01:39:29.487 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:39:29.487 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:39:29.487 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.488 DEBUG:boto:Params: {} 2024-04-20T01:39:29.488 DEBUG:boto:Token: None 2024-04-20T01:39:29.488 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:39:29 GMT /aktlji-84/obj2 2024-04-20T01:39:29.488 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gXBIRZ8az4J5ggdSWtQS7P1Cs4M= 2024-04-20T01:39:29.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gXBIRZ8az4J5ggdSWtQS7P1Cs4M='} 2024-04-20T01:39:29.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1f7412ed7d75cdb-0066231cd1-1afa-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.531 DEBUG:boto:path=/aktlji-84/ 2024-04-20T01:39:29.531 DEBUG:boto:auth_path=/aktlji-84/ 2024-04-20T01:39:29.531 DEBUG:boto:Method: HEAD 2024-04-20T01:39:29.531 DEBUG:boto:Path: /aktlji-84/ 2024-04-20T01:39:29.531 DEBUG:boto:Data: 2024-04-20T01:39:29.531 DEBUG:boto:Headers: {} 2024-04-20T01:39:29.531 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:39:29.531 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.531 DEBUG:boto:Params: {} 2024-04-20T01:39:29.532 DEBUG:boto:Token: None 2024-04-20T01:39:29.532 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:39:29 GMT /aktlji-84/ 2024-04-20T01:39:29.532 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PTXe08CBU/rEonEezwlIT/DoYF0= 2024-04-20T01:39:29.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PTXe08CBU/rEonEezwlIT/DoYF0=', 'Content-Length': '0'} 2024-04-20T01:39:29.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b01ec849296601f1-0066231cd1-1afa-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.534 DEBUG:boto:path=/aktlji-84/obj3 2024-04-20T01:39:29.534 DEBUG:boto:auth_path=/aktlji-84/obj3 2024-04-20T01:39:29.535 DEBUG:boto:Method: PUT 2024-04-20T01:39:29.535 DEBUG:boto:Path: /aktlji-84/obj3 2024-04-20T01:39:29.535 DEBUG:boto:Data: 2024-04-20T01:39:29.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:39:29.535 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:39:29.535 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.535 DEBUG:boto:Params: {} 2024-04-20T01:39:29.535 DEBUG:boto:Token: None 2024-04-20T01:39:29.535 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:39:29 GMT /aktlji-84/obj3 2024-04-20T01:39:29.535 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/RDNyHoRPPCz6MXdGMVdcRWCCAU= 2024-04-20T01:39:29.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/RDNyHoRPPCz6MXdGMVdcRWCCAU='} 2024-04-20T01:39:29.555 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005dcba42ed1e2e964-0066231cd1-1afa-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.555 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=aktlji-85 2024-04-20T01:39:29.555 DEBUG:boto:path=/aktlji-85/ 2024-04-20T01:39:29.555 DEBUG:boto:auth_path=/aktlji-85/ 2024-04-20T01:39:29.555 DEBUG:boto:Method: PUT 2024-04-20T01:39:29.555 DEBUG:boto:Path: /aktlji-85/ 2024-04-20T01:39:29.555 DEBUG:boto:Data: 2024-04-20T01:39:29.555 DEBUG:boto:Headers: {} 2024-04-20T01:39:29.556 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:29.556 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.556 DEBUG:boto:Params: {} 2024-04-20T01:39:29.556 DEBUG:boto:Token: None 2024-04-20T01:39:29.556 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:39:29 GMT /aktlji-85/ 2024-04-20T01:39:29.556 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VseUGqI3SdX/5C45LGSfHMewHvg= 2024-04-20T01:39:29.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VseUGqI3SdX/5C45LGSfHMewHvg=', 'Content-Length': '0'} 2024-04-20T01:39:29.801 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000083346015ba5118d6-0066231cd1-230b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.801 DEBUG:boto:path=/aktlji-85/ 2024-04-20T01:39:29.801 DEBUG:boto:auth_path=/aktlji-85/ 2024-04-20T01:39:29.801 DEBUG:boto:Method: HEAD 2024-04-20T01:39:29.801 DEBUG:boto:Path: /aktlji-85/ 2024-04-20T01:39:29.801 DEBUG:boto:Data: 2024-04-20T01:39:29.802 DEBUG:boto:Headers: {} 2024-04-20T01:39:29.802 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:29.802 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.802 DEBUG:boto:Params: {} 2024-04-20T01:39:29.802 DEBUG:boto:Token: None 2024-04-20T01:39:29.802 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:39:29 GMT /aktlji-85/ 2024-04-20T01:39:29.802 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WFH3enhYO3U6bq0T6oJEaghkNKY= 2024-04-20T01:39:29.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WFH3enhYO3U6bq0T6oJEaghkNKY=', 'Content-Length': '0'} 2024-04-20T01:39:29.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044a83feb71594998-0066231cd1-230b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.806 DEBUG:boto:path=/aktlji-85/obj4 2024-04-20T01:39:29.806 DEBUG:boto:auth_path=/aktlji-85/obj4 2024-04-20T01:39:29.806 DEBUG:boto:Method: PUT 2024-04-20T01:39:29.806 DEBUG:boto:Path: /aktlji-85/obj4 2024-04-20T01:39:29.806 DEBUG:boto:Data: 2024-04-20T01:39:29.806 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:39:29.806 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:29.806 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.806 DEBUG:boto:Params: {} 2024-04-20T01:39:29.807 DEBUG:boto:Token: None 2024-04-20T01:39:29.807 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:39:29 GMT /aktlji-85/obj4 2024-04-20T01:39:29.807 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:FoRwJa2TRFdFg+zFcIDXUlSr6GM= 2024-04-20T01:39:29.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FoRwJa2TRFdFg+zFcIDXUlSr6GM='} 2024-04-20T01:39:29.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8ab8bb44a16dc5a-0066231cd1-230b-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.855 DEBUG:boto:path=/aktlji-85/ 2024-04-20T01:39:29.855 DEBUG:boto:auth_path=/aktlji-85/ 2024-04-20T01:39:29.855 DEBUG:boto:Method: HEAD 2024-04-20T01:39:29.855 DEBUG:boto:Path: /aktlji-85/ 2024-04-20T01:39:29.855 DEBUG:boto:Data: 2024-04-20T01:39:29.855 DEBUG:boto:Headers: {} 2024-04-20T01:39:29.855 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:29.855 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.856 DEBUG:boto:Params: {} 2024-04-20T01:39:29.856 DEBUG:boto:Token: None 2024-04-20T01:39:29.856 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:39:29 GMT /aktlji-85/ 2024-04-20T01:39:29.856 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WFH3enhYO3U6bq0T6oJEaghkNKY= 2024-04-20T01:39:29.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WFH3enhYO3U6bq0T6oJEaghkNKY=', 'Content-Length': '0'} 2024-04-20T01:39:29.859 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4f86f3df0c0af55-0066231cd1-230b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.859 DEBUG:boto:path=/aktlji-85/obj5 2024-04-20T01:39:29.859 DEBUG:boto:auth_path=/aktlji-85/obj5 2024-04-20T01:39:29.859 DEBUG:boto:Method: PUT 2024-04-20T01:39:29.859 DEBUG:boto:Path: /aktlji-85/obj5 2024-04-20T01:39:29.859 DEBUG:boto:Data: 2024-04-20T01:39:29.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:39:29.859 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:29.859 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.860 DEBUG:boto:Params: {} 2024-04-20T01:39:29.860 DEBUG:boto:Token: None 2024-04-20T01:39:29.860 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:39:29 GMT /aktlji-85/obj5 2024-04-20T01:39:29.860 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:kMoUfM+7EiuTlwTX7tjG7+0bOSg= 2024-04-20T01:39:29.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kMoUfM+7EiuTlwTX7tjG7+0bOSg='} 2024-04-20T01:39:29.882 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fdd482859e3c233c-0066231cd1-230b-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.882 DEBUG:boto:path=/aktlji-85/ 2024-04-20T01:39:29.882 DEBUG:boto:auth_path=/aktlji-85/ 2024-04-20T01:39:29.882 DEBUG:boto:Method: HEAD 2024-04-20T01:39:29.882 DEBUG:boto:Path: /aktlji-85/ 2024-04-20T01:39:29.883 DEBUG:boto:Data: 2024-04-20T01:39:29.883 DEBUG:boto:Headers: {} 2024-04-20T01:39:29.883 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:29.883 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.883 DEBUG:boto:Params: {} 2024-04-20T01:39:29.883 DEBUG:boto:Token: None 2024-04-20T01:39:29.883 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:39:29 GMT /aktlji-85/ 2024-04-20T01:39:29.883 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WFH3enhYO3U6bq0T6oJEaghkNKY= 2024-04-20T01:39:29.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WFH3enhYO3U6bq0T6oJEaghkNKY=', 'Content-Length': '0'} 2024-04-20T01:39:29.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003efbf367a26357f7-0066231cd1-230b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.886 DEBUG:boto:path=/aktlji-85/obj6 2024-04-20T01:39:29.887 DEBUG:boto:auth_path=/aktlji-85/obj6 2024-04-20T01:39:29.887 DEBUG:boto:Method: PUT 2024-04-20T01:39:29.887 DEBUG:boto:Path: /aktlji-85/obj6 2024-04-20T01:39:29.887 DEBUG:boto:Data: 2024-04-20T01:39:29.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:39:29.887 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:39:29.887 DEBUG:boto:Port: 8000 2024-04-20T01:39:29.887 DEBUG:boto:Params: {} 2024-04-20T01:39:29.887 DEBUG:boto:Token: None 2024-04-20T01:39:29.887 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:39:29 GMT /aktlji-85/obj6 2024-04-20T01:39:29.888 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mJ8Mwq4tdqMTI0MrQE40kl+yqmE= 2024-04-20T01:39:29.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mJ8Mwq4tdqMTI0MrQE40kl+yqmE='} 2024-04-20T01:39:29.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1d9af974dcb1281-0066231cd1-230b-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:39:29.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:29.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:29.931 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:30.003 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:30.003 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:30.084 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:39:30.120 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:39:30.120 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:30.120 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:39:30.120 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:39:30.121 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:30.121 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:30.121 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:39:30.121 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:39:30.121 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:30.121 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:30.121 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577169.777139_1578.1", 2024-04-20T01:39:30.121 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:39:29.777139Z" 2024-04-20T01:39:30.121 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:39:30.121 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:39:30.121 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:39:30.121 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:39:30.122 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:39:30.122 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:39:30.123 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:30.123 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:39:30.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:30.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:30.123 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:30.197 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:30.197 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:30.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:30.279+0000 7f9680690780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:30.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:30.280+0000 7f9680690780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:30.317 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:39:30.317 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:39:30.317 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:39:30.317 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:39:30.317 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:39:30.317 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:39:30.318 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:30.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:30.318 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:39:30.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:30.318 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:39:30.318 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:30.318 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:30.318 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:39:30.318 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:30.318 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:39:30.319 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:30.319 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:39:30.319 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:30.319 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:30.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:30.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:30.319 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:39:30.319 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:30.319 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:30.319 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:39:30.320 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:30.320 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:30.320 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:30.320 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:39:30.320 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:30.320 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:30.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:30.320 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:30.320 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:39:30.320 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:30.320 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:30.321 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:39:30.321 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:30.321 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:30.321 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:30.321 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:39:30.321 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:30.321 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:30.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:30.321 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:30.321 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:39:30.321 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:30.321 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:30.322 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:39:30.322 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:30.322 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:30.322 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:30.322 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:39:30.322 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:30.322 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:30.322 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:30.322 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:39:30.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:30.323 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:39:30.323 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:39:30.323 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:39:30.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:30.323 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:39:30Z" 2024-04-20T01:39:30.323 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:39:30.324 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:39:30Z" } 2024-04-20T01:39:30.324 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:39:30.324 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:30.324 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:30.325 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:39:35.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:35.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:35.326 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:35.404 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:35.404 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:35.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:35.457+0000 7fadbc265780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:35.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:35.458+0000 7fadbc265780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:35.464 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:39:35.465 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:39:35.465 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:39:35.465 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:39:35.465 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:39:35.465 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:39:35.465 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:35.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:35.465 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:39:35.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:35.465 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:39:35.466 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:35.466 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:35.466 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:39:35.466 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:35.466 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:39:35.466 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:35.466 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:39:35.467 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:35.467 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:35.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:35.467 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:35.467 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:39:35.467 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:35.467 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:35.467 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:39:35.467 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:35.467 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:35.468 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:35.468 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:39:35.468 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:35.468 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:35.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:35.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:35.468 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:39:35.468 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:35.468 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:35.468 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:39:35.468 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:35.468 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:35.469 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:35.469 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:39:35.469 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:35.469 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:35.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:35.469 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:35.469 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:39:35.469 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:35.469 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:35.470 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:39:35.470 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:35.470 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:35.470 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:35.470 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:39:35.470 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:35.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:35.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:35.470 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:39:35.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:35.471 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:39:35.471 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:39:35.471 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:39:35.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:35.471 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:39:35Z" 2024-04-20T01:39:35.471 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:39:35.472 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:39:35Z" } 2024-04-20T01:39:35.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:39:35.472 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:35.472 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:35.472 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:39:40.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:40.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:40.474 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:40.534 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:40.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:40.594+0000 7f26b03dd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:40.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:40.596+0000 7f26b03dd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:40.602 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:39:40.602 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:39:40.602 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:39:40.602 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:39:40.602 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:39:40.602 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:39:40.602 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:40.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:40.602 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:39:40.602 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:40.603 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:39:40.603 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:40.603 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:40.603 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:39:40.603 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:40.603 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:39:40.603 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:40.603 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:39:40.603 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:40.603 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:40.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:40.604 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:40.604 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:39:40.604 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:40.604 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:40.604 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:39:40.604 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:40.604 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:40.604 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:40.604 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:39:40.604 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:40.604 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:40.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:40.605 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:40.605 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:39:40.605 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:40.605 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:40.605 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:39:40.605 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:40.605 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:40.605 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:40.605 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:39:40.605 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:40.606 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:40.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:40.606 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:40.606 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:39:40.606 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:40.606 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:40.606 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:39:40.606 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:40.606 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:40.606 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:40.607 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:39:40.607 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:40.607 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:40.607 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:40.607 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:39:40.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:40.607 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:39:40.607 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:39:40.607 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:39:40.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:40.607 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:39:40Z" 2024-04-20T01:39:40.607 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:39:40.608 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:39:40Z" } 2024-04-20T01:39:40.609 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:39:40.609 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:40.609 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:40.609 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:39:45.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:45.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:45.611 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:45.671 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:45.671 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:45.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:45.737+0000 7fc4d4c06780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:45.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:45.738+0000 7fc4d4c06780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:45.747 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:39:45.748 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:39:45.748 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:39:45.748 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:39:45.748 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:39:45.748 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:39:45.748 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:45.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:45.748 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:39:45.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:45.748 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:39:45.748 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:45.749 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:45.749 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:39:45.749 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:45.749 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:39:45.749 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:45.749 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:39:45.749 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:45.749 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:45.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:45.749 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:45.750 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:39:45.750 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:45.750 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:45.750 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:39:45.750 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:45.750 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:45.750 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:45.750 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:39:45.750 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:45.750 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:45.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:45.751 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:45.751 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:39:45.751 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:45.751 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:45.751 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:39:45.751 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:45.751 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:45.751 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:45.751 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:39:45.751 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:45.751 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:39:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:39:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:39:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:39:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:39:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:39:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:39:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:39:45Z" 2024-04-20T01:39:45.753 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:39:45.754 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:39:45Z" } 2024-04-20T01:39:45.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:39:45.755 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:45.755 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:45.755 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:39:50.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:50.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:50.757 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:50.818 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:50.818 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:50.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:50.873+0000 7fde2af5d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:50.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:50.874+0000 7fde2af5d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:50.880 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:39:50.880 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:39:50.880 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:39:50.880 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:39:50.880 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:39:50.880 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:39:50.880 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:50.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:50.881 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:39:50.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:50.881 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:39:50.882 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:50.882 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:50.882 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:39:50.882 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:50.882 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:39:50.882 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:50.882 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:39:50.882 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:50.883 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:50.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:50.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:50.883 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:39:50.883 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:50.883 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:50.883 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:39:50.883 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:50.883 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:50.883 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:50.883 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:39:50.883 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:50.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:50.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:50.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:50.884 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:39:50.884 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:50.884 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:50.884 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:39:50.884 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:50.884 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:50.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:50.884 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:39:50.885 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:50.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:50.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:50.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:50.885 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:39:50.885 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:50.885 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:50.885 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:39:50.885 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:50.885 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:50.885 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:50.885 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:39:50.886 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:50.886 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:50.886 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:50.886 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:39:50.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:50.886 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:39:50.886 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:39:50.886 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:39:50.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:50.886 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:39:50Z" 2024-04-20T01:39:50.886 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:39:50.887 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:39:50Z" } 2024-04-20T01:39:50.888 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:39:50.888 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:50.888 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:50.888 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:39:55.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:55.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:39:55.890 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:39:55.949 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:39:55.949 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:39:55.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:56.011+0000 7fd74586e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:56.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:39:56.012+0000 7fd74586e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:39:56.023 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:39:56.023 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:39:56.023 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:39:56.023 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:39:56.023 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:39:56.023 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:39:56.023 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:56.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:56.024 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:39:56.024 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:56.024 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:39:56.024 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:56.024 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:56.024 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:39:56.024 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:56.024 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:39:56.024 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:56.024 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:39:56.025 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:56.025 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:56.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:56.025 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:56.025 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:39:56.025 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:56.025 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:56.025 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:39:56.025 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:56.025 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:56.025 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:56.026 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:39:56.026 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:56.026 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:56.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:56.026 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:56.026 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:39:56.026 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:56.026 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:56.026 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:39:56.026 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:56.026 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:56.027 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:56.027 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:39:56.027 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:56.027 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:56.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:56.027 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:39:56.027 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:39:56.027 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:39:56.027 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:39:56.027 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:39:56.027 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:39:56.028 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:39:56.028 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:39:56.028 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:39:56.028 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:39:56.028 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:56.028 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:39:56.028 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:39:56.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:56.028 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:39:56.028 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:39:56.028 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:39:56.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:39:56.029 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:39:56Z" 2024-04-20T01:39:56.029 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:39:56.030 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:39:56Z" } 2024-04-20T01:39:56.030 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:39:56.030 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:56.030 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:39:56.030 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:40:01.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:01.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:01.032 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:40:01.093 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:40:01.093 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:40:01.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:01.146+0000 7f41ddab2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:01.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:01.147+0000 7f41ddab2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:01.155 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:40:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:40:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:40:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:40:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:40:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:40:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:01.155 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:40:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:40:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:40:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:40:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:40:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:01.156 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:40:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:40:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:40:01.157 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:40:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:40:01.158 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:40:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:40:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:40:01.159 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:40:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:40:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:40:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:40:01.160 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:40:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:40:01Z" 2024-04-20T01:40:01.161 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:40:01.162 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:40:01Z" } 2024-04-20T01:40:01.162 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:40:01.162 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:01.162 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:01.162 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:40:06.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:06.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:06.166 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:40:06.225 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:40:06.226 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:40:06.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:06.293+0000 7f1e99c5e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:06.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:06.294+0000 7f1e99c5e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:06.301 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:40:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:40:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:40:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:40:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:40:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:40:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:40:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:40:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:40:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:40:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:40:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:40:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:40:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:40:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:40:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:40:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:40:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:40:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:40:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:40:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:40:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:40:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:40:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:40:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:40:06Z" 2024-04-20T01:40:06.307 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:40:06.308 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:40:06Z" } 2024-04-20T01:40:06.308 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:40:06.308 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:06.308 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:06.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:40:11.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:11.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:11.310 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:40:11.370 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:40:11.370 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:40:11.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:11.438+0000 7f1d560dc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:11.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:11.439+0000 7f1d560dc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:11.448 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:40:11.448 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:40:11.448 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:40:11.448 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:40:11.448 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:40:11.448 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:40:11.448 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:11.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:11.448 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:40:11.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:11.449 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:40:11.449 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:11.449 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:11.449 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:40:11.449 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:11.449 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:40:11.449 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:11.449 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:40:11.449 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:11.450 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:11.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:11.450 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:11.450 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:40:11.450 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:11.450 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:11.450 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:40:11.450 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:11.450 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:11.450 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:11.450 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:40:11.451 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:11.451 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:11.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:11.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:11.451 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:40:11.451 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:11.451 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:11.451 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:40:11.451 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:11.451 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:11.451 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:11.452 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:40:11.452 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:11.453 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:11.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:11.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:11.453 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:40:11.454 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:11.454 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:11.454 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:40:11.454 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:11.454 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:11.454 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:11.454 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:40:11.454 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:11.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:11.456 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:11.456 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:40:11.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:11.456 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:40:11.456 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:40:11.456 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:40:11.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:11.456 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:40:11Z" 2024-04-20T01:40:11.456 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:40:11.457 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:40:11Z" } 2024-04-20T01:40:11.457 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:40:11.457 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:11.458 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:11.458 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:40:16.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:16.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:16.459 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:40:16.519 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:40:16.519 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:40:16.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:16.580+0000 7f7b55fc8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:16.582+0000 7f7b55fc8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:16.590 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:40:16.590 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:40:16.590 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:40:16.590 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:40:16.591 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:40:16.591 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:40:16.591 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:16.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:16.591 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:40:16.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:16.591 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:40:16.591 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:16.591 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:16.591 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:40:16.591 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:16.592 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:40:16.592 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:16.592 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:40:16.592 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:16.592 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:16.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:16.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:16.592 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:40:16.592 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:16.592 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:16.592 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:40:16.592 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:16.593 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:16.593 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:16.593 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:40:16.593 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:16.593 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:16.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:16.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:16.593 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:40:16.593 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:16.593 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:16.593 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:40:16.594 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:16.594 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:16.594 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:16.594 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:40:16.594 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:16.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:16.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:16.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:16.594 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:40:16.594 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:16.594 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:16.595 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:40:16.595 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:16.595 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:16.595 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:16.595 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:40:16.595 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:16.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:16.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:16.595 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:40:16.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:16.595 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:40:16.596 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:40:16.596 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:40:16.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:16.596 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:40:16Z" 2024-04-20T01:40:16.596 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:40:16.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:40:16Z" } 2024-04-20T01:40:16.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:40:16.597 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:16.597 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:16.597 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:40:21.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:21.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:21.599 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:40:21.657 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:40:21.657 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:40:21.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:21.716+0000 7f07ec19e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:21.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:21.717+0000 7f07ec19e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:21.731 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:40:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:40:21.731 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:40:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:40:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:40:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:40:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:21.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:40:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:40:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:21.733 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:40:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:40:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:21.734 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:40:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:21.735 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:40:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:40:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:21.736 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:40:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:21.737 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:40:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:40:21.738 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:40:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:21.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:40:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:21.740 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:40:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:21.741 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:40:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:40:21.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:40:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:40:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:40:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:21.743 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:40:21Z" 2024-04-20T01:40:21.744 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:40:21.745 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:40:21Z" } 2024-04-20T01:40:21.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:40:21.746 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:21.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:21.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:40:26.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:26.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:26.748 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:40:26.807 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:40:26.807 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:40:26.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:26.879+0000 7f537c1cf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:26.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:26.880+0000 7f537c1cf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:26.887 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:40:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:40:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:40:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:40:26.887 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:40:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:40:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:40:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:40:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:40:26.888 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:40:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:40:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:40:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:40:26.889 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:40:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:40:26.890 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:40:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:40:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:26.891 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:40:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:40:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:40:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:26.892 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:40:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:40:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:40:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:40:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:26.893 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:40:26Z" 2024-04-20T01:40:26.893 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:40:26.894 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:40:26Z" } 2024-04-20T01:40:26.894 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:40:26.894 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:26.895 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:26.895 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:40:31.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:31.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:31.896 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:40:31.957 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:40:31.957 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:40:32.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:32.025+0000 7f80226d9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:32.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:32.026+0000 7f80226d9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:32.036 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:40:32.036 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:40:32.036 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:40:32.036 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:40:32.036 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:40:32.036 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:40:32.037 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:32.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:32.037 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:40:32.037 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:32.037 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:40:32.037 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:32.037 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:32.037 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:40:32.037 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:32.037 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:40:32.037 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:32.037 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:40:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:40:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:40:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:40:32.040 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:32.040 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:32.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:32.040 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:32.040 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:40:32.040 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:32.040 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:32.040 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:40:32.040 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:32.040 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:32.040 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:32.040 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:40:32.041 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:32.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:32.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:32.041 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:40:32.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:32.041 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:40:32.041 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:40:32.041 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:40:32.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:32.041 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:40:32Z" 2024-04-20T01:40:32.041 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:40:32.043 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:40:32Z" } 2024-04-20T01:40:32.043 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:40:32.043 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:32.043 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:32.043 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:40:37.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:37.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:37.045 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:40:37.108 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:40:37.108 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:40:37.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:37.173+0000 7f6d59dde780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:37.174+0000 7f6d59dde780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:37.180 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:40:37.180 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:40:37.180 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:40:37.180 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:40:37.180 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:40:37.180 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:40:37.180 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:37.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:37.180 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:40:37.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:37.181 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:40:37.181 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:37.181 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:37.181 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:40:37.181 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:37.181 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:40:37.181 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:37.181 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:40:37.181 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:37.181 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:37.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:37.182 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:37.182 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:40:37.182 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:37.182 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:37.182 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:40:37.182 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:37.182 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:37.182 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:37.182 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:40:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:40:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:40:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:37.183 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:40:37.184 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:37.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:37.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:37.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:37.184 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:40:37.184 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:37.184 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:37.184 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:40:37.184 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:37.184 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:37.185 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:37.185 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:40:37.185 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:37.185 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:37.185 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:37.185 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:40:37.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:37.185 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:40:37.185 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:40:37.185 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:40:37.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:37.186 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:40:37Z" 2024-04-20T01:40:37.186 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:40:37.187 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:40:37Z" } 2024-04-20T01:40:37.187 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:40:37.187 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:37.187 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:37.187 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:40:42.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:42.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:42.189 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:40:42.248 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:40:42.248 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:40:42.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:42.303+0000 7f24bb7f2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:42.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:42.305+0000 7f24bb7f2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:42.310 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:40:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:40:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:40:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:40:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:40:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:40:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:40:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:40:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:40:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:40:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:40:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:40:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:40:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:40:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:40:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:40:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:40:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:40:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:40:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:40:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:40:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:40:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:40:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:40:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:42.316 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:40:42Z" 2024-04-20T01:40:42.316 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:40:42.317 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:40:42Z" } 2024-04-20T01:40:42.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:40:42.318 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:42.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:42.318 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:40:47.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:47.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:47.319 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:40:47.377 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:40:47.377 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:40:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:47.437+0000 7f7b966e1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:47.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:47.439+0000 7f7b966e1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:47.448 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:40:47.449 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:40:47.449 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:40:47.449 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:40:47.449 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:40:47.449 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:40:47.449 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:47.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:47.449 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:40:47.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:47.450 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:40:47.450 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:47.450 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:47.450 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:40:47.450 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:47.450 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:40:47.450 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:47.450 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:40:47.450 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:47.450 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:47.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:47.451 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:47.451 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:40:47.451 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:47.451 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:47.451 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:40:47.451 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:47.451 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:47.451 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:47.451 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:40:47.451 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:47.452 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:47.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:47.452 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:47.452 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:40:47.452 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:47.452 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:47.452 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:40:47.452 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:47.452 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:47.452 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:47.452 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:40:47.453 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:47.453 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:47.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:47.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:47.453 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:40:47.453 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:47.453 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:47.453 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:40:47.453 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:47.453 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:47.453 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:47.454 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:40:47.454 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:47.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:47.454 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:47.454 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:40:47.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:47.454 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:40:47.454 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:40:47.454 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:40:47.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:47.454 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:40:47Z" 2024-04-20T01:40:47.455 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:40:47.455 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:40:47Z" } 2024-04-20T01:40:47.456 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:40:47.456 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:47.456 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:47.456 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:40:52.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:52.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:52.458 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:40:52.517 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:40:52.517 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:40:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:52.575+0000 7f2e8ea8a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:52.577+0000 7f2e8ea8a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:52.586 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:40:52.586 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:40:52.586 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:40:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:40:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:40:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:40:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:40:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:40:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:40:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:40:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:40:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:40:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:40:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:40:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:40:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:40:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:40:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:40:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:52.592 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:40:52.592 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:40:52.592 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:40:52.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:52.592 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:40:52Z" 2024-04-20T01:40:52.592 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:40:52.593 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:40:52Z" } 2024-04-20T01:40:52.593 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:40:52.593 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:52.593 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:52.593 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:40:57.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:57.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:40:57.595 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:40:57.656 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:40:57.656 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:40:57.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:57.722+0000 7ff6e21dc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:57.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:40:57.723+0000 7ff6e21dc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:40:57.737 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:40:57.737 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:40:57.737 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:40:57.737 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:40:57.738 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:40:57.738 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:40:57.738 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:57.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:57.738 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:40:57.738 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:57.738 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:40:57.738 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:57.738 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:57.739 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:40:57.739 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:57.739 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:40:57.739 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:57.739 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:40:57.739 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:57.739 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:57.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:57.739 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:57.739 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:40:57.740 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:57.740 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:57.740 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:40:57.740 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:57.740 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:57.740 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:57.740 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:40:57.740 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:57.740 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:57.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:57.740 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:57.741 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:40:57.741 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:57.741 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:57.741 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:40:57.741 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:57.741 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:57.741 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:57.741 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:40:57.741 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:57.741 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:57.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:57.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:40:57.742 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:40:57.742 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:40:57.742 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:40:57.742 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:40:57.742 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:40:57.742 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:40:57.742 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:40:57.742 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:40:57.743 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:40:57.743 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:57.743 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:40:57.743 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:40:57.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:57.743 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:40:57.743 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:40:57.743 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:40:57.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:40:57.743 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:40:57Z" 2024-04-20T01:40:57.744 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:40:57.744 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:40:57Z" } 2024-04-20T01:40:57.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:40:57.745 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:57.745 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:40:57.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:41:02.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:02.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:02.747 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:02.808 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:02.808 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:02.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:41:02.862+0000 7f40c878b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:02.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:41:02.863+0000 7f40c878b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:02.871 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:41:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:41:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:41:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:41:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:41:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:41:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:41:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:41:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:41:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:41:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:02Z" 2024-04-20T01:41:02.877 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:02.878 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577027.057559_1543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:37:07.057559Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:41:02Z" } 2024-04-20T01:41:02.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:41:02.878 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:02.878 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:02.879 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:41:07.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:07.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:07.880 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:07.939 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:07.940 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:07.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:41:08.005+0000 7fcd07992780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:07.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:41:08.006+0000 7fcd07992780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:08.012 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:08.012 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:41:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:08.013 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:41:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:41:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:08.014 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:41:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:41:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:08.015 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577169.777139_1578.1", 2024-04-20T01:41:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:39:29.777139Z", 2024-04-20T01:41:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:08.016 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:41:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:08.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:41:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:41:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:08Z" 2024-04-20T01:41:08.018 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:08.019 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577169.777139_1578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:39:29.777139Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:41:08Z" } 2024-04-20T01:41:08.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577169.777139_1578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:29.777139Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:41:08.019 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:08.020 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:08.020 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:41:08.020 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:41:08.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:08.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:08.020 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:08.077 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:08.077 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:08.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:41:08.134+0000 7fceb74b2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:08.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:41:08.135+0000 7fceb74b2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:08.140 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:41:08.140 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:41:08.140 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:41:08.140 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:41:08.140 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:41:08.140 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:41:08.140 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:08.141 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:08.141 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:41:08.141 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:08.141 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:41:08.141 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:08.141 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:08.141 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:08.141 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:08.141 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:41:08.141 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:08.141 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:41:08.141 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:08.142 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:08.142 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:08.142 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:08.142 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:41:08.142 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:08.142 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:08.142 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:41:08.142 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:08.142 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:41:08.142 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:08.142 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:41:08.143 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:08.143 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:08.143 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:08.143 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:08.143 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:41:08.143 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:08.143 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:08.143 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577169.777139_1578.1", 2024-04-20T01:41:08.143 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:08.143 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:41:08.143 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:08.143 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:39:29.777139Z", 2024-04-20T01:41:08.144 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:08.144 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:08.144 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:08.144 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:08.144 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:41:08.144 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:08.144 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:08.144 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:08.144 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:08.144 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:41:08.144 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:08.145 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:41:08.145 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:08.145 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:08.145 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:08.145 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:08.145 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:08.145 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:41:08.145 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:41:08.145 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:41:08.145 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:08.145 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:41:08Z" 2024-04-20T01:41:08.145 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:41:08.146 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577169.777139_1578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:39:29.777139Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:41:08Z" } 2024-04-20T01:41:08.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577169.777139_1578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:29.777139Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:41:08.147 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:08.147 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:08.147 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:41:08.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:08.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:08.147 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:08.199 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:08.199 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:08.273 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:41:08.274 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:08.274 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:41:08.274 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:41:08.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:08.274 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:08.274 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:41:08.274 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:41:08.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:08.275 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:08.275 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:41:08.275 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:41:08.275 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:08.275 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:08.275 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000020738", 2024-04-20T01:41:08.275 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:39:29.514702Z" 2024-04-20T01:41:08.275 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:08.275 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:41:08.276 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:41:08.276 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:41:08.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:08.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:08.277 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:08.356 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:08.356 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:08.420 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:08.420 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:08.420 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:08.420 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:08.420 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:08.420 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:41:08.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:08.421 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:08.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:08.421 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:08.421 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:08.421 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:08.421 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:41:08.421 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:08.421 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:41:08.421 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:08.421 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:41:08.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:08.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:08.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:08.422 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:08.422 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:08.422 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:08.422 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:41:08.422 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:08.422 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:41:08.422 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:08.422 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:41:08.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:08.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:08.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:08.423 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:08.423 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:08.423 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:08.423 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:41:08.423 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:08.423 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:08.423 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:08.423 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:41:08.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:08.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:08.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:08.424 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:08.424 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:08.424 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:08.424 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:41:08.424 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:08.424 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:08.424 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:08.424 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:41:08.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:08.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:08.425 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:08.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:08.425 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:08.425 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:41:08.425 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:41:08.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:08.425 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:08Z" 2024-04-20T01:41:08.425 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:08.426 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015063", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:38:37.851769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026490", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:38:37.842528Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020598", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:39:07.899475Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:41:08Z" } 2024-04-20T01:41:08.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015063', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.851769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026490', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.842528Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020598', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:07.899475Z'}}] 2024-04-20T01:41:08.426 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:41:08.426 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020598'} 2024-04-20T01:41:08.426 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015203 target=00000000000000000000:00000000000000015063, shard=1 master=00000000000000000000:00000000000000026910 target=00000000000000000000:00000000000000026490, shard=3 master=00000000000000000000:00000000000000020738 target=00000000000000000000:00000000000000020598 2024-04-20T01:41:13.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:13.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:13.428 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:13.488 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:13.489 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:13.574 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:13.574 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:41:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:41:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:41:13.575 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:41:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:41:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:13.576 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:41:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:41:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:41:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:13.577 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:41:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:41:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:13.578 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:41:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:41:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:41:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:13.579 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:13Z" 2024-04-20T01:41:13.579 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:13.580 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015063", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:38:37.851769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026490", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:38:37.842528Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020598", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:39:07.899475Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:41:13Z" } 2024-04-20T01:41:13.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015063', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.851769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026490', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.842528Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020598', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:07.899475Z'}}] 2024-04-20T01:41:13.581 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:41:13.581 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020598'} 2024-04-20T01:41:13.581 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015203 target=00000000000000000000:00000000000000015063, shard=1 master=00000000000000000000:00000000000000026910 target=00000000000000000000:00000000000000026490, shard=3 master=00000000000000000000:00000000000000020738 target=00000000000000000000:00000000000000020598 2024-04-20T01:41:18.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:18.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:18.583 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:18.643 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:18.643 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:18.705 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:18.705 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:18.705 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:18.705 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:18.705 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:18.705 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:41:18.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:18.705 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:18.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:18.706 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:18.706 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:18.706 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:18.706 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:41:18.706 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:18.706 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:41:18.706 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:18.706 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:41:18.706 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:18.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:18.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:18.707 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:18.707 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:18.707 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:18.707 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:41:18.707 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:18.707 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:41:18.707 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:18.707 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:41:18.707 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:18.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:18.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:18.708 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:18.708 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:18.708 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:18.708 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:41:18.708 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:18.708 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:18.708 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:18.708 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:41:18.708 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:18.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:18.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:18.709 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:18.709 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:18.709 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:18.709 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:41:18.709 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:18.709 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:18.709 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:18.709 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:41:18.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:18.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:18.709 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:18.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:18.710 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:18.710 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:41:18.710 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:41:18.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:18.710 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:18Z" 2024-04-20T01:41:18.710 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:18.711 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015063", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:38:37.851769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026490", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:38:37.842528Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020598", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:39:07.899475Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:41:18Z" } 2024-04-20T01:41:18.711 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015063', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.851769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026490', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.842528Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020598', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:07.899475Z'}}] 2024-04-20T01:41:18.712 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:41:18.712 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020598'} 2024-04-20T01:41:18.712 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015203 target=00000000000000000000:00000000000000015063, shard=1 master=00000000000000000000:00000000000000026910 target=00000000000000000000:00000000000000026490, shard=3 master=00000000000000000000:00000000000000020738 target=00000000000000000000:00000000000000020598 2024-04-20T01:41:23.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:23.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:23.714 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:23.774 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:23.774 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:23.865 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:41:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:41:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:41:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:41:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:41:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:41:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:41:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:41:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:41:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:41:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:41:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:41:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:41:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:23Z" 2024-04-20T01:41:23.870 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:23.871 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015063", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:38:37.851769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026490", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:38:37.842528Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020598", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:39:07.899475Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:41:23Z" } 2024-04-20T01:41:23.871 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015063', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.851769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026490', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.842528Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020598', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:07.899475Z'}}] 2024-04-20T01:41:23.871 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:41:23.872 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020598'} 2024-04-20T01:41:23.872 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015203 target=00000000000000000000:00000000000000015063, shard=1 master=00000000000000000000:00000000000000026910 target=00000000000000000000:00000000000000026490, shard=3 master=00000000000000000000:00000000000000020738 target=00000000000000000000:00000000000000020598 2024-04-20T01:41:28.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:28.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:28.874 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:28.936 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:28.936 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:29.006 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:29.006 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:29.006 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:29.006 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:29.006 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:29.006 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:41:29.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:29.007 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:29.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:29.007 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:29.007 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:29.007 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:29.007 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:41:29.007 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:29.007 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:41:29.007 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:29.007 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:41:29.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:29.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:29.008 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:29.008 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:29.008 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:29.008 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:29.008 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:41:29.008 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:29.008 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:41:29.008 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:29.008 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:41:29.009 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:29.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:29.009 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:29.009 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:29.009 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:29.009 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:29.009 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:41:29.009 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:29.009 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:29.009 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:29.009 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:41:29.009 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:29.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:29.010 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:29.010 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:29.010 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:29.010 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:29.010 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:41:29.010 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:29.010 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:29.010 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:29.010 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:41:29.010 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:29.011 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:29.011 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:29.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:29.011 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:29.011 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:41:29.011 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:41:29.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:29.011 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:29Z" 2024-04-20T01:41:29.011 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:29.012 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015063", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:38:37.851769Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026490", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:38:37.842528Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020598", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:39:07.899475Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:41:29Z" } 2024-04-20T01:41:29.013 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015063', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.851769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026490', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.842528Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020598', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:07.899475Z'}}] 2024-04-20T01:41:29.013 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:41:29.013 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020598'} 2024-04-20T01:41:29.013 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015203 target=00000000000000000000:00000000000000015063, shard=1 master=00000000000000000000:00000000000000026910 target=00000000000000000000:00000000000000026490, shard=3 master=00000000000000000000:00000000000000020738 target=00000000000000000000:00000000000000020598 2024-04-20T01:41:34.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:34.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:34.015 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:34.082 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:34.082 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:34.162 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:34.162 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:34.162 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:34.162 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:34.162 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:34.162 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:41:34.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:34.162 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:34.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:34.163 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:34.163 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:34.163 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:34.163 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:41:34.163 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:34.163 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:41:34.163 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:34.163 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:41:34.163 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:34.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:34.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:34.164 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:34.164 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:34.164 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:34.164 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:41:34.164 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:34.164 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:41:34.164 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:34.164 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:41:34.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:34.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:41:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:41:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:34.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:41:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:41:34.166 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:34.167 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:34.167 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:34.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:34.167 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:34.167 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:41:34.167 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:41:34.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:34.167 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:34Z" 2024-04-20T01:41:34.167 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:34.168 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:41:34Z" } 2024-04-20T01:41:34.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:41:34.169 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:41:34.169 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:41:34.169 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:41:34.169 DEBUG:boto:path=/aktlji-84/ 2024-04-20T01:41:34.169 DEBUG:boto:auth_path=/aktlji-84/ 2024-04-20T01:41:34.169 DEBUG:boto:Method: HEAD 2024-04-20T01:41:34.169 DEBUG:boto:Path: /aktlji-84/ 2024-04-20T01:41:34.170 DEBUG:boto:Data: 2024-04-20T01:41:34.170 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.170 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:41:34.170 DEBUG:boto:Port: 8000 2024-04-20T01:41:34.170 DEBUG:boto:Params: {} 2024-04-20T01:41:34.170 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:41:34.171 DEBUG:boto:Token: None 2024-04-20T01:41:34.171 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:34 GMT /aktlji-84/ 2024-04-20T01:41:34.171 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QPaT0up6+kbSILGegmJVSkTDTvU= 2024-04-20T01:41:34.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QPaT0up6+kbSILGegmJVSkTDTvU=', 'Content-Length': '0'} 2024-04-20T01:41:34.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007da55f99f1bb39a3-0066231d4e-230b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.176 DEBUG:boto:path=/aktlji-84/obj1 2024-04-20T01:41:34.176 DEBUG:boto:auth_path=/aktlji-84/obj1 2024-04-20T01:41:34.176 DEBUG:boto:Method: HEAD 2024-04-20T01:41:34.176 DEBUG:boto:Path: /aktlji-84/obj1 2024-04-20T01:41:34.176 DEBUG:boto:Data: 2024-04-20T01:41:34.177 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.177 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:41:34.177 DEBUG:boto:Port: 8000 2024-04-20T01:41:34.177 DEBUG:boto:Params: {} 2024-04-20T01:41:34.177 DEBUG:boto:Token: None 2024-04-20T01:41:34.177 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:34 GMT /aktlji-84/obj1 2024-04-20T01:41:34.177 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:poQpk1qw+5z95rLoZlO+Tu+adX8= 2024-04-20T01:41:34.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:poQpk1qw+5z95rLoZlO+Tu+adX8=', 'Content-Length': '0'} 2024-04-20T01:41:34.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-84:73f46f02-c7ab-4426-a267-2273af911da5.6914.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d291140d1b59c20-0066231d4e-230b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.179 DEBUG:boto:path=/aktlji-84/obj1 2024-04-20T01:41:34.179 DEBUG:boto:auth_path=/aktlji-84/obj1 2024-04-20T01:41:34.180 DEBUG:boto:Method: GET 2024-04-20T01:41:34.180 DEBUG:boto:Path: /aktlji-84/obj1 2024-04-20T01:41:34.180 DEBUG:boto:Data: 2024-04-20T01:41:34.180 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.180 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:41:34.180 DEBUG:boto:Port: 8000 2024-04-20T01:41:34.180 DEBUG:boto:Params: {} 2024-04-20T01:41:34.180 DEBUG:boto:Token: None 2024-04-20T01:41:34.180 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:41:34 GMT /aktlji-84/obj1 2024-04-20T01:41:34.180 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Dc/wGOb5x9otFFEpQVEPEvCa9QM= 2024-04-20T01:41:34.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Dc/wGOb5x9otFFEpQVEPEvCa9QM=', 'Content-Length': '0'} 2024-04-20T01:41:34.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-84:73f46f02-c7ab-4426-a267-2273af911da5.6914.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047064a1abfbe4a9b-0066231d4e-230b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.183 DEBUG:boto:path=/aktlji-84/obj2 2024-04-20T01:41:34.183 DEBUG:boto:auth_path=/aktlji-84/obj2 2024-04-20T01:41:34.183 DEBUG:boto:Method: HEAD 2024-04-20T01:41:34.183 DEBUG:boto:Path: /aktlji-84/obj2 2024-04-20T01:41:34.183 DEBUG:boto:Data: 2024-04-20T01:41:34.183 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.183 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:41:34.183 DEBUG:boto:Port: 8000 2024-04-20T01:41:34.183 DEBUG:boto:Params: {} 2024-04-20T01:41:34.184 DEBUG:boto:Token: None 2024-04-20T01:41:34.184 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:34 GMT /aktlji-84/obj2 2024-04-20T01:41:34.184 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bK8+vwThoLlSHLsQ1fgk+RVrx3Y= 2024-04-20T01:41:34.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bK8+vwThoLlSHLsQ1fgk+RVrx3Y=', 'Content-Length': '0'} 2024-04-20T01:41:34.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-84:73f46f02-c7ab-4426-a267-2273af911da5.6914.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e2e40b8247de1e9-0066231d4e-230b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.186 DEBUG:boto:path=/aktlji-84/obj2 2024-04-20T01:41:34.186 DEBUG:boto:auth_path=/aktlji-84/obj2 2024-04-20T01:41:34.186 DEBUG:boto:Method: GET 2024-04-20T01:41:34.186 DEBUG:boto:Path: /aktlji-84/obj2 2024-04-20T01:41:34.186 DEBUG:boto:Data: 2024-04-20T01:41:34.186 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.186 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:41:34.186 DEBUG:boto:Port: 8000 2024-04-20T01:41:34.187 DEBUG:boto:Params: {} 2024-04-20T01:41:34.187 DEBUG:boto:Token: None 2024-04-20T01:41:34.187 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:41:34 GMT /aktlji-84/obj2 2024-04-20T01:41:34.187 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:UqnSCJ48ids3GsErTKl+IOrHDrE= 2024-04-20T01:41:34.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UqnSCJ48ids3GsErTKl+IOrHDrE=', 'Content-Length': '0'} 2024-04-20T01:41:34.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-84:73f46f02-c7ab-4426-a267-2273af911da5.6914.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec54cf3d3693e32e-0066231d4e-230b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.230 DEBUG:boto:path=/aktlji-84/obj3 2024-04-20T01:41:34.231 DEBUG:boto:auth_path=/aktlji-84/obj3 2024-04-20T01:41:34.231 DEBUG:boto:Method: HEAD 2024-04-20T01:41:34.231 DEBUG:boto:Path: /aktlji-84/obj3 2024-04-20T01:41:34.231 DEBUG:boto:Data: 2024-04-20T01:41:34.231 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.231 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:41:34.231 DEBUG:boto:Port: 8000 2024-04-20T01:41:34.231 DEBUG:boto:Params: {} 2024-04-20T01:41:34.231 DEBUG:boto:Token: None 2024-04-20T01:41:34.232 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:34 GMT /aktlji-84/obj3 2024-04-20T01:41:34.232 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BqVQT/4X0Xki93F45B9F3o7f9/k= 2024-04-20T01:41:34.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BqVQT/4X0Xki93F45B9F3o7f9/k=', 'Content-Length': '0'} 2024-04-20T01:41:34.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-84:73f46f02-c7ab-4426-a267-2273af911da5.6914.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fc1a25ea40f9ca0-0066231d4e-230b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.234 DEBUG:boto:path=/aktlji-84/obj3 2024-04-20T01:41:34.234 DEBUG:boto:auth_path=/aktlji-84/obj3 2024-04-20T01:41:34.234 DEBUG:boto:Method: GET 2024-04-20T01:41:34.234 DEBUG:boto:Path: /aktlji-84/obj3 2024-04-20T01:41:34.235 DEBUG:boto:Data: 2024-04-20T01:41:34.235 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.235 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:41:34.235 DEBUG:boto:Port: 8000 2024-04-20T01:41:34.235 DEBUG:boto:Params: {} 2024-04-20T01:41:34.235 DEBUG:boto:Token: None 2024-04-20T01:41:34.235 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:41:34 GMT /aktlji-84/obj3 2024-04-20T01:41:34.235 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:PBbi+VHM1CVk3qlLWVKH/KT5ECs= 2024-04-20T01:41:34.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PBbi+VHM1CVk3qlLWVKH/KT5ECs=', 'Content-Length': '0'} 2024-04-20T01:41:34.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-84:73f46f02-c7ab-4426-a267-2273af911da5.6914.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e1b8bd06c83d4f4-0066231d4e-230b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.237 DEBUG:boto:path=/aktlji-84/ 2024-04-20T01:41:34.237 DEBUG:boto:auth_path=/aktlji-84/ 2024-04-20T01:41:34.238 DEBUG:boto:Method: HEAD 2024-04-20T01:41:34.238 DEBUG:boto:Path: /aktlji-84/ 2024-04-20T01:41:34.238 DEBUG:boto:Data: 2024-04-20T01:41:34.238 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.238 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:41:34.238 DEBUG:boto:Port: 8001 2024-04-20T01:41:34.238 DEBUG:boto:Params: {} 2024-04-20T01:41:34.238 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T01:41:34.239 DEBUG:boto:Token: None 2024-04-20T01:41:34.239 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:34 GMT /aktlji-84/ 2024-04-20T01:41:34.239 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:QPaT0up6+kbSILGegmJVSkTDTvU= 2024-04-20T01:41:34.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QPaT0up6+kbSILGegmJVSkTDTvU=', 'Content-Length': '0'} 2024-04-20T01:41:34.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099d2466410cfee3a-0066231d4e-1b06-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.245 DEBUG:boto:path=/aktlji-84/obj1 2024-04-20T01:41:34.245 DEBUG:boto:auth_path=/aktlji-84/obj1 2024-04-20T01:41:34.245 DEBUG:boto:Method: HEAD 2024-04-20T01:41:34.245 DEBUG:boto:Path: /aktlji-84/obj1 2024-04-20T01:41:34.245 DEBUG:boto:Data: 2024-04-20T01:41:34.245 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.245 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:41:34.246 DEBUG:boto:Port: 8001 2024-04-20T01:41:34.246 DEBUG:boto:Params: {} 2024-04-20T01:41:34.246 DEBUG:boto:Token: None 2024-04-20T01:41:34.246 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:34 GMT /aktlji-84/obj1 2024-04-20T01:41:34.246 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:poQpk1qw+5z95rLoZlO+Tu+adX8= 2024-04-20T01:41:34.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:poQpk1qw+5z95rLoZlO+Tu+adX8=', 'Content-Length': '0'} 2024-04-20T01:41:34.248 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000008a905d5fed85776c-0066231d4e-1b06-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.248 DEBUG:boto:path=/aktlji-84/obj2 2024-04-20T01:41:34.248 DEBUG:boto:auth_path=/aktlji-84/obj2 2024-04-20T01:41:34.248 DEBUG:boto:Method: HEAD 2024-04-20T01:41:34.248 DEBUG:boto:Path: /aktlji-84/obj2 2024-04-20T01:41:34.248 DEBUG:boto:Data: 2024-04-20T01:41:34.248 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.248 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:41:34.249 DEBUG:boto:Port: 8001 2024-04-20T01:41:34.249 DEBUG:boto:Params: {} 2024-04-20T01:41:34.249 DEBUG:boto:Token: None 2024-04-20T01:41:34.249 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:34 GMT /aktlji-84/obj2 2024-04-20T01:41:34.249 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:bK8+vwThoLlSHLsQ1fgk+RVrx3Y= 2024-04-20T01:41:34.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bK8+vwThoLlSHLsQ1fgk+RVrx3Y=', 'Content-Length': '0'} 2024-04-20T01:41:34.251 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000031caa2a795cf7149-0066231d4e-1b06-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.251 DEBUG:boto:path=/aktlji-84/obj3 2024-04-20T01:41:34.251 DEBUG:boto:auth_path=/aktlji-84/obj3 2024-04-20T01:41:34.251 DEBUG:boto:Method: HEAD 2024-04-20T01:41:34.251 DEBUG:boto:Path: /aktlji-84/obj3 2024-04-20T01:41:34.251 DEBUG:boto:Data: 2024-04-20T01:41:34.251 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.251 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:41:34.252 DEBUG:boto:Port: 8001 2024-04-20T01:41:34.252 DEBUG:boto:Params: {} 2024-04-20T01:41:34.252 DEBUG:boto:Token: None 2024-04-20T01:41:34.252 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:34 GMT /aktlji-84/obj3 2024-04-20T01:41:34.252 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BqVQT/4X0Xki93F45B9F3o7f9/k= 2024-04-20T01:41:34.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BqVQT/4X0Xki93F45B9F3o7f9/k=', 'Content-Length': '0'} 2024-04-20T01:41:34.254 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000021f2526c5039bd55-0066231d4e-1b06-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.254 DEBUG:boto:path=/aktlji-85/ 2024-04-20T01:41:34.254 DEBUG:boto:auth_path=/aktlji-85/ 2024-04-20T01:41:34.254 DEBUG:boto:Method: HEAD 2024-04-20T01:41:34.254 DEBUG:boto:Path: /aktlji-85/ 2024-04-20T01:41:34.254 DEBUG:boto:Data: 2024-04-20T01:41:34.254 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.254 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:41:34.254 DEBUG:boto:Port: 8000 2024-04-20T01:41:34.254 DEBUG:boto:Params: {} 2024-04-20T01:41:34.255 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:41:34.255 DEBUG:boto:Token: None 2024-04-20T01:41:34.255 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:34 GMT /aktlji-85/ 2024-04-20T01:41:34.255 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6hVfn4Sz82C8PlLrMowU9W0g2sw= 2024-04-20T01:41:34.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6hVfn4Sz82C8PlLrMowU9W0g2sw=', 'Content-Length': '0'} 2024-04-20T01:41:34.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072ad049e2f61ed8a-0066231d4e-1afa-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.259 DEBUG:boto:path=/aktlji-85/obj4 2024-04-20T01:41:34.259 DEBUG:boto:auth_path=/aktlji-85/obj4 2024-04-20T01:41:34.259 DEBUG:boto:Method: HEAD 2024-04-20T01:41:34.259 DEBUG:boto:Path: /aktlji-85/obj4 2024-04-20T01:41:34.260 DEBUG:boto:Data: 2024-04-20T01:41:34.260 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.260 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:41:34.260 DEBUG:boto:Port: 8000 2024-04-20T01:41:34.260 DEBUG:boto:Params: {} 2024-04-20T01:41:34.260 DEBUG:boto:Token: None 2024-04-20T01:41:34.260 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:34 GMT /aktlji-85/obj4 2024-04-20T01:41:34.260 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:GSrekYlei2FJN/yI+5MPjvjd1ec= 2024-04-20T01:41:34.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GSrekYlei2FJN/yI+5MPjvjd1ec=', 'Content-Length': '0'} 2024-04-20T01:41:34.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-85:73f46f02-c7ab-4426-a267-2273af911da5.6914.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041d507968a777f20-0066231d4e-1afa-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.265 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... FAIL 2024-04-20T01:41:34.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:34.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:34.266 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:34.325 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:34.325 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:34.375 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:41:34.392 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:41:34.392 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:34.392 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:34.392 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:41:34.392 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:34.393 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:34.393 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:41:34.393 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:41:34.393 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:34.393 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:34.393 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577169.777139_1578.1", 2024-04-20T01:41:34.393 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:39:29.777139Z" 2024-04-20T01:41:34.393 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:34.393 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:34.393 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:34.394 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:41:34.394 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:34.394 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:41:34.395 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:34.395 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:41:34.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:34.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:34.395 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:34.454 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:34.454 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:34.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:41:34.512+0000 7fe2a5648780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:34.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:41:34.514+0000 7fe2a5648780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:34.520 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:34.520 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:41:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:41:34.521 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:41:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:41:34.522 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577169.777139_1578.1", 2024-04-20T01:41:34.523 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:39:29.777139Z", 2024-04-20T01:41:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:34.524 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:41:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:34.525 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:41:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:41:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:34.526 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:34Z" 2024-04-20T01:41:34.526 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:34.527 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577169.777139_1578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:39:29.777139Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:41:34Z" } 2024-04-20T01:41:34.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577169.777139_1578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:29.777139Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:41:34.527 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:34.527 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:34.527 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:41:34.527 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:41:34.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:34.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:34.529 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:34.588 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:34.588 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:34.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:41:34.638+0000 7fb6e970b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:34.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:41:34.639+0000 7fb6e970b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:34.654 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:41:34.654 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:41:34.654 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:41:34.654 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:41:34.654 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:41:34.654 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:41:34.655 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:34.655 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:34.655 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:41:34.655 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:34.655 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:41:34.655 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:34.655 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:34.655 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:34.655 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:34.655 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:41:34.655 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:34.655 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:41:34.656 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:34.656 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:34.656 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:34.656 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:34.656 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:41:34.656 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:34.656 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:34.656 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:41:34.656 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:34.656 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:41:34.656 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:34.657 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:41:34.657 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:34.657 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:34.657 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:34.657 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:34.657 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:41:34.657 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:34.657 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:34.657 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577169.777139_1578.1", 2024-04-20T01:41:34.657 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:34.657 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:41:34.658 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:34.658 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:39:29.777139Z", 2024-04-20T01:41:34.658 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:34.658 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:34.658 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:34.658 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:34.658 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:41:34.658 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:34.658 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:34.658 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:34.658 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:34.659 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:41:34.659 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:34.659 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:41:34.659 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:34.659 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:34.659 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:34.659 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:34.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:34.659 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:41:34.659 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:41:34.659 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:41:34.659 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:34.660 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:41:34Z" 2024-04-20T01:41:34.660 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:41:34.660 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577169.777139_1578.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:39:29.777139Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:41:34Z" } 2024-04-20T01:41:34.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577169.777139_1578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:29.777139Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:41:34.661 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:34.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:34.661 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:41:34.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:34.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:34.662 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:41:34.712 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:34.712 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:34.777 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:41:34.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:34.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:34.778 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:41:34.829 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:34.829 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:34.885 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:41:34.885 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-86 2024-04-20T01:41:34.885 DEBUG:boto:path=/aktlji-86/ 2024-04-20T01:41:34.885 DEBUG:boto:auth_path=/aktlji-86/ 2024-04-20T01:41:34.886 DEBUG:boto:Method: PUT 2024-04-20T01:41:34.886 DEBUG:boto:Path: /aktlji-86/ 2024-04-20T01:41:34.886 DEBUG:boto:Data: 2024-04-20T01:41:34.886 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.886 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:41:34.886 DEBUG:boto:Port: 8000 2024-04-20T01:41:34.886 DEBUG:boto:Params: {} 2024-04-20T01:41:34.886 DEBUG:boto:Token: None 2024-04-20T01:41:34.887 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:41:34 GMT /aktlji-86/ 2024-04-20T01:41:34.887 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gBTyGahrBLYh/92UHQ13deGq+pg= 2024-04-20T01:41:34.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gBTyGahrBLYh/92UHQ13deGq+pg=', 'Content-Length': '0'} 2024-04-20T01:41:34.916 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008109815029eda106-0066231d4e-1afa-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.917 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-87 2024-04-20T01:41:34.917 DEBUG:boto:path=/aktlji-87/ 2024-04-20T01:41:34.917 DEBUG:boto:auth_path=/aktlji-87/ 2024-04-20T01:41:34.917 DEBUG:boto:Method: PUT 2024-04-20T01:41:34.917 DEBUG:boto:Path: /aktlji-87/ 2024-04-20T01:41:34.917 DEBUG:boto:Data: 2024-04-20T01:41:34.917 DEBUG:boto:Headers: {} 2024-04-20T01:41:34.917 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:41:34.917 DEBUG:boto:Port: 8000 2024-04-20T01:41:34.918 DEBUG:boto:Params: {} 2024-04-20T01:41:34.918 DEBUG:boto:Token: None 2024-04-20T01:41:34.918 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:41:34 GMT /aktlji-87/ 2024-04-20T01:41:34.918 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rKBTyWY9LgStYbHY1Rl/FsoWhXo= 2024-04-20T01:41:34.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rKBTyWY9LgStYbHY1Rl/FsoWhXo=', 'Content-Length': '0'} 2024-04-20T01:41:34.945 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cf54f39ee73c2e1e-0066231d4e-1afa-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:34.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:34.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:34.946 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:35.000 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:35.000 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:35.054 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:41:35.075 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:41:35.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:35.075 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:35.075 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:41:35.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:35.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:35.076 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:41:35.076 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:41:35.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:35.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:35.076 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:41:35.076 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:34.954723Z" 2024-04-20T01:41:35.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:35.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:35.076 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:35.076 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:41:35.076 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:35.077 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:41:35.077 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:35.078 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:41:35.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:35.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:35.078 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:35.140 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:35.141 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:35.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:41:35.208+0000 7f6db38c2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:35.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:41:35.210+0000 7f6db38c2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:35.222 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:35.222 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:35.222 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:35.222 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:35.222 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:35.222 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:41:35.223 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:35.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:35.223 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:35.223 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:35.223 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:35.223 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:35.223 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:35.223 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:35.223 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:35.223 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:41:35.224 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:35.224 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:41:35.224 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:35.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:35.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:35.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:35.224 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:35.224 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:35.224 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:35.224 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:41:35.224 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:35.225 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:35.225 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:35.225 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:41:35.225 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:35.225 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:35.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:35.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:35.225 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:35.225 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:35.225 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:35.225 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.914839_1586.1", 2024-04-20T01:41:35.226 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:35.226 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:35.226 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:35.226 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.914839Z", 2024-04-20T01:41:35.226 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:35.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:35.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:35.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:35.226 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:35.226 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:35.226 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:35.227 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:35.227 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:35.227 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:35.227 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:35.227 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:41:35.227 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:35.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:35.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:35.227 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:35.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:35.227 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:35.228 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:41:35.228 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:41:35.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:35.228 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:35Z" 2024-04-20T01:41:35.228 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:35.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.914839_1586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.914839Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:41:35Z" } 2024-04-20T01:41:35.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.914839_1586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.914839Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:41:35.229 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:35.229 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.914839_1586.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:35.229 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577294.954723_1593.1 target=1_1713577294.914839_1586.1 2024-04-20T01:41:40.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:40.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:40.231 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:40.290 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:40.290 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:40.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:41:40.360+0000 7fdba8890780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:40.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:41:40.361+0000 7fdba8890780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:40.370 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:40.370 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:40.371 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:40.371 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:40.371 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:40.371 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:41:40.371 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:40.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:40.371 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:40.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:40.371 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:40.371 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:40.371 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:40.372 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:40.372 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:40.372 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:41:40.372 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:40.372 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:41:40.372 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:40.372 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:40.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:40.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:40.372 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:40.372 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:40.373 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:40.373 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:41:40.373 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:40.373 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:40.373 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:40.373 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:41:40.373 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:40.373 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:40.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:40.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:40.373 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:40.373 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:40.374 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:40.374 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:41:40.374 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:40.374 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:40.374 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:40.374 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:41:40.374 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:40.374 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:40.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:40.374 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:40.375 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:40.375 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:40.375 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:40.375 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:40.375 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:40.375 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:40.375 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:40.375 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:41:40.375 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:40.375 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:40.375 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:40.376 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:40.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:40.376 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:40.376 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:41:40.376 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:41:40.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:40.376 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:40Z" 2024-04-20T01:41:40.376 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:40.377 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:41:40Z" } 2024-04-20T01:41:40.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:41:40.377 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:40.377 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:40.378 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:41:40.378 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:41:40.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:40.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:40.378 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:40.440 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:40.440 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:40.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:41:40.504+0000 7f9d6d22d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:40.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:41:40.506+0000 7f9d6d22d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:40.508 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:41:40.508 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:41:40.508 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:41:40.508 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:41:40.508 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:41:40.508 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:41:40.509 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:40.509 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:40.509 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:41:40.509 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:40.509 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:41:40.509 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:40.510 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:40.510 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:40.510 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:40.510 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:41:40.510 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:40.510 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:41:40.510 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:40.510 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:40.510 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:40.510 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:40.510 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:41:40.511 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:40.511 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:40.511 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:41:40.511 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:40.511 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:41:40.511 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:40.511 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:41:40.511 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:40.511 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:40.511 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:40.511 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:40.512 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:41:40.512 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:40.512 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:40.512 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:41:40.512 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:40.512 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:41:40.512 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:40.512 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:41:40.513 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:40.513 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:40.513 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:40.513 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:40.513 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:41:40.513 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:40.513 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:40.513 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:40.513 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:40.513 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:41:40.513 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:40.513 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:41:40.514 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:40.514 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:40.514 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:40.514 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:40.514 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:40.514 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:41:40.514 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:41:40.514 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:41:40.514 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:40.514 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:41:40Z" 2024-04-20T01:41:40.514 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:41:40.515 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:41:40Z" } 2024-04-20T01:41:40.516 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:41:40.516 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:40.516 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:40.516 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:41:40.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:40.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:40.516 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:41:40.572 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:40.572 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:40.628 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:41:40.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:40.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:40.629 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:41:40.689 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:40.689 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:40.746 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:41:40.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--source-bucket=aktlji-86', '--dest-bucket=aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:40.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--source-bucket=aktlji-86', '--dest-bucket=aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:40.747 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --source-bucket=aktlji-86 --dest-bucket=aktlji-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:41:40.806 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:40.806 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:40.880 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"aktlji-86","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"aktlji-86","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:41:40.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:40.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:40.881 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:41:40.935 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:40.935 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:41.004 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"aktlji-86","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"aktlji-86","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:41:41.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:41.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:41.005 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:41:41.054 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:41.054 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:41.110 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"aktlji-86","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"aktlji-86","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:41:41.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:41.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:41.111 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:41:41.164 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:41.164 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:41.236 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:41:41.236 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:41:41.236 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 14, 2024-04-20T01:41:41.236 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:41:41.236 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T01:41:41.236 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T01:41:41.237 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:41:41.237 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T01:41:41.237 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.237 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T01:41:41.237 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T01:41:41.237 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T01:41:41.237 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T01:41:41.237 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:41:41.237 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:41:41.237 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:41:41.237 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T01:41:41.237 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T01:41:41.238 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:41:41.238 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:41:41.238 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.238 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:41:41.238 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T01:41:41.238 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:41:41.238 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:41:41.238 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:41:41.238 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:41:41.238 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:41:41.238 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:41:41.239 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:41:41.239 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:41:41.239 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T01:41:41.239 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:41:41.239 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:41:41.239 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:41:41.239 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:41:41.239 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:41:41.239 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:41.239 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.239 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.240 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:41:41.240 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T01:41:41.240 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:41:41.240 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:41:41.240 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:41:41.240 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:41:41.240 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:41:41.240 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:41:41.240 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:41:41.240 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:41:41.240 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:41:41.241 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:41:41.241 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:41:41.241 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:41:41.241 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:41:41.241 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:41:41.241 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:41.241 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.241 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.241 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:41:41.241 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T01:41:41.241 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:41:41.242 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:41:41.242 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:41:41.242 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:41:41.242 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:41:41.242 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:41:41.242 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:41:41.242 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:41:41.242 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:41:41.242 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:41:41.242 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:41:41.242 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:41:41.243 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:41:41.243 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:41:41.243 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:41.243 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:41.243 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:41:41.243 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T01:41:41.243 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.243 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T01:41:41.243 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T01:41:41.243 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T01:41:41.243 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T01:41:41.244 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:41.244 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:41.244 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:41:41.244 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T01:41:41.244 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:41:41.244 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T01:41:41.244 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [ 2024-04-20T01:41:41.244 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.244 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-group", 2024-04-20T01:41:41.244 INFO:teuthology.orchestra.run.smithi019.stdout: "data_flow": { 2024-04-20T01:41:41.244 INFO:teuthology.orchestra.run.smithi019.stdout: "symmetrical": [ 2024-04-20T01:41:41.245 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.245 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow1", 2024-04-20T01:41:41.245 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:41:41.245 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:41:41.245 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:41:41.245 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:41.245 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.245 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.245 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow", 2024-04-20T01:41:41.245 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:41:41.245 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:41:41.246 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:41:41.246 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:41.246 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:41.246 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:41:41.246 INFO:teuthology.orchestra.run.smithi019.stdout: "directional": [ 2024-04-20T01:41:41.246 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.246 INFO:teuthology.orchestra.run.smithi019.stdout: "source_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:41:41.246 INFO:teuthology.orchestra.run.smithi019.stdout: "dest_zone": "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:41:41.246 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:41.246 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:41.247 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.247 INFO:teuthology.orchestra.run.smithi019.stdout: "pipes": [ 2024-04-20T01:41:41.247 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.247 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe", 2024-04-20T01:41:41.247 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:41:41.247 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-86", 2024-04-20T01:41:41.247 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:41:41.247 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:41:41.247 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:41:41.247 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:41.247 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.247 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:41:41.248 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-86", 2024-04-20T01:41:41.248 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:41:41.248 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:41:41.248 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:41:41.248 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:41.248 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.248 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:41:41.248 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:41:41.248 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:41:41.248 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:41:41.248 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:41.249 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.249 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:41:41.249 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:41:41.249 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:41:41.249 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:41:41.249 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:41.249 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:41.249 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:41:41.249 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "enabled" 2024-04-20T01:41:41.249 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:41.249 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:41.249 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.250 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T01:41:41.250 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:41:41.250 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:41.250 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:41.250 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:41:41.250 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T01:41:41.250 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.250 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:41:41.250 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T01:41:41.250 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.251 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.251 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:41:41.251 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T01:41:41.251 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.251 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:41.251 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:41:41.251 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T01:41:41.251 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:41.251 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:41.251 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.251 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T01:41:41.252 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:41:41.252 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T01:41:41.252 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T01:41:41.252 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:41:41.252 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:41:41.252 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:41:41.252 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:41:41.252 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:41:41.252 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.252 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T01:41:41.252 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:41:41.252 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:41:41.253 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:41:41.253 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:41:41.253 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:41:41.253 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.253 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T01:41:41.253 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:41:41.253 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:41:41.253 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:41:41.253 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:41:41.253 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:41:41.253 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.254 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T01:41:41.254 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:41:41.254 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:41:41.254 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:41:41.254 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:41:41.254 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:41:41.254 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.263 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T01:41:41.263 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:41:41.264 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:41:41.264 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:41:41.264 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:41:41.264 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:41:41.264 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:41.264 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:41.264 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:41:41.264 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T01:41:41.264 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:41.265 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:41:41.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:41.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:41:41.266 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:41:41.321 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:41.322 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:41.414 INFO:teuthology.orchestra.run.smithi019.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"test-zone2","bucket":"aktlji-86"},"dest":{"zone":"test-zone1","bucket":"aktlji-86"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"test-zone1","bucket":"aktlji-86"},"dest":{"zone":"test-zone2","bucket":"aktlji-86"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2024-04-20T01:41:41.414 DEBUG:boto:path=/aktlji-86/ 2024-04-20T01:41:41.414 DEBUG:boto:auth_path=/aktlji-86/ 2024-04-20T01:41:41.414 DEBUG:boto:Method: HEAD 2024-04-20T01:41:41.414 DEBUG:boto:Path: /aktlji-86/ 2024-04-20T01:41:41.415 DEBUG:boto:Data: 2024-04-20T01:41:41.415 DEBUG:boto:Headers: {} 2024-04-20T01:41:41.415 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:41:41.415 DEBUG:boto:Port: 8000 2024-04-20T01:41:41.415 DEBUG:boto:Params: {} 2024-04-20T01:41:41.415 DEBUG:boto:Token: None 2024-04-20T01:41:41.415 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:41 GMT /aktlji-86/ 2024-04-20T01:41:41.415 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Easkklmq4gz9KBF5vK6hbHZaMk= 2024-04-20T01:41:41.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Easkklmq4gz9KBF5vK6hbHZaMk=', 'Content-Length': '0'} 2024-04-20T01:41:41.742 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ef400936ed55f30-0066231d55-1b8e-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:41.742 DEBUG:boto:path=/aktlji-86/obj1 2024-04-20T01:41:41.743 DEBUG:boto:auth_path=/aktlji-86/obj1 2024-04-20T01:41:41.743 DEBUG:boto:Method: PUT 2024-04-20T01:41:41.743 DEBUG:boto:Path: /aktlji-86/obj1 2024-04-20T01:41:41.743 DEBUG:boto:Data: 2024-04-20T01:41:41.743 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:41:41.743 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:41:41.743 DEBUG:boto:Port: 8000 2024-04-20T01:41:41.743 DEBUG:boto:Params: {} 2024-04-20T01:41:41.743 DEBUG:boto:Token: None 2024-04-20T01:41:41.743 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:41:41 GMT /aktlji-86/obj1 2024-04-20T01:41:41.744 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a8o3HU8dXMhASILdXa2zuiAhIGw= 2024-04-20T01:41:41.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:41:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a8o3HU8dXMhASILdXa2zuiAhIGw='} 2024-04-20T01:41:41.790 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b16cd581fac12ee-0066231d55-1b8e-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:41.791 DEBUG:boto:path=/aktlji-86/ 2024-04-20T01:41:41.791 DEBUG:boto:auth_path=/aktlji-86/ 2024-04-20T01:41:41.791 DEBUG:boto:Method: HEAD 2024-04-20T01:41:41.791 DEBUG:boto:Path: /aktlji-86/ 2024-04-20T01:41:41.791 DEBUG:boto:Data: 2024-04-20T01:41:41.791 DEBUG:boto:Headers: {} 2024-04-20T01:41:41.791 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:41:41.791 DEBUG:boto:Port: 8000 2024-04-20T01:41:41.791 DEBUG:boto:Params: {} 2024-04-20T01:41:41.792 DEBUG:boto:Token: None 2024-04-20T01:41:41.792 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:41 GMT /aktlji-86/ 2024-04-20T01:41:41.792 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Easkklmq4gz9KBF5vK6hbHZaMk= 2024-04-20T01:41:41.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Easkklmq4gz9KBF5vK6hbHZaMk=', 'Content-Length': '0'} 2024-04-20T01:41:41.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068fcd2cab8f441f7-0066231d55-1b8e-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:41.798 DEBUG:boto:path=/aktlji-86/obj2 2024-04-20T01:41:41.798 DEBUG:boto:auth_path=/aktlji-86/obj2 2024-04-20T01:41:41.798 DEBUG:boto:Method: PUT 2024-04-20T01:41:41.798 DEBUG:boto:Path: /aktlji-86/obj2 2024-04-20T01:41:41.798 DEBUG:boto:Data: 2024-04-20T01:41:41.799 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:41:41.799 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:41:41.799 DEBUG:boto:Port: 8000 2024-04-20T01:41:41.799 DEBUG:boto:Params: {} 2024-04-20T01:41:41.799 DEBUG:boto:Token: None 2024-04-20T01:41:41.799 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:41:41 GMT /aktlji-86/obj2 2024-04-20T01:41:41.799 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:okref5OyOWZLoe9mlu2Uk+K7NOU= 2024-04-20T01:41:41.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:41:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:okref5OyOWZLoe9mlu2Uk+K7NOU='} 2024-04-20T01:41:41.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008198acdb169858c7-0066231d55-1b8e-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:41.829 DEBUG:boto:path=/aktlji-86/ 2024-04-20T01:41:41.829 DEBUG:boto:auth_path=/aktlji-86/ 2024-04-20T01:41:41.829 DEBUG:boto:Method: HEAD 2024-04-20T01:41:41.830 DEBUG:boto:Path: /aktlji-86/ 2024-04-20T01:41:41.830 DEBUG:boto:Data: 2024-04-20T01:41:41.830 DEBUG:boto:Headers: {} 2024-04-20T01:41:41.830 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:41:41.830 DEBUG:boto:Port: 8000 2024-04-20T01:41:41.830 DEBUG:boto:Params: {} 2024-04-20T01:41:41.830 DEBUG:boto:Token: None 2024-04-20T01:41:41.831 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:41 GMT /aktlji-86/ 2024-04-20T01:41:41.831 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:3Easkklmq4gz9KBF5vK6hbHZaMk= 2024-04-20T01:41:41.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Easkklmq4gz9KBF5vK6hbHZaMk=', 'Content-Length': '0'} 2024-04-20T01:41:41.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043a8069a09f40282-0066231d55-1b8e-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:41.834 DEBUG:boto:path=/aktlji-86/obj3 2024-04-20T01:41:41.834 DEBUG:boto:auth_path=/aktlji-86/obj3 2024-04-20T01:41:41.835 DEBUG:boto:Method: PUT 2024-04-20T01:41:41.835 DEBUG:boto:Path: /aktlji-86/obj3 2024-04-20T01:41:41.835 DEBUG:boto:Data: 2024-04-20T01:41:41.835 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:41:41.835 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:41:41.835 DEBUG:boto:Port: 8000 2024-04-20T01:41:41.835 DEBUG:boto:Params: {} 2024-04-20T01:41:41.835 DEBUG:boto:Token: None 2024-04-20T01:41:41.835 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:41:41 GMT /aktlji-86/obj3 2024-04-20T01:41:41.835 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AVb+m5MgzjT/AV8pJLK+UxXqA3c= 2024-04-20T01:41:41.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:41:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AVb+m5MgzjT/AV8pJLK+UxXqA3c='} 2024-04-20T01:41:41.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000952c987070e80693-0066231d55-1b8e-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:41.879 DEBUG:boto:path=/aktlji-87/ 2024-04-20T01:41:41.879 DEBUG:boto:auth_path=/aktlji-87/ 2024-04-20T01:41:41.879 DEBUG:boto:Method: HEAD 2024-04-20T01:41:41.879 DEBUG:boto:Path: /aktlji-87/ 2024-04-20T01:41:41.879 DEBUG:boto:Data: 2024-04-20T01:41:41.879 DEBUG:boto:Headers: {} 2024-04-20T01:41:41.879 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:41:41.879 DEBUG:boto:Port: 8000 2024-04-20T01:41:41.879 DEBUG:boto:Params: {} 2024-04-20T01:41:41.879 DEBUG:boto:Token: None 2024-04-20T01:41:41.880 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:41:41 GMT /aktlji-87/ 2024-04-20T01:41:41.880 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RdSR8fXM5htD4sqzrLZdL1vB0ek= 2024-04-20T01:41:41.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RdSR8fXM5htD4sqzrLZdL1vB0ek=', 'Content-Length': '0'} 2024-04-20T01:41:41.888 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007185dbb4d020de02-0066231d55-1b8e-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:41.888 DEBUG:boto:path=/aktlji-87/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-20T01:41:41.889 DEBUG:boto:auth_path=/aktlji-87/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-20T01:41:41.889 DEBUG:boto:Method: PUT 2024-04-20T01:41:41.889 DEBUG:boto:Path: /aktlji-87/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-20T01:41:41.889 DEBUG:boto:Data: 2024-04-20T01:41:41.889 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:41:41.889 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:41:41.889 DEBUG:boto:Port: 8000 2024-04-20T01:41:41.889 DEBUG:boto:Params: {} 2024-04-20T01:41:41.889 DEBUG:boto:Token: None 2024-04-20T01:41:41.889 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:41:41 GMT /aktlji-87/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-20T01:41:41.890 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Hui+y1UMyGrdn1CJaGj1buXoLKI= 2024-04-20T01:41:41.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:41:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Hui+y1UMyGrdn1CJaGj1buXoLKI='} 2024-04-20T01:41:41.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dedb9614c9499627-0066231d55-1b8e-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:41:41.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:41.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:41.911 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:41.988 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:41.989 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:42.103 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:41:42.146 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:41:42.147 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:42.147 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:42.147 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:41:42.147 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:42.147 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:42.147 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:41:42.147 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:41:42.147 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:42.147 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:42.147 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:41:42.147 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:34.954723Z" 2024-04-20T01:41:42.148 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:42.148 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:42.148 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:42.148 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:41:42.148 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:42.148 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:41:42.149 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:42.149 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:41:42.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:42.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:42.150 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:42.212 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:42.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:41:42.289+0000 7efe6edee780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:42.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:41:42.291+0000 7efe6edee780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:42.309 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:41:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:42.310 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:41:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:41:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:42.311 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:41:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:41:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:42.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:41:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:41:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:41:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:41:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:41:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:42.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:41:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:41:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:42.315 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:42Z" 2024-04-20T01:41:42.315 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:42.316 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:41:42Z" } 2024-04-20T01:41:42.316 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:41:42.316 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:42.317 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:42.317 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:41:42.317 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:41:42.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:42.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:42.317 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:42.379 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:42.379 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:42.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:41:42.445+0000 7f284f386780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:42.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:41:42.446+0000 7f284f386780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:41:42.474 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:41:42.474 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:41:42.474 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:41:42.474 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:41:42.474 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:41:42.474 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:41:42.474 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:42.475 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:42.475 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:41:42.475 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:42.475 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:41:42.475 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:42.475 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:42.475 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:41:42.475 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:42.475 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:41:42.475 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:42.475 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:41:42.476 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:42.476 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:42.476 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:42.476 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:42.476 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:41:42.476 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:42.476 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:42.476 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:41:42.476 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:42.476 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:41:42.476 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:42.476 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:41:42.477 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:42.477 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:42.477 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:42.477 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:42.477 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:41:42.477 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:42.477 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:42.477 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:41:42.477 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:42.477 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:41:42.477 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:42.478 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:41:42.478 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:42.478 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:42.478 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:42.478 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:42.478 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:41:42.478 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:41:42.478 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:41:42.478 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:41:42.478 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:41:42.478 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:41:42.478 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:41:42.479 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:41:42.479 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:41:42.479 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:42.479 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:42.479 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:41:42.479 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:42.479 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:41:42.479 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:41:42.479 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:41:42.479 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:42.479 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:41:42Z" 2024-04-20T01:41:42.480 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:41:42.480 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:41:42Z" } 2024-04-20T01:41:42.481 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:41:42.481 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:42.481 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:41:42.481 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:41:42.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:42.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:42.481 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:42.542 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:42.543 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:42.614 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:41:42.614 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:42.614 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:41:42.614 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:41:42.614 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:42.614 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:42.614 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000027190", 2024-04-20T01:41:42.615 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:41.865175Z" 2024-04-20T01:41:42.615 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:42.615 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:42.615 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:41:42.615 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:41:42.615 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:41:42.615 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:41:42.615 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:41:42.615 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:41:42.615 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:41:42.615 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:41:42.616 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:41:42.617 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:41:42.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:42.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:42.617 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:42.678 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:42.678 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:42.741 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:42.742 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:42.742 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:42.742 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:42.742 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:42.742 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:41:42.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:42.742 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:42.746 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:42.746 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:42.746 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:42.746 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:42.746 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:41:42.746 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:42.746 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:41:42.746 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:42.746 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:41:42.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:42.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:42.747 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:42.747 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:42.747 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:42.747 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:42.747 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:41:42.747 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:42.747 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:41:42.747 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:42.747 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:41:42.747 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:42.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:42.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:42.748 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:42.748 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:42.748 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:42.748 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:41:42.748 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:42.748 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:42.748 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:42.748 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:41:42.748 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:42.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:42.748 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:42.749 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:42.749 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:42.749 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:42.749 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:41:42.749 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:42.749 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:42.749 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:42.749 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:41:42.749 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:42.749 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:42.749 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:42.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:42.750 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:42.750 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:41:42.750 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:41:42.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:42.750 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:42Z" 2024-04-20T01:41:42.750 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:42.751 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:41:42Z" } 2024-04-20T01:41:42.751 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:41:42.751 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:41:42.752 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:41:42.752 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:41:47.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:47.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:47.753 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:47.814 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:47.814 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:47.905 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:47.905 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:47.905 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:47.905 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:47.905 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:47.905 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:41:47.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:47.906 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:47.906 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:47.906 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:47.906 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:47.906 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:47.906 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:41:47.906 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:47.906 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:41:47.906 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:47.906 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:41:47.907 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:47.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:47.907 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:47.907 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:47.907 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:47.907 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:47.907 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:41:47.907 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:47.907 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:41:47.907 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:47.907 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:41:47.908 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:47.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:47.908 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:47.908 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:47.908 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:47.908 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:47.908 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:41:47.908 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:47.908 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:47.908 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:47.908 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:47.909 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:47.910 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:47.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:47.910 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:47.910 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:41:47.910 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:41:47.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:47.910 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:47Z" 2024-04-20T01:41:47.910 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:47.911 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:41:47Z" } 2024-04-20T01:41:47.911 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:41:47.912 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:41:47.912 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:41:47.912 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:41:52.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:52.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:52.913 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:52.974 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:52.974 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:53.056 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:53.057 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:53.057 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:53.057 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:53.057 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:53.058 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:41:53.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:53.058 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:53.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:53.058 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:53.059 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:53.059 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:53.059 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:41:53.059 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:53.059 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:41:53.060 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:53.060 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:41:53.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:53.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:53.060 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:53.061 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:53.061 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:53.061 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:53.061 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:41:53.061 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:53.062 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:41:53.062 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:53.062 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:41:53.062 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:53.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:53.063 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:53.063 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:53.063 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:53.063 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:53.063 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:41:53.064 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:53.064 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:53.064 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:53.064 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:41:53.065 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:53.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:53.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:53.065 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:53.065 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:53.066 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:53.066 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:41:53.066 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:53.066 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:53.067 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:53.067 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:41:53.067 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:53.068 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:53.068 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:53.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:53.068 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:53.068 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:41:53.069 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:41:53.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:53.069 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:53Z" 2024-04-20T01:41:53.069 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:53.071 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:41:53Z" } 2024-04-20T01:41:53.071 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:41:53.071 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:41:53.072 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:41:53.072 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:41:58.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:58.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:41:58.074 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:41:58.136 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:41:58.136 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:41:58.210 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:41:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:41:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:41:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:41:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:41:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:41:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:41:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:41:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:41:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:41:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:58.211 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:41:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:41:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:41:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:41:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:41:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:41:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:41:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:58.213 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:41:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:41:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:41:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:41:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:41:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:41:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:41:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:41:58.214 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:41:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:41:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:41:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:41:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:41:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:41:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:41:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:41:58.215 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:41:58Z" 2024-04-20T01:41:58.215 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:41:58.216 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:41:58Z" } 2024-04-20T01:41:58.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:41:58.217 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:41:58.217 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:41:58.217 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:42:03.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:03.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:03.219 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:42:03.280 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:42:03.280 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:42:03.350 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:42:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:42:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:42:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:42:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:42:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:42:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:42:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:42:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:42:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:42:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:42:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:42:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:42:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:42:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:42:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:42:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:42:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:42:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:42:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:42:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:42:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:42:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:42:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:42:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:42:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:42:03Z" 2024-04-20T01:42:03.356 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:42:03.357 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:42:03Z" } 2024-04-20T01:42:03.357 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:42:03.357 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:42:03.357 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:42:03.357 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:42:08.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:08.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:08.359 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:42:08.420 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:42:08.420 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:42:08.503 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:42:08.504 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:42:08.504 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:42:08.504 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:42:08.504 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:42:08.504 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:42:08.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:08.504 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:42:08.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:08.504 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:42:08.504 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:08.504 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:08.505 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:42:08.505 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:08.505 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:42:08.505 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:08.505 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:42:08.505 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:08.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:08.505 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:08.505 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:42:08.505 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:08.505 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:08.505 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:42:08.506 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:08.506 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:42:08.506 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:08.506 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:42:08.506 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:08.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:08.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:08.506 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:42:08.506 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:08.506 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:08.507 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:42:08.507 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:08.507 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:08.507 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:08.507 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:42:08.507 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:08.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:08.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:08.507 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:42:08.507 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:08.507 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:08.508 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:42:08.508 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:08.508 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:08.508 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:08.508 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:42:08.508 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:08.508 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:08.508 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:42:08.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:08.508 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:42:08.508 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:42:08.509 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:42:08.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:08.509 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:42:08Z" 2024-04-20T01:42:08.509 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:42:08.510 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:42:08Z" } 2024-04-20T01:42:08.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:42:08.510 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:42:08.510 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:42:08.510 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:42:13.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:13.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:13.512 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:42:13.574 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:42:13.574 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:42:13.640 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:42:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:42:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:42:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:42:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:42:13.641 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:42:13.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:13.641 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:42:13.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:13.641 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:42:13.641 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:13.641 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:13.641 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:42:13.641 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:13.641 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:42:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:42:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:42:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:42:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:13.642 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:42:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:42:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:42:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:42:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:13.643 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:42:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:13.645 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:42:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:42:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:42:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:42:13.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:42:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:42:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:42:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:13.647 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:42:13Z" 2024-04-20T01:42:13.647 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:42:13.648 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:42:13Z" } 2024-04-20T01:42:13.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:42:13.648 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:42:13.649 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:42:13.649 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:42:18.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:18.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:18.650 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:42:18.712 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:42:18.712 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:42:18.780 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:42:18.780 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:42:18.780 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:42:18.781 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:42:18.781 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:42:18.781 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:42:18.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:18.781 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:42:18.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:18.781 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:42:18.781 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:18.781 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:18.781 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:42:18.782 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:18.782 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:42:18.782 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:18.782 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:42:18.782 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:18.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:18.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:18.782 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:42:18.782 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:18.782 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:18.783 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:42:18.783 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:18.783 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:42:18.783 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:18.783 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:42:18.783 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:18.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:18.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:18.783 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:42:18.783 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:18.783 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:18.783 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:42:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:42:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:42:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:42:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:18.785 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:18.785 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:18.785 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:42:18.785 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:18.785 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:18.785 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:42:18.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:18.785 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:42:18.785 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:42:18.785 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:42:18.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:18.786 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:42:18Z" 2024-04-20T01:42:18.786 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:42:18.786 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:42:18Z" } 2024-04-20T01:42:18.787 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:42:18.787 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:42:18.787 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:42:18.787 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:42:23.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:23.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:23.788 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:42:23.850 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:42:23.851 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:42:23.912 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:42:23Z" 2024-04-20T01:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:42:23.919 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:42:23Z" } 2024-04-20T01:42:23.919 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:42:23.920 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:42:23.920 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:42:23.920 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:42:28.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:28.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:28.921 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:42:28.983 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:42:28.983 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:42:29.061 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:42:29.062 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:42:29.062 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:42:29.062 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:42:29.062 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:42:29.062 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:42:29.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:29.062 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:42:29.062 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:29.062 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:42:29.062 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:29.063 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:29.063 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:42:29.063 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:29.063 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:42:29.063 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:29.063 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:42:29.063 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:29.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:29.063 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:29.063 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:42:29.063 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:29.064 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:29.064 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:42:29.064 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:29.064 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:42:29.064 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:29.064 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:42:29.064 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:29.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:29.064 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:29.064 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:42:29.064 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:29.065 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:29.065 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:42:29.065 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:29.065 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:29.065 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:29.065 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:42:29.065 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:29.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:29.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:29.065 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:42:29.066 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:29.066 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:29.066 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:42:29.066 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:29.066 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:29.066 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:29.066 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:42:29.066 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:29.066 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:29.066 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:42:29.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:29.067 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:42:29.067 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:42:29.067 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:42:29.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:29.067 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:42:29Z" 2024-04-20T01:42:29.067 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:42:29.068 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:42:29Z" } 2024-04-20T01:42:29.068 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:42:29.069 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:42:29.069 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:42:29.069 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:42:34.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:34.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:34.071 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:42:34.133 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:42:34.133 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:42:34.197 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:42:34.197 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:42:34.197 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:42:34.198 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:42:34.199 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:34.200 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:34.201 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:42:34.202 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:34.202 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:34.202 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:34.202 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:42:34.202 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:34.202 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:34.202 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:42:34.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:34.202 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:42:34.202 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:42:34.203 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:42:34.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:34.203 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:42:34Z" 2024-04-20T01:42:34.203 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:42:34.204 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:42:34Z" } 2024-04-20T01:42:34.204 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:42:34.204 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:42:34.204 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:42:34.204 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:42:39.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:39.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:39.206 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:42:39.268 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:42:39.268 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:42:39.346 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:42:39.346 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:42:39.346 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:42:39.346 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:42:39.346 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:42:39.346 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:42:39.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:39.346 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:42:39.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:39.347 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:42:39.347 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:39.347 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:39.347 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:42:39.347 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:39.347 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:42:39.347 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:39.347 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:42:39.347 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:39.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:39.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:39.348 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:42:39.348 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:39.348 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:39.348 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:42:39.348 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:39.348 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:42:39.348 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:39.348 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:42:39.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:39.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:39.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:39.349 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:42:39.349 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:39.349 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:39.349 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:42:39.349 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:39.349 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:39.349 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:39.349 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:42:39.349 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:39.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:39.350 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:39.350 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:42:39.350 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:39.350 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:39.350 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:42:39.350 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:39.350 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:39.350 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:39.350 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:42:39.350 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:39.350 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:39.351 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:42:39.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:39.351 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:42:39.351 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:42:39.351 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:42:39.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:39.351 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:42:39Z" 2024-04-20T01:42:39.351 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:42:39.352 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:42:39Z" } 2024-04-20T01:42:39.352 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:42:39.352 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:42:39.352 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:42:39.353 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:42:44.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:44.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:44.354 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:42:44.416 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:42:44.416 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:42:44.480 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:42:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:42:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:42:44.480 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:42:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:42:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:42:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:42:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:42:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:42:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:44.481 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:42:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:42:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:42:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:42:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:42:44.482 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:42:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:42:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:42:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:42:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:42:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:42:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:44.484 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:42:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:42:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:42:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:42:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:42:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:44.485 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:42:44Z" 2024-04-20T01:42:44.485 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:42:44.486 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:42:44Z" } 2024-04-20T01:42:44.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:42:44.487 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:42:44.487 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:42:44.487 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:42:49.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:49.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:49.488 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:42:49.552 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:42:49.552 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:42:49.637 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:42:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:42:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:42:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:42:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:42:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:42:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:49.638 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:42:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:42:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:42:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:42:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:42:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:49.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:42:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:42:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:42:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:42:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:49.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:42:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:42:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:49.641 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:42:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:42:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:42:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:49.642 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:42:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:42:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:42:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:42:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:42:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:49.643 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:42:49Z" 2024-04-20T01:42:49.643 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:42:49.644 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:42:49Z" } 2024-04-20T01:42:49.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:42:49.645 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:42:49.645 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:42:49.645 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:42:54.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:54.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:54.647 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:42:54.708 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:42:54.708 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:42:54.792 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:42:54.792 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:42:54.792 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:42:54.792 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:42:54.792 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:42:54.792 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:42:54.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:54.793 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:42:54.793 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:54.793 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:42:54.793 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:54.793 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:54.793 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:42:54.793 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:54.793 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:42:54.793 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:54.793 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:42:54.793 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:54.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:54.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:54.794 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:42:54.794 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:54.794 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:54.794 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:42:54.794 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:54.794 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:42:54.794 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:54.794 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:42:54.794 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:54.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:54.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:54.795 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:42:54.795 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:54.795 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:54.795 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:42:54.795 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:54.795 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:54.795 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:54.795 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:42:54.795 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:54.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:54.796 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:54.796 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:42:54.796 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:54.796 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:54.796 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:42:54.796 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:54.796 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:54.796 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:54.796 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:42:54.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:54.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:54.796 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:42:54.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:54.797 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:42:54.797 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:42:54.797 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:42:54.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:54.797 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:42:54Z" 2024-04-20T01:42:54.797 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:42:54.798 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:42:54Z" } 2024-04-20T01:42:54.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:42:54.798 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:42:54.799 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:42:54.799 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:42:59.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:59.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:42:59.800 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:42:59.862 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:42:59.862 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:42:59.925 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:42:59.925 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:42:59.926 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:42:59.926 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:42:59.926 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:42:59.926 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:42:59.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:59.926 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:42:59.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:59.926 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:42:59.926 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:59.926 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:59.927 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:42:59.927 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:59.927 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:42:59.927 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:59.927 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:42:59.927 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:59.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:59.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:59.927 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:42:59.927 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:59.928 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:59.928 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:42:59.928 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:59.928 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:42:59.928 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:59.928 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:42:59.928 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:59.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:59.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:59.929 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:42:59.929 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:59.929 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:59.929 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:42:59.929 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:59.929 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:59.929 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:59.929 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:42:59.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:59.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:59.929 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:42:59.930 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:42:59.930 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:42:59.930 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:42:59.930 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:42:59.930 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:42:59.930 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:42:59.930 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:42:59.930 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:42:59.930 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:59.931 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:42:59.931 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:42:59.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:59.931 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:42:59.931 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:42:59.931 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:42:59.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:42:59.931 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:42:59Z" 2024-04-20T01:42:59.931 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:42:59.932 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:42:59Z" } 2024-04-20T01:42:59.933 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:42:59.933 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:42:59.933 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:42:59.933 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:43:04.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:04.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:04.935 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:04.996 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:04.996 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:05.071 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:43:05.071 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:43:05.071 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:43:05.071 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:43:05.071 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:43:05.072 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:43:05.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:05.072 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:43:05.072 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:05.072 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:43:05.072 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:05.072 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:05.072 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:43:05.072 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:05.072 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:43:05.073 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:05.073 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:43:05.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:05.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:05.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:05.073 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:43:05.073 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:05.073 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:05.073 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:43:05.074 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:05.074 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:43:05.074 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:05.074 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:43:05.074 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:05.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:05.074 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:05.074 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:43:05.074 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:05.075 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:05.075 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:43:05.075 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:05.075 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:05.075 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:05.075 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:43:05.075 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:05.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:05.075 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:05.075 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:43:05.076 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:05.076 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:05.076 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:43:05.076 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:05.076 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:05.076 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:05.076 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:43:05.076 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:05.076 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:05.076 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:43:05.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:05.077 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:43:05.077 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:43:05.077 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:43:05.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:05.077 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:43:05Z" 2024-04-20T01:43:05.077 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:43:05.078 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:43:05Z" } 2024-04-20T01:43:05.078 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:43:05.079 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:43:05.079 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:43:05.079 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:43:10.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:10.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:10.080 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:10.223 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:43:10.223 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:43:10.223 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:43:10.223 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:43:10.223 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:43:10.223 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:43:10.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:10.224 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:43:10.224 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:10.224 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:43:10.224 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:10.224 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:10.224 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:43:10.224 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:10.224 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:43:10.224 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:10.224 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:43:10.224 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:10.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:10.225 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:10.225 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:43:10.225 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:10.225 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:10.225 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:43:10.225 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:10.225 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:43:10.225 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:10.225 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:43:10.225 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:10.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:10.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:10.226 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:43:10.226 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:10.226 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:10.226 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:43:10.226 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:10.226 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:10.226 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:10.226 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:43:10.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:10.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:10.227 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:10.227 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:43:10.227 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:10.227 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:10.227 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:43:10.227 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:10.227 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:10.227 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:10.227 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:43:10.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:10.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:10.227 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:43:10.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:10.228 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:43:10.228 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:43:10.228 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:43:10.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:10.228 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:43:10Z" 2024-04-20T01:43:10.228 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:43:10.229 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:43:10Z" } 2024-04-20T01:43:10.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:43:10.229 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:43:10.230 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:43:10.230 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:43:15.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:15.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:15.231 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:15.293 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:15.293 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:16.364 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:43:16.364 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:43:16.364 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:43:16.364 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:43:16.364 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:43:16.364 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:43:16.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:16.365 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:43:16.365 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:16.365 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:43:16.365 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:16.365 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:16.365 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:43:16.365 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:16.365 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:43:16.365 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:16.365 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:43:16.365 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:16.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:16.366 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:16.366 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:43:16.366 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:16.366 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:16.366 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:43:16.366 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:16.367 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:43:16.367 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:16.367 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:43:16.367 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:16.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:16.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:16.367 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:43:16.367 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:16.367 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:16.367 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:43:16.367 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:16.368 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:16.368 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:16.368 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:43:16.368 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:16.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:16.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:16.368 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:43:16.368 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:16.368 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:16.368 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:43:16.368 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:16.369 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:16.369 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:16.369 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:43:16.369 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:16.369 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:16.369 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:43:16.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:16.369 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:43:16.369 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:43:16.369 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:43:16.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:16.369 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:43:16Z" 2024-04-20T01:43:16.370 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:43:16.371 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:43:16Z" } 2024-04-20T01:43:16.371 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:43:16.371 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:43:16.371 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:43:16.371 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:43:21.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:21.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:21.373 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:21.434 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:21.505 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:43:21.506 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:43:21.506 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:43:21.506 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:43:21.506 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:43:21.506 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:43:21.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:21.506 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:43:21.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:21.506 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:43:21.506 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:21.507 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:21.507 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:43:21.507 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:21.507 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:43:21.507 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:21.507 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:43:21.507 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:21.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:21.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:21.507 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:43:21.507 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:43:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:43:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:43:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:43:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:21.509 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:43:21.509 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:21.509 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:21.509 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:21.509 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:43:21.509 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:21.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:21.509 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:21.509 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:43:21.509 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:21.509 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:21.509 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:43:21.510 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:21.510 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:21.510 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:21.510 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:43:21.510 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:21.510 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:21.510 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:43:21.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:21.510 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:43:21.510 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:43:21.510 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:43:21.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:21.511 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:43:21Z" 2024-04-20T01:43:21.511 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:43:21.512 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:43:21Z" } 2024-04-20T01:43:21.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:43:21.512 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:43:21.512 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:43:21.512 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:43:26.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:26.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:26.513 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:26.576 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:26.576 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:26.642 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:43:26.642 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:43:26.642 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:43:26.643 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:43:26.643 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:43:26.643 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:43:26.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:26.643 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:43:26.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:26.643 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:43:26.643 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:26.643 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:26.643 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:43:26.644 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:26.644 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:43:26.644 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:26.644 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:43:26.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:26.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:26.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:26.645 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:43:26.645 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:26.645 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:26.645 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:43:26.645 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:26.646 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:43:26.646 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:26.646 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:43:26.646 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:26.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:26.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:26.646 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:43:26.646 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:26.646 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:26.646 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:43:26.646 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:26.647 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:26.647 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:26.647 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:43:26.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:26.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:26.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:26.647 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:43:26.647 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:26.647 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:26.647 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:43:26.647 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:26.648 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:26.648 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:26.648 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:43:26.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:26.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:26.648 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:43:26.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:26.648 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:43:26.648 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:43:26.648 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:43:26.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:26.648 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:43:26Z" 2024-04-20T01:43:26.649 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:43:26.650 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:43:26Z" } 2024-04-20T01:43:26.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:43:26.650 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:43:26.650 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:43:26.650 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:43:31.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:31.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:31.652 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:31.714 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:31.787 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:43:31.787 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:43:31.787 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:43:31.787 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:43:31.787 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:43:31.787 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:43:31.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:31.787 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:43:31.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:31.788 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:43:31.788 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:31.788 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:31.788 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:43:31.788 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:31.788 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:43:31.788 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:31.788 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:43:31.788 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:31.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:31.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:31.789 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:43:31.789 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:31.789 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:31.789 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:43:31.789 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:31.789 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:43:31.789 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:31.789 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:43:31.789 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:31.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:31.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:31.790 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:43:31.790 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:31.790 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:31.790 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:43:31.790 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:31.790 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:31.790 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:31.790 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:43:31.790 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:31.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:31.790 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:31.791 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:43:31.791 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:31.791 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:31.791 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:43:31.791 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:31.791 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:31.791 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:31.791 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:43:31.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:31.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:31.792 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:43:31.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:31.792 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:43:31.792 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:43:31.792 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:43:31.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:31.792 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:43:31Z" 2024-04-20T01:43:31.792 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:43:31.793 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:43:31Z" } 2024-04-20T01:43:31.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:43:31.793 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:43:31.794 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:43:31.794 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:43:36.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:36.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:36.798 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:36.860 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:36.924 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:43:36.924 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:43:36.924 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:43:36.925 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:43:36.925 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:43:36.925 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:43:36.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:36.925 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:43:36.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:36.925 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:43:36.925 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:36.925 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:36.925 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:43:36.925 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:36.925 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:43:36.926 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:36.926 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:43:36.926 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:36.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:36.926 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:36.926 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:43:36.926 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:36.926 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:36.926 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:43:36.926 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:36.926 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:43:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:43:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:43:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:43:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:36.928 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:43:36.928 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:36.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:36.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:36.928 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:43:36.928 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:36.928 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:36.928 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:43:36.928 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:36.928 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:36.928 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:36.928 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:43:36.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:36.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:36.929 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:43:36.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:36.929 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:43:36.929 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:43:36.929 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:43:36.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:36.929 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:43:36Z" 2024-04-20T01:43:36.929 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:43:36.930 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:43:36Z" } 2024-04-20T01:43:36.930 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:43:36.931 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:43:36.931 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:43:36.931 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:43:41.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:41.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:41.932 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:41.993 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:42.061 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:43:42.061 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:43:42.061 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:43:42.061 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:43:42.061 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:43:42.062 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:43:42.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:42.062 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:43:42.062 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:42.062 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:43:42.062 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:42.062 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:42.062 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:43:42.062 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:42.062 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:43:42.062 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:42.063 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:43:42.063 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:42.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:42.063 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:42.063 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:43:42.063 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:42.063 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:42.063 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:43:42.063 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:42.063 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:43:42.063 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:42.064 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:43:42.064 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:42.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:42.064 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:42.064 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:43:42.064 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:42.064 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:42.064 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:43:42.064 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:42.064 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:42.064 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:42.064 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:43:42.065 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:42.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:42.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:42.065 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:43:42.065 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:42.065 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:42.065 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:43:42.065 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:42.065 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:42.065 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:42.065 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:43:42.066 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:42.066 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:42.066 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:43:42.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:42.066 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:43:42.066 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:43:42.066 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:43:42.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:42.066 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:43:42Z" 2024-04-20T01:43:42.066 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:43:42.067 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026910", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:00.551471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024649", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:27:17.869908Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020878", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:30.495154Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:43:42Z" } 2024-04-20T01:43:42.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:43:42.068 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:43:42.068 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:43:42.068 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027190 target=00000000000000000000:00000000000000026910, shard=2 master=00000000000000000000:00000000000000024790 target=00000000000000000000:00000000000000024649, shard=3 master=00000000000000000000:00000000000000021018 target=00000000000000000000:00000000000000020878 2024-04-20T01:43:47.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:47.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:47.070 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:47.131 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:47.131 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:47.198 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:43:47.199 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:43:47.199 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:43:47.199 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:43:47.199 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:43:47.199 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:43:47.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:47.199 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:43:47.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:47.199 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:43:47.199 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:47.200 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:47.200 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:43:47.200 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:47.200 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:43:47.200 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:47.200 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:43:47.200 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:47.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:47.200 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:47.200 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:43:47.200 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:47.201 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:47.201 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027190", 2024-04-20T01:43:47.201 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:47.201 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:43:47.201 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:47.201 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.865175Z" 2024-04-20T01:43:47.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:47.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:47.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:47.201 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:43:47.201 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:47.201 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:47.202 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:43:47.202 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:47.202 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:47.202 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:47.202 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:43:47.202 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:47.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:47.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:47.202 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:43:47.202 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:47.202 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:43:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:43:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:43:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:43:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:43:47.203 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:43:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:43:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:47.204 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:43:47Z" 2024-04-20T01:43:47.204 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:43:47.205 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027190", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:41:41.865175Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:43:47Z" } 2024-04-20T01:43:47.205 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027190', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.865175Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:43:47.205 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:43:47.205 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027190', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:43:47.205 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:43:47.206 DEBUG:boto:path=/aktlji-86/ 2024-04-20T01:43:47.206 DEBUG:boto:auth_path=/aktlji-86/ 2024-04-20T01:43:47.206 DEBUG:boto:Method: HEAD 2024-04-20T01:43:47.206 DEBUG:boto:Path: /aktlji-86/ 2024-04-20T01:43:47.206 DEBUG:boto:Data: 2024-04-20T01:43:47.206 DEBUG:boto:Headers: {} 2024-04-20T01:43:47.206 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:43:47.206 DEBUG:boto:Port: 8000 2024-04-20T01:43:47.206 DEBUG:boto:Params: {} 2024-04-20T01:43:47.207 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:43:47.207 DEBUG:boto:Token: None 2024-04-20T01:43:47.207 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:43:47 GMT /aktlji-86/ 2024-04-20T01:43:47.207 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:J7FBqrwi85OYkLb/KkZttWy7E/E= 2024-04-20T01:43:47.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:J7FBqrwi85OYkLb/KkZttWy7E/E=', 'Content-Length': '0'} 2024-04-20T01:43:47.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3ac74574f657f58-0066231dd3-23f7-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:43:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:47.214 DEBUG:boto:path=/aktlji-86/obj1 2024-04-20T01:43:47.214 DEBUG:boto:auth_path=/aktlji-86/obj1 2024-04-20T01:43:47.215 DEBUG:boto:Method: HEAD 2024-04-20T01:43:47.215 DEBUG:boto:Path: /aktlji-86/obj1 2024-04-20T01:43:47.215 DEBUG:boto:Data: 2024-04-20T01:43:47.215 DEBUG:boto:Headers: {} 2024-04-20T01:43:47.215 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:43:47.215 DEBUG:boto:Port: 8000 2024-04-20T01:43:47.215 DEBUG:boto:Params: {} 2024-04-20T01:43:47.215 DEBUG:boto:Token: None 2024-04-20T01:43:47.215 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:43:47 GMT /aktlji-86/obj1 2024-04-20T01:43:47.215 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:m1W16dJiTCXnfWivsYzIjh0lHmA= 2024-04-20T01:43:47.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m1W16dJiTCXnfWivsYzIjh0lHmA=', 'Content-Length': '0'} 2024-04-20T01:43:47.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-86:73f46f02-c7ab-4426-a267-2273af911da5.6914.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd22994aa07523ff-0066231dd3-23f7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:43:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:47.218 DEBUG:boto:path=/aktlji-86/obj1 2024-04-20T01:43:47.218 DEBUG:boto:auth_path=/aktlji-86/obj1 2024-04-20T01:43:47.218 DEBUG:boto:Method: GET 2024-04-20T01:43:47.218 DEBUG:boto:Path: /aktlji-86/obj1 2024-04-20T01:43:47.219 DEBUG:boto:Data: 2024-04-20T01:43:47.219 DEBUG:boto:Headers: {} 2024-04-20T01:43:47.219 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:43:47.219 DEBUG:boto:Port: 8000 2024-04-20T01:43:47.219 DEBUG:boto:Params: {} 2024-04-20T01:43:47.219 DEBUG:boto:Token: None 2024-04-20T01:43:47.219 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:43:47 GMT /aktlji-86/obj1 2024-04-20T01:43:47.219 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zu6XqNXbN8QMiz/IKMK4H/akmxk= 2024-04-20T01:43:47.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zu6XqNXbN8QMiz/IKMK4H/akmxk=', 'Content-Length': '0'} 2024-04-20T01:43:47.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-86:73f46f02-c7ab-4426-a267-2273af911da5.6914.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da595dbbd8cfbccd-0066231dd3-23f7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:43:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:47.222 DEBUG:boto:path=/aktlji-86/obj2 2024-04-20T01:43:47.222 DEBUG:boto:auth_path=/aktlji-86/obj2 2024-04-20T01:43:47.222 DEBUG:boto:Method: HEAD 2024-04-20T01:43:47.222 DEBUG:boto:Path: /aktlji-86/obj2 2024-04-20T01:43:47.222 DEBUG:boto:Data: 2024-04-20T01:43:47.222 DEBUG:boto:Headers: {} 2024-04-20T01:43:47.222 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:43:47.222 DEBUG:boto:Port: 8000 2024-04-20T01:43:47.223 DEBUG:boto:Params: {} 2024-04-20T01:43:47.223 DEBUG:boto:Token: None 2024-04-20T01:43:47.223 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:43:47 GMT /aktlji-86/obj2 2024-04-20T01:43:47.223 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:s9dApukXi8OWvgyBpMGOwAlO4lA= 2024-04-20T01:43:47.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:s9dApukXi8OWvgyBpMGOwAlO4lA=', 'Content-Length': '0'} 2024-04-20T01:43:47.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-86:73f46f02-c7ab-4426-a267-2273af911da5.6914.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051c2c1cd4b79885d-0066231dd3-23f7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:43:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:47.225 DEBUG:boto:path=/aktlji-86/obj2 2024-04-20T01:43:47.225 DEBUG:boto:auth_path=/aktlji-86/obj2 2024-04-20T01:43:47.225 DEBUG:boto:Method: GET 2024-04-20T01:43:47.226 DEBUG:boto:Path: /aktlji-86/obj2 2024-04-20T01:43:47.226 DEBUG:boto:Data: 2024-04-20T01:43:47.226 DEBUG:boto:Headers: {} 2024-04-20T01:43:47.226 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:43:47.226 DEBUG:boto:Port: 8000 2024-04-20T01:43:47.227 DEBUG:boto:Params: {} 2024-04-20T01:43:47.227 DEBUG:boto:Token: None 2024-04-20T01:43:47.227 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:43:47 GMT /aktlji-86/obj2 2024-04-20T01:43:47.227 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Vuf3/Fc1BM8epP75tFifYbwIotk= 2024-04-20T01:43:47.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Vuf3/Fc1BM8epP75tFifYbwIotk=', 'Content-Length': '0'} 2024-04-20T01:43:47.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-86:73f46f02-c7ab-4426-a267-2273af911da5.6914.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000537381503a90eef7-0066231dd3-23f7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:43:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:47.229 DEBUG:boto:path=/aktlji-86/obj3 2024-04-20T01:43:47.230 DEBUG:boto:auth_path=/aktlji-86/obj3 2024-04-20T01:43:47.230 DEBUG:boto:Method: HEAD 2024-04-20T01:43:47.230 DEBUG:boto:Path: /aktlji-86/obj3 2024-04-20T01:43:47.230 DEBUG:boto:Data: 2024-04-20T01:43:47.230 DEBUG:boto:Headers: {} 2024-04-20T01:43:47.230 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:43:47.230 DEBUG:boto:Port: 8000 2024-04-20T01:43:47.230 DEBUG:boto:Params: {} 2024-04-20T01:43:47.231 DEBUG:boto:Token: None 2024-04-20T01:43:47.231 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:43:47 GMT /aktlji-86/obj3 2024-04-20T01:43:47.231 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WpAjHDUlKVMzoS2fIi/CkgPzeug= 2024-04-20T01:43:47.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WpAjHDUlKVMzoS2fIi/CkgPzeug=', 'Content-Length': '0'} 2024-04-20T01:43:47.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-86:73f46f02-c7ab-4426-a267-2273af911da5.6914.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fadccd1180ed864-0066231dd3-23f7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:43:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:47.233 DEBUG:boto:path=/aktlji-86/obj3 2024-04-20T01:43:47.233 DEBUG:boto:auth_path=/aktlji-86/obj3 2024-04-20T01:43:47.233 DEBUG:boto:Method: GET 2024-04-20T01:43:47.234 DEBUG:boto:Path: /aktlji-86/obj3 2024-04-20T01:43:47.234 DEBUG:boto:Data: 2024-04-20T01:43:47.234 DEBUG:boto:Headers: {} 2024-04-20T01:43:47.234 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:43:47.234 DEBUG:boto:Port: 8000 2024-04-20T01:43:47.234 DEBUG:boto:Params: {} 2024-04-20T01:43:47.235 DEBUG:boto:Token: None 2024-04-20T01:43:47.235 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:43:47 GMT /aktlji-86/obj3 2024-04-20T01:43:47.235 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:DWtE7UTiMyE0IU2bGpZRFgkce3w= 2024-04-20T01:43:47.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DWtE7UTiMyE0IU2bGpZRFgkce3w=', 'Content-Length': '0'} 2024-04-20T01:43:47.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-86:73f46f02-c7ab-4426-a267-2273af911da5.6914.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cacd9b1f55d3561e-0066231dd3-23f7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:43:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:47.237 DEBUG:boto:path=/aktlji-87/ 2024-04-20T01:43:47.237 DEBUG:boto:auth_path=/aktlji-87/ 2024-04-20T01:43:47.237 DEBUG:boto:Method: HEAD 2024-04-20T01:43:47.238 DEBUG:boto:Path: /aktlji-87/ 2024-04-20T01:43:47.238 DEBUG:boto:Data: 2024-04-20T01:43:47.238 DEBUG:boto:Headers: {} 2024-04-20T01:43:47.238 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:43:47.238 DEBUG:boto:Port: 8000 2024-04-20T01:43:47.238 DEBUG:boto:Params: {} 2024-04-20T01:43:47.238 DEBUG:boto:Token: None 2024-04-20T01:43:47.239 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:43:47 GMT /aktlji-87/ 2024-04-20T01:43:47.239 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g9vRKGUWYFL4uqsOoh8/r5Cjg8Y= 2024-04-20T01:43:47.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g9vRKGUWYFL4uqsOoh8/r5Cjg8Y=', 'Content-Length': '0'} 2024-04-20T01:43:47.241 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f87b0128c52a21dd-0066231dd3-23f7-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:43:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:47.241 DEBUG:boto:path=/aktlji-87/obj1 2024-04-20T01:43:47.242 DEBUG:boto:auth_path=/aktlji-87/obj1 2024-04-20T01:43:47.242 DEBUG:boto:Method: HEAD 2024-04-20T01:43:47.242 DEBUG:boto:Path: /aktlji-87/obj1 2024-04-20T01:43:47.242 DEBUG:boto:Data: 2024-04-20T01:43:47.242 DEBUG:boto:Headers: {} 2024-04-20T01:43:47.242 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:43:47.242 DEBUG:boto:Port: 8000 2024-04-20T01:43:47.242 DEBUG:boto:Params: {} 2024-04-20T01:43:47.242 DEBUG:boto:Token: None 2024-04-20T01:43:47.243 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:43:47 GMT /aktlji-87/obj1 2024-04-20T01:43:47.243 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HudP1hQ5G0netw0BQjdCUSw+rb8= 2024-04-20T01:43:47.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HudP1hQ5G0netw0BQjdCUSw+rb8=', 'Content-Length': '0'} 2024-04-20T01:43:47.244 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a188f922c9175e5a-0066231dd3-23f7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:43:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:47.244 DEBUG:boto:path=/aktlji-87/obj2 2024-04-20T01:43:47.244 DEBUG:boto:auth_path=/aktlji-87/obj2 2024-04-20T01:43:47.245 DEBUG:boto:Method: HEAD 2024-04-20T01:43:47.245 DEBUG:boto:Path: /aktlji-87/obj2 2024-04-20T01:43:47.245 DEBUG:boto:Data: 2024-04-20T01:43:47.245 DEBUG:boto:Headers: {} 2024-04-20T01:43:47.245 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:43:47.245 DEBUG:boto:Port: 8000 2024-04-20T01:43:47.245 DEBUG:boto:Params: {} 2024-04-20T01:43:47.245 DEBUG:boto:Token: None 2024-04-20T01:43:47.245 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:43:47 GMT /aktlji-87/obj2 2024-04-20T01:43:47.245 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cHnUoA5QbxUGzmFl/MBlAw+Xtd4= 2024-04-20T01:43:47.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cHnUoA5QbxUGzmFl/MBlAw+Xtd4=', 'Content-Length': '0'} 2024-04-20T01:43:47.247 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000c4d3ada6b70fc002-0066231dd3-23f7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:43:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:47.247 DEBUG:boto:path=/aktlji-87/obj3 2024-04-20T01:43:47.247 DEBUG:boto:auth_path=/aktlji-87/obj3 2024-04-20T01:43:47.247 DEBUG:boto:Method: HEAD 2024-04-20T01:43:47.247 DEBUG:boto:Path: /aktlji-87/obj3 2024-04-20T01:43:47.247 DEBUG:boto:Data: 2024-04-20T01:43:47.248 DEBUG:boto:Headers: {} 2024-04-20T01:43:47.248 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:43:47.248 DEBUG:boto:Port: 8000 2024-04-20T01:43:47.248 DEBUG:boto:Params: {} 2024-04-20T01:43:47.248 DEBUG:boto:Token: None 2024-04-20T01:43:47.248 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:43:47 GMT /aktlji-87/obj3 2024-04-20T01:43:47.248 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:g0XW3PAWhi+WRpRdA0PI/I0McYo= 2024-04-20T01:43:47.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:47 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g0XW3PAWhi+WRpRdA0PI/I0McYo=', 'Content-Length': '0'} 2024-04-20T01:43:47.250 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000008b00b8360c7b3bbb-0066231dd3-23f7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:43:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:47.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:47.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:47.251 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:43:47.313 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:47.313 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:47.388 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"enabled"}]} 2024-04-20T01:43:47.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:47.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:47.389 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:43:47.447 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:47.447 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:47.504 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:43:47.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:47.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:47.504 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:43:47.558 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:47.558 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:47.622 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:43:47.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:47.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:47.623 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:43:47.677 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:47.677 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:47.747 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:43:47.748 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:43:47.748 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 15, 2024-04-20T01:43:47.748 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:43:47.748 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T01:43:47.748 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T01:43:47.748 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:43:47.748 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T01:43:47.748 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.748 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T01:43:47.748 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T01:43:47.749 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T01:43:47.749 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T01:43:47.749 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:43:47.749 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:43:47.749 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:43:47.749 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T01:43:47.749 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T01:43:47.749 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:43:47.749 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:43:47.749 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.750 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:43:47.750 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T01:43:47.750 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:43:47.750 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:43:47.750 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:43:47.750 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:43:47.750 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:43:47.750 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:43:47.750 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:43:47.750 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:43:47.751 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T01:43:47.751 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:43:47.751 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:43:47.751 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:43:47.751 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:43:47.751 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:43:47.751 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:43:47.751 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.751 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.751 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:43:47.751 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T01:43:47.751 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:43:47.752 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:43:47.752 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:43:47.752 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:43:47.752 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:43:47.752 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:43:47.752 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:43:47.752 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:43:47.752 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:43:47.752 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:43:47.752 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:43:47.752 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:43:47.752 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:43:47.753 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:43:47.753 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:43:47.753 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.753 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.753 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:43:47.753 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T01:43:47.753 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:43:47.753 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:43:47.753 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:43:47.753 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:43:47.753 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:43:47.753 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:43:47.754 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:43:47.754 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:43:47.754 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:43:47.754 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:43:47.754 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:43:47.754 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:43:47.754 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:43:47.754 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:43:47.754 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:43:47.754 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:43:47.754 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:43:47.755 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T01:43:47.755 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.755 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T01:43:47.755 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T01:43:47.755 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T01:43:47.755 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T01:43:47.755 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:43:47.755 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:43:47.755 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:43:47.755 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T01:43:47.755 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:43:47.755 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T01:43:47.756 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [ 2024-04-20T01:43:47.756 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.756 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-group", 2024-04-20T01:43:47.756 INFO:teuthology.orchestra.run.smithi019.stdout: "data_flow": { 2024-04-20T01:43:47.756 INFO:teuthology.orchestra.run.smithi019.stdout: "symmetrical": [ 2024-04-20T01:43:47.756 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.756 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow1", 2024-04-20T01:43:47.756 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:43:47.756 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:43:47.756 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:43:47.756 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:43:47.757 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.757 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.757 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow", 2024-04-20T01:43:47.757 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:43:47.757 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:43:47.757 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:43:47.757 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:43:47.757 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:43:47.757 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:43:47.757 INFO:teuthology.orchestra.run.smithi019.stdout: "directional": [ 2024-04-20T01:43:47.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.758 INFO:teuthology.orchestra.run.smithi019.stdout: "source_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:43:47.758 INFO:teuthology.orchestra.run.smithi019.stdout: "dest_zone": "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:43:47.758 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:43:47.758 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:43:47.758 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.758 INFO:teuthology.orchestra.run.smithi019.stdout: "pipes": [ 2024-04-20T01:43:47.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.758 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe", 2024-04-20T01:43:47.758 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:43:47.759 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "*", 2024-04-20T01:43:47.759 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:43:47.759 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:43:47.759 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:43:47.759 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:43:47.759 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.759 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:43:47.759 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "*", 2024-04-20T01:43:47.759 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:43:47.759 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:43:47.759 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:43:47.759 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:43:47.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.760 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:43:47.760 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:43:47.760 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:43:47.760 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:43:47.760 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:43:47.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.760 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:43:47.760 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:43:47.760 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:43:47.760 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:43:47.761 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:43:47.761 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:43:47.761 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:43:47.761 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "allowed" 2024-04-20T01:43:47.761 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:43:47.761 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:43:47.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.761 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T01:43:47.761 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:43:47.761 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:43:47.761 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:43:47.762 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:43:47.762 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T01:43:47.762 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.762 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:43:47.762 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T01:43:47.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.762 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.762 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:43:47.762 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T01:43:47.762 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.762 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:47.762 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:43:47.763 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T01:43:47.763 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:43:47.763 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:43:47.763 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.763 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T01:43:47.763 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:43:47.763 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T01:43:47.763 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T01:43:47.763 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:43:47.763 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:43:47.763 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:43:47.763 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:43:47.764 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:43:47.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.764 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T01:43:47.764 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:43:47.764 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:43:47.764 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:43:47.764 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:43:47.764 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:43:47.764 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.764 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T01:43:47.764 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:43:47.764 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:43:47.765 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:43:47.765 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:43:47.765 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:43:47.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.765 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T01:43:47.765 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:43:47.765 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:43:47.765 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:43:47.765 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:43:47.765 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:43:47.765 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.766 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T01:43:47.766 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:43:47.766 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:43:47.766 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:43:47.766 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:43:47.766 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:43:47.766 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:43:47.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:47.767 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:43:47.767 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T01:43:47.767 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:43:47.767 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:43:47.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:47.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:47.768 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:43:47.826 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:47.826 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:47.927 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:43:47.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:47.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:47.928 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket aktlji-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:43:47.999 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:47.999 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:48.178 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-20T01:43:48.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:48.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:48.179 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket aktlji-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:43:48.271 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:48.271 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:48.443 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-20T01:43:48.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:48.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:48.443 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=aktlji-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:43:48.517 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:48.517 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:48.642 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-20T01:43:48.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:48.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:48.646 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket aktlji-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:43:48.713 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:48.713 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:48.819 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-20T01:43:48.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:48.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:43:48.882 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket aktlji-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:43:48.943 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:48.944 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:49.014 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-20T01:43:49.015 DEBUG:boto:path=/aktlji-86/ 2024-04-20T01:43:49.015 DEBUG:boto:auth_path=/aktlji-86/ 2024-04-20T01:43:49.015 DEBUG:boto:Method: HEAD 2024-04-20T01:43:49.015 DEBUG:boto:Path: /aktlji-86/ 2024-04-20T01:43:49.015 DEBUG:boto:Data: 2024-04-20T01:43:49.015 DEBUG:boto:Headers: {} 2024-04-20T01:43:49.015 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:43:49.015 DEBUG:boto:Port: 8000 2024-04-20T01:43:49.015 DEBUG:boto:Params: {} 2024-04-20T01:43:49.016 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:43:49.016 DEBUG:boto:Token: None 2024-04-20T01:43:49.016 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:43:49 GMT /aktlji-86/ 2024-04-20T01:43:49.016 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RhvWkR91NBV6agG0BaYn2Fr/+9k= 2024-04-20T01:43:49.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RhvWkR91NBV6agG0BaYn2Fr/+9k=', 'Content-Length': '0'} 2024-04-20T01:43:49.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097a7cacfb34b773a-0066231dd5-1bc9-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:43:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:49.024 DEBUG:boto:path=/aktlji-86/obj3 2024-04-20T01:43:49.024 DEBUG:boto:auth_path=/aktlji-86/obj3 2024-04-20T01:43:49.024 DEBUG:boto:Method: PUT 2024-04-20T01:43:49.024 DEBUG:boto:Path: /aktlji-86/obj3 2024-04-20T01:43:49.024 DEBUG:boto:Data: 2024-04-20T01:43:49.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:43:49.024 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:43:49.024 DEBUG:boto:Port: 8000 2024-04-20T01:43:49.025 DEBUG:boto:Params: {} 2024-04-20T01:43:49.025 DEBUG:boto:Token: None 2024-04-20T01:43:49.025 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:43:49 GMT /aktlji-86/obj3 2024-04-20T01:43:49.025 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:rhW+Pt4+azZC3s7RPDHO1L1uqpw= 2024-04-20T01:43:49.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:43:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rhW+Pt4+azZC3s7RPDHO1L1uqpw='} 2024-04-20T01:43:49.037 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b998943339f3fcf3-0066231dd5-1bc9-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:43:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:49.037 DEBUG:boto:path=/aktlji-87/ 2024-04-20T01:43:49.037 DEBUG:boto:auth_path=/aktlji-87/ 2024-04-20T01:43:49.037 DEBUG:boto:Method: HEAD 2024-04-20T01:43:49.037 DEBUG:boto:Path: /aktlji-87/ 2024-04-20T01:43:49.037 DEBUG:boto:Data: 2024-04-20T01:43:49.037 DEBUG:boto:Headers: {} 2024-04-20T01:43:49.038 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:43:49.038 DEBUG:boto:Port: 8000 2024-04-20T01:43:49.038 DEBUG:boto:Params: {} 2024-04-20T01:43:49.038 DEBUG:boto:Token: None 2024-04-20T01:43:49.038 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:43:49 GMT /aktlji-87/ 2024-04-20T01:43:49.038 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aPu0miU6Kh9ZBPm1z4gUb1aHaRo= 2024-04-20T01:43:49.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:43:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aPu0miU6Kh9ZBPm1z4gUb1aHaRo=', 'Content-Length': '0'} 2024-04-20T01:43:49.041 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092f9f78665edc2ae-0066231dd5-1bc9-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:43:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:49.042 DEBUG:boto:path=/aktlji-87/obj3 2024-04-20T01:43:49.042 DEBUG:boto:auth_path=/aktlji-87/obj3 2024-04-20T01:43:49.042 DEBUG:boto:Method: PUT 2024-04-20T01:43:49.042 DEBUG:boto:Path: /aktlji-87/obj3 2024-04-20T01:43:49.042 DEBUG:boto:Data: 2024-04-20T01:43:49.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:43:49.042 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:43:49.042 DEBUG:boto:Port: 8000 2024-04-20T01:43:49.042 DEBUG:boto:Params: {} 2024-04-20T01:43:49.043 DEBUG:boto:Token: None 2024-04-20T01:43:49.043 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:43:49 GMT /aktlji-87/obj3 2024-04-20T01:43:49.043 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VO/RTFfThx6euBIelticAyttwiU= 2024-04-20T01:43:49.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:43:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VO/RTFfThx6euBIelticAyttwiU='} 2024-04-20T01:43:49.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039ec9d69d2f37d5a-0066231dd5-1bc9-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:43:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:43:49.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:49.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:49.059 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:49.113 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:49.113 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:49.159 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:43:49.176 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:43:49.176 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:49.176 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:43:49.177 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:43:49.177 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:49.177 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:49.177 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:43:49.177 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:43:49.177 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:49.177 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:49.177 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577428.787391_1643.1", 2024-04-20T01:43:49.177 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:43:48.787391Z" 2024-04-20T01:43:49.177 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:43:49.177 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:43:49.177 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:43:49.178 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:43:49.178 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:43:49.178 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:43:49.179 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:43:49.179 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:43:49.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:49.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:49.179 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:49.239 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:49.239 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:49.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:43:49.304+0000 7f23f3b59780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:43:49.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:43:49.306+0000 7f23f3b59780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:43:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:43:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:43:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:43:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:43:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:43:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:43:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:43:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:43:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:43:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:43:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:43:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:43:49.316 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:43:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:43:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:43:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:49.317 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:43:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:43:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:43:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:43:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:49.318 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:43:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:43:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:43:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:43:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:49.319 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:43:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:43:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:43:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:43:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:43:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:49.320 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:43:49Z" 2024-04-20T01:43:49.321 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:43:49.322 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:43:49Z" } 2024-04-20T01:43:49.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:43:49.322 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:43:49.322 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:43:49.322 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:43:54.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:54.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:54.324 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:54.383 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:54.384 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:43:54.449+0000 7f5d40356780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:43:54.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:43:54.450+0000 7f5d40356780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:43:54.460 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:43:54.461 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:43:54.461 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:43:54.461 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:43:54.461 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:43:54.461 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:43:54.461 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:54.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:54.461 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:43:54.462 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:54.462 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:43:54.462 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:54.462 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:43:54.462 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:43:54.462 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:54.462 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:43:54.462 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:54.462 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:43:54.462 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:54.463 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:54.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:54.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:54.463 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:43:54.463 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:54.463 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:43:54.463 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:43:54.463 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:54.463 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:54.463 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:54.463 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:43:54.464 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:54.464 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:54.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:54.464 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:54.464 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:43:54.464 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:54.464 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:43:54.464 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:43:54.464 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:54.464 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:54.464 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:54.465 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:43:54.465 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:54.465 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:54.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:54.465 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:54.465 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:43:54.465 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:54.465 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:43:54.465 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:43:54.465 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:54.465 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:54.466 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:54.466 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:43:54.466 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:54.466 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:54.466 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:54.466 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:43:54.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:54.466 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:43:54.466 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:43:54.466 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:43:54.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:54.467 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:43:54Z" 2024-04-20T01:43:54.467 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:43:54.467 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:43:54Z" } 2024-04-20T01:43:54.468 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:43:54.468 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:43:54.468 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:43:54.468 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:43:59.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:59.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:43:59.470 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:43:59.529 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:43:59.530 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:43:59.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:43:59.589+0000 7f454c463780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:43:59.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:43:59.590+0000 7f454c463780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:43:59.595 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:43:59.595 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:43:59.595 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:43:59.595 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:43:59.595 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:43:59.595 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:43:59.595 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:59.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:59.595 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:43:59.596 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:59.596 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:43:59.596 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:59.596 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:43:59.596 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:43:59.596 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:59.596 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:43:59.596 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:59.596 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:43:59.596 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:59.597 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:59.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:59.597 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:59.597 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:43:59.597 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:59.597 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:43:59.597 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:43:59.597 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:59.597 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:59.597 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:59.598 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:43:59.598 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:59.598 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:59.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:59.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:59.599 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:43:59.599 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:59.599 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:43:59.599 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:43:59.599 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:59.599 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:59.599 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:59.599 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:43:59.599 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:59.599 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:59.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:59.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:43:59.600 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:43:59.600 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:43:59.600 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:43:59.600 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:43:59.600 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:43:59.600 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:43:59.600 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:43:59.600 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:43:59.600 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:43:59.600 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:59.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:43:59.601 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:43:59.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:59.601 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:43:59.601 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:43:59.601 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:43:59.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:43:59.601 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:43:59Z" 2024-04-20T01:43:59.601 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:43:59.602 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:43:59Z" } 2024-04-20T01:43:59.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:43:59.603 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:43:59.603 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:43:59.603 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:44:04.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:04.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:04.604 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:44:04.663 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:44:04.663 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:44:04.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:04.721+0000 7f4bdc801780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:04.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:04.722+0000 7f4bdc801780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:04.728 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:44:04.729 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:44:04.729 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:44:04.729 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:44:04.729 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:44:04.729 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:44:04.729 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:04.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:04.729 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:44:04.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:04.730 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:44:04.730 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:04.730 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:04.730 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:44:04.730 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:04.730 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:44:04.730 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:04.730 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:44:04.730 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:04.730 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:04.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:04.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:04.731 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:44:04.731 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:04.731 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:04.731 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:44:04.731 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:04.731 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:04.731 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:04.731 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:44:04.731 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:04.732 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:04.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:04.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:04.732 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:44:04.732 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:04.732 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:04.732 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:44:04.732 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:04.732 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:04.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:04.732 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:44:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:44:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:44:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:04.733 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:04.734 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:44:04.734 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:04.734 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:04.734 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:04.734 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:44:04.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:04.734 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:44:04.734 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:44:04.734 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:44:04.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:04.734 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:44:04Z" 2024-04-20T01:44:04.735 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:44:04.735 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:44:04Z" } 2024-04-20T01:44:04.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:44:04.736 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:04.736 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:04.736 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:44:09.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:09.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:09.738 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:44:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:44:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:44:09.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:09.868+0000 7efdd14d2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:09.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:09.869+0000 7efdd14d2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:09.882 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:44:09.882 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:44:09.882 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:44:09.882 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:44:09.882 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:44:09.882 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:44:09.882 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:09.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:09.883 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:44:09.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:09.883 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:44:09.883 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:09.883 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:09.883 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:44:09.883 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:09.883 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:44:09.883 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:09.883 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:44:09.884 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:09.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:09.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:09.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:09.884 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:44:09.884 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:09.884 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:09.884 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:44:09.884 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:09.884 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:09.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:09.884 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:44:09.885 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:09.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:09.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:09.885 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:09.885 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:44:09.885 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:09.885 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:09.885 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:44:09.885 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:09.885 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:09.885 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:09.886 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:44:09.886 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:09.886 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:09.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:09.886 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:09.886 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:44:09.886 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:09.886 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:09.886 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:44:09.886 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:44:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:44:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:44:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:44:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:44:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:09.888 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:44:09Z" 2024-04-20T01:44:09.888 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:44:09.889 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:44:09Z" } 2024-04-20T01:44:09.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:44:09.889 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:09.889 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:09.889 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:44:14.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:14.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:14.891 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:44:14.963 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:44:14.963 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:44:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:15.029+0000 7f9380fc0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:15.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:15.030+0000 7f9380fc0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:15.036 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:44:15.036 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:44:15.036 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:44:15.036 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:44:15.036 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:44:15.036 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:44:15.036 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:15.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:15.036 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:44:15.037 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:15.037 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:44:15.037 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:15.037 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:15.037 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:44:15.037 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:15.037 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:44:15.038 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:15.038 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:44:15.038 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:15.038 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:15.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:15.038 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:15.038 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:44:15.038 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:15.038 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:15.038 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:44:15.039 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:15.039 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:15.039 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:15.039 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:44:15.039 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:15.039 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:15.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:15.039 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:15.039 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:44:15.039 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:15.039 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:15.039 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:44:15.040 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:15.040 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:15.040 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:15.040 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:44:15.040 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:15.040 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:15.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:15.040 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:15.040 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:44:15.040 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:15.040 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:15.041 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:44:15.041 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:15.041 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:15.041 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:15.041 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:44:15.041 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:15.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:15.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:15.041 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:44:15.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:15.042 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:44:15.042 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:44:15.042 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:44:15.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:15.042 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:44:15Z" 2024-04-20T01:44:15.042 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:44:15.043 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:44:15Z" } 2024-04-20T01:44:15.043 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:44:15.043 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:15.043 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:15.044 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:44:20.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:20.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:20.044 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:44:20.104 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:44:20.104 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:44:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:20.166+0000 7f426abdf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:20.167+0000 7f426abdf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:20.173 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:44:20.173 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:44:20.173 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:44:20.173 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:44:20.173 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:44:20.174 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:44:20.174 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:20.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:20.174 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:44:20.174 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:20.174 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:44:20.174 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:20.174 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:20.174 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:44:20.174 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:20.175 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:44:20.175 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:20.175 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:44:20.175 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:20.175 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:20.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:20.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:20.175 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:44:20.175 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:20.175 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:20.175 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:44:20.176 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:20.176 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:20.176 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:20.176 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:44:20.176 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:20.176 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:20.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:20.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:20.176 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:44:20.176 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:20.177 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:20.177 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:44:20.177 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:20.177 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:20.177 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:20.177 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:44:20.177 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:20.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:20.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:20.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:20.178 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:44:20.178 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:20.178 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:20.178 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:44:20.178 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:20.178 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:20.178 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:20.178 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:44:20.178 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:20.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:20.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:20.178 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:44:20.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:20.179 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:44:20.179 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:44:20.179 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:44:20.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:20.179 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:44:20Z" 2024-04-20T01:44:20.179 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:44:20.180 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:44:20Z" } 2024-04-20T01:44:20.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:44:20.180 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:20.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:20.181 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:44:25.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:25.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:25.183 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:44:25.242 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:44:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:44:25.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:25.304+0000 7f1201150780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:25.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:25.305+0000 7f1201150780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:25.310 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:44:25.310 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:44:25.310 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:44:25.310 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:44:25.311 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:44:25.311 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:44:25.311 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:25.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:25.311 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:44:25.311 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:25.311 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:44:25.311 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:25.311 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:25.311 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:44:25.311 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:25.312 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:44:25.312 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:25.312 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:44:25.312 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:25.312 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:25.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:25.312 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:25.312 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:44:25.312 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:25.312 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:25.312 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:44:25.313 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:25.313 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:25.313 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:25.313 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:44:25.313 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:25.313 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:25.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:25.313 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:25.313 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:44:25.313 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:25.313 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:25.313 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:44:25.314 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:25.314 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:25.314 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:25.314 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:44:25.314 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:25.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:25.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:25.314 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:25.314 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:44:25.314 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:25.314 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:25.315 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:44:25.315 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:25.315 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:25.315 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:25.315 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:44:25.315 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:25.315 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:25.315 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:25.315 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:44:25.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:25.315 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:44:25.316 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:44:25.316 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:44:25.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:25.316 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:44:25Z" 2024-04-20T01:44:25.316 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:44:25.317 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:44:25Z" } 2024-04-20T01:44:25.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:44:25.317 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:25.317 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:25.317 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:44:30.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:30.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:30.318 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:44:30.378 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:44:30.378 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:44:30.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:30.433+0000 7f99748c2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:30.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:30.435+0000 7f99748c2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:30.443 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:44:30.443 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:44:30.444 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:44:30.444 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:44:30.444 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:44:30.444 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:44:30.444 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:30.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:30.444 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:44:30.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:30.444 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:44:30.444 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:30.444 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:30.444 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:44:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:44:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:44:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:44:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:30.445 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:44:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:44:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:44:30.446 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:30.447 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:30.447 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:44:30.447 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:30.447 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:30.447 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:30.447 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:44:30.447 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:30.447 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:30.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:30.447 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:30.447 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:44:30.447 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:30.448 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:30.448 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:44:30.448 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:30.448 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:30.448 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:30.448 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:44:30.448 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:30.448 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:30.448 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:30.448 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:44:30.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:30.448 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:44:30.449 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:44:30.449 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:44:30.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:30.449 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:44:30Z" 2024-04-20T01:44:30.449 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:44:30.450 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:44:30Z" } 2024-04-20T01:44:30.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:44:30.450 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:30.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:30.450 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:44:35.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:35.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:35.454 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:44:35.513 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:44:35.513 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:44:35.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:35.580+0000 7f304b47e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:35.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:35.581+0000 7f304b47e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:35.591 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:44:35.591 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:44:35.591 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:44:35.591 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:44:35.591 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:44:35.591 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:44:35.591 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:35.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:35.591 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:44:35.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:35.592 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:44:35.592 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:35.592 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:35.592 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:44:35.592 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:35.592 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:44:35.592 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:35.592 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:44:35.592 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:35.593 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:35.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:35.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:35.593 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:44:35.593 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:35.593 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:35.593 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:44:35.593 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:35.593 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:35.593 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:35.593 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:44:35.593 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:35.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:35.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:35.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:35.594 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:44:35.594 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:35.594 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:35.594 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:44:35.594 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:35.595 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:35.595 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:35.595 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:44:35.595 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:35.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:35.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:35.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:35.595 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:44:35.595 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:35.595 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:35.595 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:44:35.596 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:35.596 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:35.596 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:35.596 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:44:35.596 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:35.596 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:35.596 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:35.596 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:44:35.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:35.596 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:44:35.596 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:44:35.596 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:44:35.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:35.597 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:44:35Z" 2024-04-20T01:44:35.597 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:44:35.598 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:44:35Z" } 2024-04-20T01:44:35.598 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:44:35.598 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:35.598 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:35.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:44:40.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:40.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:40.600 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:44:40.659 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:44:40.659 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:44:40.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:40.728+0000 7f0ce5567780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:40.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:40.729+0000 7f0ce5567780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:40.740 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:44:40.740 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:44:40.740 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:44:40.740 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:44:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:44:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:44:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:44:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:44:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:40.741 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:44:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:44:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:44:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:44:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:40.742 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:44:40.743 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:40.743 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:40.743 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:40.743 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:44:40.743 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:40.743 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:40.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:40.743 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:40.743 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:44:40.743 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:40.744 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:40.744 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:44:40.744 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:40.744 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:40.744 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:40.744 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:44:40.744 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:40.744 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:40.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:40.745 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:40.745 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:44:40.745 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:40.745 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:40.745 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:44:40.745 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:40.745 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:40.745 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:40.745 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:44:40.746 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:40.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:40.746 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:40.746 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:44:40.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:40.746 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:44:40.746 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:44:40.746 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:44:40.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:40.747 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:44:40Z" 2024-04-20T01:44:40.747 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:44:40.748 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:44:40Z" } 2024-04-20T01:44:40.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:44:40.748 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:40.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:40.748 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:44:45.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:45.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:45.750 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:44:45.810 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:44:45.810 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:44:45.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:45.872+0000 7f9727309780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:45.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:45.873+0000 7f9727309780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:45.879 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:44:45.879 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:44:45.880 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:44:45.880 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:44:45.880 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:44:45.880 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:44:45.880 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:45.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:45.880 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:44:45.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:45.880 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:44:45.880 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:45.880 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:45.881 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:44:45.881 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:45.881 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:44:45.881 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:45.881 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:44:45.881 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:45.881 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:45.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:45.881 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:45.881 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:44:45.881 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:45.881 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:45.882 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:44:45.882 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:45.882 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:45.882 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:45.883 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:44:45.883 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:45.883 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:45.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:45.883 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:45.883 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:44:45.883 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:45.883 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:45.883 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:44:45.883 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:45.883 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:44:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:44:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:44:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:44:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:44:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:44:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:44:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:44:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:45.886 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:44:45Z" 2024-04-20T01:44:45.886 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:44:45.887 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:44:45Z" } 2024-04-20T01:44:45.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:44:45.887 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:45.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:45.887 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:44:50.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:50.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:50.888 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:44:50.947 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:44:50.947 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:44:50.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:51.011+0000 7fc9c9e77780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:50.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:51.012+0000 7fc9c9e77780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:51.018 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:44:51.018 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:44:51.018 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:44:51.018 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:44:51.018 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:44:51.018 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:44:51.018 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:51.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:51.018 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:44:51.019 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:51.019 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:44:51.019 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:51.019 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:51.019 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:44:51.019 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:51.019 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:44:51.019 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:51.019 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:44:51.019 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:51.019 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:51.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:51.020 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:51.020 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:44:51.020 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:51.020 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:51.020 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:44:51.020 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:51.020 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:51.020 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:51.020 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:44:51.020 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:51.021 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:51.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:51.021 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:51.021 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:44:51.021 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:51.021 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:51.021 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:44:51.021 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:51.021 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:51.021 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:51.021 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:44:51.021 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:51.022 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:51.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:51.022 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:51.022 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:44:51.022 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:51.022 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:51.022 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:44:51.022 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:51.022 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:51.022 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:51.023 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:44:51.023 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:51.023 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:51.023 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:51.023 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:44:51.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:51.023 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:44:51.023 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:44:51.023 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:44:51.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:51.023 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:44:51Z" 2024-04-20T01:44:51.024 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:44:51.024 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:44:51Z" } 2024-04-20T01:44:51.025 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:44:51.025 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:51.025 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:51.025 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:44:56.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:56.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:44:56.027 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:44:56.088 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:44:56.088 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:44:56.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:56.143+0000 7f23613bd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:56.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:44:56.145+0000 7f23613bd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:44:56.149 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:44:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:44:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:44:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:44:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:44:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:44:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:44:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:44:56.151 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:56.151 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:56.151 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:44:56.151 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:56.151 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:44:56.151 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:56.151 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:44:56.151 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:56.151 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:56.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:56.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:56.151 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:44:56.152 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:56.152 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:56.152 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:44:56.152 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:56.152 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:56.152 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:56.152 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:44:56.152 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:56.152 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:56.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:56.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:56.152 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:44:56.153 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:56.153 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:56.153 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:44:56.153 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:56.153 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:56.153 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:56.153 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:44:56.153 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:56.153 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:56.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:56.154 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:44:56.154 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:44:56.154 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:44:56.154 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:44:56.154 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:44:56.154 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:44:56.154 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:44:56.154 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:44:56.154 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:44:56.154 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:44:56.154 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:56.155 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:44:56.155 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:44:56.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:56.155 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:44:56.155 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:44:56.155 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:44:56.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:44:56.155 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:44:56Z" 2024-04-20T01:44:56.155 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:44:56.156 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:44:56Z" } 2024-04-20T01:44:56.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:44:56.157 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:56.157 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:44:56.157 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:45:01.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:01.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:01.158 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:01.218 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:01.218 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:01.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:01.289+0000 7fd46da6c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:01.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:01.290+0000 7fd46da6c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:01.297 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:01.298 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:01.298 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:01.298 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:01.298 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:01.298 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:01.298 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:01.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:01.298 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:01.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:01.298 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:01.299 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:01.299 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:01.299 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:01.299 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:01.299 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:01.299 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:01.299 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:01.299 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:01.299 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:01.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:01.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:01.300 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:01.300 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:01.300 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:01.300 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:01.300 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:01.300 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:01.300 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:01.300 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:01.300 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:01.300 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:01.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:01.301 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:01.301 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:01.301 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:01.301 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:01.301 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:45:01.301 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:01.301 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:01.301 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:01.301 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:45:01.301 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:01.301 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:01.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:01.302 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:01.302 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:01.302 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:01.302 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:01.302 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:01.302 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:01.303 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:01.303 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:01.303 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:01.303 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:01.303 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:01.303 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:01.303 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:01.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:01.303 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:01.303 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:01.303 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:01.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:01.304 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:01Z" 2024-04-20T01:45:01.304 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:01.305 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:01Z" } 2024-04-20T01:45:01.305 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:01.305 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:01.305 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:01.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:45:06.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:06.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:06.307 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:06.366 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:06.366 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:06.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:06.429+0000 7f82492e0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:06.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:06.430+0000 7f82492e0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:06.438 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:45:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:45:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:06Z" 2024-04-20T01:45:06.444 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:06.444 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:06Z" } 2024-04-20T01:45:06.445 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:06.445 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:06.445 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:06.445 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:45:11.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:11.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:11.446 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:11.506 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:11.506 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:11.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:11.565+0000 7f446ac3b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:11.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:11.566+0000 7f446ac3b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:11.572 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:11.572 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:11.572 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:11.573 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:11.573 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:11.573 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:11.573 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:11.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:11.573 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:11.573 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:11.573 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:11.573 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:11.573 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:11.573 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:11.574 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:11.574 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:11.574 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:11.574 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:11.574 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:11.574 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:11.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:11.574 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:11.574 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:11.574 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:11.574 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:11.575 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:11.575 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:11.575 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:11.575 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:11.575 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:11.575 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:11.575 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:11.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:11.575 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:11.575 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:11.575 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:11.575 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:11.576 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:45:11.576 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:11.576 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:11.576 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:11.576 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:45:11.576 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:11.576 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:11.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:11.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:11.576 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:11.576 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:11.576 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:11.577 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:11.577 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:11.577 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:11.577 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:11.577 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:11.577 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:11.577 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:11.577 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:11.577 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:11.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:11.577 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:11.577 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:11.578 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:11.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:11.578 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:11Z" 2024-04-20T01:45:11.578 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:11.579 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:11Z" } 2024-04-20T01:45:11.579 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:11.579 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:11.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:11.579 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:45:16.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:16.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:16.580 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:16.640 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:16.640 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:16.706+0000 7fed61d7e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:16.707+0000 7fed61d7e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:16.715 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:16.716 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:16.716 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:16.716 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:16.716 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:16.716 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:16.716 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:16.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:16.716 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:16.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:16.716 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:16.716 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:16.716 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:16.717 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:16.717 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:16.717 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:16.717 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:16.717 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:16.717 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:16.717 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:16.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:16.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:16.717 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:16.717 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:16.718 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:16.718 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:16.718 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:16.718 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:16.718 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:16.718 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:16.718 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:16.718 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:16.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:16.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:16.718 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:16.718 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:16.719 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:16.719 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:45:16.719 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:16.719 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:16.719 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:16.719 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:45:16.719 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:16.719 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:16.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:16.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:16.719 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:16.720 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:16.720 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:16.720 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:16.720 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:16.720 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:16.720 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:16.720 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:16.720 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:16.720 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:16.720 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:16.720 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:16.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:16.721 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:16.721 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:16.721 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:16.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:16.721 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:16Z" 2024-04-20T01:45:16.721 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:16.722 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:16Z" } 2024-04-20T01:45:16.722 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:16.722 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:16.722 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:16.722 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:45:21.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:21.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:21.724 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:21.784 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:21.785 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:21.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:21.849+0000 7fd5709f4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:21.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:21.850+0000 7fd5709f4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:21.859 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:21.860 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:21.860 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:21.860 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:21.860 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:21.860 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:21.860 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:21.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:21.860 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:21.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:21.860 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:21.860 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:21.861 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:21.861 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:21.861 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:21.861 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:21.861 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:21.861 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:21.861 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:21.861 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:21.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:21.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:21.862 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:21.862 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:21.862 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:21.862 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:21.862 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:21.862 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:21.862 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:21.862 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:21.862 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:21.862 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:21.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:21.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:21.863 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:21.863 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:21.863 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:21.863 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:45:21.863 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:21.863 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:21.863 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:21.863 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:45:21.863 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:21.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:21.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:21.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:21.864 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:21.864 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:21.864 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:21.864 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:21.864 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:21.864 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:21.864 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:21.864 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:21.865 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:21.865 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:21.865 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:21.865 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:21.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:21.865 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:21.865 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:21.865 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:21.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:21.865 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:21Z" 2024-04-20T01:45:21.865 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:21.866 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:21Z" } 2024-04-20T01:45:21.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:21.867 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:21.867 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:21.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:45:26.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:26.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:26.869 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:26.929 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:26.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:26.999+0000 7f2bc62e7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:26.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:27.000+0000 7f2bc62e7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:27.005 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:27.005 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:27.006 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:27.007 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:45:27.008 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:45:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:27.009 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:27.010 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:27.011 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:27Z" 2024-04-20T01:45:27.011 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:27.012 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:27Z" } 2024-04-20T01:45:27.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:27.012 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:27.012 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:27.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:45:32.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:32.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:32.013 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:32.073 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:32.073 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:32.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:32.143+0000 7f208617b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:32.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:32.144+0000 7f208617b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:32.156 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:32.156 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:32.156 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:32.156 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:32.156 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:32.156 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:32.156 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:32.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:32.157 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:32.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:32.157 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:32.157 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:32.157 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:32.157 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:32.157 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:32.157 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:32.157 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:32.157 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:32.157 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:32.158 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:32.159 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:32.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:32.159 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:32.159 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:32.159 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:32.159 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:32.159 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:45:32.159 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:32.159 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:32.159 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:32.159 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:45:32.160 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:32.160 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:32.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:32.160 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:32.160 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:32.160 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:32.160 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:32.160 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:32.160 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:32.160 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:32.160 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:32.160 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:32.161 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:32Z" 2024-04-20T01:45:32.161 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:32.162 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:32Z" } 2024-04-20T01:45:32.162 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:32.163 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:32.163 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:32.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:45:37.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:37.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:37.164 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:37.223 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:37.224 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:37.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:37.297+0000 7f46370ab780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:37.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:37.298+0000 7f46370ab780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:37.304 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:37.304 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:37.304 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:37.304 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:37.304 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:37.304 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:37.304 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:37.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:37.304 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:37.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:37.305 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:37.305 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:37.305 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:37.305 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:37.305 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:37.305 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:37.305 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:37.305 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:37.305 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:37.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:37.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:37.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:37.306 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:37.306 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:37.306 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:37.306 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:37.306 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:37.306 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:37.306 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:37.306 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:37.306 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:37.307 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:37.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:37.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:37.308 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:37.308 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:37.309 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:37.309 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:45:37.309 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:37.309 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:37.309 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:37.309 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:45:37.309 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:37.309 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:37.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:37.309 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:37.309 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:37.310 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:37.310 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:37.310 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:37.310 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:37.310 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:37.310 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:37.310 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:37.310 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:37.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:37.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:37.310 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:37.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:37.311 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:37.311 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:37.311 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:37.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:37.311 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:37Z" 2024-04-20T01:45:37.311 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:37.312 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:37Z" } 2024-04-20T01:45:37.312 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:37.312 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:37.312 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:37.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:45:42.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:42.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:42.318 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:42.377 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:42.377 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:42.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:42.432+0000 7fc00b71b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:42.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:42.433+0000 7fc00b71b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:42.441 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:42.441 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:42.441 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:42.441 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:42.441 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:42.441 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:42.441 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:42.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:42.441 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:42.442 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:42.442 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:42.442 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:42.443 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:42.443 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:42.443 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:42.443 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:42.443 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:42.443 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:42.443 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:42.443 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:42.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:42.443 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:42.443 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:42.444 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:42.444 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:42.444 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:42.444 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:42.444 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:42.444 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:42.444 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:42.444 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:42.444 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:42.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:42.444 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:42.444 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:42.445 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:42.445 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:42.445 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:45:42.445 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:42.445 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:42.445 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:42.445 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:45:42.445 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:42.445 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:42.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:42.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:42.446 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:42.446 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:42.446 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:42.446 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:42.446 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:42.446 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:42.446 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:42.446 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:42.446 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:42.446 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:42.446 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:42.447 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:42.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:42.447 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:42.447 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:42.447 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:42.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:42.447 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:42Z" 2024-04-20T01:45:42.447 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:42.448 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:42Z" } 2024-04-20T01:45:42.448 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:42.448 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:42.448 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:42.449 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:45:47.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:47.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:47.450 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:47.509 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:47.510 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:47.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:47.579+0000 7fe1581c1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:47.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:47.580+0000 7fe1581c1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:47.590 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:47.591 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:47.591 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:47.591 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:47.591 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:47.591 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:47.591 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:47.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:47.591 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:47.591 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:47.591 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:47.591 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:47.592 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:47.592 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:47.592 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:47.592 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:47.592 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:47.592 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:47.592 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:47.592 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:47.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:47.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:47.592 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:47.593 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:47.593 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:47.593 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:47.593 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:47.593 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:47.593 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:47.593 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:47.593 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:47.593 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:47.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:47.593 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:47.593 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:47.594 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:47.594 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:47.594 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577294.954723_1593.1", 2024-04-20T01:45:47.594 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:47.594 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:47.594 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:47.594 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:34.954723Z", 2024-04-20T01:45:47.594 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:47.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:47.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:47.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:47.595 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:47.595 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:47.595 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:47.595 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:47.595 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:47.595 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:47.595 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:47.595 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:47.595 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:47.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:47.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:47.596 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:47.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:47.596 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:47.596 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:47.596 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:47.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:47.596 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:47Z" 2024-04-20T01:45:47.596 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:47.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577294.954723_1593.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:34.954723Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:47Z" } 2024-04-20T01:45:47.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577294.954723_1593.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:34.954723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:47.597 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:47.598 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577294.954723_1593.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:47.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577428.787391_1643.1 target=1_1713577294.954723_1593.1 2024-04-20T01:45:52.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:52.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:52.599 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:52.658 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:52.658 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:52.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:52.723+0000 7ff4e2d2c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:52.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:52.724+0000 7ff4e2d2c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:52.728 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:52.729 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:52.729 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:52.729 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:52.729 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:52.729 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:52.729 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:52.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:52.729 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:52.729 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:52.730 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:52.730 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:52.730 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:52.730 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:52.730 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:52.730 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:52.730 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:52.730 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:52.730 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:52.730 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:52.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:52.731 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:52.731 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:52.731 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:52.731 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:52.731 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:52.731 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:52.731 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:52.731 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:52.731 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:52.732 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:52.732 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:52.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:52.732 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:52.732 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:52.732 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:52.732 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:52.732 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577428.787391_1643.1", 2024-04-20T01:45:52.732 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:52.732 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:52.732 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:52.732 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:48.787391Z", 2024-04-20T01:45:52.733 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:52.733 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:52.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:52.733 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:52.733 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:52.733 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:52.733 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:52.733 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:52.733 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:52.733 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:52.733 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:52.734 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:52.734 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:52.734 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:52.734 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:52.734 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:52.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:52.734 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:52.734 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:52.734 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:52.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:52.734 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:52Z" 2024-04-20T01:45:52.735 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:52.735 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577428.787391_1643.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:43:48.787391Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:52Z" } 2024-04-20T01:45:52.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577428.787391_1643.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:43:48.787391Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:52.736 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:52.736 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:52.736 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:45:52.736 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:45:52.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:52.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:52.737 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:52.797 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:52.797 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:52.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:45:52.846+0000 7ff90caa9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:52.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:45:52.848+0000 7ff90caa9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:52.853 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:45:52.853 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:45:52.853 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:45:52.853 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:45:52.853 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:45:52.853 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:52.853 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:45:52.853 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:45:52.853 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:45:52.854 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:45:52.854 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:45:52.854 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:45:52.854 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:45:52.854 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:52.854 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:45:52.854 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:45:52.854 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:45:52.854 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:52.854 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:45:52.855 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:45:52.855 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:45:52.855 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:45:52.855 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:45:52.855 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:45:52.855 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:45:52.855 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:45:52.855 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:45:52.855 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:45:52.855 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:45:52.856 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:45:52.856 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:45:52.856 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:45:52.856 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:45:52.856 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:45:52.856 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:45:52.856 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:45:52.856 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:45:52.856 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577428.787391_1643.1", 2024-04-20T01:45:52.856 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:45:52.856 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:45:52.856 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:45:52.857 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:43:48.787391Z", 2024-04-20T01:45:52.857 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:45:52.857 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:45:52.857 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:45:52.857 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:45:52.857 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:45:52.857 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:45:52.857 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:45:52.857 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:52.857 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:45:52.858 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:45:52.858 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:45:52.858 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:52.858 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:45:52.858 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:45:52.858 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:45:52.858 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:45:52.858 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:45:52.858 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:45:52.858 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:45:52.859 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:45:52.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:45:52.859 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:45:52Z" 2024-04-20T01:45:52.859 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:45:52.860 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577428.787391_1643.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:43:48.787391Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:52Z" } 2024-04-20T01:45:52.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577428.787391_1643.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:43:48.787391Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:52.860 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:52.860 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:52.860 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:45:52.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:52.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:52.861 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:52.914 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:52.992 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:45:52.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:45:52.992 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:45:52.992 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:45:52.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:45:52.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:45:52.992 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:45:52.992 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:45:52.993 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:45:52.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:45:52.993 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:45:52.993 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:45:52.993 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:45:52.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:45:52.993 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:45:52.993 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:45:52.993 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:45:52.993 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:45:52.995 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:45:52.995 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:45:52.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:52.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:52.995 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:53.058 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:53.058 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:53.121 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:53.121 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:53.121 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:53.121 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:53.121 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:53.122 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:45:53.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:53.122 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:53.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:53.122 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:53.122 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:53.122 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:45:53.122 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:45:53.122 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:45:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:45:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:45:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:45:53.123 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:45:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:45:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:45:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:45:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:45:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:53.125 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:45:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:45:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:45:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:45:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:45:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:53.127 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:53Z" 2024-04-20T01:45:53.127 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:53.128 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:45:53Z" } 2024-04-20T01:45:53.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:45:53.129 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:45:53.129 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:45:53.129 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:45:53.129 DEBUG:boto:path=/aktlji-86/ 2024-04-20T01:45:53.129 DEBUG:boto:auth_path=/aktlji-86/ 2024-04-20T01:45:53.130 DEBUG:boto:Method: HEAD 2024-04-20T01:45:53.130 DEBUG:boto:Path: /aktlji-86/ 2024-04-20T01:45:53.130 DEBUG:boto:Data: 2024-04-20T01:45:53.130 DEBUG:boto:Headers: {} 2024-04-20T01:45:53.130 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:45:53.130 DEBUG:boto:Port: 8000 2024-04-20T01:45:53.130 DEBUG:boto:Params: {} 2024-04-20T01:45:53.131 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:45:53.131 DEBUG:boto:Token: None 2024-04-20T01:45:53.131 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:45:53 GMT /aktlji-86/ 2024-04-20T01:45:53.132 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Q5/urw3IuWRI2nEe1ho2BppI6A4= 2024-04-20T01:45:53.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:45:53 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q5/urw3IuWRI2nEe1ho2BppI6A4=', 'Content-Length': '0'} 2024-04-20T01:45:53.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005dacba84f7616020-0066231e51-24cb-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:45:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:45:53.139 DEBUG:boto:path=/aktlji-86/obj3 2024-04-20T01:45:53.139 DEBUG:boto:auth_path=/aktlji-86/obj3 2024-04-20T01:45:53.139 DEBUG:boto:Method: HEAD 2024-04-20T01:45:53.139 DEBUG:boto:Path: /aktlji-86/obj3 2024-04-20T01:45:53.139 DEBUG:boto:Data: 2024-04-20T01:45:53.139 DEBUG:boto:Headers: {} 2024-04-20T01:45:53.139 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:45:53.139 DEBUG:boto:Port: 8000 2024-04-20T01:45:53.139 DEBUG:boto:Params: {} 2024-04-20T01:45:53.140 DEBUG:boto:Token: None 2024-04-20T01:45:53.140 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:45:53 GMT /aktlji-86/obj3 2024-04-20T01:45:53.140 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zjGOL0XG9yP3W5vqMRSnn520cF8= 2024-04-20T01:45:53.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:45:53 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zjGOL0XG9yP3W5vqMRSnn520cF8=', 'Content-Length': '0'} 2024-04-20T01:45:53.142 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-86:73f46f02-c7ab-4426-a267-2273af911da5.6914.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffd8c74762a709bc-0066231e51-24cb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:45:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:45:53.143 DEBUG:boto:path=/aktlji-86/obj3 2024-04-20T01:45:53.143 DEBUG:boto:auth_path=/aktlji-86/obj3 2024-04-20T01:45:53.143 DEBUG:boto:Method: GET 2024-04-20T01:45:53.143 DEBUG:boto:Path: /aktlji-86/obj3 2024-04-20T01:45:53.144 DEBUG:boto:Data: 2024-04-20T01:45:53.144 DEBUG:boto:Headers: {} 2024-04-20T01:45:53.144 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:45:53.144 DEBUG:boto:Port: 8000 2024-04-20T01:45:53.144 DEBUG:boto:Params: {} 2024-04-20T01:45:53.145 DEBUG:boto:Token: None 2024-04-20T01:45:53.145 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:45:53 GMT /aktlji-86/obj3 2024-04-20T01:45:53.145 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:D3OR81ppIt/60tCGVS5Ki1rTaxM= 2024-04-20T01:45:53.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:45:53 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:D3OR81ppIt/60tCGVS5Ki1rTaxM=', 'Content-Length': '0'} 2024-04-20T01:45:53.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:41:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-86:73f46f02-c7ab-4426-a267-2273af911da5.6914.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034f5886373c510e1-0066231e51-24cb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:45:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:45:53.147 DEBUG:boto:path=/aktlji-87/ 2024-04-20T01:45:53.148 DEBUG:boto:auth_path=/aktlji-87/ 2024-04-20T01:45:53.148 DEBUG:boto:Method: HEAD 2024-04-20T01:45:53.148 DEBUG:boto:Path: /aktlji-87/ 2024-04-20T01:45:53.148 DEBUG:boto:Data: 2024-04-20T01:45:53.148 DEBUG:boto:Headers: {} 2024-04-20T01:45:53.148 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:45:53.149 DEBUG:boto:Port: 8000 2024-04-20T01:45:53.149 DEBUG:boto:Params: {} 2024-04-20T01:45:53.149 DEBUG:boto:Token: None 2024-04-20T01:45:53.149 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:45:53 GMT /aktlji-87/ 2024-04-20T01:45:53.149 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RWUPvmM6V2bUkTFThOqy9IOpxOI= 2024-04-20T01:45:53.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:45:53 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RWUPvmM6V2bUkTFThOqy9IOpxOI=', 'Content-Length': '0'} 2024-04-20T01:45:53.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f30f77552c8cc905-0066231e51-24cb-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:45:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:45:53.152 DEBUG:boto:path=/aktlji-87/obj3 2024-04-20T01:45:53.152 DEBUG:boto:auth_path=/aktlji-87/obj3 2024-04-20T01:45:53.152 DEBUG:boto:Method: HEAD 2024-04-20T01:45:53.153 DEBUG:boto:Path: /aktlji-87/obj3 2024-04-20T01:45:53.153 DEBUG:boto:Data: 2024-04-20T01:45:53.153 DEBUG:boto:Headers: {} 2024-04-20T01:45:53.153 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:45:53.153 DEBUG:boto:Port: 8000 2024-04-20T01:45:53.153 DEBUG:boto:Params: {} 2024-04-20T01:45:53.153 DEBUG:boto:Token: None 2024-04-20T01:45:53.153 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:45:53 GMT /aktlji-87/obj3 2024-04-20T01:45:53.153 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Z6rqTt5HOE9id2NLtD92Na9qJiU= 2024-04-20T01:45:53.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:45:53 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z6rqTt5HOE9id2NLtD92Na9qJiU=', 'Content-Length': '0'} 2024-04-20T01:45:53.155 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000007e4c7751232afbda-0066231e51-24cb-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:45:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:45:53.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:45:53.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'aktlji-86', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:45:53.155 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket aktlji-86 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:45:53.215 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:53.215 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:53.279 INFO:teuthology.orchestra.run.smithi019.stdout:{"sync_policy":{"groups":[]}} 2024-04-20T01:45:53.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:45:53.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:45:53.280 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:45:53.334 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:53.334 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:53.402 INFO:teuthology.orchestra.run.smithi019.stdout:{"sync_policy":{"groups":[]}} 2024-04-20T01:45:53.403 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ok 2024-04-20T01:45:53.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:53.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:53.404 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:53.458 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:53.458 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:53.495 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:45:53.512 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:45:53.512 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:45:53.512 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:53.512 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:45:53.512 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:45:53.512 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:45:53.512 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:45:53.512 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:45:53.513 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:45:53.513 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:45:53.513 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577553.275530_1647.1", 2024-04-20T01:45:53.513 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:45:53.275530Z" 2024-04-20T01:45:53.513 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:45:53.513 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:45:53.513 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:53.513 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:45:53.513 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:45:53.513 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:45:53.514 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577553.275530_1647.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:53.514 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:45:53.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:53.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:53.515 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:53.575 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:53.575 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:53.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:53.639+0000 7f0e081a3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:53.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:53.641+0000 7f0e081a3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:53.646 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:53.646 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:53.646 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:53.646 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:53.647 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:53.648 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:53.648 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:53.648 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:53.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:53.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:53.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:53.648 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:53.648 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:53.648 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:53.648 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:53.648 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:53.649 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:53.649 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:53.649 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:53.649 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:53.649 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:53.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:53.649 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:53.649 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:53.649 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:53.649 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:53.649 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577428.787391_1643.1", 2024-04-20T01:45:53.650 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:53.650 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:53.650 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:53.650 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:48.787391Z", 2024-04-20T01:45:53.650 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:53.650 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:53.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:53.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:53.650 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:53.650 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:53.650 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:53.650 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:53.651 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:53.651 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:53.651 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:53.651 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:53.651 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:53.651 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:53.651 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:53.651 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:53.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:53.651 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:53.652 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:53.652 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:53.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:53.652 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:53Z" 2024-04-20T01:45:53.652 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:53.653 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577428.787391_1643.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:43:48.787391Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:53Z" } 2024-04-20T01:45:53.653 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577428.787391_1643.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:43:48.787391Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:53.653 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577553.275530_1647.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:53.653 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:53.653 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577553.275530_1647.1 target=1_1713577428.787391_1643.1 2024-04-20T01:45:58.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:58.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:45:58.655 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:45:58.714 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:45:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:45:58.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:58.773+0000 7f64b87e0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:58.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:45:58.774+0000 7f64b87e0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:45:58.780 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:45:58.780 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:45:58.780 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:45:58.780 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:45:58.780 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:45:58.780 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:45:58.780 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:58.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:58.780 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:45:58.781 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:58.781 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:45:58.781 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:58.781 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:58.781 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:45:58.781 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:58.781 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:45:58.781 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:58.781 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:45:58.781 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:58.782 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:58.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:58.782 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:58.782 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:45:58.782 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:58.782 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:58.782 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:45:58.782 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:58.782 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:58.782 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:58.782 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:45:58.783 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:58.783 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:58.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:58.783 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:58.783 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:45:58.783 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:58.783 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:58.784 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577428.787391_1643.1", 2024-04-20T01:45:58.784 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:58.784 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:58.784 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:58.784 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:48.787391Z", 2024-04-20T01:45:58.785 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:58.785 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:58.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:58.785 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:45:58.785 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:45:58.785 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:45:58.785 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:45:58.785 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:45:58.785 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:45:58.785 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:45:58.785 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:45:58.786 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:45:58.786 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:45:58.786 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:58.786 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:45:58.786 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:45:58.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:58.786 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:45:58.786 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:45:58.786 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:45:58.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:45:58.787 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:45:58Z" 2024-04-20T01:45:58.787 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:45:58.787 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577428.787391_1643.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:43:48.787391Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:45:58Z" } 2024-04-20T01:45:58.788 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577428.787391_1643.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:43:48.787391Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:45:58.788 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577553.275530_1647.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:58.788 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:45:58.789 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577553.275530_1647.1 target=1_1713577428.787391_1643.1 2024-04-20T01:46:03.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:03.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:03.790 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:03.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:46:03.918+0000 7f6d9a523780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:03.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:46:03.919+0000 7f6d9a523780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:03.930 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:46:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:46:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:46:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:46:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:46:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:46:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577428.787391_1643.1", 2024-04-20T01:46:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:48.787391Z", 2024-04-20T01:46:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:46:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:46:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:46:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:46:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:03Z" 2024-04-20T01:46:03.936 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:03.937 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577428.787391_1643.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:43:48.787391Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:46:03Z" } 2024-04-20T01:46:03.937 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577428.787391_1643.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:43:48.787391Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:46:03.937 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577553.275530_1647.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:03.937 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:03.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577553.275530_1647.1 target=1_1713577428.787391_1643.1 2024-04-20T01:46:08.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:08.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:08.939 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:08.998 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:08.998 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:09.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:46:09.081+0000 7f37c9cc2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:09.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:46:09.082+0000 7f37c9cc2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:09.110 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:09.111 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:09.111 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:09.111 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:09.111 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:09.111 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:46:09.111 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:09.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:09.111 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:09.111 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:09.112 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:09.112 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:09.112 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:09.112 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:46:09.112 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:09.112 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:46:09.112 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:09.112 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:46:09.112 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:09.112 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:09.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:09.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:09.113 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:09.113 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:09.113 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:09.113 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:46:09.113 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:09.113 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:09.113 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:09.113 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:46:09.113 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:09.113 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577428.787391_1643.1", 2024-04-20T01:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:48.787391Z", 2024-04-20T01:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:09.114 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:46:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:46:09.115 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:46:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:46:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:09.116 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:09Z" 2024-04-20T01:46:09.116 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:09.117 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577428.787391_1643.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:43:48.787391Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:46:09Z" } 2024-04-20T01:46:09.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577428.787391_1643.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:43:48.787391Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:46:09.118 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577553.275530_1647.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:09.118 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577428.787391_1643.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:09.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577553.275530_1647.1 target=1_1713577428.787391_1643.1 2024-04-20T01:46:14.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:14.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:14.119 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:14.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:46:14.235+0000 7fd33fb30780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:14.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:46:14.236+0000 7fd33fb30780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:14.241 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:14.241 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:14.241 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:14.241 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:14.241 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:14.241 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:46:14.241 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:14.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:14.241 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:14.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:14.242 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:14.242 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:14.242 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:14.242 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:46:14.242 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:14.242 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:46:14.242 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:14.242 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:46:14.242 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:14.242 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:14.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:14.243 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:14.243 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:14.243 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:14.243 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:14.243 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:46:14.243 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:14.243 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:14.243 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:14.243 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:46:14.243 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:14.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:14.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:14.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:14.244 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:14.244 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:14.244 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:14.244 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577553.275530_1647.1", 2024-04-20T01:46:14.244 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:14.244 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:14.244 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:14.244 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:45:53.275530Z", 2024-04-20T01:46:14.244 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:14.245 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:14.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:14.245 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:14.245 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:14.245 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:14.245 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:14.245 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:46:14.245 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:14.245 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:14.245 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:14.245 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:46:14.246 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:14.246 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:14.246 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:14.246 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:14.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:14.246 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:14.246 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:46:14.246 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:46:14.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:14.246 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:14Z" 2024-04-20T01:46:14.247 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:14.247 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577553.275530_1647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:45:53.275530Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:46:14Z" } 2024-04-20T01:46:14.248 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577553.275530_1647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:45:53.275530Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:46:14.248 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577553.275530_1647.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:14.248 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577553.275530_1647.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:14.248 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:46:14.248 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:46:14.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:14.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:14.249 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:14.310 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:14.310 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:14.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:46:14.359+0000 7fe58495a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:14.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:46:14.360+0000 7fe58495a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:14.362 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:46:14.363 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:46:14.363 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:46:14.363 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:46:14.363 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:46:14.363 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:46:14.363 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:14.363 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:14.363 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:46:14.363 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:14.363 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:46:14.364 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:46:14.364 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:46:14.364 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:46:14.364 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:46:14.364 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:46:14.364 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:46:14.364 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:46:14.364 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:14.364 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:14.364 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:14.364 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:14.365 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:46:14.365 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:46:14.365 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:46:14.365 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:46:14.365 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:46:14.365 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:46:14.365 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:46:14.365 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:46:14.365 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:14.365 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:14.365 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:14.366 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:14.366 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:46:14.366 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:46:14.366 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:46:14.366 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577553.275530_1647.1", 2024-04-20T01:46:14.366 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:46:14.366 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:46:14.366 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:46:14.366 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:45:53.275530Z", 2024-04-20T01:46:14.366 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:14.367 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:14.367 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:14.367 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:14.367 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:46:14.367 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:46:14.367 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:46:14.367 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:46:14.367 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:46:14.367 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:46:14.367 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:46:14.367 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:46:14.368 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:14.368 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:14.368 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:14.368 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:14.368 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:14.368 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:46:14.368 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:46:14.368 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:46:14.368 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:14.368 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:46:14Z" 2024-04-20T01:46:14.369 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:46:14.370 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577553.275530_1647.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:45:53.275530Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:46:14Z" } 2024-04-20T01:46:14.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577553.275530_1647.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:45:53.275530Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:46:14.370 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577553.275530_1647.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:14.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577553.275530_1647.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:14.370 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:46:14.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:14.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:14.371 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:46:14.421 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:14.421 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:14.483 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[]} 2024-04-20T01:46:14.483 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-88 2024-04-20T01:46:14.484 DEBUG:boto:path=/aktlji-88/ 2024-04-20T01:46:14.484 DEBUG:boto:auth_path=/aktlji-88/ 2024-04-20T01:46:14.484 DEBUG:boto:Method: PUT 2024-04-20T01:46:14.484 DEBUG:boto:Path: /aktlji-88/ 2024-04-20T01:46:14.484 DEBUG:boto:Data: 2024-04-20T01:46:14.484 DEBUG:boto:Headers: {} 2024-04-20T01:46:14.484 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:46:14.484 DEBUG:boto:Port: 8000 2024-04-20T01:46:14.484 DEBUG:boto:Params: {} 2024-04-20T01:46:14.485 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:46:14.485 DEBUG:boto:Token: None 2024-04-20T01:46:14.485 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:46:14 GMT /aktlji-88/ 2024-04-20T01:46:14.485 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:e8JGHFKwgQG4aEK44ofxiQ4MnfM= 2024-04-20T01:46:14.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:46:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:e8JGHFKwgQG4aEK44ofxiQ4MnfM=', 'Content-Length': '0'} 2024-04-20T01:46:14.518 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ca7c910ec63fb9e0-0066231e66-1bc9-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:46:14.518 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-89 2024-04-20T01:46:14.518 DEBUG:boto:path=/aktlji-89/ 2024-04-20T01:46:14.518 DEBUG:boto:auth_path=/aktlji-89/ 2024-04-20T01:46:14.518 DEBUG:boto:Method: PUT 2024-04-20T01:46:14.518 DEBUG:boto:Path: /aktlji-89/ 2024-04-20T01:46:14.518 DEBUG:boto:Data: 2024-04-20T01:46:14.519 DEBUG:boto:Headers: {} 2024-04-20T01:46:14.519 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:46:14.519 DEBUG:boto:Port: 8000 2024-04-20T01:46:14.519 DEBUG:boto:Params: {} 2024-04-20T01:46:14.519 DEBUG:boto:Token: None 2024-04-20T01:46:14.519 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:46:14 GMT /aktlji-89/ 2024-04-20T01:46:14.519 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vK6WsLr5OMR7hOSnxxhdtBhRNBk= 2024-04-20T01:46:14.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:46:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vK6WsLr5OMR7hOSnxxhdtBhRNBk=', 'Content-Length': '0'} 2024-04-20T01:46:14.547 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000057452f2621217318-0066231e66-1bc9-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:46:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:46:14.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:14.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:14.548 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:14.603 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:14.603 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:14.665 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:46:14.687 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:46:14.687 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:14.687 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:46:14.687 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:46:14.687 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:14.688 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:14.688 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:46:14.688 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:46:14.688 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:14.688 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:14.688 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577574.559830_1655.1", 2024-04-20T01:46:14.688 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:46:14.559830Z" 2024-04-20T01:46:14.688 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:14.688 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:14.688 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:46:14.688 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:46:14.689 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:14.689 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:46:14.690 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:14.690 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:46:14.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:14.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:14.690 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:14.749 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:14.749 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:14.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:46:14.815+0000 7f4c0ad8e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:14.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:46:14.816+0000 7f4c0ad8e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:14.826 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:14.826 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:14.826 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:14.826 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:46:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:46:14.827 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:46:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:46:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:14.828 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:46:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:46:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:14.829 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577574.516978_1648.1", 2024-04-20T01:46:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:14.516978Z", 2024-04-20T01:46:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:14.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:46:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:46:14.831 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:46:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:46:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:14.832 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:14Z" 2024-04-20T01:46:14.832 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:14.833 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577574.516978_1648.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:14.516978Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:46:14Z" } 2024-04-20T01:46:14.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577574.516978_1648.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:14.516978Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:46:14.834 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:14.834 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.516978_1648.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:14.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577574.559830_1655.1 target=1_1713577574.516978_1648.1 2024-04-20T01:46:19.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:19.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:19.836 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:19.896 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:46:19.958+0000 7fc61ac77780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:19.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:46:19.959+0000 7fc61ac77780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:19.967 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:19.967 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:19.967 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:19.967 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:19.967 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:19.967 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:46:19.967 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:19.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:19.968 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:19.968 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:19.968 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:19.968 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:19.968 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:19.968 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:46:19.968 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:19.968 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:46:19.968 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:19.968 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:46:19.969 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:19.969 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:19.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:19.969 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:19.969 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:19.969 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:19.969 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:19.969 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:46:19.969 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:19.969 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:46:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:19.970 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577574.559830_1655.1", 2024-04-20T01:46:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:14.559830Z", 2024-04-20T01:46:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:19.971 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:46:19.972 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:19.972 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:19.972 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:19.972 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:46:19.972 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:19.972 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:19.972 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:19.972 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:19.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:19.972 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:19.972 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:46:19.973 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:46:19.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:19.973 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:19Z" 2024-04-20T01:46:19.973 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:19.974 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577574.559830_1655.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:14.559830Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:46:19Z" } 2024-04-20T01:46:19.974 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577574.559830_1655.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:14.559830Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:46:19.974 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:19.974 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:19.974 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:46:19.974 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:46:19.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:19.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:19.975 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:20.034 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:20.035 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:20.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:46:20.090+0000 7f4f30c58780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:20.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:46:20.092+0000 7f4f30c58780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:20.097 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:46:20.097 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:46:20.097 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:46:20.097 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:46:20.098 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:46:20.098 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:46:20.098 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:20.098 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.099 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:46:20.099 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.099 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:46:20.099 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:46:20.099 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:46:20.100 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:46:20.100 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:46:20.100 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:46:20.100 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:46:20.101 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:46:20.101 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:20.101 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.102 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.102 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.102 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:46:20.102 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:46:20.102 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:46:20.103 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:46:20.103 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:46:20.103 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:46:20.103 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:46:20.103 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:46:20.104 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:20.104 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.104 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.104 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.105 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:46:20.105 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:46:20.105 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:46:20.105 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577574.559830_1655.1", 2024-04-20T01:46:20.105 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:46:20.106 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:46:20.106 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:46:20.106 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:46:14.559830Z", 2024-04-20T01:46:20.106 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:20.106 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.107 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.107 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.107 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:46:20.107 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:46:20.108 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:46:20.108 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:46:20.108 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:46:20.108 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:46:20.108 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:46:20.109 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:46:20.109 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:20.109 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.109 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.109 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:20.110 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.110 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:46:20.110 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:46:20.110 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:46:20.111 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.111 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:46:20Z" 2024-04-20T01:46:20.111 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:46:20.113 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577574.559830_1655.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:14.559830Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:46:20Z" } 2024-04-20T01:46:20.114 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577574.559830_1655.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:14.559830Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:46:20.114 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:20.115 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:20.115 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:46:20.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:20.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:20.116 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:46:20.173 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:20.173 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:20.243 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-20T01:46:20.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:20.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:20.244 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:46:20.304 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:20.304 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:20.369 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-20T01:46:20.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-88', '--dest-bucket=aktlji-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:20.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-88', '--dest-bucket=aktlji-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:20.370 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=aktlji-88 --dest-bucket=aktlji-89 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:46:20.429 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:20.430 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:20.488 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"aktlji-88","zones":["test-zone1"]},"dest":{"bucket":"aktlji-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:46:20.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:20.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:20.488 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:46:20.543 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:20.543 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:20.617 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"aktlji-88","zones":["test-zone1"]},"dest":{"bucket":"aktlji-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:46:20.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:20.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:20.617 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:46:20.673 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:20.674 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:20.750 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:46:20.751 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:46:20.751 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 16, 2024-04-20T01:46:20.751 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:46:20.751 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T01:46:20.751 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T01:46:20.751 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:46:20.751 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T01:46:20.751 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.751 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T01:46:20.751 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T01:46:20.752 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T01:46:20.752 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T01:46:20.752 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:46:20.752 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:46:20.752 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:46:20.752 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T01:46:20.752 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T01:46:20.752 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:46:20.752 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:46:20.752 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.752 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:46:20.753 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T01:46:20.753 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:46:20.753 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:46:20.753 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:46:20.753 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:46:20.753 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:46:20.753 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:46:20.753 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:46:20.753 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:46:20.754 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T01:46:20.754 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:46:20.754 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:46:20.754 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:46:20.754 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:46:20.754 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:46:20.754 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:20.754 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.754 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.754 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:46:20.754 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T01:46:20.755 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:46:20.755 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:46:20.755 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:46:20.755 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:46:20.755 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:46:20.755 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:46:20.755 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:46:20.755 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:46:20.755 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:46:20.755 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:46:20.755 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:46:20.756 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:46:20.756 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:46:20.756 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:46:20.756 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:20.756 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.756 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.756 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:46:20.756 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T01:46:20.756 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:46:20.756 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:46:20.756 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:46:20.757 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:46:20.757 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:46:20.757 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:46:20.757 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:46:20.757 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:46:20.757 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:46:20.757 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:46:20.757 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:46:20.757 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:46:20.757 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:46:20.758 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:46:20.758 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:20.758 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.758 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:46:20.758 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T01:46:20.758 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.758 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T01:46:20.758 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T01:46:20.759 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T01:46:20.759 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T01:46:20.759 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:20.759 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.759 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:46:20.759 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T01:46:20.759 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:46:20.759 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T01:46:20.759 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [ 2024-04-20T01:46:20.759 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.759 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-group", 2024-04-20T01:46:20.759 INFO:teuthology.orchestra.run.smithi019.stdout: "data_flow": { 2024-04-20T01:46:20.760 INFO:teuthology.orchestra.run.smithi019.stdout: "symmetrical": [ 2024-04-20T01:46:20.760 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.760 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow1", 2024-04-20T01:46:20.760 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:46:20.760 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:46:20.767 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:46:20.767 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:20.767 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.767 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:20.767 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.768 INFO:teuthology.orchestra.run.smithi019.stdout: "pipes": [ 2024-04-20T01:46:20.768 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.768 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe", 2024-04-20T01:46:20.768 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:46:20.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-88", 2024-04-20T01:46:20.769 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:46:20.769 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:46:20.769 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:20.769 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.769 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:46:20.769 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-89", 2024-04-20T01:46:20.770 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:46:20.770 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:46:20.770 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:20.770 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.770 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:46:20.770 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:46:20.770 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:46:20.770 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:46:20.771 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.771 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.771 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:46:20.771 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:46:20.771 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:46:20.771 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:46:20.771 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.771 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.771 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:46:20.771 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "enabled" 2024-04-20T01:46:20.771 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.771 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:20.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.772 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T01:46:20.772 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:46:20.772 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:20.772 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.772 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:46:20.772 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T01:46:20.772 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.772 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:46:20.772 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T01:46:20.772 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.773 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.773 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:46:20.773 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T01:46:20.773 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.773 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:20.773 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:46:20.773 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T01:46:20.773 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.773 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:20.773 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.773 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T01:46:20.773 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:46:20.774 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T01:46:20.774 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T01:46:20.774 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:46:20.774 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:46:20.774 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:46:20.774 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:46:20.774 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:46:20.774 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.774 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T01:46:20.774 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:46:20.774 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:46:20.775 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:46:20.775 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:46:20.775 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:46:20.775 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.775 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T01:46:20.775 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:46:20.775 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:46:20.775 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:46:20.775 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:46:20.775 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:46:20.775 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.775 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T01:46:20.776 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:46:20.776 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:46:20.776 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:46:20.776 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:46:20.776 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:46:20.776 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.776 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T01:46:20.776 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:46:20.776 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:46:20.776 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:46:20.776 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:46:20.776 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:46:20.777 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:20.777 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:20.777 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:46:20.777 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T01:46:20.777 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:20.777 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:46:20.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:20.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:46:20.779 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:46:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:20.961 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"aktlji-88","zones":["test-zone1"]},"dest":{"bucket":"aktlji-89","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:46:20.962 DEBUG:boto:path=/aktlji-88/ 2024-04-20T01:46:20.962 DEBUG:boto:auth_path=/aktlji-88/ 2024-04-20T01:46:20.962 DEBUG:boto:Method: HEAD 2024-04-20T01:46:20.962 DEBUG:boto:Path: /aktlji-88/ 2024-04-20T01:46:20.962 DEBUG:boto:Data: 2024-04-20T01:46:20.962 DEBUG:boto:Headers: {} 2024-04-20T01:46:20.962 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:46:20.962 DEBUG:boto:Port: 8000 2024-04-20T01:46:20.963 DEBUG:boto:Params: {} 2024-04-20T01:46:20.963 DEBUG:boto:Token: None 2024-04-20T01:46:20.963 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:46:20 GMT /aktlji-88/ 2024-04-20T01:46:20.963 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jZLzMK+++WDgJxzuW2El+p+izmY= 2024-04-20T01:46:20.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:46:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jZLzMK+++WDgJxzuW2El+p+izmY=', 'Content-Length': '0'} 2024-04-20T01:46:21.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002324d91003ba95f8-0066231e6d-1c7d-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:46:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:46:21.064 DEBUG:boto:path=/aktlji-88/obj1 2024-04-20T01:46:21.065 DEBUG:boto:auth_path=/aktlji-88/obj1 2024-04-20T01:46:21.065 DEBUG:boto:Method: PUT 2024-04-20T01:46:21.065 DEBUG:boto:Path: /aktlji-88/obj1 2024-04-20T01:46:21.065 DEBUG:boto:Data: 2024-04-20T01:46:21.065 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:46:21.065 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:46:21.065 DEBUG:boto:Port: 8000 2024-04-20T01:46:21.065 DEBUG:boto:Params: {} 2024-04-20T01:46:21.065 DEBUG:boto:Token: None 2024-04-20T01:46:21.066 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:46:21 GMT /aktlji-88/obj1 2024-04-20T01:46:21.066 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Gb8IL+GMccYLSGxtv7DCE2VMZUk= 2024-04-20T01:46:21.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:46:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Gb8IL+GMccYLSGxtv7DCE2VMZUk='} 2024-04-20T01:46:21.105 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3bb08ab2982c7f9-0066231e6d-1c7d-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:46:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:46:21.105 DEBUG:boto:path=/aktlji-88/ 2024-04-20T01:46:21.105 DEBUG:boto:auth_path=/aktlji-88/ 2024-04-20T01:46:21.105 DEBUG:boto:Method: HEAD 2024-04-20T01:46:21.105 DEBUG:boto:Path: /aktlji-88/ 2024-04-20T01:46:21.106 DEBUG:boto:Data: 2024-04-20T01:46:21.106 DEBUG:boto:Headers: {} 2024-04-20T01:46:21.106 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:46:21.106 DEBUG:boto:Port: 8000 2024-04-20T01:46:21.106 DEBUG:boto:Params: {} 2024-04-20T01:46:21.106 DEBUG:boto:Token: None 2024-04-20T01:46:21.106 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:46:21 GMT /aktlji-88/ 2024-04-20T01:46:21.106 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Ukvh3DrVc4/ASqHTsS/rpE1Wskw= 2024-04-20T01:46:21.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:46:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ukvh3DrVc4/ASqHTsS/rpE1Wskw=', 'Content-Length': '0'} 2024-04-20T01:46:21.112 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d18b28edc82f0639-0066231e6d-1c7d-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:46:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:46:21.112 DEBUG:boto:path=/aktlji-88/obj2 2024-04-20T01:46:21.112 DEBUG:boto:auth_path=/aktlji-88/obj2 2024-04-20T01:46:21.112 DEBUG:boto:Method: PUT 2024-04-20T01:46:21.112 DEBUG:boto:Path: /aktlji-88/obj2 2024-04-20T01:46:21.112 DEBUG:boto:Data: 2024-04-20T01:46:21.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:46:21.113 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:46:21.113 DEBUG:boto:Port: 8000 2024-04-20T01:46:21.113 DEBUG:boto:Params: {} 2024-04-20T01:46:21.113 DEBUG:boto:Token: None 2024-04-20T01:46:21.113 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:46:21 GMT /aktlji-88/obj2 2024-04-20T01:46:21.113 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:xr9oP+g04o4/S6ugvpfjFmSJwCc= 2024-04-20T01:46:21.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:46:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xr9oP+g04o4/S6ugvpfjFmSJwCc='} 2024-04-20T01:46:21.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097f9429dfb268319-0066231e6d-1c7d-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:46:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:46:21.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:21.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:21.159 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:21.246 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:21.246 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:21.381 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:46:21.439 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:46:21.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:21.439 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:46:21.439 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:46:21.439 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:21.439 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:21.439 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:46:21.439 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:46:21.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:21.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:21.440 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577574.559830_1655.1", 2024-04-20T01:46:21.440 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:46:14.559830Z" 2024-04-20T01:46:21.440 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:21.440 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:21.440 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:46:21.440 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:46:21.440 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:21.440 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:46:21.440 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:21.441 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:46:21.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:21.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:21.441 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:21.501 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:21.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:46:21.565+0000 7fc517308780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:21.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:46:21.566+0000 7fc517308780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:21.581 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:21.581 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:21.581 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:21.582 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:21.582 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:21.582 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:46:21.582 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:21.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:21.582 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:21.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:21.582 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:21.582 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:21.582 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:21.583 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:46:21.583 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:21.583 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:46:21.583 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:21.583 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:46:21.583 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:21.583 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:21.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:21.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:21.583 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:21.583 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:21.584 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:21.584 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:46:21.584 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:21.584 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:21.584 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:21.584 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:46:21.584 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:21.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:21.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:21.584 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:21.584 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:21.585 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:21.585 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:21.585 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577574.559830_1655.1", 2024-04-20T01:46:21.585 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:21.585 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:21.585 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:21.585 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:14.559830Z", 2024-04-20T01:46:21.585 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:21.585 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:21.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:21.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:21.586 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:21.586 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:21.586 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:46:21.586 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:46:21.586 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:21.586 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:21.586 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:21.586 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:46:21.586 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:46:21.587 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:21.587 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:21.587 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:21.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:21.587 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:21.587 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:46:21.587 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:46:21.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:21.587 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:21Z" 2024-04-20T01:46:21.588 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:21.590 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577574.559830_1655.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:14.559830Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:46:21Z" } 2024-04-20T01:46:21.590 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577574.559830_1655.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:14.559830Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:46:21.590 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:21.590 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:21.590 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:46:21.590 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:46:21.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:21.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:21.591 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:21.672 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:21.672 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:21.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:46:21.776+0000 7ffa5485f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:21.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:46:21.778+0000 7ffa5485f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:46:21.811 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:46:21.811 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:46:21.811 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:46:21.811 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:46:21.811 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:46:21.811 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:46:21.811 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:21.811 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:21.812 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:46:21.812 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:21.812 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:46:21.812 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:46:21.812 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:46:21.812 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:46:21.812 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:46:21.812 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:46:21.812 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:46:21.812 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:46:21.812 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:21.813 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:21.813 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:21.813 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:21.813 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:46:21.813 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:46:21.813 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:46:21.813 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:46:21.813 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:46:21.813 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:46:21.813 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:46:21.813 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:46:21.814 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:21.814 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:21.814 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:21.814 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:21.814 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:46:21.814 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:46:21.814 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:46:21.814 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577574.559830_1655.1", 2024-04-20T01:46:21.814 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:46:21.814 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:46:21.815 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:46:21.815 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:46:14.559830Z", 2024-04-20T01:46:21.815 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:21.815 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:21.815 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:21.815 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:21.815 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:46:21.815 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:46:21.815 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:46:21.815 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:46:21.815 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:46:21.815 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:46:21.816 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:46:21.816 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:46:21.816 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:46:21.816 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:21.816 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:21.816 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:46:21.816 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:21.816 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:46:21.816 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:46:21.816 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:46:21.816 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:21.817 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:46:21Z" 2024-04-20T01:46:21.817 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:46:21.818 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577574.559830_1655.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:14.559830Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:46:21Z" } 2024-04-20T01:46:21.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577574.559830_1655.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:14.559830Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:46:21.818 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:21.818 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:46:21.818 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:46:21.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:21.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:21.819 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:21.882 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:21.882 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:22.000 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:46:22.000 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:22.000 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:46:22.001 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:46:22.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:22.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:22.001 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:46:22.001 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:46:22.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:22.001 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:22.001 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:46:22.001 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:46:22.001 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:46:22.002 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:46:22.002 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:46:22.002 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:46:22.002 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:46:22.002 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:46:22.003 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:46:22.003 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:46:22.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:22.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:22.004 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:22.064 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:22.065 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:22.129 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:22.129 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:22.129 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:22.129 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:22.129 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:22.130 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:46:22.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:22.130 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:22.130 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:22.130 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:22.130 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:22.130 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:22.130 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:46:22.130 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:22.130 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:46:22.131 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:22.131 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:46:22.131 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:22.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:22.131 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:22.131 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:22.131 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:22.132 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:22.132 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:46:22.132 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:22.132 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:46:22.132 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:22.132 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:46:22.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:46:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:46:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:22.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:22.134 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:22.134 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:22.134 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:22.134 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:22.134 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:46:22.134 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:22.134 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:22.134 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:22.134 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:46:22.134 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:22.135 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:22.135 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:22.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:22.135 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:22.135 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:46:22.135 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:46:22.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:22.135 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:22Z" 2024-04-20T01:46:22.135 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:22.136 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:46:22Z" } 2024-04-20T01:46:22.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:46:22.137 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:46:22.137 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:46:22.137 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:46:27.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:27.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:27.138 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:27.200 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:27.200 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:27.263 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:46:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:27.264 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:46:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:46:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:46:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:27.265 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:46:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:46:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:46:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:27.266 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:46:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:46:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:27.267 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:46:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:46:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:27.268 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:46:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:46:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:27.269 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:27Z" 2024-04-20T01:46:27.269 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:27.270 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:46:27Z" } 2024-04-20T01:46:27.270 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:46:27.270 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:46:27.270 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:46:27.270 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:46:32.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:32.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:32.272 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:32.333 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:32.333 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:32.393 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:32.394 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:32.394 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:32.394 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:32.394 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:32.394 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:46:32.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:32.394 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:32.394 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:32.394 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:32.395 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:32.395 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:32.395 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:46:32.395 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:32.395 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:46:32.395 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:32.395 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:46:32.395 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:32.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:32.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:32.396 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:32.396 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:32.396 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:32.396 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:46:32.396 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:32.396 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:46:32.396 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:32.396 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:46:32.396 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:32.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:32.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:32.397 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:32.397 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:32.397 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:32.397 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:46:32.397 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:32.397 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:32.397 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:32.397 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:46:32.397 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:32.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:32.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:32.398 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:32.398 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:32.398 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:32.398 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:46:32.398 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:32.398 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:32.398 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:32.398 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:46:32.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:32.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:32.398 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:32.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:32.399 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:32.399 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:46:32.399 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:46:32.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:32.399 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:32Z" 2024-04-20T01:46:32.399 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:32.400 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:46:32Z" } 2024-04-20T01:46:32.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:46:32.400 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:46:32.400 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:46:32.401 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:46:37.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:37.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:37.402 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:37.464 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:37.464 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:37.543 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:37.544 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:37.544 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:37.544 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:37.544 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:37.544 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:46:37.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:37.544 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:37.544 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:37.544 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:37.545 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:37.545 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:37.545 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:46:37.545 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:37.545 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:46:37.545 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:37.545 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:46:37.545 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:37.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:37.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:37.545 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:37.546 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:37.546 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:37.546 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:46:37.546 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:37.546 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:46:37.546 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:37.546 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:46:37.546 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:37.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:37.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:37.546 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:37.546 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:37.547 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:37.547 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:46:37.547 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:37.547 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:37.547 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:37.547 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:46:37.547 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:37.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:37.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:37.547 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:37.547 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:37.548 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:37.548 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:46:37.548 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:37.548 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:37.548 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:37.548 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:46:37.548 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:37.548 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:37.548 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:37.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:37.548 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:37.548 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:46:37.549 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:46:37.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:37.549 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:37Z" 2024-04-20T01:46:37.549 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:37.551 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:46:37Z" } 2024-04-20T01:46:37.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:46:37.551 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:46:37.551 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:46:37.551 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:46:42.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:42.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:42.553 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:42.615 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:42.615 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:42.695 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:42.695 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:42.696 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:42.696 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:42.696 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:42.696 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:46:42.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:42.696 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:42.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:42.696 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:42.696 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:42.696 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:42.697 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:46:42.697 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:42.697 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:46:42.697 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:42.697 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:46:42.697 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:42.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:42.697 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:42.697 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:42.697 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:42.697 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:42.698 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:46:42.698 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:42.698 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:46:42.698 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:42.698 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:46:42.698 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:42.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:42.698 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:42.698 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:42.698 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:42.698 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:42.699 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:46:42.699 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:42.699 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:42.699 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:42.699 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:46:42.699 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:42.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:42.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:42.699 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:42.699 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:42.699 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:42.699 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:46:42.700 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:42.700 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:42.700 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:42.700 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:46:42.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:42.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:42.700 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:42.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:42.700 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:42.700 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:46:42.700 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:46:42.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:42.701 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:42Z" 2024-04-20T01:46:42.701 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:42.702 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:46:42Z" } 2024-04-20T01:46:42.702 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:46:42.702 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:46:42.702 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:46:42.702 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:46:47.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:47.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:47.704 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:47.766 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:47.766 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:47.831 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:47.831 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:47.831 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:47.831 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:47.831 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:47.831 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:46:47.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:47.832 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:47.832 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:47.832 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:47.832 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:47.832 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:47.832 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:46:47.832 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:47.832 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:46:47.832 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:47.833 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:46:47.833 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:47.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:47.833 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:47.833 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:47.833 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:47.833 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:47.833 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:46:47.833 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:47.833 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:46:47.833 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:47.834 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:46:47.834 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:47.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:47.834 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:47.834 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:47.834 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:47.834 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:47.834 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:46:47.834 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:47.834 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:47.835 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:46:47.836 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:47.836 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:47.836 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:47.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:47.836 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:47.836 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:46:47.836 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:46:47.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:47.836 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:47Z" 2024-04-20T01:46:47.836 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:47.837 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:46:47Z" } 2024-04-20T01:46:47.837 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:46:47.838 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:46:47.838 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:46:47.838 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:46:52.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:52.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:52.839 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:52.902 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:52.903 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:52.974 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:52.975 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:52.975 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:52.975 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:52.975 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:52.975 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:46:52.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:52.975 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:52.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:52.975 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:52.975 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:52.976 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:52.976 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:46:52.976 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:52.976 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:46:52.976 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:52.976 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:46:52.976 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:52.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:52.976 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:52.976 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:52.976 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:52.976 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:52.977 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:46:52.977 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:52.977 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:46:52.977 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:52.977 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:46:52.977 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:52.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:52.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:52.977 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:52.977 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:52.977 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:52.978 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:46:52.978 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:52.978 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:52.978 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:52.978 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:46:52.978 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:52.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:52.978 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:52.978 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:52.978 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:52.978 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:52.979 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:46:52.979 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:52.979 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:52.979 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:52.979 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:46:52.979 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:52.979 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:52.979 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:52.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:52.979 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:52.979 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:46:52.980 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:46:52.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:52.980 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:52Z" 2024-04-20T01:46:52.980 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:52.981 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:46:52Z" } 2024-04-20T01:46:52.981 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:46:52.981 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:46:52.982 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:46:52.982 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:46:57.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:57.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:46:57.983 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:46:58.046 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:46:58.046 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:46:58.115 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:46:58.115 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:46:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:46:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:46:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:46:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:46:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:46:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:46:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:58.116 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:46:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:46:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:46:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:46:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:58.117 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:46:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:46:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:46:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:46:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:58.118 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:46:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:46:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:46:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:46:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:46:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:46:58.119 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:46:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:46:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:46:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:46:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:46:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:46:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:46:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:46:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:46:58.120 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:46:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:46:58.121 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:46:58Z" 2024-04-20T01:46:58.121 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:46:58.122 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:46:58Z" } 2024-04-20T01:46:58.122 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:46:58.122 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:46:58.122 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:46:58.122 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:47:03.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:03.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:03.124 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:03.186 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:03.186 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:03.249 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:47:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:47:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:47:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:47:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:47:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:47:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:47:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:47:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:47:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:47:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:47:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:47:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:47:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:47:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:47:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:47:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:47:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:47:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:47:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:47:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:47:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:47:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:47:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:47:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:47:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:47:03Z" 2024-04-20T01:47:03.254 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:47:03.255 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:47:03Z" } 2024-04-20T01:47:03.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:47:03.256 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:47:03.256 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:47:03.256 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:47:08.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:08.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:08.258 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:08.320 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:08.320 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:08.394 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:47:08.394 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:47:08.394 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:47:08.394 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:47:08.395 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:47:08.395 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:47:08.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:08.395 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:47:08.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:08.395 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:47:08.395 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:08.395 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:08.395 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:47:08.396 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:08.396 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:47:08.396 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:08.396 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:47:08.396 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:08.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:08.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:08.396 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:47:08.396 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:08.396 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:08.397 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:47:08.397 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:08.397 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:47:08.397 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:08.397 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:47:08.397 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:08.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:08.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:08.397 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:47:08.397 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:08.397 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:08.398 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:47:08.398 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:08.398 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:08.398 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:08.398 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:47:08.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:08.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:08.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:08.398 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:47:08.398 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:08.398 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:08.399 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:47:08.399 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:08.399 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:08.399 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:08.399 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:47:08.399 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:08.399 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:08.399 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:47:08.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:08.399 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:47:08.399 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:47:08.399 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:47:08.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:08.400 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:47:08Z" 2024-04-20T01:47:08.400 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:47:08.401 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:47:08Z" } 2024-04-20T01:47:08.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:47:08.401 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:47:08.401 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:47:08.401 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:47:13.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:13.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:13.402 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:13.463 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:13.464 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:13.534 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:47:13.534 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:47:13.535 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:47:13.535 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:47:13.535 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:47:13.535 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:47:13.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:13.535 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:47:13.535 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:13.535 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:47:13.535 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:13.535 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:13.535 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:47:13.535 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:13.536 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:47:13.536 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:13.536 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:47:13.536 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:13.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:13.536 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:13.536 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:47:13.536 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:13.536 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:13.536 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:47:13.536 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:13.536 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:47:13.537 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:13.537 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:47:13.537 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:13.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:13.537 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:13.537 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:47:13.537 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:13.537 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:13.537 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:47:13.537 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:13.537 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:13.538 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:13.538 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:47:13.538 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:13.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:13.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:13.538 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:47:13.538 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:13.538 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:13.538 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:47:13.538 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:13.538 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:13.538 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:13.539 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:47:13.539 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:13.539 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:13.539 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:47:13.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:13.539 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:47:13.539 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:47:13.539 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:47:13.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:13.539 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:47:13Z" 2024-04-20T01:47:13.539 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:47:13.540 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:47:13Z" } 2024-04-20T01:47:13.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:47:13.541 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:47:13.541 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:47:13.541 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:47:18.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:18.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:18.542 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:18.667 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:47:18.667 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:47:18.667 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:47:18.667 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:47:18.667 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:47:18.667 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:47:18.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:18.668 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:47:18.668 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:18.668 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:47:18.668 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:18.668 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:18.668 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:47:18.668 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:18.668 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:47:18.668 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:18.669 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:47:18.669 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:18.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:18.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:18.669 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:47:18.669 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:18.669 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:18.669 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:47:18.669 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:18.669 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:47:18.669 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:18.670 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:47:18.670 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:18.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:18.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:18.670 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:47:18.670 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:18.670 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:18.670 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:47:18.670 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:18.670 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:18.670 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:18.671 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:47:18.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:18.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:18.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:18.671 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:47:18.671 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:18.671 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:18.671 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:47:18.671 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:18.671 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:18.671 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:18.672 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:47:18.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:18.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:18.672 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:47:18.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:18.672 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:47:18.672 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:47:18.672 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:47:18.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:18.672 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:47:18Z" 2024-04-20T01:47:18.673 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:47:18.674 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:47:18Z" } 2024-04-20T01:47:18.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:47:18.674 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:47:18.674 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:47:18.674 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:47:23.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:23.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:23.676 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:23.818 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:47:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:47:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:47:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:47:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:47:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:47:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:47:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:47:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:23.821 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:23.821 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:47:23.821 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:23.821 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:47:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:47:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:47:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:47:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:47:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:47:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:47:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:47:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:47:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:47:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:47:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:47:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:47:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:23.827 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:47:23.827 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:47:23.827 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:47:23.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:23.827 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:47:23Z" 2024-04-20T01:47:23.827 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:47:23.828 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:47:23Z" } 2024-04-20T01:47:23.828 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:47:23.828 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:47:23.829 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:47:23.829 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:47:28.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:28.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:28.831 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:28.893 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:28.893 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:28.958 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:47:28.958 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:47:28.958 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:47:28.958 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:47:28.958 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:47:28.958 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:47:28.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:28.958 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:47:28.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:28.959 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:47:28.959 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:28.959 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:28.959 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:47:28.959 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:28.959 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:47:28.959 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:28.959 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:47:28.959 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:28.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:28.960 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:28.960 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:47:28.960 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:28.960 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:28.960 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:47:28.960 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:28.960 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:47:28.960 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:28.960 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:47:28.960 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:28.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:28.961 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:28.961 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:47:28.961 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:28.961 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:28.961 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:47:28.961 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:28.961 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:28.961 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:28.961 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:47:28.961 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:28.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:28.962 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:28.962 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:47:28.962 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:28.962 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:28.962 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:47:28.962 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:28.962 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:28.962 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:28.962 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:47:28.963 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:28.963 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:28.963 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:47:28.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:28.963 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:47:28.963 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:47:28.963 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:47:28.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:28.963 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:47:28Z" 2024-04-20T01:47:28.963 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:47:28.964 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:47:28Z" } 2024-04-20T01:47:28.964 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:47:28.965 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:47:28.965 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:47:28.965 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:47:33.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:33.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:33.966 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:34.028 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:34.028 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:34.098 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:47:34.098 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:47:34.098 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:47:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:47:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:47:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:47:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:47:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:47:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:47:34.100 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:34.100 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:47:34.100 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:34.100 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:47:34.100 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:34.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:34.100 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:34.100 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:47:34.100 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:47:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:47:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:47:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:47:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:34.102 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:34.102 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:47:34.102 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:34.102 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:34.102 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:34.102 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:47:34.102 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:34.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:34.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:34.103 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:47:34.103 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:34.103 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:34.103 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:47:34.103 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:34.103 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:34.103 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:34.103 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:47:34.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:34.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:34.103 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:47:34.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:34.104 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:47:34.104 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:47:34.104 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:47:34.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:34.104 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:47:34Z" 2024-04-20T01:47:34.104 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:47:34.105 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:47:34Z" } 2024-04-20T01:47:34.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:47:34.105 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:47:34.105 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:47:34.105 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:47:39.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:39.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:39.107 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:39.248 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:47:39.249 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:47:39.249 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:47:39.249 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:47:39.249 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:47:39.249 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:47:39.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:39.249 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:47:39.249 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:39.249 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:47:39.249 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:39.249 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:39.250 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:47:39.250 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:39.250 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:47:39.250 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:39.250 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:47:39.250 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:39.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:39.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:39.250 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:47:39.251 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:39.251 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:39.251 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:47:39.251 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:39.251 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:47:39.251 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:39.251 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:47:39.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:39.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:39.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:39.251 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:47:39.252 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:39.252 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:39.252 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:47:39.252 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:39.252 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:39.252 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:39.252 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:47:39.252 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:39.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:39.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:39.253 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:47:39.253 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:39.253 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:39.253 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:47:39.253 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:39.253 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:39.253 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:39.253 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:47:39.254 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:39.254 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:39.254 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:47:39.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:39.254 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:47:39.254 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:47:39.254 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:47:39.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:39.254 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:47:39Z" 2024-04-20T01:47:39.254 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:47:39.255 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:47:39Z" } 2024-04-20T01:47:39.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:47:39.256 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:47:39.256 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:47:39.256 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:47:44.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:44.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:44.257 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:44.322 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:44.322 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:44.386 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:47:44.568 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:47:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:47:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:47:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:47:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:47:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:47:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:47:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:44.569 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:47:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:47:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:47:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:47:44.570 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:47:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:47:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:47:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:44.571 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:47:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:47:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:47:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:44.572 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:47:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:47:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:47:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:47:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:44.573 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:47:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:47:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:47:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:44.574 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:47:44Z" 2024-04-20T01:47:44.574 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:47:44.575 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:47:44Z" } 2024-04-20T01:47:44.575 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:47:44.576 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:47:44.576 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:47:44.576 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:47:49.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:49.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:49.577 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:49.639 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:49.639 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:49.702 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:47:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:47:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:47:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:47:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:47:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:47:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:47:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:47:49.703 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:47:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:47:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:47:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:47:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:49.704 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027470", 2024-04-20T01:47:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:47:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:43:49.066000Z" 2024-04-20T01:47:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:47:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:49.705 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:47:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:47:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:47:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:49.706 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021018", 2024-04-20T01:47:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.836632Z" 2024-04-20T01:47:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:47:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:47:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:47:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:47:49.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:49.708 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:47:49Z" 2024-04-20T01:47:49.708 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:47:49.709 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027470", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:43:49.066000Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.836632Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:47:49Z" } 2024-04-20T01:47:49.709 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027470', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:43:49.066000Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.836632Z'}}] 2024-04-20T01:47:49.709 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:47:49.709 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027470', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021018'} 2024-04-20T01:47:49.709 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027610 target=00000000000000000000:00000000000000027470, shard=3 master=00000000000000000000:00000000000000021158 target=00000000000000000000:00000000000000021018 2024-04-20T01:47:54.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:54.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:54.711 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:54.773 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:54.773 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:54.834 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:47:54.834 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:47:54.835 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:47:54.835 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:47:54.835 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:47:54.835 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:47:54.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:54.835 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:47:54.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:54.836 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:47:54.836 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:54.836 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:54.836 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:47:54.836 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:54.836 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:47:54.837 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:54.837 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:47:54.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:54.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:54.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:54.837 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:47:54.838 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:54.838 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:54.838 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:47:54.838 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:54.838 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:47:54.838 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:54.838 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:47:54.838 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:54.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:54.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:54.839 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:47:54.839 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:54.839 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:54.839 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:47:54.839 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:54.839 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:54.839 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:54.839 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:47:54.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:54.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:54.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:54.840 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:47:54.841 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:54.841 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:47:54.841 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:47:54.841 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:54.842 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:54.842 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:54.842 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:47:54.842 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:54.843 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:54.843 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:47:54.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:54.843 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:47:54.843 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:47:54.843 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:47:54.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:54.844 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:47:54Z" 2024-04-20T01:47:54.844 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:47:54.845 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:47:54Z" } 2024-04-20T01:47:54.845 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:47:54.846 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:47:54.846 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:47:54.846 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:47:54.847 DEBUG:boto:path=/aktlji-88/ 2024-04-20T01:47:54.847 DEBUG:boto:auth_path=/aktlji-88/ 2024-04-20T01:47:54.847 DEBUG:boto:Method: HEAD 2024-04-20T01:47:54.847 DEBUG:boto:Path: /aktlji-88/ 2024-04-20T01:47:54.847 DEBUG:boto:Data: 2024-04-20T01:47:54.847 DEBUG:boto:Headers: {} 2024-04-20T01:47:54.847 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:47:54.847 DEBUG:boto:Port: 8000 2024-04-20T01:47:54.848 DEBUG:boto:Params: {} 2024-04-20T01:47:54.848 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:47:54.848 DEBUG:boto:Token: None 2024-04-20T01:47:54.848 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:47:54 GMT /aktlji-88/ 2024-04-20T01:47:54.849 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:w2++NGPZfPAb7MYD8MwMZfws1+4= 2024-04-20T01:47:54.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:47:54 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:w2++NGPZfPAb7MYD8MwMZfws1+4=', 'Content-Length': '0'} 2024-04-20T01:47:54.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010e18b5ccb8332d9-0066231eca-25d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:47:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:47:54.856 DEBUG:boto:path=/aktlji-88/obj1 2024-04-20T01:47:54.856 DEBUG:boto:auth_path=/aktlji-88/obj1 2024-04-20T01:47:54.856 DEBUG:boto:Method: HEAD 2024-04-20T01:47:54.856 DEBUG:boto:Path: /aktlji-88/obj1 2024-04-20T01:47:54.856 DEBUG:boto:Data: 2024-04-20T01:47:54.856 DEBUG:boto:Headers: {} 2024-04-20T01:47:54.857 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:47:54.857 DEBUG:boto:Port: 8000 2024-04-20T01:47:54.857 DEBUG:boto:Params: {} 2024-04-20T01:47:54.857 DEBUG:boto:Token: None 2024-04-20T01:47:54.857 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:47:54 GMT /aktlji-88/obj1 2024-04-20T01:47:54.857 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:XdOQPyCls78jTjJ4AlUcyE0kW/s= 2024-04-20T01:47:54.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:47:54 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XdOQPyCls78jTjJ4AlUcyE0kW/s=', 'Content-Length': '0'} 2024-04-20T01:47:54.859 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000006feadd870941d910-0066231eca-25d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:47:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:47:54.859 DEBUG:boto:path=/aktlji-88/obj2 2024-04-20T01:47:54.859 DEBUG:boto:auth_path=/aktlji-88/obj2 2024-04-20T01:47:54.859 DEBUG:boto:Method: HEAD 2024-04-20T01:47:54.859 DEBUG:boto:Path: /aktlji-88/obj2 2024-04-20T01:47:54.859 DEBUG:boto:Data: 2024-04-20T01:47:54.859 DEBUG:boto:Headers: {} 2024-04-20T01:47:54.859 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:47:54.859 DEBUG:boto:Port: 8000 2024-04-20T01:47:54.860 DEBUG:boto:Params: {} 2024-04-20T01:47:54.860 DEBUG:boto:Token: None 2024-04-20T01:47:54.860 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:47:54 GMT /aktlji-88/obj2 2024-04-20T01:47:54.860 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wlhUkNYq5rnGHBJsNawt3/vjP54= 2024-04-20T01:47:54.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:47:54 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wlhUkNYq5rnGHBJsNawt3/vjP54=', 'Content-Length': '0'} 2024-04-20T01:47:54.861 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000008cde75450e5290fb-0066231eca-25d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:47:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:47:54.862 DEBUG:boto:path=/aktlji-89/ 2024-04-20T01:47:54.862 DEBUG:boto:auth_path=/aktlji-89/ 2024-04-20T01:47:54.862 DEBUG:boto:Method: HEAD 2024-04-20T01:47:54.862 DEBUG:boto:Path: /aktlji-89/ 2024-04-20T01:47:54.862 DEBUG:boto:Data: 2024-04-20T01:47:54.862 DEBUG:boto:Headers: {} 2024-04-20T01:47:54.863 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:47:54.863 DEBUG:boto:Port: 8000 2024-04-20T01:47:54.863 DEBUG:boto:Params: {} 2024-04-20T01:47:54.863 DEBUG:boto:Token: None 2024-04-20T01:47:54.864 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:47:54 GMT /aktlji-89/ 2024-04-20T01:47:54.864 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BBPGebtXSo3glYvw8ZNgjLuARh4= 2024-04-20T01:47:54.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:47:54 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BBPGebtXSo3glYvw8ZNgjLuARh4=', 'Content-Length': '0'} 2024-04-20T01:47:54.867 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e678966dd3d942b0-0066231eca-25d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:47:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:47:54.867 DEBUG:boto:path=/aktlji-89/obj1 2024-04-20T01:47:54.867 DEBUG:boto:auth_path=/aktlji-89/obj1 2024-04-20T01:47:54.867 DEBUG:boto:Method: HEAD 2024-04-20T01:47:54.867 DEBUG:boto:Path: /aktlji-89/obj1 2024-04-20T01:47:54.867 DEBUG:boto:Data: 2024-04-20T01:47:54.867 DEBUG:boto:Headers: {} 2024-04-20T01:47:54.867 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:47:54.867 DEBUG:boto:Port: 8000 2024-04-20T01:47:54.868 DEBUG:boto:Params: {} 2024-04-20T01:47:54.868 DEBUG:boto:Token: None 2024-04-20T01:47:54.868 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:47:54 GMT /aktlji-89/obj1 2024-04-20T01:47:54.868 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:2nP3iOa8jK65kV7zgd+fawqGzXQ= 2024-04-20T01:47:54.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:47:54 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2nP3iOa8jK65kV7zgd+fawqGzXQ=', 'Content-Length': '0'} 2024-04-20T01:47:54.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:46:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-88:73f46f02-c7ab-4426-a267-2273af911da5.7121.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027328dfab344e569-0066231eca-25d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:47:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:47:54.870 DEBUG:boto:path=/aktlji-89/obj1 2024-04-20T01:47:54.871 DEBUG:boto:auth_path=/aktlji-89/obj1 2024-04-20T01:47:54.871 DEBUG:boto:Method: GET 2024-04-20T01:47:54.871 DEBUG:boto:Path: /aktlji-89/obj1 2024-04-20T01:47:54.871 DEBUG:boto:Data: 2024-04-20T01:47:54.871 DEBUG:boto:Headers: {} 2024-04-20T01:47:54.871 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:47:54.872 DEBUG:boto:Port: 8000 2024-04-20T01:47:54.872 DEBUG:boto:Params: {} 2024-04-20T01:47:54.872 DEBUG:boto:Token: None 2024-04-20T01:47:54.872 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:47:54 GMT /aktlji-89/obj1 2024-04-20T01:47:54.872 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:jjuZq04//+hKBppcLBdBeZlV6/E= 2024-04-20T01:47:54.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:47:54 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jjuZq04//+hKBppcLBdBeZlV6/E=', 'Content-Length': '0'} 2024-04-20T01:47:54.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:46:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-88:73f46f02-c7ab-4426-a267-2273af911da5.7121.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000190be851d553790b-0066231eca-25d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:47:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:47:54.918 DEBUG:boto:path=/aktlji-89/obj2 2024-04-20T01:47:54.919 DEBUG:boto:auth_path=/aktlji-89/obj2 2024-04-20T01:47:54.919 DEBUG:boto:Method: HEAD 2024-04-20T01:47:54.919 DEBUG:boto:Path: /aktlji-89/obj2 2024-04-20T01:47:54.919 DEBUG:boto:Data: 2024-04-20T01:47:54.919 DEBUG:boto:Headers: {} 2024-04-20T01:47:54.919 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:47:54.920 DEBUG:boto:Port: 8000 2024-04-20T01:47:54.920 DEBUG:boto:Params: {} 2024-04-20T01:47:54.920 DEBUG:boto:Token: None 2024-04-20T01:47:54.920 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:47:54 GMT /aktlji-89/obj2 2024-04-20T01:47:54.920 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:X81u2lnd2AphYqYDeEUACM8vJe0= 2024-04-20T01:47:54.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:47:54 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X81u2lnd2AphYqYDeEUACM8vJe0=', 'Content-Length': '0'} 2024-04-20T01:47:54.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:46:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-88:73f46f02-c7ab-4426-a267-2273af911da5.7121.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d04883a3507bd03-0066231eca-25d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:47:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:47:54.922 DEBUG:boto:path=/aktlji-89/obj2 2024-04-20T01:47:54.922 DEBUG:boto:auth_path=/aktlji-89/obj2 2024-04-20T01:47:54.922 DEBUG:boto:Method: GET 2024-04-20T01:47:54.923 DEBUG:boto:Path: /aktlji-89/obj2 2024-04-20T01:47:54.923 DEBUG:boto:Data: 2024-04-20T01:47:54.923 DEBUG:boto:Headers: {} 2024-04-20T01:47:54.923 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:47:54.923 DEBUG:boto:Port: 8000 2024-04-20T01:47:54.923 DEBUG:boto:Params: {} 2024-04-20T01:47:54.923 DEBUG:boto:Token: None 2024-04-20T01:47:54.923 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:47:54 GMT /aktlji-89/obj2 2024-04-20T01:47:54.924 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:q8QLfLNxJjao6JreC31gWEon91M= 2024-04-20T01:47:54.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:47:54 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q8QLfLNxJjao6JreC31gWEon91M=', 'Content-Length': '0'} 2024-04-20T01:47:54.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:46:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-88:73f46f02-c7ab-4426-a267-2273af911da5.7121.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039bd52d9921ffde4-0066231eca-25d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:47:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:47:54.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:54.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:54.926 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:47:54.987 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:54.987 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:55.059 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-20T01:47:55.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:55.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:55.060 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:47:55.117 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:55.117 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:55.179 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:47:55.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:55.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:55.180 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:47:55.234 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:55.235 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:55.309 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:47:55.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:55.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:55.310 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:47:55.363 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:55.364 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:55.440 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:47:55.440 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:47:55.440 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 17, 2024-04-20T01:47:55.440 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:47:55.440 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T01:47:55.440 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T01:47:55.441 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:47:55.441 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T01:47:55.441 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:55.441 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T01:47:55.441 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T01:47:55.441 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T01:47:55.441 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T01:47:55.441 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:47:55.441 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:47:55.441 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:47:55.441 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T01:47:55.442 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T01:47:55.442 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:47:55.442 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:47:55.442 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:55.442 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:47:55.442 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T01:47:55.442 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:47:55.442 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:47:55.442 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:47:55.443 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:47:55.443 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:47:55.443 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:47:55.443 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:47:55.443 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:47:55.443 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T01:47:55.443 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:47:55.443 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:47:55.443 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:47:55.443 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:47:55.443 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:47:55.444 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:47:55.444 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.444 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:55.444 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:47:55.444 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T01:47:55.444 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:47:55.444 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:47:55.444 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:47:55.444 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:47:55.444 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:47:55.444 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:47:55.445 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:47:55.445 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:47:55.445 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:47:55.445 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:47:55.445 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:47:55.445 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:47:55.445 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:47:55.445 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:47:55.445 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:47:55.445 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.445 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:55.446 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:47:55.446 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T01:47:55.446 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:47:55.446 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:47:55.446 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:47:55.446 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:47:55.446 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:47:55.447 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:47:55.447 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:47:55.447 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:47:55.448 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:47:55.448 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:47:55.448 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:47:55.448 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:47:55.448 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:47:55.449 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:47:55.449 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:47:55.449 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:47:55.449 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:47:55.449 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T01:47:55.450 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:55.450 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T01:47:55.450 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T01:47:55.450 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T01:47:55.450 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T01:47:55.451 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:47:55.451 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:47:55.451 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:47:55.451 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T01:47:55.451 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:47:55.451 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T01:47:55.451 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [ 2024-04-20T01:47:55.451 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:55.451 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-group", 2024-04-20T01:47:55.452 INFO:teuthology.orchestra.run.smithi019.stdout: "data_flow": { 2024-04-20T01:47:55.452 INFO:teuthology.orchestra.run.smithi019.stdout: "symmetrical": [ 2024-04-20T01:47:55.452 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:55.452 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow1", 2024-04-20T01:47:55.452 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:47:55.452 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:47:55.465 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:47:55.465 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:47:55.466 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:47:55.466 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:47:55.466 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.466 INFO:teuthology.orchestra.run.smithi019.stdout: "pipes": [ 2024-04-20T01:47:55.466 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:55.466 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe", 2024-04-20T01:47:55.466 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:47:55.466 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "*", 2024-04-20T01:47:55.467 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:47:55.467 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:47:55.467 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:47:55.467 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:47:55.467 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.467 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:47:55.467 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "*", 2024-04-20T01:47:55.467 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:47:55.467 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:47:55.467 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:47:55.467 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:47:55.467 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.468 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:47:55.468 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:47:55.468 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:47:55.468 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:47:55.468 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:47:55.468 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.468 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:47:55.468 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:47:55.468 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:47:55.468 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:47:55.468 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:47:55.468 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "allowed" 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:55.469 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:47:55.470 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T01:47:55.470 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.470 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:55.470 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:47:55.470 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T01:47:55.470 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.470 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:55.470 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:47:55.471 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T01:47:55.471 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:47:55.471 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:47:55.471 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.471 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T01:47:55.471 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:47:55.471 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T01:47:55.471 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T01:47:55.471 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:47:55.471 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:47:55.471 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:47:55.471 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:47:55.472 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:47:55.472 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.472 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T01:47:55.472 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:47:55.472 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:47:55.472 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:47:55.472 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:47:55.472 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:47:55.473 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.473 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T01:47:55.473 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:47:55.473 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:47:55.473 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:47:55.473 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:47:55.473 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:47:55.474 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.474 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T01:47:55.474 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:47:55.474 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:47:55.474 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:47:55.474 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:47:55.474 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:47:55.474 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.474 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T01:47:55.474 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:47:55.474 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:47:55.475 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:47:55.475 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:47:55.475 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:47:55.475 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:47:55.475 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:55.475 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:47:55.475 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T01:47:55.475 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:47:55.475 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:47:55.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:55.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:55.477 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:47:55.536 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:55.537 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:55.608 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:47:55.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:55.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:55.608 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket aktlji-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:47:55.680 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:55.680 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:55.842 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-20T01:47:55.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:55.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:55.843 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket aktlji-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:47:55.920 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:55.920 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:56.096 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-20T01:47:56.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=aktlji-88', '--source-bucket=*', '--dest-bucket=aktlji-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:56.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=aktlji-88', '--source-bucket=*', '--dest-bucket=aktlji-89', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:56.096 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=aktlji-88 '--source-bucket=*' --dest-bucket=aktlji-89 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:47:56.179 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:56.179 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:56.346 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"aktlji-89","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-20T01:47:56.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:56.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:56.347 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket aktlji-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:47:56.412 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:56.413 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:56.542 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"aktlji-89","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-20T01:47:56.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:56.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:47:56.543 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket aktlji-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:47:56.606 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:56.606 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:56.705 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"aktlji-89","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-20T01:47:56.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:56.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:56.706 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:56.765 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:56.765 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:56.823 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:47:56.840 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:47:56.840 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:56.840 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:47:56.840 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:47:56.840 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:56.841 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:56.841 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:47:56.841 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:47:56.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:56.841 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:56.841 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577676.516091_1706.1", 2024-04-20T01:47:56.841 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:47:56.516091Z" 2024-04-20T01:47:56.841 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:47:56.841 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:47:56.841 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:47:56.841 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:47:56.842 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:47:56.842 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:47:56.843 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:47:56.843 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:47:56.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:56.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:47:56.843 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:47:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:47:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:47:56.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:47:56.960+0000 7f4056371780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:47:56.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:47:56.961+0000 7f4056371780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:47:56.964 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:47:56.964 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:47:56.964 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:47:56.965 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:47:56.965 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:47:56.965 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:47:56.965 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:47:56.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:56.965 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:47:56.965 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:56.965 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:47:56.965 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:56.965 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:47:56.965 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:47:56.966 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:56.966 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:47:56.966 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:56.966 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:47:56.966 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:47:56.966 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:56.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:56.966 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:56.966 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:47:56.966 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:56.966 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:47:56.966 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:47:56.967 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:56.967 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:56.967 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:56.967 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:47:56.967 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:47:56.967 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:56.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:56.967 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:56.967 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:47:56.967 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:56.967 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:47:56.968 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577574.559830_1655.1", 2024-04-20T01:47:56.968 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:56.968 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:56.968 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:56.968 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:14.559830Z", 2024-04-20T01:47:56.968 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:47:56.968 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:56.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:56.968 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:47:56.968 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:47:56.968 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:47:56.969 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:47:56.969 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:47:56.969 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:47:56.969 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:47:56.969 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:47:56.969 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:47:56.969 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:47:56.969 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:56.969 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:47:56.969 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:47:56.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:56.969 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:47:56.970 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:47:56.970 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:47:56.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:47:56.970 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:47:56Z" 2024-04-20T01:47:56.970 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:47:56.971 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577574.559830_1655.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:14.559830Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:47:56Z" } 2024-04-20T01:47:56.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577574.559830_1655.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:14.559830Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:47:56.971 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:47:56.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:47:56.972 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577676.516091_1706.1 target=1_1713577574.559830_1655.1 2024-04-20T01:48:01.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:01.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:01.973 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:02.095+0000 7fb0d1565780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:02.097+0000 7fb0d1565780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:02.100 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:02.101 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:02.101 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:02.101 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:02.101 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:02.101 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:48:02.101 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:02.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:02.101 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:02.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:02.101 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:02.101 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:02.102 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:02.102 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:48:02.102 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:02.102 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:48:02.102 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:02.102 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:48:02.102 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:02.102 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:02.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:02.102 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:02.103 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:02.103 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:02.103 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:02.103 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:48:02.103 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:02.103 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:02.103 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:02.103 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:48:02.103 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:02.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:02.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:02.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:02.104 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:02.104 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:02.104 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:02.104 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577574.559830_1655.1", 2024-04-20T01:48:02.104 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:02.104 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:02.104 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:02.104 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:14.559830Z", 2024-04-20T01:48:02.104 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:02.104 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:02.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:02.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:02.105 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:02.105 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:02.105 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:02.105 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:48:02.105 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:02.105 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:02.105 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:02.105 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:48:02.105 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:02.106 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:02.106 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:02.106 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:02.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:02.106 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:02.106 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:48:02.106 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:48:02.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:02.106 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:02Z" 2024-04-20T01:48:02.106 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:02.107 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577574.559830_1655.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:14.559830Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:48:02Z" } 2024-04-20T01:48:02.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577574.559830_1655.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:14.559830Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:48:02.108 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:02.108 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:02.108 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577676.516091_1706.1 target=1_1713577574.559830_1655.1 2024-04-20T01:48:07.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:07.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:07.109 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:07.169 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:07.169 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:07.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:07.233+0000 7feaee6a6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:07.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:07.234+0000 7feaee6a6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:07.238 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:07.238 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:07.238 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:07.238 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:07.238 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:07.238 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:48:07.239 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:07.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:07.239 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:07.239 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:07.239 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:07.239 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:07.239 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:07.239 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:48:07.239 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:07.239 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:48:07.239 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:07.239 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:48:07.240 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:07.240 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:07.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:07.240 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:07.240 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:07.240 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:07.240 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:07.240 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:48:07.240 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:07.240 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:07.240 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:07.240 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:48:07.241 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:07.241 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:07.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:07.241 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:07.241 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:07.241 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:07.241 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:07.241 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577574.559830_1655.1", 2024-04-20T01:48:07.241 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:07.241 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:07.241 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:07.242 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:14.559830Z", 2024-04-20T01:48:07.242 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:07.242 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:07.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:07.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:07.242 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:07.242 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:07.242 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:07.242 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:48:07.242 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:07.242 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:07.243 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:07.243 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:48:07.243 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:07.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:07.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:07.243 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:07.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:07.243 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:07.243 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:48:07.243 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:48:07.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:07.243 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:07Z" 2024-04-20T01:48:07.244 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:07.244 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577574.559830_1655.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:14.559830Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:48:07Z" } 2024-04-20T01:48:07.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577574.559830_1655.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:14.559830Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:48:07.245 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:07.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:07.245 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577676.516091_1706.1 target=1_1713577574.559830_1655.1 2024-04-20T01:48:12.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:12.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:12.246 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:12.306 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:12.306 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:12.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:12.362+0000 7f5b90a0c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:12.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:12.363+0000 7f5b90a0c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:12.366 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:12.367 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:12.367 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:12.367 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:12.367 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:12.367 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:48:12.367 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:12.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:12.367 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:12.367 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:12.367 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:12.367 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:48:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:48:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:48:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:12.368 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:48:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:48:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:12.369 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577574.559830_1655.1", 2024-04-20T01:48:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:14.559830Z", 2024-04-20T01:48:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:12.370 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:48:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:48:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:12.371 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:48:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:48:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:12.372 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:12Z" 2024-04-20T01:48:12.372 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:12.373 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577574.559830_1655.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:14.559830Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:48:12Z" } 2024-04-20T01:48:12.373 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577574.559830_1655.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:14.559830Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:48:12.373 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:12.373 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:12.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577676.516091_1706.1 target=1_1713577574.559830_1655.1 2024-04-20T01:48:17.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:17.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:17.375 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:17.433 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:17.434 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:17.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:17.495+0000 7fab4fec1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:17.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:17.496+0000 7fab4fec1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:17.504 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:17.504 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:17.504 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:17.504 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:17.504 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:17.504 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:48:17.504 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:48:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:48:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:17.505 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:48:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:48:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:17.506 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:48:17.507 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:17.507 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:17.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:17.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:17.507 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:17.507 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:17.507 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:17.507 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577574.559830_1655.1", 2024-04-20T01:48:17.507 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:17.507 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:17.507 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:17.507 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:14.559830Z", 2024-04-20T01:48:17.508 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:17.508 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:17.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:17.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:17.508 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:17.508 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:17.508 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:17.508 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:48:17.508 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:17.508 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:17.508 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:17.508 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:48:17.509 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:17.509 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:17.509 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:17.509 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:17.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:17.509 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:17.509 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:48:17.509 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:48:17.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:17.509 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:17Z" 2024-04-20T01:48:17.509 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:17.510 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577574.559830_1655.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:14.559830Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:48:17Z" } 2024-04-20T01:48:17.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577574.559830_1655.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:14.559830Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:48:17.511 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:17.511 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:17.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577676.516091_1706.1 target=1_1713577574.559830_1655.1 2024-04-20T01:48:22.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:22.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:22.512 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:22.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:22.635+0000 7f2df9586780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:22.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:22.636+0000 7f2df9586780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:22.646 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:48:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:22.647 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:48:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:48:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:48:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:22.648 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:48:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:48:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:22.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:22.650 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:22.650 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:22.650 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:22.650 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:22.650 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577574.559830_1655.1", 2024-04-20T01:48:22.650 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:22.650 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:22.650 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:22.650 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:14.559830Z", 2024-04-20T01:48:22.650 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:22.651 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:22.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:22.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:22.651 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:22.651 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:22.651 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:22.651 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:48:22.651 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:22.651 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:22.651 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:22.651 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:48:22.651 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:22.652 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:22.652 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:22.652 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:22.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:22.652 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:22.652 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:48:22.652 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:48:22.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:22.652 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:22Z" 2024-04-20T01:48:22.652 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:22.653 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577574.559830_1655.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:14.559830Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:48:22Z" } 2024-04-20T01:48:22.653 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577574.559830_1655.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:14.559830Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:48:22.654 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:22.654 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577574.559830_1655.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:22.654 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577676.516091_1706.1 target=1_1713577574.559830_1655.1 2024-04-20T01:48:27.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:27.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:27.656 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:27.715 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:27.715 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:27.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:27.775+0000 7fc08cc62780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:27.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:27.776+0000 7fc08cc62780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:27.784 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:48:27.785 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:27.786 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:48:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:48:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:48:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:27.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:48:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:27.788 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:48:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:27.790 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577676.516091_1706.1", 2024-04-20T01:48:27.790 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:27.790 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:47:56.516091Z", 2024-04-20T01:48:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:27.791 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:48:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:48:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:27.792 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:48:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:48:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:27.793 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:27Z" 2024-04-20T01:48:27.793 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:27.794 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577676.516091_1706.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:47:56.516091Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:48:27Z" } 2024-04-20T01:48:27.794 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577676.516091_1706.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:47:56.516091Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:48:27.794 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:27.794 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:27.794 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:48:27.795 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:48:27.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:27.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:27.795 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:27.854 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:27.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:48:27.910+0000 7f4877464780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:27.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:48:27.911+0000 7f4877464780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:27.913 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:48:27.913 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:48:27.913 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:48:27.913 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:48:27.913 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:48:27.914 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:48:27.914 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:48:27.914 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:27.914 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:48:27.914 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:27.914 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:48:27.914 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:48:27.914 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:48:27.914 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:48:27.914 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:48:27.915 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:48:27.915 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:48:27.915 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:48:27.915 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:48:27.915 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:48:27.915 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:27.915 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:27.915 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:48:27.915 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:48:27.915 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:48:27.915 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:48:27.915 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:48:27.916 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:48:27.916 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:48:27.916 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:48:27.916 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:48:27.916 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:48:27.916 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:27.916 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:27.916 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:48:27.916 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:48:27.916 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:48:27.916 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577676.516091_1706.1", 2024-04-20T01:48:27.917 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:48:27.917 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:48:27.917 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:48:27.917 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:47:56.516091Z", 2024-04-20T01:48:27.917 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:48:27.917 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:48:27.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:27.917 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:27.917 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:48:27.917 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:48:27.917 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:48:27.917 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:48:27.918 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:48:27.918 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:48:27.918 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:48:27.918 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:48:27.918 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:48:27.918 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:48:27.918 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:48:27.918 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:48:27.918 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:27.918 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:48:27.918 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:48:27.919 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:48:27.919 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:27.919 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:48:27Z" 2024-04-20T01:48:27.919 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:48:27.922 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577676.516091_1706.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:47:56.516091Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:48:27Z" } 2024-04-20T01:48:27.922 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577676.516091_1706.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:47:56.516091Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:48:27.922 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:27.922 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:27.922 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:48:27.922 DEBUG:boto:path=/aktlji-88/ 2024-04-20T01:48:27.923 DEBUG:boto:auth_path=/aktlji-88/ 2024-04-20T01:48:27.923 DEBUG:boto:Method: HEAD 2024-04-20T01:48:27.923 DEBUG:boto:Path: /aktlji-88/ 2024-04-20T01:48:27.923 DEBUG:boto:Data: 2024-04-20T01:48:27.923 DEBUG:boto:Headers: {} 2024-04-20T01:48:27.923 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:48:27.923 DEBUG:boto:Port: 8000 2024-04-20T01:48:27.923 DEBUG:boto:Params: {} 2024-04-20T01:48:27.924 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:48:27.924 DEBUG:boto:Token: None 2024-04-20T01:48:27.924 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:48:27 GMT /aktlji-88/ 2024-04-20T01:48:27.924 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HbtIy7p8+hddrrJSZZNwFWABJH0= 2024-04-20T01:48:27.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:48:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HbtIy7p8+hddrrJSZZNwFWABJH0=', 'Content-Length': '0'} 2024-04-20T01:48:27.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1d31d854f255137-0066231eeb-1cbe-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:48:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:48:27.931 DEBUG:boto:path=/aktlji-88/obj5 2024-04-20T01:48:27.931 DEBUG:boto:auth_path=/aktlji-88/obj5 2024-04-20T01:48:27.931 DEBUG:boto:Method: PUT 2024-04-20T01:48:27.931 DEBUG:boto:Path: /aktlji-88/obj5 2024-04-20T01:48:27.931 DEBUG:boto:Data: 2024-04-20T01:48:27.931 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:48:27.931 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:48:27.932 DEBUG:boto:Port: 8000 2024-04-20T01:48:27.932 DEBUG:boto:Params: {} 2024-04-20T01:48:27.932 DEBUG:boto:Token: None 2024-04-20T01:48:27.932 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:48:27 GMT /aktlji-88/obj5 2024-04-20T01:48:27.932 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:VwJBu+vMXwEbyQ+DSkARKnPwCQk= 2024-04-20T01:48:27.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:48:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VwJBu+vMXwEbyQ+DSkARKnPwCQk='} 2024-04-20T01:48:27.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084bc363bae2a7364-0066231eeb-1cbe-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:48:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:48:27.943 DEBUG:boto:path=/aktlji-88/ 2024-04-20T01:48:27.943 DEBUG:boto:auth_path=/aktlji-88/ 2024-04-20T01:48:27.943 DEBUG:boto:Method: HEAD 2024-04-20T01:48:27.944 DEBUG:boto:Path: /aktlji-88/ 2024-04-20T01:48:27.944 DEBUG:boto:Data: 2024-04-20T01:48:27.944 DEBUG:boto:Headers: {} 2024-04-20T01:48:27.944 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:48:27.944 DEBUG:boto:Port: 8000 2024-04-20T01:48:27.944 DEBUG:boto:Params: {} 2024-04-20T01:48:27.944 DEBUG:boto:Token: None 2024-04-20T01:48:27.944 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:48:27 GMT /aktlji-88/ 2024-04-20T01:48:27.944 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HbtIy7p8+hddrrJSZZNwFWABJH0= 2024-04-20T01:48:27.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:48:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HbtIy7p8+hddrrJSZZNwFWABJH0=', 'Content-Length': '0'} 2024-04-20T01:48:27.947 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7a4d7446fd21774-0066231eeb-1cbe-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:48:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:48:27.947 DEBUG:boto:path=/aktlji-88/obj6 2024-04-20T01:48:27.947 DEBUG:boto:auth_path=/aktlji-88/obj6 2024-04-20T01:48:27.948 DEBUG:boto:Method: PUT 2024-04-20T01:48:27.948 DEBUG:boto:Path: /aktlji-88/obj6 2024-04-20T01:48:27.948 DEBUG:boto:Data: 2024-04-20T01:48:27.948 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:48:27.948 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:48:27.948 DEBUG:boto:Port: 8000 2024-04-20T01:48:27.948 DEBUG:boto:Params: {} 2024-04-20T01:48:27.948 DEBUG:boto:Token: None 2024-04-20T01:48:27.948 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:48:27 GMT /aktlji-88/obj6 2024-04-20T01:48:27.948 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:NAH0Fie/ANtyca+6LqvaMPOPhZc= 2024-04-20T01:48:27.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:48:27 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NAH0Fie/ANtyca+6LqvaMPOPhZc='} 2024-04-20T01:48:27.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b89080a4cb9c6933-0066231eeb-1cbe-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:48:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:48:27.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:27.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:27.995 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:28.052 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:28.052 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:28.099 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:48:28.117 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:48:28.117 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:28.117 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:48:28.117 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:48:28.117 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:28.117 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:28.117 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:48:28.117 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:48:28.117 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:28.118 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:28.118 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577676.516091_1706.1", 2024-04-20T01:48:28.118 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:47:56.516091Z" 2024-04-20T01:48:28.118 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:28.118 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:28.118 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:48:28.118 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:48:28.119 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:48:28.119 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:48:28.120 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:28.120 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:48:28.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:28.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:28.120 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:28.179 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:28.179 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:28.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:28.254+0000 7f7ed8c69780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:28.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:48:28.255+0000 7f7ed8c69780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:28.261 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:28.261 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:48:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:48:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:48:28.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:48:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:48:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:28.263 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:48:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577676.516091_1706.1", 2024-04-20T01:48:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:28.264 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:47:56.516091Z", 2024-04-20T01:48:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:48:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:48:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:28.265 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:48:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:48:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:48:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:48:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:28.266 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:28Z" 2024-04-20T01:48:28.267 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:28.267 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577676.516091_1706.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:47:56.516091Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:48:28Z" } 2024-04-20T01:48:28.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577676.516091_1706.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:47:56.516091Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:48:28.268 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:28.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:28.268 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:48:28.268 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:48:28.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:28.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:28.269 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:28.329 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:28.330 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:28.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:48:28.381+0000 7f5e2c14e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:28.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:48:28.382+0000 7f5e2c14e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:48:28.385 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:48:28.385 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:48:28.385 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:48:28.385 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:48:28.385 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:48:28.385 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:48:28.385 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:48:28.385 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:28.385 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:48:28.385 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:28.386 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:48:28.386 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:48:28.386 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:48:28.386 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:48:28.386 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:48:28.386 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:48:28.386 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:48:28.386 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:48:28.386 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:48:28.386 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:48:28.386 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:28.387 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:28.387 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:48:28.387 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:48:28.387 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:48:28.387 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:48:28.387 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:48:28.387 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:48:28.387 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:48:28.387 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:48:28.388 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:48:28.388 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:48:28.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:28.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:28.388 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:48:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:48:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:48:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577676.516091_1706.1", 2024-04-20T01:48:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:48:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:48:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:48:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:47:56.516091Z", 2024-04-20T01:48:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:48:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:48:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:28.390 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:28.390 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:48:28.390 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:48:28.390 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:48:28.390 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:48:28.390 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:48:28.390 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:48:28.390 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:48:28.390 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:48:28.390 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:48:28.390 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:48:28.391 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:48:28.391 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:48:28.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:28.391 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:48:28.391 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:48:28.391 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:48:28.391 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:28.391 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:48:28Z" 2024-04-20T01:48:28.391 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:48:28.392 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577676.516091_1706.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:47:56.516091Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:48:28Z" } 2024-04-20T01:48:28.393 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577676.516091_1706.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:47:56.516091Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:48:28.394 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:28.394 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:48:28.394 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:48:28.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:28.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:28.394 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:28.448 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:28.448 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:28.518 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:48:28.519 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:28.519 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:48:28.519 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:48:28.519 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:28.519 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:28.519 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:48:28.519 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:48:28.519 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:28.519 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:28.519 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:48:28.520 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:48:28.520 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:48:28.520 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:48:28.520 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:48:28.520 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:48:28.520 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:48:28.520 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:48:28.522 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:48:28.522 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:48:28.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:28.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:28.523 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:28.583 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:28.584 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:28.650 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:28.651 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:28.651 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:28.651 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:28.651 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:28.651 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:48:28.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:28.651 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:28.651 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:28.651 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:28.652 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:28.652 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:28.652 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:48:28.652 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:28.652 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:48:28.652 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:28.652 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:48:28.652 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:28.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:28.652 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:28.652 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:28.652 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:28.653 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:28.653 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:48:28.653 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:28.653 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:48:28.653 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:28.653 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:48:28.653 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:28.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:28.653 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:28.653 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:28.653 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:28.654 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:28.654 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:48:28.654 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:28.654 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:28.654 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:28.654 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:48:28.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:28.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:28.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:28.654 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:28.654 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:28.655 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:28.655 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:48:28.655 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:28.655 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:28.655 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:28.655 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:48:28.655 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:28.655 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:28.655 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:28.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:28.655 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:28.656 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:48:28.656 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:48:28.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:28.656 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:28Z" 2024-04-20T01:48:28.656 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:28.657 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:48:28Z" } 2024-04-20T01:48:28.657 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:48:28.657 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:48:28.657 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:48:28.657 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:48:33.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:33.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:33.659 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:33.721 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:33.721 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:33.792 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:33.792 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:33.792 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:33.792 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:33.792 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:33.792 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:48:33.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:33.792 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:33.792 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:33.793 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:33.793 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:33.793 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:33.793 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:48:33.793 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:33.793 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:48:33.793 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:33.793 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:48:33.793 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:33.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:33.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:33.794 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:33.794 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:33.794 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:33.794 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:48:33.794 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:33.794 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:48:33.794 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:33.794 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:48:33.794 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:33.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:33.794 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:33.795 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:33.795 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:33.795 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:33.795 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:48:33.795 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:33.795 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:33.795 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:33.795 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:48:33.795 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:33.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:33.795 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:33.796 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:33.796 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:33.796 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:33.796 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:48:33.796 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:33.796 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:33.796 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:33.796 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:48:33.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:33.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:33.796 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:33.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:33.797 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:33.797 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:48:33.797 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:48:33.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:33.797 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:33Z" 2024-04-20T01:48:33.797 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:33.798 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:48:33Z" } 2024-04-20T01:48:33.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:48:33.798 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:48:33.798 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:48:33.798 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:48:38.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:38.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:38.800 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:38.864 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:38.864 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:38.940 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:38.941 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:38.941 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:38.941 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:38.941 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:38.941 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:48:38.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:38.941 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:38.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:38.941 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:38.942 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:38.942 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:38.942 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:48:38.942 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:38.942 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:48:38.942 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:38.942 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:48:38.942 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:38.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:38.942 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:38.942 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:38.943 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:38.943 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:38.943 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:48:38.943 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:38.943 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:48:38.943 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:38.943 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:48:38.943 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:38.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:38.943 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:38.945 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:38.945 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:38.945 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:38.945 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:48:38.945 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:38.945 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:38.946 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:38.946 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:48:38.946 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:38.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:38.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:38.946 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:38.946 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:38.946 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:38.946 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:48:38.946 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:38.946 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:38.947 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:38.947 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:48:38.947 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:38.947 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:38.947 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:38.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:38.947 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:38.947 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:48:38.947 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:48:38.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:38.947 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:38Z" 2024-04-20T01:48:38.947 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:38.948 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:48:38Z" } 2024-04-20T01:48:38.949 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:48:38.949 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:48:38.949 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:48:38.949 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:48:43.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:43.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:43.951 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:44.012 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:44.012 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:44.073 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:44.073 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:48:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:44.074 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:48:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:48:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:48:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:44.075 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:48:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:48:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:48:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:44.076 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:48:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:48:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:48:44.077 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:48:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:48:44.078 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:48:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:44.079 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:44Z" 2024-04-20T01:48:44.079 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:44.080 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:48:44Z" } 2024-04-20T01:48:44.080 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:48:44.080 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:48:44.080 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:48:44.080 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:48:49.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:49.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:49.081 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:49.143 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:49.143 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:49.226 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:49.227 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:49.227 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:49.227 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:49.228 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:49.228 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:48:49.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:49.228 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:49.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:49.229 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:49.229 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:49.229 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:49.229 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:48:49.229 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:49.230 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:48:49.230 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:49.230 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:48:49.230 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:49.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:49.231 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:49.231 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:49.231 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:49.231 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:49.232 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:48:49.232 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:49.232 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:48:49.232 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:49.233 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:48:49.233 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:49.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:49.233 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:49.233 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:49.234 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:49.234 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:49.234 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:48:49.234 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:49.235 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:49.235 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:49.235 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:48:49.235 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:49.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:49.236 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:49.236 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:49.236 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:49.236 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:49.237 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:48:49.237 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:49.237 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:49.237 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:49.237 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:48:49.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:49.238 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:49.238 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:49.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:49.239 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:49.239 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:48:49.239 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:48:49.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:49.239 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:49Z" 2024-04-20T01:48:49.240 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:49.242 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:48:49Z" } 2024-04-20T01:48:49.242 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:48:49.242 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:48:49.243 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:48:49.243 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:48:54.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:54.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:54.245 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:54.306 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:54.307 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:54.370 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:54.370 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:54.370 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:54.370 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:54.370 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:54.370 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:48:54.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:54.370 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:54.371 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:54.371 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:54.371 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:54.371 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:54.371 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:48:54.371 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:54.371 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:48:54.371 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:54.371 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:48:54.371 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:54.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:54.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:54.372 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:54.372 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:54.372 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:54.372 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:48:54.372 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:54.372 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:48:54.372 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:54.372 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:48:54.372 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:54.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:54.372 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:54.373 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:54.373 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:54.373 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:54.373 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:48:54.373 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:54.373 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:54.373 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:54.373 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:48:54.373 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:54.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:54.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:54.374 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:54.374 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:54.374 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:54.374 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:48:54.374 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:54.374 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:54.374 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:54.374 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:48:54.374 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:54.374 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:54.374 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:54.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:54.375 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:54.375 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:48:54.375 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:48:54.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:54.375 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:54Z" 2024-04-20T01:48:54.375 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:54.376 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:48:54Z" } 2024-04-20T01:48:54.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:48:54.377 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:48:54.377 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:48:54.377 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:48:59.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:59.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:48:59.379 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:48:59.441 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:48:59.441 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:48:59.502 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:48:59.503 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:48:59.503 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:48:59.503 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:48:59.503 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:48:59.503 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:48:59.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:59.504 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:48:59.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:59.504 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:48:59.504 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:59.504 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:59.504 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:48:59.504 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:59.504 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:48:59.504 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:59.505 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:48:59.505 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:59.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:59.505 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:59.505 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:48:59.505 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:59.505 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:59.505 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:48:59.505 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:59.505 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:48:59.505 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:59.506 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:48:59.506 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:59.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:59.506 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:59.506 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:48:59.506 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:59.506 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:59.506 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:48:59.506 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:59.506 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:59.507 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:59.507 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:48:59.507 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:59.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:59.507 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:48:59.507 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:48:59.507 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:48:59.507 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:48:59.507 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:48:59.507 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:48:59.507 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:48:59.508 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:48:59.508 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:48:59.508 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:59.508 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:48:59.508 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:48:59.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:59.508 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:48:59.508 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:48:59.508 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:48:59.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:48:59.509 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:48:59Z" 2024-04-20T01:48:59.509 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:48:59.510 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:48:59Z" } 2024-04-20T01:48:59.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:48:59.510 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:48:59.510 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:48:59.510 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:49:04.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:04.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:04.511 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:49:04.573 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:49:04.573 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:49:04.642 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:49:04.642 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:49:04.642 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:49:04.642 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:49:04.643 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:49:04.643 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:49:04.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:04.643 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:49:04.643 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:04.643 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:49:04.643 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:04.643 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:04.643 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:49:04.643 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:04.643 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:49:04.644 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:04.644 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:49:04.644 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:04.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:04.644 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:04.644 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:49:04.644 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:04.644 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:04.644 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:49:04.644 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:04.644 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:49:04.644 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:04.645 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:49:04.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:04.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:04.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:04.645 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:49:04.645 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:04.645 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:04.645 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:49:04.645 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:04.645 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:04.645 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:04.646 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:49:04.646 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:04.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:04.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:04.646 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:49:04.646 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:04.646 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:04.646 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:49:04.646 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:04.646 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:04.646 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:04.647 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:49:04.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:04.647 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:04.647 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:49:04.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:04.647 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:49:04.647 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:49:04.647 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:49:04.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:04.647 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:49:04Z" 2024-04-20T01:49:04.647 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:49:04.648 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:49:04Z" } 2024-04-20T01:49:04.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:49:04.649 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:49:04.649 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:49:04.649 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:49:09.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:09.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:09.650 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:49:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:49:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:49:09.772 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:49:09.772 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:49:09.772 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:49:09.772 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:49:09.772 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:49:09.772 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:49:09.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:09.773 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:49:09.773 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:09.773 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:49:09.773 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:09.773 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:09.773 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:49:09.773 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:09.773 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:49:09.773 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:09.773 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:49:09.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:09.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:09.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:09.774 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:49:09.774 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:09.774 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:09.774 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:49:09.774 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:09.774 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:49:09.774 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:09.774 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:49:09.774 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:09.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:09.775 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:09.775 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:49:09.775 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:09.775 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:09.775 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:49:09.775 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:09.775 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:09.775 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:09.775 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:49:09.775 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:09.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:09.776 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:09.776 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:49:09.776 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:09.776 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:09.776 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:49:09.776 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:09.776 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:09.776 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:09.776 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:49:09.776 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:09.776 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:09.777 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:49:09.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:09.777 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:49:09.777 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:49:09.777 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:49:09.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:09.777 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:49:09Z" 2024-04-20T01:49:09.777 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:49:09.778 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:49:09Z" } 2024-04-20T01:49:09.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:49:09.778 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:49:09.779 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:49:09.779 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:49:14.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:14.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:14.780 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:49:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:49:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:49:14.916 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:49:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:49:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:49:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:49:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:49:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:49:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:49:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:49:14.917 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:49:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:49:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:49:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:14.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:49:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:49:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:49:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:49:14.919 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:49:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:49:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:49:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:49:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:49:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:49:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:49:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:49:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:49:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:49:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:14.922 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:49:14Z" 2024-04-20T01:49:14.922 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:49:14.923 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:49:14Z" } 2024-04-20T01:49:14.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:49:14.923 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:49:14.924 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:49:14.924 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:49:19.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:19.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:19.925 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:49:19.987 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:49:19.987 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:49:20.055 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:49:20.055 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:49:20.055 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:49:20.055 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:49:20.056 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:49:20.056 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:49:20.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:20.056 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:49:20.056 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:20.056 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:49:20.056 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:20.056 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:20.056 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:49:20.057 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:20.057 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:49:20.057 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:20.057 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:49:20.057 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:20.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:20.057 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:20.057 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:49:20.057 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:20.057 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:20.057 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:49:20.058 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:20.058 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:49:20.058 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:20.058 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:49:20.058 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:20.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:20.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:20.058 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:49:20.058 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:20.058 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:20.059 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:49:20.059 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:20.059 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:20.059 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:20.059 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:49:20.059 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:20.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:20.059 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:20.059 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:49:20.059 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:20.060 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:20.060 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:49:20.060 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:20.060 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:20.060 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:20.060 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:49:20.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:20.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:20.060 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:49:20.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:20.060 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:49:20.061 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:49:20.061 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:49:20.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:20.061 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:49:20Z" 2024-04-20T01:49:20.061 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:49:20.062 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:49:20Z" } 2024-04-20T01:49:20.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:49:20.062 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:49:20.062 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:49:20.062 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:49:25.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:25.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:25.064 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:49:25.125 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:49:25.125 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:49:25.192 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:49:25.192 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:49:25.192 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:49:25.192 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:49:25.192 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:49:25.192 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:49:25.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:25.192 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:49:25.193 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:25.193 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:49:25.193 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:25.193 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:25.193 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:49:25.193 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:25.193 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:49:25.193 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:25.193 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:49:25.193 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:25.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:25.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:25.194 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:49:25.194 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:25.194 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:25.194 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:49:25.194 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:25.194 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:49:25.194 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:25.194 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:49:25.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:25.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:25.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:25.195 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:49:25.195 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:25.195 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:25.195 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:49:25.195 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:25.195 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:25.195 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:25.195 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:49:25.195 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:25.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:25.196 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:25.196 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:49:25.196 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:25.196 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:25.196 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:49:25.196 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:25.196 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:25.197 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:25.197 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:49:25.197 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:25.197 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:25.197 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:49:25.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:25.197 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:49:25.197 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:49:25.198 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:49:25.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:25.198 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:49:25Z" 2024-04-20T01:49:25.198 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:49:25.199 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:49:25Z" } 2024-04-20T01:49:25.199 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:49:25.199 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:49:25.199 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:49:25.200 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:49:30.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:30.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:30.201 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:49:30.262 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:49:30.263 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:49:30.327 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:49:30.327 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:49:30.327 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:49:30.327 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:49:30.328 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:49:30.328 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:49:30.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:30.328 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:49:30.328 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:30.328 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:49:30.328 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:30.328 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:30.328 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:49:30.328 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:30.328 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:49:30.328 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:30.329 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:49:30.329 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:30.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:30.329 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:30.329 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:49:30.329 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:30.329 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:30.329 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:49:30.329 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:30.329 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:49:30.329 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:30.329 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:49:30.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:30.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:30.330 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:30.330 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:49:30.330 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:30.330 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:30.330 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:49:30.330 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:30.330 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:30.330 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:30.330 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:49:30.331 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:30.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:30.331 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:30.331 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:49:30.331 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:30.331 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:30.331 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:49:30.331 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:30.331 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:30.331 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:30.331 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:49:30.331 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:30.332 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:30.332 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:49:30.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:30.332 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:49:30.332 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:49:30.332 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:49:30.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:30.332 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:49:30Z" 2024-04-20T01:49:30.332 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:49:30.333 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:49:30Z" } 2024-04-20T01:49:30.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:49:30.333 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:49:30.334 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:49:30.334 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:49:35.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:35.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:35.335 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:49:35.397 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:49:35.397 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:49:35.480 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:49:35.480 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:49:35.480 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:49:35.480 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:49:35.480 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:49:35.480 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:49:35.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:35.481 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:49:35.481 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:35.481 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:49:35.481 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:35.481 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:35.481 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:49:35.481 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:35.481 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:49:35.481 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:35.481 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:49:35.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:35.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:35.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:35.482 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:49:35.482 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:35.482 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:35.482 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:49:35.482 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:35.482 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:49:35.482 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:35.482 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:49:35.482 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:35.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:35.483 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:35.483 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:49:35.483 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:35.483 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:35.483 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:49:35.483 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:35.483 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:35.483 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:35.483 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:49:35.483 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:35.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:35.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:35.484 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:49:35.484 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:35.484 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:35.484 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:49:35.484 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:35.484 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:35.484 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:35.484 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:49:35.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:35.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:35.485 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:49:35.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:35.485 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:49:35.485 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:49:35.485 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:49:35.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:35.485 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:49:35Z" 2024-04-20T01:49:35.485 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:49:35.486 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:49:35Z" } 2024-04-20T01:49:35.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:49:35.487 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:49:35.487 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:49:35.487 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:49:40.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:40.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:40.489 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:49:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:49:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:49:40.621 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:49:40.622 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:49:40.622 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:49:40.622 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:49:40.622 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:49:40.622 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:49:40.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:40.622 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:49:40.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:40.622 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:49:40.622 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:40.623 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:40.623 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:49:40.623 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:40.623 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:49:40.623 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:40.623 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:49:40.623 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:40.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:40.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:40.623 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:49:40.623 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:40.624 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:40.624 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:49:40.624 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:40.624 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:49:40.624 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:40.624 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:49:40.624 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:40.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:40.624 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:40.624 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:49:40.624 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:40.624 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:40.625 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:49:40.625 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:40.625 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:40.625 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:40.625 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:49:40.625 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:40.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:40.625 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:40.625 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:49:40.625 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:40.625 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:40.626 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:49:40.626 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:40.626 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:40.626 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:40.626 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:49:40.626 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:40.626 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:40.626 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:49:40.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:40.626 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:49:40.626 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:49:40.626 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:49:40.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:40.627 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:49:40Z" 2024-04-20T01:49:40.627 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:49:40.628 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:49:40Z" } 2024-04-20T01:49:40.628 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:49:40.628 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:49:40.628 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:49:40.628 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:49:45.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:45.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:45.630 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:49:45.691 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:49:45.691 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:49:45.752 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:49:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:49:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:49:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:49:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:49:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:49:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:49:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:49:45.752 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:49:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:49:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:49:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:49:45.753 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:45.754 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:45.754 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:49:45.754 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:45.754 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:49:45.754 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:45.754 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:49:45.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:45.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:45.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:45.754 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:49:45.754 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:45.755 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:45.755 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:49:45.755 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:45.755 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:45.755 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:45.755 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:49:45.755 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:45.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:45.755 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:45.755 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:49:45.755 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:45.756 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:45.756 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:49:45.756 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:45.756 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:45.756 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:45.756 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:49:45.756 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:45.756 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:45.756 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:49:45.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:45.756 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:49:45.757 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:49:45.757 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:49:45.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:45.757 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:49:45Z" 2024-04-20T01:49:45.757 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:49:45.758 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:49:45Z" } 2024-04-20T01:49:45.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:49:45.758 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:49:45.758 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:49:45.758 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:49:50.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:50.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:50.760 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:49:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:49:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:49:50.898 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:49:50.899 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:49:50.899 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:49:50.899 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:49:50.899 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:49:50.899 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:49:50.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:50.899 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:49:50.899 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:50.899 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:49:50.899 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:50.899 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:50.900 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:49:50.900 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:50.900 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:49:50.900 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:50.900 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:49:50.900 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:50.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:50.900 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:50.900 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:49:50.900 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:50.900 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:50.901 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:49:50.901 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:50.901 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:49:50.901 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:50.901 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:49:50.901 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:50.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:50.901 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:50.901 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:49:50.901 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:50.901 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:50.902 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:49:50.902 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:50.902 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:50.902 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:50.902 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:49:50.902 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:50.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:50.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:50.902 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:49:50.902 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:50.902 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:50.903 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:49:50.903 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:50.903 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:50.903 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:50.903 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:49:50.903 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:50.903 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:50.903 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:49:50.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:50.903 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:49:50.903 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:49:50.903 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:49:50.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:50.904 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:49:50Z" 2024-04-20T01:49:50.904 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:49:50.905 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:49:50Z" } 2024-04-20T01:49:50.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:49:50.905 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:49:50.905 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:49:50.905 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:49:55.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:55.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:49:55.906 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:49:55.968 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:49:55.968 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:49:56.044 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:49:56.044 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:49:56.044 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:49:56.044 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:49:56.045 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:49:56.045 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:49:56.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:56.045 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:49:56.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:56.045 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:49:56.045 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:56.045 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:56.045 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:49:56.045 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:56.046 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:49:56.046 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:56.046 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:49:56.046 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:56.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:56.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:56.046 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:49:56.046 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:56.046 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:56.046 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027610", 2024-04-20T01:49:56.046 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:49:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.113990Z" 2024-04-20T01:49:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:49:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:49:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:49:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:49:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:49:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:49:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:49:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021158", 2024-04-20T01:49:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:49:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:49:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:49:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:46:21.154533Z" 2024-04-20T01:49:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:49:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:49:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:49:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:49:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:49:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:49:56.049 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:49:56Z" 2024-04-20T01:49:56.049 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:49:56.050 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027610", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:46:21.113990Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:46:21.154533Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:49:56Z" } 2024-04-20T01:49:56.050 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027610', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.113990Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:46:21.154533Z'}}] 2024-04-20T01:49:56.051 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:49:56.051 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027610', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021158'} 2024-04-20T01:49:56.051 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000027750 target=00000000000000000000:00000000000000027610, shard=3 master=00000000000000000000:00000000000000021298 target=00000000000000000000:00000000000000021158 2024-04-20T01:50:01.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:01.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:01.052 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:01.113 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:01.113 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:01.200 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:01.200 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:01.201 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:01.201 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:01.201 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:01.201 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:50:01.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:01.202 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:01.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:01.202 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:01.203 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:01.203 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:50:01.203 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:50:01.203 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:01.203 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:50:01.204 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:01.204 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:50:01.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:01.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:01.205 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:01.205 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:01.205 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:01.205 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:50:01.205 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:50:01.206 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:01.206 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:50:01.206 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:01.206 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:50:01.207 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:01.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:01.207 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:01.207 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:01.207 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:01.208 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:50:01.208 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024790", 2024-04-20T01:50:01.208 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:01.208 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:01.208 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:01.209 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:41:41.923131Z" 2024-04-20T01:50:01.209 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:01.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:01.209 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:01.209 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:01.210 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:01.210 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:50:01.210 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:50:01.210 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:01.211 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:01.211 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:01.211 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:50:01.211 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:01.211 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:01.212 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:01.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:01.212 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:01.212 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:50:01.212 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:50:01.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:01.213 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:01Z" 2024-04-20T01:50:01.213 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:01.215 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015203", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:41:00.554653Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024790", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:41:41.923131Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:50:01Z" } 2024-04-20T01:50:01.215 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024790', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:41.923131Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:50:01.215 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:50:01.215 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024790', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:50:01.216 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:50:01.216 DEBUG:boto:path=/aktlji-88/ 2024-04-20T01:50:01.216 DEBUG:boto:auth_path=/aktlji-88/ 2024-04-20T01:50:01.217 DEBUG:boto:Method: HEAD 2024-04-20T01:50:01.217 DEBUG:boto:Path: /aktlji-88/ 2024-04-20T01:50:01.217 DEBUG:boto:Data: 2024-04-20T01:50:01.217 DEBUG:boto:Headers: {} 2024-04-20T01:50:01.218 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:01.218 DEBUG:boto:Port: 8000 2024-04-20T01:50:01.218 DEBUG:boto:Params: {} 2024-04-20T01:50:01.219 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:50:01.219 DEBUG:boto:Token: None 2024-04-20T01:50:01.220 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:01 GMT /aktlji-88/ 2024-04-20T01:50:01.220 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zpNYFjGrPxndE/hLzgtXK1cGQFE= 2024-04-20T01:50:01.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zpNYFjGrPxndE/hLzgtXK1cGQFE=', 'Content-Length': '0'} 2024-04-20T01:50:01.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2b0a843ed2de213-0066231f49-267b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:01.230 DEBUG:boto:path=/aktlji-88/obj5 2024-04-20T01:50:01.230 DEBUG:boto:auth_path=/aktlji-88/obj5 2024-04-20T01:50:01.231 DEBUG:boto:Method: HEAD 2024-04-20T01:50:01.231 DEBUG:boto:Path: /aktlji-88/obj5 2024-04-20T01:50:01.231 DEBUG:boto:Data: 2024-04-20T01:50:01.231 DEBUG:boto:Headers: {} 2024-04-20T01:50:01.231 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:01.232 DEBUG:boto:Port: 8000 2024-04-20T01:50:01.232 DEBUG:boto:Params: {} 2024-04-20T01:50:01.232 DEBUG:boto:Token: None 2024-04-20T01:50:01.232 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:01 GMT /aktlji-88/obj5 2024-04-20T01:50:01.233 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:AwQEScxopcBZ6e+3X3GPvG/FrS4= 2024-04-20T01:50:01.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AwQEScxopcBZ6e+3X3GPvG/FrS4=', 'Content-Length': '0'} 2024-04-20T01:50:01.235 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000090c2747ed9d509bd-0066231f49-267b-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:01.235 DEBUG:boto:path=/aktlji-88/obj6 2024-04-20T01:50:01.235 DEBUG:boto:auth_path=/aktlji-88/obj6 2024-04-20T01:50:01.235 DEBUG:boto:Method: HEAD 2024-04-20T01:50:01.236 DEBUG:boto:Path: /aktlji-88/obj6 2024-04-20T01:50:01.236 DEBUG:boto:Data: 2024-04-20T01:50:01.236 DEBUG:boto:Headers: {} 2024-04-20T01:50:01.236 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:01.236 DEBUG:boto:Port: 8000 2024-04-20T01:50:01.237 DEBUG:boto:Params: {} 2024-04-20T01:50:01.237 DEBUG:boto:Token: None 2024-04-20T01:50:01.237 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:01 GMT /aktlji-88/obj6 2024-04-20T01:50:01.237 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ND7R8CBf3UlDtlgHMigLhP3GXLA= 2024-04-20T01:50:01.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ND7R8CBf3UlDtlgHMigLhP3GXLA=', 'Content-Length': '0'} 2024-04-20T01:50:01.240 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000003a0cf408689fa2cc-0066231f49-267b-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:01.240 DEBUG:boto:path=/aktlji-89/ 2024-04-20T01:50:01.240 DEBUG:boto:auth_path=/aktlji-89/ 2024-04-20T01:50:01.240 DEBUG:boto:Method: HEAD 2024-04-20T01:50:01.240 DEBUG:boto:Path: /aktlji-89/ 2024-04-20T01:50:01.241 DEBUG:boto:Data: 2024-04-20T01:50:01.241 DEBUG:boto:Headers: {} 2024-04-20T01:50:01.241 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:01.241 DEBUG:boto:Port: 8000 2024-04-20T01:50:01.241 DEBUG:boto:Params: {} 2024-04-20T01:50:01.242 DEBUG:boto:Token: None 2024-04-20T01:50:01.242 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:01 GMT /aktlji-89/ 2024-04-20T01:50:01.242 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:j9Gbel3gpunIRTtf6uawY1Ni/gw= 2024-04-20T01:50:01.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:j9Gbel3gpunIRTtf6uawY1Ni/gw=', 'Content-Length': '0'} 2024-04-20T01:50:01.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8462a286bde5789-0066231f49-267b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:01.245 DEBUG:boto:path=/aktlji-89/obj5 2024-04-20T01:50:01.246 DEBUG:boto:auth_path=/aktlji-89/obj5 2024-04-20T01:50:01.246 DEBUG:boto:Method: HEAD 2024-04-20T01:50:01.246 DEBUG:boto:Path: /aktlji-89/obj5 2024-04-20T01:50:01.246 DEBUG:boto:Data: 2024-04-20T01:50:01.247 DEBUG:boto:Headers: {} 2024-04-20T01:50:01.247 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:01.247 DEBUG:boto:Port: 8000 2024-04-20T01:50:01.247 DEBUG:boto:Params: {} 2024-04-20T01:50:01.248 DEBUG:boto:Token: None 2024-04-20T01:50:01.248 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:01 GMT /aktlji-89/obj5 2024-04-20T01:50:01.248 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:M7+GWJviHbNhTkX9JQZHDFdEiXw= 2024-04-20T01:50:01.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:M7+GWJviHbNhTkX9JQZHDFdEiXw=', 'Content-Length': '0'} 2024-04-20T01:50:01.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:48:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-88:73f46f02-c7ab-4426-a267-2273af911da5.7121.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c6cc598608f8cfb-0066231f49-267b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:01.251 DEBUG:boto:path=/aktlji-89/obj5 2024-04-20T01:50:01.251 DEBUG:boto:auth_path=/aktlji-89/obj5 2024-04-20T01:50:01.252 DEBUG:boto:Method: GET 2024-04-20T01:50:01.252 DEBUG:boto:Path: /aktlji-89/obj5 2024-04-20T01:50:01.252 DEBUG:boto:Data: 2024-04-20T01:50:01.252 DEBUG:boto:Headers: {} 2024-04-20T01:50:01.252 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:01.253 DEBUG:boto:Port: 8000 2024-04-20T01:50:01.253 DEBUG:boto:Params: {} 2024-04-20T01:50:01.253 DEBUG:boto:Token: None 2024-04-20T01:50:01.253 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:50:01 GMT /aktlji-89/obj5 2024-04-20T01:50:01.254 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:tgQguFLBWQh7oo+ySIJ9/TVYEgA= 2024-04-20T01:50:01.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tgQguFLBWQh7oo+ySIJ9/TVYEgA=', 'Content-Length': '0'} 2024-04-20T01:50:01.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:48:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-88:73f46f02-c7ab-4426-a267-2273af911da5.7121.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7953fb390c0a198-0066231f49-267b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:01.298 DEBUG:boto:path=/aktlji-89/obj6 2024-04-20T01:50:01.298 DEBUG:boto:auth_path=/aktlji-89/obj6 2024-04-20T01:50:01.299 DEBUG:boto:Method: HEAD 2024-04-20T01:50:01.299 DEBUG:boto:Path: /aktlji-89/obj6 2024-04-20T01:50:01.299 DEBUG:boto:Data: 2024-04-20T01:50:01.299 DEBUG:boto:Headers: {} 2024-04-20T01:50:01.299 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:01.299 DEBUG:boto:Port: 8000 2024-04-20T01:50:01.299 DEBUG:boto:Params: {} 2024-04-20T01:50:01.299 DEBUG:boto:Token: None 2024-04-20T01:50:01.299 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:01 GMT /aktlji-89/obj6 2024-04-20T01:50:01.299 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:+1qqPyO+0OBUNNyS2ygWMtlhiKo= 2024-04-20T01:50:01.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+1qqPyO+0OBUNNyS2ygWMtlhiKo=', 'Content-Length': '0'} 2024-04-20T01:50:01.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:48:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-88:73f46f02-c7ab-4426-a267-2273af911da5.7121.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000401126f24be46043-0066231f49-267b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:01.302 DEBUG:boto:path=/aktlji-89/obj6 2024-04-20T01:50:01.302 DEBUG:boto:auth_path=/aktlji-89/obj6 2024-04-20T01:50:01.302 DEBUG:boto:Method: GET 2024-04-20T01:50:01.302 DEBUG:boto:Path: /aktlji-89/obj6 2024-04-20T01:50:01.302 DEBUG:boto:Data: 2024-04-20T01:50:01.302 DEBUG:boto:Headers: {} 2024-04-20T01:50:01.302 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:01.302 DEBUG:boto:Port: 8000 2024-04-20T01:50:01.302 DEBUG:boto:Params: {} 2024-04-20T01:50:01.303 DEBUG:boto:Token: None 2024-04-20T01:50:01.303 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:50:01 GMT /aktlji-89/obj6 2024-04-20T01:50:01.303 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WLshzD41OovUS+gVYaw7okjTMwo= 2024-04-20T01:50:01.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WLshzD41OovUS+gVYaw7okjTMwo=', 'Content-Length': '0'} 2024-04-20T01:50:01.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:48:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-88:73f46f02-c7ab-4426-a267-2273af911da5.7121.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9d70a4e627285b7-0066231f49-267b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:50:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:01.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:01.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:01.306 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket aktlji-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:50:01.365 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:01.365 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:01.449 INFO:teuthology.orchestra.run.smithi019.stdout:{"sync_policy":{"groups":[]}} 2024-04-20T01:50:01.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:01.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:01.449 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:01.503 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:01.504 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:01.543 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:50:01.560 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:50:01.560 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:01.561 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:01.561 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:50:01.561 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:01.561 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:01.561 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:50:01.561 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:50:01.561 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:01.561 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:01.561 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577801.439434_1708.1", 2024-04-20T01:50:01.561 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:50:01.439434Z" 2024-04-20T01:50:01.562 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:01.562 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:01.562 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:01.562 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:50:01.562 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:01.562 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:50:01.563 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577801.439434_1708.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:01.563 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:50:01.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:01.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:01.564 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:01.623 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:01.624 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:01.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:01.688+0000 7ff164a99780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:01.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:01.689+0000 7ff164a99780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:01.698 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:01.698 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:01.698 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:01.698 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:01.698 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:01.699 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:01.699 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:01.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:01.699 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:01.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:01.699 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:01.699 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:01.699 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:01.700 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:01.700 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:01.700 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:01.700 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:01.700 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:01.700 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:01.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:01.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:01.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:01.700 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:01.700 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:01.701 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:01.701 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:01.701 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:01.701 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:01.701 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:01.701 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:01.701 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:01.701 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:01.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:01.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:01.701 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:01.702 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:01.702 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:01.702 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577676.516091_1706.1", 2024-04-20T01:50:01.702 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:01.702 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:01.702 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:01.702 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:47:56.516091Z", 2024-04-20T01:50:01.702 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:01.702 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:01.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:01.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:01.703 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:01.703 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:01.703 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:01.703 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:01.703 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:01.703 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:01.703 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:01.703 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:01.703 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:01.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:01.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:01.704 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:01.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:01.704 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:01.704 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:01.704 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:01.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:01.704 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:01Z" 2024-04-20T01:50:01.704 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:01.705 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577676.516091_1706.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:47:56.516091Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:01Z" } 2024-04-20T01:50:01.705 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577676.516091_1706.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:47:56.516091Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:01.706 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577801.439434_1708.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:01.706 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:01.706 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577801.439434_1708.1 target=1_1713577676.516091_1706.1 2024-04-20T01:50:06.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:06.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:06.707 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:06.767 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:06.768 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:06.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:06.826+0000 7fcf522f5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:06.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:06.829+0000 7fcf522f5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:06.835 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:06.835 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:06.835 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:06.835 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:06.835 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:06.835 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:06.835 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:06.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:06.836 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:06.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:06.836 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:06.836 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:06.836 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:06.836 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:06.836 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:06.836 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:06.836 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:06.836 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:06.836 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:06.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:06.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:06.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:06.837 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:06.837 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:06.837 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:06.837 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:06.837 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:06.837 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:06.837 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:06.837 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577676.516091_1706.1", 2024-04-20T01:50:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:47:56.516091Z", 2024-04-20T01:50:06.839 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:06.839 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:06.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:06.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:06.839 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:06.839 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:06.839 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:06.839 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:06.839 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:06.839 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:06.839 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:06.840 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:06.840 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:06.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:06.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:06.840 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:06.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:06.840 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:06.840 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:06.840 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:06.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:06.840 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:06Z" 2024-04-20T01:50:06.840 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:06.841 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577676.516091_1706.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:47:56.516091Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:06Z" } 2024-04-20T01:50:06.842 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577676.516091_1706.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:47:56.516091Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:06.842 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577801.439434_1708.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:06.842 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577676.516091_1706.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:06.842 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577801.439434_1708.1 target=1_1713577676.516091_1706.1 2024-04-20T01:50:11.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:11.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:11.846 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:11.904 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:11.904 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:11.969+0000 7f53091f5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:11.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:11.970+0000 7f53091f5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:11.978 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:11.978 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:11.978 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:11.979 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:11.979 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:11.979 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:11.979 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:11.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:11.979 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:11.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:11.979 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:11.979 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:11.979 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:11.980 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:11.980 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:11.980 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:11.980 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:11.980 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:11.980 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:11.980 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:11.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:11.980 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:11.980 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:11.980 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:11.981 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:11.981 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:11.981 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:11.981 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:11.981 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:11.981 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:11.981 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:11.981 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:11.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:11.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:11.982 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:11.982 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:11.982 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:11.982 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577801.439434_1708.1", 2024-04-20T01:50:11.982 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:11.982 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:11.982 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:11.982 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:01.439434Z", 2024-04-20T01:50:11.982 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:11.982 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:11.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:11.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:11.983 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:11.983 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:11.983 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:11.983 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:11.983 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:11.983 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:11.983 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:11.983 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:11.983 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:11.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:11.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:11.984 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:11.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:11.984 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:11.984 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:11.984 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:11.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:11.984 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:11Z" 2024-04-20T01:50:11.984 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:11.985 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577801.439434_1708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:01.439434Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:11Z" } 2024-04-20T01:50:11.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577801.439434_1708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:01.439434Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:11.986 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577801.439434_1708.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:11.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577801.439434_1708.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:11.986 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:50:11.986 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:50:11.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:11.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:11.987 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:12.046 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:12.046 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:12.096 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:50:12.110+0000 7f76a0688780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:12.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:50:12.112+0000 7f76a0688780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:12.117 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:50:12.117 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:50:12.117 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:50:12.117 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:50:12.117 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:50:12.117 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:12.117 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:12.118 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:12.118 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:50:12.118 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:12.118 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:50:12.118 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:12.118 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:50:12.118 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:12.118 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:12.118 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:50:12.118 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:12.118 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:12.119 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:12.119 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:12.119 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:12.119 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:12.119 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:50:12.119 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:12.119 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:50:12.119 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:50:12.119 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:12.119 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:12.119 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:12.119 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:50:12.120 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:12.120 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:12.120 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:12.120 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:12.120 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:50:12.120 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:12.120 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:50:12.120 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577801.439434_1708.1", 2024-04-20T01:50:12.120 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:12.120 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:12.120 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:12.121 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:50:01.439434Z", 2024-04-20T01:50:12.121 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:12.121 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:12.121 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:12.121 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:12.121 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:50:12.121 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:12.121 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:50:12.121 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:12.121 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:12.121 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:12.121 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:12.122 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:12.122 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:12.122 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:12.122 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:12.122 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:50:12.122 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:12.122 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:50:12.122 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:50:12.122 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:50:12.122 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:12.122 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:50:12Z" 2024-04-20T01:50:12.123 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:50:12.123 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577801.439434_1708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:01.439434Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:12Z" } 2024-04-20T01:50:12.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577801.439434_1708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:01.439434Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:12.124 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577801.439434_1708.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:12.124 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577801.439434_1708.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:12.124 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:50:12.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:12.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:12.124 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket aktlji-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:50:12.179 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:12.179 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:12.249 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[]} 2024-04-20T01:50:12.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:12.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:12.249 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket aktlji-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:50:12.307 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:12.307 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:12.373 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-20T01:50:12.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:12.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:12.374 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=test-zone1,test-zone2 --bucket aktlji-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:50:12.435 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:12.435 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:12.507 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-20T01:50:12.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=aktlji-88', '--source-bucket=aktlji-89', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:12.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--bucket=aktlji-88', '--source-bucket=aktlji-89', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:12.508 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --bucket=aktlji-88 --source-bucket=aktlji-89 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:50:12.568 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:12.568 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:12.641 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"aktlji-89","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-20T01:50:12.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:12.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:12.642 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket aktlji-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:50:12.700 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:12.700 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:12.777 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"aktlji-89","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-20T01:50:12.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:12.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:12.778 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket aktlji-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:50:12.831 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:12.832 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:12.904 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"aktlji-89","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-20T01:50:12.904 DEBUG:boto:path=/aktlji-88/ 2024-04-20T01:50:12.904 DEBUG:boto:auth_path=/aktlji-88/ 2024-04-20T01:50:12.904 DEBUG:boto:Method: HEAD 2024-04-20T01:50:12.904 DEBUG:boto:Path: /aktlji-88/ 2024-04-20T01:50:12.905 DEBUG:boto:Data: 2024-04-20T01:50:12.905 DEBUG:boto:Headers: {} 2024-04-20T01:50:12.905 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:12.905 DEBUG:boto:Port: 8000 2024-04-20T01:50:12.905 DEBUG:boto:Params: {} 2024-04-20T01:50:12.905 DEBUG:boto:Token: None 2024-04-20T01:50:12.905 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:12 GMT /aktlji-88/ 2024-04-20T01:50:12.905 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OUeV2XXom98/M4tcbqot2etI/c0= 2024-04-20T01:50:12.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OUeV2XXom98/M4tcbqot2etI/c0=', 'Content-Length': '0'} 2024-04-20T01:50:12.908 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ba7bfbb359c77d8-0066231f54-267b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:12.908 DEBUG:boto:path=/aktlji-88/obj7 2024-04-20T01:50:12.909 DEBUG:boto:auth_path=/aktlji-88/obj7 2024-04-20T01:50:12.909 DEBUG:boto:Method: PUT 2024-04-20T01:50:12.909 DEBUG:boto:Path: /aktlji-88/obj7 2024-04-20T01:50:12.909 DEBUG:boto:Data: 2024-04-20T01:50:12.909 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:50:12.909 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:12.909 DEBUG:boto:Port: 8000 2024-04-20T01:50:12.909 DEBUG:boto:Params: {} 2024-04-20T01:50:12.909 DEBUG:boto:Token: None 2024-04-20T01:50:12.910 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:50:12 GMT /aktlji-88/obj7 2024-04-20T01:50:12.910 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:CvR5UNazMO0RSiAct85SvxLXZRU= 2024-04-20T01:50:12.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:50:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CvR5UNazMO0RSiAct85SvxLXZRU='} 2024-04-20T01:50:12.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c360616ccbcecd4-0066231f54-267b-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:12.920 DEBUG:boto:path=/aktlji-88/ 2024-04-20T01:50:12.920 DEBUG:boto:auth_path=/aktlji-88/ 2024-04-20T01:50:12.920 DEBUG:boto:Method: HEAD 2024-04-20T01:50:12.921 DEBUG:boto:Path: /aktlji-88/ 2024-04-20T01:50:12.921 DEBUG:boto:Data: 2024-04-20T01:50:12.921 DEBUG:boto:Headers: {} 2024-04-20T01:50:12.921 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:12.921 DEBUG:boto:Port: 8000 2024-04-20T01:50:12.921 DEBUG:boto:Params: {} 2024-04-20T01:50:12.921 DEBUG:boto:Token: None 2024-04-20T01:50:12.921 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:12 GMT /aktlji-88/ 2024-04-20T01:50:12.921 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OUeV2XXom98/M4tcbqot2etI/c0= 2024-04-20T01:50:12.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OUeV2XXom98/M4tcbqot2etI/c0=', 'Content-Length': '0'} 2024-04-20T01:50:12.924 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044406e62331d5aee-0066231f54-267b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:12.924 DEBUG:boto:path=/aktlji-88/obj8 2024-04-20T01:50:12.924 DEBUG:boto:auth_path=/aktlji-88/obj8 2024-04-20T01:50:12.924 DEBUG:boto:Method: PUT 2024-04-20T01:50:12.925 DEBUG:boto:Path: /aktlji-88/obj8 2024-04-20T01:50:12.925 DEBUG:boto:Data: 2024-04-20T01:50:12.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:50:12.925 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:12.925 DEBUG:boto:Port: 8000 2024-04-20T01:50:12.925 DEBUG:boto:Params: {} 2024-04-20T01:50:12.925 DEBUG:boto:Token: None 2024-04-20T01:50:12.925 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:50:12 GMT /aktlji-88/obj8 2024-04-20T01:50:12.926 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/Ss7fvcly1bSsLgHv7p1I1Ti17Y= 2024-04-20T01:50:12.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:50:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/Ss7fvcly1bSsLgHv7p1I1Ti17Y='} 2024-04-20T01:50:12.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004453138f43072103-0066231f54-267b-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:12.971 DEBUG:boto:path=/aktlji-89/ 2024-04-20T01:50:12.971 DEBUG:boto:auth_path=/aktlji-89/ 2024-04-20T01:50:12.971 DEBUG:boto:Method: HEAD 2024-04-20T01:50:12.971 DEBUG:boto:Path: /aktlji-89/ 2024-04-20T01:50:12.971 DEBUG:boto:Data: 2024-04-20T01:50:12.971 DEBUG:boto:Headers: {} 2024-04-20T01:50:12.971 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:12.971 DEBUG:boto:Port: 8000 2024-04-20T01:50:12.972 DEBUG:boto:Params: {} 2024-04-20T01:50:12.972 DEBUG:boto:Token: None 2024-04-20T01:50:12.972 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:12 GMT /aktlji-89/ 2024-04-20T01:50:12.972 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gpvQxL/akbO36My90hsf2hxutY8= 2024-04-20T01:50:12.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gpvQxL/akbO36My90hsf2hxutY8=', 'Content-Length': '0'} 2024-04-20T01:50:12.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000683cd8bd448110cb-0066231f54-267b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:50:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:12.975 DEBUG:boto:path=/aktlji-89/obj9 2024-04-20T01:50:12.975 DEBUG:boto:auth_path=/aktlji-89/obj9 2024-04-20T01:50:12.975 DEBUG:boto:Method: PUT 2024-04-20T01:50:12.975 DEBUG:boto:Path: /aktlji-89/obj9 2024-04-20T01:50:12.975 DEBUG:boto:Data: 2024-04-20T01:50:12.975 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:50:12.975 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:12.975 DEBUG:boto:Port: 8000 2024-04-20T01:50:12.975 DEBUG:boto:Params: {} 2024-04-20T01:50:12.976 DEBUG:boto:Token: None 2024-04-20T01:50:12.976 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:50:12 GMT /aktlji-89/obj9 2024-04-20T01:50:12.976 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:73/7n2t5YPZVLeqlExECA+JdqcE= 2024-04-20T01:50:12.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:50:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:73/7n2t5YPZVLeqlExECA+JdqcE='} 2024-04-20T01:50:12.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042da8e85aa1cce42-0066231f54-267b-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:12.985 DEBUG:boto:path=/aktlji-89/ 2024-04-20T01:50:12.985 DEBUG:boto:auth_path=/aktlji-89/ 2024-04-20T01:50:12.985 DEBUG:boto:Method: HEAD 2024-04-20T01:50:12.986 DEBUG:boto:Path: /aktlji-89/ 2024-04-20T01:50:12.986 DEBUG:boto:Data: 2024-04-20T01:50:12.986 DEBUG:boto:Headers: {} 2024-04-20T01:50:12.986 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:12.986 DEBUG:boto:Port: 8000 2024-04-20T01:50:12.986 DEBUG:boto:Params: {} 2024-04-20T01:50:12.986 DEBUG:boto:Token: None 2024-04-20T01:50:12.986 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:12 GMT /aktlji-89/ 2024-04-20T01:50:12.986 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:gpvQxL/akbO36My90hsf2hxutY8= 2024-04-20T01:50:12.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gpvQxL/akbO36My90hsf2hxutY8=', 'Content-Length': '0'} 2024-04-20T01:50:12.989 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f76c445bec6ab49-0066231f55-267b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:12.989 DEBUG:boto:path=/aktlji-89/obj10 2024-04-20T01:50:12.989 DEBUG:boto:auth_path=/aktlji-89/obj10 2024-04-20T01:50:12.990 DEBUG:boto:Method: PUT 2024-04-20T01:50:12.990 DEBUG:boto:Path: /aktlji-89/obj10 2024-04-20T01:50:12.990 DEBUG:boto:Data: 2024-04-20T01:50:12.990 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:50:12.990 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:50:12.990 DEBUG:boto:Port: 8000 2024-04-20T01:50:12.990 DEBUG:boto:Params: {} 2024-04-20T01:50:12.990 DEBUG:boto:Token: None 2024-04-20T01:50:12.990 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:50:12 GMT /aktlji-89/obj10 2024-04-20T01:50:12.991 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:6iPBJfR2V1wq7HrVpp/r3DkdT0k= 2024-04-20T01:50:12.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:50:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6iPBJfR2V1wq7HrVpp/r3DkdT0k='} 2024-04-20T01:50:13.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000063a6df43327ec91-0066231f55-267b-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:50:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:13.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:13.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:13.035 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:13.095 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:13.095 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:13.134 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:50:13.151 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:50:13.151 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:13.151 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:13.151 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:50:13.151 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:13.152 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:13.152 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:50:13.152 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:50:13.152 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:13.152 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:13.152 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:50:13.152 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:50:12.772163Z" 2024-04-20T01:50:13.152 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:13.152 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:13.152 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:13.152 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:50:13.153 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:13.153 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:50:13.154 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:13.154 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:50:13.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:13.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:13.154 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:13.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:13.269+0000 7fa8f01fa780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:13.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:13.270+0000 7fa8f01fa780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:13.273 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:13.273 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:13.273 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:13.273 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:13.273 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:13.273 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:13.273 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:13.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:13.274 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:13.274 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:13.274 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:13.275 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:13.275 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:13.275 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:13.275 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:13.275 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:13.275 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:13.275 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:13.276 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:13.276 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:13.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:13.276 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:13.276 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:13.276 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:13.276 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:13.276 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:13.276 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:13.276 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:13.276 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:13.276 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:13.277 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:13.277 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:13.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:13.277 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:13.277 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:13.277 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:13.277 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:13.277 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577801.439434_1708.1", 2024-04-20T01:50:13.277 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:13.277 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:13.278 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:13.278 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:01.439434Z", 2024-04-20T01:50:13.278 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:13.278 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:13.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:13.278 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:13.279 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:13.279 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:13.279 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:13.279 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:13.279 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:13.280 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:13.280 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:13.280 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:13.280 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:13.280 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:13.280 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:13.280 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:13.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:13.280 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:13.280 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:13.280 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:13.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:13.281 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:13Z" 2024-04-20T01:50:13.281 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:13.282 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577801.439434_1708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:01.439434Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:13Z" } 2024-04-20T01:50:13.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577801.439434_1708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:01.439434Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:13.282 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:13.282 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577801.439434_1708.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:13.282 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577812.772163_1716.1 target=1_1713577801.439434_1708.1 2024-04-20T01:50:18.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:18.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:18.284 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:18.343 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:18.343 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:18.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:18.407+0000 7f88585ef780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:18.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:18.409+0000 7f88585ef780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:18.419 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:18.419 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:18.419 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:18.420 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:18.420 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:18.420 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:18.420 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:18.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:18.420 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:18.420 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:18.420 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:18.420 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:18.420 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:18.420 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:18.421 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:18.421 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:18.421 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:18.421 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:18.421 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:18.421 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:18.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:18.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:18.421 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:18.421 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:18.421 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:18.421 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:18.422 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:18.422 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:18.422 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:18.422 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:18.422 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:18.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:18.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:18.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:18.422 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:18.422 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:18.422 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:18.423 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577801.439434_1708.1", 2024-04-20T01:50:18.423 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:18.423 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:18.423 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:18.423 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:01.439434Z", 2024-04-20T01:50:18.423 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:18.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:18.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:18.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:18.423 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:18.423 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:18.424 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:18.424 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:18.424 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:18.424 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:18.424 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:18.424 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:18.424 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:18.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:18.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:18.424 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:18.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:18.424 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:18.425 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:18.425 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:18.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:18.425 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:18Z" 2024-04-20T01:50:18.425 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:18.426 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577801.439434_1708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:01.439434Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:18Z" } 2024-04-20T01:50:18.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577801.439434_1708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:01.439434Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:18.426 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:18.426 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577801.439434_1708.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:18.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577812.772163_1716.1 target=1_1713577801.439434_1708.1 2024-04-20T01:50:23.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:23.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:23.427 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:23.487 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:23.487 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:23.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:23.554+0000 7efcacc5c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:23.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:23.555+0000 7efcacc5c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:23.560 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577801.439434_1708.1", 2024-04-20T01:50:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:01.439434Z", 2024-04-20T01:50:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:23.566 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:23Z" 2024-04-20T01:50:23.566 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:23.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577801.439434_1708.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:01.439434Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:23Z" } 2024-04-20T01:50:23.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577801.439434_1708.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:01.439434Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:23.567 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:23.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577801.439434_1708.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:23.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577812.772163_1716.1 target=1_1713577801.439434_1708.1 2024-04-20T01:50:28.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:28.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:28.568 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:28.628 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:28.628 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:28.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:28.693+0000 7f84915b9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:28.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:28.694+0000 7f84915b9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:28.698 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:28.699 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:28.699 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:28.699 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:28.699 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:28.699 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:28.699 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:28.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:28.699 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:28.699 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:28.699 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:28.699 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:28.699 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:28.700 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:28.700 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:28.700 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:28.700 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:28.700 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:28.700 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:28.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:28.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:28.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:28.700 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:28.700 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:28.700 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:28.701 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:28.701 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:28.701 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:28.701 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:28.701 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:28.701 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:28.701 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:28.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:28.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:28.701 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:28.701 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:28.702 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:28.702 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577812.367165_1709.1", 2024-04-20T01:50:28.702 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:28.702 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:28.702 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:28.702 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:12.367165Z", 2024-04-20T01:50:28.702 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:28.702 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:28.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:28.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:28.703 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:28.703 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:28.703 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:28.703 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:28.703 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:28.703 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:28.703 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:28.703 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:28.703 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:28.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:28.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:28.704 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:28.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:28.704 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:28.704 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:28.704 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:28.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:28.704 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:28Z" 2024-04-20T01:50:28.704 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:28.705 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577812.367165_1709.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:12.367165Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:28Z" } 2024-04-20T01:50:28.705 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577812.367165_1709.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.367165Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:28.705 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:28.705 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.367165_1709.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:28.706 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577812.772163_1716.1 target=1_1713577812.367165_1709.1 2024-04-20T01:50:33.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:33.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:33.707 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:33.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:33.840+0000 7f4cb2daf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:33.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:33.841+0000 7f4cb2daf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:33.854 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:33.854 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:33.855 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:33.856 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:33.857 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:50:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:50:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:33.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:33.859 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:33.860 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:33Z" 2024-04-20T01:50:33.860 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:33.861 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577812.772163_1716.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:12.772163Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:33Z" } 2024-04-20T01:50:33.861 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577812.772163_1716.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.772163Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:33.861 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:33.861 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:33.862 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:50:33.862 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:50:33.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:33.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:33.862 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:33.923 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:33.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:50:33.984+0000 7f61d0082780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:33.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:50:33.986+0000 7f61d0082780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:33.988 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:50:33.988 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:50:33.988 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:50:33.989 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:50:33.989 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:50:33.989 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:33.989 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:33.989 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:33.989 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:50:33.989 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:33.989 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:50:33.989 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:33.989 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:50:33.989 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:33.989 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:33.990 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:50:33.990 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:33.990 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:33.990 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:33.990 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:33.990 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:33.990 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:33.990 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:50:33.990 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:33.990 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:50:33.991 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:50:33.991 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:33.991 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:33.991 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:33.991 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:50:33.991 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:33.991 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:33.991 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:33.992 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:33.992 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:50:33.992 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:33.992 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:50:33.992 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:50:33.992 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:33.992 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:33.992 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:33.992 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:50:33.992 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:33.992 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:33.992 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:33.993 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:33.993 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:50:33.993 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:33.993 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:50:33.993 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:33.993 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:33.993 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:33.993 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:33.993 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:33.993 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:33.993 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:33.994 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:33.994 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:50:33.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:33.994 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:50:33.994 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:50:33.994 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:50:33.994 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:33.994 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:50:33Z" 2024-04-20T01:50:33.994 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:50:33.995 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577812.772163_1716.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:12.772163Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:33Z" } 2024-04-20T01:50:33.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577812.772163_1716.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.772163Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:33.996 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:33.996 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:33.996 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:50:33.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:33.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:33.996 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:34.055 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:34.055 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:34.139 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-04-20T01:50:34.139 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:34.139 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000007694", 2024-04-20T01:50:34.139 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-20T01:50:13.012288Z" 2024-04-20T01:50:34.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:34.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:34.140 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000014169", 2024-04-20T01:50:34.140 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-20T01:50:12.932726Z" 2024-04-20T01:50:34.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:34.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:34.140 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000012910", 2024-04-20T01:50:34.140 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-20T01:50:12.999339Z" 2024-04-20T01:50:34.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:34.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:34.140 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000009544", 2024-04-20T01:50:34.141 INFO:teuthology.orchestra.run.smithi031.stdout: "last_update": "2024-04-20T01:49:56.803591Z" 2024-04-20T01:50:34.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:34.141 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-04-20T01:50:34.142 DEBUG:rgw_multi.tests:data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000007694', 1: '00000000000000000000:00000000000000014169', 2: '00000000000000000000:00000000000000012910', 3: '00000000000000000000:00000000000000009544'} 2024-04-20T01:50:34.142 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-20T01:50:34.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:34.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:34.142 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:34.204 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:34.204 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:34.274 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:50:34.274 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:50:34.274 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:50:34.274 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:50:34.274 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:50:34.274 INFO:teuthology.orchestra.run.smithi019.stdout: "instance_id": 11239922052735367740 2024-04-20T01:50:34.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:34.275 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:50:34.275 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:34.275 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:50:34.275 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:34.275 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:50:34.275 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000007694", 2024-04-20T01:50:34.275 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:34.275 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:34.275 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:34.275 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:50:13.012288Z" 2024-04-20T01:50:34.275 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:34.275 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:34.276 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:34.276 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:50:34.276 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:34.276 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:50:34.276 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000014169", 2024-04-20T01:50:34.276 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:34.276 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:34.276 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:34.276 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:50:12.932726Z" 2024-04-20T01:50:34.276 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:34.276 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:34.276 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000012910", 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:50:12.999339Z" 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:50:34.277 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:34.278 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "incremental-sync", 2024-04-20T01:50:34.278 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000009544", 2024-04-20T01:50:34.278 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:34.278 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:34.278 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:34.278 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:49:56.803591Z" 2024-04-20T01:50:34.278 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:34.278 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:34.278 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:50:34.278 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:34.279 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:50:34.279 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 0, 2024-04-20T01:50:34.279 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 0 2024-04-20T01:50:34.279 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:34.279 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:50:34Z" 2024-04-20T01:50:34.279 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:50:34.280 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11239922052735367740 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007694", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:13.012288Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014169", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:12.932726Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012910", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:12.999339Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009544", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:49:56.803591Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-20T01:50:34Z" } 2024-04-20T01:50:34.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007694', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:13.012288Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014169', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.932726Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012910', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.999339Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009544', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:49:56.803591Z'}}] 2024-04-20T01:50:34.280 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007694', 1: '00000000000000000000:00000000000000014169', 2: '00000000000000000000:00000000000000012910', 3: '00000000000000000000:00000000000000009544'} 2024-04-20T01:50:34.280 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007694', 1: '00000000000000000000:00000000000000014169', 2: '00000000000000000000:00000000000000012910', 3: '00000000000000000000:00000000000000009544'} 2024-04-20T01:50:34.280 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-20T01:50:34.281 DEBUG:boto:path=/aktlji-88/ 2024-04-20T01:50:34.281 DEBUG:boto:auth_path=/aktlji-88/ 2024-04-20T01:50:34.281 DEBUG:boto:Method: HEAD 2024-04-20T01:50:34.281 DEBUG:boto:Path: /aktlji-88/ 2024-04-20T01:50:34.281 DEBUG:boto:Data: 2024-04-20T01:50:34.281 DEBUG:boto:Headers: {} 2024-04-20T01:50:34.281 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:50:34.281 DEBUG:boto:Port: 8000 2024-04-20T01:50:34.281 DEBUG:boto:Params: {} 2024-04-20T01:50:34.282 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:50:34.282 DEBUG:boto:Token: None 2024-04-20T01:50:34.282 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:34 GMT /aktlji-88/ 2024-04-20T01:50:34.282 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:SQGzPW238v/lbJX5+chSxUDs04E= 2024-04-20T01:50:34.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SQGzPW238v/lbJX5+chSxUDs04E=', 'Content-Length': '0'} 2024-04-20T01:50:34.287 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008919a528c0efcf60-0066231f6a-1cbe-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:34.287 DEBUG:boto:path=/aktlji-88/obj7 2024-04-20T01:50:34.287 DEBUG:boto:auth_path=/aktlji-88/obj7 2024-04-20T01:50:34.288 DEBUG:boto:Method: HEAD 2024-04-20T01:50:34.288 DEBUG:boto:Path: /aktlji-88/obj7 2024-04-20T01:50:34.288 DEBUG:boto:Data: 2024-04-20T01:50:34.288 DEBUG:boto:Headers: {} 2024-04-20T01:50:34.288 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:50:34.288 DEBUG:boto:Port: 8000 2024-04-20T01:50:34.288 DEBUG:boto:Params: {} 2024-04-20T01:50:34.288 DEBUG:boto:Token: None 2024-04-20T01:50:34.288 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:34 GMT /aktlji-88/obj7 2024-04-20T01:50:34.288 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:EjTJlr+wskY5MNPmgtd9V0bMz1Q= 2024-04-20T01:50:34.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EjTJlr+wskY5MNPmgtd9V0bMz1Q=', 'Content-Length': '0'} 2024-04-20T01:50:34.290 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a6933690fb7cb0f7-0066231f6a-1cbe-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:34.290 DEBUG:boto:path=/aktlji-88/obj8 2024-04-20T01:50:34.290 DEBUG:boto:auth_path=/aktlji-88/obj8 2024-04-20T01:50:34.290 DEBUG:boto:Method: HEAD 2024-04-20T01:50:34.290 DEBUG:boto:Path: /aktlji-88/obj8 2024-04-20T01:50:34.291 DEBUG:boto:Data: 2024-04-20T01:50:34.291 DEBUG:boto:Headers: {} 2024-04-20T01:50:34.291 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:50:34.291 DEBUG:boto:Port: 8000 2024-04-20T01:50:34.291 DEBUG:boto:Params: {} 2024-04-20T01:50:34.291 DEBUG:boto:Token: None 2024-04-20T01:50:34.291 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:34 GMT /aktlji-88/obj8 2024-04-20T01:50:34.291 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:u+rAc5hmdq4c0vxJ4HNizbMjNoI= 2024-04-20T01:50:34.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u+rAc5hmdq4c0vxJ4HNizbMjNoI=', 'Content-Length': '0'} 2024-04-20T01:50:34.293 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000072369fcaf790cf13-0066231f6a-1cbe-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:34.293 DEBUG:boto:path=/aktlji-88/obj9 2024-04-20T01:50:34.293 DEBUG:boto:auth_path=/aktlji-88/obj9 2024-04-20T01:50:34.293 DEBUG:boto:Method: HEAD 2024-04-20T01:50:34.293 DEBUG:boto:Path: /aktlji-88/obj9 2024-04-20T01:50:34.293 DEBUG:boto:Data: 2024-04-20T01:50:34.293 DEBUG:boto:Headers: {} 2024-04-20T01:50:34.294 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:50:34.294 DEBUG:boto:Port: 8000 2024-04-20T01:50:34.294 DEBUG:boto:Params: {} 2024-04-20T01:50:34.294 DEBUG:boto:Token: None 2024-04-20T01:50:34.294 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:34 GMT /aktlji-88/obj9 2024-04-20T01:50:34.294 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OgTftLnz0mVVhMoqk+veFVU0pSc= 2024-04-20T01:50:34.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OgTftLnz0mVVhMoqk+veFVU0pSc=', 'Content-Length': '0'} 2024-04-20T01:50:34.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:50:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-89:73f46f02-c7ab-4426-a267-2273af911da5.7121.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070d4f4ca16d3dc5e-0066231f6a-1cbe-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:34.296 DEBUG:boto:path=/aktlji-88/obj9 2024-04-20T01:50:34.296 DEBUG:boto:auth_path=/aktlji-88/obj9 2024-04-20T01:50:34.296 DEBUG:boto:Method: GET 2024-04-20T01:50:34.297 DEBUG:boto:Path: /aktlji-88/obj9 2024-04-20T01:50:34.297 DEBUG:boto:Data: 2024-04-20T01:50:34.297 DEBUG:boto:Headers: {} 2024-04-20T01:50:34.297 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:50:34.297 DEBUG:boto:Port: 8000 2024-04-20T01:50:34.297 DEBUG:boto:Params: {} 2024-04-20T01:50:34.297 DEBUG:boto:Token: None 2024-04-20T01:50:34.297 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:50:34 GMT /aktlji-88/obj9 2024-04-20T01:50:34.297 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:BB2p33UuCmGyKq4Hpla232KaBjM= 2024-04-20T01:50:34.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BB2p33UuCmGyKq4Hpla232KaBjM=', 'Content-Length': '0'} 2024-04-20T01:50:34.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:50:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-89:73f46f02-c7ab-4426-a267-2273af911da5.7121.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008433d8fcbf0841a-0066231f6a-1cbe-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:34.300 DEBUG:boto:path=/aktlji-88/obj10 2024-04-20T01:50:34.300 DEBUG:boto:auth_path=/aktlji-88/obj10 2024-04-20T01:50:34.300 DEBUG:boto:Method: HEAD 2024-04-20T01:50:34.300 DEBUG:boto:Path: /aktlji-88/obj10 2024-04-20T01:50:34.300 DEBUG:boto:Data: 2024-04-20T01:50:34.300 DEBUG:boto:Headers: {} 2024-04-20T01:50:34.300 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:50:34.301 DEBUG:boto:Port: 8000 2024-04-20T01:50:34.301 DEBUG:boto:Params: {} 2024-04-20T01:50:34.301 DEBUG:boto:Token: None 2024-04-20T01:50:34.301 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:50:34 GMT /aktlji-88/obj10 2024-04-20T01:50:34.301 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mzOpeU/93GD6memIiENaasFYFEA= 2024-04-20T01:50:34.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mzOpeU/93GD6memIiENaasFYFEA=', 'Content-Length': '0'} 2024-04-20T01:50:34.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:50:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-89:73f46f02-c7ab-4426-a267-2273af911da5.7121.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b0e763078aac1fc-0066231f6a-1cbe-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:34.303 DEBUG:boto:path=/aktlji-88/obj10 2024-04-20T01:50:34.303 DEBUG:boto:auth_path=/aktlji-88/obj10 2024-04-20T01:50:34.303 DEBUG:boto:Method: GET 2024-04-20T01:50:34.303 DEBUG:boto:Path: /aktlji-88/obj10 2024-04-20T01:50:34.303 DEBUG:boto:Data: 2024-04-20T01:50:34.303 DEBUG:boto:Headers: {} 2024-04-20T01:50:34.303 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:50:34.304 DEBUG:boto:Port: 8000 2024-04-20T01:50:34.304 DEBUG:boto:Params: {} 2024-04-20T01:50:34.304 DEBUG:boto:Token: None 2024-04-20T01:50:34.304 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:50:34 GMT /aktlji-88/obj10 2024-04-20T01:50:34.304 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WWpK5FbTOzeg+UfqOKPe5WnhXo4= 2024-04-20T01:50:34.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WWpK5FbTOzeg+UfqOKPe5WnhXo4=', 'Content-Length': '0'} 2024-04-20T01:50:34.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:50:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-89:73f46f02-c7ab-4426-a267-2273af911da5.7121.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1760b3333dd18f5-0066231f6a-1cbe-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:50:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:34.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:34.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'aktlji-88', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:34.347 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket aktlji-88 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:50:34.405 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:34.405 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:34.473 INFO:teuthology.orchestra.run.smithi019.stdout:{"sync_policy":{"groups":[]}} 2024-04-20T01:50:34.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:34.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:34.474 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:50:34.527 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:34.527 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:34.585 INFO:teuthology.orchestra.run.smithi019.stdout:{"sync_policy":{"groups":[]}} 2024-04-20T01:50:34.586 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ok 2024-04-20T01:50:34.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:34.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:34.588 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:34.641 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:34.641 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:34.695 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:50:34.717 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:50:34.717 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:50:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:50:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:50:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577834.466139_1718.1", 2024-04-20T01:50:34.718 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:50:34.466139Z" 2024-04-20T01:50:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:50:34.719 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:34.719 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:50:34.720 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:34.720 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:50:34.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:34.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:34.720 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:34.779 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:34.779 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:34.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:34.834+0000 7f6b1d63f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:34.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:34.835+0000 7f6b1d63f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:34.841 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:34.841 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:34.841 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:34.841 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:34.841 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:34.843 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:34.843 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:34.843 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:34.843 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:34.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:34.843 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:34.843 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:34.843 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:34.843 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:34.843 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:34.843 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:34.843 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:34.844 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:34.844 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:34.844 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:34.844 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:34.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:34.844 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:34.844 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:34.844 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:34.844 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:34.844 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:50:34.844 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:50:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:34.845 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:34.846 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:34.847 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:34Z" 2024-04-20T01:50:34.847 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:34.848 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577812.772163_1716.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:12.772163Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:34Z" } 2024-04-20T01:50:34.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577812.772163_1716.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.772163Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:34.848 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:34.848 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:34.848 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577834.466139_1718.1 target=1_1713577812.772163_1716.1 2024-04-20T01:50:39.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:39.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:39.850 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:39.910 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:39.910 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:39.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:39.983+0000 7f6224b6b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:39.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:39.985+0000 7f6224b6b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:39.994 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:39.994 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:39.994 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:39.994 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:39.994 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:39.994 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:39.994 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:39.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:39.995 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:39.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:39.995 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:39.995 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:39.995 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:39.995 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:39.995 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:39.995 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:39.995 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:39.995 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:39.995 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:39.996 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:39.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:39.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:39.996 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:39.996 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:39.996 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:39.996 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:39.996 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:39.996 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:39.996 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:39.996 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:39.996 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:39.997 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:39.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:39.997 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:39.997 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:39.997 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:39.997 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:39.997 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:50:39.997 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:39.997 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:39.997 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:39.997 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:50:39.997 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:39.998 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:39.999 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:39Z" 2024-04-20T01:50:39.999 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:40.000 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577812.772163_1716.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:12.772163Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:39Z" } 2024-04-20T01:50:40.000 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577812.772163_1716.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.772163Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:40.001 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:40.001 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:40.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577834.466139_1718.1 target=1_1713577812.772163_1716.1 2024-04-20T01:50:45.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:45.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:45.003 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:45.117+0000 7f01be384780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:45.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:45.118+0000 7f01be384780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:45.121 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:45.121 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:45.122 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:45.123 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:45.124 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:50:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:45.125 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:50:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:45.126 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:45.127 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:45.128 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:45Z" 2024-04-20T01:50:45.128 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:45.129 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577812.772163_1716.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:12.772163Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:45Z" } 2024-04-20T01:50:45.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577812.772163_1716.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.772163Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:45.129 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:45.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:45.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577834.466139_1718.1 target=1_1713577812.772163_1716.1 2024-04-20T01:50:50.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:50.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:50.131 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:50.190 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:50.190 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:50.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:50.249+0000 7f8bd843e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:50.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:50.250+0000 7f8bd843e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:50.257 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:50.257 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:50.258 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:50.259 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:50:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:50.260 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:50:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:50.261 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:50.262 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:50.263 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:50Z" 2024-04-20T01:50:50.263 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:50.264 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577812.772163_1716.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:12.772163Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:50Z" } 2024-04-20T01:50:50.264 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577812.772163_1716.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.772163Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:50.264 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:50.264 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:50.264 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577834.466139_1718.1 target=1_1713577812.772163_1716.1 2024-04-20T01:50:55.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:55.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:55.266 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:55.325 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:55.325 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:55.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:55.387+0000 7fe6b0d75780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:55.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:55.388+0000 7fe6b0d75780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:55.394 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:55.394 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:55.394 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:55.394 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:55.394 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:55.394 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:55.394 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:55.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:55.395 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:55.395 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:55.395 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:55.395 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:55.395 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:55.395 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:55.395 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:55.395 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:55.395 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:55.395 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:55.396 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:55.396 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:55.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:55.396 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:55.396 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:55.396 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:55.396 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:55.396 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:55.396 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:55.396 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:55.396 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:55.397 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:55.397 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:55.397 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:55.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:55.397 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:55.397 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:55.397 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:55.397 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:55.397 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577834.466139_1718.1", 2024-04-20T01:50:55.397 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:55.397 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:55.398 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:55.398 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:34.466139Z", 2024-04-20T01:50:55.398 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:55.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:55.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:55.398 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:55.398 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:55.398 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:55.398 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:55.398 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:55.399 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:55.399 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:55.399 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:55.399 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:55.399 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:55.399 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:55.399 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:55.399 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:55.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:55.399 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:55.400 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:55.400 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:55.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:55.400 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:55Z" 2024-04-20T01:50:55.400 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:55.401 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577834.466139_1718.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:34.466139Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:55Z" } 2024-04-20T01:50:55.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577834.466139_1718.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:34.466139Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:55.401 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:55.401 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:55.401 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:50:55.401 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:50:55.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:55.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:55.402 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:55.462 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:55.462 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:55.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:50:55.520+0000 7f1620fb9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:55.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:50:55.521+0000 7f1620fb9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:55.528 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:50:55.528 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:50:55.528 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:50:55.528 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:50:55.528 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:50:55.528 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:55.528 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:55.528 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:50:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:50:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:50:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:50:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:50:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:50:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:50:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:55.530 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:50:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:50:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:50:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577834.466139_1718.1", 2024-04-20T01:50:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:55.531 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:50:34.466139Z", 2024-04-20T01:50:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:50:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:50:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:50:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:50:55.532 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:50:55Z" 2024-04-20T01:50:55.533 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:50:55.534 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577834.466139_1718.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:34.466139Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:55Z" } 2024-04-20T01:50:55.535 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577834.466139_1718.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:34.466139Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:55.535 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:55.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:55.535 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:50:55.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:55.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:50:55.535 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:50:55.586 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:55.586 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:55.653 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[]} 2024-04-20T01:50:55.654 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-90 2024-04-20T01:50:55.654 DEBUG:boto:path=/aktlji-90/ 2024-04-20T01:50:55.654 DEBUG:boto:auth_path=/aktlji-90/ 2024-04-20T01:50:55.654 DEBUG:boto:Method: PUT 2024-04-20T01:50:55.654 DEBUG:boto:Path: /aktlji-90/ 2024-04-20T01:50:55.654 DEBUG:boto:Data: 2024-04-20T01:50:55.655 DEBUG:boto:Headers: {} 2024-04-20T01:50:55.655 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:50:55.655 DEBUG:boto:Port: 8000 2024-04-20T01:50:55.655 DEBUG:boto:Params: {} 2024-04-20T01:50:55.655 DEBUG:boto:Token: None 2024-04-20T01:50:55.655 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:50:55 GMT /aktlji-90/ 2024-04-20T01:50:55.655 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:16i0W4/ROY1NXCxd4KXb7xFNncA= 2024-04-20T01:50:55.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:55 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:16i0W4/ROY1NXCxd4KXb7xFNncA=', 'Content-Length': '0'} 2024-04-20T01:50:55.685 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000022682f4dc252fb7d-0066231f7f-1cbe-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:50:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:55.685 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-91 2024-04-20T01:50:55.685 DEBUG:boto:path=/aktlji-91/ 2024-04-20T01:50:55.685 DEBUG:boto:auth_path=/aktlji-91/ 2024-04-20T01:50:55.686 DEBUG:boto:Method: PUT 2024-04-20T01:50:55.686 DEBUG:boto:Path: /aktlji-91/ 2024-04-20T01:50:55.686 DEBUG:boto:Data: 2024-04-20T01:50:55.686 DEBUG:boto:Headers: {} 2024-04-20T01:50:55.686 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:50:55.686 DEBUG:boto:Port: 8000 2024-04-20T01:50:55.686 DEBUG:boto:Params: {} 2024-04-20T01:50:55.686 DEBUG:boto:Token: None 2024-04-20T01:50:55.686 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:50:55 GMT /aktlji-91/ 2024-04-20T01:50:55.686 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Iq7tDJv7utXl0B/jsJCOkNWEY/w= 2024-04-20T01:50:55.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:55 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Iq7tDJv7utXl0B/jsJCOkNWEY/w=', 'Content-Length': '0'} 2024-04-20T01:50:55.722 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001f3c088f3fbc49ab-0066231f7f-1cbe-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:50:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:50:55.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:55.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:55.723 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:55.785 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:55.785 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:55.824 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:50:55.843 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:50:55.843 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:55.843 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:55.844 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:50:55.844 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:55.844 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:55.844 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:50:55.844 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:50:55.844 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:55.844 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:55.844 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:50:55.844 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:50:55.729112Z" 2024-04-20T01:50:55.844 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:50:55.845 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:50:55.845 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:55.845 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:50:55.845 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:50:55.845 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:50:55.846 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:55.846 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:50:55.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:55.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:50:55.847 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:50:55.905 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:50:55.905 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:50:55.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:55.971+0000 7fc49cdad780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:55.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:50:55.972+0000 7fc49cdad780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:50:55.976 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:50:55.977 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:50:55.977 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:50:55.977 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:50:55.977 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:50:55.977 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:50:55.977 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:55.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:55.977 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:50:55.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:55.977 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:50:55.977 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:55.978 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:55.978 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:50:55.978 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:55.978 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:50:55.978 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:55.978 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:50:55.978 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:55.978 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:55.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:55.979 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:55.979 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:50:55.979 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:55.979 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:55.979 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:50:55.979 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:55.979 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:55.979 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:55.979 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:50:55.979 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:55.979 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:55.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:55.980 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:55.980 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:50:55.980 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:55.980 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:55.980 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577855.686103_1719.1", 2024-04-20T01:50:55.980 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:55.980 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:55.980 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:55.980 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:55.686103Z", 2024-04-20T01:50:55.980 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:55.980 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:50:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:50:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:50:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:50:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:50:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:50:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:50:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:50:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:50:55.981 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:50:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:50:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:50:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:50:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:50:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:50:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:50:55.982 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:50:55Z" 2024-04-20T01:50:55.982 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:50:55.983 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577855.686103_1719.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:55.686103Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:50:55Z" } 2024-04-20T01:50:55.983 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.686103_1719.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.686103Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:50:55.984 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:55.984 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.686103_1719.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:50:55.984 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1713577855.729112_1726.1 target=1_1713577855.686103_1719.1 2024-04-20T01:51:00.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:00.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:00.985 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:01.044 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:01.044 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:01.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:51:01.107+0000 7f08ed7d4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:51:01.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:51:01.108+0000 7f08ed7d4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:51:01.112 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:01.112 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:51:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:51:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:51:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:51:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:01.113 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:51:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:51:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:51:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:51:01.114 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:51:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:51:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:51:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:51:01.115 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:51:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:51:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:51:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:01.116 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:51:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:51:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:51:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:51:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:51:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:51:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:01.118 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:01Z" 2024-04-20T01:51:01.118 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:01.119 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577855.729112_1726.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:55.729112Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:51:01Z" } 2024-04-20T01:51:01.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.729112_1726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.729112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:51:01.120 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:51:01.120 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:51:01.120 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:51:01.120 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:51:01.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:01.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:01.120 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:01.179 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:01.180 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:01.238 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:51:01.252+0000 7f3e439c9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:51:01.239 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:51:01.253+0000 7f3e439c9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:51:01.262 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:51:01.263 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:51:01.263 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:51:01.263 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:51:01.263 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:51:01.263 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:51:01.263 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:51:01.263 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:01.263 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:51:01.263 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:01.263 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:51:01.263 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:51:01.264 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:51:01.264 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:51:01.264 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:51:01.264 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:51:01.264 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:51:01.264 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:51:01.264 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:51:01.264 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:01.265 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:01.265 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:01.265 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:51:01.265 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:51:01.265 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:51:01.265 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:51:01.265 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:51:01.265 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:51:01.265 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:51:01.265 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:51:01.265 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:51:01.265 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:01.266 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:01.266 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:01.266 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:51:01.266 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:51:01.266 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:51:01.266 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:51:01.266 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:51:01.266 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:51:01.266 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:51:01.266 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:51:01.267 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:51:01.267 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:01.267 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:01.267 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:01.267 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:51:01.267 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:51:01.267 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:51:01.267 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:51:01.267 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:51:01.267 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:51:01.267 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:51:01.267 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:51:01.268 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:51:01.268 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:01.268 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:01.268 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:01.268 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:01.268 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:51:01.268 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:51:01.268 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:51:01.268 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:01.268 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:51:01Z" 2024-04-20T01:51:01.268 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:51:01.269 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577855.729112_1726.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:55.729112Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:51:01Z" } 2024-04-20T01:51:01.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.729112_1726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.729112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:51:01.270 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:51:01.270 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:51:01.270 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:51:01.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:01.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:01.270 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:51:01.326 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:01.327 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:01.394 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-20T01:51:01.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:01.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:01.395 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:51:01.455 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:01.455 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:01.516 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"allowed"}]} 2024-04-20T01:51:01.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-90', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-90', '--dest-bucket=aktlji-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:01.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-90', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-90', '--dest-bucket=aktlji-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:01.517 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-aktlji-90 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=aktlji-90 --dest-bucket=aktlji-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:51:01.577 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:01.577 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:01.664 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:51:01.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-91', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-91', '--dest-bucket=aktlji-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:01.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-91', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-91', '--dest-bucket=aktlji-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:01.665 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-aktlji-91 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=aktlji-91 --dest-bucket=aktlji-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:51:01.728 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:01.728 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:01.808 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:51:01.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:01.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:01.809 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:51:01.865 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:01.866 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:01.926 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:51:01.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:01.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:01.927 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:51:01.980 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:01.980 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:02.048 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:51:02.048 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:51:02.048 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 18, 2024-04-20T01:51:02.048 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:51:02.049 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T01:51:02.049 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T01:51:02.049 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:51:02.049 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T01:51:02.049 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.049 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T01:51:02.049 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T01:51:02.049 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T01:51:02.049 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T01:51:02.049 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:51:02.050 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:51:02.050 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:51:02.050 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T01:51:02.050 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T01:51:02.050 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:51:02.050 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:51:02.050 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.050 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:51:02.050 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T01:51:02.050 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:51:02.050 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:51:02.051 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:51:02.051 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:51:02.051 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:51:02.051 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:51:02.051 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:51:02.051 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:51:02.051 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T01:51:02.051 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:51:02.051 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:51:02.051 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:51:02.052 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:51:02.052 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:51:02.052 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:02.052 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.052 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.052 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:51:02.052 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T01:51:02.052 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:51:02.052 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:51:02.052 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:51:02.052 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:51:02.053 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:51:02.053 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:51:02.053 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:51:02.053 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:51:02.053 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:51:02.053 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:51:02.053 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:51:02.053 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:51:02.054 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:51:02.054 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:51:02.054 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:02.054 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.054 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.054 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:51:02.054 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T01:51:02.054 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:51:02.055 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:51:02.055 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:51:02.055 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:51:02.055 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:51:02.055 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:51:02.055 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:51:02.055 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:51:02.055 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:51:02.055 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:51:02.055 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:51:02.055 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:51:02.056 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:51:02.056 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:51:02.056 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:02.056 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.056 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:51:02.056 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T01:51:02.056 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.056 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T01:51:02.056 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T01:51:02.056 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T01:51:02.056 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T01:51:02.057 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:02.057 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.057 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:51:02.057 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T01:51:02.057 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:51:02.058 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T01:51:02.058 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [ 2024-04-20T01:51:02.058 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.058 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-group", 2024-04-20T01:51:02.058 INFO:teuthology.orchestra.run.smithi019.stdout: "data_flow": { 2024-04-20T01:51:02.059 INFO:teuthology.orchestra.run.smithi019.stdout: "directional": [ 2024-04-20T01:51:02.059 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.059 INFO:teuthology.orchestra.run.smithi019.stdout: "source_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:51:02.059 INFO:teuthology.orchestra.run.smithi019.stdout: "dest_zone": "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:51:02.059 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.060 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:02.060 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.060 INFO:teuthology.orchestra.run.smithi019.stdout: "pipes": [ 2024-04-20T01:51:02.060 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.060 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe-aktlji-90", 2024-04-20T01:51:02.061 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:51:02.061 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-90", 2024-04-20T01:51:02.061 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:51:02.061 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:51:02.061 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:02.062 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.062 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:51:02.062 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-91", 2024-04-20T01:51:02.062 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:51:02.062 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:51:02.063 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:02.063 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.063 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:51:02.063 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:51:02.064 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:51:02.064 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:51:02.064 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.064 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.064 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:51:02.065 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:51:02.065 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:51:02.065 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:51:02.065 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.065 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.066 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.066 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe-aktlji-91", 2024-04-20T01:51:02.066 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:51:02.066 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-91", 2024-04-20T01:51:02.067 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:51:02.067 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:51:02.067 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:02.067 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.067 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:51:02.068 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-91", 2024-04-20T01:51:02.068 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:51:02.068 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:51:02.068 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:02.068 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.069 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:51:02.069 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:51:02.069 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:51:02.069 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:51:02.069 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.070 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.070 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:51:02.070 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:51:02.070 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:51:02.070 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:51:02.071 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.071 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.071 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:51:02.071 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "enabled" 2024-04-20T01:51:02.072 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.072 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:02.072 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.072 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T01:51:02.072 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:51:02.073 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:02.073 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.073 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:51:02.073 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T01:51:02.073 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.074 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:51:02.074 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T01:51:02.074 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.074 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.074 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:51:02.075 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T01:51:02.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.075 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:51:02.076 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T01:51:02.076 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.076 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:02.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.076 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T01:51:02.077 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:51:02.077 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T01:51:02.077 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T01:51:02.077 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:51:02.077 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:51:02.078 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:51:02.078 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:51:02.078 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:51:02.078 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.079 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T01:51:02.079 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:51:02.079 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:51:02.079 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:51:02.079 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:51:02.080 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:51:02.080 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.080 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T01:51:02.080 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:51:02.080 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:51:02.083 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:51:02.083 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:51:02.083 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:51:02.083 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.083 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T01:51:02.084 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:51:02.084 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:51:02.084 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:51:02.084 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:51:02.085 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:51:02.085 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.085 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T01:51:02.085 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:51:02.085 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:51:02.085 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:51:02.086 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:51:02.086 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:51:02.086 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.086 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.086 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:51:02.087 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T01:51:02.087 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:51:02.087 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:51:02.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:02.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:51:02.088 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:51:02.142 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:02.142 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:02.232 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:51:02.232 DEBUG:boto:path=/aktlji-90/ 2024-04-20T01:51:02.232 DEBUG:boto:auth_path=/aktlji-90/ 2024-04-20T01:51:02.232 DEBUG:boto:Method: HEAD 2024-04-20T01:51:02.233 DEBUG:boto:Path: /aktlji-90/ 2024-04-20T01:51:02.233 DEBUG:boto:Data: 2024-04-20T01:51:02.233 DEBUG:boto:Headers: {} 2024-04-20T01:51:02.233 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:51:02.233 DEBUG:boto:Port: 8000 2024-04-20T01:51:02.233 DEBUG:boto:Params: {} 2024-04-20T01:51:02.233 DEBUG:boto:Token: None 2024-04-20T01:51:02.233 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:51:02 GMT /aktlji-90/ 2024-04-20T01:51:02.234 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sIvuZQ+J3ndacA6H5KRx3dpQ/XE= 2024-04-20T01:51:02.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sIvuZQ+J3ndacA6H5KRx3dpQ/XE=', 'Content-Length': '0'} 2024-04-20T01:51:02.335 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1d8a182a3483104-0066231f86-1dec-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:51:02.335 DEBUG:boto:path=/aktlji-90/obj1 2024-04-20T01:51:02.335 DEBUG:boto:auth_path=/aktlji-90/obj1 2024-04-20T01:51:02.336 DEBUG:boto:Method: PUT 2024-04-20T01:51:02.336 DEBUG:boto:Path: /aktlji-90/obj1 2024-04-20T01:51:02.336 DEBUG:boto:Data: 2024-04-20T01:51:02.336 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:51:02.336 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:51:02.336 DEBUG:boto:Port: 8000 2024-04-20T01:51:02.336 DEBUG:boto:Params: {} 2024-04-20T01:51:02.336 DEBUG:boto:Token: None 2024-04-20T01:51:02.336 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:51:02 GMT /aktlji-90/obj1 2024-04-20T01:51:02.337 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Dc7brSe5PRN6JD8Lm1xM70anyuM= 2024-04-20T01:51:02.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Dc7brSe5PRN6JD8Lm1xM70anyuM='} 2024-04-20T01:51:02.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe40ebbf376afaca-0066231f86-1dec-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:51:02.357 DEBUG:boto:path=/aktlji-90/ 2024-04-20T01:51:02.357 DEBUG:boto:auth_path=/aktlji-90/ 2024-04-20T01:51:02.357 DEBUG:boto:Method: HEAD 2024-04-20T01:51:02.358 DEBUG:boto:Path: /aktlji-90/ 2024-04-20T01:51:02.358 DEBUG:boto:Data: 2024-04-20T01:51:02.358 DEBUG:boto:Headers: {} 2024-04-20T01:51:02.358 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:51:02.358 DEBUG:boto:Port: 8000 2024-04-20T01:51:02.358 DEBUG:boto:Params: {} 2024-04-20T01:51:02.358 DEBUG:boto:Token: None 2024-04-20T01:51:02.358 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:51:02 GMT /aktlji-90/ 2024-04-20T01:51:02.358 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sIvuZQ+J3ndacA6H5KRx3dpQ/XE= 2024-04-20T01:51:02.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sIvuZQ+J3ndacA6H5KRx3dpQ/XE=', 'Content-Length': '0'} 2024-04-20T01:51:02.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d2f663d110720fe-0066231f86-1dec-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:51:02.364 DEBUG:boto:path=/aktlji-90/obj2 2024-04-20T01:51:02.364 DEBUG:boto:auth_path=/aktlji-90/obj2 2024-04-20T01:51:02.364 DEBUG:boto:Method: PUT 2024-04-20T01:51:02.364 DEBUG:boto:Path: /aktlji-90/obj2 2024-04-20T01:51:02.364 DEBUG:boto:Data: 2024-04-20T01:51:02.365 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:51:02.365 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:51:02.365 DEBUG:boto:Port: 8000 2024-04-20T01:51:02.365 DEBUG:boto:Params: {} 2024-04-20T01:51:02.365 DEBUG:boto:Token: None 2024-04-20T01:51:02.365 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:51:02 GMT /aktlji-90/obj2 2024-04-20T01:51:02.365 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:/v/R/wUicGqWo0CjagyAm+ExVPY= 2024-04-20T01:51:02.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/v/R/wUicGqWo0CjagyAm+ExVPY='} 2024-04-20T01:51:02.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000296e79f502584d9d-0066231f86-1dec-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:51:02.415 DEBUG:boto:path=/aktlji-91/ 2024-04-20T01:51:02.415 DEBUG:boto:auth_path=/aktlji-91/ 2024-04-20T01:51:02.415 DEBUG:boto:Method: HEAD 2024-04-20T01:51:02.415 DEBUG:boto:Path: /aktlji-91/ 2024-04-20T01:51:02.415 DEBUG:boto:Data: 2024-04-20T01:51:02.415 DEBUG:boto:Headers: {} 2024-04-20T01:51:02.415 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:51:02.415 DEBUG:boto:Port: 8000 2024-04-20T01:51:02.415 DEBUG:boto:Params: {} 2024-04-20T01:51:02.416 DEBUG:boto:Token: None 2024-04-20T01:51:02.416 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:51:02 GMT /aktlji-91/ 2024-04-20T01:51:02.416 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mXfbVpFiuIYYSG2SmI3/TLGNM2E= 2024-04-20T01:51:02.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mXfbVpFiuIYYSG2SmI3/TLGNM2E=', 'Content-Length': '0'} 2024-04-20T01:51:02.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5a9f8ff9d901398-0066231f86-1dec-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:51:02.421 DEBUG:boto:path=/aktlji-91/obj3 2024-04-20T01:51:02.421 DEBUG:boto:auth_path=/aktlji-91/obj3 2024-04-20T01:51:02.421 DEBUG:boto:Method: PUT 2024-04-20T01:51:02.421 DEBUG:boto:Path: /aktlji-91/obj3 2024-04-20T01:51:02.421 DEBUG:boto:Data: 2024-04-20T01:51:02.421 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:51:02.421 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:51:02.422 DEBUG:boto:Port: 8000 2024-04-20T01:51:02.422 DEBUG:boto:Params: {} 2024-04-20T01:51:02.422 DEBUG:boto:Token: None 2024-04-20T01:51:02.422 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:51:02 GMT /aktlji-91/obj3 2024-04-20T01:51:02.422 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:43IifLKJSfFLPGqjCJ05TNnPwng= 2024-04-20T01:51:02.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:43IifLKJSfFLPGqjCJ05TNnPwng='} 2024-04-20T01:51:02.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b71f8e87951152b1-0066231f86-1dec-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:51:02.444 DEBUG:boto:path=/aktlji-91/ 2024-04-20T01:51:02.444 DEBUG:boto:auth_path=/aktlji-91/ 2024-04-20T01:51:02.444 DEBUG:boto:Method: HEAD 2024-04-20T01:51:02.444 DEBUG:boto:Path: /aktlji-91/ 2024-04-20T01:51:02.444 DEBUG:boto:Data: 2024-04-20T01:51:02.444 DEBUG:boto:Headers: {} 2024-04-20T01:51:02.444 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:51:02.444 DEBUG:boto:Port: 8000 2024-04-20T01:51:02.445 DEBUG:boto:Params: {} 2024-04-20T01:51:02.445 DEBUG:boto:Token: None 2024-04-20T01:51:02.445 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:51:02 GMT /aktlji-91/ 2024-04-20T01:51:02.445 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:mXfbVpFiuIYYSG2SmI3/TLGNM2E= 2024-04-20T01:51:02.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mXfbVpFiuIYYSG2SmI3/TLGNM2E=', 'Content-Length': '0'} 2024-04-20T01:51:02.448 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000596dd50c0dc61bcb-0066231f86-1dec-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:51:02.448 DEBUG:boto:path=/aktlji-91/obj4 2024-04-20T01:51:02.448 DEBUG:boto:auth_path=/aktlji-91/obj4 2024-04-20T01:51:02.448 DEBUG:boto:Method: PUT 2024-04-20T01:51:02.448 DEBUG:boto:Path: /aktlji-91/obj4 2024-04-20T01:51:02.448 DEBUG:boto:Data: 2024-04-20T01:51:02.448 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:51:02.448 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:51:02.449 DEBUG:boto:Port: 8000 2024-04-20T01:51:02.449 DEBUG:boto:Params: {} 2024-04-20T01:51:02.449 DEBUG:boto:Token: None 2024-04-20T01:51:02.449 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:51:02 GMT /aktlji-91/obj4 2024-04-20T01:51:02.449 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:hIAxjr2JMDYqiumSro12g5sph3E= 2024-04-20T01:51:02.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hIAxjr2JMDYqiumSro12g5sph3E='} 2024-04-20T01:51:02.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6c81492b860fac3-0066231f86-1dec-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:51:02.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:02.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:02.499 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:02.572 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:02.572 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:02.735 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:51:02.803 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:51:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:51:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:51:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.803 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:51:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:51:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:51:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:50:55.729112Z" 2024-04-20T01:51:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:51:02.804 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:51:02.805 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:02.805 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:51:02.806 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:51:02.806 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:51:02.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:02.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:02.807 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:02.874 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:02.875 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:02.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:51:02.959+0000 7fbb89b18780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:51:02.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:51:02.961+0000 7fbb89b18780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:51:02.981 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:51:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:51:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:51:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:51:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:51:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:51:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:51:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:51:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:51:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:51:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:51:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:51:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:51:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:51:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:51:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:51:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:51:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:51:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:51:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:51:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:51:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:02Z" 2024-04-20T01:51:02.987 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:02.987 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577855.729112_1726.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:55.729112Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:51:02Z" } 2024-04-20T01:51:02.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.729112_1726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.729112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:51:02.988 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:51:02.988 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:51:02.988 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:51:02.988 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:51:02.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:02.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:02.989 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:03.070 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:03.071 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:03.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:51:03.182+0000 7f897ab27780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:51:03.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:51:03.184+0000 7f897ab27780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:51:03.215 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:51:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:51:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:51:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:51:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:51:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:51:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:51:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:03.216 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:51:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:51:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:51:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:51:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:51:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:51:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:51:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:51:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:51:03.217 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:51:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:51:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:51:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:51:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:51:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:51:03.218 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:51:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:51:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:51:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:51:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:51:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:51:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:51:03.219 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:51:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:51:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:51:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:51:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:51:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:51:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:51:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:51:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:51:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:51:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:51:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:51:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:51:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:51:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:51:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:03.221 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:51:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:51:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:51:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:51:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:03.222 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:51:03Z" 2024-04-20T01:51:03.222 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:51:03.223 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577855.729112_1726.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:55.729112Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:51:03Z" } 2024-04-20T01:51:03.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.729112_1726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.729112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:51:03.224 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:51:03.224 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:51:03.224 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:51:03.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:03.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:03.224 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:03.282 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:03.282 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:03.387 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:51:03.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:03.388 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:51:03.388 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:51:03.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:03.388 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:03.388 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:51:03.388 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:51:03.388 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:03.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:03.389 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:51:03.389 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:51:03.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:51:03.389 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:51:03.389 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:51:03.389 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:51:03.389 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:51:03.389 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:51:03.391 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:51:03.391 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:51:03.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:03.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:03.391 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:03.455 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:03.455 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:03.544 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:51:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:51:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:51:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:51:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:51:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:51:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:51:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:51:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:51:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:51:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:51:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:51:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:51:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:03Z" 2024-04-20T01:51:03.549 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:03.550 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:51:03Z" } 2024-04-20T01:51:03.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:51:03.550 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:51:03.550 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:51:03.551 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:51:08.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:08.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:08.552 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:08.613 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:08.614 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:08.691 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:08.691 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:51:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:51:08.692 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:51:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:51:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:51:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:08.693 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:51:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:51:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:51:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:08.694 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:51:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:51:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:08.695 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:51:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:51:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:51:08.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:08.697 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:08Z" 2024-04-20T01:51:08.697 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:08.697 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:51:08Z" } 2024-04-20T01:51:08.698 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:51:08.698 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:51:08.698 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:51:08.698 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:51:13.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:13.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:13.699 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:13.761 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:13.761 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:13.836 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:13.837 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:13.837 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:13.837 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:13.837 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:13.837 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:51:13.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:13.837 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:13.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:13.837 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:13.837 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:13.838 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:13.838 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:51:13.838 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:13.838 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:51:13.838 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:13.838 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:51:13.838 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:13.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:13.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:13.838 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:13.838 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:13.839 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:13.839 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:51:13.839 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:13.839 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:51:13.839 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:13.839 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:51:13.839 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:13.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:13.839 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:13.839 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:13.839 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:13.840 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:13.840 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:51:13.840 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:13.840 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:13.840 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:13.840 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:51:13.840 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:13.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:13.840 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:13.840 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:13.841 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:13.841 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:13.841 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:51:13.841 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:13.841 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:13.841 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:13.841 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:51:13.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:13.841 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:13.841 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:13.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:13.842 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:13.842 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:51:13.842 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:51:13.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:13.842 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:13Z" 2024-04-20T01:51:13.842 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:13.843 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:51:13Z" } 2024-04-20T01:51:13.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:51:13.843 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:51:13.844 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:51:13.844 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:51:18.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:18.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:18.845 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:18.907 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:18.908 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:18.972 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:51:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:18.972 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:51:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:51:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:51:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:18.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:51:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:51:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:51:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:18.974 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:51:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:51:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:18.975 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:51:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:51:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:18.976 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:51:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:51:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:18.977 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:18Z" 2024-04-20T01:51:18.977 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:18.978 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:51:18Z" } 2024-04-20T01:51:18.978 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:51:18.978 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:51:18.978 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:51:18.978 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:51:23.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:23.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:23.980 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:24.122 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:51:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:51:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:51:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:51:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:51:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:51:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:51:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:51:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:51:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:51:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:51:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:51:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:51:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:24Z" 2024-04-20T01:51:24.128 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:24.129 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:51:24Z" } 2024-04-20T01:51:24.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:51:24.129 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:51:24.129 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:51:24.129 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:51:29.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:29.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:29.131 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:29.278 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:29.279 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:29.279 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:29.279 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:29.279 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:29.279 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:51:29.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:29.279 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:29.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:29.279 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:29.279 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:29.279 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:29.280 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:51:29.280 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:29.280 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:51:29.280 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:29.280 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:51:29.280 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:29.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:29.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:29.280 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:29.280 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:29.280 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:29.281 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:51:29.281 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:29.281 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:51:29.281 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:29.281 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:51:29.281 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:29.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:29.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:29.281 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:29.281 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:29.281 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:29.282 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:51:29.282 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:29.282 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:29.282 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:29.282 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:51:29.282 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:29.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:29.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:29.282 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:29.282 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:29.282 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:51:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:51:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:51:29.284 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:51:29.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:29.284 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:29Z" 2024-04-20T01:51:29.284 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:29.285 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:51:29Z" } 2024-04-20T01:51:29.285 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:51:29.285 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:51:29.285 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:51:29.285 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:51:34.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:34.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:34.287 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:34.348 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:34.349 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:34.410 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:34.410 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:34.410 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:34.410 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:34.410 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:34.410 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:51:34.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:34.410 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:34.411 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:34.411 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:34.411 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:34.411 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:34.411 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:51:34.411 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:34.411 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:51:34.411 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:34.411 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:51:34.411 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:34.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:34.412 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:34.412 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:34.412 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:34.412 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:34.412 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:51:34.412 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:34.412 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:51:34.412 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:34.412 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:51:34.412 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:34.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:34.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:34.413 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:34.413 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:34.413 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:34.413 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:51:34.413 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:34.413 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:34.413 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:34.413 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:51:34.413 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:34.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:34.413 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:34.414 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:34.414 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:34.414 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:34.414 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:51:34.414 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:34.414 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:34.414 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:34.414 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:51:34.414 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:34.414 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:34.414 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:34.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:34.415 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:34.415 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:51:34.415 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:51:34.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:34.415 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:34Z" 2024-04-20T01:51:34.415 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:34.417 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:51:34Z" } 2024-04-20T01:51:34.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:51:34.417 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:51:34.417 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:51:34.417 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:51:39.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:39.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:39.419 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:39.482 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:39.483 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:39.550 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:51:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:39.551 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:51:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:51:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:51:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:39.552 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:39.553 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:39.553 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:51:39.553 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:39.553 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:51:39.553 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:39.553 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:51:39.553 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:39.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:39.553 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:39.553 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:39.554 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:39.554 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:39.554 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:51:39.554 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:39.554 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:39.554 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:39.554 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:51:39.554 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:39.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:39.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:39.554 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:39.555 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:39.555 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:39.555 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:51:39.555 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:39.555 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:39.555 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:39.555 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:51:39.555 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:39.555 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:39.555 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:39.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:39.556 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:39.556 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:51:39.556 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:51:39.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:39.556 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:39Z" 2024-04-20T01:51:39.556 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:39.557 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:51:39Z" } 2024-04-20T01:51:39.557 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:51:39.557 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:51:39.558 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:51:39.558 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:51:44.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:44.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:44.559 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:44.621 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:44.621 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:44.684 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:51:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:44.685 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:51:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:51:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:51:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:44.686 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:51:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:51:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:51:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:44.687 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:51:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:51:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:44.688 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:51:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:51:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:44.689 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:51:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:51:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:44.690 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:44Z" 2024-04-20T01:51:44.690 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:44.691 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:51:44Z" } 2024-04-20T01:51:44.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:51:44.691 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:51:44.691 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:51:44.691 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:51:49.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:49.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:49.693 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:49.757 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:49.757 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:49.847 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:51:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:49.848 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:51:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:51:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:51:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:49.849 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:51:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:51:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:51:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:49.850 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:51:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:51:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:49.851 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:51:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:51:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:49.852 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:51:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:51:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:49.853 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:49Z" 2024-04-20T01:51:49.853 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:49.854 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:51:49Z" } 2024-04-20T01:51:49.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:51:49.854 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:51:49.854 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:51:49.854 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:51:54.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:54.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:51:54.856 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:51:54.918 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:51:54.918 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:51:54.998 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:51:54.999 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:51:54.999 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:51:54.999 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:51:54.999 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:51:54.999 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:51:54.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:54.999 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:51:54.999 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:54.999 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:51:54.999 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:54.999 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:55.000 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:51:55.000 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:55.000 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:51:55.000 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:55.000 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:51:55.000 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:55.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:55.000 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:55.000 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:51:55.000 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:55.000 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:55.001 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:51:55.001 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:55.001 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:51:55.001 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:55.001 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:51:55.001 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:55.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:55.001 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:55.001 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:51:55.001 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:55.001 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:55.002 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:51:55.002 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:55.002 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:55.002 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:55.002 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:51:55.002 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:55.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:55.002 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:51:55.002 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:51:55.002 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:51:55.002 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:51:55.002 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:51:55.003 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:51:55.003 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:51:55.003 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:51:55.003 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:51:55.003 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:55.003 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:51:55.003 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:51:55.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:55.003 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:51:55.003 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:51:55.003 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:51:55.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:51:55.004 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:51:54Z" 2024-04-20T01:51:55.004 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:51:55.005 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:51:54Z" } 2024-04-20T01:51:55.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:51:55.005 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:51:55.005 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:51:55.005 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:52:00.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:00.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:00.007 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:52:00.068 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:52:00.068 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:52:00.143 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:52:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:52:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:52:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:52:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:52:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:52:00.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:52:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:52:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:52:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:52:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:52:00.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:52:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:52:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:52:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:52:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:00.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:00.146 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:00.146 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:52:00.146 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:00.146 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:00.146 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:52:00.146 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:00.146 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:00.146 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:00.146 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:52:00.146 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:00.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:00.147 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:00.147 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:52:00.147 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:00.147 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:00.147 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:52:00.147 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:00.147 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:00.147 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:00.147 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:52:00.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:00.147 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:00.148 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:52:00.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:00.149 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:52:00.149 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:52:00.149 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:52:00.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:00.149 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:52:00Z" 2024-04-20T01:52:00.149 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:52:00.150 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:52:00Z" } 2024-04-20T01:52:00.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:52:00.151 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:52:00.151 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:52:00.151 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:52:05.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:05.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:05.153 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:52:05.214 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:52:05.215 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:52:05.296 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:52:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:52:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:52:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:52:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:52:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:52:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:52:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:52:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:52:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:52:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:52:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:52:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:52:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:52:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:52:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:52:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:52:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:52:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:52:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:52:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:52:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:52:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:52:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:52:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:52:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:52:05Z" 2024-04-20T01:52:05.302 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:52:05.303 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:52:05Z" } 2024-04-20T01:52:05.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:52:05.303 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:52:05.303 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:52:05.303 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:52:10.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:10.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:10.305 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:52:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:52:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:52:10.432 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:52:10.432 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:52:10.432 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:52:10.432 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:52:10.432 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:52:10.432 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:52:10.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:10.432 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:52:10.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:10.433 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:52:10.433 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:10.433 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:10.433 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:52:10.433 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:10.433 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:52:10.433 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:10.433 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:52:10.433 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:10.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:10.433 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:10.434 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:52:10.434 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:10.434 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:10.434 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:52:10.434 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:10.434 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:52:10.434 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:10.434 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:52:10.434 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:10.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:10.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:10.435 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:52:10.435 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:10.435 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:10.435 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:52:10.435 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:10.435 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:10.435 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:10.435 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:52:10.435 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:52:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:52:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:52:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:10.437 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:10.437 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:52:10.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:10.437 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:52:10.437 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:52:10.437 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:52:10.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:10.437 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:52:10Z" 2024-04-20T01:52:10.438 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:52:10.439 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:52:10Z" } 2024-04-20T01:52:10.439 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:52:10.439 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:52:10.439 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:52:10.439 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:52:15.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:15.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:15.441 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:52:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:52:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:52:15.579 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:52:15.579 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:52:15.580 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:52:15.580 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:52:15.580 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:52:15.580 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:52:15.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:15.580 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:52:15.580 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:15.580 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:52:15.580 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:15.580 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:15.580 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:52:15.581 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:15.581 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:52:15.581 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:15.581 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:52:15.581 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:15.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:15.581 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:15.581 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:52:15.581 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:15.581 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:15.582 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:52:15.582 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:15.582 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:52:15.582 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:15.582 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:52:15.582 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:15.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:15.582 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:15.582 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:52:15.582 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:52:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:52:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:52:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:15.584 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:52:15.584 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:15.584 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:15.584 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:15.584 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:52:15.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:15.584 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:15.584 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:52:15.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:15.584 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:52:15.585 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:52:15.585 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:52:15.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:15.585 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:52:15Z" 2024-04-20T01:52:15.585 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:52:15.586 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:52:15Z" } 2024-04-20T01:52:15.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:52:15.586 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:52:15.586 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:52:15.586 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:52:20.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:20.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:20.588 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:52:20.649 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:52:20.649 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:52:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:52:20.716 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:52:20.716 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:52:20.716 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:52:20.716 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:52:20.716 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:52:20.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:20.716 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:52:20.716 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:20.716 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:52:20.716 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:20.716 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:20.717 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:52:20.717 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:20.717 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:52:20.717 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:20.717 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:52:20.717 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:20.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:20.717 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:20.717 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:52:20.717 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:20.717 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:20.718 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:52:20.718 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:20.718 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:52:20.718 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:20.718 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:52:20.718 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:20.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:20.718 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:20.718 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:52:20.718 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:20.718 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:20.719 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:52:20.719 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:20.719 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:20.719 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:20.719 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:52:20.719 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:20.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:20.719 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:20.719 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:52:20.719 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:20.719 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:20.720 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:52:20.720 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:20.720 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:20.720 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:20.720 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:52:20.720 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:20.720 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:20.720 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:52:20.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:20.720 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:52:20.720 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:52:20.721 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:52:20.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:20.721 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:52:20Z" 2024-04-20T01:52:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:52:20.722 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:52:20Z" } 2024-04-20T01:52:20.722 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:52:20.722 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:52:20.722 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:52:20.723 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:52:25.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:25.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:25.724 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:52:25.786 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:52:25.786 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:52:25.855 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:52:25.855 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:52:25.855 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:52:25.855 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:52:25.855 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:52:25.855 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:52:25.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:25.855 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:52:25.856 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:25.856 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:52:25.856 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:25.856 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:25.856 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:52:25.856 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:25.856 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:52:25.856 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:25.856 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:52:25.856 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:25.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:25.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:25.857 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:52:25.857 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:25.857 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:25.857 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:52:25.857 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:25.857 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:52:25.857 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:25.857 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:52:25.857 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:25.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:25.857 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:25.859 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:52:25.859 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:25.859 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:25.859 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:52:25.859 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:25.859 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:25.859 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:25.859 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:52:25.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:25.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:25.859 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:52:25.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:25.860 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:52:25.860 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:52:25.860 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:52:25.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:25.860 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:52:25Z" 2024-04-20T01:52:25.861 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:52:25.862 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:52:25Z" } 2024-04-20T01:52:25.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:52:25.862 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:52:25.862 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:52:25.862 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:52:30.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:30.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:30.865 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:52:30.929 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:52:30.929 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:52:32.014 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:52:32.015 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:52:32.015 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:52:32.015 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:52:32.015 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:52:32.015 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:52:32.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:32.015 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:52:32.015 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:32.015 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:52:32.016 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:32.016 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:32.016 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:52:32.016 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:32.016 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:52:32.016 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:32.016 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:52:32.017 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:32.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:32.017 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:32.017 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:52:32.017 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:32.017 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:32.017 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:52:32.017 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:32.017 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:52:32.017 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:32.018 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:52:32.018 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:32.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:32.018 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:32.018 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:52:32.018 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:32.018 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:32.018 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:52:32.018 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:32.018 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:32.019 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:32.019 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:52:32.019 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:32.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:32.019 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:32.019 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:52:32.019 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:32.019 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:32.019 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:52:32.019 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:32.019 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:32.020 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:32.020 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:52:32.020 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:32.020 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:32.020 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:52:32.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:32.020 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:52:32.020 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:52:32.020 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:52:32.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:32.020 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:52:32Z" 2024-04-20T01:52:32.020 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:52:32.021 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:52:32Z" } 2024-04-20T01:52:32.022 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:52:32.022 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:52:32.022 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:52:32.022 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:52:37.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:37.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:37.024 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:52:37.087 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:52:37.087 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:52:37.155 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:52:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:52:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:52:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:52:37.155 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:52:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:52:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:52:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:37.156 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:52:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:52:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:37.157 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:52:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:52:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:37.158 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:52:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:52:37.159 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:52:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:52:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:52:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:52:37.161 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:52:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:37.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:52:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:37.163 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:52:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:52:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:52:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:52:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:52:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:52:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:37.166 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:52:37Z" 2024-04-20T01:52:37.166 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:52:37.167 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:52:37Z" } 2024-04-20T01:52:37.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:52:37.168 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:52:37.168 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:52:37.168 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:52:42.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:42.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:42.170 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:52:42.232 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:52:42.232 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:52:42.302 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:52:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:52:42.302 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:52:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:52:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:52:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:52:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:52:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:52:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:42.303 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:52:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:52:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:52:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:52:42.304 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:52:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:52:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:52:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:42.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:52:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:52:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:52:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:42.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:52:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:52:42.307 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:52:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:52:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:52:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:52:42.308 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:52:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:42.309 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:52:42Z" 2024-04-20T01:52:42.309 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:52:42.310 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:52:42Z" } 2024-04-20T01:52:42.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:52:42.310 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:52:42.310 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:52:42.310 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:52:47.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:47.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:47.311 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:52:47.373 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:52:47.373 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:52:47.436 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:52:47.437 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:52:47.437 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:52:47.437 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:52:47.437 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:52:47.437 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:52:47.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:47.437 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:52:47.437 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:47.437 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:52:47.437 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:47.437 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:47.438 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:52:47.438 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:47.438 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:52:47.438 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:47.438 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:52:47.438 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:47.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:47.438 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:47.438 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:52:47.438 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:47.438 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:47.439 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:52:47.439 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:47.439 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:52:47.439 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:47.439 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:52:47.439 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:47.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:47.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:47.439 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:52:47.439 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:47.439 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:47.440 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:52:47.440 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:47.440 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:47.440 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:47.440 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:52:47.440 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:47.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:47.440 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:47.440 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:52:47.440 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:47.440 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:47.440 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:52:47.441 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:47.441 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:47.441 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:47.441 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:52:47.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:47.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:47.441 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:52:47.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:47.441 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:52:47.441 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:52:47.441 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:52:47.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:47.442 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:52:47Z" 2024-04-20T01:52:47.442 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:52:47.443 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:52:47Z" } 2024-04-20T01:52:47.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:52:47.443 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:52:47.443 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:52:47.443 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:52:52.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:52.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:52.445 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:52:52.506 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:52:52.506 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:52:52.582 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:52:52.582 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:52:52.582 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:52:52.583 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:52:52.583 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:52:52.583 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:52:52.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:52.586 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:52:52.586 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:52.586 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:52:52.586 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:52.586 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:52:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:52:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:52:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:52.587 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:52:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:52:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:52:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:52:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:52.588 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:52:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:52:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:52:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:52:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:52:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:52:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:52:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:52.591 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:52:52.592 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:52:52.592 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:52:52.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:52.592 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:52:52Z" 2024-04-20T01:52:52.592 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:52:52.593 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:52:52Z" } 2024-04-20T01:52:52.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:52:52.594 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:52:52.594 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:52:52.594 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:52:57.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:57.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:52:57.596 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:52:57.658 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:52:57.659 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:52:57.722 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:52:57.722 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:52:57.723 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:52:57.723 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:52:57.723 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:52:57.723 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:52:57.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:57.723 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:52:57.723 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:57.723 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:52:57.723 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:57.724 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:57.724 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:52:57.724 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:57.724 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:52:57.724 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:57.724 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:52:57.724 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:57.724 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:57.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:57.724 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:52:57.724 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:57.725 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:57.725 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:52:57.725 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:57.725 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:52:57.725 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:57.725 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:52:57.725 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:57.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:57.725 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:57.725 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:52:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:52:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:52:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:52:57.726 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:52:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:52:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:52:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:52:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:52:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:52:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:52:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:52:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:52:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:52:57.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:52:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:52:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:52:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:52:57.728 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:52:57Z" 2024-04-20T01:52:57.728 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:52:57.729 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:52:57Z" } 2024-04-20T01:52:57.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:52:57.729 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:52:57.730 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:52:57.730 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:53:02.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:02.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:02.731 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:02.793 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:02.794 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:02.876 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:53:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:53:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:53:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:53:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:53:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:53:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:53:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:53:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:53:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:53:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:53:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:02Z" 2024-04-20T01:53:02.882 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:02.883 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:50:15.891072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027750", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:48:27.957440Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021298", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:48:27.971508Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:53:02Z" } 2024-04-20T01:53:02.883 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:53:02.883 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:53:02.883 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:53:02.883 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:53:07.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:07.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:07.885 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:08.019 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:53:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:53:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:53:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:53:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:08.021 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:53:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:53:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:53:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:08.022 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:53:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:53:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:08.023 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:53:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:53:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:08.024 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:53:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:53:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:08.025 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:08Z" 2024-04-20T01:53:08.025 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:08.026 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:53:08Z" } 2024-04-20T01:53:08.026 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:53:08.027 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:53:08.027 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:53:08.027 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:53:08.027 DEBUG:boto:path=/aktlji-90/ 2024-04-20T01:53:08.027 DEBUG:boto:auth_path=/aktlji-90/ 2024-04-20T01:53:08.027 DEBUG:boto:Method: HEAD 2024-04-20T01:53:08.028 DEBUG:boto:Path: /aktlji-90/ 2024-04-20T01:53:08.028 DEBUG:boto:Data: 2024-04-20T01:53:08.028 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.028 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.028 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.028 DEBUG:boto:Params: {} 2024-04-20T01:53:08.028 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:53:08.029 DEBUG:boto:Token: None 2024-04-20T01:53:08.029 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:53:08 GMT /aktlji-90/ 2024-04-20T01:53:08.029 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:sBy4ske2mPxpy2n6OfhXLinH78s= 2024-04-20T01:53:08.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sBy4ske2mPxpy2n6OfhXLinH78s=', 'Content-Length': '0'} 2024-04-20T01:53:08.038 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000169c6b0493fa95a4-0066232004-27d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.039 DEBUG:boto:path=/aktlji-90/obj1 2024-04-20T01:53:08.039 DEBUG:boto:auth_path=/aktlji-90/obj1 2024-04-20T01:53:08.039 DEBUG:boto:Method: HEAD 2024-04-20T01:53:08.039 DEBUG:boto:Path: /aktlji-90/obj1 2024-04-20T01:53:08.039 DEBUG:boto:Data: 2024-04-20T01:53:08.039 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.039 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.039 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.040 DEBUG:boto:Params: {} 2024-04-20T01:53:08.040 DEBUG:boto:Token: None 2024-04-20T01:53:08.040 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:53:08 GMT /aktlji-90/obj1 2024-04-20T01:53:08.040 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:wp+yaJ7FJjBtGLXUJeJ+dFlUQEA= 2024-04-20T01:53:08.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wp+yaJ7FJjBtGLXUJeJ+dFlUQEA=', 'Content-Length': '0'} 2024-04-20T01:53:08.042 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000187977713b3bf610-0066232004-27d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.042 DEBUG:boto:path=/aktlji-90/obj2 2024-04-20T01:53:08.042 DEBUG:boto:auth_path=/aktlji-90/obj2 2024-04-20T01:53:08.042 DEBUG:boto:Method: HEAD 2024-04-20T01:53:08.042 DEBUG:boto:Path: /aktlji-90/obj2 2024-04-20T01:53:08.042 DEBUG:boto:Data: 2024-04-20T01:53:08.042 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.042 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.043 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.043 DEBUG:boto:Params: {} 2024-04-20T01:53:08.043 DEBUG:boto:Token: None 2024-04-20T01:53:08.043 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:53:08 GMT /aktlji-90/obj2 2024-04-20T01:53:08.043 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a5O7uZCabDpLzti9+67bTSiY7cc= 2024-04-20T01:53:08.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a5O7uZCabDpLzti9+67bTSiY7cc=', 'Content-Length': '0'} 2024-04-20T01:53:08.045 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d0cf673e7dcc3a3d-0066232004-27d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.045 DEBUG:boto:path=/aktlji-90/obj3 2024-04-20T01:53:08.045 DEBUG:boto:auth_path=/aktlji-90/obj3 2024-04-20T01:53:08.045 DEBUG:boto:Method: HEAD 2024-04-20T01:53:08.045 DEBUG:boto:Path: /aktlji-90/obj3 2024-04-20T01:53:08.045 DEBUG:boto:Data: 2024-04-20T01:53:08.045 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.045 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.045 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.046 DEBUG:boto:Params: {} 2024-04-20T01:53:08.046 DEBUG:boto:Token: None 2024-04-20T01:53:08.046 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:53:08 GMT /aktlji-90/obj3 2024-04-20T01:53:08.046 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ywIkfox+hkrZCJ9S9M/R2vfuDQs= 2024-04-20T01:53:08.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ywIkfox+hkrZCJ9S9M/R2vfuDQs=', 'Content-Length': '0'} 2024-04-20T01:53:08.048 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000eed9210ca6b0ba3e-0066232004-27d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.048 DEBUG:boto:path=/aktlji-90/obj4 2024-04-20T01:53:08.048 DEBUG:boto:auth_path=/aktlji-90/obj4 2024-04-20T01:53:08.049 DEBUG:boto:Method: HEAD 2024-04-20T01:53:08.049 DEBUG:boto:Path: /aktlji-90/obj4 2024-04-20T01:53:08.049 DEBUG:boto:Data: 2024-04-20T01:53:08.049 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.049 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.049 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.049 DEBUG:boto:Params: {} 2024-04-20T01:53:08.049 DEBUG:boto:Token: None 2024-04-20T01:53:08.049 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:53:08 GMT /aktlji-90/obj4 2024-04-20T01:53:08.049 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:U/LONsj0hFtM4Ph2zUcJdUn7zww= 2024-04-20T01:53:08.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:U/LONsj0hFtM4Ph2zUcJdUn7zww=', 'Content-Length': '0'} 2024-04-20T01:53:08.051 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000ba79d1693d625176-0066232004-27d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.051 DEBUG:boto:path=/aktlji-91/ 2024-04-20T01:53:08.051 DEBUG:boto:auth_path=/aktlji-91/ 2024-04-20T01:53:08.051 DEBUG:boto:Method: HEAD 2024-04-20T01:53:08.051 DEBUG:boto:Path: /aktlji-91/ 2024-04-20T01:53:08.051 DEBUG:boto:Data: 2024-04-20T01:53:08.051 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.051 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.052 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.052 DEBUG:boto:Params: {} 2024-04-20T01:53:08.052 DEBUG:boto:Token: None 2024-04-20T01:53:08.052 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:53:08 GMT /aktlji-91/ 2024-04-20T01:53:08.052 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:54EqijFpEwVF2ELHPaocF/em7Mk= 2024-04-20T01:53:08.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:54EqijFpEwVF2ELHPaocF/em7Mk=', 'Content-Length': '0'} 2024-04-20T01:53:08.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016ba72d0faa67c60-0066232004-27d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.055 DEBUG:boto:path=/aktlji-91/obj1 2024-04-20T01:53:08.055 DEBUG:boto:auth_path=/aktlji-91/obj1 2024-04-20T01:53:08.055 DEBUG:boto:Method: HEAD 2024-04-20T01:53:08.055 DEBUG:boto:Path: /aktlji-91/obj1 2024-04-20T01:53:08.055 DEBUG:boto:Data: 2024-04-20T01:53:08.055 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.055 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.055 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.055 DEBUG:boto:Params: {} 2024-04-20T01:53:08.055 DEBUG:boto:Token: None 2024-04-20T01:53:08.056 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:53:08 GMT /aktlji-91/obj1 2024-04-20T01:53:08.056 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:A6uaoZun8/dAJUKAte+dYFbbOlU= 2024-04-20T01:53:08.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:A6uaoZun8/dAJUKAte+dYFbbOlU=', 'Content-Length': '0'} 2024-04-20T01:53:08.058 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-90:73f46f02-c7ab-4426-a267-2273af911da5.7366.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5b4ba39e18cfedf-0066232004-27d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.058 DEBUG:boto:path=/aktlji-91/obj1 2024-04-20T01:53:08.058 DEBUG:boto:auth_path=/aktlji-91/obj1 2024-04-20T01:53:08.059 DEBUG:boto:Method: GET 2024-04-20T01:53:08.059 DEBUG:boto:Path: /aktlji-91/obj1 2024-04-20T01:53:08.059 DEBUG:boto:Data: 2024-04-20T01:53:08.059 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.059 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.059 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.059 DEBUG:boto:Params: {} 2024-04-20T01:53:08.059 DEBUG:boto:Token: None 2024-04-20T01:53:08.059 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:53:08 GMT /aktlji-91/obj1 2024-04-20T01:53:08.059 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:fBLmDHKIkRTdpYTMbClXoqCc/B0= 2024-04-20T01:53:08.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fBLmDHKIkRTdpYTMbClXoqCc/B0=', 'Content-Length': '0'} 2024-04-20T01:53:08.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-90:73f46f02-c7ab-4426-a267-2273af911da5.7366.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000016f315d50671f00-0066232004-27d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.107 DEBUG:boto:path=/aktlji-91/obj2 2024-04-20T01:53:08.107 DEBUG:boto:auth_path=/aktlji-91/obj2 2024-04-20T01:53:08.107 DEBUG:boto:Method: HEAD 2024-04-20T01:53:08.107 DEBUG:boto:Path: /aktlji-91/obj2 2024-04-20T01:53:08.107 DEBUG:boto:Data: 2024-04-20T01:53:08.107 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.107 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.108 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.108 DEBUG:boto:Params: {} 2024-04-20T01:53:08.108 DEBUG:boto:Token: None 2024-04-20T01:53:08.108 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:53:08 GMT /aktlji-91/obj2 2024-04-20T01:53:08.108 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:vpOrk73glIWeBwOSkMyEf+y2lP8= 2024-04-20T01:53:08.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vpOrk73glIWeBwOSkMyEf+y2lP8=', 'Content-Length': '0'} 2024-04-20T01:53:08.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-90:73f46f02-c7ab-4426-a267-2273af911da5.7366.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006105163525d73614-0066232004-27d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.111 DEBUG:boto:path=/aktlji-91/obj2 2024-04-20T01:53:08.111 DEBUG:boto:auth_path=/aktlji-91/obj2 2024-04-20T01:53:08.111 DEBUG:boto:Method: GET 2024-04-20T01:53:08.111 DEBUG:boto:Path: /aktlji-91/obj2 2024-04-20T01:53:08.111 DEBUG:boto:Data: 2024-04-20T01:53:08.111 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.112 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.112 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.112 DEBUG:boto:Params: {} 2024-04-20T01:53:08.112 DEBUG:boto:Token: None 2024-04-20T01:53:08.112 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:53:08 GMT /aktlji-91/obj2 2024-04-20T01:53:08.112 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:OeFqJxBaMdKDmRvBM48z2XbaWtI= 2024-04-20T01:53:08.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OeFqJxBaMdKDmRvBM48z2XbaWtI=', 'Content-Length': '0'} 2024-04-20T01:53:08.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-90:73f46f02-c7ab-4426-a267-2273af911da5.7366.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f8c32d83c8c92a0-0066232004-27d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.116 DEBUG:boto:path=/aktlji-91/obj3 2024-04-20T01:53:08.116 DEBUG:boto:auth_path=/aktlji-91/obj3 2024-04-20T01:53:08.116 DEBUG:boto:Method: HEAD 2024-04-20T01:53:08.116 DEBUG:boto:Path: /aktlji-91/obj3 2024-04-20T01:53:08.117 DEBUG:boto:Data: 2024-04-20T01:53:08.117 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.117 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.117 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.117 DEBUG:boto:Params: {} 2024-04-20T01:53:08.117 DEBUG:boto:Token: None 2024-04-20T01:53:08.117 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:53:08 GMT /aktlji-91/obj3 2024-04-20T01:53:08.117 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:WtovSonjPNYCUfwljF7Y+iOPx94= 2024-04-20T01:53:08.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WtovSonjPNYCUfwljF7Y+iOPx94=', 'Content-Length': '0'} 2024-04-20T01:53:08.119 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-91:73f46f02-c7ab-4426-a267-2273af911da5.7366.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004029042808fcf3f8-0066232004-27d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.120 DEBUG:boto:path=/aktlji-91/obj3 2024-04-20T01:53:08.120 DEBUG:boto:auth_path=/aktlji-91/obj3 2024-04-20T01:53:08.120 DEBUG:boto:Method: GET 2024-04-20T01:53:08.120 DEBUG:boto:Path: /aktlji-91/obj3 2024-04-20T01:53:08.121 DEBUG:boto:Data: 2024-04-20T01:53:08.121 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.121 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.121 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.121 DEBUG:boto:Params: {} 2024-04-20T01:53:08.121 DEBUG:boto:Token: None 2024-04-20T01:53:08.121 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:53:08 GMT /aktlji-91/obj3 2024-04-20T01:53:08.121 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:aCH/mUlOojNePGs6rpb2MjsOw7M= 2024-04-20T01:53:08.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aCH/mUlOojNePGs6rpb2MjsOw7M=', 'Content-Length': '0'} 2024-04-20T01:53:08.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-91:73f46f02-c7ab-4426-a267-2273af911da5.7366.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069fa83dd53e5e9d4-0066232004-27d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.166 DEBUG:boto:path=/aktlji-91/obj4 2024-04-20T01:53:08.167 DEBUG:boto:auth_path=/aktlji-91/obj4 2024-04-20T01:53:08.167 DEBUG:boto:Method: HEAD 2024-04-20T01:53:08.167 DEBUG:boto:Path: /aktlji-91/obj4 2024-04-20T01:53:08.167 DEBUG:boto:Data: 2024-04-20T01:53:08.167 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.167 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:53:08.167 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.167 DEBUG:boto:Params: {} 2024-04-20T01:53:08.167 DEBUG:boto:Token: None 2024-04-20T01:53:08.168 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:53:08 GMT /aktlji-91/obj4 2024-04-20T01:53:08.168 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:HokDBMfYIxhDp8IuUua6TyNkltU= 2024-04-20T01:53:08.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HokDBMfYIxhDp8IuUua6TyNkltU=', 'Content-Length': '0'} 2024-04-20T01:53:08.169 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000077f5d438f8aa46c4-0066232004-27d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.173 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2024-04-20T01:53:08.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:08.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:08.175 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:08.234 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:08.234 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:08.287 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:53:08.307 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:53:08.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:08.308 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:53:08.308 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:53:08.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:08.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:08.308 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:53:08.308 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:53:08.308 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:08.308 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:08.309 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:53:08.309 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:50:55.729112Z" 2024-04-20T01:53:08.309 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:08.309 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:08.309 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:53:08.309 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:53:08.309 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:08.309 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:53:08.310 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:08.310 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:53:08.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:08.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:08.311 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:08.369 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:08.370 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:53:08.426+0000 7f3563500780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:53:08.427+0000 7f3563500780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:08.431 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:08.432 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:08.432 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:08.432 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:08.432 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:08.432 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:53:08.432 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:08.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:08.432 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:08.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:08.433 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:08.433 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:08.433 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:53:08.433 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:53:08.433 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:08.433 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:53:08.433 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:08.433 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:53:08.433 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:08.433 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:08.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:08.434 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:08.434 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:08.434 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:08.434 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:53:08.434 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:53:08.434 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:08.434 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:08.434 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:08.434 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:53:08.434 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:08.434 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:08.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:08.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:08.435 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:08.435 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:08.435 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:53:08.435 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:53:08.435 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:08.435 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:08.435 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:08.435 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:53:08.435 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:08.436 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:08.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:08.436 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:08.436 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:08.436 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:08.436 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:53:08.436 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:53:08.436 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:08.436 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:08.436 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:08.436 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:53:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:53:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:53:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:08Z" 2024-04-20T01:53:08.437 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:08.438 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577855.729112_1726.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:55.729112Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:53:08Z" } 2024-04-20T01:53:08.439 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.729112_1726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.729112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:53:08.439 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:08.439 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:08.439 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:53:08.440 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:53:08.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:08.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:08.440 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:08.499 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:08.500 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:08.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:53:08.565+0000 7f3211255780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:08.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:53:08.566+0000 7f3211255780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:08.571 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:53:08.571 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:53:08.571 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:53:08.571 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:53:08.571 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:53:08.571 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:53:08.571 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:08.571 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:08.571 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:53:08.572 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:08.572 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:53:08.572 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:08.572 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:08.572 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:53:08.572 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:08.572 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:53:08.572 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:08.572 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:53:08.572 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:08.572 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:08.573 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:08.573 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:08.573 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:53:08.573 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:08.573 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:08.573 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:53:08.573 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:08.573 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:53:08.573 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:08.573 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:53:08.574 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:08.574 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:08.574 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:08.574 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:08.574 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:53:08.574 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:08.574 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:08.574 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:53:08.574 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:08.574 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:53:08.575 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:08.575 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:53:08.575 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:08.575 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:08.575 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:08.575 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:08.575 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:53:08.575 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:08.575 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:08.575 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:53:08.575 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:08.575 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:53:08.576 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:08.576 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:53:08.576 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:08.576 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:08.576 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:08.576 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:08.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:08.576 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:53:08.576 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:53:08.576 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:53:08.576 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:08.577 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:53:08Z" 2024-04-20T01:53:08.577 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:53:08.578 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577855.729112_1726.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:55.729112Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:53:08Z" } 2024-04-20T01:53:08.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.729112_1726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.729112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:53:08.578 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:08.578 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:08.578 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:53:08.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:08.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:08.579 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:53:08.630 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:08.630 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:08.693 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:53:08.693 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-92 2024-04-20T01:53:08.694 DEBUG:boto:path=/aktlji-92/ 2024-04-20T01:53:08.694 DEBUG:boto:auth_path=/aktlji-92/ 2024-04-20T01:53:08.694 DEBUG:boto:Method: PUT 2024-04-20T01:53:08.694 DEBUG:boto:Path: /aktlji-92/ 2024-04-20T01:53:08.694 DEBUG:boto:Data: 2024-04-20T01:53:08.694 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.694 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:53:08.694 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.694 DEBUG:boto:Params: {} 2024-04-20T01:53:08.695 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:53:08.695 DEBUG:boto:Token: None 2024-04-20T01:53:08.695 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:53:08 GMT /aktlji-92/ 2024-04-20T01:53:08.695 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:Vj8QkI9ZRSfpRlUPzWfQtTij3VU= 2024-04-20T01:53:08.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Vj8QkI9ZRSfpRlUPzWfQtTij3VU=', 'Content-Length': '0'} 2024-04-20T01:53:08.717 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dda16759288e2032-0066232004-1dec-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.717 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=aktlji-93 2024-04-20T01:53:08.717 DEBUG:boto:path=/aktlji-93/ 2024-04-20T01:53:08.717 DEBUG:boto:auth_path=/aktlji-93/ 2024-04-20T01:53:08.717 DEBUG:boto:Method: PUT 2024-04-20T01:53:08.718 DEBUG:boto:Path: /aktlji-93/ 2024-04-20T01:53:08.718 DEBUG:boto:Data: 2024-04-20T01:53:08.718 DEBUG:boto:Headers: {} 2024-04-20T01:53:08.718 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:53:08.718 DEBUG:boto:Port: 8000 2024-04-20T01:53:08.718 DEBUG:boto:Params: {} 2024-04-20T01:53:08.718 DEBUG:boto:Token: None 2024-04-20T01:53:08.718 DEBUG:boto:StringToSign: PUT Sat, 20 Apr 2024 01:53:08 GMT /aktlji-93/ 2024-04-20T01:53:08.718 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:cfwTYpc3YiQ6c3bkh6ydoO4kXwE= 2024-04-20T01:53:08.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cfwTYpc3YiQ6c3bkh6ydoO4kXwE=', 'Content-Length': '0'} 2024-04-20T01:53:08.745 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b4812be046d6506-0066232004-1dec-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:08.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:08.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:08.746 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:08.806 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:08.806 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:08.870 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:53:08.891 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:53:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:53:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:53:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:53:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:53:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:08.892 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:53:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:53:08.757488Z" 2024-04-20T01:53:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:53:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:53:08.893 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:08.893 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:53:08.894 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:08.894 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:53:08.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:08.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:08.895 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:09.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:53:09.024+0000 7f1bd85ee780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:09.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:53:09.025+0000 7f1bd85ee780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:09.031 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:09.031 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:09.031 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:09.031 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:09.031 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:09.032 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:53:09.032 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:09.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:09.032 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:09.032 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:09.032 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:09.032 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:09.032 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:53:09.032 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:53:09.032 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:09.032 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:53:09.032 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:09.033 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:53:09.033 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:09.033 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:09.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:09.033 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:09.033 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:09.033 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:09.033 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:53:09.033 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:53:09.033 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:09.033 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:09.034 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:09.034 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:53:09.034 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:09.034 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:09.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:09.034 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:09.034 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:09.034 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:09.034 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:53:09.034 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:53:09.034 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:09.035 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:09.035 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:09.035 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:53:08.757488Z", 2024-04-20T01:53:09.035 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:09.035 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:09.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:09.035 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:09.035 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:09.035 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:09.035 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:53:09.035 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:53:09.036 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:09.036 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:09.036 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:09.036 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:53:09.036 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:09.036 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:09.036 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:09.036 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:09.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:09.036 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:09.036 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:53:09.036 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:53:09.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:09.037 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:09Z" 2024-04-20T01:53:09.037 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:09.038 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577988.757488_1755.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:53:08.757488Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:53:09Z" } 2024-04-20T01:53:09.038 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577988.757488_1755.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:53:08.757488Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:53:09.038 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:09.038 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:09.038 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:53:09.039 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:53:09.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:09.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:09.039 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:09.098 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:09.099 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:09.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:53:09.152+0000 7f0877c32780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:09.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:53:09.153+0000 7f0877c32780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:09.154 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:53:09.155 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:53:09.155 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:53:09.155 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:53:09.155 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:53:09.155 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:53:09.155 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:09.155 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:09.155 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:53:09.155 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:09.155 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:53:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:53:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:53:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:53:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:53:09.157 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:09.157 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:09.157 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:53:09.157 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:09.157 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:53:09.158 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:09.158 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:53:09.158 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:09.158 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:09.158 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:09.158 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:09.158 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:53:09.159 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:09.159 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:09.159 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577988.723277_1748.1", 2024-04-20T01:53:09.159 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:09.159 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:53:09.159 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:09.159 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:53:08.723277Z", 2024-04-20T01:53:09.159 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:09.159 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:09.159 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:09.159 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:09.160 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:53:09.160 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:09.160 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:09.160 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:53:09.160 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:09.160 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:53:09.160 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:09.160 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:53:09.160 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:09.160 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:09.160 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:09.160 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:09.161 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:09.161 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:53:09.161 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:53:09.161 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:53:09.161 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:09.161 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:53:09Z" 2024-04-20T01:53:09.161 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:53:09.162 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577988.723277_1748.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:53:08.723277Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:53:09Z" } 2024-04-20T01:53:09.162 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577988.723277_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:53:08.723277Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:53:09.162 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:09.162 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.723277_1748.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:09.163 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1713577988.757488_1755.1 target=1_1713577988.723277_1748.1 2024-04-20T01:53:14.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:14.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:14.164 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:14.224 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:14.224 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:14.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:53:14.284+0000 7fbe55b96780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:14.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:53:14.286+0000 7fbe55b96780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:14.287 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:53:14.288 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:53:14.288 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:53:14.288 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:53:14.288 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:53:14.288 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:53:14.288 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:14.288 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:14.288 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:53:14.289 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:14.289 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:53:14.289 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:14.289 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:14.289 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:53:14.289 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:14.289 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:53:14.289 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:14.289 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:53:14.290 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:14.290 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:14.290 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:14.290 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:14.290 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:53:14.290 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:14.290 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:14.290 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:53:14.290 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:14.290 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:53:14.291 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:14.291 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:53:14.291 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:14.291 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:14.291 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:14.291 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:14.291 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:53:14.291 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:14.291 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:14.291 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:53:14.292 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:14.292 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:53:14.292 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:14.292 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:53:08.757488Z", 2024-04-20T01:53:14.292 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:14.292 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:14.292 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:14.292 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:14.292 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:53:14.292 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:14.293 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:14.293 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:53:14.293 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:14.293 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:53:14.293 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:14.293 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:53:14.293 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:14.293 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:14.293 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:14.294 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:14.294 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:14.294 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:53:14.294 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:53:14.294 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:53:14.294 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:14.294 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:53:14Z" 2024-04-20T01:53:14.294 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:53:14.295 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577988.757488_1755.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:53:08.757488Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:53:14Z" } 2024-04-20T01:53:14.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577988.757488_1755.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:53:08.757488Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:53:14.296 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:14.296 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:14.296 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:53:14.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:14.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:14.296 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:53:14.353 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:14.353 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:14.425 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:53:14.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:14.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:14.426 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:53:14.485 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:14.485 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:14.580 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:53:14.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-92', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-92', '--dest-bucket=aktlji-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:14.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-92', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-92', '--dest-bucket=aktlji-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:14.580 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-aktlji-92 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=aktlji-92 --dest-bucket=aktlji-92 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:53:14.641 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:14.641 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:14.722 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-92","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:53:14.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-93', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-92', '--dest-bucket=aktlji-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:14.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-93', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-92', '--dest-bucket=aktlji-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:14.723 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-aktlji-93 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=aktlji-92 --dest-bucket=aktlji-93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:53:14.783 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:14.783 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:14.864 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-92","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-93","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:53:14.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-92', '--dest-bucket=aktlji-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:14.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-92', '--dest-bucket=aktlji-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:14.865 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=aktlji-92 --dest-bucket=aktlji-93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:53:14.925 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:14.925 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:14.987 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-92","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-93","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:53:14.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:14.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:14.988 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:53:15.043 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:15.044 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:15.117 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-92","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-93","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:53:15.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:15.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:15.118 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:53:15.176 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:15.176 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:15.250 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:53:15.250 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:53:15.250 INFO:teuthology.orchestra.run.smithi019.stdout: "epoch": 19, 2024-04-20T01:53:15.250 INFO:teuthology.orchestra.run.smithi019.stdout: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:53:15.250 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": [], 2024-04-20T01:53:15.251 INFO:teuthology.orchestra.run.smithi019.stdout: "period_map": { 2024-04-20T01:53:15.251 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:53:15.251 INFO:teuthology.orchestra.run.smithi019.stdout: "zonegroups": [ 2024-04-20T01:53:15.251 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.251 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "test-zonegroup", 2024-04-20T01:53:15.251 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zonegroup", 2024-04-20T01:53:15.251 INFO:teuthology.orchestra.run.smithi019.stdout: "api_name": "", 2024-04-20T01:53:15.251 INFO:teuthology.orchestra.run.smithi019.stdout: "is_master": true, 2024-04-20T01:53:15.251 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:53:15.251 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:53:15.251 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:53:15.252 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames": [], 2024-04-20T01:53:15.252 INFO:teuthology.orchestra.run.smithi019.stdout: "hostnames_s3website": [], 2024-04-20T01:53:15.252 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:53:15.252 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:53:15.252 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.252 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:53:15.252 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone2", 2024-04-20T01:53:15.252 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:53:15.252 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:53:15.252 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:53:15.252 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:53:15.253 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:53:15.253 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:53:15.253 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:53:15.253 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:53:15.253 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": false, 2024-04-20T01:53:15.253 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:53:15.253 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:53:15.253 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:53:15.253 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:53:15.253 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:53:15.253 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.253 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.254 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.254 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:53:15.254 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone1", 2024-04-20T01:53:15.254 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:53:15.254 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:53:15.254 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:53:15.254 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:53:15.254 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:53:15.254 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:53:15.254 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:53:15.254 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:53:15.255 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:53:15.255 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:53:15.255 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:53:15.255 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:53:15.255 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:53:15.255 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:53:15.255 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.255 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.255 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.255 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:53:15.255 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "test-zone3", 2024-04-20T01:53:15.255 INFO:teuthology.orchestra.run.smithi019.stdout: "endpoints": [ 2024-04-20T01:53:15.256 INFO:teuthology.orchestra.run.smithi019.stdout: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:53:15.256 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:53:15.256 INFO:teuthology.orchestra.run.smithi019.stdout: "log_meta": false, 2024-04-20T01:53:15.256 INFO:teuthology.orchestra.run.smithi019.stdout: "log_data": true, 2024-04-20T01:53:15.256 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_index_max_shards": 11, 2024-04-20T01:53:15.256 INFO:teuthology.orchestra.run.smithi019.stdout: "read_only": false, 2024-04-20T01:53:15.256 INFO:teuthology.orchestra.run.smithi019.stdout: "tier_type": "", 2024-04-20T01:53:15.256 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from_all": true, 2024-04-20T01:53:15.256 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_from": [], 2024-04-20T01:53:15.256 INFO:teuthology.orchestra.run.smithi019.stdout: "redirect_zone": "", 2024-04-20T01:53:15.256 INFO:teuthology.orchestra.run.smithi019.stdout: "supported_features": [ 2024-04-20T01:53:15.257 INFO:teuthology.orchestra.run.smithi019.stdout: "compress-encrypted", 2024-04-20T01:53:15.257 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:53:15.257 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.257 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.257 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:53:15.257 INFO:teuthology.orchestra.run.smithi019.stdout: "placement_targets": [ 2024-04-20T01:53:15.257 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.257 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "default-placement", 2024-04-20T01:53:15.257 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [], 2024-04-20T01:53:15.257 INFO:teuthology.orchestra.run.smithi019.stdout: "storage_classes": [ 2024-04-20T01:53:15.257 INFO:teuthology.orchestra.run.smithi019.stdout: "STANDARD" 2024-04-20T01:53:15.257 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.258 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.258 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:53:15.258 INFO:teuthology.orchestra.run.smithi019.stdout: "default_placement": "default-placement", 2024-04-20T01:53:15.258 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:53:15.258 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_policy": { 2024-04-20T01:53:15.258 INFO:teuthology.orchestra.run.smithi019.stdout: "groups": [ 2024-04-20T01:53:15.258 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.258 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-group", 2024-04-20T01:53:15.258 INFO:teuthology.orchestra.run.smithi019.stdout: "data_flow": { 2024-04-20T01:53:15.258 INFO:teuthology.orchestra.run.smithi019.stdout: "symmetrical": [ 2024-04-20T01:53:15.258 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.259 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-flow1", 2024-04-20T01:53:15.259 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:53:15.259 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:53:15.259 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:53:15.259 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.259 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.259 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:53:15.259 INFO:teuthology.orchestra.run.smithi019.stdout: "directional": [ 2024-04-20T01:53:15.259 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.259 INFO:teuthology.orchestra.run.smithi019.stdout: "source_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:53:15.259 INFO:teuthology.orchestra.run.smithi019.stdout: "dest_zone": "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:53:15.259 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.260 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.260 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.260 INFO:teuthology.orchestra.run.smithi019.stdout: "pipes": [ 2024-04-20T01:53:15.260 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.260 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe-aktlji-90", 2024-04-20T01:53:15.260 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:53:15.260 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-90", 2024-04-20T01:53:15.260 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:53:15.260 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:53:15.260 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.260 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.260 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:53:15.261 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-91", 2024-04-20T01:53:15.261 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:53:15.261 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:53:15.261 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.261 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.261 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:53:15.261 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:53:15.261 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:53:15.261 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:53:15.261 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.261 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.262 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:53:15.262 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:53:15.262 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:53:15.262 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:53:15.262 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.262 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.262 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.262 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe-aktlji-91", 2024-04-20T01:53:15.262 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:53:15.262 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-91", 2024-04-20T01:53:15.262 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:53:15.262 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:53:15.263 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.263 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.263 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:53:15.263 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-91", 2024-04-20T01:53:15.263 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:53:15.263 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:53:15.263 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.263 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.263 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:53:15.263 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:53:15.263 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:53:15.263 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:53:15.264 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.264 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.264 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:53:15.264 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:53:15.264 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:53:15.264 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:53:15.264 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.264 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.264 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.264 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe-aktlji-92", 2024-04-20T01:53:15.264 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:53:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-92", 2024-04-20T01:53:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:53:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:53:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:53:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-92", 2024-04-20T01:53:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:53:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:53:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:53:15.266 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:53:15.266 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:53:15.266 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:53:15.266 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.266 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.266 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:53:15.266 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:53:15.266 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:53:15.266 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:53:15.266 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.266 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.267 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.267 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe-aktlji-93", 2024-04-20T01:53:15.267 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:53:15.267 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-92", 2024-04-20T01:53:15.267 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:53:15.267 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:53:15.267 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.267 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.267 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:53:15.267 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-93", 2024-04-20T01:53:15.267 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:53:15.267 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:53:15.268 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.268 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.268 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:53:15.268 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:53:15.268 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:53:15.268 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:53:15.268 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.268 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.268 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:53:15.268 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:53:15.268 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:53:15.268 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:53:15.269 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.269 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.269 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.269 INFO:teuthology.orchestra.run.smithi019.stdout: "id": "sync-pipe", 2024-04-20T01:53:15.269 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:53:15.269 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-92", 2024-04-20T01:53:15.269 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:53:15.269 INFO:teuthology.orchestra.run.smithi019.stdout: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:53:15.269 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.269 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.269 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": { 2024-04-20T01:53:15.269 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket": "aktlji-93", 2024-04-20T01:53:15.270 INFO:teuthology.orchestra.run.smithi019.stdout: "zones": [ 2024-04-20T01:53:15.270 INFO:teuthology.orchestra.run.smithi019.stdout: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:53:15.270 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.270 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.271 INFO:teuthology.orchestra.run.smithi019.stdout: "params": { 2024-04-20T01:53:15.271 INFO:teuthology.orchestra.run.smithi019.stdout: "source": { 2024-04-20T01:53:15.271 INFO:teuthology.orchestra.run.smithi019.stdout: "filter": { 2024-04-20T01:53:15.271 INFO:teuthology.orchestra.run.smithi019.stdout: "tags": [] 2024-04-20T01:53:15.271 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.271 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.272 INFO:teuthology.orchestra.run.smithi019.stdout: "dest": {}, 2024-04-20T01:53:15.272 INFO:teuthology.orchestra.run.smithi019.stdout: "priority": 0, 2024-04-20T01:53:15.272 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "system", 2024-04-20T01:53:15.272 INFO:teuthology.orchestra.run.smithi019.stdout: "user": "" 2024-04-20T01:53:15.272 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.272 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.272 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:53:15.272 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "enabled" 2024-04-20T01:53:15.272 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.273 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.273 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.273 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled_features": [ 2024-04-20T01:53:15.273 INFO:teuthology.orchestra.run.smithi019.stdout: "resharding" 2024-04-20T01:53:15.273 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.273 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.273 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-04-20T01:53:15.273 INFO:teuthology.orchestra.run.smithi019.stdout: "short_zone_ids": [ 2024-04-20T01:53:15.273 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.273 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:53:15.273 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 2010863401 2024-04-20T01:53:15.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.274 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.274 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:53:15.274 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1221149675 2024-04-20T01:53:15.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.274 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:15.274 INFO:teuthology.orchestra.run.smithi019.stdout: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:53:15.274 INFO:teuthology.orchestra.run.smithi019.stdout: "val": 1478903101 2024-04-20T01:53:15.274 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.274 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:15.274 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.274 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zonegroup": "test-zonegroup", 2024-04-20T01:53:15.275 INFO:teuthology.orchestra.run.smithi019.stdout: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:53:15.275 INFO:teuthology.orchestra.run.smithi019.stdout: "period_config": { 2024-04-20T01:53:15.275 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_quota": { 2024-04-20T01:53:15.275 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:53:15.276 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:53:15.276 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:53:15.277 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:53:15.277 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:53:15.277 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.277 INFO:teuthology.orchestra.run.smithi019.stdout: "user_quota": { 2024-04-20T01:53:15.277 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false, 2024-04-20T01:53:15.277 INFO:teuthology.orchestra.run.smithi019.stdout: "check_on_raw": false, 2024-04-20T01:53:15.277 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size": -1, 2024-04-20T01:53:15.277 INFO:teuthology.orchestra.run.smithi019.stdout: "max_size_kb": 0, 2024-04-20T01:53:15.277 INFO:teuthology.orchestra.run.smithi019.stdout: "max_objects": -1 2024-04-20T01:53:15.277 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.277 INFO:teuthology.orchestra.run.smithi019.stdout: "user_ratelimit": { 2024-04-20T01:53:15.278 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:53:15.278 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:53:15.278 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:53:15.278 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:53:15.278 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:53:15.278 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.278 INFO:teuthology.orchestra.run.smithi019.stdout: "bucket_ratelimit": { 2024-04-20T01:53:15.278 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:53:15.278 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:53:15.278 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:53:15.278 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:53:15.278 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:53:15.279 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.279 INFO:teuthology.orchestra.run.smithi019.stdout: "anonymous_ratelimit": { 2024-04-20T01:53:15.279 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_ops": 0, 2024-04-20T01:53:15.279 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_ops": 0, 2024-04-20T01:53:15.279 INFO:teuthology.orchestra.run.smithi019.stdout: "max_read_bytes": 0, 2024-04-20T01:53:15.279 INFO:teuthology.orchestra.run.smithi019.stdout: "max_write_bytes": 0, 2024-04-20T01:53:15.279 INFO:teuthology.orchestra.run.smithi019.stdout: "enabled": false 2024-04-20T01:53:15.279 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:15.279 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:15.279 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:53:15.279 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_name": "test-realm", 2024-04-20T01:53:15.279 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:15.280 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:53:15.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:15.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:53:15.280 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:53:15.332 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:15.332 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:15.426 INFO:teuthology.orchestra.run.smithi019.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-92","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-93","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:53:15.427 DEBUG:boto:path=/aktlji-92/ 2024-04-20T01:53:15.427 DEBUG:boto:auth_path=/aktlji-92/ 2024-04-20T01:53:15.427 DEBUG:boto:Method: HEAD 2024-04-20T01:53:15.427 DEBUG:boto:Path: /aktlji-92/ 2024-04-20T01:53:15.427 DEBUG:boto:Data: 2024-04-20T01:53:15.427 DEBUG:boto:Headers: {} 2024-04-20T01:53:15.427 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:53:15.428 DEBUG:boto:Port: 8000 2024-04-20T01:53:15.428 DEBUG:boto:Params: {} 2024-04-20T01:53:15.428 DEBUG:boto:Token: None 2024-04-20T01:53:15.428 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:53:15 GMT /aktlji-92/ 2024-04-20T01:53:15.428 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qVmR/AVK/5r/GZ2c9pbUywXmzic= 2024-04-20T01:53:15.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qVmR/AVK/5r/GZ2c9pbUywXmzic=', 'Content-Length': '0'} 2024-04-20T01:53:15.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fdf581878a43dff2-006623200b-1e80-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:15.657 DEBUG:boto:path=/aktlji-92/obj1 2024-04-20T01:53:15.657 DEBUG:boto:auth_path=/aktlji-92/obj1 2024-04-20T01:53:15.657 DEBUG:boto:Method: PUT 2024-04-20T01:53:15.658 DEBUG:boto:Path: /aktlji-92/obj1 2024-04-20T01:53:15.658 DEBUG:boto:Data: 2024-04-20T01:53:15.658 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:53:15.658 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:53:15.658 DEBUG:boto:Port: 8000 2024-04-20T01:53:15.658 DEBUG:boto:Params: {} 2024-04-20T01:53:15.658 DEBUG:boto:Token: None 2024-04-20T01:53:15.659 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:53:15 GMT /aktlji-92/obj1 2024-04-20T01:53:15.659 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:zII3qIqFsLNeRxYV11N/xxImEio= 2024-04-20T01:53:15.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:53:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zII3qIqFsLNeRxYV11N/xxImEio='} 2024-04-20T01:53:15.692 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f57f74ddafa93020-006623200b-1e80-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:15.692 DEBUG:boto:path=/aktlji-92/ 2024-04-20T01:53:15.692 DEBUG:boto:auth_path=/aktlji-92/ 2024-04-20T01:53:15.692 DEBUG:boto:Method: HEAD 2024-04-20T01:53:15.692 DEBUG:boto:Path: /aktlji-92/ 2024-04-20T01:53:15.692 DEBUG:boto:Data: 2024-04-20T01:53:15.693 DEBUG:boto:Headers: {} 2024-04-20T01:53:15.693 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:53:15.693 DEBUG:boto:Port: 8000 2024-04-20T01:53:15.693 DEBUG:boto:Params: {} 2024-04-20T01:53:15.693 DEBUG:boto:Token: None 2024-04-20T01:53:15.693 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:53:15 GMT /aktlji-92/ 2024-04-20T01:53:15.693 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:qVmR/AVK/5r/GZ2c9pbUywXmzic= 2024-04-20T01:53:15.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qVmR/AVK/5r/GZ2c9pbUywXmzic=', 'Content-Length': '0'} 2024-04-20T01:53:15.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022773a0ff92d9f46-006623200b-1e80-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:15.701 DEBUG:boto:path=/aktlji-92/obj2 2024-04-20T01:53:15.701 DEBUG:boto:auth_path=/aktlji-92/obj2 2024-04-20T01:53:15.701 DEBUG:boto:Method: PUT 2024-04-20T01:53:15.701 DEBUG:boto:Path: /aktlji-92/obj2 2024-04-20T01:53:15.702 DEBUG:boto:Data: 2024-04-20T01:53:15.702 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:53:15.702 DEBUG:boto:Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:53:15.702 DEBUG:boto:Port: 8000 2024-04-20T01:53:15.702 DEBUG:boto:Params: {} 2024-04-20T01:53:15.702 DEBUG:boto:Token: None 2024-04-20T01:53:15.702 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 20 Apr 2024 01:53:15 GMT /aktlji-92/obj2 2024-04-20T01:53:15.702 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:a0maXniRS1UJN3PGkWpm6hb5ldA= 2024-04-20T01:53:15.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:53:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a0maXniRS1UJN3PGkWpm6hb5ldA='} 2024-04-20T01:53:15.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000601dd7760de354f1-006623200b-1e80-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:53:15.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:15.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:15.751 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:15.840 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:15.840 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:16.008 INFO:teuthology.orchestra.run.smithi019.stderr:No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:53:16.074 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:53:16.074 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:16.074 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:53:16.074 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:53:16.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:16.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:16.075 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:53:16.075 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "0.000000" 2024-04-20T01:53:16.075 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:16.075 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:16.075 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:53:16.075 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:53:08.757488Z" 2024-04-20T01:53:16.076 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:16.076 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:16.076 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:53:16.076 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:53:16.076 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:16.076 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:53:16.077 DEBUG:rgw_multi.tests:master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:16.077 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-04-20T01:53:16.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:16.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:16.078 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:16.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:53:16.264+0000 7fbe351fb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:16.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-20T01:53:16.266+0000 7fbe351fb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:16.295 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:53:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:16.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:53:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:53:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 1, 2024-04-20T01:53:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:53:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:16.297 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:53:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "", 2024-04-20T01:53:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:16.298 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "0.000000", 2024-04-20T01:53:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:53:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:53:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:16.299 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:53:08.757488Z", 2024-04-20T01:53:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "state": 1, 2024-04-20T01:53:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:53:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:16.300 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:53:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "realm_epoch": 2 2024-04-20T01:53:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 1, 2024-04-20T01:53:16.301 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 1 2024-04-20T01:53:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:16.302 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:16Z" 2024-04-20T01:53:16.302 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:16.303 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577988.757488_1755.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:53:08.757488Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:53:16Z" } 2024-04-20T01:53:16.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577988.757488_1755.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:53:08.757488Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:53:16.304 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:16.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:16.304 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-04-20T01:53:16.304 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-04-20T01:53:16.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:16.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:16.304 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:16.380 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:16.381 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:16.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:53:16.471+0000 7fd75af0f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:16.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-04-20T01:53:16.473+0000 7fd75af0f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:53:16.506 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-04-20T01:53:16.506 INFO:teuthology.orchestra.run.smithi019.stdout: "sync_status": { 2024-04-20T01:53:16.506 INFO:teuthology.orchestra.run.smithi019.stdout: "info": { 2024-04-20T01:53:16.506 INFO:teuthology.orchestra.run.smithi019.stdout: "status": "sync", 2024-04-20T01:53:16.506 INFO:teuthology.orchestra.run.smithi019.stdout: "num_shards": 4, 2024-04-20T01:53:16.506 INFO:teuthology.orchestra.run.smithi019.stdout: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:53:16.507 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:16.507 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:16.507 INFO:teuthology.orchestra.run.smithi019.stdout: "markers": [ 2024-04-20T01:53:16.507 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:16.507 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 0, 2024-04-20T01:53:16.507 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:16.507 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:16.507 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:53:16.507 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:16.507 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 1, 2024-04-20T01:53:16.507 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:16.508 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:53:16.508 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:16.508 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:16.508 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:16.508 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:16.508 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 1, 2024-04-20T01:53:16.508 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:16.508 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:16.508 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "", 2024-04-20T01:53:16.508 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:16.508 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:53:16.508 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:16.509 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "0.000000", 2024-04-20T01:53:16.509 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:16.509 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:16.509 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:16.509 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:16.509 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 2, 2024-04-20T01:53:16.509 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:16.509 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:16.509 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:53:16.509 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:16.509 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:53:16.509 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:16.510 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:53:08.757488Z", 2024-04-20T01:53:16.510 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:16.510 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:16.510 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:16.510 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:16.510 INFO:teuthology.orchestra.run.smithi019.stdout: "key": 3, 2024-04-20T01:53:16.510 INFO:teuthology.orchestra.run.smithi019.stdout: "val": { 2024-04-20T01:53:16.510 INFO:teuthology.orchestra.run.smithi019.stdout: "state": 1, 2024-04-20T01:53:16.510 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:53:16.510 INFO:teuthology.orchestra.run.smithi019.stdout: "next_step_marker": "", 2024-04-20T01:53:16.510 INFO:teuthology.orchestra.run.smithi019.stdout: "total_entries": 0, 2024-04-20T01:53:16.511 INFO:teuthology.orchestra.run.smithi019.stdout: "pos": 0, 2024-04-20T01:53:16.511 INFO:teuthology.orchestra.run.smithi019.stdout: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:53:16.511 INFO:teuthology.orchestra.run.smithi019.stdout: "realm_epoch": 2 2024-04-20T01:53:16.511 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:16.511 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:16.511 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-04-20T01:53:16.511 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:16.511 INFO:teuthology.orchestra.run.smithi019.stdout: "full_sync": { 2024-04-20T01:53:16.511 INFO:teuthology.orchestra.run.smithi019.stdout: "total": 1, 2024-04-20T01:53:16.511 INFO:teuthology.orchestra.run.smithi019.stdout: "complete": 1 2024-04-20T01:53:16.511 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:16.512 INFO:teuthology.orchestra.run.smithi019.stdout: "current_time": "2024-04-20T01:53:16Z" 2024-04-20T01:53:16.512 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-04-20T01:53:16.513 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "f494ba41-e721-4411-8938-a297d95f7246", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1713568837.840174_14.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-19T23:20:37.840174Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1713577988.757488_1755.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:53:08.757488Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1713575003.344070_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:03:23.344070Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-20T01:53:16Z" } 2024-04-20T01:53:16.513 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577988.757488_1755.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:53:08.757488Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:53:16.513 DEBUG:rgw_multi.tests:log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:16.513 DEBUG:rgw_multi.tests:sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:53:16.513 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-04-20T01:53:16.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:16.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:16.514 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:16.574 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:16.574 INFO:teuthology.orchestra.run.smithi019.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:16.680 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-04-20T01:53:16.680 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:16.680 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:53:16.680 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:53:16.680 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:16.680 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:16.681 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000028170", 2024-04-20T01:53:16.681 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:53:15.700250Z" 2024-04-20T01:53:16.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:16.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:16.681 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:53:16.681 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:53:16.681 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-04-20T01:53:16.681 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-04-20T01:53:16.681 INFO:teuthology.orchestra.run.smithi019.stdout: "marker": "00000000000000000000:00000000000000021578", 2024-04-20T01:53:16.681 INFO:teuthology.orchestra.run.smithi019.stdout: "last_update": "2024-04-20T01:53:15.749553Z" 2024-04-20T01:53:16.682 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-04-20T01:53:16.682 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-04-20T01:53:16.682 DEBUG:rgw_multi.tests:data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:53:16.682 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:53:16.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:16.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:16.683 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:16.816 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:53:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:16.817 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:53:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:53:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:53:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:16.818 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:53:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:53:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:53:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:16.819 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:53:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:53:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:16.820 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:53:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:53:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:16.821 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:53:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:53:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:16Z" 2024-04-20T01:53:16.822 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:16.823 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:53:16Z" } 2024-04-20T01:53:16.824 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:53:16.824 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:53:16.824 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:53:16.824 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:53:21.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:21.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:21.826 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:21.889 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:21.889 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:21.972 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:21.972 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:21.972 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:21.972 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:21.972 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:21.973 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:53:21.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:21.973 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:21.973 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:21.973 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:21.973 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:21.973 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:21.973 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:53:21.973 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:21.973 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:53:21.974 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:21.974 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:53:21.974 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:21.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:21.974 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:21.974 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:21.974 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:21.974 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:21.974 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:53:21.974 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:21.974 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:53:21.975 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:21.975 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:53:21.975 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:21.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:21.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:21.975 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:21.975 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:21.975 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:21.975 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:53:21.975 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:21.975 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:21.976 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:21.976 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:53:21.976 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:21.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:21.976 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:21.976 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:21.976 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:21.976 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:21.976 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:53:21.976 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:21.976 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:21.977 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:21.977 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:53:21.977 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:21.977 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:21.977 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:21.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:21.977 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:21.977 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:53:21.977 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:53:21.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:21.977 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:21Z" 2024-04-20T01:53:21.977 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:21.978 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:53:21Z" } 2024-04-20T01:53:21.979 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:53:21.979 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:53:21.979 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:53:21.979 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:53:26.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:26.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:26.980 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:27.041 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:27.042 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:27.103 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:53:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:27.104 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:53:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:53:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:53:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:27.105 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:53:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:53:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:53:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:27.106 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:53:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:53:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:27.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:53:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:53:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:27.108 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:53:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:53:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:27.109 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:27Z" 2024-04-20T01:53:27.109 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:27.110 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:53:27Z" } 2024-04-20T01:53:27.110 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:53:27.110 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:53:27.110 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:53:27.110 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:53:32.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:32.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:32.112 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:32.172 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:32.173 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:32.249 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:32.249 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:32.249 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:32.250 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:32.250 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:32.250 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:53:32.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:32.250 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:32.250 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:32.250 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:32.250 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:32.250 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:32.250 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:53:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:53:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:53:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:53:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:53:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:53:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:53:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:53:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:32.254 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:32.254 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:53:32.254 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:32.254 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:32.254 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:32.254 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:53:32.254 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:32.254 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:32.254 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:32.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:32.254 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:32.254 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:53:32.255 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:53:32.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:32.255 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:32Z" 2024-04-20T01:53:32.255 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:32.256 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:53:32Z" } 2024-04-20T01:53:32.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:53:32.256 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:53:32.256 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:53:32.256 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:53:37.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:37.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:37.258 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:37.320 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:37.321 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:37.384 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:37.384 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:53:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:37.385 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:53:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:53:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:53:37.386 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:53:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:53:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:53:37.387 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:53:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:53:37.388 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:53:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:37.389 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:53:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:37.391 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:53:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:53:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:37.392 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:37Z" 2024-04-20T01:53:37.392 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:37.393 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:53:37Z" } 2024-04-20T01:53:37.393 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:53:37.393 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:53:37.393 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:53:37.393 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:53:42.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:42.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:42.395 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:42.456 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:42.456 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:42.534 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:42.534 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:42.535 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:42.535 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:42.535 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:42.535 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:53:42.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:42.535 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:42.535 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:42.535 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:42.535 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:42.535 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:42.535 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:53:42.536 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:42.536 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:53:42.536 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:42.536 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:53:42.536 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:42.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:42.536 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:42.536 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:42.536 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:42.536 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:42.536 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:53:42.537 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:42.537 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:53:42.537 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:42.537 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:53:42.537 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:42.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:42.537 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:42.537 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:42.537 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:42.537 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:42.537 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:53:42.537 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:42.538 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:42.538 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:42.538 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:53:42.538 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:42.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:42.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:42.538 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:42.538 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:42.538 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:42.538 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:53:42.538 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:53:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:53:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:53:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:42.539 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:42Z" 2024-04-20T01:53:42.540 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:42.540 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:53:42Z" } 2024-04-20T01:53:42.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:53:42.541 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:53:42.541 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:53:42.541 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:53:47.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:47.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:47.543 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:47.605 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:47.605 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:47.668 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:47.668 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:47.668 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:47.668 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:47.668 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:47.669 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:53:47.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:47.669 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:47.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:47.669 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:47.669 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:47.669 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:47.669 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:53:47.669 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:47.669 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:53:47.669 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:47.670 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:53:47.670 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:47.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:47.670 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:47.670 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:47.670 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:47.670 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:47.670 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:53:47.670 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:47.670 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:53:47.670 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:47.671 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:53:47.671 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:47.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:47.671 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:47.671 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:47.671 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:47.671 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:47.671 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:53:47.671 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:47.671 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:47.671 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:47.671 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:53:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:53:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:53:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:47.673 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:47.673 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:47.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:47.673 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:47.673 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:53:47.673 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:53:47.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:47.673 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:47Z" 2024-04-20T01:53:47.673 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:47.674 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:53:47Z" } 2024-04-20T01:53:47.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:53:47.675 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:53:47.675 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:53:47.675 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:53:52.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:52.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:52.677 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:52.740 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:52.740 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:52.826 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:52.826 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:52.827 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:52.827 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:52.827 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:52.827 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:53:52.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:52.827 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:52.827 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:52.827 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:52.827 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:52.827 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:52.828 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:53:52.828 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:52.828 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:53:52.828 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:52.828 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:53:52.828 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:52.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:52.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:52.828 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:52.828 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:52.828 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:52.829 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:53:52.829 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:52.829 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:53:52.829 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:52.829 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:53:52.829 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:52.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:52.829 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:52.829 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:52.829 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:52.830 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:52.830 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:53:52.830 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:52.830 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:52.830 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:52.830 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:53:52.830 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:52.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:52.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:52.830 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:52.830 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:52.830 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:52.831 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:53:52.831 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:52.831 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:52.831 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:52.831 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:53:52.831 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:52.831 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:52.831 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:52.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:52.832 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:52.832 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:53:52.832 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:53:52.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:52.832 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:52Z" 2024-04-20T01:53:52.832 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:52.833 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:53:52Z" } 2024-04-20T01:53:52.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:53:52.843 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:53:52.843 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:53:52.843 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:53:57.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:57.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:53:57.844 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:53:57.908 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:53:57.908 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:53:57.986 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:53:57.987 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:53:57.987 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:53:57.987 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:53:57.987 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:53:57.987 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:53:57.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:57.987 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:53:57.987 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:57.987 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:53:57.987 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:57.988 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:57.988 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:53:57.988 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:57.988 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:53:57.988 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:57.988 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:53:57.988 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:57.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:57.988 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:57.988 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:53:57.988 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:57.989 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:57.989 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:53:57.989 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:57.989 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:53:57.989 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:57.989 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:53:57.989 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:57.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:57.989 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:57.989 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:53:57.989 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:57.990 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:57.990 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:53:57.990 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:57.990 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:57.990 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:57.990 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:53:57.990 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:57.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:57.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:53:57.990 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:53:57.991 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:53:57.991 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:53:57.991 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:53:57.991 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:53:57.991 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:53:57.991 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:53:57.991 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:53:57.991 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:57.991 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:53:57.991 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:53:57.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:57.991 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:53:57.992 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:53:57.992 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:53:57.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:53:57.992 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:53:57Z" 2024-04-20T01:53:57.992 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:53:57.993 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:53:57Z" } 2024-04-20T01:53:57.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:53:57.993 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:53:57.993 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:53:57.994 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:54:02.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:02.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:02.995 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:54:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:54:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:54:03.123 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:54:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:54:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:54:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:54:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:54:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:54:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:54:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:54:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:54:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:54:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:54:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:54:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:54:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:54:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:54:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:54:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:54:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:54:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:54:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:54:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:54:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:54:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:54:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:54:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:54:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:54:03Z" 2024-04-20T01:54:03.129 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:54:03.130 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:54:03Z" } 2024-04-20T01:54:03.130 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:54:03.130 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:54:03.130 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:54:03.131 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:54:08.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:08.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:08.132 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:54:08.193 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:54:08.194 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:54:08.265 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:54:08.265 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:54:08.266 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:54:08.266 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:54:08.266 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:54:08.266 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:54:08.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:08.266 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:54:08.266 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:08.266 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:54:08.266 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:08.266 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:54:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:54:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:54:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:08.267 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:54:08.268 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:08.268 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:08.268 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:54:08.268 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:08.268 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:54:08.268 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:08.268 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:54:08.268 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:08.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:08.268 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:08.268 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:54:08.269 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:08.269 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:08.269 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:54:08.269 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:08.269 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:08.269 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:08.269 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:54:08.269 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:08.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:08.270 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:08.270 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:54:08.270 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:08.270 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:08.270 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:54:08.270 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:08.270 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:08.270 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:08.270 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:54:08.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:08.271 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:08.271 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:54:08.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:08.271 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:54:08.271 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:54:08.271 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:54:08.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:08.271 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:54:08Z" 2024-04-20T01:54:08.272 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:54:08.273 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:54:08Z" } 2024-04-20T01:54:08.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:54:08.273 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:54:08.273 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:54:08.273 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:54:13.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:13.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:13.274 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:54:13.336 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:54:13.336 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:54:13.398 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:54:13.398 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:54:13.398 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:54:13.398 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:54:13.399 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:54:13.399 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:54:13.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:13.399 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:54:13.399 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:13.399 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:54:13.399 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:13.399 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:13.399 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:54:13.399 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:13.399 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:54:13.400 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:13.400 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:54:13.400 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:13.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:13.400 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:13.400 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:54:13.400 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:13.400 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:13.400 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:54:13.400 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:13.400 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:54:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:54:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:54:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:54:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:13.402 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:13.402 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:54:13.402 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:13.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:13.402 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:13.402 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:54:13.402 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:13.402 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:13.402 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:54:13.402 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:13.403 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:13.403 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:13.403 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:54:13.403 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:13.403 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:13.403 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:54:13.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:13.403 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:54:13.403 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:54:13.403 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:54:13.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:13.403 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:54:13Z" 2024-04-20T01:54:13.404 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:54:13.405 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:54:13Z" } 2024-04-20T01:54:13.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:54:13.405 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:54:13.405 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:54:13.405 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:54:18.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:18.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:18.406 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:54:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:54:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:54:18.556 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:54:18.557 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:54:18.557 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:54:18.557 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:54:18.557 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:54:18.557 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:54:18.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:18.558 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:54:18.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:18.558 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:54:18.558 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:18.559 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:18.559 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:54:18.559 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:18.559 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:54:18.559 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:18.559 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:54:18.560 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:18.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:18.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:18.560 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:54:18.560 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:18.561 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:18.561 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:54:18.561 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:18.561 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:54:18.561 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:18.562 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:54:18.562 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:18.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:18.562 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:18.562 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:54:18.563 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:18.563 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:18.563 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:54:18.563 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:18.564 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:18.564 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:18.564 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:54:18.564 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:18.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:18.564 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:18.564 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:54:18.564 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:18.565 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:18.565 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:54:18.565 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:18.565 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:18.565 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:18.565 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:54:18.566 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:18.566 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:18.566 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:54:18.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:18.566 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:54:18.566 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:54:18.566 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:54:18.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:18.567 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:54:18Z" 2024-04-20T01:54:18.567 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:54:18.568 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:54:18Z" } 2024-04-20T01:54:18.568 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:54:18.568 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:54:18.568 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:54:18.568 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:54:23.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:23.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:23.570 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:54:23.632 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:54:23.632 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:54:23.700 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:54:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:54:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:54:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:54:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:54:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:54:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:54:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:54:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:54:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:54:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:54:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:54:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:54:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:54:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:54:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:54:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:54:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:54:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:54:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:54:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:54:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:54:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:54:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:54:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:54:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:54:23Z" 2024-04-20T01:54:23.705 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:54:23.706 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:54:23Z" } 2024-04-20T01:54:23.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:54:23.706 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:54:23.706 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:54:23.707 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:54:28.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:28.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:28.708 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:54:28.769 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:54:28.770 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:54:28.834 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:54:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:54:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:54:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:54:28.834 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:54:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:54:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:54:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:54:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:54:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:54:28.835 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:54:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:54:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:54:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:54:28.836 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:54:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:54:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:54:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:28.837 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:54:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:54:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:54:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:28.838 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:28.839 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:54:28.839 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:28.839 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:28.839 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:54:28.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:28.839 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:54:28.839 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:54:28.839 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:54:28.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:28.839 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:54:28Z" 2024-04-20T01:54:28.839 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:54:28.840 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:54:28Z" } 2024-04-20T01:54:28.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:54:28.841 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:54:28.841 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:54:28.841 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:54:33.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:33.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:33.843 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:54:33.905 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:54:33.905 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:54:33.993 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:54:33.993 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:54:33.993 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:54:33.993 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:54:33.993 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:54:33.993 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:54:33.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:33.994 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:54:33.994 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:33.994 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:54:33.994 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:33.994 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:33.994 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:54:33.994 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:33.994 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:54:33.994 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:33.994 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:54:33.995 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:33.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:33.995 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:33.995 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:54:33.995 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:33.995 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:33.995 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:54:33.995 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:33.995 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:54:33.995 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:33.995 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:54:33.996 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:33.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:33.996 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:33.996 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:54:33.996 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:33.996 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:33.996 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:54:33.996 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:33.996 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:33.996 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:33.997 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:54:33.997 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:33.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:33.997 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:33.997 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:54:33.997 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:33.997 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:33.997 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:54:33.997 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:33.997 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:33.997 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:33.998 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:54:33.998 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:33.998 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:33.998 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:54:33.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:33.998 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:54:33.998 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:54:33.998 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:54:33.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:33.998 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:54:33Z" 2024-04-20T01:54:33.998 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:54:33.999 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:54:33Z" } 2024-04-20T01:54:34.000 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:54:34.000 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:54:34.000 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:54:34.000 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:54:39.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:39.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:39.001 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:54:39.064 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:54:39.064 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:54:39.139 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:54:39.139 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:54:39.139 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:54:39.139 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:54:39.140 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:54:39.140 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:54:39.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:39.140 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:54:39.140 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:39.140 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:54:39.140 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:39.140 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:39.140 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:54:39.140 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:39.141 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:54:39.141 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:39.141 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:54:39.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:39.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:39.141 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:39.141 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:54:39.141 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:39.141 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:39.141 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:54:39.141 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:39.142 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:54:39.142 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:39.142 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:54:39.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:39.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:39.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:39.142 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:54:39.142 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:39.142 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:39.142 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:54:39.142 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:39.143 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:39.143 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:39.143 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:54:39.143 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:39.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:39.143 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:39.143 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:54:39.143 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:39.143 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:39.143 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:54:39.143 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:39.144 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:39.144 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:39.144 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:54:39.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:39.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:39.144 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:54:39.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:39.144 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:54:39.144 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:54:39.144 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:54:39.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:39.144 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:54:39Z" 2024-04-20T01:54:39.145 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:54:39.146 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:54:39Z" } 2024-04-20T01:54:39.146 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:54:39.146 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:54:39.146 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:54:39.146 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:54:44.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:44.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:44.150 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:54:44.211 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:54:44.211 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:54:44.278 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:54:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:54:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:54:44.278 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:54:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:54:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:54:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:54:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:54:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:54:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:44.279 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:54:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:54:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:54:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:54:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:44.280 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:54:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:54:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:54:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:54:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:44.281 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:54:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:54:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:54:44.282 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:54:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:54:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:54:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:54:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:54:44.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:54:44Z" 2024-04-20T01:54:44.284 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:54:44.285 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:54:44Z" } 2024-04-20T01:54:44.285 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:54:44.285 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:54:44.285 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:54:44.285 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:54:49.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:49.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:49.287 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:54:49.349 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:54:49.349 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:54:49.420 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:54:49.420 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:54:49.420 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:54:49.420 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:54:49.420 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:54:49.420 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:54:49.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:49.421 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:54:49.421 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:49.421 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:54:49.421 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:49.421 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:49.421 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:54:49.421 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:49.421 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:54:49.421 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:54:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:54:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:54:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:49.422 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:54:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:54:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:54:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:54:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:49.423 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:54:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:54:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:54:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:49.424 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:49.425 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:54:49.425 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:49.425 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:49.425 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:54:49.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:49.425 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:54:49.425 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:54:49.425 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:54:49.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:49.426 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:54:49Z" 2024-04-20T01:54:49.426 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:54:49.427 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:54:49Z" } 2024-04-20T01:54:49.427 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:54:49.427 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:54:49.427 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:54:49.427 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:54:54.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:54.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:54.429 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:54:54.491 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:54:54.491 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:54:54.557 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:54:54.557 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:54:54.557 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:54:54.557 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:54:54.557 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:54:54.557 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:54:54.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:54.557 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:54:54.558 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:54.558 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:54:54.558 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:54.558 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:54.558 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:54:54.558 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:54.558 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:54:54.558 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:54.558 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:54:54.558 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:54.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:54.559 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:54.559 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:54:54.559 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:54.559 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:54.559 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:54:54.559 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:54.559 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:54:54.559 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:54.559 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:54:54.559 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:54.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:54.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:54.560 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:54:54.560 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:54.560 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:54.560 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:54:54.560 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:54.560 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:54.560 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:54.560 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:54:54.560 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:54.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:54.560 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:54.561 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:54:54.561 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:54.561 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:54.561 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:54:54.561 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:54.561 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:54.561 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:54.561 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:54:54.561 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:54.561 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:54.561 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:54:54.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:54.562 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:54:54.562 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:54:54.562 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:54:54.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:54.562 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:54:54Z" 2024-04-20T01:54:54.562 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:54:54.563 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:54:54Z" } 2024-04-20T01:54:54.563 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:54:54.564 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:54:54.564 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:54:54.564 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:54:59.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:59.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:54:59.565 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:54:59.626 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:54:59.627 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:54:59.706 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:54:59.706 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:54:59.706 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:54:59.706 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:54:59.706 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:54:59.706 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:54:59.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:59.707 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:54:59.707 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:59.707 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:54:59.707 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:59.707 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:59.707 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:54:59.707 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:59.707 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:54:59.707 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:59.707 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:54:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:54:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:54:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:54:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:54:59.708 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:59.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:59.709 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:59.709 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:54:59.709 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:59.709 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:59.709 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:54:59.709 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:59.709 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:59.709 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:59.709 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:54:59.709 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:59.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:59.710 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:54:59.710 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:54:59.710 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:54:59.710 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:54:59.710 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:54:59.710 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:54:59.710 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:54:59.710 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:54:59.710 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:54:59.710 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:59.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:54:59.711 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:54:59.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:59.711 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:54:59.711 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:54:59.711 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:54:59.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:54:59.711 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:54:59Z" 2024-04-20T01:54:59.711 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:54:59.712 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:54:59Z" } 2024-04-20T01:54:59.712 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:54:59.713 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:54:59.713 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:54:59.713 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:04.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:04.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:04.714 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:55:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:55:04.857 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:55:04.857 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:55:04.857 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:55:04.857 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:55:04.858 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:55:04.858 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:55:04.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:04.858 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:55:04.858 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:04.858 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:55:04.858 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:04.858 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:04.858 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:04.858 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:04.859 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:55:04.859 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:04.859 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:04.859 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:04.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:04.859 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:04.859 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:55:04.859 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:04.859 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:04.859 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:04.859 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:04.859 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:55:04.860 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:04.860 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:04.860 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:04.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:04.860 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:04.860 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:55:04.860 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:04.860 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:04.860 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:04.860 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:04.860 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:55:04.861 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:04.861 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:04.861 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:04.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:04.861 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:04.861 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:55:04.861 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:04.861 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:04.862 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:04.862 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:04.862 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:55:04.862 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:04.863 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:04.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:04.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:04.863 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:55:04.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:04.863 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:55:04.863 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:55:04.863 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:55:04.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:04.864 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:55:04Z" 2024-04-20T01:55:04.864 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:55:04.865 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:55:04Z" } 2024-04-20T01:55:04.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:04.865 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:04.865 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:04.865 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:09.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:09.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:09.870 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:09.931 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:55:09.932 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:55:10.008 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:55:10.008 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:55:10.008 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:55:10.008 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:55:10.008 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:55:10.008 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:55:10.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:10.009 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:55:10.009 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:10.009 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:55:10.009 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:10.009 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:10.009 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:10.009 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:10.009 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:55:10.009 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:10.009 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:10.009 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:10.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:10.010 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:10.010 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:55:10.010 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:10.010 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:10.010 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:10.010 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:10.010 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:55:10.010 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:10.010 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:10.011 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:10.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:10.011 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:10.011 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:55:10.011 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:10.011 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:10.011 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:10.011 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:10.012 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:55:10.012 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:10.012 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:10.012 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:10.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:10.012 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:10.012 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:55:10.012 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:10.012 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:10.012 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:10.012 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:10.013 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:55:10.013 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:10.013 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:10.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:10.014 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:10.014 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:55:10.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:10.014 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:55:10.014 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:55:10.014 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:55:10.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:10.014 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:55:09Z" 2024-04-20T01:55:10.015 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:55:10.015 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:55:09Z" } 2024-04-20T01:55:10.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:10.017 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:10.017 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:10.017 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:15.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:15.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:15.018 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:15.081 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:55:15.082 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:55:15.148 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:55:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:55:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:55:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:55:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:55:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:55:15.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:55:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:55:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:55:15.149 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:55:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:15.150 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:55:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:55:15.151 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:55:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:55:15.152 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:55:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:55:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:15.153 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:55:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:55:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:55:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:15.154 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:55:15Z" 2024-04-20T01:55:15.154 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:55:15.155 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028030", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:51:02.453429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021438", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:51:02.403172Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:55:15Z" } 2024-04-20T01:55:15.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:15.155 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:15.155 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:15.156 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:20.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.157 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:20.218 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setuser ceph since I am not root 2024-04-20T01:55:20.218 INFO:teuthology.orchestra.run.smithi031.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T01:55:20.292 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-20T01:55:20.293 INFO:teuthology.orchestra.run.smithi031.stdout: "sync_status": { 2024-04-20T01:55:20.293 INFO:teuthology.orchestra.run.smithi031.stdout: "info": { 2024-04-20T01:55:20.293 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "sync", 2024-04-20T01:55:20.293 INFO:teuthology.orchestra.run.smithi031.stdout: "num_shards": 4, 2024-04-20T01:55:20.293 INFO:teuthology.orchestra.run.smithi031.stdout: "instance_id": 17812922715370875371 2024-04-20T01:55:20.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:20.293 INFO:teuthology.orchestra.run.smithi031.stdout: "markers": [ 2024-04-20T01:55:20.293 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:20.293 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 0, 2024-04-20T01:55:20.294 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:20.294 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:20.294 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:20.294 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:20.294 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 9, 2024-04-20T01:55:20.294 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:20.294 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:20.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:20.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:20.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:20.295 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 1, 2024-04-20T01:55:20.295 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:20.295 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:20.295 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000028170", 2024-04-20T01:55:20.295 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:20.295 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 70, 2024-04-20T01:55:20.295 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:20.295 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:53:15.700250Z" 2024-04-20T01:55:20.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:20.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:20.296 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:20.296 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 2, 2024-04-20T01:55:20.296 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:20.296 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:20.296 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:20.296 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:20.296 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:55:20.296 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:20.296 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:20.296 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:20.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:20.297 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-20T01:55:20.297 INFO:teuthology.orchestra.run.smithi031.stdout: "key": 3, 2024-04-20T01:55:20.297 INFO:teuthology.orchestra.run.smithi031.stdout: "val": { 2024-04-20T01:55:20.297 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "incremental-sync", 2024-04-20T01:55:20.297 INFO:teuthology.orchestra.run.smithi031.stdout: "marker": "00000000000000000000:00000000000000021578", 2024-04-20T01:55:20.297 INFO:teuthology.orchestra.run.smithi031.stdout: "next_step_marker": "", 2024-04-20T01:55:20.297 INFO:teuthology.orchestra.run.smithi031.stdout: "total_entries": 0, 2024-04-20T01:55:20.297 INFO:teuthology.orchestra.run.smithi031.stdout: "pos": 0, 2024-04-20T01:55:20.297 INFO:teuthology.orchestra.run.smithi031.stdout: "timestamp": "2024-04-20T01:53:15.749553Z" 2024-04-20T01:55:20.297 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:20.297 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-20T01:55:20.298 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-20T01:55:20.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:20.298 INFO:teuthology.orchestra.run.smithi031.stdout: "full_sync": { 2024-04-20T01:55:20.298 INFO:teuthology.orchestra.run.smithi031.stdout: "total": 79, 2024-04-20T01:55:20.298 INFO:teuthology.orchestra.run.smithi031.stdout: "complete": 79 2024-04-20T01:55:20.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-20T01:55:20.298 INFO:teuthology.orchestra.run.smithi031.stdout: "current_time": "2024-04-20T01:55:20Z" 2024-04-20T01:55:20.298 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-20T01:55:20.299 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17812922715370875371 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015483", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-20T01:51:02.485059Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028170", "next_step_marker": "", "total_entries": 70, "pos": 0, "timestamp": "2024-04-20T01:53:15.700250Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024930", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:50:15.891278Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000021578", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-20T01:53:15.749553Z" } } ] }, "full_sync": { "total": 79, "complete": 79 }, "current_time": "2024-04-20T01:55:20Z" } 2024-04-20T01:55:20.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028170', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:53:15.700250Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021578', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:53:15.749553Z'}}] 2024-04-20T01:55:20.299 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:20.300 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:20.300 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:20.300 DEBUG:boto:path=/aktlji-92/ 2024-04-20T01:55:20.300 DEBUG:boto:auth_path=/aktlji-92/ 2024-04-20T01:55:20.300 DEBUG:boto:Method: HEAD 2024-04-20T01:55:20.300 DEBUG:boto:Path: /aktlji-92/ 2024-04-20T01:55:20.300 DEBUG:boto:Data: 2024-04-20T01:55:20.301 DEBUG:boto:Headers: {} 2024-04-20T01:55:20.301 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.301 DEBUG:boto:Port: 8000 2024-04-20T01:55:20.301 DEBUG:boto:Params: {} 2024-04-20T01:55:20.302 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:20.302 DEBUG:boto:Token: None 2024-04-20T01:55:20.302 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:55:20 GMT /aktlji-92/ 2024-04-20T01:55:20.302 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:slSwALBBm9Fj87g8ENESDuXrTYg= 2024-04-20T01:55:20.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:slSwALBBm9Fj87g8ENESDuXrTYg=', 'Content-Length': '0'} 2024-04-20T01:55:20.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd5784dea44da6ee-0066232088-28bb-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.311 DEBUG:boto:path=/aktlji-92/obj1 2024-04-20T01:55:20.311 DEBUG:boto:auth_path=/aktlji-92/obj1 2024-04-20T01:55:20.311 DEBUG:boto:Method: HEAD 2024-04-20T01:55:20.311 DEBUG:boto:Path: /aktlji-92/obj1 2024-04-20T01:55:20.311 DEBUG:boto:Data: 2024-04-20T01:55:20.311 DEBUG:boto:Headers: {} 2024-04-20T01:55:20.311 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.311 DEBUG:boto:Port: 8000 2024-04-20T01:55:20.311 DEBUG:boto:Params: {} 2024-04-20T01:55:20.312 DEBUG:boto:Token: None 2024-04-20T01:55:20.312 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:55:20 GMT /aktlji-92/obj1 2024-04-20T01:55:20.312 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ol8bukn6m9vlt1gGZe+7Fd1fLHo= 2024-04-20T01:55:20.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ol8bukn6m9vlt1gGZe+7Fd1fLHo=', 'Content-Length': '0'} 2024-04-20T01:55:20.315 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-92:73f46f02-c7ab-4426-a267-2273af911da5.7664.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005351028d89ba8175-0066232088-28bb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.316 DEBUG:boto:path=/aktlji-92/obj1 2024-04-20T01:55:20.317 DEBUG:boto:auth_path=/aktlji-92/obj1 2024-04-20T01:55:20.317 DEBUG:boto:Method: GET 2024-04-20T01:55:20.317 DEBUG:boto:Path: /aktlji-92/obj1 2024-04-20T01:55:20.317 DEBUG:boto:Data: 2024-04-20T01:55:20.317 DEBUG:boto:Headers: {} 2024-04-20T01:55:20.317 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.317 DEBUG:boto:Port: 8000 2024-04-20T01:55:20.317 DEBUG:boto:Params: {} 2024-04-20T01:55:20.317 DEBUG:boto:Token: None 2024-04-20T01:55:20.318 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:55:20 GMT /aktlji-92/obj1 2024-04-20T01:55:20.318 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:RntvL+cK2GCjd+EyEwe5kxi2j08= 2024-04-20T01:55:20.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RntvL+cK2GCjd+EyEwe5kxi2j08=', 'Content-Length': '0'} 2024-04-20T01:55:20.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-92:73f46f02-c7ab-4426-a267-2273af911da5.7664.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052e28215de0c37f9-0066232088-28bb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.320 DEBUG:boto:path=/aktlji-92/obj2 2024-04-20T01:55:20.320 DEBUG:boto:auth_path=/aktlji-92/obj2 2024-04-20T01:55:20.320 DEBUG:boto:Method: HEAD 2024-04-20T01:55:20.321 DEBUG:boto:Path: /aktlji-92/obj2 2024-04-20T01:55:20.321 DEBUG:boto:Data: 2024-04-20T01:55:20.321 DEBUG:boto:Headers: {} 2024-04-20T01:55:20.321 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.321 DEBUG:boto:Port: 8000 2024-04-20T01:55:20.321 DEBUG:boto:Params: {} 2024-04-20T01:55:20.321 DEBUG:boto:Token: None 2024-04-20T01:55:20.321 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:55:20 GMT /aktlji-92/obj2 2024-04-20T01:55:20.321 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:iW34LuqJ9whOAY32a0/JeYjQnkc= 2024-04-20T01:55:20.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iW34LuqJ9whOAY32a0/JeYjQnkc=', 'Content-Length': '0'} 2024-04-20T01:55:20.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-92:73f46f02-c7ab-4426-a267-2273af911da5.7664.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006364a2d23a6a34e5-0066232088-28bb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.323 DEBUG:boto:path=/aktlji-92/obj2 2024-04-20T01:55:20.323 DEBUG:boto:auth_path=/aktlji-92/obj2 2024-04-20T01:55:20.324 DEBUG:boto:Method: GET 2024-04-20T01:55:20.324 DEBUG:boto:Path: /aktlji-92/obj2 2024-04-20T01:55:20.324 DEBUG:boto:Data: 2024-04-20T01:55:20.324 DEBUG:boto:Headers: {} 2024-04-20T01:55:20.324 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.324 DEBUG:boto:Port: 8000 2024-04-20T01:55:20.324 DEBUG:boto:Params: {} 2024-04-20T01:55:20.324 DEBUG:boto:Token: None 2024-04-20T01:55:20.324 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:55:20 GMT /aktlji-92/obj2 2024-04-20T01:55:20.325 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:W+kENkDcMnesPdNTQIEK8+U27hk= 2024-04-20T01:55:20.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W+kENkDcMnesPdNTQIEK8+U27hk=', 'Content-Length': '0'} 2024-04-20T01:55:20.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-92:73f46f02-c7ab-4426-a267-2273af911da5.7664.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdd251937e8de5a0-0066232088-28bb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.374 DEBUG:boto:path=/aktlji-93/ 2024-04-20T01:55:20.375 DEBUG:boto:auth_path=/aktlji-93/ 2024-04-20T01:55:20.375 DEBUG:boto:Method: HEAD 2024-04-20T01:55:20.375 DEBUG:boto:Path: /aktlji-93/ 2024-04-20T01:55:20.375 DEBUG:boto:Data: 2024-04-20T01:55:20.375 DEBUG:boto:Headers: {} 2024-04-20T01:55:20.375 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.376 DEBUG:boto:Port: 8000 2024-04-20T01:55:20.376 DEBUG:boto:Params: {} 2024-04-20T01:55:20.376 DEBUG:boto:Token: None 2024-04-20T01:55:20.376 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:55:20 GMT /aktlji-93/ 2024-04-20T01:55:20.377 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:ShlfjV73ipt3eZY8+uPx45eEFsA= 2024-04-20T01:55:20.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ShlfjV73ipt3eZY8+uPx45eEFsA=', 'Content-Length': '0'} 2024-04-20T01:55:20.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c75a49d653e536d4-0066232088-28bb-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.380 DEBUG:boto:path=/aktlji-93/obj1 2024-04-20T01:55:20.380 DEBUG:boto:auth_path=/aktlji-93/obj1 2024-04-20T01:55:20.381 DEBUG:boto:Method: HEAD 2024-04-20T01:55:20.381 DEBUG:boto:Path: /aktlji-93/obj1 2024-04-20T01:55:20.381 DEBUG:boto:Data: 2024-04-20T01:55:20.381 DEBUG:boto:Headers: {} 2024-04-20T01:55:20.381 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.382 DEBUG:boto:Port: 8000 2024-04-20T01:55:20.382 DEBUG:boto:Params: {} 2024-04-20T01:55:20.382 DEBUG:boto:Token: None 2024-04-20T01:55:20.382 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:55:20 GMT /aktlji-93/obj1 2024-04-20T01:55:20.383 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:w4ww0G51T4OEnuTKu9p+zMWyrp4= 2024-04-20T01:55:20.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:w4ww0G51T4OEnuTKu9p+zMWyrp4=', 'Content-Length': '0'} 2024-04-20T01:55:20.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-92:73f46f02-c7ab-4426-a267-2273af911da5.7664.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000723edad7b0294638-0066232088-28bb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.385 DEBUG:boto:path=/aktlji-93/obj1 2024-04-20T01:55:20.385 DEBUG:boto:auth_path=/aktlji-93/obj1 2024-04-20T01:55:20.386 DEBUG:boto:Method: GET 2024-04-20T01:55:20.386 DEBUG:boto:Path: /aktlji-93/obj1 2024-04-20T01:55:20.386 DEBUG:boto:Data: 2024-04-20T01:55:20.386 DEBUG:boto:Headers: {} 2024-04-20T01:55:20.386 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.386 DEBUG:boto:Port: 8000 2024-04-20T01:55:20.386 DEBUG:boto:Params: {} 2024-04-20T01:55:20.386 DEBUG:boto:Token: None 2024-04-20T01:55:20.386 DEBUG:boto:StringToSign: GET Sat, 20 Apr 2024 01:55:20 GMT /aktlji-93/obj1 2024-04-20T01:55:20.387 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:o7eHfSBAxxy//YB94p0L/JOOCxg= 2024-04-20T01:55:20.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o7eHfSBAxxy//YB94p0L/JOOCxg=', 'Content-Length': '0'} 2024-04-20T01:55:20.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-92:73f46f02-c7ab-4426-a267-2273af911da5.7664.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099360a3a70dd4d1b-0066232088-28bb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.389 DEBUG:boto:path=/aktlji-93/obj2 2024-04-20T01:55:20.389 DEBUG:boto:auth_path=/aktlji-93/obj2 2024-04-20T01:55:20.389 DEBUG:boto:Method: HEAD 2024-04-20T01:55:20.389 DEBUG:boto:Path: /aktlji-93/obj2 2024-04-20T01:55:20.389 DEBUG:boto:Data: 2024-04-20T01:55:20.389 DEBUG:boto:Headers: {} 2024-04-20T01:55:20.389 DEBUG:boto:Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.390 DEBUG:boto:Port: 8000 2024-04-20T01:55:20.390 DEBUG:boto:Params: {} 2024-04-20T01:55:20.390 DEBUG:boto:Token: None 2024-04-20T01:55:20.390 DEBUG:boto:StringToSign: HEAD Sat, 20 Apr 2024 01:55:20 GMT /aktlji-93/obj2 2024-04-20T01:55:20.390 DEBUG:boto:Signature: AWS T4PF8H1DJQT67HZSKX5F:J/v5zu5qmYipXULkPQvod0NFn+M= 2024-04-20T01:55:20.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:J/v5zu5qmYipXULkPQvod0NFn+M=', 'Content-Length': '0'} 2024-04-20T01:55:20.392 DEBUG:boto:Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a7624b452ef0c9d3-0066232088-28bb-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.395 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... FAIL 2024-04-20T01:55:20.395 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.396 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:20.396 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_encrypted_object_sync 2024-04-20T01:55:20.396 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:20.396 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:20.396 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:20.396 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:20.396 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1319, in test_encrypted_object_sync 2024-04-20T01:55:20.396 INFO:tasks.rgw_multisite_tests: eq(data, key.get_contents_as_string(headers=sse_c_headers, encoding='ascii')) 2024-04-20T01:55:20.396 INFO:tasks.rgw_multisite_tests:AttributeError: 'NoneType' object has no attribute 'get_contents_as_string' 2024-04-20T01:55:20.396 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-67 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-67/ 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-67/ 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-67/ 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests:/aktlji-67/ 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Q0f2EBTSb8yD5E83s5WCCGgxV+M= 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q0f2EBTSb8yD5E83s5WCCGgxV+M=', 'Content-Length': '0'} 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000777525ac7a25ec23-0066230cb0-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-67/testobj-sse-c 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-67/testobj-sse-c 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-67/testobj-sse-c 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-algorithm:AES256 2024-04-20T01:55:20.399 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:/aktlji-67/testobj-sse-c 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:w1vJCP+ahsdL3zAFW/RmmgtpOp4= 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:w1vJCP+ahsdL3zAFW/RmmgtpOp4='} 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000637b91c77d52e874-0066230cb0-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-67/testobj-sse-kms 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-67/testobj-sse-kms 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-67/testobj-sse-kms 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption:aws:kms 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-aws-kms-key-id:testkey-1 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:/aktlji-67/testobj-sse-kms 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:mP01sqkX0HyD9vIhfkJRuSbKiYs= 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mP01sqkX0HyD9vIhfkJRuSbKiYs='} 2024-04-20T01:55:20.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6YWt0bGppLTY3L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000bbe0f69eed3d74dd-0066230cb0-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713573040.185141_826.1", 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:30:40.185141Z" 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.185141_826.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:20.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:20.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:30:40.454+0000 7fa7da69c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:30:40.455+0000 7fa7da69c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:20.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:20.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:20.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713573040.185141_826.1", 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:20.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:30:40.185141Z", 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:30:40Z" 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:20.409 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:20.410 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:20.411 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: "marker": "1_1713573040.185141_826.1", 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:30:40.185141Z", 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.412 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.413 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:30:40Z" 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573040.185141_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:30:40.185141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.185141_826.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.185141_826.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:20.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:20.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:20.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:20.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:20.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:40.579+0000 7f1ef9178780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:20.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:40.580+0000 7f1ef9178780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:20.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:20.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:20.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:20.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:20.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:20.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:20.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713573040.185141_826.1", 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T00:30:40.185141Z", 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:20.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T00:30:40Z" 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.421 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.422 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: "marker": "1_1713573040.185141_826.1", 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.423 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:30:40.185141Z", 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:20.424 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.425 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:20.425 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.425 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.425 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.425 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:20.426 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.426 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:20.426 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:20.426 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:20.426 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.426 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:30:40Z" 2024-04-20T01:55:20.426 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:20.426 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573040.185141_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:30:40.185141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.185141_826.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.185141_826.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-67', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-67 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-67 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-67/ 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-67/ 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-67/ 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests:/aktlji-67/ 2024-04-20T01:55:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:V+gO8br0FghZeQiTWWU2Mf5B5+E= 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:V+gO8br0FghZeQiTWWU2Mf5B5+E=', 'Content-Length': '0'} 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000050d707ad67179bd-0066230cb0-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-67/testobj-sse-c 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-67/testobj-sse-c 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-67/testobj-sse-c 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-algorithm:AES256 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests:/aktlji-67/testobj-sse-c 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:j3X9fr3JDMo+4Arf1Du67igZqYM= 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:j3X9fr3JDMo+4Arf1Du67igZqYM=', 'Content-Length': '0'} 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000599ea9f098d4938a-0066230cb0-1bfc-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.430 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:20.431 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.431 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:20.431 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_object_delete 2024-04-20T01:55:20.431 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:20.435 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:20.435 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:20.435 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:20.435 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 687, in test_multi_object_delete 2024-04-20T01:55:20.435 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2024-04-20T01:55:20.435 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:20.435 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:20.435 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:20.435 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:20.435 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-16 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Fdy5qym9tKkRbJ1L+Q6B5u/LOsk= 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Fdy5qym9tKkRbJ1L+Q6B5u/LOsk=', 'Content-Length': '0'} 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000006d87aa821579882b-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=aktlji-17 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:9gwSrDQDB+HVMdc52KIdsDB3uXw= 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9gwSrDQDB+HVMdc52KIdsDB3uXw=', 'Content-Length': '0'} 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009da37aeeb92c4353-006622fd94-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=aktlji-18 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cj3QUeeRUqB8FBww8AxMcALs68g= 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cj3QUeeRUqB8FBww8AxMcALs68g=', 'Content-Length': '0'} 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000001a6cdd7e58d01ed2-006622fd94-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f30de14e5c648e5-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj1 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj1 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:NAwmiyIP1jRPTqlIdlYvGm5OUck= 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NAwmiyIP1jRPTqlIdlYvGm5OUck='} 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a858f89ab0f1a607-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.445 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da2a575b3af48338-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2 2024-04-20T01:55:20.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2 2024-04-20T01:55:20.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj2 2024-04-20T01:55:20.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.447 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.447 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.447 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.447 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.447 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj2 2024-04-20T01:55:20.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:5X0whiAt2Nj1qvB71b3I8NFjpaE= 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5X0whiAt2Nj1qvB71b3I8NFjpaE='} 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d86642fae1e9a58-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.449 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.449 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.449 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.449 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.449 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.449 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021524f144042bd41-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3 2024-04-20T01:55:20.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj3 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.450 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.451 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj3 2024-04-20T01:55:20.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.451 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:PbmNrvau/Xv2tKmWI2aj1+msVQM= 2024-04-20T01:55:20.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PbmNrvau/Xv2tKmWI2aj1+msVQM='} 2024-04-20T01:55:20.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4bb612760e4b69d-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.453 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a53e547da9435b76-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4 2024-04-20T01:55:20.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4 2024-04-20T01:55:20.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj4 2024-04-20T01:55:20.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj4 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:VueLeQX6Yogg4LNN9ZjogZzZbAk= 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VueLeQX6Yogg4LNN9ZjogZzZbAk='} 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d7d4cad04770dae-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000176abfc364fcd2ae-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5 2024-04-20T01:55:20.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5 2024-04-20T01:55:20.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj5 2024-04-20T01:55:20.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.461 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.461 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.461 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.461 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.461 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj5 2024-04-20T01:55:20.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.461 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:kdzaMR2GZaOywdOSYyKYbIeFV0c= 2024-04-20T01:55:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kdzaMR2GZaOywdOSYyKYbIeFV0c='} 2024-04-20T01:55:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006959d6d0ece6bfb8-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e4e5a317ace199a-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6 2024-04-20T01:55:20.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj6 2024-04-20T01:55:20.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.464 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.465 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.465 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.465 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.465 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj6 2024-04-20T01:55:20.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.465 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:r5WpE3zObe4vLVY4E/Uu9e800wA= 2024-04-20T01:55:20.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r5WpE3zObe4vLVY4E/Uu9e800wA='} 2024-04-20T01:55:20.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000160a111885a8d2d6-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.467 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.467 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.467 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.467 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.467 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9df2f260fa7728a-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj7 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.469 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.469 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.469 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.469 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.469 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj7 2024-04-20T01:55:20.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.469 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:OWwN2FtAwf9NTCQleVyYeeKe0C4= 2024-04-20T01:55:20.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OWwN2FtAwf9NTCQleVyYeeKe0C4='} 2024-04-20T01:55:20.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001bc0ef2bc02ad06-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.470 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.470 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.471 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.471 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.471 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.471 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3e68fe6fc4c9a16-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj8 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.472 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj8 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:NU+IJJz4lB0TaqDW2RcGOscElVc= 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NU+IJJz4lB0TaqDW2RcGOscElVc='} 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f97a7767c89d6df-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.474 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.474 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.474 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.474 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.474 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.474 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aba5fd520e9eecf5-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9 2024-04-20T01:55:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9 2024-04-20T01:55:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj9 2024-04-20T01:55:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.476 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.476 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.477 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.477 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.477 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj9 2024-04-20T01:55:20.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.477 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cKE5TKoykuRyOp49vUq6kllG168= 2024-04-20T01:55:20.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cKE5TKoykuRyOp49vUq6kllG168='} 2024-04-20T01:55:20.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009cc4080120c58147-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.478 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.478 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.478 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8f2edf84602c4d3-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj10 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.480 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.480 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.480 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.480 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.480 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj10 2024-04-20T01:55:20.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.480 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sk1lYCzJ5Vx3tum7LNsGxRbHWXM= 2024-04-20T01:55:20.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sk1lYCzJ5Vx3tum7LNsGxRbHWXM='} 2024-04-20T01:55:20.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003fed8bd8893b3c99-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.482 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.482 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.482 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.482 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.482 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.482 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d88e5de0f334b4a8-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj11 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.483 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj11 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:0qZneRnU/sdUjrHty05q3q6Oz4Q= 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0qZneRnU/sdUjrHty05q3q6Oz4Q='} 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0724c873e36dbf8-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5267516ab0bb6b7-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12 2024-04-20T01:55:20.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12 2024-04-20T01:55:20.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj12 2024-04-20T01:55:20.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.487 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.487 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.488 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.488 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.488 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj12 2024-04-20T01:55:20.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.488 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BcAItnDa+7ddczxARgfn0Jd6oUQ= 2024-04-20T01:55:20.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BcAItnDa+7ddczxARgfn0Jd6oUQ='} 2024-04-20T01:55:20.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000685fa46c9ccac02a-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.489 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b19d6f26e778d3a-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj13 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj13 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Y7i9o6zNThPJosAMQDYirb2x2lk= 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y7i9o6zNThPJosAMQDYirb2x2lk='} 2024-04-20T01:55:20.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d56325476abfc889-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037195bd767548e9b-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14 2024-04-20T01:55:20.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj14 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.494 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj14 2024-04-20T01:55:20.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.495 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hOkk8hfHTnsf9bS6cv/5o2E8uLw= 2024-04-20T01:55:20.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hOkk8hfHTnsf9bS6cv/5o2E8uLw='} 2024-04-20T01:55:20.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed5f7e29fb2dbbc2-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.496 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.496 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.496 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.496 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.496 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.497 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY= 2024-04-20T01:55:20.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HrrpCrTpNLAuAhkbN4LA73OaNBY=', 'Content-Length': '0'} 2024-04-20T01:55:20.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d291b4156d4b495-006622fd94-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15 2024-04-20T01:55:20.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15 2024-04-20T01:55:20.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj15 2024-04-20T01:55:20.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.499 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.499 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.500 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.500 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:12 GMT 2024-04-20T01:55:20.500 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj15 2024-04-20T01:55:20.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.500 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:c4mM2EpNBY8eUf7WlYEjbBpmpZY= 2024-04-20T01:55:20.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:12 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c4mM2EpNBY8eUf7WlYEjbBpmpZY='} 2024-04-20T01:55:20.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000472b10cd9c0d537f-006622fd94-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1a79c035f4c4b0d-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj16 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.504 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj16 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/8B32ntaHTU2iIo43FoIaxh9E9E= 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/8B32ntaHTU2iIo43FoIaxh9E9E='} 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f487a4fa98a5acde-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a532d964c60a4a9-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17 2024-04-20T01:55:20.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj17 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.507 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj17 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EgsoaJrxor7zMQm7ternFnTS+Pk= 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EgsoaJrxor7zMQm7ternFnTS+Pk='} 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b29d9a5fc7908427-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4c0c8952661a4c9-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18 2024-04-20T01:55:20.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj18 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj18 2024-04-20T01:55:20.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:E5YFg1R00gImpRK8V9FNaH23Lxg= 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E5YFg1R00gImpRK8V9FNaH23Lxg='} 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067447fa4e72006dd-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a13cb628acc613c8-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj19 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj19 2024-04-20T01:55:20.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:vZKsyGeWQ1xQK1ziv2+XTVKSM1s= 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vZKsyGeWQ1xQK1ziv2+XTVKSM1s='} 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f87bb5de21d73ed5-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054c5b0ccaa9f1e29-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj20 2024-04-20T01:55:20.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj20 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.516 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:7E8VrFDMIsOAzrh16DI1KMjezEA= 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7E8VrFDMIsOAzrh16DI1KMjezEA='} 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8b240dbbaba704c-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.517 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000003f0146cfddb3c7-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj21 2024-04-20T01:55:20.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj21 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8Dj4SFSOyA+NqAsraGlyuSM9RGk= 2024-04-20T01:55:20.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8Dj4SFSOyA+NqAsraGlyuSM9RGk='} 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055271d97a174723b-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.520 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cd9459e8570a7be-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj22 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj22 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:9JOTjg1pN06skhW2e+Ev1iiBIxI= 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9JOTjg1pN06skhW2e+Ev1iiBIxI='} 2024-04-20T01:55:20.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ee85cd8d6b05044-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.523 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a5831ec1f0b8512-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj23 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj23 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:B3wzCtoPYlB3dgaQ1AhMdXSq+/4= 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B3wzCtoPYlB3dgaQ1AhMdXSq+/4='} 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f33ae953747bbd7-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.526 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2adf18b28e5be3d-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj24 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj24 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4gv7O0ENkNwPHyF2ITHq+Jz4k+A= 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4gv7O0ENkNwPHyF2ITHq+Jz4k+A='} 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2cd3bdb34f7648b-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.529 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d73a439f877597dc-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj25 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.531 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.531 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.531 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.531 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.531 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj25 2024-04-20T01:55:20.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.531 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:pUzcHYaE78TqEdkrgZyVPUcf7DM= 2024-04-20T01:55:20.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pUzcHYaE78TqEdkrgZyVPUcf7DM='} 2024-04-20T01:55:20.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000543e2c252a910eb3-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbf8f5afbbb3e978-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj26 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj26 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.536 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:T4zvxCTU4b06pNJEeQMQXXm1v5w= 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T4zvxCTU4b06pNJEeQMQXXm1v5w='} 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ebc8c44de4a494f-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.537 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cdd7e74afbccc14-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj27 2024-04-20T01:55:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj27 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GFLKbOWrtSMf+9xoN8H90r8prE0= 2024-04-20T01:55:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GFLKbOWrtSMf+9xoN8H90r8prE0='} 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbc7abba149b3347-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.540 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e77690ae9476c44-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj28 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj28 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:E6j3X480HgIyW8Cwr/OC8i3euYQ= 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E6j3X480HgIyW8Cwr/OC8i3euYQ='} 2024-04-20T01:55:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6cf322bde94d404-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.543 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009989e3c5f16c9244-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj29 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj29 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:puXjnMzbFORwHJpYcf2epxskhF0= 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:puXjnMzbFORwHJpYcf2epxskhF0='} 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003a06a3d22bc3200-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.546 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024cea54ce25296a9-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj30 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj30 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hG51rjXT+J49UXS6KXCGiOd/zsk= 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hG51rjXT+J49UXS6KXCGiOd/zsk='} 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bd38f356d707637-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e2631c622851295-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj31 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj31 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:g2mBpl78cWx5oAD4rA40UNA0CI0= 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g2mBpl78cWx5oAD4rA40UNA0CI0='} 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000741b74009c4d88fb-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3d19efdbef7bba7-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj32 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.553 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj32 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:aHg+nE0cV+0fi+uhGh8SDdUcI50= 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aHg+nE0cV+0fi+uhGh8SDdUcI50='} 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6391c3345f2f76e-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020eeb57a2bde8857-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj33 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.556 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj33 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:pK/+sYNFIO7ZG0TiRYSbStxsj5o= 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pK/+sYNFIO7ZG0TiRYSbStxsj5o='} 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bc5253809091a61-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013c6cd2b7274e702-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj34 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.559 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj34 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1lYk4n7y419jjrDqkfRIOSTeQ14= 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1lYk4n7y419jjrDqkfRIOSTeQ14='} 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7fba3537354c123-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000456814b552afe91b-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35 2024-04-20T01:55:20.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj35 2024-04-20T01:55:20.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.562 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.562 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.562 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj35 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:vCuI5njxZ/XVn6/BOdTA000aHAE= 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vCuI5njxZ/XVn6/BOdTA000aHAE='} 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f15b18d5e20e7990-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.564 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.564 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.564 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.564 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.564 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5dfc7ff960a5619-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj36 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.566 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.566 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.566 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.566 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.566 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj36 2024-04-20T01:55:20.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.566 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:FjHT0EfwQGD8ZvM9RqdiRnoNhdY= 2024-04-20T01:55:20.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FjHT0EfwQGD8ZvM9RqdiRnoNhdY='} 2024-04-20T01:55:20.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a094e3669b83443-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.567 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.567 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.567 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.567 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000604b8886fcb3f564-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj37 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.569 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.569 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.569 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.569 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.569 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj37 2024-04-20T01:55:20.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.569 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:PeSpm2Bm89aj3vLyT2UpcwbD6KI= 2024-04-20T01:55:20.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PeSpm2Bm89aj3vLyT2UpcwbD6KI='} 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f1c70b4cbe5f75a-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.570 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084f9756c9734cacd-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj38 2024-04-20T01:55:20.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj38 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GruXFFJDPDJ+oeJBk9zxFCVgpfo= 2024-04-20T01:55:20.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GruXFFJDPDJ+oeJBk9zxFCVgpfo='} 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007bd9eb69d07711ec-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.573 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000014a8c7e85719b0a-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj39 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj39 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6ReF5i75TFtw6femqMFS5xyO/eA= 2024-04-20T01:55:20.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ReF5i75TFtw6femqMFS5xyO/eA='} 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000393eb47622798bc5-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.576 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d30aedb198cdf2c5-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj40 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj40 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EErmUrS+SsBhJLn0Ud3ML7QVN1I= 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EErmUrS+SsBhJLn0Ud3ML7QVN1I='} 2024-04-20T01:55:20.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a794e489a221409-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.579 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005be27f2376894f9b-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj41 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj41 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZYXEkoxpPvNuytXIxz4g8st1vVc= 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZYXEkoxpPvNuytXIxz4g8st1vVc='} 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e57436f54bbaff50-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.582 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c589e43ca9b4cda-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj42 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj42 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GZT7aZjfC2dQPh42jq8J581UQgQ= 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GZT7aZjfC2dQPh42jq8J581UQgQ='} 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000918c4196581d59d7-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.585 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a99ab4653130000-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj43 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj43 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:X1M7/rRWV1Tld4d1TbM34t4k02A= 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X1M7/rRWV1Tld4d1TbM34t4k02A='} 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a57844bc600332c-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d25b684d94d830d-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj44 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.589 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj44 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2r6QIkuFHIylAFubQre5IPPXMiM= 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2r6QIkuFHIylAFubQre5IPPXMiM='} 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cf2d3a0207cad3d-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.591 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.591 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.591 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.591 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.591 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.591 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b21ab38925acbe7-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj45 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj45 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:S+zu7L6RpN4hEO/bXkFxlSvTBDg= 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:S+zu7L6RpN4hEO/bXkFxlSvTBDg='} 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000371f8584106fdffe-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a53feba8fa14508b-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj46 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.595 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj46 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BNQJ/JOXwY/YzXvd31cB/SSIvGo= 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BNQJ/JOXwY/YzXvd31cB/SSIvGo='} 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cfda88f8e957d8c-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042117a310a95af87-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47 2024-04-20T01:55:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47 2024-04-20T01:55:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj47 2024-04-20T01:55:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.598 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.598 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj47 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:MpCznzp2+XNZ/6nCgw251J5hoqE= 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MpCznzp2+XNZ/6nCgw251J5hoqE='} 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001594a264cc10f7fb-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004225f081890b1461-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj48 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.601 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj48 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Yo9KrHU+1F0rujx+TgpkQbiG3SE= 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Yo9KrHU+1F0rujx+TgpkQbiG3SE='} 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b86a8702d1e45a38-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8= 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dc+TjWahBL54ioUrDD7x4fK/jZ8=', 'Content-Length': '0'} 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000671b8f0ee82db957-006622fd95-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj49 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.604 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj49 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:F8+QA60Y3MVYAIsGXO6RGqsO2DY= 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:F8+QA60Y3MVYAIsGXO6RGqsO2DY='} 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b370c078912b4d3-006622fd95-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3DE1rSFZ03+Qf5qd9bhyfT7oJGc= 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3DE1rSFZ03+Qf5qd9bhyfT7oJGc=', 'Content-Length': '0'} 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a2c7641715e61fd-006622fd95-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj1 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj1 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj1 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj1 2024-04-20T01:55:20.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:O5Q/PYjortp6nyeGsLV0fIfQsxQ= 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O5Q/PYjortp6nyeGsLV0fIfQsxQ='} 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5a7a6c64cd09066-006622fd95-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3DE1rSFZ03+Qf5qd9bhyfT7oJGc= 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3DE1rSFZ03+Qf5qd9bhyfT7oJGc=', 'Content-Length': '0'} 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032c463672ba32824-006622fd95-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj2 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj2 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj2 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:13 GMT 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj2 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.610 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tOo1ew6zRSuquO6wf46RXSfpMcw= 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tOo1ew6zRSuquO6wf46RXSfpMcw='} 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b77e07a2886a545-006622fd95-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.611 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a52d588373699721-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj3 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj3 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj3 2024-04-20T01:55:20.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj3 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hsbr4AqCH6ISSBsUs4mdddIQ5vU= 2024-04-20T01:55:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hsbr4AqCH6ISSBsUs4mdddIQ5vU='} 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093159f5d94cf4bf0-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.614 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a48d3e9fd7e26e94-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj4 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj4 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj4 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj4 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:S9NMizasssifnE8teLsG//49xBM= 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:S9NMizasssifnE8teLsG//49xBM='} 2024-04-20T01:55:20.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b43e643207784d1-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.617 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000802a67d6bf671682-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj5 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj5 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj5 2024-04-20T01:55:20.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj5 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.619 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:E/WVdpfRhlP+4+HgfRVUJzhkZmY= 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E/WVdpfRhlP+4+HgfRVUJzhkZmY='} 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d2917dc35177d45-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.620 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000320723f01d78811-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj6 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj6 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj6 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj6 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:UIhaC3B/6P2ls+qXb9VTeRY+Fy8= 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UIhaC3B/6P2ls+qXb9VTeRY+Fy8='} 2024-04-20T01:55:20.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b16160e21ae3650-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.623 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a693bef5416f49df-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj7 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj7 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj7 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj7 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:lJg9LClQiT1yhdsh9QI8o64GcNM= 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lJg9LClQiT1yhdsh9QI8o64GcNM='} 2024-04-20T01:55:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000566433853f55944d-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.626 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfef5db118260df0-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj8 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj8 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj8 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj8 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:IIRMRdxDGfRJh8mT4Vdbp+AU+g0= 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IIRMRdxDGfRJh8mT4Vdbp+AU+g0='} 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bab453503ec51378-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.629 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000722f483e33a45949-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj9 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj9 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj9 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj9 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:SOD6aCLakjilIMrSrzIyhgEvKm8= 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SOD6aCLakjilIMrSrzIyhgEvKm8='} 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044bf0c11a267beab-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4ffce0cd8654fab-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj10 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj10 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj10 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj10 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BYcUSPoBtWx5xd1q8/JHMdw+P+E= 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BYcUSPoBtWx5xd1q8/JHMdw+P+E='} 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f70781643f6edf6-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.635 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2a80d1643496283-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj11 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj11 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj11 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.636 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj11 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:JHS7wCQ52H1G94E4Ste7i17b7L4= 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JHS7wCQ52H1G94E4Ste7i17b7L4='} 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000049bbcd086a38c25-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000443f6fd93093ac53-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj12 2024-04-20T01:55:20.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj12 2024-04-20T01:55:20.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj12 2024-04-20T01:55:20.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj12 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZBhgwfk1CnflJA3LZrsiLQ++V5w= 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZBhgwfk1CnflJA3LZrsiLQ++V5w='} 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022aa2cd49a2259c7-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.644 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca9233845ea31213-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj13 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj13 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj13 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj13 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zr1LNGipUDnLdRrhdcsWU6YvBHY= 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zr1LNGipUDnLdRrhdcsWU6YvBHY='} 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a797da1da684de1f-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.647 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d9d9d703720740c-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj14 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj14 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj14 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.648 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj14 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2NZPmfyA0aCTTL+G+LP7S/lG5bE= 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2NZPmfyA0aCTTL+G+LP7S/lG5bE='} 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd16188396f1c35d-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a182f76eb47f9d0-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj15 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj15 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj15 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.651 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj15 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:RX1k3mpY57FOhLwM9R82OF8cBa0= 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RX1k3mpY57FOhLwM9R82OF8cBa0='} 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d8173f285c26493-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c68a92fccd490ea0-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj16 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj16 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj16 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.654 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj16 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:jA5MEK6WfPRYstoXiH2EcbE2gu0= 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jA5MEK6WfPRYstoXiH2EcbE2gu0='} 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea70a1d467624789-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1b56e08f5c3d132-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj17 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj17 2024-04-20T01:55:20.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj17 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.657 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj17 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:m56esVs2YIioovLUDaWwmSFBJiM= 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m56esVs2YIioovLUDaWwmSFBJiM='} 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000777f77ce45291002-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001aa57846606134dc-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj18 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj18 2024-04-20T01:55:20.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj18 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj18 2024-04-20T01:55:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:K7Mi6rl/M4NhWhkdhiiDWZT9A2w= 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K7Mi6rl/M4NhWhkdhiiDWZT9A2w='} 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043a2f93172a1a8c8-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.662 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.662 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.662 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.662 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.662 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.662 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bc8e784d68c5998-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj19 2024-04-20T01:55:20.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj19 2024-04-20T01:55:20.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj19 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj19 2024-04-20T01:55:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AwqqdMqK9sEO9kCsNgj1KLsaug8= 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AwqqdMqK9sEO9kCsNgj1KLsaug8='} 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0581b798f8ae649-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015490f2889e072b0-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj20 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj20 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj20 2024-04-20T01:55:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj20 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.666 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:MxXmhkLI9WD0ngZoiqdqXoVpIJo= 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MxXmhkLI9WD0ngZoiqdqXoVpIJo='} 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ef7a0eb2aed67fd-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000898537f0ec24d744-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj21 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj21 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj21 2024-04-20T01:55:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj21 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.669 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:G27s6TOna8BbxiEDWeas2GXF7gc= 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:G27s6TOna8BbxiEDWeas2GXF7gc='} 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cc3c4fa5e64753f-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.670 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa23b30c72f3e2d2-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj22 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj22 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj22 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj22 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:uXM8ijlmJHaM2e9ECdRCiMzTYQo= 2024-04-20T01:55:20.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uXM8ijlmJHaM2e9ECdRCiMzTYQo='} 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa3e92a6db25ac18-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.673 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f9f1cbe0fb5333f-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj23 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj23 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj23 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj23 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/9JBsknAhtptiIHRR87M7//0uc4= 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/9JBsknAhtptiIHRR87M7//0uc4='} 2024-04-20T01:55:20.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d96137e395d7931-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.676 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004be0ea7206ff7c9-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj24 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj24 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj24 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj24 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qF+cXbeQfRwxp2rDFtmu6tnKIOk= 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qF+cXbeQfRwxp2rDFtmu6tnKIOk='} 2024-04-20T01:55:20.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000363c59e60f736969-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.679 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000494262e4a5ccd54a-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj25 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj25 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj25 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj25 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:UPZxTzE0qveDKb29vcE0m57i6BM= 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UPZxTzE0qveDKb29vcE0m57i6BM='} 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000396d51068a107d94-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.682 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001580667f6b91370f-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj26 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj26 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj26 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj26 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Oo223t4cBZw6tFw4tLYUbKzXiF8= 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Oo223t4cBZw6tFw4tLYUbKzXiF8='} 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000503b2a1d77a89deb-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.685 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061512adaaa65deb9-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj27 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj27 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj27 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj27 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6wJR3sfJ2qH5gtrk5TiOXCluQfE= 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6wJR3sfJ2qH5gtrk5TiOXCluQfE='} 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b8e099b25fbc48d-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a60767afc8493da-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj28 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj28 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj28 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj28 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:s09QlKKeU8SGZvjEr5PNGJpmhJk= 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:s09QlKKeU8SGZvjEr5PNGJpmhJk='} 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001032eb7b3d3854d1-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002760a040d3efffd4-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj29 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj29 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj29 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj29 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ImXdDxY/yoNId4EplSCSvIw+eqM= 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ImXdDxY/yoNId4EplSCSvIw+eqM='} 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001220b7457b3dded5-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.694 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060b554843eaba598-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj30 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj30 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj30 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.695 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj30 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:MTWGFuGuxuV442YHUcpCsK+g0Cc= 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MTWGFuGuxuV442YHUcpCsK+g0Cc='} 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000dee92bd1c0d7192-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df497eb8271a805b-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj31 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj31 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj31 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.698 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj31 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BMQl0x8o0iwb6WUa7AA6rgB1BXE= 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BMQl0x8o0iwb6WUa7AA6rgB1BXE='} 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1d0437998be04e9-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c168e97897e12ef-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj32 2024-04-20T01:55:20.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj32 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj32 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.701 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj32 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:O4TCVpu+aHGELesGdbva2Mk8QL4= 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:O4TCVpu+aHGELesGdbva2Mk8QL4='} 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5a2eba7dfb2e4a8-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1680c11a643c86e-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj33 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj33 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj33 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj33 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.704 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:0R0OTKUWWU8+isymrXCIGulCOt0= 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0R0OTKUWWU8+isymrXCIGulCOt0='} 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009dc832288d8be83d-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.705 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cb214f72c357799-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj34 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj34 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj34 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj34 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dtgm+JQF68bxvGKgBif0kwmpUF0= 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dtgm+JQF68bxvGKgBif0kwmpUF0='} 2024-04-20T01:55:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2cbd0953aaa346d-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.708 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE= 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z8UFV3IScXkLOV3gykwIdBZ78jE=', 'Content-Length': '0'} 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6726fa94c3bd26c-006622fd96-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj35 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj35 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj35 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:14 GMT 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj35 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:u58gI82upxJpHbNVcivcFirnrtc= 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u58gI82upxJpHbNVcivcFirnrtc='} 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c866024420c4006-006622fd96-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.711 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c181f621709ef72-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj36 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj36 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj36 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj36 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:L10kwF74t0F1UneocU39NZDc9SU= 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L10kwF74t0F1UneocU39NZDc9SU='} 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb08ecaa41b2c7e3-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.714 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.714 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.714 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.715 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.715 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.715 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098629b3781f0ac64-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj37 2024-04-20T01:55:20.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj37 2024-04-20T01:55:20.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj37 2024-04-20T01:55:20.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj37 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+EOMmiFwukcCzlPxwGke+1M2bRY= 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+EOMmiFwukcCzlPxwGke+1M2bRY='} 2024-04-20T01:55:20.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f169253e9679968-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.717 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7da0798b6ed4d98-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj38 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj38 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj38 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj38 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:JeBN1NTFEcYom6agK3gwWNIQ9M4= 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JeBN1NTFEcYom6agK3gwWNIQ9M4='} 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009165b32c1217af66-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.720 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097048091f51a46a6-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj39 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj39 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj39 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj39 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HyMYVoqDJbfi25ca5zqcpgumrCk= 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HyMYVoqDJbfi25ca5zqcpgumrCk='} 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfd7eb2f31c6c053-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.723 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001573993fdad66427-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj40 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj40 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj40 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj40 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:uoT0qpq4/JrCeX1QYSnp9O/WjXo= 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uoT0qpq4/JrCeX1QYSnp9O/WjXo='} 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbfdafa56839e87b-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008ca024969909069-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj41 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj41 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj41 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj41 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:shEuJ1l9pdfkqDWo/7R7JGkr8Fg= 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:shEuJ1l9pdfkqDWo/7R7JGkr8Fg='} 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1e0f704e9726ea0-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddd84036c96228f1-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj42 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj42 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj42 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.730 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj42 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:jM1ZO3iNjFwhZUV6lEWpHZTiyaM= 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jM1ZO3iNjFwhZUV6lEWpHZTiyaM='} 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6a8866c0983ee5c-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c2563b6a4d68af4-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj43 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj43 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj43 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.733 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj43 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hi3EG6MKOAJNkVsRtha3J58976I= 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hi3EG6MKOAJNkVsRtha3J58976I='} 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040fc93da88acd5ea-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000483c030b96ea5cc-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj44 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj44 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj44 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.736 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj44 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/hE6Jc9JM5ic02eFaulJ5T3965A= 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/hE6Jc9JM5ic02eFaulJ5T3965A='} 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044aa7c2516d01eb9-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000431bff69b9cc1c6d-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj45 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj45 2024-04-20T01:55:20.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj45 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj45 2024-04-20T01:55:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:YCviTTsyhptBvtExr2bhUYeM7Ww= 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YCviTTsyhptBvtExr2bhUYeM7Ww='} 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4c7944fe2999c57-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b308badc38279e12-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj46 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj46 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj46 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj46 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.742 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:No711zyjEkOkleO/h+12gLFN3LM= 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:No711zyjEkOkleO/h+12gLFN3LM='} 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011dae0c8aa4be76b-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.743 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ae0fa1085007024-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj47 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj47 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj47 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj47 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zpmKlmIl2Aj8AwT3uLjn74Kmbo4= 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zpmKlmIl2Aj8AwT3uLjn74Kmbo4='} 2024-04-20T01:55:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000736e407748dd87b0-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.746 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0538a4317e9f89a-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj48 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj48 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj48 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj48 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:lhRoUD1YXh17gVHzNA0XQfjgfa0= 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lhRoUD1YXh17gVHzNA0XQfjgfa0='} 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddec167df5c46e8b-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/ 2024-04-20T01:55:20.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/ 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/ 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.749 INFO:tasks.rgw_multisite_tests:/aktlji-17/ 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo= 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3Hc4Qs7tyyaMvSftYj/VbuAJNKo=', 'Content-Length': '0'} 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000682a5b11c8d9faa5-006622fd97-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-17/obj49 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-17/obj49 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-17/obj49 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:/aktlji-17/obj49 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Xye4pUdRDWE96O9I5Xpj7UHTXYk= 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Xye4pUdRDWE96O9I5Xpj7UHTXYk='} 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c8c6f701da13424-006622fd97-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6a03bb368e1789d-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj1 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj1 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj1 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj1 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EzzlANHvoz2SC+yNgbOmXAByzI8= 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EzzlANHvoz2SC+yNgbOmXAByzI8='} 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024c946e42175d2ef-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.755 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000916d1771dc8a7daa-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj2 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj2 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj2 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.756 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj2 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1tJXKrqMtGA51lp8lBUkDd4CVg8= 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1tJXKrqMtGA51lp8lBUkDd4CVg8='} 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009fbb54fb08a5b296-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001db6983606ac01bc-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj3 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj3 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj3 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.759 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj3 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Tp77ofTsLFsUf1IO9qbNyple3os= 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Tp77ofTsLFsUf1IO9qbNyple3os='} 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000244f00d46c4361d1-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012e8a9c906234425-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj4 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj4 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj4 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.762 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj4 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:jUNcbEu4rkpKAc/jLPhUZ3gNP1A= 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jUNcbEu4rkpKAc/jLPhUZ3gNP1A='} 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a165b4b832e317ed-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085d9b48c1854294e-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj5 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj5 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj5 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.765 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj5 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:A4fpJURRtZFKPPoTmdsAt3LFNko= 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:A4fpJURRtZFKPPoTmdsAt3LFNko='} 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009dbbeccec282ce2c-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073dc70944b564783-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj6 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj6 2024-04-20T01:55:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj6 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj6 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.768 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:FThfJx43ljM1ZvpP5H8u0cK2GVM= 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FThfJx43ljM1ZvpP5H8u0cK2GVM='} 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6d2ffac51d6fcad-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.769 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bc7d628ea5b930e-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj7 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj7 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj7 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj7 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:JMqLBDZikKsrD/hYZZwkWz4KkYE= 2024-04-20T01:55:20.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JMqLBDZikKsrD/hYZZwkWz4KkYE='} 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000350c54daa4fd105e-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.772 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed148de40c907539-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj8 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj8 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj8 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj8 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:oADasiTYZJ9X7W3dMhgMsYEuQ4g= 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oADasiTYZJ9X7W3dMhgMsYEuQ4g='} 2024-04-20T01:55:20.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c63620f1fe84b7c1-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.775 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d90b3049770b504-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj9 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj9 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj9 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj9 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6awl+O73y5Yr98TPwCca8WbYcdI= 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6awl+O73y5Yr98TPwCca8WbYcdI='} 2024-04-20T01:55:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000403e421dedd57d84-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.778 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008488b1e6c28eb39d-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj10 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj10 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj10 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj10 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:euXsdLrcYoY3ARZi7cgcJwSt/Og= 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:euXsdLrcYoY3ARZi7cgcJwSt/Og='} 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5e05cfb8272eea8-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.781 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002de9aa21579b0493-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj11 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj11 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj11 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj11 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZBqvZcswP2k5E3ahdcLAAnKTD00= 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZBqvZcswP2k5E3ahdcLAAnKTD00='} 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bec5ca025b95699-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.784 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000308adb801676a816-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj12 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj12 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj12 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj12 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:w7r3j2PfjkkVeIOx+7VXTh0ZIn8= 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:w7r3j2PfjkkVeIOx+7VXTh0ZIn8='} 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068deea84e2738d42-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000128c001b80866f1f-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj13 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj13 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj13 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj13 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qVITO1YIoMtQCoNPAoyiMqgYlqE= 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qVITO1YIoMtQCoNPAoyiMqgYlqE='} 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7918ef00c3bf173-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005a0459f23f4c461-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj14 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj14 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj14 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.791 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj14 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HpcItr8ChjJ1dJGrPRJInacS9uk= 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HpcItr8ChjJ1dJGrPRJInacS9uk='} 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001370d57edbde094d-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e3739fc3ea3bb9a-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj15 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj15 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj15 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.794 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj15 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8PvfRoHPMIrGgl86hdRWetJvelo= 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8PvfRoHPMIrGgl86hdRWetJvelo='} 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061dd27e5755963d1-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cc628b6981e7d99-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj16 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj16 2024-04-20T01:55:20.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj16 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj16 2024-04-20T01:55:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:wAEKlbFndGLTvOpYubKvMzs+JI0= 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wAEKlbFndGLTvOpYubKvMzs+JI0='} 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033c238a859235298-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000203bdbbb72bed969-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj17 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj17 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj17 2024-04-20T01:55:20.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj17 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.800 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LWlyW0s2zQNEXEKKxUG+3ahKiqg= 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LWlyW0s2zQNEXEKKxUG+3ahKiqg='} 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a5b35344a99a88c-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.801 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000104c81ec5c99e63b-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj18 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj18 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj18 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj18 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:g5peiNdug1v2IRCtznBQ8M9DJN8= 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:g5peiNdug1v2IRCtznBQ8M9DJN8='} 2024-04-20T01:55:20.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d65b07650b1c3e8e-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.804 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000816323bca1f41a3f-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj19 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj19 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj19 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj19 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1M28GfQPOWc3lOqSLnrSvE90KDs= 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1M28GfQPOWc3lOqSLnrSvE90KDs='} 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebfe121d3ab9d795-006622fd97-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.807 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ= 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XfetPq/9sJ58s27PakwzyARnrKQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f8173cd99aea7fc-006622fd97-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj20 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj20 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj20 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:15 GMT 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj20 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:JwWq06MUogFrjiRbz9yQXxa3vac= 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JwWq06MUogFrjiRbz9yQXxa3vac='} 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6f24a551e166889-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.810 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003190502b2cbff0c-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj21 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj21 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj21 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj21 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:OyKDZ9JglhWGjzflPEp19RCh6aM= 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OyKDZ9JglhWGjzflPEp19RCh6aM='} 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000138cd8fcb9c085d6-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a929d48b4e03c32-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj22 2024-04-20T01:55:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj22 2024-04-20T01:55:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj22 2024-04-20T01:55:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj22 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1xBUus32uTqE0kg55fflNSxJ7MA= 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1xBUus32uTqE0kg55fflNSxJ7MA='} 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4ed40e414cd2adc-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096f0550a9cff5ae6-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj23 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj23 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj23 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.817 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj23 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:H2444GM3GGY6WZOym+AbzNaU0Dk= 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:H2444GM3GGY6WZOym+AbzNaU0Dk='} 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007449a4514c60f395-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4780b381874b315-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj24 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj24 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj24 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.820 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj24 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Y7NIh3JcVSuibJBe90xfRjK+dqY= 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y7NIh3JcVSuibJBe90xfRjK+dqY='} 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000345a73f36b7bd7e2-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085701fae72a1c85b-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj25 2024-04-20T01:55:20.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj25 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj25 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.823 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj25 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+45YH47SHGoDwBuejIkiuFQoiy8= 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+45YH47SHGoDwBuejIkiuFQoiy8='} 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f098d6f3c250d58-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088ffc4367b58f352-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj26 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj26 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj26 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj26 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.826 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EKuElpgQkhNRMqPrP22UMOncDdk= 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EKuElpgQkhNRMqPrP22UMOncDdk='} 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000281e1b0700891178-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.827 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fed36d57da16c33-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj27 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj27 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj27 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj27 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Li0/Aj494HJcekIeuPYoGu9//7g= 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Li0/Aj494HJcekIeuPYoGu9//7g='} 2024-04-20T01:55:20.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b97e92a963a5ecf4-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.830 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000032a9ca5b87a3d3c-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj28 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj28 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj28 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj28 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nTFt8K/1tRyydQEQVDe+p0Ssd48= 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nTFt8K/1tRyydQEQVDe+p0Ssd48='} 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002271ffc24ad76b3-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.833 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088ac754b41e1a069-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj29 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj29 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj29 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj29 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:DR4Cmpk2ptSNd2NcpEN5BVbUvN8= 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DR4Cmpk2ptSNd2NcpEN5BVbUvN8='} 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009af3b3f3ff8a4b03-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.836 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000904fbc638475bed4-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj30 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj30 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj30 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj30 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Z9toIhsYZKvkjMRJ5rHClXbxVgo= 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z9toIhsYZKvkjMRJ5rHClXbxVgo='} 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000354e6eca8b5e6002-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000456f15e36a510c14-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj31 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj31 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj31 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj31 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:X0G6b6m9gyCFzeHZJEGNatlBZaM= 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X0G6b6m9gyCFzeHZJEGNatlBZaM='} 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f63c42bd839611fc-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000269e9e0bd4ba3cbb-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj32 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj32 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj32 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.843 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj32 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4IBA+u+/NjegAOcGdq8SIjvlxXM= 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4IBA+u+/NjegAOcGdq8SIjvlxXM='} 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d84cb34e45f06e8c-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f548d2ab2afec58-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj33 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj33 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj33 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.846 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj33 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XEMUm7QZO2uBXr4djCyJ5h7skpE= 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XEMUm7QZO2uBXr4djCyJ5h7skpE='} 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b56c6730ea3fcff-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d93ad28164bc8d3e-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj34 2024-04-20T01:55:20.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj34 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj34 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.849 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj34 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:pX1Glq8+bo1dsQn5rQcJZSKziLU= 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pX1Glq8+bo1dsQn5rQcJZSKziLU='} 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6228e22d4bccdb9-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005166b65817011787-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj35 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj35 2024-04-20T01:55:20.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj35 2024-04-20T01:55:20.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.853 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.853 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.853 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.853 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.853 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj35 2024-04-20T01:55:20.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BY4/TSaFrQ6HJ7gqZOqSB2IgxZY= 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BY4/TSaFrQ6HJ7gqZOqSB2IgxZY='} 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa362e74ed72089a-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f1ee0b0abe89c45-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj36 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj36 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj36 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj36 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.856 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AAHQb8bShg3zaFM6Gc0ENJ3/xfQ= 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AAHQb8bShg3zaFM6Gc0ENJ3/xfQ='} 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a3b12320f63977a-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.857 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000135c1675381cf3f9-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj37 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj37 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj37 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj37 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:UcEWM5pZG7u/RgtB11nnkV5KAqw= 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UcEWM5pZG7u/RgtB11nnkV5KAqw='} 2024-04-20T01:55:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc1af73bac530013-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.860 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b993e51052fa73d0-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj38 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj38 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj38 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj38 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:DqDMJG3WA6GkEYMkjrqMdKpT3p0= 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DqDMJG3WA6GkEYMkjrqMdKpT3p0='} 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec0a43c07ba6db65-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1a65fab7d103414-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj39 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj39 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj39 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj39 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zis0z0lIF6OPsMtQ/bCytnv2lAo= 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zis0z0lIF6OPsMtQ/bCytnv2lAo='} 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008175277f5f15b80a-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.866 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7945cbf7c5c9df1-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj40 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj40 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj40 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.867 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj40 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EVfg1M6XqYXoN8I+BpPOFqknAYE= 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EVfg1M6XqYXoN8I+BpPOFqknAYE='} 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092c1a6540419725b-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfcff0b01a495a95-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj41 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj41 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj41 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.870 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj41 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eKP0bC+NQWrGEtlJClIS3S4plAU= 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eKP0bC+NQWrGEtlJClIS3S4plAU='} 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ac685037373ecaf-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ce6f526ea362128-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj42 2024-04-20T01:55:20.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj42 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj42 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.873 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj42 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HvY49/quLj+pVeQCGrkf915AEvA= 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HvY49/quLj+pVeQCGrkf915AEvA='} 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a69aac434ee470b-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd788d467526b96a-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj43 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj43 2024-04-20T01:55:20.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj43 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj43 2024-04-20T01:55:20.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:7nS8rDOP5SbZHpaAH8NolVg5VTo= 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7nS8rDOP5SbZHpaAH8NolVg5VTo='} 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff3a49931a839712-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e96050e0bf3e03e9-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj44 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj44 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj44 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj44 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.879 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:THZJlQMekW1D6l4w4+Od2APXGG4= 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:THZJlQMekW1D6l4w4+Od2APXGG4='} 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b63ab855b95012f8-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.880 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecd97848bf6da389-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj45 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj45 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj45 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj45 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:X8gSmTp9mB1FPYsJZOoV8SxIgnQ= 2024-04-20T01:55:20.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:X8gSmTp9mB1FPYsJZOoV8SxIgnQ='} 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000abec812fb87b63c4-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.883 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035d4f18f072abad9-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj46 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj46 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj46 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj46 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4vyEPJ2KtqEtxDcS15BZP5QudDw= 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4vyEPJ2KtqEtxDcS15BZP5QudDw='} 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083e63804ef38d1a2-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.886 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c4bba95ae607919-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj47 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj47 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj47 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj47 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LI9gLUYIS0i2YJNdJkEj40e5w0s= 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LI9gLUYIS0i2YJNdJkEj40e5w0s='} 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c79cb796e5552a9-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9ba6975b3dcfbdf-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj48 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj48 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj48 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj48 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ach+lJTqbzOUNRNg14qUIhZXdtY= 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ach+lJTqbzOUNRNg14qUIhZXdtY='} 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a77f7622a44e8678-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/ 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/ 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/ 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests:/aktlji-18/ 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.892 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08= 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZkU/4Zr6VgyobQC1fKVJoPDVQ08=', 'Content-Length': '0'} 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f70439d710d58ca8-006622fd98-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-18/obj49 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-18/obj49 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-18/obj49 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:16 GMT 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:/aktlji-18/obj49 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qpGVEyRPFneq10BlyykjPuMzpSk= 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:26:16 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qpGVEyRPFneq10BlyykjPuMzpSk='} 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8832f2cf7137b0d-006622fd98-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:26:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:20.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713569172.547279_68.1", 2024-04-20T01:55:20.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:26:12.547279Z" 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569172.547279_68.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:20.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:26:16.998+0000 7f94eea35780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:26:17.002+0000 7f94eea35780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:20.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:20.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713569172.547279_68.1", 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:12.547279Z", 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:26:17Z" 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:20.901 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.902 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.903 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: "marker": "1_1713569172.547279_68.1", 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:12.547279Z", 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.904 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.905 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:26:17Z" 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569172.547279_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:12.547279Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569172.547279_68.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569172.547279_68.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:20.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:26:17.164+0000 7f8ca0460780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:26:17.165+0000 7f8ca0460780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:20.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713569172.547279_68.1", 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:26:12.547279Z", 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:20.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:20.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-19T23:26:17Z" 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:20.912 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:20.913 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.914 INFO:tasks.rgw_multisite_tests: "marker": "1_1713569172.547279_68.1", 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:12.547279Z", 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:20.915 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:26:17Z" 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:20.916 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713569172.547279_68.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:12.547279Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569172.547279_68.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713569172.547279_68.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:26:17.297+0000 7f0bf22d2780 1 waiting for incremental sync to catch up: 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: local status: [00000000007.39.6, , 00000000002.11.6, , 00000000006.16.6, 00000000001.5.6, , 00000000004.17.6, 00000000002.18.6, , 00000000001.17.6] 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: remote markers: [00000000007.39.6, 00000000004.41.6, 00000000004.15.6, 00000000004.26.6, 00000000006.16.6, 00000000005.13.6, 00000000004.21.6, 00000000005.19.6, 00000000002.18.6, 00000000004.22.6, 00000000004.35.6] 2024-04-20T01:55:20.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:26:22.298+0000 7f0bf22d2780 1 waiting for incremental sync to catch up: 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: local status: [00000000007.39.6, , 00000000002.11.6, , 00000000006.16.6, 00000000001.5.6, , 00000000004.17.6, 00000000002.18.6, , 00000000001.17.6] 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: remote markers: [00000000007.39.6, 00000000004.41.6, 00000000004.15.6, 00000000004.26.6, 00000000006.16.6, 00000000005.13.6, 00000000004.21.6, 00000000005.19.6, 00000000002.18.6, 00000000004.22.6, 00000000004.35.6] 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:26:27.299+0000 7f0bf22d2780 1 waiting for incremental sync to catch up: 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: local status: [00000000007.39.6, , 00000000002.11.6, , 00000000006.16.6, 00000000001.5.6, , 00000000004.17.6, 00000000002.18.6, , 00000000001.17.6] 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: remote markers: [00000000007.39.6, 00000000004.41.6, 00000000004.15.6, 00000000004.26.6, 00000000006.16.6, 00000000005.13.6, 00000000004.21.6, 00000000005.19.6, 00000000002.18.6, 00000000004.22.6, 00000000004.35.6] 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:26:32.300+0000 7f0bf22d2780 1 waiting for incremental sync to catch up: 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: local status: [00000000007.39.6, , 00000000004.15.6, , 00000000006.16.6, 00000000001.5.6, 00000000004.21.6, 00000000004.17.6, 00000000002.18.6, , 00000000003.25.6] 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: remote markers: [00000000007.39.6, 00000000004.41.6, 00000000004.15.6, 00000000004.26.6, 00000000006.16.6, 00000000005.13.6, 00000000004.21.6, 00000000005.19.6, 00000000002.18.6, 00000000004.22.6, 00000000004.35.6] 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:26:37.302+0000 7f0bf22d2780 1 bucket sync caught up with source: 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: local status: [00000000007.39.6, 00000000004.41.6, 00000000004.15.6, 00000000004.26.6, 00000000006.16.6, 00000000005.13.6, 00000000004.21.6, 00000000005.19.6, 00000000002.18.6, 00000000004.22.6, 00000000004.35.6] 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: remote markers: [00000000007.39.6, 00000000004.41.6, 00000000004.15.6, 00000000004.26.6, 00000000006.16.6, 00000000005.13.6, 00000000004.21.6, 00000000005.19.6, 00000000002.18.6, 00000000004.22.6, 00000000004.35.6] 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:26:37.302+0000 7f0bf22d2780 0 bucket checkpoint complete 2024-04-20T01:55:20.918 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=aktlji-16 zones={test-zone2, test-zone1} 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.919 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:u2pNxeEWJmssPF+lFy9B9vFghh4= 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u2pNxeEWJmssPF+lFy9B9vFghh4=', 'Content-Length': '0'} 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '294'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e19773c4b608bdd5-006622fdad-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/ 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:u2pNxeEWJmssPF+lFy9B9vFghh4= 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u2pNxeEWJmssPF+lFy9B9vFghh4=', 'Content-Length': '0'} 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '294'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1644e597b1a2bb3-006622fdad-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.921 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?versions&max-keys=999 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?versions&max-keys=999 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?versions&max-keys=999 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests:/aktlji-16/?versions 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ= 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c45ea7f43351c271-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16999falseobj1nulltrue2024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj1 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj10 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-20T01:55:20.923 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj12 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj13 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj14 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj15 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj16 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj17 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj18 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj19 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj2 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj20 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj21 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj23 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj24 2024-04-20T01:55:20.924 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj25 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj26 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj27 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj28 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj29 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj3 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj30 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj31 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj32 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj34 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj35 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj36 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj37 2024-04-20T01:55:20.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj38 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj39 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj4 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj40 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj41 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj42 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj43 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj45 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj46 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj47 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj48 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj49 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj5 2024-04-20T01:55:20.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj6 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj7 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj8 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj9 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?versions&max-keys=999 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?versions&max-keys=999 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?versions&max-keys=999 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests:/aktlji-16/?versions 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ= 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c463b7d7cbeb904-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16999falseobj1nulltrue2024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj1 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj10 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj12 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj13 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj14 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj15 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj16 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj17 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj18 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj19 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj2 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj20 2024-04-20T01:55:20.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj21 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj23 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj24 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj25 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj26 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj27 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj28 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj29 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj3 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj30 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj31 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj32 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-20T01:55:20.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj34 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj35 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj36 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj37 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj38 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj39 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj4 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj40 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj41 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj42 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj43 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj45 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj46 2024-04-20T01:55:20.931 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj47 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj48 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj49 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj5 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj6 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj7 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj8 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj9 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?versions&max-keys=999 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?versions&max-keys=999 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?versions&max-keys=999 2024-04-20T01:55:20.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests:/aktlji-16/?versions 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ= 2024-04-20T01:55:20.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb147473ab8424ae-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16999falseobj1nulltrue2024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?versions&max-keys=999 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?versions&max-keys=999 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?versions&max-keys=999 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests:/aktlji-16/?versions 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ= 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8d9Ijo9RQyb6VkCcJiPWaMu7/eQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000136f8ce1b356b4ea-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16999falseobj1nulltrue2024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj1 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.936 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E= 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E=', 'Content-Length': '0'} 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eac4907f72c6c64d-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E= 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E=', 'Content-Length': '0'} 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0fd8a3cc70aa3e7-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.940 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.940 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.940 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.940 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.940 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.942 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WRHkkeevAGFZqmF8I9uvqI6rOnk= 2024-04-20T01:55:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WRHkkeevAGFZqmF8I9uvqI6rOnk=', 'Content-Length': '0'} 2024-04-20T01:55:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000028a1632e5468659-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1 2024-04-20T01:55:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1 2024-04-20T01:55:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WRHkkeevAGFZqmF8I9uvqI6rOnk= 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WRHkkeevAGFZqmF8I9uvqI6rOnk=', 'Content-Length': '0'} 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ab3eecceb369d5b-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.943 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj1 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.944 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E= 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E=', 'Content-Length': '0'} 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf9c97caa4f3e57a-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj1?versionId=null 2024-04-20T01:55:20.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E= 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a+cM0h7cXwg4XXfGorFhXRn+j2E=', 'Content-Length': '0'} 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc0b62e64ee9aa44-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj1 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj1 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj1 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed8f7312935d80a8-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj11000falseobj12024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj1 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj1 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj1 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.949 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3ddcac8bce4e2e9-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj11000falseobj12024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj1 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj1 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KCCU2H3Yrr8eZJYdTf21DQKSVfA= 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KCCU2H3Yrr8eZJYdTf21DQKSVfA=', 'Content-Length': '0'} 2024-04-20T01:55:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090d9b8d8c888d9d9-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj1 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.952 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj1 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KCCU2H3Yrr8eZJYdTf21DQKSVfA= 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KCCU2H3Yrr8eZJYdTf21DQKSVfA=', 'Content-Length': '0'} 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000536b517f07bae7df-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj1 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj1 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj1 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Q/AU8UO+g40/05l5K/wr3OZfmEg= 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q/AU8UO+g40/05l5K/wr3OZfmEg=', 'Content-Length': '0'} 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a18f620c12a12e60-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj1 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj1 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj1 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.955 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj1 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Q/AU8UO+g40/05l5K/wr3OZfmEg= 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q/AU8UO+g40/05l5K/wr3OZfmEg=', 'Content-Length': '0'} 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083552fbc0e0f5299-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj1 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj1 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj1 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:20.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028da5ab90b8b91ad-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj11000falseobj12024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj1 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj1 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj1 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001581ec14335d1323-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj11000falseobj12024-04-19T23:26:12.566Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj10 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10 2024-04-20T01:55:20.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.960 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8= 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8=', 'Content-Length': '0'} 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5e6a95c3d5a6413-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8= 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8=', 'Content-Length': '0'} 2024-04-20T01:55:20.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008816d51005a31dc2-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.963 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AqtqIO0CfxAa2Xx/FqIwSqxBkDc= 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqtqIO0CfxAa2Xx/FqIwSqxBkDc=', 'Content-Length': '0'} 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7b64ce4e40f396b-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AqtqIO0CfxAa2Xx/FqIwSqxBkDc= 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqtqIO0CfxAa2Xx/FqIwSqxBkDc=', 'Content-Length': '0'} 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb39abb4a18d1526-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj10 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8= 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8=', 'Content-Length': '0'} 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cb0401b9bc14b95-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.968 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj10?versionId=null 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8= 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uNAn/WMn72lzqMdoOYUPzl/ppE8=', 'Content-Length': '0'} 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d9b9a806f0c33ba-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj10 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj10 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj10 2024-04-20T01:55:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066b25d90f571f07f-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj101000falseobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj10 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj10 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj10 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026c8cafaf5e9c8a2-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj101000falseobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10 2024-04-20T01:55:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj10 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj10 2024-04-20T01:55:20.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:u5YRXT1T/TwTeQKn76ue0TUYwC0= 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u5YRXT1T/TwTeQKn76ue0TUYwC0=', 'Content-Length': '0'} 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb9b7013254be24c-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj10 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj10 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:u5YRXT1T/TwTeQKn76ue0TUYwC0= 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u5YRXT1T/TwTeQKn76ue0TUYwC0=', 'Content-Length': '0'} 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012312477e7de46ff-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj10 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj10 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj10 2024-04-20T01:55:20.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WQe5mKX7mQLQNzmtEtLFuPAQyPA= 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WQe5mKX7mQLQNzmtEtLFuPAQyPA=', 'Content-Length': '0'} 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9c244a05b4967bf-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj10 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj10 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj10 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj10 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WQe5mKX7mQLQNzmtEtLFuPAQyPA= 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WQe5mKX7mQLQNzmtEtLFuPAQyPA=', 'Content-Length': '0'} 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000001884a692f7cd60-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj10 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj10 2024-04-20T01:55:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj10 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.979 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000327e412eb3da5a07-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj101000falseobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj10 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj10 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj10 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000701312c5a82038ca-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj101000falseobj102024-04-19T23:26:12.836Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI= 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI=', 'Content-Length': '0'} 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f34309d91f6eaf62-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.984 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI= 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI=', 'Content-Length': '0'} 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b42c50fc4243647d-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1RWwxselmKhavHETsSEAugeMNqQ= 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1RWwxselmKhavHETsSEAugeMNqQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096f4335698de80e2-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.987 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1RWwxselmKhavHETsSEAugeMNqQ= 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1RWwxselmKhavHETsSEAugeMNqQ=', 'Content-Length': '0'} 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5f15399958c7eaf-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.989 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI= 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI=', 'Content-Length': '0'} 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009cf8238059e97e6-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj11?versionId=null 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI= 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5pCkRdUQqC0ySzC0kG/AZUV2LOI=', 'Content-Length': '0'} 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000215844e24ec434d3-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj11 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj11 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj11 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.992 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000269875472c15e668-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj111000falseobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj11 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj11 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj11 2024-04-20T01:55:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:20.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e54ce20b1fe8b584-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj111000falseobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj11 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.995 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj11 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LoIjMd+TJkqxIdEed9CjYa1F3aI= 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LoIjMd+TJkqxIdEed9CjYa1F3aI=', 'Content-Length': '0'} 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066467b2b21684c94-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj11 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj11 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LoIjMd+TJkqxIdEed9CjYa1F3aI= 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LoIjMd+TJkqxIdEed9CjYa1F3aI=', 'Content-Length': '0'} 2024-04-20T01:55:20.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c56f7b609e5f6666-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj11 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:20.998 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj11 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:b1UbXEXDcF07K6+NScReW1L5wFs= 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b1UbXEXDcF07K6+NScReW1L5wFs=', 'Content-Length': '0'} 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e0fb464d9db8d55-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj11 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj11 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj11 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj11 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:b1UbXEXDcF07K6+NScReW1L5wFs= 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:b1UbXEXDcF07K6+NScReW1L5wFs=', 'Content-Length': '0'} 2024-04-20T01:55:21.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005afba0fa101dda2f-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj11 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj11 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj11 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.001 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1b55db7cd6554b9-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj111000falseobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj11 2024-04-20T01:55:21.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj11 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj11 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.003 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ba15a764df090cf-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj111000falseobj112024-04-19T23:26:12.848Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj12 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc= 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc=', 'Content-Length': '0'} 2024-04-20T01:55:21.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053cef70fa241b58a-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc= 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc=', 'Content-Length': '0'} 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001732fdd64ca27cf3-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:u9m+LNwO0o7Sh5tUHttHsNZQEMw= 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u9m+LNwO0o7Sh5tUHttHsNZQEMw=', 'Content-Length': '0'} 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b82491bcf128a1b4-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:u9m+LNwO0o7Sh5tUHttHsNZQEMw= 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u9m+LNwO0o7Sh5tUHttHsNZQEMw=', 'Content-Length': '0'} 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af9e6178cfabd94e-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj12 2024-04-20T01:55:21.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.011 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc= 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc=', 'Content-Length': '0'} 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d18e7209f78b7c54-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj12?versionId=null 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc= 2024-04-20T01:55:21.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FN5Pm8ygZ5Xy4ImdKmBgUyYOpgc=', 'Content-Length': '0'} 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001797a9976d9fe624-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj12 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj12 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj12 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009722e4e80b3b49a1-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj121000falseobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj12 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj12 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj12 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.016 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a12535ce9f4cebac-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj121000falseobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj12 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj12 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+3MdO0phVU+xitqxbbEoLmsbZaM= 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+3MdO0phVU+xitqxbbEoLmsbZaM=', 'Content-Length': '0'} 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000beebccdada001954-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj12 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.019 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj12 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+3MdO0phVU+xitqxbbEoLmsbZaM= 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+3MdO0phVU+xitqxbbEoLmsbZaM=', 'Content-Length': '0'} 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000937381bc3ac958c7-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj12 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj12 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj12 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:DUKfNDThPBNbWLt3E94KBuyAE4M= 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DUKfNDThPBNbWLt3E94KBuyAE4M=', 'Content-Length': '0'} 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e5121b7a7b4f5e0-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj12 2024-04-20T01:55:21.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj12 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj12 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.022 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj12 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:DUKfNDThPBNbWLt3E94KBuyAE4M= 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DUKfNDThPBNbWLt3E94KBuyAE4M=', 'Content-Length': '0'} 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ff32ba48ba123a7-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj12 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj12 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj12 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000368fd1c4ddeb2d35-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj121000falseobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj12 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj12 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj12 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.025 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfff76ac5995f934-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj121000falseobj122024-04-19T23:26:12.898Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj13 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13 2024-04-20T01:55:21.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.027 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k= 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k=', 'Content-Length': '0'} 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da0f77e172d47bd8-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k= 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k=', 'Content-Length': '0'} 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d9a1b8a2aa59e9d-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.030 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HBFjf2wUQV8eZNkku7iK3D81QIo= 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HBFjf2wUQV8eZNkku7iK3D81QIo=', 'Content-Length': '0'} 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9442cbff09dfab9-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HBFjf2wUQV8eZNkku7iK3D81QIo= 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HBFjf2wUQV8eZNkku7iK3D81QIo=', 'Content-Length': '0'} 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f79f801f32aef7bd-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj13 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k= 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k=', 'Content-Length': '0'} 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000500e9a049d0a2b83-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.035 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj13?versionId=null 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k= 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c947EMeBpNrkNrefJzgHHcc/05k=', 'Content-Length': '0'} 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cdfe8fd0cf38175-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj13 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj13 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj13 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:21.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f03a6e95579240d-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj131000falseobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj13 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj13 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj13 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90= 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGXz7pUOxHOwV+Fe24fsUTauK90=', 'Content-Length': '0'} 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fffa5e9a1ccc40dd-006622fdad-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj131000falseobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13 2024-04-20T01:55:21.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj13 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.040 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj13 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:43N/TZyO7/FbVNF9eR5Lvvr6JNk= 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:43N/TZyO7/FbVNF9eR5Lvvr6JNk=', 'Content-Length': '0'} 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6986e7ca89d6427-006622fdad-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj13 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj13 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:43N/TZyO7/FbVNF9eR5Lvvr6JNk= 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:43N/TZyO7/FbVNF9eR5Lvvr6JNk=', 'Content-Length': '0'} 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6da219bda440860-006622fdad-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj13 2024-04-20T01:55:21.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj13 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.043 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj13 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nZVfLDYF22fcG1Yp78UYlBG4yCo= 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nZVfLDYF22fcG1Yp78UYlBG4yCo=', 'Content-Length': '0'} 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbe45a9b2a6397f6-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj13 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj13 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj13 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:37 GMT 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj13 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nZVfLDYF22fcG1Yp78UYlBG4yCo= 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:37 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nZVfLDYF22fcG1Yp78UYlBG4yCo=', 'Content-Length': '0'} 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a058b187bc97393-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj13 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj13 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj13 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.046 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7cc2b26bb1c8a0e-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj131000falseobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj13 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj13 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj13 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.048 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ddb0809b66ed7c2-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj131000falseobj132024-04-19T23:26:12.908Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj14 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY= 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY=', 'Content-Length': '0'} 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000045dc715bffbd2d7-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.051 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY= 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY=', 'Content-Length': '0'} 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000392f016d9552301a-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:OPz2E/MXlCGEqkyYYAdjKy/MFyo= 2024-04-20T01:55:21.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OPz2E/MXlCGEqkyYYAdjKy/MFyo=', 'Content-Length': '0'} 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000458c1718ca518483-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:OPz2E/MXlCGEqkyYYAdjKy/MFyo= 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OPz2E/MXlCGEqkyYYAdjKy/MFyo=', 'Content-Length': '0'} 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f443cab64c57c8f1-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj14 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14 2024-04-20T01:55:21.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.056 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY= 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY=', 'Content-Length': '0'} 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006841b7bcc406452b-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj14?versionId=null 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY= 2024-04-20T01:55:21.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BjvbvTGQyQjyERyechfmLKABOOY=', 'Content-Length': '0'} 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c555a8f5603b90b8-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj14 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj14 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj14 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065fd0ec3f9b64183-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj141000falseobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj14 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj14 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj14 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.061 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7cacc24471827b4-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj141000falseobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj14 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj14 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:0RKRSF4W85aEi6HmYWJTrSBpmiw= 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0RKRSF4W85aEi6HmYWJTrSBpmiw=', 'Content-Length': '0'} 2024-04-20T01:55:21.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b535b7d05e999ad5-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj14 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.064 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj14 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:0RKRSF4W85aEi6HmYWJTrSBpmiw= 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0RKRSF4W85aEi6HmYWJTrSBpmiw=', 'Content-Length': '0'} 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024e9e8505d7ec048-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj14 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj14 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj14 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:iE8fTNz96M7vStGnybiEI/cSqUw= 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iE8fTNz96M7vStGnybiEI/cSqUw=', 'Content-Length': '0'} 2024-04-20T01:55:21.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e89aa8de612aa14-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj14 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj14 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj14 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.067 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj14 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:iE8fTNz96M7vStGnybiEI/cSqUw= 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iE8fTNz96M7vStGnybiEI/cSqUw=', 'Content-Length': '0'} 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f90d616fd6c6c25-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj14 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj14 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj14 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d50af4d7b078b974-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj141000falseobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj14 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj14 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj14 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a37acf56199ba5bc-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj141000falseobj142024-04-19T23:26:12.954Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj15 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15 2024-04-20T01:55:21.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.072 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ= 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c8847797bfa41b9-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ= 2024-04-20T01:55:21.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e661f03cb64c675-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dUu+9oc8+RqEeqTMvL45iN0qru8= 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dUu+9oc8+RqEeqTMvL45iN0qru8=', 'Content-Length': '0'} 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc40580297a66c47-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dUu+9oc8+RqEeqTMvL45iN0qru8= 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dUu+9oc8+RqEeqTMvL45iN0qru8=', 'Content-Length': '0'} 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035349728ca8e66b9-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj15 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ= 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eeebb1caac1fcb3d-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15 2024-04-20T01:55:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.080 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj15?versionId=null 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ= 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ie/TCWwgqpCkB1/ktLyajKGhTaQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c0a4457bbf1dc62-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj15 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj15 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj15 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8f3055151fc0f8d-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj151000falseobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj15 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj15 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj15 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bcdeb3928e6325b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj151000falseobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15 2024-04-20T01:55:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj15 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.085 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj15 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nBoeWzh+ibTCbREgF2bdO/qAJM8= 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nBoeWzh+ibTCbREgF2bdO/qAJM8=', 'Content-Length': '0'} 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000947774888b2becc9-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj15 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj15 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nBoeWzh+ibTCbREgF2bdO/qAJM8= 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nBoeWzh+ibTCbREgF2bdO/qAJM8=', 'Content-Length': '0'} 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ad1fa7013da011f-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj15 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15 2024-04-20T01:55:21.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj15 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.088 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj15 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:uzVZ/NJqjX1+rHsk4hVdCmFqPrs= 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uzVZ/NJqjX1+rHsk4hVdCmFqPrs=', 'Content-Length': '0'} 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a050f1e54330ccd3-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj15 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj15 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj15 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.090 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.090 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.090 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.090 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.090 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj15 2024-04-20T01:55:21.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.090 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:uzVZ/NJqjX1+rHsk4hVdCmFqPrs= 2024-04-20T01:55:21.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uzVZ/NJqjX1+rHsk4hVdCmFqPrs=', 'Content-Length': '0'} 2024-04-20T01:55:21.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000093bd408c8cbef13-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj15 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj15 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj15 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.091 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c31c2d77516c6204-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj151000falseobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj15 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj15 2024-04-20T01:55:21.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj15 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000027bf28e24f3d03-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj151000falseobj152024-04-19T23:26:12.965Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj16 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo= 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo=', 'Content-Length': '0'} 2024-04-20T01:55:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037a34fd4270f8d4b-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.096 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo= 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo=', 'Content-Length': '0'} 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002de36c73243ce49b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:aoIXtyBa7q/maP1xYAuzNgk0IDU= 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aoIXtyBa7q/maP1xYAuzNgk0IDU=', 'Content-Length': '0'} 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a99ca535afd4dd21-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:aoIXtyBa7q/maP1xYAuzNgk0IDU= 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aoIXtyBa7q/maP1xYAuzNgk0IDU=', 'Content-Length': '0'} 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ad00f70380d725c-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj16 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16 2024-04-20T01:55:21.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.101 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo= 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo=', 'Content-Length': '0'} 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000661a0fe359ab7196-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj16?versionId=null 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo= 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/J+YU14cnb+bH6vmjEbNQ4A46Jo=', 'Content-Length': '0'} 2024-04-20T01:55:21.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b9347fa366809c9-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj16 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj16 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj16 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.104 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048691a09b6b5b41d-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj161000falseobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj16 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj16 2024-04-20T01:55:21.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj16 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000882ce372d55fadd1-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj161000falseobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj16 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj16 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8fbRVNpMUhODOz/bSabz8Wausro= 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8fbRVNpMUhODOz/bSabz8Wausro=', 'Content-Length': '0'} 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000804c75dbb66bb317-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16 2024-04-20T01:55:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj16 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj16 2024-04-20T01:55:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8fbRVNpMUhODOz/bSabz8Wausro= 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8fbRVNpMUhODOz/bSabz8Wausro=', 'Content-Length': '0'} 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080860c6a3f1c2403-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj16 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj16 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj16 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hBS11ASULHnosTZ9F5oFl7seqiU= 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hBS11ASULHnosTZ9F5oFl7seqiU=', 'Content-Length': '0'} 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000adbc9411b84bc93-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj16 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj16 2024-04-20T01:55:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj16 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.112 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj16 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hBS11ASULHnosTZ9F5oFl7seqiU= 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hBS11ASULHnosTZ9F5oFl7seqiU=', 'Content-Length': '0'} 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f7e840bf8260a13-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj16 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj16 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj16 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004833cb3a5ff8ff11-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj161000falseobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj16 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj16 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj16 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.115 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ab02ffabf7c611d-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj161000falseobj162024-04-19T23:26:13.019Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0= 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0=', 'Content-Length': '0'} 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5ca60effc1e9337-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0= 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0=', 'Content-Length': '0'} 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a867ae27bc2d06c-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17 2024-04-20T01:55:21.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.120 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fZ3Kr6/0qub8zWi+NPVf8BflQsI= 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fZ3Kr6/0qub8zWi+NPVf8BflQsI=', 'Content-Length': '0'} 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001eb6ce3aec602f35-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fZ3Kr6/0qub8zWi+NPVf8BflQsI= 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fZ3Kr6/0qub8zWi+NPVf8BflQsI=', 'Content-Length': '0'} 2024-04-20T01:55:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001655c45b5845ed82-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0= 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0=', 'Content-Length': '0'} 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba1ff91da9bb28c7-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj17?versionId=null 2024-04-20T01:55:21.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0= 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K+7AfMAARcRLGGdKX1lyCTi5yE0=', 'Content-Length': '0'} 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049944a88a702b706-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj17 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj17 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj17 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efdd72c3a7c0ff2f-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj171000falseobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj17 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj17 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj17 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.128 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050e3a5815f807580-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj171000falseobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj17 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj17 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tH8rwQxe7MoJgK/KeqoEMToY12o= 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tH8rwQxe7MoJgK/KeqoEMToY12o=', 'Content-Length': '0'} 2024-04-20T01:55:21.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b74f1acef798760d-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj17 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.131 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj17 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tH8rwQxe7MoJgK/KeqoEMToY12o= 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tH8rwQxe7MoJgK/KeqoEMToY12o=', 'Content-Length': '0'} 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008697d02e976c4e8a-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj17 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj17 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:djgMuTyHIaNInOJnGn1xwEYOtoA= 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:djgMuTyHIaNInOJnGn1xwEYOtoA=', 'Content-Length': '0'} 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0a3cfbf22ac30b5-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj17 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj17 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj17 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.134 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj17 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:djgMuTyHIaNInOJnGn1xwEYOtoA= 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:djgMuTyHIaNInOJnGn1xwEYOtoA=', 'Content-Length': '0'} 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e51a784fc53b329d-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj17 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj17 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj17 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093892da2d803fcfe-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj171000falseobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj17 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj17 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj17 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf479447eb948634-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj171000falseobj172024-04-19T23:26:13.030Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj18 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18 2024-04-20T01:55:21.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.139 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw= 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw=', 'Content-Length': '0'} 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e31a4b289a132e4a-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw= 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw=', 'Content-Length': '0'} 2024-04-20T01:55:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ee49123c4587647-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.145 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:9BlM1/0Aww32tf801X3f1RzUsdA= 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9BlM1/0Aww32tf801X3f1RzUsdA=', 'Content-Length': '0'} 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e199b8525fc4f73e-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.147 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:9BlM1/0Aww32tf801X3f1RzUsdA= 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9BlM1/0Aww32tf801X3f1RzUsdA=', 'Content-Length': '0'} 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5c26d4d6825bd10-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj18 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw= 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw=', 'Content-Length': '0'} 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000316102b46a6ddb9f-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.150 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj18?versionId=null 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw= 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KedeVEsCaVzyaa08d47RZjJumjw=', 'Content-Length': '0'} 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026546691c4e2c52b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj18 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj18 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj18 2024-04-20T01:55:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003590c9ba931579a6-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj181000falseobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj18 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj18 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj18 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004de727c2ce2b9ddf-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj181000falseobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18 2024-04-20T01:55:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj18 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj18 2024-04-20T01:55:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Zzu+mNwHdffBtjX354X2GzVp8Z0= 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zzu+mNwHdffBtjX354X2GzVp8Z0=', 'Content-Length': '0'} 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcc346167d5294d0-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj18 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj18 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Zzu+mNwHdffBtjX354X2GzVp8Z0= 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Zzu+mNwHdffBtjX354X2GzVp8Z0=', 'Content-Length': '0'} 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e1c9d4158c06728-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj18 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj18 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj18 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.158 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:YpviMq0WuPNIyMfFeMd6+v9symU= 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpviMq0WuPNIyMfFeMd6+v9symU=', 'Content-Length': '0'} 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074edd147fed816fe-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj18 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj18 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj18 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.159 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj18 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:YpviMq0WuPNIyMfFeMd6+v9symU= 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:YpviMq0WuPNIyMfFeMd6+v9symU=', 'Content-Length': '0'} 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe917ed409f93e2f-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj18 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj18 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj18 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.161 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d41d56e9d3f847b4-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj181000falseobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj18 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj18 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj18 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9822f573ddf47dc-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj181000falseobj182024-04-19T23:26:13.041Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.163 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj19 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.164 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw= 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw=', 'Content-Length': '0'} 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d85c4a19c49c27ef-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.166 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw= 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw=', 'Content-Length': '0'} 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f87121a9b4f18a1d-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HazYyvj85FjqxT5y0Aid8Xb3cAY= 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HazYyvj85FjqxT5y0Aid8Xb3cAY=', 'Content-Length': '0'} 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d7e60da5218809f-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.169 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HazYyvj85FjqxT5y0Aid8Xb3cAY= 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HazYyvj85FjqxT5y0Aid8Xb3cAY=', 'Content-Length': '0'} 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd1460779545206b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj19 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw= 2024-04-20T01:55:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw=', 'Content-Length': '0'} 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0355cb4bde0f233-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj19?versionId=null 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw= 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LWTOLyv68cWRQVWAy9UI/EmBTRw=', 'Content-Length': '0'} 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000671423ab9a14c40c-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj19 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj19 2024-04-20T01:55:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj19 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b7cd68f53e2d86b-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj191000falseobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj19 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj19 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj19 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9d17abfc860353b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj191000falseobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj19 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.177 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj19 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EQ4hjtqKLA1Z+IMZoj/GBja9M9s= 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EQ4hjtqKLA1Z+IMZoj/GBja9M9s=', 'Content-Length': '0'} 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000095dc486b0abeda2-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj19 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj19 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EQ4hjtqKLA1Z+IMZoj/GBja9M9s= 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EQ4hjtqKLA1Z+IMZoj/GBja9M9s=', 'Content-Length': '0'} 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c40885d10715d49-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.179 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj19 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj19 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.180 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj19 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Uf+9+w68gqXM94LysRb10TSG5DI= 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Uf+9+w68gqXM94LysRb10TSG5DI=', 'Content-Length': '0'} 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034ec301da8e5ba9b-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj19 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj19 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj19 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj19 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Uf+9+w68gqXM94LysRb10TSG5DI= 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Uf+9+w68gqXM94LysRb10TSG5DI=', 'Content-Length': '0'} 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000399949c5971021f7-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj19 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj19 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj19 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.183 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000741c2bd3fa94ab9-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj191000falseobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj19 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj19 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj19 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.185 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b61666fe1939c2b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj191000falseobj192024-04-19T23:26:13.095Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj2 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA= 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA=', 'Content-Length': '0'} 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9d745ee4105bc32-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.188 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA= 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA=', 'Content-Length': '0'} 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f83002a0cf3cf640-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.190 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qOh0DBEhQvgnQZDOPdjtHRCzYlg= 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qOh0DBEhQvgnQZDOPdjtHRCzYlg=', 'Content-Length': '0'} 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae82722e32254b7c-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qOh0DBEhQvgnQZDOPdjtHRCzYlg= 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qOh0DBEhQvgnQZDOPdjtHRCzYlg=', 'Content-Length': '0'} 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fcf88b244f33a27-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj2 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2 2024-04-20T01:55:21.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.193 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA= 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA=', 'Content-Length': '0'} 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012d4eab4e5a6cc49-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj2?versionId=null 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA= 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AROAV40IkcFKH/HPaL4WPZZXzlA=', 'Content-Length': '0'} 2024-04-20T01:55:21.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d2b17e8fbb183eb-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj2 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj2 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj2 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.196 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000153be6504a2c1c33-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj21000falseobj22024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj2 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj2 2024-04-20T01:55:21.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj2 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c00f627763d8b0e3-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj21000falseobj22024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj2 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj2 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Mc2uk9IARTNO4v2AsoZfKq3hjzs= 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Mc2uk9IARTNO4v2AsoZfKq3hjzs=', 'Content-Length': '0'} 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d9be398748785a9-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2 2024-04-20T01:55:21.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj2 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj2 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.201 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Mc2uk9IARTNO4v2AsoZfKq3hjzs= 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Mc2uk9IARTNO4v2AsoZfKq3hjzs=', 'Content-Length': '0'} 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000653b0b2a79cb33c-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj2 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj2 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj2 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rmD/OVfx768/greWwHE8tVdkrC4= 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rmD/OVfx768/greWwHE8tVdkrC4=', 'Content-Length': '0'} 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afb53c2e0831b0b7-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj2 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj2 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj2 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj2 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.204 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rmD/OVfx768/greWwHE8tVdkrC4= 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rmD/OVfx768/greWwHE8tVdkrC4=', 'Content-Length': '0'} 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbe84b29afa00cab-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj2 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj2 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj2 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d0616ff01878221-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj21000falseobj22024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj2 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj2 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj2 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.207 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a01c1504a6f23a1-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj21000falseobj22024-04-19T23:26:12.577Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj20 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.209 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14= 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14=', 'Content-Length': '0'} 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003694025d03ff0129-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14= 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14=', 'Content-Length': '0'} 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072f9f38b23e2455b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.212 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1DSkS2pdAHOnzIPj1WN4SsCr/es= 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1DSkS2pdAHOnzIPj1WN4SsCr/es=', 'Content-Length': '0'} 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006265432a42a5954f-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1DSkS2pdAHOnzIPj1WN4SsCr/es= 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1DSkS2pdAHOnzIPj1WN4SsCr/es=', 'Content-Length': '0'} 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c14039f03a39e616-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj20 2024-04-20T01:55:21.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.215 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14= 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14=', 'Content-Length': '0'} 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aee2efcd2e51f8d5-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj20?versionId=null 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.217 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14= 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r5SZQrQAvQWx5C+Ha2SOOHA9w14=', 'Content-Length': '0'} 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074d8d459bb59301d-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj20 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj20 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj20 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5008239b4cc6bae-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj201000falseobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj20 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj20 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj20 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.220 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090d14a137569c439-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj201000falseobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj20 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj20 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sVASCfz51ZyQzp/VefvI77+J/kw= 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sVASCfz51ZyQzp/VefvI77+J/kw=', 'Content-Length': '0'} 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f78f96fe09ec915-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj20 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.223 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj20 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sVASCfz51ZyQzp/VefvI77+J/kw= 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sVASCfz51ZyQzp/VefvI77+J/kw=', 'Content-Length': '0'} 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d8411e69d57c84b-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj20 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj20 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj20 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qeDPhB5yq7ZD0YA8az30o6cAW6w= 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qeDPhB5yq7ZD0YA8az30o6cAW6w=', 'Content-Length': '0'} 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e2df0f6db97c5a8-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj20 2024-04-20T01:55:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj20 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj20 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.226 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj20 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qeDPhB5yq7ZD0YA8az30o6cAW6w= 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qeDPhB5yq7ZD0YA8az30o6cAW6w=', 'Content-Length': '0'} 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000424dea6409783783-006622fdae-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj20 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj20 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj20 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000c777bbcbedd125-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj201000falseobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj20 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj20 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj20 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.229 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4= 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d2D7TvqOfdW6pyIm7Tklpv4tYz4=', 'Content-Length': '0'} 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040380aa8bda896ac-006622fdae-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj201000falseobj202024-04-19T23:26:13.105Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj21 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:38 GMT 2024-04-20T01:55:21.231 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.232 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:u488v733Fw92mSGAWfeXusrgYyg= 2024-04-20T01:55:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:38 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:u488v733Fw92mSGAWfeXusrgYyg=', 'Content-Length': '0'} 2024-04-20T01:55:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df23d329556cbbde-006622fdae-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:38 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21 2024-04-20T01:55:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21 2024-04-20T01:55:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ddoSZzT7/LHyKbw2oj5SllHeRdA= 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ddoSZzT7/LHyKbw2oj5SllHeRdA=', 'Content-Length': '0'} 2024-04-20T01:55:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c1432c4d3d1c9fc-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.234 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:UMgmIzmz5t5zJlByBsJY0+qKg4I= 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UMgmIzmz5t5zJlByBsJY0+qKg4I=', 'Content-Length': '0'} 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000309a4113fbe4c0ee-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:UMgmIzmz5t5zJlByBsJY0+qKg4I= 2024-04-20T01:55:21.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UMgmIzmz5t5zJlByBsJY0+qKg4I=', 'Content-Length': '0'} 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068ae06adcfba6a44-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj21 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ddoSZzT7/LHyKbw2oj5SllHeRdA= 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ddoSZzT7/LHyKbw2oj5SllHeRdA=', 'Content-Length': '0'} 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b29d4e8d20ada028-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.239 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj21?versionId=null 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ddoSZzT7/LHyKbw2oj5SllHeRdA= 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ddoSZzT7/LHyKbw2oj5SllHeRdA=', 'Content-Length': '0'} 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021bbaadec8afe5b6-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj21 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj21 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj21 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000057a9d6e54011310-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj211000falseobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj21 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj21 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj21 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.242 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062cab148c7510608-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj211000falseobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj21 2024-04-20T01:55:21.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj21 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8pIl+ISpl7iql7XRcM0H1zseptE= 2024-04-20T01:55:21.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8pIl+ISpl7iql7XRcM0H1zseptE=', 'Content-Length': '0'} 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004432b9442b78a634-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj21 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.245 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj21 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8pIl+ISpl7iql7XRcM0H1zseptE= 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8pIl+ISpl7iql7XRcM0H1zseptE=', 'Content-Length': '0'} 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d594960045b90936-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj21 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj21 2024-04-20T01:55:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj21 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:mGQatwtgeRAtAUz/O2neLzPulfA= 2024-04-20T01:55:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mGQatwtgeRAtAUz/O2neLzPulfA=', 'Content-Length': '0'} 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000833dbd9b87e1b39c-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj21 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj21 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj21 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.248 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj21 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:mGQatwtgeRAtAUz/O2neLzPulfA= 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mGQatwtgeRAtAUz/O2neLzPulfA=', 'Content-Length': '0'} 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000464f5cbd6f6715fe-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj21 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj21 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj21 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.250 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a35c30fae5b90ca9-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj211000falseobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj21 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj21 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj21 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000753f3a79bf3515c6-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj211000falseobj212024-04-19T23:26:13.116Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-20T01:55:21.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.253 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0= 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0=', 'Content-Length': '0'} 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d65593f3bf760cdf-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.255 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0= 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0=', 'Content-Length': '0'} 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c824670dfb4551d4-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fwJRdN5aQIT0A/zB1S6AtCwlSJc= 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fwJRdN5aQIT0A/zB1S6AtCwlSJc=', 'Content-Length': '0'} 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdc7984b1ea3ca3e-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.258 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fwJRdN5aQIT0A/zB1S6AtCwlSJc= 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fwJRdN5aQIT0A/zB1S6AtCwlSJc=', 'Content-Length': '0'} 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb47f4cef51684b8-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0= 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0=', 'Content-Length': '0'} 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad3c413068c96867-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.261 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj22?versionId=null 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0= 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tITmM7W9Hwr7XBaOnN36FtMAkJ0=', 'Content-Length': '0'} 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000158037c7d39bcf84-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj22 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj22 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj22 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.263 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002e5cb08ed524e93-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj221000falseobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj22 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj22 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj22 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c7454df610f9800-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj221000falseobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj22 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.266 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj22 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WL7RhfzDw1pLoroMDyP3lXgv70s= 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WL7RhfzDw1pLoroMDyP3lXgv70s=', 'Content-Length': '0'} 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000306ddaf0b6cb0ffd-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj22 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj22 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WL7RhfzDw1pLoroMDyP3lXgv70s= 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WL7RhfzDw1pLoroMDyP3lXgv70s=', 'Content-Length': '0'} 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000468951e1a8995ea1-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22 2024-04-20T01:55:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj22 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.269 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj22 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dJFa7gXb1tTrQudS/yeWYCzM+VA= 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dJFa7gXb1tTrQudS/yeWYCzM+VA=', 'Content-Length': '0'} 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044a4b1e79ea6306c-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj22 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj22 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj22 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj22 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dJFa7gXb1tTrQudS/yeWYCzM+VA= 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dJFa7gXb1tTrQudS/yeWYCzM+VA=', 'Content-Length': '0'} 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dd12092021f769e-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj22 2024-04-20T01:55:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj22 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj22 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.272 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000474b06f90042beb3-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj221000falseobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj22 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj22 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj22 2024-04-20T01:55:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043101fdfa2f9e07d-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj221000falseobj222024-04-19T23:26:13.163Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj23 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34= 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34=', 'Content-Length': '0'} 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9a9840d29968428-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23 2024-04-20T01:55:21.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.277 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34= 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34=', 'Content-Length': '0'} 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000348ed0d0c624089f-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:NlYF3RZ4OY9zuYTrxKjBE2x/fsg= 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NlYF3RZ4OY9zuYTrxKjBE2x/fsg=', 'Content-Length': '0'} 2024-04-20T01:55:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b399f08d309a752b-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23 2024-04-20T01:55:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23 2024-04-20T01:55:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.281 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.281 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.281 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.281 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.281 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.281 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:NlYF3RZ4OY9zuYTrxKjBE2x/fsg= 2024-04-20T01:55:21.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NlYF3RZ4OY9zuYTrxKjBE2x/fsg=', 'Content-Length': '0'} 2024-04-20T01:55:21.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000431c274c4e408b66-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.281 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj23 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.283 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.283 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.283 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.283 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.283 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.283 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34= 2024-04-20T01:55:21.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34=', 'Content-Length': '0'} 2024-04-20T01:55:21.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083dd3c70d3e35cd1-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23 2024-04-20T01:55:21.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23 2024-04-20T01:55:21.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.284 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj23?versionId=null 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34= 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2+IhZ8lnWFTBmuUQYR1XwYAuw34=', 'Content-Length': '0'} 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dc296d90d71fc15-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj23 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj23 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj23 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.286 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.286 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.286 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.286 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.286 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053c2f8f37bc1ed66-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj231000falseobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj23 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj23 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj23 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.288 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.288 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.288 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.288 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.288 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.288 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d612fbc5986fa6d-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj231000falseobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj23 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj23 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LPmhOcidz2Eki57bSq0cuvGHGMM= 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LPmhOcidz2Eki57bSq0cuvGHGMM=', 'Content-Length': '0'} 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000786c00e593af769a-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj23 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj23 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.291 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LPmhOcidz2Eki57bSq0cuvGHGMM= 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LPmhOcidz2Eki57bSq0cuvGHGMM=', 'Content-Length': '0'} 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9576f91d160c181-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj23 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj23 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj23 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Qj3Tv3Cxsnnq3vY75Q+SinroJ40= 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Qj3Tv3Cxsnnq3vY75Q+SinroJ40=', 'Content-Length': '0'} 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007cbf2aa96218d9b2-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj23 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj23 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj23 2024-04-20T01:55:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj23 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.294 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Qj3Tv3Cxsnnq3vY75Q+SinroJ40= 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Qj3Tv3Cxsnnq3vY75Q+SinroJ40=', 'Content-Length': '0'} 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a27a2c177bebef8-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj23 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj23 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj23 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015dcd4dc52c2b550-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj231000falseobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj23 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj23 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj23 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.297 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054a28fe5a6a1663f-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj231000falseobj232024-04-19T23:26:13.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj24 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.299 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o= 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o=', 'Content-Length': '0'} 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f798c9ccc1720c5d-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o= 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o=', 'Content-Length': '0'} 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba625e9f36ddc4fa-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.302 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:MyU+XMttDiay5XfzTxYphkMW6Kc= 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MyU+XMttDiay5XfzTxYphkMW6Kc=', 'Content-Length': '0'} 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e34ad0d0fc7fcfa-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:MyU+XMttDiay5XfzTxYphkMW6Kc= 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MyU+XMttDiay5XfzTxYphkMW6Kc=', 'Content-Length': '0'} 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000248777b7e16c2a4e-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj24 2024-04-20T01:55:21.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.305 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o= 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o=', 'Content-Length': '0'} 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d7c15d4c50b10ef-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj24?versionId=null 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o= 2024-04-20T01:55:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KeKSPpWiNpnHWr4oNQly6aqpY3o=', 'Content-Length': '0'} 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006398ec760b366a83-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj24 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj24 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj24 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bc5eb618eaab5e2-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj241000falseobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj24 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj24 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj24 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.310 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d13c800c711153c-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj241000falseobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj24 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj24 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:NKJ+ERI1zY8/zgHiCz8vydyPjlI= 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NKJ+ERI1zY8/zgHiCz8vydyPjlI=', 'Content-Length': '0'} 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eea0898b20cdaa15-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj24 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.313 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj24 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:NKJ+ERI1zY8/zgHiCz8vydyPjlI= 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NKJ+ERI1zY8/zgHiCz8vydyPjlI=', 'Content-Length': '0'} 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016f49dd54f7a3969-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj24 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj24 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj24 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+Fw0cXjT/uShfolYLVT4e4DFj3o= 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+Fw0cXjT/uShfolYLVT4e4DFj3o=', 'Content-Length': '0'} 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f3973514ca06883-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj24 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj24 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj24 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.316 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj24 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+Fw0cXjT/uShfolYLVT4e4DFj3o= 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+Fw0cXjT/uShfolYLVT4e4DFj3o=', 'Content-Length': '0'} 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef85b35ca771f60f-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj24 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj24 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj24 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009356729953a40f8b-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj241000falseobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj24 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj24 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj24 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.319 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d0917c0abc910a2-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj241000falseobj242024-04-19T23:26:13.228Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj25 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo= 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo=', 'Content-Length': '0'} 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c4462efca65d86e-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo= 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo=', 'Content-Length': '0'} 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e281f691fad261d-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25 2024-04-20T01:55:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.324 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:wUInX1rIY1X4k/gjuL7c9jB+HTc= 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wUInX1rIY1X4k/gjuL7c9jB+HTc=', 'Content-Length': '0'} 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3f1b4deb37031d2-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:wUInX1rIY1X4k/gjuL7c9jB+HTc= 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wUInX1rIY1X4k/gjuL7c9jB+HTc=', 'Content-Length': '0'} 2024-04-20T01:55:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c1a0fcfd5a156c8-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj25 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo= 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo=', 'Content-Length': '0'} 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058873134c543b9b9-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj25?versionId=null 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.329 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo= 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ek2mayQQzVXlzz16Aqk3SPKHYOo=', 'Content-Length': '0'} 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000574c57e2cd0bf0c2-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj25 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj25 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj25 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004eed11fc20bc7e65-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj251000falseobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj25 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj25 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj25 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.332 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0c49b9f347b220f-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj251000falseobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj25 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj25 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Goy24wK2sREjHvDks4fofHAbFGI= 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Goy24wK2sREjHvDks4fofHAbFGI=', 'Content-Length': '0'} 2024-04-20T01:55:21.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036e6a42d64af7943-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj25 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.335 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj25 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Goy24wK2sREjHvDks4fofHAbFGI= 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Goy24wK2sREjHvDks4fofHAbFGI=', 'Content-Length': '0'} 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d92534ae4eb9f52a-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj25 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj25 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj25 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2Jr7pfIjN26yWqTnT/k2plX2mLc= 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Jr7pfIjN26yWqTnT/k2plX2mLc=', 'Content-Length': '0'} 2024-04-20T01:55:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007632d82d2f538723-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj25 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj25 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj25 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.338 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj25 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2Jr7pfIjN26yWqTnT/k2plX2mLc= 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2Jr7pfIjN26yWqTnT/k2plX2mLc=', 'Content-Length': '0'} 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077456c84009aacbd-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj25 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj25 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj25 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4471a28bef3f515-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj251000falseobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj25 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj25 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj25 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b6bd07a78d94046-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj251000falseobj252024-04-19T23:26:13.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj26 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26 2024-04-20T01:55:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.343 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U= 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U=', 'Content-Length': '0'} 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa89023607c018ba-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U= 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U=', 'Content-Length': '0'} 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bc0f0b8c0f5a530-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.346 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rQYUECDsS/lLIF1MbR1BHq1A9Nc= 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rQYUECDsS/lLIF1MbR1BHq1A9Nc=', 'Content-Length': '0'} 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000815f5a76b0dd19ff-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.350 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.350 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.350 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.350 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.350 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.350 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rQYUECDsS/lLIF1MbR1BHq1A9Nc= 2024-04-20T01:55:21.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rQYUECDsS/lLIF1MbR1BHq1A9Nc=', 'Content-Length': '0'} 2024-04-20T01:55:21.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010fc788b1aaa4c40-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj26 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U= 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U=', 'Content-Length': '0'} 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000264d00f379bf8b78-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.353 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj26?versionId=null 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U= 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9u/m+2UrR2/qtXDmjxsxpfWOf0U=', 'Content-Length': '0'} 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000356deee4b306f74c-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj26 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj26 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj26 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000603e9701595ed440-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj261000falseobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj26 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj26 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj26 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.356 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003aa3d5334975bad5-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj261000falseobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj26 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj26 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.358 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Trq6kWHbkyFYZJQFK7T9s4ly3LI= 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Trq6kWHbkyFYZJQFK7T9s4ly3LI=', 'Content-Length': '0'} 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070207abc08336999-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj26 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.359 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj26 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Trq6kWHbkyFYZJQFK7T9s4ly3LI= 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Trq6kWHbkyFYZJQFK7T9s4ly3LI=', 'Content-Length': '0'} 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c571bdba46fae41-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj26 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj26 2024-04-20T01:55:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj26 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:B+15N8xHoAErXKIVmyjXDq/w5As= 2024-04-20T01:55:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B+15N8xHoAErXKIVmyjXDq/w5As=', 'Content-Length': '0'} 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000405253a3b89a2055-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj26 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj26 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj26 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.362 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj26 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:B+15N8xHoAErXKIVmyjXDq/w5As= 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B+15N8xHoAErXKIVmyjXDq/w5As=', 'Content-Length': '0'} 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b89a6fd0e3df0ff5-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj26 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj26 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj26 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.364 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075a049ea862a1e2e-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj261000falseobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj26 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj26 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj26 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076874c827f44a70b-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj261000falseobj262024-04-19T23:26:13.287Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.366 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj27 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.367 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk= 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk=', 'Content-Length': '0'} 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ceb5d21a4b15afae-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.369 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk= 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk=', 'Content-Length': '0'} 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009da1581ca0bcbc9d-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1rzJsO7eXgE4NFwRisbZdejV5QU= 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1rzJsO7eXgE4NFwRisbZdejV5QU=', 'Content-Length': '0'} 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1b97ec40b805844-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27 2024-04-20T01:55:21.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.372 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1rzJsO7eXgE4NFwRisbZdejV5QU= 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1rzJsO7eXgE4NFwRisbZdejV5QU=', 'Content-Length': '0'} 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3691f8f878d4961-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj27 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk= 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk=', 'Content-Length': '0'} 2024-04-20T01:55:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000054b7e5ca97c2351-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.375 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj27?versionId=null 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk= 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yMgdq6irQ2AKiOdWJVWoLRh3ckk=', 'Content-Length': '0'} 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b06283d9e5f56d21-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj27 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj27 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj27 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c46ef8a59156bc7-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj271000falseobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj27 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj27 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj27 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7c56b510526f7c-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj271000falseobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27 2024-04-20T01:55:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj27 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.380 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj27 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KckFcHUeHICmaSOx9SH7U9zjDCo= 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KckFcHUeHICmaSOx9SH7U9zjDCo=', 'Content-Length': '0'} 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014c3ca2cb10dcef5-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj27 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj27 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KckFcHUeHICmaSOx9SH7U9zjDCo= 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KckFcHUeHICmaSOx9SH7U9zjDCo=', 'Content-Length': '0'} 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089bde4e1fd062828-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj27 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27 2024-04-20T01:55:21.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj27 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj27 2024-04-20T01:55:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nW0IbLU3TTagnEGGtPlA7mMDJSw= 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nW0IbLU3TTagnEGGtPlA7mMDJSw=', 'Content-Length': '0'} 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e71292b9cc771546-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj27 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj27 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj27 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj27 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nW0IbLU3TTagnEGGtPlA7mMDJSw= 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nW0IbLU3TTagnEGGtPlA7mMDJSw=', 'Content-Length': '0'} 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb05122f47c8af51-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj27 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj27 2024-04-20T01:55:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj27 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647b15f707d36db5-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj271000falseobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj27 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj27 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj27 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f0f4804ac46767d-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj271000falseobj272024-04-19T23:26:13.297Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.388 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj28 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.389 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA= 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA=', 'Content-Length': '0'} 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000958c8655a35efa27-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA= 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA=', 'Content-Length': '0'} 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077ace566115f447b-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:IB2EKtkPVLxkCY/OZ0jWyNXWN3A= 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IB2EKtkPVLxkCY/OZ0jWyNXWN3A=', 'Content-Length': '0'} 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091170bad3ba58905-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.394 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:IB2EKtkPVLxkCY/OZ0jWyNXWN3A= 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IB2EKtkPVLxkCY/OZ0jWyNXWN3A=', 'Content-Length': '0'} 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cce45ed216980d2-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj28 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA= 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA=', 'Content-Length': '0'} 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063edb07884045b13-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.397 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj28?versionId=null 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA= 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zeQ5V+TG2sUmQr6bhuzjJq+ydXA=', 'Content-Length': '0'} 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081ebb1c02a18f24e-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj28 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj28 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj28 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081e1ff8ab11a0e4d-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj281000falseobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj28 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj28 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj28 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo= 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zhgJUV3tU2rY7dM1wEb53TPd0zo=', 'Content-Length': '0'} 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000950e7f308148f9f4-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj281000falseobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28 2024-04-20T01:55:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj28 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj28 2024-04-20T01:55:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Z3bQ5a2MgHif541EjhFnN6Hip2Y= 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z3bQ5a2MgHif541EjhFnN6Hip2Y=', 'Content-Length': '0'} 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067023b426af9daa0-006622fdaf-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj28 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj28 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Z3bQ5a2MgHif541EjhFnN6Hip2Y= 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z3bQ5a2MgHif541EjhFnN6Hip2Y=', 'Content-Length': '0'} 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba2d8ec9213f2ee8-006622fdaf-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj28 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj28 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:39 GMT 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj28 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.405 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1CBoBNGbZOQlZ5ITkhAfGOqXwxM= 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:39 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1CBoBNGbZOQlZ5ITkhAfGOqXwxM=', 'Content-Length': '0'} 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093733b5b16b11c17-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj28 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj28 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj28 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.406 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj28 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Wdeb8IGTFWWbxYQrL2XgENz7JPU= 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Wdeb8IGTFWWbxYQrL2XgENz7JPU=', 'Content-Length': '0'} 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000758f0a372d951b5f-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj28 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj28 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj28 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.408 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f7f29892d03c034-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj281000falseobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj28 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj28 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj28 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050021151af770eb8-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj281000falseobj282024-04-19T23:26:13.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.410 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj29 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.411 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c= 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c=', 'Content-Length': '0'} 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009aff63fcfdffc8a9-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.413 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c= 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c=', 'Content-Length': '0'} 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062d46e2b45c9870b-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZiCisE2SNZ0UlZQirdJ/6liOffQ= 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZiCisE2SNZ0UlZQirdJ/6liOffQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000664c00c4765e7779-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29 2024-04-20T01:55:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.416 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZiCisE2SNZ0UlZQirdJ/6liOffQ= 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZiCisE2SNZ0UlZQirdJ/6liOffQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007af55ba818d57e93-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj29 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c= 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c=', 'Content-Length': '0'} 2024-04-20T01:55:21.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d247bd9efec89524-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj29?versionId=null 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c= 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qfuismZgaq9JG/j3w9YFhTpWk7c=', 'Content-Length': '0'} 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f30a1c606d7c8ef-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj29 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj29 2024-04-20T01:55:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj29 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008507e2d7fd5e6e6f-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj291000falseobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj29 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj29 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj29 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026d71d27ce1a4f0d-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj291000falseobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj29 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.424 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj29 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EraQY/yInyykUjTaYn0Sw8ZC+pY= 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EraQY/yInyykUjTaYn0Sw8ZC+pY=', 'Content-Length': '0'} 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000992c391367ffe2cb-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj29 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj29 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EraQY/yInyykUjTaYn0Sw8ZC+pY= 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EraQY/yInyykUjTaYn0Sw8ZC+pY=', 'Content-Length': '0'} 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfdfceee1d990082-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.426 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj29 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj29 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.427 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj29 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WVx1CcDA4XDcitlFhthPwEwadzA= 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WVx1CcDA4XDcitlFhthPwEwadzA=', 'Content-Length': '0'} 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a40c9035d989abe-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj29 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj29 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj29 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj29 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WVx1CcDA4XDcitlFhthPwEwadzA= 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WVx1CcDA4XDcitlFhthPwEwadzA=', 'Content-Length': '0'} 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000931aa794a977d0db-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj29 2024-04-20T01:55:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj29 2024-04-20T01:55:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj29 2024-04-20T01:55:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.431 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.431 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.431 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.431 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.431 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.431 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003104d41e23f1603f-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj291000falseobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj29 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj29 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj29 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.432 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddd951a8559e4841-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj291000falseobj292024-04-19T23:26:13.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj3 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3 2024-04-20T01:55:21.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.434 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss= 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss=', 'Content-Length': '0'} 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f7ce2d6e22ff5df-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss= 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss=', 'Content-Length': '0'} 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d898ce304db1da2e-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.437 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:IH5YlwSv+zpCDEHX9Fg+DCgmhRo= 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IH5YlwSv+zpCDEHX9Fg+DCgmhRo=', 'Content-Length': '0'} 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006624818534d63075-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.439 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:IH5YlwSv+zpCDEHX9Fg+DCgmhRo= 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IH5YlwSv+zpCDEHX9Fg+DCgmhRo=', 'Content-Length': '0'} 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddbab46da24fe5b9-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj3 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss= 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss=', 'Content-Length': '0'} 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7402aa7ae62c3a9-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3 2024-04-20T01:55:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.442 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj3?versionId=null 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss= 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JnmL+EohF3e+KQAnmG97sSFDQss=', 'Content-Length': '0'} 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd257b72db045772-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj3 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj3 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj3 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040ac69e56c081420-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj31000falseobj32024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj3 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj3 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj3 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083941db154047aab-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj31000falseobj32024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj3 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj3 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.447 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XQI4Whz5Z/pkDMye3pDEF6AFjbI= 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XQI4Whz5Z/pkDMye3pDEF6AFjbI=', 'Content-Length': '0'} 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fc5b53b7ae489ff-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj3 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.448 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj3 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XQI4Whz5Z/pkDMye3pDEF6AFjbI= 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XQI4Whz5Z/pkDMye3pDEF6AFjbI=', 'Content-Length': '0'} 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b797ce35c074490-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj3 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj3 2024-04-20T01:55:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj3 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.450 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3k17iRo2XJGQs0/nP7oZ57vXgsU= 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3k17iRo2XJGQs0/nP7oZ57vXgsU=', 'Content-Length': '0'} 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004394ecc8be3b298-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj3 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj3 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj3 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.451 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj3 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3k17iRo2XJGQs0/nP7oZ57vXgsU= 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3k17iRo2XJGQs0/nP7oZ57vXgsU=', 'Content-Length': '0'} 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcb5344ec6285a97-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj3 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj3 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj3 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.453 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbded74056940ea2-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj31000falseobj32024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj3 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj3 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj3 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ee9530054eca798-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj31000falseobj32024-04-19T23:26:12.630Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.455 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj30 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk= 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk=', 'Content-Length': '0'} 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a087a07bce56480c-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk= 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk=', 'Content-Length': '0'} 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000896e2f231bfba69-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HkgMPSGyucZGSK8AwHwyquhDZYc= 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HkgMPSGyucZGSK8AwHwyquhDZYc=', 'Content-Length': '0'} 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d2739eddffde520-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30 2024-04-20T01:55:21.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.461 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HkgMPSGyucZGSK8AwHwyquhDZYc= 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HkgMPSGyucZGSK8AwHwyquhDZYc=', 'Content-Length': '0'} 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e8fb23f9dc21090-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj30 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk= 2024-04-20T01:55:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk=', 'Content-Length': '0'} 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f54a74d8015b8bf-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj30?versionId=null 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk= 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yrAG3tD6Nog8fjWfHMWmmi8i6Sk=', 'Content-Length': '0'} 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008275ddf986ab27fb-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj30 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj30 2024-04-20T01:55:21.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj30 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.466 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f9a2c42e7386c9d-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj301000falseobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj30 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj30 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj30 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c754da078a1c399e-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj301000falseobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj30 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.469 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj30 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8DJLqC+sf2h19QqWvrh0eDajWuo= 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8DJLqC+sf2h19QqWvrh0eDajWuo=', 'Content-Length': '0'} 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7c357b3545ba231-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj30 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj30 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8DJLqC+sf2h19QqWvrh0eDajWuo= 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8DJLqC+sf2h19QqWvrh0eDajWuo=', 'Content-Length': '0'} 2024-04-20T01:55:21.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008925d0dc7ed18d75-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj30 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj30 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.472 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj30 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ly0FbRvyVKIbRGccr88h92pm76s= 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ly0FbRvyVKIbRGccr88h92pm76s=', 'Content-Length': '0'} 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb56cdaa925986ba-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj30 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj30 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj30 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj30 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ly0FbRvyVKIbRGccr88h92pm76s= 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ly0FbRvyVKIbRGccr88h92pm76s=', 'Content-Length': '0'} 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d59caf2e6cf2da8a-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj30 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj30 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj30 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.475 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015b00c6cdb3fb667-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj301000falseobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj30 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj30 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj30 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.477 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d95f65c0a8369a-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj301000falseobj302024-04-19T23:26:13.371Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj31 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8= 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8=', 'Content-Length': '0'} 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8adc7eecf8ee94d-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.480 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8= 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8=', 'Content-Length': '0'} 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6d856b96eecb673-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KtK5TniNDv+kZXvXSJ/fi5KIZlE= 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KtK5TniNDv+kZXvXSJ/fi5KIZlE=', 'Content-Length': '0'} 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f7c7cc32dd42ae6-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KtK5TniNDv+kZXvXSJ/fi5KIZlE= 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KtK5TniNDv+kZXvXSJ/fi5KIZlE=', 'Content-Length': '0'} 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dc74625f4810f5c-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj31 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31 2024-04-20T01:55:21.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.485 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8= 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8=', 'Content-Length': '0'} 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b189937cb83fd92-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj31?versionId=null 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8= 2024-04-20T01:55:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gsuasQeDvectZw/h8jNGljACvG8=', 'Content-Length': '0'} 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000568a36ebbe100395-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj31 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj31 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj31 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d568379604953ebf-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj311000falseobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj31 2024-04-20T01:55:21.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj31 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj31 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.490 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb703a2cfc399ba4-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj311000falseobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj31 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj31 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:0acb4s3hT61LXkPKQSHr97UHlyg= 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0acb4s3hT61LXkPKQSHr97UHlyg=', 'Content-Length': '0'} 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006713d0926f76f3d2-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31 2024-04-20T01:55:21.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj31 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.493 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj31 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:0acb4s3hT61LXkPKQSHr97UHlyg= 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0acb4s3hT61LXkPKQSHr97UHlyg=', 'Content-Length': '0'} 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bda11d2c42830410-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj31 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj31 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj31 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Z6aIj4PFM+iih7iSsQQ4M+XAbOc= 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z6aIj4PFM+iih7iSsQQ4M+XAbOc=', 'Content-Length': '0'} 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c379b53148e9cc28-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj31 2024-04-20T01:55:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj31 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj31 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.496 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj31 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Z6aIj4PFM+iih7iSsQQ4M+XAbOc= 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z6aIj4PFM+iih7iSsQQ4M+XAbOc=', 'Content-Length': '0'} 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b025778f9705bfd9-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj31 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj31 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj31 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026e57ce3ad3f2f74-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj311000falseobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj31 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj31 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj31 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.499 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003b2f481a4d93760-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj311000falseobj312024-04-19T23:26:13.419Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj32 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32 2024-04-20T01:55:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.501 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE= 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE=', 'Content-Length': '0'} 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003072ec9c504d2fbb-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE= 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE=', 'Content-Length': '0'} 2024-04-20T01:55:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be4b12f5ec6a2d7d-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.504 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:wqk4Wh3f2iaJUyPI0x8nlatsfb4= 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wqk4Wh3f2iaJUyPI0x8nlatsfb4=', 'Content-Length': '0'} 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9057cad8a6641bf-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:wqk4Wh3f2iaJUyPI0x8nlatsfb4= 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wqk4Wh3f2iaJUyPI0x8nlatsfb4=', 'Content-Length': '0'} 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c87dae9244ad0da-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj32 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE= 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE=', 'Content-Length': '0'} 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfb4d1ee31b53065-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32 2024-04-20T01:55:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.509 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj32?versionId=null 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE= 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sTcAeZoj34tzCWfSG/QiZaJXQSE=', 'Content-Length': '0'} 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085a53b6baf2d8433-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj32 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj32 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj32 2024-04-20T01:55:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3b337d8476505a9-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj321000falseobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj32 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj32 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj32 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060bb425549145708-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj321000falseobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32 2024-04-20T01:55:21.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj32 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.514 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj32 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZykiHAQAFoJjhK/R7fgSYYDlSw0= 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZykiHAQAFoJjhK/R7fgSYYDlSw0=', 'Content-Length': '0'} 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b39c9ef5109f93f-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj32 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj32 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZykiHAQAFoJjhK/R7fgSYYDlSw0= 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZykiHAQAFoJjhK/R7fgSYYDlSw0=', 'Content-Length': '0'} 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000943174b8a5088a61-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj32 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32 2024-04-20T01:55:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj32 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj32 2024-04-20T01:55:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ik74UYDwT5VQS665Fux86sJkqIQ= 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ik74UYDwT5VQS665Fux86sJkqIQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fe4b9dac3392cab-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj32 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj32 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj32 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj32 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ik74UYDwT5VQS665Fux86sJkqIQ= 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ik74UYDwT5VQS665Fux86sJkqIQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c1a7f202eb7c5f7-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj32 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj32 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj32 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.520 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9e761c6a3ede467-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj321000falseobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj32 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj32 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj32 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b113af14c24b1de5-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj321000falseobj322024-04-19T23:26:13.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI= 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI=', 'Content-Length': '0'} 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f9bb4c530846832-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33 2024-04-20T01:55:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.525 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI= 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI=', 'Content-Length': '0'} 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081eba2b5a26239c5-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Likvd2hK7rp6bkk94ncOxfSdkbE= 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Likvd2hK7rp6bkk94ncOxfSdkbE=', 'Content-Length': '0'} 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e741147b2663e4b-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.528 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Likvd2hK7rp6bkk94ncOxfSdkbE= 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Likvd2hK7rp6bkk94ncOxfSdkbE=', 'Content-Length': '0'} 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d6cb96964505de4-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI= 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI=', 'Content-Length': '0'} 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000072ba07265b7419a-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj33?versionId=null 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI= 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KlAgUcki7wwlrRrXZ1x+eS82cWI=', 'Content-Length': '0'} 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d1b436c7d62b46d-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj33 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj33 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj33 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.533 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090ed0da9b2860fb1-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj331000falseobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj33 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj33 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj33 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.535 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dac6439131e66d74-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj331000falseobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj33 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj33 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nCUGKwm4j/5vTHxonKFKXN9+uCM= 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nCUGKwm4j/5vTHxonKFKXN9+uCM=', 'Content-Length': '0'} 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e083b6d14ca383d7-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj33 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj33 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.538 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nCUGKwm4j/5vTHxonKFKXN9+uCM= 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nCUGKwm4j/5vTHxonKFKXN9+uCM=', 'Content-Length': '0'} 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090bc2c8921315a9a-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj33 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj33 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:o2CJUTwM24L8Oq3lw5XY/0enNHU= 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o2CJUTwM24L8Oq3lw5XY/0enNHU=', 'Content-Length': '0'} 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b39be1222f463f3f-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj33 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj33 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj33 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj33 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.541 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:o2CJUTwM24L8Oq3lw5XY/0enNHU= 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o2CJUTwM24L8Oq3lw5XY/0enNHU=', 'Content-Length': '0'} 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9385fbf434e2b91-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj33 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj33 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj33 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008ec3684742dda9a-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj331000falseobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj33 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj33 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj33 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.544 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f46726d4eeb40566-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj331000falseobj332024-04-19T23:26:13.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0= 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0=', 'Content-Length': '0'} 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031abac1c34b2ca56-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0= 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0=', 'Content-Length': '0'} 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e2bee4413a20907-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.549 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:k6LTzXxcaK3GNMYQjGNhXBwBes4= 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k6LTzXxcaK3GNMYQjGNhXBwBes4=', 'Content-Length': '0'} 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c27452c9bc0062b1-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:k6LTzXxcaK3GNMYQjGNhXBwBes4= 2024-04-20T01:55:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:k6LTzXxcaK3GNMYQjGNhXBwBes4=', 'Content-Length': '0'} 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce5ce4b7afe767b4-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.553 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.553 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.553 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.553 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.553 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.555 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0= 2024-04-20T01:55:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0=', 'Content-Length': '0'} 2024-04-20T01:55:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092b3bab6e081ffeb-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34 2024-04-20T01:55:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34 2024-04-20T01:55:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj34?versionId=null 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.556 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0= 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LKHXEohN7cpvr7sgUigebt+y+Z0=', 'Content-Length': '0'} 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001089572b8f4ec7af-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj34 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj34 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj34 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049fdf48c578da6f5-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj341000falseobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj34 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj34 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj34 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.559 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006059397b82367b45-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj341000falseobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj34 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj34 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KNLYOGxyfURCYdY0b8btpDiskcw= 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KNLYOGxyfURCYdY0b8btpDiskcw=', 'Content-Length': '0'} 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000565e78694eee2a2f-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj34 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.562 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj34 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KNLYOGxyfURCYdY0b8btpDiskcw= 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KNLYOGxyfURCYdY0b8btpDiskcw=', 'Content-Length': '0'} 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee9a3791e3078d15-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj34 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj34 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:wg0U/0Qy2oBzNuV3vjZfw/U9ijA= 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wg0U/0Qy2oBzNuV3vjZfw/U9ijA=', 'Content-Length': '0'} 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016c576c420daf9a9-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj34 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj34 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj34 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.565 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj34 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:wg0U/0Qy2oBzNuV3vjZfw/U9ijA= 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wg0U/0Qy2oBzNuV3vjZfw/U9ijA=', 'Content-Length': '0'} 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c9aa85b48db89e1-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj34 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj34 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj34 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010330d1f4e793bdf-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj341000falseobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj34 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj34 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj34 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004042a16a2bdd2f26-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj341000falseobj342024-04-19T23:26:13.492Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj35 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35 2024-04-20T01:55:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.570 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA= 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA=', 'Content-Length': '0'} 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1ca6652f6e0d08c-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA= 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA=', 'Content-Length': '0'} 2024-04-20T01:55:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090665b171e35794e-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35 2024-04-20T01:55:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35 2024-04-20T01:55:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QytyXqBo9BN4nxnHFScQ9vl3p9I= 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QytyXqBo9BN4nxnHFScQ9vl3p9I=', 'Content-Length': '0'} 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d460dab2f3ec3b13-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.575 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QytyXqBo9BN4nxnHFScQ9vl3p9I= 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QytyXqBo9BN4nxnHFScQ9vl3p9I=', 'Content-Length': '0'} 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7b7214b90fcb014-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj35 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA= 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA=', 'Content-Length': '0'} 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f67fe97e3579090c-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.578 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.579 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.579 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.579 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.579 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj35?versionId=null 2024-04-20T01:55:21.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.579 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA= 2024-04-20T01:55:21.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kYC4/ZczRdhJr/zpLuca7cV1YtA=', 'Content-Length': '0'} 2024-04-20T01:55:21.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000498f15a17b71861-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj35 2024-04-20T01:55:21.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj35 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj35 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.580 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077de279d547ecf39-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj351000falseobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj35 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj35 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj35 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004457ce372fe90d98-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj351000falseobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj35 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.583 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj35 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:n6aSx7JXg/+fIa4gYHUL9aJ1I5s= 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:n6aSx7JXg/+fIa4gYHUL9aJ1I5s=', 'Content-Length': '0'} 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b283a17575ae90c3-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj35 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj35 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:n6aSx7JXg/+fIa4gYHUL9aJ1I5s= 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:n6aSx7JXg/+fIa4gYHUL9aJ1I5s=', 'Content-Length': '0'} 2024-04-20T01:55:21.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a54ca095fe55e41a-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj35 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj35 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.586 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj35 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AmOun2s389rU3j1gilJUHKoZGbA= 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AmOun2s389rU3j1gilJUHKoZGbA=', 'Content-Length': '0'} 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7cfa6dae46c1ca1-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj35 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj35 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj35 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj35 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AmOun2s389rU3j1gilJUHKoZGbA= 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AmOun2s389rU3j1gilJUHKoZGbA=', 'Content-Length': '0'} 2024-04-20T01:55:21.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0e68498988026d1-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj35 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj35 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj35 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.589 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089cfea7c3b1994be-006622fdb0-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj351000falseobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj35 2024-04-20T01:55:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj35 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj35 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:40 GMT 2024-04-20T01:55:21.591 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk= 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KchV0xVmOr9A93h2mPKnJBfOPdk=', 'Content-Length': '0'} 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eb02fa96751d436-006622fdb0-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj351000falseobj352024-04-19T23:26:13.547Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj36 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0= 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0=', 'Content-Length': '0'} 2024-04-20T01:55:21.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034bd8d293e07c3e4-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0= 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0=', 'Content-Length': '0'} 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000237cdae0a618ce17-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tN6fe4rQr+5rHkZP5UwBxUKqFyI= 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tN6fe4rQr+5rHkZP5UwBxUKqFyI=', 'Content-Length': '0'} 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3cfc3976b2827fc-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tN6fe4rQr+5rHkZP5UwBxUKqFyI= 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tN6fe4rQr+5rHkZP5UwBxUKqFyI=', 'Content-Length': '0'} 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009514a1bca6bc9a7-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj36 2024-04-20T01:55:21.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.599 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0= 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0=', 'Content-Length': '0'} 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f109ccc7b0b60b8-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj36?versionId=null 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0= 2024-04-20T01:55:21.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bDWU1j2V/8y7ltJKKmVoc+hOYi0=', 'Content-Length': '0'} 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d63b6a54d499a7ce-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj36 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj36 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj36 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2922c6aeafe5c49-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj361000falseobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj36 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj36 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj36 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.604 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7347d9a51752745-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj361000falseobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj36 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj36 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dLf0AhMr014+dm+83tkZkdJnlk0= 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dLf0AhMr014+dm+83tkZkdJnlk0=', 'Content-Length': '0'} 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fd666940a791dbd-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj36 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.607 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj36 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dLf0AhMr014+dm+83tkZkdJnlk0= 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dLf0AhMr014+dm+83tkZkdJnlk0=', 'Content-Length': '0'} 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a1670f0a6496c14-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj36 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj36 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj36 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3VKMbRYqwU6HhtoTdwvo3B1LW7M= 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3VKMbRYqwU6HhtoTdwvo3B1LW7M=', 'Content-Length': '0'} 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005928f09638f3f10e-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj36 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj36 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj36 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.610 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj36 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3VKMbRYqwU6HhtoTdwvo3B1LW7M= 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3VKMbRYqwU6HhtoTdwvo3B1LW7M=', 'Content-Length': '0'} 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f049c2b334183ab-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj36 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj36 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj36 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b3fec427662ad19-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj361000falseobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj36 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj36 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj36 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.613 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037a6a8a65e7ab002-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj361000falseobj362024-04-19T23:26:13.556Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj37 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37 2024-04-20T01:55:21.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.615 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0= 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0=', 'Content-Length': '0'} 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000564fc71c45f482de-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0= 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0=', 'Content-Length': '0'} 2024-04-20T01:55:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b431c59d8e8efb0-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.618 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:K6oDwdvPzN4jhQX5lvfdEyWtxTI= 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K6oDwdvPzN4jhQX5lvfdEyWtxTI=', 'Content-Length': '0'} 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f76e3596eb4b9295-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.620 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:K6oDwdvPzN4jhQX5lvfdEyWtxTI= 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:K6oDwdvPzN4jhQX5lvfdEyWtxTI=', 'Content-Length': '0'} 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bc93d67c298b8af-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj37 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0= 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0=', 'Content-Length': '0'} 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca09dc9c72c13a77-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37 2024-04-20T01:55:21.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.623 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj37?versionId=null 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0= 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ewO7tG8mSjhWlEODwHuJu4CV3+0=', 'Content-Length': '0'} 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f98751ab5fec3029-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj37 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj37 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj37 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000015615b1effc5cd3-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj371000falseobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj37 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj37 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj37 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ef8dda4ad5d9d3e-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj371000falseobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37 2024-04-20T01:55:21.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj37 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj37 2024-04-20T01:55:21.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XlV5PnxnXqBMuScnZhB2+Fzdpik= 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XlV5PnxnXqBMuScnZhB2+Fzdpik=', 'Content-Length': '0'} 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c6fd09e30153133-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj37 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj37 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XlV5PnxnXqBMuScnZhB2+Fzdpik= 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XlV5PnxnXqBMuScnZhB2+Fzdpik=', 'Content-Length': '0'} 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050b6603aa7f719cb-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj37 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37 2024-04-20T01:55:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj37 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj37 2024-04-20T01:55:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:x5IQvbUZIZ9kFDN7OdLzAyHMBaA= 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x5IQvbUZIZ9kFDN7OdLzAyHMBaA=', 'Content-Length': '0'} 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca82b3eeebdeb04c-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj37 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj37 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj37 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj37 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:x5IQvbUZIZ9kFDN7OdLzAyHMBaA= 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x5IQvbUZIZ9kFDN7OdLzAyHMBaA=', 'Content-Length': '0'} 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f95e6a5165d2774-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj37 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj37 2024-04-20T01:55:21.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj37 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.634 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d52be11d12f8d86-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj371000falseobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj37 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj37 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj37 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cbbc79d456e5faf-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj371000falseobj372024-04-19T23:26:13.607Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj38 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM= 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM=', 'Content-Length': '0'} 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000697513f3650f134d-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.639 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM= 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM=', 'Content-Length': '0'} 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5bda94716750b21-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XZuwX8PSqyPEVtEkwBkZmc1BmzA= 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XZuwX8PSqyPEVtEkwBkZmc1BmzA=', 'Content-Length': '0'} 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000449f06d43677c243-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.642 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XZuwX8PSqyPEVtEkwBkZmc1BmzA= 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XZuwX8PSqyPEVtEkwBkZmc1BmzA=', 'Content-Length': '0'} 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be23a9713606df88-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj38 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.644 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM= 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM=', 'Content-Length': '0'} 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8be4dea28e3fdea-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj38?versionId=null 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM= 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l84ABgncJ/Hdz64bNBb+gj7VfvM=', 'Content-Length': '0'} 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c0d81f485691331-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj38 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj38 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj38 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.647 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4d70369ece99509-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj381000falseobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj38 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj38 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj38 2024-04-20T01:55:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.649 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037c7a13a473646ec-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj381000falseobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj38 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj38 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hLv6OF+8hRhOpGalwHZ1ZwMRNCA= 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hLv6OF+8hRhOpGalwHZ1ZwMRNCA=', 'Content-Length': '0'} 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2f9101b4f3fd148-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj38 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj38 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.652 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hLv6OF+8hRhOpGalwHZ1ZwMRNCA= 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hLv6OF+8hRhOpGalwHZ1ZwMRNCA=', 'Content-Length': '0'} 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d8cd7981d09bb8b-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj38 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj38 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj38 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:IQx/tf1NQ1MlyXoBsLeEYTCQS30= 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IQx/tf1NQ1MlyXoBsLeEYTCQS30=', 'Content-Length': '0'} 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ccf852d85242d72-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj38 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj38 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj38 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj38 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.655 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:IQx/tf1NQ1MlyXoBsLeEYTCQS30= 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IQx/tf1NQ1MlyXoBsLeEYTCQS30=', 'Content-Length': '0'} 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e3cc8e5d8612bbe-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj38 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj38 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj38 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004b361aaa6d4c141-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj381000falseobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj38 2024-04-20T01:55:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj38 2024-04-20T01:55:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj38 2024-04-20T01:55:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.658 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.659 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.659 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.659 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.659 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.659 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008580cf57815a9ac-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj381000falseobj382024-04-19T23:26:13.616Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.659 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj39 2024-04-20T01:55:21.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39 2024-04-20T01:55:21.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.660 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ= 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd14a080361af5c8-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ= 2024-04-20T01:55:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff5b2bcdf7c8eb9c-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LICDCeCJN6hsUn2y/KSfze85lgA= 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LICDCeCJN6hsUn2y/KSfze85lgA=', 'Content-Length': '0'} 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008532d74584dd1ec2-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.665 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LICDCeCJN6hsUn2y/KSfze85lgA= 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LICDCeCJN6hsUn2y/KSfze85lgA=', 'Content-Length': '0'} 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe90dbedb66e340d-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj39 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ= 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2a40f8df764dd10-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.668 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj39?versionId=null 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ= 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3FQtoUFblHP1s8fIT5IRs0tXHUQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fed09aae678a502-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj39 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj39 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj39 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.670 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d4fa27d792bf2f6-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj391000falseobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj39 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj39 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj39 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000358d867c66bb4fb7-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj391000falseobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39 2024-04-20T01:55:21.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj39 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.673 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj39 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tGUQWo75glRYv9UAklIrrhNeSUU= 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGUQWo75glRYv9UAklIrrhNeSUU=', 'Content-Length': '0'} 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bd583cfce2be853-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj39 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj39 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tGUQWo75glRYv9UAklIrrhNeSUU= 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGUQWo75glRYv9UAklIrrhNeSUU=', 'Content-Length': '0'} 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011676e39754d9c9f-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj39 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39 2024-04-20T01:55:21.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj39 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj39 2024-04-20T01:55:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:03aKqdiCeGD34jUoqUeQ0/zu+ps= 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:03aKqdiCeGD34jUoqUeQ0/zu+ps=', 'Content-Length': '0'} 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068c1c84b71843c05-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj39 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj39 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj39 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.678 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.678 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.678 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.678 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.678 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj39 2024-04-20T01:55:21.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.678 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:03aKqdiCeGD34jUoqUeQ0/zu+ps= 2024-04-20T01:55:21.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:03aKqdiCeGD34jUoqUeQ0/zu+ps=', 'Content-Length': '0'} 2024-04-20T01:55:21.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcc08fed1953f88c-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj39 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj39 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj39 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.679 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cdf163700dcfd59-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj391000falseobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj39 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj39 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj39 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8531f1130ef67c2-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj391000falseobj392024-04-19T23:26:13.667Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj4 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0= 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0=', 'Content-Length': '0'} 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a833ef7fb2943313-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4 2024-04-20T01:55:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.684 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0= 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0=', 'Content-Length': '0'} 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f63fd7a3d9d56c2-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HSUMhs6NTyS2N4jhNr1uLu5y43o= 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HSUMhs6NTyS2N4jhNr1uLu5y43o=', 'Content-Length': '0'} 2024-04-20T01:55:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0f647032d8ad92a-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.687 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HSUMhs6NTyS2N4jhNr1uLu5y43o= 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HSUMhs6NTyS2N4jhNr1uLu5y43o=', 'Content-Length': '0'} 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070e94dfa865975f8-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj4 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0= 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0=', 'Content-Length': '0'} 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001094413ba83ca05a-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj4?versionId=null 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0= 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:7odXOROyGpgSJ3I7cH0JySDQoS0=', 'Content-Length': '0'} 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005811f662a647428d-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj4 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj4 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj4 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.692 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000147409facaa6aa2f-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj41000falseobj42024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj4 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj4 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj4 2024-04-20T01:55:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.694 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097a4a55896367c2b-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj41000falseobj42024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj4 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj4 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:DTzQF6tNillgL7wYNLDuO76Ov9w= 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DTzQF6tNillgL7wYNLDuO76Ov9w=', 'Content-Length': '0'} 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ad3b4afd6aa111b-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj4 2024-04-20T01:55:21.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj4 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:DTzQF6tNillgL7wYNLDuO76Ov9w= 2024-04-20T01:55:21.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DTzQF6tNillgL7wYNLDuO76Ov9w=', 'Content-Length': '0'} 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ee959a67855d7a3-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj4 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj4 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj4 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:0a9JXPqLwgOS0y6Ci6Q7LOB3/no= 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0a9JXPqLwgOS0y6Ci6Q7LOB3/no=', 'Content-Length': '0'} 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014f4311483b40491-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj4 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj4 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj4 2024-04-20T01:55:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj4 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:0a9JXPqLwgOS0y6Ci6Q7LOB3/no= 2024-04-20T01:55:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0a9JXPqLwgOS0y6Ci6Q7LOB3/no=', 'Content-Length': '0'} 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000271c6866d7a34c8a-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj4 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj4 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj4 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ad01d019f0280dc-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj41000falseobj42024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj4 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj4 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj4 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.703 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb246f983da71f53-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj41000falseobj42024-04-19T23:26:12.641Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj40 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.705 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA= 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA=', 'Content-Length': '0'} 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001609f35a7667ffea-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA= 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA=', 'Content-Length': '0'} 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf60424c13c3211b-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40 2024-04-20T01:55:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.708 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:keYTqHzUCbUBjZWQvwNDq96nCGM= 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:keYTqHzUCbUBjZWQvwNDq96nCGM=', 'Content-Length': '0'} 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a87d48f2d093cd8-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:keYTqHzUCbUBjZWQvwNDq96nCGM= 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:keYTqHzUCbUBjZWQvwNDq96nCGM=', 'Content-Length': '0'} 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040acb52c92a36d4c-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.710 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj40 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.711 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA= 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA=', 'Content-Length': '0'} 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c749952ec29f47d8-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj40?versionId=null 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.713 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA= 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MFFje9eGPktoNGboPc6q+pEqnEA=', 'Content-Length': '0'} 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094707d317896e89c-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj40 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj40 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj40 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000338e70b7cadb755e-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj401000falseobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj40 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj40 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj40 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.716 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008730f0c59f5cffcd-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj401000falseobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj40 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj40 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:p8jeU/5J8oBCkTE3uE56bemHvVU= 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:p8jeU/5J8oBCkTE3uE56bemHvVU=', 'Content-Length': '0'} 2024-04-20T01:55:21.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095008dd55abfa498-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj40 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.719 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj40 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:p8jeU/5J8oBCkTE3uE56bemHvVU= 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:p8jeU/5J8oBCkTE3uE56bemHvVU=', 'Content-Length': '0'} 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b1243508bb9f3c6-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj40 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj40 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj40 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EknWK+Xky9UlBG9NtHCIQmxkSqk= 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EknWK+Xky9UlBG9NtHCIQmxkSqk=', 'Content-Length': '0'} 2024-04-20T01:55:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ab6bcad9b61ac7a-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj40 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj40 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj40 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.722 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj40 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EknWK+Xky9UlBG9NtHCIQmxkSqk= 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EknWK+Xky9UlBG9NtHCIQmxkSqk=', 'Content-Length': '0'} 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b533d05ee49d9f84-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj40 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj40 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj40 2024-04-20T01:55:21.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006cee459d6a81700-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj401000falseobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj40 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj40 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj40 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000794eaac5d22273e8-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj401000falseobj402024-04-19T23:26:13.677Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj41 2024-04-20T01:55:21.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.727 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4= 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4=', 'Content-Length': '0'} 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d222b200ff215f52-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.729 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4= 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4=', 'Content-Length': '0'} 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c144a46a7dfd5d04-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rSOHZWGdgjhbfnt2sllqY9vfCek= 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rSOHZWGdgjhbfnt2sllqY9vfCek=', 'Content-Length': '0'} 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076bcfe9e88ee775a-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41 2024-04-20T01:55:21.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.732 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rSOHZWGdgjhbfnt2sllqY9vfCek= 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rSOHZWGdgjhbfnt2sllqY9vfCek=', 'Content-Length': '0'} 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030aeb4d2f908f339-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj41 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4= 2024-04-20T01:55:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4=', 'Content-Length': '0'} 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d96b290caf082f9-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj41?versionId=null 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4= 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:poMM9a0Ftky8PJKxAdTlek9K9C4=', 'Content-Length': '0'} 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008671d36e9c1afee3-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj41 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj41 2024-04-20T01:55:21.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj41 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ae9c7b668e22d8-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj411000falseobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj41 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj41 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj41 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096a0bbfe4aabf91d-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj411000falseobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj41 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.740 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj41 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:vSgTRPFrdVZeygwMeBzQO1XIXho= 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vSgTRPFrdVZeygwMeBzQO1XIXho=', 'Content-Length': '0'} 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a778d7d73f64221e-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj41 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj41 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:vSgTRPFrdVZeygwMeBzQO1XIXho= 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vSgTRPFrdVZeygwMeBzQO1XIXho=', 'Content-Length': '0'} 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027f8f125715c91a3-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.742 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj41 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj41 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.743 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj41 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:OW5WcKJu3+D4W+w4FB7GOOn935M= 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OW5WcKJu3+D4W+w4FB7GOOn935M=', 'Content-Length': '0'} 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ea6791d6c1b78e4-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj41 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj41 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj41 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj41 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:OW5WcKJu3+D4W+w4FB7GOOn935M= 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OW5WcKJu3+D4W+w4FB7GOOn935M=', 'Content-Length': '0'} 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000634b10cb9d4cab6a-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj41 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj41 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj41 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.746 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073c0dc0d265c01bb-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj411000falseobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj41 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj41 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj41 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.748 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6897833fc42130b-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj411000falseobj412024-04-19T23:26:13.730Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj42 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s= 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s=', 'Content-Length': '0'} 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000922af8d1bdc1c42d-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.751 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s= 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s=', 'Content-Length': '0'} 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a3b7bfae5a5a0e3-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.753 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ra/uats+1ZnOz9YNrhNu3U/t8I8= 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ra/uats+1ZnOz9YNrhNu3U/t8I8=', 'Content-Length': '0'} 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a414936df569b11-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.755 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.755 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.755 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.755 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.755 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.755 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ra/uats+1ZnOz9YNrhNu3U/t8I8= 2024-04-20T01:55:21.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ra/uats+1ZnOz9YNrhNu3U/t8I8=', 'Content-Length': '0'} 2024-04-20T01:55:21.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a92f0799f89b73d-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.755 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj42 2024-04-20T01:55:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42 2024-04-20T01:55:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42 2024-04-20T01:55:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s= 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s=', 'Content-Length': '0'} 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0c79c8a7ed1862d-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42 2024-04-20T01:55:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.760 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj42?versionId=null 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s= 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TxBf5Kzn/6zmD9p/11oeYehk+3s=', 'Content-Length': '0'} 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a22c66ad9967863-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj42 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj42 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj42 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab45c0103387eec-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj421000falseobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj42 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj42 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj42 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.763 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020cfa761beadeba2-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj421000falseobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj42 2024-04-20T01:55:21.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj42 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Pxi+FiXd4nteQYhW+G2wcqt7aP4= 2024-04-20T01:55:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Pxi+FiXd4nteQYhW+G2wcqt7aP4=', 'Content-Length': '0'} 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000322f2b2da4bb7a6c-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj42 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.766 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj42 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Pxi+FiXd4nteQYhW+G2wcqt7aP4= 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Pxi+FiXd4nteQYhW+G2wcqt7aP4=', 'Content-Length': '0'} 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f101cb1d157856ba-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj42 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj42 2024-04-20T01:55:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj42 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GJPzOPitAfDJEKzKQ7/CJGjB7KQ= 2024-04-20T01:55:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GJPzOPitAfDJEKzKQ7/CJGjB7KQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cba9919f248bd65a-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj42 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj42 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj42 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.769 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj42 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GJPzOPitAfDJEKzKQ7/CJGjB7KQ= 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GJPzOPitAfDJEKzKQ7/CJGjB7KQ=', 'Content-Length': '0'} 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bac2253e55308c52-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj42 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj42 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj42 2024-04-20T01:55:21.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c144f99829aed52-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj421000falseobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj42 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj42 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj42 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd1ce911f5a36af7-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj421000falseobj422024-04-19T23:26:13.740Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj43 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43 2024-04-20T01:55:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.774 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak= 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak=', 'Content-Length': '0'} 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064f4122a93daaf94-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak= 2024-04-20T01:55:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak=', 'Content-Length': '0'} 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001dd93ae6c9efddb-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fk96L56YigQkhG2Aroo+77vxbqo= 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fk96L56YigQkhG2Aroo+77vxbqo=', 'Content-Length': '0'} 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7437e2f83179166-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fk96L56YigQkhG2Aroo+77vxbqo= 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fk96L56YigQkhG2Aroo+77vxbqo=', 'Content-Length': '0'} 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011bd85f19c53a027-006622fdb1-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj43 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak= 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak=', 'Content-Length': '0'} 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091319d63dbe40b61-006622fdb1-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43 2024-04-20T01:55:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.782 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj43?versionId=null 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak= 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+aAh6kttqob0TfyfzXtpZOs6Lak=', 'Content-Length': '0'} 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f40cbff711f7a88-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj43 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj43 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj43 2024-04-20T01:55:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:41 GMT 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE= 2024-04-20T01:55:21.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qaOWHeC6YKYmaUJ/xSBikkUqqFE=', 'Content-Length': '0'} 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d03a9ba42eae56a-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj431000falseobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj43 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj43 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj43 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beef9ef74f9f6b15-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj431000falseobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43 2024-04-20T01:55:21.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj43 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj43 2024-04-20T01:55:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZwLhwpBmvtlavpfEqsph0RwnAjY= 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZwLhwpBmvtlavpfEqsph0RwnAjY=', 'Content-Length': '0'} 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfe0430759239c62-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj43 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj43 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ZwLhwpBmvtlavpfEqsph0RwnAjY= 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ZwLhwpBmvtlavpfEqsph0RwnAjY=', 'Content-Length': '0'} 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c232b2e594367e86-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj43 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj43 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj43 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.790 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:0o9p0D30IuytxplDVzrc7Vm5als= 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0o9p0D30IuytxplDVzrc7Vm5als=', 'Content-Length': '0'} 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b20615039285a14e-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj43 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj43 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj43 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.791 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj43 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:0o9p0D30IuytxplDVzrc7Vm5als= 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0o9p0D30IuytxplDVzrc7Vm5als=', 'Content-Length': '0'} 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000997a166cd6c6ab7a-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj43 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj43 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj43 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.793 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7abaa9c4fefde44-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj431000falseobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj43 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj43 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj43 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e58523bef9cacfc6-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj431000falseobj432024-04-19T23:26:13.791Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.796 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA= 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA=', 'Content-Length': '0'} 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ced12870502aa6a6-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA= 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA=', 'Content-Length': '0'} 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2ddb6956888b07f-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Qhc+M3g+f533oMy/asvV+KRpHjc= 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Qhc+M3g+f533oMy/asvV+KRpHjc=', 'Content-Length': '0'} 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9c2d47fe512bf53-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.801 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Qhc+M3g+f533oMy/asvV+KRpHjc= 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Qhc+M3g+f533oMy/asvV+KRpHjc=', 'Content-Length': '0'} 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bed961fbe7fd17cb-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA= 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA=', 'Content-Length': '0'} 2024-04-20T01:55:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3d544383d91b98c-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.804 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj44?versionId=null 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA= 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FdhL1SATtUnxyxAtlkv9XZccwdA=', 'Content-Length': '0'} 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b650f35e200ce6c-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj44 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj44 2024-04-20T01:55:21.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj44 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095882cce684d041d-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj441000falseobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj44 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj44 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj44 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001780cfe6359f65e3-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj441000falseobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj44 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.809 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj44 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:DIGLAkFU51o24FhZlFy4U6gOBoU= 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DIGLAkFU51o24FhZlFy4U6gOBoU=', 'Content-Length': '0'} 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a34fd2778dc8b1bf-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj44 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj44 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:DIGLAkFU51o24FhZlFy4U6gOBoU= 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DIGLAkFU51o24FhZlFy4U6gOBoU=', 'Content-Length': '0'} 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bebed758cb5f9b92-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-20T01:55:21.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj44 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.812 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj44 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:SjA8iq123dcWVepe64XIr3saFIA= 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SjA8iq123dcWVepe64XIr3saFIA=', 'Content-Length': '0'} 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c7ad6a2b757e364-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj44 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj44 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj44 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj44 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:SjA8iq123dcWVepe64XIr3saFIA= 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SjA8iq123dcWVepe64XIr3saFIA=', 'Content-Length': '0'} 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d86086fdf0fe6ddb-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj44 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj44 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj44 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.815 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f803dd832b7c230c-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj441000falseobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj44 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj44 2024-04-20T01:55:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj44 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebece6c8ec97553c-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj441000falseobj442024-04-19T23:26:13.801Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj45 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0= 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0=', 'Content-Length': '0'} 2024-04-20T01:55:21.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071d9652a279d7929-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0= 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0=', 'Content-Length': '0'} 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b2bf5013916af95-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zmmZypW4JBfLqspFvAQZyLugX48= 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zmmZypW4JBfLqspFvAQZyLugX48=', 'Content-Length': '0'} 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e2bd7dd0823d3a9-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zmmZypW4JBfLqspFvAQZyLugX48= 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zmmZypW4JBfLqspFvAQZyLugX48=', 'Content-Length': '0'} 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0f37d0cbf63960c-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj45 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45 2024-04-20T01:55:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.825 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.826 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.826 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.826 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0= 2024-04-20T01:55:21.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0=', 'Content-Length': '0'} 2024-04-20T01:55:21.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5a1e6f239fe55a1-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45 2024-04-20T01:55:21.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45 2024-04-20T01:55:21.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj45?versionId=null 2024-04-20T01:55:21.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0= 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VU/pVaTrbNBM0Mb2JKgVRHEFwq0=', 'Content-Length': '0'} 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e410585349efbd0f-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj45 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj45 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj45 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064ac8960642d8541-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj451000falseobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj45 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj45 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj45 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.830 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c088d5db2d003848-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj451000falseobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj45 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj45 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:bmwdV+rEIAbHM/+ysoHD041OCUw= 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bmwdV+rEIAbHM/+ysoHD041OCUw=', 'Content-Length': '0'} 2024-04-20T01:55:21.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000515c46cebce9bee3-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj45 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.833 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj45 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:bmwdV+rEIAbHM/+ysoHD041OCUw= 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bmwdV+rEIAbHM/+ysoHD041OCUw=', 'Content-Length': '0'} 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038d7ce295ce19219-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj45 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj45 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj45 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/9En4I36Ej589rqc7/gxX19MVzg= 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/9En4I36Ej589rqc7/gxX19MVzg=', 'Content-Length': '0'} 2024-04-20T01:55:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050969d20ddf2a7cc-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj45 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj45 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj45 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.836 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj45 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/9En4I36Ej589rqc7/gxX19MVzg= 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/9En4I36Ej589rqc7/gxX19MVzg=', 'Content-Length': '0'} 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ffaa6f101e445f3-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj45 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj45 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj45 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009344a159b069d2f8-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj451000falseobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj45 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj45 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj45 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009836123faafa6e2c-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj451000falseobj452024-04-19T23:26:13.851Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj46 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46 2024-04-20T01:55:21.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.841 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY= 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY=', 'Content-Length': '0'} 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b94bda5ce0ad8123-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY= 2024-04-20T01:55:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY=', 'Content-Length': '0'} 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adc75fb8c19fd445-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zAZD3tzKTHrXZZB9iFO8J+9lvpc= 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zAZD3tzKTHrXZZB9iFO8J+9lvpc=', 'Content-Length': '0'} 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e588260284d201cc-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.846 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zAZD3tzKTHrXZZB9iFO8J+9lvpc= 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zAZD3tzKTHrXZZB9iFO8J+9lvpc=', 'Content-Length': '0'} 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa5a98a89a511622-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj46 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY= 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY=', 'Content-Length': '0'} 2024-04-20T01:55:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036c38b2364ba1e3b-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.849 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj46?versionId=null 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY= 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VNt1AbY1q/EXuvHF4Aj4kMZqpdY=', 'Content-Length': '0'} 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4b7f97a61da9378-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj46 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj46 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj46 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.851 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530e6b43630fe7cd-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj461000falseobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj46 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj46 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj46 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000031db4999a0105b7-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj461000falseobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj46 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.854 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj46 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:CvzPswguHImVzOaiXWqiQmv0kFU= 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CvzPswguHImVzOaiXWqiQmv0kFU=', 'Content-Length': '0'} 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a68e1458d997464-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj46 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj46 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:CvzPswguHImVzOaiXWqiQmv0kFU= 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CvzPswguHImVzOaiXWqiQmv0kFU=', 'Content-Length': '0'} 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000460dbb1dc6b97d7a-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj46 2024-04-20T01:55:21.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj46 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.857 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj46 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QT0tAxPcWAul5bzIOuOkXzXg5yo= 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QT0tAxPcWAul5bzIOuOkXzXg5yo=', 'Content-Length': '0'} 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082ada89ba37ca108-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj46 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj46 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj46 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj46 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QT0tAxPcWAul5bzIOuOkXzXg5yo= 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QT0tAxPcWAul5bzIOuOkXzXg5yo=', 'Content-Length': '0'} 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eefa112e0b17488f-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj46 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj46 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj46 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.860 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d41927ce2ecab91-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj461000falseobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj46 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj46 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj46 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eccd71cb94a1cc5b-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj461000falseobj462024-04-19T23:26:13.861Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj47 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY= 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY=', 'Content-Length': '0'} 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc11a14722a3a764-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47 2024-04-20T01:55:21.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.865 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY= 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY=', 'Content-Length': '0'} 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070d3f40c13286b71-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sPprbB2bcvgxEyejuDjZugxBERA= 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sPprbB2bcvgxEyejuDjZugxBERA=', 'Content-Length': '0'} 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a08148e10dace5dd-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.868 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sPprbB2bcvgxEyejuDjZugxBERA= 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sPprbB2bcvgxEyejuDjZugxBERA=', 'Content-Length': '0'} 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f679cf61945e49e-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj47 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY= 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY=', 'Content-Length': '0'} 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006947a99e1a84544-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj47?versionId=null 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY= 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KldEaiR4iH9Sd815/XpfDHwKJGY=', 'Content-Length': '0'} 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dea9ad4930cd571-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj47 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj47 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj47 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.873 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad8e005ad53d6a84-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj471000falseobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj47 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj47 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj47 2024-04-20T01:55:21.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee4bbf8624eab54c-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj471000falseobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj47 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj47 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:xJNAZhPhZLDHWLEsg7/bZs0tWsw= 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xJNAZhPhZLDHWLEsg7/bZs0tWsw=', 'Content-Length': '0'} 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001413ea85fec4b2fa-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj47 2024-04-20T01:55:21.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj47 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:xJNAZhPhZLDHWLEsg7/bZs0tWsw= 2024-04-20T01:55:21.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xJNAZhPhZLDHWLEsg7/bZs0tWsw=', 'Content-Length': '0'} 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f008cc73e32e3de1-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj47 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj47 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj47 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3rIXwprxpB/nZlAehMuUPQDWF4w= 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3rIXwprxpB/nZlAehMuUPQDWF4w=', 'Content-Length': '0'} 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b689ac69b708dd11-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj47 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj47 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj47 2024-04-20T01:55:21.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj47 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3rIXwprxpB/nZlAehMuUPQDWF4w= 2024-04-20T01:55:21.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3rIXwprxpB/nZlAehMuUPQDWF4w=', 'Content-Length': '0'} 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b5f3dbec6210459-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj47 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj47 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj47 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000249d2400fed99424-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj471000falseobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj47 2024-04-20T01:55:21.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj47 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj47 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.884 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c0756cabb8b1135-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj471000falseobj472024-04-19T23:26:13.911Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj48 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0= 2024-04-20T01:55:21.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0=', 'Content-Length': '0'} 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5bc8c9f24795e46-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0= 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0=', 'Content-Length': '0'} 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfc412b344421447-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EO9SGck1cNQ37eBxu4ynRBtZXD4= 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EO9SGck1cNQ37eBxu4ynRBtZXD4=', 'Content-Length': '0'} 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d3dc31daeb330ba-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EO9SGck1cNQ37eBxu4ynRBtZXD4= 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EO9SGck1cNQ37eBxu4ynRBtZXD4=', 'Content-Length': '0'} 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093b1388be12c5f81-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj48 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48 2024-04-20T01:55:21.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.892 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0= 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0=', 'Content-Length': '0'} 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024acd86a8cb5b733-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj48?versionId=null 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0= 2024-04-20T01:55:21.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5XA+N7QjXs66PSte9iYScc5bgq0=', 'Content-Length': '0'} 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1355a20268dc3eb-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj48 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj48 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj48 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000342afb78931c1323-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj481000falseobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj48 2024-04-20T01:55:21.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj48 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj48 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.897 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055bf177d12431fae-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj481000falseobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj48 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj48 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ll75/inp3fKPjPJ97vHAirGCEPI= 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ll75/inp3fKPjPJ97vHAirGCEPI=', 'Content-Length': '0'} 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008645cd8b085de073-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48 2024-04-20T01:55:21.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj48 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.900 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj48 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ll75/inp3fKPjPJ97vHAirGCEPI= 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ll75/inp3fKPjPJ97vHAirGCEPI=', 'Content-Length': '0'} 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8c109210f7b40d3-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj48 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj48 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj48 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ynuqLU9p/NIFuaNGKgeL2/UssvI= 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ynuqLU9p/NIFuaNGKgeL2/UssvI=', 'Content-Length': '0'} 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bf8929ca8e0c4aa-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj48 2024-04-20T01:55:21.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj48 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj48 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.903 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj48 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ynuqLU9p/NIFuaNGKgeL2/UssvI= 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ynuqLU9p/NIFuaNGKgeL2/UssvI=', 'Content-Length': '0'} 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c237533273a4e6f-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj48 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj48 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj48 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd8dac8eca44e48d-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj481000falseobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj48 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj48 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj48 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.906 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003923766e29db8559-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj481000falseobj482024-04-19T23:26:13.920Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj49 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.908 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI= 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI=', 'Content-Length': '0'} 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8481b08eab0d9f0-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI= 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI=', 'Content-Length': '0'} 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6d319101edbdb77-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49 2024-04-20T01:55:21.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.911 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eAfIWoNF+elD5FILfnHDAvQ9EpY= 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eAfIWoNF+elD5FILfnHDAvQ9EpY=', 'Content-Length': '0'} 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e882f9f4bbc81ece-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eAfIWoNF+elD5FILfnHDAvQ9EpY= 2024-04-20T01:55:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eAfIWoNF+elD5FILfnHDAvQ9EpY=', 'Content-Length': '0'} 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000399d4226a04fd2ab-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj49 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI= 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI=', 'Content-Length': '0'} 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5fb0ebaf31eaa7f-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49 2024-04-20T01:55:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.916 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj49?versionId=null 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI= 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ENwSJophqVAmUTlC2ifVA/zeEfI=', 'Content-Length': '0'} 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d092bab75ff945f9-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj49 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj49 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj49 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e23056429e20b0de-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj491000falseobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj49 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj49 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj49 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a972ee1a4b12dc83-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj491000falseobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj49 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj49 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.921 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:W6+QcL41hgR9811tBstfuxigVYs= 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W6+QcL41hgR9811tBstfuxigVYs=', 'Content-Length': '0'} 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea43a2df53e8f3e3-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj49 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.922 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj49 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:W6+QcL41hgR9811tBstfuxigVYs= 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W6+QcL41hgR9811tBstfuxigVYs=', 'Content-Length': '0'} 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005518c2e844e6ae97-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj49 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj49 2024-04-20T01:55:21.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj49 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:kYOhBOx+bUcfAKVXRExL41Y39pY= 2024-04-20T01:55:21.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kYOhBOx+bUcfAKVXRExL41Y39pY=', 'Content-Length': '0'} 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('x-rgw-replicated-from', 'bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2abdd0517ca36e7-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj49 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj49 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj49 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.925 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj49 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:kYOhBOx+bUcfAKVXRExL41Y39pY= 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kYOhBOx+bUcfAKVXRExL41Y39pY=', 'Content-Length': '0'} 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030bad422308f6d1b-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj49 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj49 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj49 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.927 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d820b3bd35715b75-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj491000falseobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj49 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj49 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj49 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd44fa9c82315611-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj491000falseobj492024-04-19T23:26:13.971Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj5 2024-04-20T01:55:21.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.930 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U= 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U=', 'Content-Length': '0'} 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c04162890856f2e-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.932 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U= 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U=', 'Content-Length': '0'} 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079bff67ceca0fc42-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:264yvT9MNgk3+Dpz1XP7Wo1CKkM= 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:264yvT9MNgk3+Dpz1XP7Wo1CKkM=', 'Content-Length': '0'} 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000554de72105cc10a2-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5 2024-04-20T01:55:21.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.935 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:264yvT9MNgk3+Dpz1XP7Wo1CKkM= 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:264yvT9MNgk3+Dpz1XP7Wo1CKkM=', 'Content-Length': '0'} 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5c7205bd120e42e-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj5 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U= 2024-04-20T01:55:21.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U=', 'Content-Length': '0'} 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ed9f45e130e771e-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj5?versionId=null 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U= 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AxyjtJblfeXyHOS4BwZElb+CM6U=', 'Content-Length': '0'} 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d5c8a223771b05e-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj5 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj5 2024-04-20T01:55:21.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj5 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a91cce509ff2891-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj51000falseobj52024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj5 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj5 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj5 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc3185633d95477d-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj51000falseobj52024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj5 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.943 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj5 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:VnwqfiYQs0bdtOKknVcHZsm3slk= 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VnwqfiYQs0bdtOKknVcHZsm3slk=', 'Content-Length': '0'} 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e46e15b25cf017e5-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj5 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj5 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:VnwqfiYQs0bdtOKknVcHZsm3slk= 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VnwqfiYQs0bdtOKknVcHZsm3slk=', 'Content-Length': '0'} 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b94e6a88b131b1de-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj5 2024-04-20T01:55:21.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj5 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.946 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj5 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2TjVUMKpM9v7YV9X8qLRfI4HUVE= 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2TjVUMKpM9v7YV9X8qLRfI4HUVE=', 'Content-Length': '0'} 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000091afed372342ea0-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj5 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj5 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj5 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj5 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2TjVUMKpM9v7YV9X8qLRfI4HUVE= 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2TjVUMKpM9v7YV9X8qLRfI4HUVE=', 'Content-Length': '0'} 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006dcf75f2e7552a1f-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj5 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj5 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj5 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.949 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4cefb886e8ef9bb-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj51000falseobj52024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj5 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj5 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj5 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ec49e74a1ed2509-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj51000falseobj52024-04-19T23:26:12.652Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj6 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0= 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0=', 'Content-Length': '0'} 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005119ea733477f806-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.954 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0= 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0=', 'Content-Length': '0'} 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022680fc9b2295355-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4VFW70TiVsYBjtSQh5whwu73Lcs= 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4VFW70TiVsYBjtSQh5whwu73Lcs=', 'Content-Length': '0'} 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f675d6cd8aff0ed7-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6 2024-04-20T01:55:21.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.957 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.958 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.958 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.958 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4VFW70TiVsYBjtSQh5whwu73Lcs= 2024-04-20T01:55:21.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4VFW70TiVsYBjtSQh5whwu73Lcs=', 'Content-Length': '0'} 2024-04-20T01:55:21.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a869f93ae9450a5-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.958 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj6 2024-04-20T01:55:21.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6 2024-04-20T01:55:21.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6 2024-04-20T01:55:21.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.962 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0= 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0=', 'Content-Length': '0'} 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee945ad2e16000d9-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj6?versionId=null 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0= 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aYPSUF8V6XluSj9r1/WaIoVOjL0=', 'Content-Length': '0'} 2024-04-20T01:55:21.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001032021d458a074a-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj6 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj6 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj6 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.965 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c27f280b4a428c2b-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj61000falseobj62024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj6 2024-04-20T01:55:21.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj6 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj6 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.967 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE= 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z7HGaztf1CQwdQdbpEF5Zthi8kE=', 'Content-Length': '0'} 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8bf69adda6c6795-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj61000falseobj62024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj6 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj6 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rkGclTUbeGG4nnCEB3O2j+WG7BI= 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rkGclTUbeGG4nnCEB3O2j+WG7BI=', 'Content-Length': '0'} 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e1b5b3e539b14c0-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6 2024-04-20T01:55:21.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj6 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj6 2024-04-20T01:55:21.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rkGclTUbeGG4nnCEB3O2j+WG7BI= 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rkGclTUbeGG4nnCEB3O2j+WG7BI=', 'Content-Length': '0'} 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006abea39868d40951-006622fdb2-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj6 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj6 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:42 GMT 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj6 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:CNeqFsfuArjmeq/mFiyz3NzSZM0= 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CNeqFsfuArjmeq/mFiyz3NzSZM0=', 'Content-Length': '0'} 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e467df76ad63f82c-006622fdb2-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj6 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj6 2024-04-20T01:55:21.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj6 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj6 2024-04-20T01:55:21.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:aVNBVYJQVgQIIKRDLy8E6hapi1E= 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aVNBVYJQVgQIIKRDLy8E6hapi1E=', 'Content-Length': '0'} 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064eb63da20251ac6-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj6 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj6 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj6 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee9114de33a16903-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj61000falseobj62024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj6 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj6 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj6 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.976 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ab5524241828e3d-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj61000falseobj62024-04-19T23:26:12.703Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj7 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40= 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40=', 'Content-Length': '0'} 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006866a629b4b4f7d3-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40= 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40=', 'Content-Length': '0'} 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e56132450ad5945d-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7 2024-04-20T01:55:21.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.981 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:p/2EIeiW/CHijxd3EhN+u8YOLC4= 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:p/2EIeiW/CHijxd3EhN+u8YOLC4=', 'Content-Length': '0'} 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9c1427f16e80807-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:p/2EIeiW/CHijxd3EhN+u8YOLC4= 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:p/2EIeiW/CHijxd3EhN+u8YOLC4=', 'Content-Length': '0'} 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe749f83bb5a7f4c-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.983 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj7 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.984 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40= 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40=', 'Content-Length': '0'} 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c38b13bd3150c4b2-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj7?versionId=null 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.986 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40= 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ovHEMukWS3Wb2zwsdbsNye0Xc40=', 'Content-Length': '0'} 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038861d19f7a974ca-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj7 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj7 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj7 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fdb1ff116b19bb3-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj71000falseobj72024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj7 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj7 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj7 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.989 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ea3d04fcef4e0db-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj71000falseobj72024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj7 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj7 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ErVvsYFBdx8o7Ofj44LmRetub9U= 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ErVvsYFBdx8o7Ofj44LmRetub9U=', 'Content-Length': '0'} 2024-04-20T01:55:21.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a139a6381abb66c0-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj7 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.992 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj7 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ErVvsYFBdx8o7Ofj44LmRetub9U= 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ErVvsYFBdx8o7Ofj44LmRetub9U=', 'Content-Length': '0'} 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f605169fb81038bd-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj7 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj7 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj7 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:l4qKxijHU/gu/8uauJx02dUGLks= 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l4qKxijHU/gu/8uauJx02dUGLks=', 'Content-Length': '0'} 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024e83fae83b95d18-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj7 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj7 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj7 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.995 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj7 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:l4qKxijHU/gu/8uauJx02dUGLks= 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:l4qKxijHU/gu/8uauJx02dUGLks=', 'Content-Length': '0'} 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c5afc38c1607f66-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj7 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj7 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj7 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de4f57095c7c79c4-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj71000falseobj72024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj7 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj7 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj7 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:21.998 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f17f3a53e9b867a-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj71000falseobj72024-04-19T23:26:12.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj8 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8?versionId=null 2024-04-20T01:55:21.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.000 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0= 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0=', 'Content-Length': '0'} 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068624c53f97abf98-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0= 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0=', 'Content-Length': '0'} 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4c2bd71473e4bda-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8 2024-04-20T01:55:22.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.003 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qsknR8i4CcVYoCg+Qvr8fg71cIU= 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qsknR8i4CcVYoCg+Qvr8fg71cIU=', 'Content-Length': '0'} 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b567e8ec9c43070-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qsknR8i4CcVYoCg+Qvr8fg71cIU= 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qsknR8i4CcVYoCg+Qvr8fg71cIU=', 'Content-Length': '0'} 2024-04-20T01:55:22.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fc8b7eea6253ccf-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj8 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0= 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0=', 'Content-Length': '0'} 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002516b05754cc277c-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.008 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj8?versionId=null 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0= 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4JqjsjA4Gba9mgByOxZEChYjYv0=', 'Content-Length': '0'} 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c41bff5503c7c19-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj8 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj8 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj8 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004afecc669040dca6-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj81000falseobj82024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:22.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj8 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj8 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj8 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.011 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043cf67e63c2006a2-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj81000falseobj82024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj8 2024-04-20T01:55:22.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj8 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:q1gH94jCOB5lpLe70qwI9uQZdNI= 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q1gH94jCOB5lpLe70qwI9uQZdNI=', 'Content-Length': '0'} 2024-04-20T01:55:22.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f59d5ebe06fc6afe-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj8 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:22.014 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj8 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:q1gH94jCOB5lpLe70qwI9uQZdNI= 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q1gH94jCOB5lpLe70qwI9uQZdNI=', 'Content-Length': '0'} 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086bfa682059cfd56-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj8 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj8 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj8 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:iUKSjOyGIIQ9ntimhnYKBENOk5Y= 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iUKSjOyGIIQ9ntimhnYKBENOk5Y=', 'Content-Length': '0'} 2024-04-20T01:55:22.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000871c41e003e300e1-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj8 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj8 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj8 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.017 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj8 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:iUKSjOyGIIQ9ntimhnYKBENOk5Y= 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iUKSjOyGIIQ9ntimhnYKBENOk5Y=', 'Content-Length': '0'} 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005468491bc38fd345-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj8 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj8 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj8 2024-04-20T01:55:22.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:22.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078d9572d26713fdb-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj81000falseobj82024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj8 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj8 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj8 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5a9650271aae5df-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj81000falseobj82024-04-19T23:26:12.775Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj9 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9 2024-04-20T01:55:22.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9 2024-04-20T01:55:22.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw= 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw=', 'Content-Length': '0'} 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbcab50f83431963-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9 2024-04-20T01:55:22.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.024 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw= 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw=', 'Content-Length': '0'} 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb81a6a68bc42d26-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hQGa/VY4gsSwFnBDoR3hY/60WQ0= 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hQGa/VY4gsSwFnBDoR3hY/60WQ0=', 'Content-Length': '0'} 2024-04-20T01:55:22.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc0813979c5f9210-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:22.027 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hQGa/VY4gsSwFnBDoR3hY/60WQ0= 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hQGa/VY4gsSwFnBDoR3hY/60WQ0=', 'Content-Length': '0'} 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9196269deb671e0-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj9 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw= 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw=', 'Content-Length': '0'} 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009179a24acb37ab71-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj9?versionId=null 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw= 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SRh4FsVmYUKLfT/BJyQIo+LanTw=', 'Content-Length': '0'} 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000328fdad63a91bcaa-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:22.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj9 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj9 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj9 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.032 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000388b8e71c23c4040-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj91000falseobj92024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj9 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj9 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj9 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:22.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000201dd4858c606354-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj91000falseobj92024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj9 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.035 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj9 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6AZk+2qK4sQrxulwLPzUVx11IfE= 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6AZk+2qK4sQrxulwLPzUVx11IfE=', 'Content-Length': '0'} 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000218147c6d719f88a-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj9 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj9 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6AZk+2qK4sQrxulwLPzUVx11IfE= 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6AZk+2qK4sQrxulwLPzUVx11IfE=', 'Content-Length': '0'} 2024-04-20T01:55:22.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc3d48c105f53d2f-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj9 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj9 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.038 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj9 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+GHMYnL65H2K11TsBzfy9X5QAPU= 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+GHMYnL65H2K11TsBzfy9X5QAPU=', 'Content-Length': '0'} 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bd538a807a2b389-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/obj9 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/obj9 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/obj9 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests:/aktlji-16/obj9 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+GHMYnL65H2K11TsBzfy9X5QAPU= 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+GHMYnL65H2K11TsBzfy9X5QAPU=', 'Content-Length': '0'} 2024-04-20T01:55:22.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 19 Apr 2024 23:26:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae722fd95b80e17f-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj9 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj9 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj9 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b0b6cc0c9f5a72f-006622fdb3-10d7-test-zone2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj91000falseobj92024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/ 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/ 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-16/?prefix=obj9 2024-04-20T01:55:22.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-16/?prefix=obj9 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-16/?prefix=obj9 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:26:43 GMT 2024-04-20T01:55:22.043 INFO:tasks.rgw_multisite_tests:/aktlji-16/ 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8= 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:26:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eEqNB+7nYG8Qo06nWLwpvXMDcP8=', 'Content-Length': '0'} 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073cb91d1a4781a13-006622fdb3-1100-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 19 Apr 2024 23:26:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-16obj91000falseobj92024-04-19T23:26:12.785Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=aktlji-16 zones={test-zone2, test-zone1} 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:26:43.511+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:26:48.512+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:26:53.512+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:26:58.513+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:27:03.514+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:27:08.514+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:27:13.515+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:27:18.516+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:27:23.517+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:27:28.518+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:27:33.519+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:27:38.520+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:27:43.520+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:27:48.521+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:27:53.521+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:27:58.521+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:28:03.522+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:28:08.522+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:28:13.523+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:28:18.524+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:28:23.525+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:28:28.525+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:28:33.526+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:28:38.527+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:28:43.528+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:28:48.528+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:28:53.529+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi031.stdout: INFO: 2024-04-19T23:28:54.490+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-20T01:55:22.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:28:58.530+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:29:03.530+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:29:08.531+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:29:13.533+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:29:18.533+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:29:23.534+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:29:28.535+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:29:33.535+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:29:38.536+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:29:43.536+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:29:48.536+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:29:53.537+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:29:58.538+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:30:03.538+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:30:08.539+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:30:13.540+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:30:18.541+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:30:23.541+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:30:28.542+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:30:33.543+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:30:38.544+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:30:43.544+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:30:48.545+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:30:53.547+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:30:58.547+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:31:03.548+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:31:08.549+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:31:13.549+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:31:18.550+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:31:23.549+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:31:28.550+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:31:33.551+0000 7f1ae9ded780 1 waiting to reach incremental sync.. 2024-04-20T01:55:22.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:31:38.552+0000 7f1ae9ded780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:22.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:31:38.552+0000 7f1ae9ded780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:22.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-19T23:31:38.552+0000 7f1ae9ded780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:22.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:22.049 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:22.050 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.050 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:22.051 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_delete_notempty 2024-04-20T01:55:22.051 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:22.053 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:22.053 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:22.053 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:22.053 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 900, in test_bucket_delete_notempty 2024-04-20T01:55:22.053 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-20T01:55:22.054 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-04-20T01:55:22.054 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-20T01:55:22.054 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-04-20T01:55:22.054 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-04-20T01:55:22.054 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-04-20T01:55:22.054 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:22.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-39 2024-04-20T01:55:22.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-39/ 2024-04-20T01:55:22.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-39/ 2024-04-20T01:55:22.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:22.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-39/ 2024-04-20T01:55:22.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.057 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:22.057 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.057 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.057 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:44:05 GMT 2024-04-20T01:55:22.058 INFO:tasks.rgw_multisite_tests:/aktlji-39/ 2024-04-20T01:55:22.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.058 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d8SN9GWgVBNEChRLFfHwoH/weZQ= 2024-04-20T01:55:22.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d8SN9GWgVBNEChRLFfHwoH/weZQ=', 'Content-Length': '0'} 2024-04-20T01:55:22.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000aa8cda59e075e825-00662301c5-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=aktlji-40 2024-04-20T01:55:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-40/ 2024-04-20T01:55:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-40/ 2024-04-20T01:55:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-40/ 2024-04-20T01:55:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.060 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:22.060 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.060 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.061 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:44:05 GMT 2024-04-20T01:55:22.061 INFO:tasks.rgw_multisite_tests:/aktlji-40/ 2024-04-20T01:55:22.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.061 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:gPMzWRdUt2uvhWzlDWzfb1b45cY= 2024-04-20T01:55:22.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gPMzWRdUt2uvhWzlDWzfb1b45cY=', 'Content-Length': '0'} 2024-04-20T01:55:22.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000004faa31999e953b2e-00662301c5-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=aktlji-41 2024-04-20T01:55:22.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-41/ 2024-04-20T01:55:22.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-41/ 2024-04-20T01:55:22.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:22.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-41/ 2024-04-20T01:55:22.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:22.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:22.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.062 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:22.062 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.062 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.062 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:44:05 GMT 2024-04-20T01:55:22.062 INFO:tasks.rgw_multisite_tests:/aktlji-41/ 2024-04-20T01:55:22.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.063 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:PaAvvMSr9lbfioqobDMMH3gj+Yo= 2024-04-20T01:55:22.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:44:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PaAvvMSr9lbfioqobDMMH3gj+Yo=', 'Content-Length': '0'} 2024-04-20T01:55:22.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000dbcce448e20eca0c-00662301c5-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:44:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.063 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.063 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:22.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713570245.168285_359.1", 2024-04-20T01:55:22.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:44:05.168285Z" 2024-04-20T01:55:22.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:22.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:22.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:22.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:22.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:22.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:22.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:05.434+0000 7f21d3c02780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:05.435+0000 7f21d3c02780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:44:05Z" 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.073 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.074 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.074 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.074 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.074 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.074 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.074 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.074 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.076 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.076 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.076 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.076 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.076 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.077 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.077 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.077 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.077 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.077 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.077 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.078 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.078 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.078 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.078 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.078 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.078 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.079 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.079 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.079 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.079 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.079 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.080 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.080 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.080 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.080 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.080 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.080 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.081 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.081 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.081 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.081 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.081 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.081 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:44:05Z" 2024-04-20T01:55:22.082 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.082 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.082 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:22.082 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.082 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:10.554+0000 7fc4963de780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:10.555+0000 7fc4963de780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:22.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:22.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:44:10Z" 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.092 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.093 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.093 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.093 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.093 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.093 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.093 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.094 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.094 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.094 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.094 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.094 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.094 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.094 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.094 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.094 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.095 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.096 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.097 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.097 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.097 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.097 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.097 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.097 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.097 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.097 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.097 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.098 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.098 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.098 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.098 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.098 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.098 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.099 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.099 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:44:10Z" 2024-04-20T01:55:22.099 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.099 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.609968_303.1 2024-04-20T01:55:22.100 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi031.stdout: INFO: 2024-04-19T23:44:14.513+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-20T01:55:22.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:15.723+0000 7f58e4fc9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:15.724+0000 7f58e4fc9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:44:15Z" 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.107 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.109 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.110 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.111 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:44:15Z" 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:20.843+0000 7f3841b67780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:20.844+0000 7f3841b67780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:44:20Z" 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.117 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.118 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.118 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.118 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.118 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.118 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.118 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.118 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.118 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.119 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.120 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.121 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:44:20Z" 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.122 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:25.997+0000 7f252c371780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:25.998+0000 7f252c371780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:44:26Z" 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.128 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.129 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.130 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.131 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.132 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:44:26Z" 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:31.113+0000 7f624950c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:31.114+0000 7f624950c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:22.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:44:31Z" 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.138 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.139 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.140 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.141 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.142 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:44:31Z" 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.738086_316.1 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:36.307+0000 7f45351eb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:36.308+0000 7f45351eb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:44:36Z" 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.148 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.149 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.150 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.151 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.152 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:44:36Z" 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:41.426+0000 7f0f69542780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:41.428+0000 7f0f69542780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:44:41Z" 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.158 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.159 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.161 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.162 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:44:41Z" 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:46.547+0000 7f25a18d4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:46.548+0000 7f25a18d4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:44:46Z" 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.172 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.173 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.174 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.175 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:44:46Z" 2024-04-20T01:55:22.176 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:51.675+0000 7fd0d03f2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:51.676+0000 7fd0d03f2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:44:51Z" 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.182 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.183 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.184 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.185 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.186 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:44:51Z" 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.577447_300.1 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:56.877+0000 7f545bb59780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:44:56.878+0000 7f545bb59780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:44:56Z" 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.192 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.193 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.194 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.195 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.196 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:44:56Z" 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.572795_298.1 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:01.996+0000 7fc161f2e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:01.997+0000 7fc161f2e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:45:01Z" 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.203 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.204 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.205 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.206 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:45:01Z" 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.572795_298.1 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:07.114+0000 7fa3cca2d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:07.115+0000 7fa3cca2d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:45:07Z" 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.213 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.214 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.215 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:45:07Z" 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.217 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:12.231+0000 7f7e2bf5a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:12.232+0000 7f7e2bf5a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:45:12Z" 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.223 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.224 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.225 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.226 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.227 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:45:12Z" 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.580723_301.1 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:17.355+0000 7f512f4c5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:17.356+0000 7f512f4c5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:45:17Z" 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.233 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.234 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.235 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.236 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.236 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.236 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.236 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.236 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.236 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.236 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:22.236 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.236 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.236 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:22.236 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.237 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:45:17Z" 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.555632_296.1 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:22.479+0000 7f23e5659780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:22.480+0000 7f23e5659780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:45:22Z" 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.244 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.245 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.246 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.247 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:45:22Z" 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.248 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:27.695+0000 7f3ad3e8d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:27.696+0000 7f3ad3e8d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:45:27Z" 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.254 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.255 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.256 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.257 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.258 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:45:27Z" 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.572795_298.1 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:32.813+0000 7fed69e7e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:32.814+0000 7fed69e7e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:45:32Z" 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.264 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.265 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.266 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.267 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.268 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:45:32Z" 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:37.932+0000 7faea99b9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:37.933+0000 7faea99b9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:45:37Z" 2024-04-20T01:55:22.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.275 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.276 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:22.277 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.278 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:45:37Z" 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.577447_300.1 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:43.052+0000 7fc481dd1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:43.053+0000 7fc481dd1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:45:43Z" 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.285 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.286 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.287 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.288 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:45:43Z" 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.289 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:48.172+0000 7f73478fb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:48.173+0000 7f73478fb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.698420_310.1", 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:45:48Z" 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.295 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.296 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.297 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.698420_310.1", 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.298 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.299 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:45:48Z" 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.698420_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.698420Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.698420_310.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.698420_310.1 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:53.329+0000 7efe6fe54780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:53.330+0000 7efe6fe54780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:45:53Z" 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.305 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.306 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.307 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.308 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.309 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:45:53Z" 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.580723_301.1 2024-04-20T01:55:22.310 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:58.501+0000 7f2da4ca4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:45:58.503+0000 7f2da4ca4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:45:58Z" 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.316 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.317 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.318 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.319 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:45:58Z" 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:03.622+0000 7f635cdb3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:03.623+0000 7f635cdb3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:46:03Z" 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.326 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.327 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.328 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.329 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.330 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:46:03Z" 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.606676_302.1 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:08.741+0000 7f550a29b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:08.742+0000 7f550a29b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:46:08Z" 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.336 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.338 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.339 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.340 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:46:08Z" 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.555632_296.1 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:13.861+0000 7efd7bc0c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:13.862+0000 7efd7bc0c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:46:13Z" 2024-04-20T01:55:22.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.347 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.348 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.349 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.350 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:46:13Z" 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:18.983+0000 7ff240d17780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:18.985+0000 7ff240d17780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:46:18Z" 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.357 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.358 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.359 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.360 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:46:18Z" 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.361 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:22.362 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.362 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:24.105+0000 7f18fd80d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:24.106+0000 7f18fd80d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:46:24Z" 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.367 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.368 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.368 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.368 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.368 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.368 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.368 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.371 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.371 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.371 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.371 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.371 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.371 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.371 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.371 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.371 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.371 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.372 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.373 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:46:24Z" 2024-04-20T01:55:22.374 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:29.227+0000 7f2b76b3d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:29.229+0000 7f2b76b3d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:46:29Z" 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.380 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.381 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.382 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.383 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.384 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:46:29Z" 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:34.412+0000 7f744977b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:34.413+0000 7f744977b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:46:34Z" 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:22.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.394 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:46:34Z" 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:39.530+0000 7f82e4399780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:39.531+0000 7f82e4399780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:46:39Z" 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.401 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.402 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.403 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.404 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:46:39Z" 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.405 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:44.652+0000 7efc02203780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:44.653+0000 7efc02203780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.687754_309.1", 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-20T01:55:22.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:46:44Z" 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.411 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.412 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.413 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.687754_309.1", 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.414 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.415 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:46:44Z" 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.687754_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.687754Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.687754_309.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.687754_309.1 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:49.772+0000 7fe9b490c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:49.773+0000 7fe9b490c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:46:49Z" 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.425 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:46:49Z" 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:54.892+0000 7fadb325b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:46:54.893+0000 7fadb325b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:46:54Z" 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.432 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.433 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:22.434 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.435 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:46:54Z" 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.436 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.632785_307.1 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:00.013+0000 7f552eecf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:00.019+0000 7f552eecf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:47:00Z" 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.442 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.443 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.444 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.445 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.446 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:47:00Z" 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.738086_316.1 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:05.318+0000 7ff4e0f37780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:05.319+0000 7ff4e0f37780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:47:05Z" 2024-04-20T01:55:22.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.453 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.454 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.455 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.456 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:47:05Z" 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:10.439+0000 7fb553fd7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:10.440+0000 7fb553fd7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:47:10Z" 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.463 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.464 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:22.465 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.466 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:47:10Z" 2024-04-20T01:55:22.467 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.614143_305.1 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:15.589+0000 7f0d671de780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:15.590+0000 7f0d671de780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:47:15Z" 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.473 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.474 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.475 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.476 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.477 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:47:15Z" 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:20.710+0000 7f3b2a4a5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:20.711+0000 7f3b2a4a5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.720922_313.1", 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.720922Z", 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:47:20Z" 2024-04-20T01:55:22.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.484 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.485 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.720922_313.1", 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.486 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.720922Z", 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.487 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:47:20Z" 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.720922_313.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.720922Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.720922_313.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.488 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.720922_313.1 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:25.915+0000 7fdbcc1b2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:25.916+0000 7fdbcc1b2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:47:25Z" 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.494 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.496 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.497 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:47:25Z" 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.577447_300.1 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:31.035+0000 7f31ca362780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:31.036+0000 7f31ca362780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:47:31Z" 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.504 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.505 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.506 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.507 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.508 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:47:31Z" 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.580723_301.1 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:36.155+0000 7fe152c7a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:36.156+0000 7fe152c7a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:47:36Z" 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.514 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.515 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.516 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.517 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.518 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:47:36Z" 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.577447_300.1 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:41.276+0000 7fcc7c794780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:41.277+0000 7fcc7c794780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:47:41Z" 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.527 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.528 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:47:41Z" 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.580723_301.1 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:46.400+0000 7fa40903b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:46.401+0000 7fa40903b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:47:46Z" 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.535 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.536 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.537 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.538 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:47:46Z" 2024-04-20T01:55:22.539 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.580723_301.1 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:51.521+0000 7f65a5ea4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:51.523+0000 7f65a5ea4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:47:51Z" 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.545 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.546 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.547 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.548 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.549 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:47:51Z" 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:56.731+0000 7fdcdb6d5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:47:56.733+0000 7fdcdb6d5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:22.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:47:56Z" 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.557 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.558 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.558 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.558 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.558 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.558 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.558 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.558 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.558 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.559 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:47:56Z" 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.560 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.797017_325.1 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:01.844+0000 7f28eba46780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:01.845+0000 7f28eba46780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:22.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:48:01Z" 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.566 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.568 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.569 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.570 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:48:01Z" 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.572795_298.1 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:06.963+0000 7f1c2a392780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:06.964+0000 7f1c2a392780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:48:06Z" 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.578 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.579 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.580 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.581 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:48:06Z" 2024-04-20T01:55:22.582 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:12.084+0000 7f422ef7d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:12.086+0000 7f422ef7d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:48:12Z" 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.588 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.589 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.590 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.590 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.590 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.590 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.590 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.590 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.590 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.590 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.590 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.590 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.590 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.591 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.592 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:48:12Z" 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.632785_307.1 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:17.203+0000 7f51b1229780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:17.204+0000 7f51b1229780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.720922_313.1", 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.720922Z", 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:48:17Z" 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.599 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.600 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.720922_313.1", 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.601 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.720922Z", 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.602 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:48:17Z" 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.720922_313.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.720922Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.720922_313.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.720922_313.1 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:22.331+0000 7f315381f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:22.333+0000 7f315381f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:48:22Z" 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.609 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.610 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.611 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.612 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:48:22Z" 2024-04-20T01:55:22.613 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.555632_296.1 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:27.453+0000 7f9285b8c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:27.455+0000 7f9285b8c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:22.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:48:27Z" 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.619 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.620 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.621 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.622 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.623 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:48:27Z" 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:32.619+0000 7f04418ab780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:32.620+0000 7f04418ab780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:48:32Z" 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.629 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.630 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.631 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.632 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.633 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:48:32Z" 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.577447_300.1 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:37.742+0000 7f989d0f7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:37.743+0000 7f989d0f7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.868452_328.1", 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:48:37Z" 2024-04-20T01:55:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.640 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.641 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.868452_328.1", 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.642 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.643 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:48:37Z" 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.868452_328.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.868452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.868452_328.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.868452_328.1 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:42.863+0000 7f96a49ee780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:42.865+0000 7f96a49ee780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:48:42Z" 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.650 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.651 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.652 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:48:42Z" 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.609968_303.1 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:47.984+0000 7f9987691780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:47.986+0000 7f9987691780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.717624_312.1", 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:48:47Z" 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.660 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.661 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.662 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.717624_312.1", 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.663 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.664 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:48:47Z" 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.717624_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.717624Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.717624_312.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.717624_312.1 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:53.105+0000 7fa00a404780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:53.107+0000 7fa00a404780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.717624_312.1", 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:48:53Z" 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.670 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.671 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.672 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.717624_312.1", 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.673 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.674 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:48:53Z" 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.717624_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.717624Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.717624_312.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.717624_312.1 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:58.314+0000 7fc2dd18d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:48:58.315+0000 7fc2dd18d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:48:58Z" 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.680 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.681 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.682 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:22.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.684 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:48:58Z" 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.555632_296.1 2024-04-20T01:55:22.685 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:03.442+0000 7fe3db9eb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:03.443+0000 7fe3db9eb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:49:03Z" 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.691 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.692 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.693 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.694 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:49:03Z" 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:08.561+0000 7faefff54780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:08.563+0000 7faefff54780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:49:08Z" 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.701 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.702 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.703 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.704 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:49:08Z" 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.705 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.168285_359.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570245.168285_359.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:22.706 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:22.706 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.706 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:22.706 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_period_incremental_sync 2024-04-20T01:55:22.707 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:22.708 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:22.708 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:22.708 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:22.708 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 933, in test_multi_period_incremental_sync 2024-04-20T01:55:22.708 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-20T01:55:22.708 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-04-20T01:55:22.708 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-20T01:55:22.708 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-04-20T01:55:22.708 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-04-20T01:55:22.708 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-04-20T01:55:22.708 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-42 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-42/ 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-42/ 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-42/ 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.709 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:49:13 GMT 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests:/aktlji-42/ 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:NlQ1vw3o1KldTw18QhGAZ4d9iH0= 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:49:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:NlQ1vw3o1KldTw18QhGAZ4d9iH0=', 'Content-Length': '0'} 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002c8bc13064425a82-00662302f9-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:49:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=aktlji-43 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-43/ 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-43/ 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:22.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-43/ 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:49:13 GMT 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:/aktlji-43/ 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.711 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Y6Bz09C2QlItxMkZmUCYPaAPFeU= 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:49:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y6Bz09C2QlItxMkZmUCYPaAPFeU=', 'Content-Length': '0'} 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000de191e880de62143-00662302f9-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:49:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=aktlji-44 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-44/ 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-44/ 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-44/ 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T01:55:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:49:13 GMT 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests:/aktlji-44/ 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cs02GKKGT24bVlhiOle5zAkwxTI= 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:49:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cs02GKKGT24bVlhiOle5zAkwxTI=', 'Content-Length': '0'} 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000af4dd71f0197ccf7-00662302f9-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:49:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713570553.765147_405.1", 2024-04-20T01:55:22.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:49:13.765147Z" 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:14.026+0000 7f265e6ac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:14.028+0000 7f265e6ac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:49:14Z" 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.720 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.721 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.722 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.723 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.724 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:49:14Z" 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi031.stdout: INFO: 2024-04-19T23:49:14.520+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:19.221+0000 7fcffc384780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:19.222+0000 7fcffc384780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:49:19Z" 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.731 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.732 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.733 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.734 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:49:19Z" 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.606676_302.1 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:24.340+0000 7f7cea89c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:24.341+0000 7f7cea89c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:49:24Z" 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.741 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.742 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:22.743 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.744 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:49:24Z" 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:29.463+0000 7fedbda88780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:29.464+0000 7fedbda88780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:49:29Z" 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.751 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.752 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.753 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.754 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.755 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:49:29Z" 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.555632_296.1 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:34.587+0000 7f1ffee77780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:34.588+0000 7f1ffee77780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:49:34Z" 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.761 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.763 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.764 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.765 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:49:34Z" 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.580723_301.1 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.766 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:39.709+0000 7fd8cb34a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:39.710+0000 7fd8cb34a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:49:39Z" 2024-04-20T01:55:22.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.772 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.773 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.774 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.775 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:49:39Z" 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.580723_301.1 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:44.830+0000 7fc3b3042780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:44.831+0000 7fc3b3042780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:49:44Z" 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.782 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.783 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.784 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:49:44Z" 2024-04-20T01:55:22.786 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:50.026+0000 7fb7a6811780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:50.027+0000 7fb7a6811780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:22.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:49:50Z" 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.792 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.793 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.794 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.795 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.796 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:49:50Z" 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.606676_302.1 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:55.146+0000 7ff6c9a4f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:49:55.147+0000 7ff6c9a4f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.808187_326.1", 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:49:55Z" 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.802 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.803 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.804 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.808187_326.1", 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.805 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.806 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:49:55Z" 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.808187_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.808187Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.808187_326.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.808187_326.1 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:00.267+0000 7f09780fd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:00.268+0000 7f09780fd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:50:00Z" 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.813 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.814 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:22.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.816 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:50:00Z" 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.614143_305.1 2024-04-20T01:55:22.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:05.386+0000 7f9d1faa4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:05.387+0000 7f9d1faa4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:50:05Z" 2024-04-20T01:55:22.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.823 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.823 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.823 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.823 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.823 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.823 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.823 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.824 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.824 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.824 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.824 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.824 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.825 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.825 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.825 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.825 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.825 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.826 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.826 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.826 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.826 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.826 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.826 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.827 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.827 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.827 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.827 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.827 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.827 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.827 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.828 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:50:05Z" 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.829 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.609968_303.1 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:10.509+0000 7fbe8f642780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:10.511+0000 7fbe8f642780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:50:10Z" 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.837 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.838 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.839 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:50:10Z" 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.580723_301.1 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:15.635+0000 7f41469a4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:15.636+0000 7f41469a4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:50:15Z" 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.845 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.847 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.848 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.849 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:50:15Z" 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.797017_325.1 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:20.754+0000 7f205cf8a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:20.755+0000 7f205cf8a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570244.063607_345.1", 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:04.063607Z", 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:50:20Z" 2024-04-20T01:55:22.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.856 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.857 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570244.063607_345.1", 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.858 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:04.063607Z", 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.859 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:50:20Z" 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570244.063607_345.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:04.063607Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570244.063607_345.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570244.063607_345.1 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:25.900+0000 7f5095aa4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:25.901+0000 7f5095aa4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.698420_310.1", 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:50:25Z" 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.866 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.867 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.698420_310.1", 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.868 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.869 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:50:25Z" 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.698420_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.698420Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.698420_310.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.698420_310.1 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:31.040+0000 7f00d3c62780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:31.041+0000 7f00d3c62780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:50:31Z" 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.876 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.877 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.878 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.879 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.880 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:50:31Z" 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.572795_298.1 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:36.161+0000 7fd1aac5c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:36.162+0000 7fd1aac5c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.701491_311.1", 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.701491Z", 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:50:36Z" 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.886 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.888 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.701491_311.1", 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.701491Z", 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.889 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.890 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:50:36Z" 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.701491_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.701491Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.701491_311.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.701491_311.1 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:41.282+0000 7f243c0b8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:41.284+0000 7f243c0b8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.717624_312.1", 2024-04-20T01:55:22.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:50:41Z" 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.896 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.898 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.717624_312.1", 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-20T01:55:22.899 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.900 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:50:41Z" 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.717624_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.717624Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.717624_312.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.717624_312.1 2024-04-20T01:55:22.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:46.405+0000 7f8daa416780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:46.406+0000 7f8daa416780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.769121_320.1", 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.769121Z", 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:50:46Z" 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.907 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.908 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.769121_320.1", 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.909 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.769121Z", 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.910 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:50:46Z" 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.769121_320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.769121Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.769121_320.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.769121_320.1 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:51.525+0000 7fbc69b11780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:51.526+0000 7fbc69b11780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.687754_309.1", 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:50:51Z" 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.917 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.918 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.687754_309.1", 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.920 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:50:51Z" 2024-04-20T01:55:22.921 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.687754_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.687754Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.687754_309.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.687754_309.1 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:56.650+0000 7fd4facbd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:50:56.651+0000 7fd4facbd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:50:56Z" 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.927 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.928 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.929 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.930 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.931 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:50:56Z" 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:01.815+0000 7fc794cd9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:01.816+0000 7fc794cd9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:51:01Z" 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.937 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.938 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.939 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.940 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.941 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:51:01Z" 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.580723_301.1 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:06.939+0000 7f4d6aa88780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:06.940+0000 7f4d6aa88780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.725126_315.1", 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.725126Z", 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:51:06Z" 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.949 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.950 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.725126_315.1", 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.725126Z", 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.951 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.952 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:51:06Z" 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.725126_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.725126Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.725126_315.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.725126_315.1 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:12.058+0000 7fc782330780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:12.059+0000 7fc782330780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:51:12Z" 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.958 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.960 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.961 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.962 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:51:12Z" 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.606676_302.1 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:17.184+0000 7f170feab780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:17.185+0000 7f170feab780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.725126_315.1", 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.725126Z", 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:51:17Z" 2024-04-20T01:55:22.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.969 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.970 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.725126_315.1", 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.971 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.725126Z", 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.972 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:51:17Z" 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.725126_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.725126Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.725126_315.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.725126_315.1 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:22.308+0000 7f768de75780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:22.309+0000 7f768de75780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:51:22Z" 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.979 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.980 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:22.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:51:22Z" 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.983 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:27.546+0000 7f84a4f06780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:27.548+0000 7f84a4f06780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:51:27Z" 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.990 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.991 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:22.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:22.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:51:27Z" 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.609968_303.1 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:22.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:32.670+0000 7f3a733ca780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:32.671+0000 7f3a733ca780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:22.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:22.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:51:32Z" 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:22.999 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.000 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.001 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.003 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:51:32Z" 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:37.800+0000 7f897f354780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:37.801+0000 7f897f354780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:51:37Z" 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.010 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:51:37Z" 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:42.922+0000 7f90b1bfd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:42.923+0000 7f90b1bfd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:51:42Z" 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.020 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.021 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.022 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.023 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:51:42Z" 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.024 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.577447_300.1 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:48.044+0000 7f2b6648b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:48.045+0000 7f2b6648b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:51:48Z" 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.030 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.031 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.032 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.033 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.034 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:51:48Z" 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.572795_298.1 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:53.166+0000 7f3314e24780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:53.167+0000 7f3314e24780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:51:53Z" 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.040 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.041 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.042 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.043 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.044 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:51:53Z" 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.738086_316.1 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:58.368+0000 7f697e645780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:51:58.370+0000 7f697e645780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:51:58Z" 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.050 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.051 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.052 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.053 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.054 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:51:58Z" 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.572795_298.1 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:03.489+0000 7f3172ad0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:03.490+0000 7f3172ad0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:52:03Z" 2024-04-20T01:55:23.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.061 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.061 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.061 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.061 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.061 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.061 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.061 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.061 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.062 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:52:03Z" 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.065 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:08.615+0000 7f3a10365780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:08.617+0000 7f3a10365780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.687754_309.1", 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:52:08Z" 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.071 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.072 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.073 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.687754_309.1", 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.074 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.075 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:52:08Z" 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.687754_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.687754Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.687754_309.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.687754_309.1 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:13.736+0000 7fc6dcb6d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:13.738+0000 7fc6dcb6d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:52:13Z" 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.081 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.082 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.083 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.084 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.085 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:52:13Z" 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:23.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:18.863+0000 7f177eb4f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:18.865+0000 7f177eb4f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:52:18Z" 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.092 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.093 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.094 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:52:18Z" 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:23.983+0000 7fe4a5175780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:23.984+0000 7fe4a5175780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:52:23Z" 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.102 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.103 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.104 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.105 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.106 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:52:23Z" 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:29.208+0000 7f094fcd4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:29.209+0000 7f094fcd4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:52:29Z" 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.112 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.113 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.114 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.115 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.116 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:52:29Z" 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.572795_298.1 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:34.326+0000 7f4342985780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:34.327+0000 7f4342985780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:52:34Z" 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.123 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.124 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.125 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.126 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:52:34Z" 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:23.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:39.455+0000 7f17a7357780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:39.456+0000 7f17a7357780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:52:39Z" 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.133 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.134 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.135 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.136 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:52:39Z" 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:44.579+0000 7fe0b2336780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:44.580+0000 7fe0b2336780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:52:44Z" 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.143 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.144 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.145 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.146 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:52:44Z" 2024-04-20T01:55:23.147 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:49.703+0000 7f5c1e902780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:49.704+0000 7f5c1e902780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:52:49Z" 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.153 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.154 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.155 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.156 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.157 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:52:49Z" 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:54.833+0000 7f8f13c74780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:54.835+0000 7f8f13c74780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:52:54Z" 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.163 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.164 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.165 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.166 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.167 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:52:54Z" 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:59.958+0000 7f9da9240780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:52:59.960+0000 7f9da9240780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:52:59Z" 2024-04-20T01:55:23.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.174 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.175 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.176 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.177 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:52:59Z" 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.632785_307.1 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:05.091+0000 7fc971500780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:05.092+0000 7fc971500780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:53:05Z" 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.184 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.185 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.186 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.187 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.188 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:53:05Z" 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:10.222+0000 7f52b705a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:10.223+0000 7f52b705a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:53:10Z" 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.194 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.197 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.198 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:53:10Z" 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.572795_298.1 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:15.355+0000 7f3439753780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:15.356+0000 7f3439753780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:53:15Z" 2024-04-20T01:55:23.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.205 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.206 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.207 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.208 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:53:15Z" 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:20.477+0000 7fa57b407780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:20.478+0000 7fa57b407780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:53:20Z" 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.215 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:23.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.218 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:53:20Z" 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.219 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.580723_301.1 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:25.604+0000 7f41ba102780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:25.605+0000 7f41ba102780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.611175_304.1", 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.611175Z", 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:53:25Z" 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.225 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.226 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.227 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.611175_304.1", 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.611175Z", 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.228 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.229 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:53:25Z" 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.611175_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.611175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.611175_304.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.611175_304.1 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:30.729+0000 7f03bae86780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:30.731+0000 7f03bae86780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:53:30Z" 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.235 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.236 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.237 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.238 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.239 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:53:30Z" 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:35.862+0000 7f21259ef780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:35.863+0000 7f21259ef780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:53:35Z" 2024-04-20T01:55:23.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.246 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.247 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.248 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.249 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:53:35Z" 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.606676_302.1 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:40.983+0000 7f35e1756780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:40.984+0000 7f35e1756780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:53:40Z" 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.256 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.257 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.258 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.259 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:53:40Z" 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.260 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:46.105+0000 7f9497317780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:46.107+0000 7f9497317780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:23.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:53:46Z" 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.266 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.267 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.268 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.269 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.270 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:53:46Z" 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.555632_296.1 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:51.239+0000 7fddac538780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:51.241+0000 7fddac538780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:53:51Z" 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.276 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.277 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.278 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.279 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.280 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:53:51Z" 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.632785_307.1 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:56.377+0000 7f3dcf350780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:53:56.378+0000 7f3dcf350780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:53:56Z" 2024-04-20T01:55:23.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.287 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.288 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.289 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.290 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:53:56Z" 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:23.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:01.500+0000 7fb3aaf16780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:01.501+0000 7fb3aaf16780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:54:01Z" 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.297 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.298 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.299 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.300 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:54:01Z" 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:06.631+0000 7fc0e81b0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:06.632+0000 7fc0e81b0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:54:06Z" 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.307 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.308 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.309 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.310 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.311 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:54:06Z" 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.632785_307.1 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:11.756+0000 7f12f50d1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:11.757+0000 7f12f50d1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:54:11Z" 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.317 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.318 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.319 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.320 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.321 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:54:11Z" 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi031.stdout: INFO: 2024-04-19T23:54:14.525+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:16.984+0000 7f0416f14780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:16.993+0000 7f0416f14780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:54:16Z" 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.327 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.328 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.329 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.330 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.331 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:54:16Z" 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570553.765147_405.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570553.765147_405.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:23.332 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:23.333 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.333 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:23.333 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_datalog_autotrim 2024-04-20T01:55:23.333 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:23.334 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:23.334 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:23.334 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:23.334 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1018, in test_datalog_autotrim 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-45 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-45/ 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-45/ 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-45/ 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:23.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:54:22 GMT 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests:/aktlji-45/ 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:23.336 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:C/5ve7yvmI6QZ7M1gbSTYCHFPj8= 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:C/5ve7yvmI6QZ7M1gbSTYCHFPj8=', 'Content-Length': '0'} 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000d9b57a0acbec57fb-006623042e-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=aktlji-46 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-46/ 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-46/ 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-46/ 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:54:22 GMT 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests:/aktlji-46/ 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XHarvH3wDkIJW/6kRgGMEno6I8Q= 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XHarvH3wDkIJW/6kRgGMEno6I8Q=', 'Content-Length': '0'} 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002b82244dac32d6d3-006623042e-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=aktlji-47 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-47/ 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-47/ 2024-04-20T01:55:23.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-47/ 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:54:22 GMT 2024-04-20T01:55:23.339 INFO:tasks.rgw_multisite_tests:/aktlji-47/ 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:DcJ9oHBOcIgVx4uu5pi/I7L56Rk= 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DcJ9oHBOcIgVx4uu5pi/I7L56Rk=', 'Content-Length': '0'} 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e0b8f71b1bedf772-006623042e-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-45/ 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-45/ 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-45/ 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:54:22 GMT 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests:/aktlji-45/ 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:9z+sXwF8si1DD04e46fegyCcTnw= 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9z+sXwF8si1DD04e46fegyCcTnw=', 'Content-Length': '0'} 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000504d4fdb4637c0ea-006623042e-1100-test-zone1'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-45/key 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-45/key 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-45/key 2024-04-20T01:55:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:54:22 GMT 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:/aktlji-45/key 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Vd31iakfXXWGP83F6rNVB6QWMQo= 2024-04-20T01:55:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Vd31iakfXXWGP83F6rNVB6QWMQo='} 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a86c97d4411e40c-006623042e-1100-test-zone1'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-46/ 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-46/ 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-46/ 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:23.343 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:54:22 GMT 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:/aktlji-46/ 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:soFx5r61eGhSdhlOdHt8t/oeH98= 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:soFx5r61eGhSdhlOdHt8t/oeH98=', 'Content-Length': '0'} 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006dc6394c2aa02ccf-006623042e-10d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-46/key 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-46/key 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-46/key 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:23.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:54:22 GMT 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:/aktlji-46/key 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1mh1Q4bI35b5sKn4JGH1yw5ZWOw= 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1mh1Q4bI35b5sKn4JGH1yw5ZWOw='} 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec27c9101f6712fe-006623042e-10d7-test-zone2'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-47/ 2024-04-20T01:55:23.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-47/ 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-47/ 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:54:22 GMT 2024-04-20T01:55:23.346 INFO:tasks.rgw_multisite_tests:/aktlji-47/ 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tF81CjUAu1YlvWGVtulKaqqE6hE= 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tF81CjUAu1YlvWGVtulKaqqE6hE=', 'Content-Length': '0'} 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060a1b4d1ae47246b-006623042e-1106-test-zone3'), ('Content-Length', '0'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-47/key 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-47/key 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-47/key 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:Fri, 19 Apr 2024 23:54:22 GMT 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:/aktlji-47/key 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ApjnyE/+777brPOGMV3K9WGbZOk= 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 19 Apr 2024 23:54:22 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ApjnyE/+777brPOGMV3K9WGbZOk='} 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000193dc8ff69b733e5-006623042e-1106-test-zone3'), ('Date', 'Fri, 19 Apr 2024 23:54:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713570862.263117_452.1", 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:54:22.263117Z" 2024-04-20T01:55:23.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:22.567+0000 7f10af769780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:22.569+0000 7f10af769780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:54:22Z" 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.355 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.357 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.358 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.359 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:54:22Z" 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.632785_307.1 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:27.721+0000 7f047fdf7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:27.722+0000 7f047fdf7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:54:27Z" 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.365 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.366 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.368 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.369 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:54:27Z" 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.555632_296.1 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.370 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:32.859+0000 7fa9ee78c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:32.860+0000 7fa9ee78c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:54:32Z" 2024-04-20T01:55:23.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.376 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.377 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.378 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.379 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:54:32Z" 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.797017_325.1 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:37.994+0000 7f4157ed3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:37.995+0000 7f4157ed3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:54:37Z" 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.386 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.387 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.387 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.387 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.387 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.387 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.389 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.389 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.389 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.389 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.390 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.391 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.392 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:54:37Z" 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.572795_298.1 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:43.234+0000 7f2cb08b2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:43.235+0000 7f2cb08b2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:54:43Z" 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.398 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.399 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.400 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.401 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.402 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:54:43Z" 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:48.486+0000 7f68737c0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:48.488+0000 7f68737c0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:54:48Z" 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.410 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.411 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.412 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.413 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:54:48Z" 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:53.654+0000 7f3c926bb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:53.655+0000 7f3c926bb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.793769_324.1", 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.793769Z", 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:54:53Z" 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.420 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.421 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.793769_324.1", 2024-04-20T01:55:23.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.793769Z", 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:54:53Z" 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.793769_324.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.793769Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.793769_324.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.793769_324.1 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:58.793+0000 7f4c25dc0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:54:58.794+0000 7f4c25dc0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:54:58Z" 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.430 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.431 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.432 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:54:58Z" 2024-04-20T01:55:23.434 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.555632_296.1 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:04.104+0000 7f2737aa0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:04.106+0000 7f2737aa0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.717624_312.1", 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:55:04Z" 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.440 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.441 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.442 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.717624_312.1", 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.443 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:55:04Z" 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.717624_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.717624Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.717624_312.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.717624_312.1 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:09.238+0000 7f9faff91780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:09.239+0000 7f9faff91780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:55:09Z" 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.450 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.451 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.452 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.453 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.454 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:55:09Z" 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.738086_316.1 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:14.367+0000 7f9bf8d28780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:14.368+0000 7f9bf8d28780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:55:14Z" 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.461 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.463 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.464 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:55:14Z" 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:19.536+0000 7f1a552ba780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:19.546+0000 7f1a552ba780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:55:19Z" 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.471 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.472 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.473 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.474 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.475 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:55:19Z" 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.614143_305.1 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:24.783+0000 7ff6da420780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:24.785+0000 7ff6da420780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:55:24Z" 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.481 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.482 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.483 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.484 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.485 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:55:24Z" 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.606676_302.1 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:29.919+0000 7f3f8a453780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:29.920+0000 7f3f8a453780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:55:29Z" 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.492 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.495 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:55:29Z" 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:35.047+0000 7f559c411780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:35.048+0000 7f559c411780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:55:35Z" 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.502 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.503 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.580723_301.1", 2024-04-20T01:55:23.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.580723Z", 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:55:35Z" 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.506 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.580723_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.580723Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.580723_301.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.580723_301.1 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:40.180+0000 7fec463cc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:40.181+0000 7fec463cc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:55:40Z" 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.512 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.513 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.514 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.516 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:55:40Z" 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.606676_302.1 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:45.332+0000 7fc16e212780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:45.333+0000 7fc16e212780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.698420_310.1", 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:55:45Z" 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.522 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.523 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.524 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.698420_310.1", 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.525 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.526 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.527 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.527 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.527 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.527 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:55:45Z" 2024-04-20T01:55:23.527 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.527 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.698420_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.698420Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.698420_310.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.698420_310.1 2024-04-20T01:55:23.527 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:50.469+0000 7f19f2ea2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:50.470+0000 7f19f2ea2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:55:50Z" 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.533 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.534 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.535 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:55:50Z" 2024-04-20T01:55:23.537 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:55.715+0000 7fde48e63780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:55:55.716+0000 7fde48e63780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:55:55Z" 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.543 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.544 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.545 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.546 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.547 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:55:55Z" 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:00.853+0000 7f922d11a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:00.854+0000 7f922d11a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:56:00Z" 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.553 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.554 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.555 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.556 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.557 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:56:00Z" 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.555632_296.1 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:05.995+0000 7f451a72f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:05.996+0000 7f451a72f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:56:05Z" 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.563 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.564 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.565 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.566 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.567 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:56:05Z" 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.568 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:11.123+0000 7f4d8c83e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:11.124+0000 7f4d8c83e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:56:11Z" 2024-04-20T01:55:23.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.574 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.575 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.576 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.577 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:56:11Z" 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.555632_296.1 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:16.253+0000 7fcf34b68780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:16.254+0000 7fcf34b68780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:56:16Z" 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.584 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.585 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:23.586 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.587 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:56:16Z" 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.614143_305.1 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:21.380+0000 7f5015a1b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:21.382+0000 7f5015a1b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.717624_312.1", 2024-04-20T01:55:23.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:56:21Z" 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.595 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.596 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.597 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.717624_312.1", 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.717624Z", 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.598 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.599 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:56:21Z" 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.717624_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.717624Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.717624_312.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.717624_312.1 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:26.509+0000 7f0b3eefa780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:26.511+0000 7f0b3eefa780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:56:26Z" 2024-04-20T01:55:23.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.606 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.607 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.608 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.609 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:56:26Z" 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.609968_303.1 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:31.644+0000 7f038ecbc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:31.645+0000 7f038ecbc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:56:31Z" 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.616 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.617 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.618 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.619 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:56:31Z" 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.572795_298.1 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:36.771+0000 7f2c05316780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:36.772+0000 7f2c05316780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:56:36Z" 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.626 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.627 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.628 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.555632_296.1", 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.555632Z", 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.629 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:56:36Z" 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.630 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.555632_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.555632Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.555632_296.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.555632_296.1 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:41.900+0000 7f22ec4f4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:41.901+0000 7f22ec4f4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.611175_304.1", 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.611175Z", 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:56:41Z" 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.636 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.637 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.638 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.611175_304.1", 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.611175Z", 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.639 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.640 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:56:41Z" 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.611175_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.611175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.611175_304.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.611175_304.1 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:47.194+0000 7fcd9efa6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:47.195+0000 7fcd9efa6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:56:47Z" 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.647 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.648 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.649 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.650 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:56:47Z" 2024-04-20T01:55:23.651 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.609968_303.1 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:52.341+0000 7faef03e3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:52.343+0000 7faef03e3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:23.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:56:52Z" 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.657 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.658 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.659 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.660 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.661 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:56:52Z" 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.632785_307.1 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:57.481+0000 7ff169710780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:56:57.482+0000 7ff169710780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:56:57Z" 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.667 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.668 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.669 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.670 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.671 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:56:57Z" 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.672 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:02.631+0000 7f5c9cbfc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:02.632+0000 7f5c9cbfc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:57:02Z" 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.678 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.679 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.680 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.681 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:57:02Z" 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:07.776+0000 7fab12a78780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:07.777+0000 7fab12a78780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:57:07Z" 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.688 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.689 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.690 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.691 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:57:07Z" 2024-04-20T01:55:23.692 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.632785_307.1 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:12.918+0000 7f44df41c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:12.920+0000 7f44df41c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:57:12Z" 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.698 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.700 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.701 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.702 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:57:12Z" 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:18.192+0000 7fcea8339780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:18.195+0000 7fcea8339780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:57:18Z" 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.710 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.712 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:57:18Z" 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:23.326+0000 7f8cb1624780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:23.327+0000 7f8cb1624780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:57:23Z" 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.719 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.720 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:57:23Z" 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:28.485+0000 7f830d0ff780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:28.486+0000 7f830d0ff780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:57:28Z" 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.729 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.730 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.731 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.732 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.733 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:57:28Z" 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:33.648+0000 7f742f504780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:33.649+0000 7f742f504780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:57:33Z" 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.739 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.740 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.741 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.742 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.743 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:57:33Z" 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:38.808+0000 7f378b848780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:38.810+0000 7f378b848780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:57:38Z" 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.749 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.750 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.751 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.752 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.753 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:57:38Z" 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:43.962+0000 7f27d40b0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:43.963+0000 7f27d40b0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:57:43Z" 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:23.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.763 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:57:43Z" 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.577447_300.1 2024-04-20T01:55:23.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:49.107+0000 7f8e1b47a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:49.108+0000 7f8e1b47a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:57:49Z" 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.770 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.771 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.773 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:57:49Z" 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:54.243+0000 7f54a639d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:54.244+0000 7f54a639d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:57:54Z" 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.780 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.781 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:23.782 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.783 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:57:54Z" 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.784 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:59.399+0000 7f917a8ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:57:59.415+0000 7f917a8ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:57:59Z" 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.790 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.791 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.792 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.793 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.794 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:57:59Z" 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:04.688+0000 7f97ddc50780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:04.696+0000 7f97ddc50780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:58:04Z" 2024-04-20T01:55:23.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.802 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.803 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.804 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.805 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:58:04Z" 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:10.005+0000 7fe74ac4b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:10.006+0000 7fe74ac4b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.687754_309.1", 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:58:10Z" 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.812 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.813 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.687754_309.1", 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.814 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.815 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:58:10Z" 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.687754_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.687754Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.687754_309.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.687754_309.1 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:15.139+0000 7f1dfef6a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:15.140+0000 7f1dfef6a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:58:15Z" 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.822 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.823 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.825 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:58:15Z" 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.826 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:20.277+0000 7f1d976f5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:20.278+0000 7f1d976f5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:58:20Z" 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.832 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.834 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.836 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:58:20Z" 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.609968_303.1 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:25.414+0000 7fc80e0a4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:25.415+0000 7fc80e0a4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:58:25Z" 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.842 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.843 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.844 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.845 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.846 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:58:25Z" 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.606676_302.1 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:30.560+0000 7fc85ecb1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:30.561+0000 7fc85ecb1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:58:30Z" 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.852 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.853 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.854 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.554348_295.1", 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.554348Z", 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.855 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.856 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:58:30Z" 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.554348_295.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.554348Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.554348_295.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.554348_295.1 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:35.697+0000 7f2b6e422780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:35.698+0000 7f2b6e422780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.742452_317.1", 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.742452Z", 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:58:35Z" 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.863 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.742452_317.1", 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.742452Z", 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.866 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:58:35Z" 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.742452_317.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.742452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.742452_317.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.742452_317.1 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:40.841+0000 7fb7b96bc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:40.843+0000 7fb7b96bc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.793769_324.1", 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.793769Z", 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:58:40Z" 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.873 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.874 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.875 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.793769_324.1", 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.793769Z", 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.876 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:58:40Z" 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.877 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.793769_324.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.793769Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.793769_324.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.793769_324.1 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:45.996+0000 7f6e952c7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:45.998+0000 7f6e952c7780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:58:45Z" 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.883 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.884 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.885 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.609968_303.1", 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.609968Z", 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.886 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.887 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:58:45Z" 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.609968_303.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.609968Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.609968_303.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.609968_303.1 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:51.151+0000 7fe2b0a20780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:51.152+0000 7fe2b0a20780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:58:51Z" 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.893 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.894 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.896 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.897 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:58:51Z" 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.606676_302.1 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:56.280+0000 7f6311a32780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:58:56.281+0000 7f6311a32780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:58:56Z" 2024-04-20T01:55:23.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.904 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.905 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.906 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:58:56Z" 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.614143_305.1 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:59:01.541+0000 7f0b5eb80780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:59:01.543+0000 7f0b5eb80780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:59:01Z" 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.914 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.915 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.916 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.606676_302.1", 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.606676Z", 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:59:01Z" 2024-04-20T01:55:23.918 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.606676_302.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.606676Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.606676_302.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.606676_302.1 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:59:06.698+0000 7f60ed7e2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:59:06.699+0000 7f60ed7e2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:59:06Z" 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.924 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.925 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.926 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.577447_300.1", 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.577447Z", 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.927 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.928 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:59:06Z" 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.577447_300.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.577447Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.577447_300.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.577447_300.1 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:59:11.848+0000 7eff81138780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:59:11.849+0000 7eff81138780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:59:11Z" 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.934 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.935 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.936 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.572795_298.1", 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.572795Z", 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.937 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.938 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:59:11Z" 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.572795_298.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.572795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.572795_298.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.572795_298.1 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi031.stdout: INFO: 2024-04-19T23:59:14.529+0000 7fe8bf445700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:59:16.991+0000 7f4597930780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:59:16.992+0000 7f4597930780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:59:16Z" 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.945 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.946 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.576049_299.1", 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.947 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.576049Z", 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.948 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:59:16Z" 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.576049_299.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.576049Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.576049_299.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.576049_299.1 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:59:22.134+0000 7f6761611780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:59:22.135+0000 7f6761611780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:59:22Z" 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.955 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.956 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:23.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:59:22Z" 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.614143_305.1 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:59:27.269+0000 7f612d2c2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-19T23:59:27.270+0000 7f612d2c2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-19T23:59:27Z" 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.965 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.966 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.559046_297.1", 2024-04-20T01:55:23.967 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.559046Z", 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.968 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-19T23:59:27Z" 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.559046_297.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.559046Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:23.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570862.263117_452.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.559046_297.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713570862.263117_452.1 target=1_1713570242.559046_297.1 2024-04-20T01:55:23.970 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:23.970 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.970 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:23.970 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_zone_redirect 2024-04-20T01:55:23.971 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:23.971 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1061, in test_multi_zone_redirect 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:23.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "name": "test-zone2", 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "domain_root": "test-zone2.rgw.meta:root", 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "control_pool": "test-zone2.rgw.control", 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "gc_pool": "test-zone2.rgw.log:gc", 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "lc_pool": "test-zone2.rgw.log:lc", 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "log_pool": "test-zone2.rgw.log", 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "roles_pool": "test-zone2.rgw.meta:roles", 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-04-20T01:55:23.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "otp_pool": "test-zone2.rgw.otp", 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "system_key": { 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "access_key": "TWO98NBPIQJFTC6N", 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "secret_key": "WVtqgebnqURqV1qdDSut7FusWVqSrgr8" 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "placement_pools": [ 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": "default-placement", 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "index_pool": "test-zone2.rgw.buckets.index", 2024-04-20T01:55:23.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "storage_classes": { 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "STANDARD": { 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "data_pool": "test-zone2.rgw.buckets.data", 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "compression_type": "random" 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "index_type": 0, 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "inline_data": true 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ], 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "notif_pool": "test-zone2.rgw.log:notif" 2024-04-20T01:55:23.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: Sending period to new master zone 73f46f02-c7ab-4426-a267-2273af911da5 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "epoch": 4, 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": [], 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period_map": { 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "zonegroups": [ 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "id": "test-zonegroup", 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "name": "test-zonegroup", 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "api_name": "", 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "is_master": true, 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "endpoints": [ 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ], 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "hostnames": [], 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "hostnames_s3website": [], 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "zones": [ 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "name": "test-zone2", 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "endpoints": [ 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ], 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "log_meta": false, 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "log_data": true, 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "read_only": false, 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "tier_type": "", 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_from_all": false, 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_from": [], 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "supported_features": [ 2024-04-20T01:55:23.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "resharding" 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "name": "test-zone1", 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "endpoints": [ 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ], 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "log_meta": false, 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "log_data": true, 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "read_only": false, 2024-04-20T01:55:23.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "tier_type": "", 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_from_all": true, 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_from": [], 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "supported_features": [ 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "resharding" 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "name": "test-zone3", 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "endpoints": [ 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ], 2024-04-20T01:55:23.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "log_meta": false, 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "log_data": true, 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "read_only": false, 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "tier_type": "", 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_from_all": true, 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_from": [], 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "supported_features": [ 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "resharding" 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ], 2024-04-20T01:55:23.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "placement_targets": [ 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "name": "default-placement", 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "tags": [], 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "storage_classes": [ 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "STANDARD" 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ], 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "default_placement": "default-placement", 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_policy": { 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "groups": [] 2024-04-20T01:55:23.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "enabled_features": [ 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "resharding" 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ], 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "short_zone_ids": [ 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": 2010863401 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": 1221149675 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": 1478903101 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "master_zonegroup": "test-zonegroup", 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period_config": { 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket_quota": { 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "enabled": false, 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "check_on_raw": false, 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_size": -1, 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_size_kb": 0, 2024-04-20T01:55:23.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_objects": -1 2024-04-20T01:55:23.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "user_quota": { 2024-04-20T01:55:23.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "enabled": false, 2024-04-20T01:55:23.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "check_on_raw": false, 2024-04-20T01:55:23.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_size": -1, 2024-04-20T01:55:23.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_size_kb": 0, 2024-04-20T01:55:23.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_objects": -1 2024-04-20T01:55:23.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "user_ratelimit": { 2024-04-20T01:55:23.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:23.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:23.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "enabled": false 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket_ratelimit": { 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "enabled": false 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "anonymous_ratelimit": { 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:23.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "enabled": false 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_name": "test-realm", 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: Set sync_from_all flag on zone test-zone2 to false 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-48 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-48/ 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-48/ 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:23.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-48/ 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:00:32 GMT 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests:/aktlji-48/ 2024-04-20T01:55:23.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:9v2636GG6Ua7LLdBB0Jctb4PvWw= 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:00:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9v2636GG6Ua7LLdBB0Jctb4PvWw=', 'Content-Length': '0'} 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b062126b2bc8d60a-00662305a0-153e-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:00:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-48/testredirect 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-48/testredirect 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-48/testredirect 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:23.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:00:32 GMT 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:/aktlji-48/testredirect 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GeDC9kyOg3OrEFAmrztSNFpi8X0= 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:00:32 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GeDC9kyOg3OrEFAmrztSNFpi8X0='} 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000277fc3e1cc2baf7e-00662305a0-153e-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:00:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:00:32.655231Z" 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:00:32.968+0000 7f3feb595780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:00:32.969+0000 7f3feb595780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:00:32Z" 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:23.997 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:23.998 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:23.999 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.000 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.001 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.001 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.001 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:00:32Z" 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:00:38.099+0000 7f65752f3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:00:38.100+0000 7f65752f3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.808187_326.1", 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:00:38Z" 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.009 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.010 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.808187_326.1", 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:00:38Z" 2024-04-20T01:55:24.013 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.808187_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.808187Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.808187_326.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.808187_326.1 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:00:43.244+0000 7f83eac58780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:00:43.246+0000 7f83eac58780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.687754_309.1", 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-20T01:55:24.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:00:43Z" 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.019 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.020 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.021 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.687754_309.1", 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.687754Z", 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.022 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.023 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:00:43Z" 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.687754_309.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.687754Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.687754_309.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.687754_309.1 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:00:48.372+0000 7f8e33a59780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:00:48.373+0000 7f8e33a59780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:00:48Z" 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.029 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.030 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.031 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.032 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.033 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:00:48Z" 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:24.034 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:00:53.500+0000 7f6e9f7f0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:00:53.501+0000 7f6e9f7f0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:00:53Z" 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.040 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.041 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.042 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.043 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:00:53Z" 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.738086_316.1 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:00:58.725+0000 7fa6b9581780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:00:58.726+0000 7fa6b9581780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:00:58Z" 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.050 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.051 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.052 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.053 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.054 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:00:58Z" 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.632785_307.1 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:03.920+0000 7f8b0523e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:03.922+0000 7f8b0523e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:01:03Z" 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.060 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.063 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.064 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:01:03Z" 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:09.080+0000 7f654f3dd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:09.081+0000 7f654f3dd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:01:09Z" 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.071 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.072 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.073 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:01:09Z" 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:14.230+0000 7f7acefe5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:14.232+0000 7f7acefe5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:01:14Z" 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.081 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.082 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.083 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.084 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:01:14Z" 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.085 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.738086_316.1 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:19.366+0000 7f00467bb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:19.367+0000 7f00467bb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:01:19Z" 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.091 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.092 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.093 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.094 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.095 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:01:19Z" 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:24.499+0000 7fd9d5d9b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:24.501+0000 7fd9d5d9b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:01:24Z" 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.101 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.102 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.102 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.102 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.102 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.102 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.102 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.102 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.102 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.102 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.102 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.102 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.103 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.104 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.105 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:01:24Z" 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:29.646+0000 7f872ce5a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:29.647+0000 7f872ce5a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.793769_324.1", 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.793769Z", 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:01:29Z" 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.112 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.113 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.793769_324.1", 2024-04-20T01:55:24.114 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.793769Z", 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.115 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:01:29Z" 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.793769_324.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.793769Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.793769_324.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.793769_324.1 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:34.794+0000 7f574a63b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:34.795+0000 7f574a63b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:01:34Z" 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.122 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.123 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.124 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.125 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.126 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:01:34Z" 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:39.947+0000 7fe94d7dc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:39.949+0000 7fe94d7dc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.698420_310.1", 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:01:39Z" 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.132 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.133 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.134 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.698420_310.1", 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.135 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.136 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:01:39Z" 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.698420_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.698420Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.698420_310.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.698420_310.1 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:45.169+0000 7ff97cecd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:45.170+0000 7ff97cecd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:24.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:01:45Z" 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.143 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.144 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.629562_306.1", 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.629562Z", 2024-04-20T01:55:24.145 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.146 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:01:45Z" 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.629562_306.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.629562Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.629562_306.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.629562_306.1 2024-04-20T01:55:24.147 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:50.309+0000 7fd927cb2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:50.310+0000 7fd927cb2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:01:50Z" 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.153 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.154 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.682721_308.1", 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.155 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.682721Z", 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.156 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:01:50Z" 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.682721_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.682721Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.682721_308.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.682721_308.1 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:55.445+0000 7fd3e20b9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:01:55.446+0000 7fd3e20b9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:01:55Z" 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.163 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.164 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.165 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.738086_316.1", 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.738086Z", 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.166 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:01:55Z" 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.167 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.738086_316.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.738086Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.738086_316.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.738086_316.1 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:00.590+0000 7f1665e40780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:00.591+0000 7f1665e40780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.701491_311.1", 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.701491Z", 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:02:00Z" 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.173 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.174 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.175 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.701491_311.1", 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.701491Z", 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.176 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:02:00Z" 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.177 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.701491_311.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.701491Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.701491_311.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.701491_311.1 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:05.740+0000 7f0c1ab45780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:05.742+0000 7f0c1ab45780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:02:05Z" 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.184 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.185 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.632785_307.1", 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.632785Z", 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.186 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.187 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:02:05Z" 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.632785_307.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.632785Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.632785_307.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.632785_307.1 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:10.902+0000 7fe1363e6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:10.903+0000 7fe1363e6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:02:10Z" 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.193 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.194 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.195 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.614143_305.1", 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.614143Z", 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.196 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.197 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:02:10Z" 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.614143_305.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.614143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.614143_305.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.614143_305.1 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:16.161+0000 7f08d49a3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:16.165+0000 7f08d49a3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.698420_310.1", 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-20T01:55:24.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:02:16Z" 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.207 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.208 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.698420_310.1", 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.209 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.698420Z", 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.210 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:02:16Z" 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.698420_310.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.698420Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.698420_310.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.211 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.698420_310.1 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:21.310+0000 7f9e8c6b9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:21.311+0000 7f9e8c6b9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.973078_341.1", 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.973078Z", 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:02:21Z" 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.217 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.218 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.218 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.218 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.218 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.218 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.218 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.218 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.218 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.218 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.218 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.219 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.219 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.219 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.219 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.219 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.219 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.219 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.219 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.219 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.220 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.220 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.220 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.220 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.220 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.220 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.220 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.973078_341.1", 2024-04-20T01:55:24.220 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.221 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.221 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.221 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.973078Z", 2024-04-20T01:55:24.221 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.221 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.221 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.221 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.221 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.221 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.222 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.222 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.222 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.222 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.222 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.222 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.222 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.223 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.223 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.223 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.223 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.223 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.223 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.223 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.223 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:02:21Z" 2024-04-20T01:55:24.224 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.224 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.973078_341.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.973078Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.973078_341.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.973078_341.1 2024-04-20T01:55:24.224 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.224 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:26.453+0000 7f9e20b6f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:26.454+0000 7f9e20b6f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:02:26Z" 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.229 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.230 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.231 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.232 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.233 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:02:26Z" 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:31.591+0000 7fef5a1b3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:31.592+0000 7fef5a1b3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.907433_331.1", 2024-04-20T01:55:24.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.907433Z", 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:02:31Z" 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.239 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.240 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.241 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.907433_331.1", 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.907433Z", 2024-04-20T01:55:24.242 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.243 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:02:31Z" 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.907433_331.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.907433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.907433_331.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.907433_331.1 2024-04-20T01:55:24.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:36.735+0000 7fd737fd8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:36.736+0000 7fd737fd8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.987431_342.1", 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.987431Z", 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:02:36Z" 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.250 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.987431_342.1", 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.252 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.987431Z", 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.253 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:02:36Z" 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.987431_342.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.987431Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.987431_342.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.987431_342.1 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:41.884+0000 7f6d01543780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:41.887+0000 7f6d01543780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.868452_328.1", 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:02:41Z" 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.260 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.261 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.262 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.868452_328.1", 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.263 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:02:41Z" 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.264 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.868452_328.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.868452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.868452_328.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.868452_328.1 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:47.016+0000 7fb95287e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:47.017+0000 7fb95287e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.987431_342.1", 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.987431Z", 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:02:47Z" 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.271 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.272 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.987431_342.1", 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.987431Z", 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.273 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.274 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:02:47Z" 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.987431_342.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.987431Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.987431_342.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.987431_342.1 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:52.147+0000 7f8729515780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:52.148+0000 7f8729515780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.968897_339.1", 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.968897Z", 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:02:52Z" 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.280 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.282 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.968897_339.1", 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.968897Z", 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.283 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.284 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:02:52Z" 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.968897_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.968897Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.968897_339.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.968897_339.1 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:57.287+0000 7f1cf2f02780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:02:57.288+0000 7f1cf2f02780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.868452_328.1", 2024-04-20T01:55:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:02:57Z" 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.291 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.292 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.868452_328.1", 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T01:55:24.293 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.294 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:02:57Z" 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.868452_328.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.868452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.868452_328.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.868452_328.1 2024-04-20T01:55:24.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:02.433+0000 7f61025d0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:02.434+0000 7f61025d0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:03:02Z" 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.301 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.302 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.303 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.304 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:03:02Z" 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:07.679+0000 7f2a8f94d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:07.680+0000 7f2a8f94d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:03:07Z" 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.311 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.312 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:03:07Z" 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.315 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:12.833+0000 7f54499ef780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:12.834+0000 7f54499ef780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.868452_328.1", 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:03:12Z" 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.321 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.322 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.323 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.868452_328.1", 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.324 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.325 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:03:12Z" 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.868452_328.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.868452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.868452_328.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.868452_328.1 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:17.977+0000 7fe94e2ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:17.978+0000 7fe94e2ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.907433_331.1", 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.907433Z", 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:03:17Z" 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.331 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.333 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.907433_331.1", 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.907433Z", 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.334 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.335 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:03:17Z" 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.907433_331.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.907433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.907433_331.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.907433_331.1 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:23.129+0000 7f6671dda780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:23.130+0000 7f6671dda780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.811395_327.1", 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:03:23Z" 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.342 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.343 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.344 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.811395_327.1", 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.345 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.346 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:03:23Z" 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.811395_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.811395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.811395_327.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.811395_327.1 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:28.387+0000 7f1cfc42c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:28.388+0000 7f1cfc42c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.907433_331.1", 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.907433Z", 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:03:28Z" 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.352 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.353 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.354 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.907433_331.1", 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.907433Z", 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.355 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.356 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:03:28Z" 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.907433_331.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.907433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.907433_331.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.907433_331.1 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:33.530+0000 7fbf9ba21780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:33.531+0000 7fbf9ba21780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.907433_331.1", 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.907433Z", 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:03:33Z" 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.362 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.363 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.364 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.907433_331.1", 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.907433Z", 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.365 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.366 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:03:33Z" 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.907433_331.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.907433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.907433_331.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.907433_331.1 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.367 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:38.672+0000 7f0064f67780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:38.673+0000 7f0064f67780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570245.107533_348.1", 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:05.107533Z", 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:03:38Z" 2024-04-20T01:55:24.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.373 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.374 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570245.107533_348.1", 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.375 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:05.107533Z", 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.376 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:03:38Z" 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570245.107533_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:05.107533Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.107533_348.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570245.107533_348.1 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:43.816+0000 7f9f3923e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:43.817+0000 7f9f3923e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.808187_326.1", 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:03:43Z" 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.383 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.384 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.385 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.808187_326.1", 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.386 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.387 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:03:43Z" 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.808187_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.808187Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.808187_326.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.808187_326.1 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:49.025+0000 7fb866639780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:49.026+0000 7fb866639780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.970082_340.1", 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.970082Z", 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:03:49Z" 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.393 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.394 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.395 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.970082_340.1", 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.970082Z", 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.396 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.397 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:03:49Z" 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.970082_340.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.970082Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.970082_340.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.970082_340.1 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:54.162+0000 7f6acc971780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:54.163+0000 7f6acc971780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:03:54Z" 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.403 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.404 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.406 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:03:54Z" 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:59.291+0000 7f689f865780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:03:59.292+0000 7f689f865780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570245.087989_346.1", 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:05.087989Z", 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:03:59Z" 2024-04-20T01:55:24.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.416 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570245.087989_346.1", 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:05.087989Z", 2024-04-20T01:55:24.418 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.419 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:03:59Z" 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570245.087989_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:05.087989Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.087989_346.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570245.087989_346.1 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:04.438+0000 7f3e1685f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:04.439+0000 7f3e1685f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.877554_329.1", 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.877554Z", 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:04:04Z" 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.426 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.877554_329.1", 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.877554Z", 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.429 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:04:04Z" 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.877554_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.877554Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.877554_329.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.877554_329.1 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:09.592+0000 7f5e1be85780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:09.593+0000 7f5e1be85780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.811395_327.1", 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:04:09Z" 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.436 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.437 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.811395_327.1", 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.438 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.439 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:04:09Z" 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.811395_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.811395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.811395_327.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.811395_327.1 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:14.730+0000 7fe5c642b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:14.731+0000 7fe5c642b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.936115_334.1", 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.936115Z", 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:04:14Z" 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.446 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.447 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.448 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.936115_334.1", 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.936115Z", 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.449 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:04:14Z" 2024-04-20T01:55:24.450 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.936115_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.936115Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.936115_334.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.936115_334.1 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:19.858+0000 7fc240e36780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:19.859+0000 7fc240e36780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:04:19Z" 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.457 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.458 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.460 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:04:19Z" 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:25.000+0000 7ff740778780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:25.002+0000 7ff740778780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.868452_328.1", 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:04:25Z" 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.467 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.468 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.868452_328.1", 2024-04-20T01:55:24.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.868452Z", 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:04:25Z" 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.868452_328.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.868452Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.868452_328.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.868452_328.1 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:30.136+0000 7fc795779780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:30.137+0000 7fc795779780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570245.114997_351.1", 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:05.114997Z", 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:04:30Z" 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.477 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.478 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.479 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570245.114997_351.1", 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:05.114997Z", 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.480 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:04:30Z" 2024-04-20T01:55:24.481 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570245.114997_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:05.114997Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570245.114997_351.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570245.114997_351.1 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:35.277+0000 7f2585b1b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:35.279+0000 7f2585b1b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:04:35Z" 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.487 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.488 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.489 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.490 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.491 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:04:35Z" 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:40.420+0000 7ff5a313b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:40.421+0000 7ff5a313b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.808187_326.1", 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:04:40Z" 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.498 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.499 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.808187_326.1", 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T01:55:24.500 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.501 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:04:40Z" 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.808187_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.808187Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.808187_326.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.808187_326.1 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:45.647+0000 7fb19d54a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:45.648+0000 7fb19d54a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.914458_333.1", 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.914458Z", 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:04:45Z" 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.508 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.509 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.914458_333.1", 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.510 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.914458Z", 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.511 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:04:45Z" 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.914458_333.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.914458Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.914458_333.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.914458_333.1 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:50.797+0000 7fdb93f6c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:50.799+0000 7fdb93f6c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.811395_327.1", 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:04:50Z" 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.518 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.519 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.520 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.811395_327.1", 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.521 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:04:50Z" 2024-04-20T01:55:24.522 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.811395_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.811395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.811395_327.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.811395_327.1 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:55.933+0000 7f21b8fc0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:04:55.934+0000 7f21b8fc0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:04:55Z" 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.528 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.529 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.530 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.531 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.532 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:04:55Z" 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:01.075+0000 7febce631780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:01.077+0000 7febce631780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.909976_332.1", 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.909976Z", 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:05:01Z" 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.538 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.539 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.540 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.909976_332.1", 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.909976Z", 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.541 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.542 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:05:01Z" 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.909976_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.909976Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.909976_332.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.909976_332.1 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:06.217+0000 7f22535f3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:06.218+0000 7f22535f3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.877554_329.1", 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.877554Z", 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:05:06Z" 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.549 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.550 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.877554_329.1", 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.551 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.877554Z", 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.552 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:05:06Z" 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.877554_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.877554Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.877554_329.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.877554_329.1 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:11.349+0000 7f001c795780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:11.351+0000 7f001c795780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.808187_326.1", 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:05:11Z" 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.559 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.560 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.561 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.808187_326.1", 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.808187Z", 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.562 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:05:11Z" 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.563 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.808187_326.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.808187Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.808187_326.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.808187_326.1 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:16.483+0000 7f367c78e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:16.485+0000 7f367c78e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.942889_337.1", 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.942889Z", 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:05:16Z" 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.570 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.571 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.942889_337.1", 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.942889Z", 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.572 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.573 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.574 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:05:16Z" 2024-04-20T01:55:24.574 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.574 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.942889_337.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.942889Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.942889_337.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.942889_337.1 2024-04-20T01:55:24.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:21.620+0000 7f81dc95c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:21.622+0000 7f81dc95c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.811395_327.1", 2024-04-20T01:55:24.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:05:21Z" 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.579 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.580 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.581 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.811395_327.1", 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.811395Z", 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.582 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.583 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:05:21Z" 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.811395_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.811395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.811395_327.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.811395_327.1 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.584 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:26.765+0000 7ff339dda780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:26.766+0000 7ff339dda780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:05:26Z" 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.590 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.591 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.797017_325.1", 2024-04-20T01:55:24.592 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.797017Z", 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:05:26Z" 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.797017_325.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.797017Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.797017_325.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.797017_325.1 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:31.904+0000 7f2ce65af780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:31.906+0000 7f2ce65af780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.877554_329.1", 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.877554Z", 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:05:31Z" 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.600 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.601 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.602 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.877554_329.1", 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.877554Z", 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.603 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.604 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:05:31Z" 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.877554_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.877554Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.877554_329.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.877554_329.1 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:37.056+0000 7fe6ff346780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:05:37.058+0000 7fe6ff346780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713570242.968897_339.1", 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:44:02.968897Z", 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:05:37Z" 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.610 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.611 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.612 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.612 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.612 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.612 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.612 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.613 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.614 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: "marker": "1_1713570242.968897_339.1", 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:44:02.968897Z", 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.615 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:05:37Z" 2024-04-20T01:55:24.616 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.617 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713570242.968897_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:44:02.968897Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713570242.968897_339.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571232.655231_514.1 target=1_1713570242.968897_339.1 2024-04-20T01:55:24.617 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:24.617 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.617 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:24.618 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable 2024-04-20T01:55:24.618 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:24.621 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:24.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:24.621 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:24.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1181, in test_bucket_sync_disable 2024-04-20T01:55:24.621 INFO:tasks.rgw_multisite_tests: zonegroup_data_checkpoint(zonegroup_conns) 2024-04-20T01:55:24.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 375, in zonegroup_data_checkpoint 2024-04-20T01:55:24.621 INFO:tasks.rgw_multisite_tests: zone_data_checkpoint(target_conn.zone, source_conn.zone) 2024-04-20T01:55:24.621 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 366, in zone_data_checkpoint 2024-04-20T01:55:24.621 INFO:tasks.rgw_multisite_tests: assert False, 'failed data checkpoint for target_zone=%s source_zone=%s' % \ 2024-04-20T01:55:24.621 INFO:tasks.rgw_multisite_tests:AssertionError: failed data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:24.621 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-55 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-55/ 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-55/ 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-55/ 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:24.622 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:06:41 GMT 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:/aktlji-55/ 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+5KPnXY2rfd2jt3X+akmKE1pcc8= 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+5KPnXY2rfd2jt3X+akmKE1pcc8=', 'Content-Length': '0'} 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002afe204477f910b2-0066230711-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:06:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=aktlji-56 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-56/ 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-56/ 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-56/ 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:24.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:06:41 GMT 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests:/aktlji-56/ 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:jiHKUF81rP8trOToOtuspjXkhQo= 2024-04-20T01:55:24.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:jiHKUF81rP8trOToOtuspjXkhQo=', 'Content-Length': '0'} 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002d5869ccd9b32e11-0066230711-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:06:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=aktlji-57 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-57/ 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-57/ 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-57/ 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:24.625 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:06:41 GMT 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests:/aktlji-57/ 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rbM7pVaE/9RCizk4mqycHUOF7mg= 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:06:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rbM7pVaE/9RCizk4mqycHUOF7mg=', 'Content-Length': '0'} 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000056145bc34cd8efe7-0066230711-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:06:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713571601.228671_641.1", 2024-04-20T01:55:24.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:06:41.228671Z" 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:06:41.477+0000 7f1993e6f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:06:41.478+0000 7f1993e6f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:06:41Z" 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.633 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.635 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.636 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.637 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:06:41Z" 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:06:46.616+0000 7f050aed5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:06:46.617+0000 7f050aed5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:06:46Z" 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.645 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.646 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.647 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.648 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:06:46Z" 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:06:51.760+0000 7fcfd2bdd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:06:51.761+0000 7fcfd2bdd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:06:51Z" 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.654 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.655 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.656 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.657 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.658 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:06:51Z" 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:06:56.910+0000 7f86e4b27780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:06:56.911+0000 7f86e4b27780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:06:56Z" 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.667 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.668 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:06:56Z" 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:02.043+0000 7fae588b3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:02.044+0000 7fae588b3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:07:02Z" 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.675 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.676 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.677 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.678 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:07:02Z" 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.679 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:07.196+0000 7f83382e8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:07.197+0000 7f83382e8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:07:07Z" 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.685 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.687 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.688 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.689 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.690 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.690 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.690 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:07:07Z" 2024-04-20T01:55:24.690 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.690 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.690 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.690 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:12.344+0000 7fc909adf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:12.345+0000 7fc909adf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:07:12Z" 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.696 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.697 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.699 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.700 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.700 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.700 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.700 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.700 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.700 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.700 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.700 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:07:12Z" 2024-04-20T01:55:24.700 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.700 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.700 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:17.474+0000 7fda18899780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:17.476+0000 7fda18899780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:07:17Z" 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.706 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.707 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.708 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.709 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:07:17Z" 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.710 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:22.614+0000 7f42def7e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:22.615+0000 7f42def7e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:07:22Z" 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.716 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.717 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.718 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.719 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.720 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:07:22Z" 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:27.760+0000 7f6be4d6a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:27.761+0000 7f6be4d6a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:07:27Z" 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.726 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.728 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.729 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.730 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:07:27Z" 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.731 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:32.888+0000 7f17b730c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:32.889+0000 7f17b730c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:07:32Z" 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.737 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.738 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.739 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.740 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:07:32Z" 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:38.017+0000 7fb0f09c5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:38.018+0000 7fb0f09c5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:07:38Z" 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.747 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.748 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.749 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:07:38Z" 2024-04-20T01:55:24.751 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:43.168+0000 7f20d47e5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:43.169+0000 7f20d47e5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:07:43Z" 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.757 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.758 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.759 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.760 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.761 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:07:43Z" 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:48.305+0000 7fdef4114780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:48.306+0000 7fdef4114780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:07:48Z" 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.768 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.769 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.771 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:07:48Z" 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:53.435+0000 7fdf295a8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:53.436+0000 7fdf295a8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:07:53Z" 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.778 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.779 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.780 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.781 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:07:53Z" 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.782 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:58.591+0000 7fb7af6bc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:07:58.592+0000 7fb7af6bc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:07:58Z" 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.788 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.789 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.790 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.791 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.792 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:07:58Z" 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:03.727+0000 7fd227e22780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:03.728+0000 7fd227e22780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:08:03Z" 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.799 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.800 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.801 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:08:03Z" 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:08.869+0000 7fbcae659780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:08.870+0000 7fbcae659780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:08:08Z" 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.809 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.810 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.811 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.812 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:08:08Z" 2024-04-20T01:55:24.813 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:14.004+0000 7fb59434d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:14.006+0000 7fb59434d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:08:14Z" 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.821 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.824 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.825 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:08:14Z" 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:19.149+0000 7f61cbc61780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:19.150+0000 7f61cbc61780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:08:19Z" 2024-04-20T01:55:24.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.832 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.833 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:08:19Z" 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.836 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:24.289+0000 7f099d08f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:24.290+0000 7f099d08f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:08:24Z" 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.842 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.843 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.844 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:08:24Z" 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.846 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:29.425+0000 7fc2ca07a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:29.426+0000 7fc2ca07a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:08:29Z" 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.852 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.853 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.854 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.855 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.856 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:08:29Z" 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:34.572+0000 7fd3ea5e4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:34.573+0000 7fd3ea5e4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:08:34Z" 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.862 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.863 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.864 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.865 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.865 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.865 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.865 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.865 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.865 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.865 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.865 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:08:34Z" 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:39.703+0000 7f38a7870780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:39.705+0000 7f38a7870780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:08:39Z" 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.873 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.874 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.875 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571232.655231_514.1", 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:00:32.655231Z", 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.876 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:08:39Z" 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571232.655231_514.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:00:32.655231Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571232.655231_514.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571232.655231_514.1 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:44.847+0000 7fa697f25780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:44.848+0000 7fa697f25780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571598.305342_610.1", 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:38.305342Z", 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:08:44Z" 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.883 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.884 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.885 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571598.305342_610.1", 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:38.305342Z", 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.886 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.887 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:08:44Z" 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.305342_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.305342Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.305342_610.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.305342_610.1 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:50.001+0000 7fee43b91780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:50.003+0000 7fee43b91780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571597.910175_581.1", 2024-04-20T01:55:24.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:37.910175Z", 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:08:50Z" 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.893 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.894 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.895 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571597.910175_581.1", 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:37.910175Z", 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.896 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.897 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:08:50Z" 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571597.910175_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:37.910175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571597.910175_581.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571597.910175_581.1 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.898 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:55.151+0000 7fbf654f3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:08:55.153+0000 7fbf654f3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571598.253928_606.1", 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:38.253928Z", 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:08:55Z" 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.904 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571598.253928_606.1", 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:38.253928Z", 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:08:55Z" 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.253928_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.253928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.253928_606.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.253928_606.1 2024-04-20T01:55:24.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:00.283+0000 7f4f0977f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:00.284+0000 7f4f0977f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571598.253928_606.1", 2024-04-20T01:55:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:38.253928Z", 2024-04-20T01:55:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:09:00Z" 2024-04-20T01:55:24.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.915 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.915 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.916 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.916 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.916 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.916 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.916 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.916 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.916 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.916 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.916 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.916 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.916 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.917 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.917 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.917 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.917 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.917 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.917 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.917 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.917 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.917 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.917 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.918 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.918 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.918 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.918 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.918 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.918 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.919 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571598.253928_606.1", 2024-04-20T01:55:24.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.919 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:38.253928Z", 2024-04-20T01:55:24.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.919 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.919 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.919 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.919 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.919 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.920 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.920 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.920 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.920 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.920 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.920 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.920 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.920 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.920 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.920 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.921 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.921 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.921 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.921 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:09:00Z" 2024-04-20T01:55:24.921 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.921 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.253928_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.253928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.253928_606.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.253928_606.1 2024-04-20T01:55:24.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:05.424+0000 7fe3dc27d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:05.426+0000 7fe3dc27d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571598.253928_606.1", 2024-04-20T01:55:24.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:38.253928Z", 2024-04-20T01:55:24.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:09:05Z" 2024-04-20T01:55:24.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.928 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.928 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.928 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.928 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.928 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.929 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.929 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.929 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.929 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.929 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.929 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.929 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.929 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.930 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.930 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.930 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.930 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.930 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.930 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.930 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.930 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.930 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.931 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.931 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.931 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.931 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.931 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.931 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.931 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.931 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.931 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571598.253928_606.1", 2024-04-20T01:55:24.931 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.931 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.932 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.932 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:38.253928Z", 2024-04-20T01:55:24.932 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.932 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.932 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.932 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.932 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.932 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.932 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.932 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.932 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.933 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.933 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.933 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.933 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.933 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.933 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.933 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.933 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.933 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.933 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.933 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.934 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.934 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:09:05Z" 2024-04-20T01:55:24.934 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.934 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.253928_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.253928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.253928_606.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.253928_606.1 2024-04-20T01:55:24.934 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.934 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:10.568+0000 7f84023ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:10.569+0000 7f84023ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571598.263539_607.1", 2024-04-20T01:55:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:38.263539Z", 2024-04-20T01:55:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:09:10Z" 2024-04-20T01:55:24.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.941 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.941 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.941 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.941 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.941 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.941 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.941 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.942 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.942 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.942 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.942 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.942 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.942 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.942 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.942 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.942 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.942 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.942 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.943 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.943 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.943 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.943 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.943 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.943 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.943 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.943 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.943 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.943 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.943 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.944 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.944 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.944 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.944 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.944 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.944 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.944 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571598.263539_607.1", 2024-04-20T01:55:24.944 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.944 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.944 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.944 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:38.263539Z", 2024-04-20T01:55:24.945 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.945 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.945 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.945 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.945 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.945 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.945 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.945 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.945 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.945 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.945 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.946 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.946 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.946 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.946 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.946 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.946 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.946 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.946 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.946 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.946 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:09:10Z" 2024-04-20T01:55:24.947 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.947 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.263539_607.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.263539Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.263539_607.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.263539_607.1 2024-04-20T01:55:24.947 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.947 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:15.731+0000 7f6b7cbfe780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:15.732+0000 7f6b7cbfe780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571598.253928_606.1", 2024-04-20T01:55:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:38.253928Z", 2024-04-20T01:55:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:09:15Z" 2024-04-20T01:55:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.954 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.954 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.954 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.954 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.954 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.954 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.954 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.954 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.954 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.954 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.954 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.955 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.955 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.955 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.955 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.955 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.955 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.955 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.955 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.955 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.955 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.955 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.956 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.956 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.956 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.956 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.956 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.956 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.956 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.956 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.956 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.956 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571598.253928_606.1", 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:38.253928Z", 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.957 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.958 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.958 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.958 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.958 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.958 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.958 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.958 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.958 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.958 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.959 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.959 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.959 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.959 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:09:15Z" 2024-04-20T01:55:24.959 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.959 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.253928_606.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.253928Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.253928_606.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.253928_606.1 2024-04-20T01:55:24.960 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.960 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:20.942+0000 7f64c3101780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:20.943+0000 7f64c3101780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571597.910175_581.1", 2024-04-20T01:55:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:37.910175Z", 2024-04-20T01:55:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:09:20Z" 2024-04-20T01:55:24.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.966 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.966 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.966 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.966 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.966 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.967 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.967 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.967 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.967 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.967 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.967 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.967 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.967 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.967 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.967 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.967 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.967 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.968 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.968 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.968 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.968 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.968 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.968 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.968 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.968 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.968 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.968 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.968 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.969 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.969 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.969 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.969 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.969 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.969 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.969 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571597.910175_581.1", 2024-04-20T01:55:24.969 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.969 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.969 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.970 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:37.910175Z", 2024-04-20T01:55:24.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.970 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.970 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.970 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.970 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.970 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.970 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.971 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.971 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.971 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.971 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.971 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.971 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.971 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.971 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.972 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.972 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:09:20Z" 2024-04-20T01:55:24.972 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.972 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.972 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571597.910175_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:37.910175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.972 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.972 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571597.910175_581.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.972 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571597.910175_581.1 2024-04-20T01:55:24.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:26.104+0000 7f73b1a27780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:26.106+0000 7f73b1a27780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571598.266030_608.1", 2024-04-20T01:55:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:38.266030Z", 2024-04-20T01:55:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:09:26Z" 2024-04-20T01:55:24.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.979 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.979 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.979 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.979 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.979 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.979 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.979 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.980 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.980 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.980 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.980 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.980 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.980 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.980 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.980 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.981 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.981 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.981 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.981 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.981 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.981 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.981 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.982 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.982 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.982 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.982 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.982 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.982 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571598.266030_608.1", 2024-04-20T01:55:24.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:38.266030Z", 2024-04-20T01:55:24.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.983 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.983 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.983 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.983 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.983 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.983 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.984 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.984 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.984 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.984 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.984 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.984 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.984 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.984 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.984 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:09:26Z" 2024-04-20T01:55:24.984 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.985 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.266030_608.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.266030Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.266030_608.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.266030_608.1 2024-04-20T01:55:24.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:31.246+0000 7fefd66c4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:31.247+0000 7fefd66c4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571597.919367_583.1", 2024-04-20T01:55:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:37.919367Z", 2024-04-20T01:55:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:24.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:24.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:24.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:24.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:09:31Z" 2024-04-20T01:55:24.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:24.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:24.992 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:24.992 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:24.992 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:24.992 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:24.992 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.992 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.992 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:24.992 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.992 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:24.992 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.993 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.993 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:24.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.993 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:24.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.993 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:24.993 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.993 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.993 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.993 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:24.994 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.994 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.994 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:24.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.994 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.994 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:24.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.994 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.994 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.994 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.995 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:24.995 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.995 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.995 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571597.919367_583.1", 2024-04-20T01:55:24.995 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.995 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.995 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.995 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:37.919367Z", 2024-04-20T01:55:24.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.995 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.995 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.995 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:24.996 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:24.996 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:24.996 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:24.996 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:24.996 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:24.996 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:24.996 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:24.996 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:24.996 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:24.996 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.997 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:24.997 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:24.997 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.997 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:24.997 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:24.997 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:24.997 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:24.997 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:09:31Z" 2024-04-20T01:55:24.997 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:24.997 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:24.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571597.919367_583.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:37.919367Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:24.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571597.919367_583.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:24.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571597.919367_583.1 2024-04-20T01:55:24.998 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.998 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:24.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:24.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:24.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:24.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:36.398+0000 7f56f928b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:36.399+0000 7f56f928b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:24.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571598.106288_599.1", 2024-04-20T01:55:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:38.106288Z", 2024-04-20T01:55:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:25.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:25.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:09:36Z" 2024-04-20T01:55:25.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:25.004 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.004 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.004 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.004 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.004 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.004 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.005 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.005 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.005 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.005 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.005 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.005 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.005 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.005 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:25.005 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.005 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.005 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.006 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.006 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.006 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.006 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.006 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.006 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.006 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:25.006 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.006 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.006 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.007 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:25.007 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.007 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.007 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.007 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.007 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.007 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.007 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.007 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571598.106288_599.1", 2024-04-20T01:55:25.007 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.007 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.007 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.008 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:38.106288Z", 2024-04-20T01:55:25.008 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.008 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.008 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.008 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.008 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.008 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.008 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.008 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.008 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.008 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.008 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.009 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.009 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.009 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.009 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.009 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.009 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.009 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:25.009 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:25.009 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.009 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:09:36Z" 2024-04-20T01:55:25.009 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.010 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.106288_599.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.106288Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:25.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.106288_599.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.106288_599.1 2024-04-20T01:55:25.010 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.010 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:41.540+0000 7ff1867e5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:41.541+0000 7ff1867e5780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571598.305342_610.1", 2024-04-20T01:55:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:38.305342Z", 2024-04-20T01:55:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:09:41Z" 2024-04-20T01:55:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:25.016 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.016 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.017 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.017 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.017 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.017 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.017 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.017 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.017 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.017 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.017 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.017 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.017 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.017 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.018 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:25.018 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.018 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.018 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.018 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.018 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.018 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.018 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.018 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.018 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.019 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:25.019 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.019 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.019 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.019 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:25.019 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.019 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.019 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.019 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.019 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.019 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.019 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.020 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571598.305342_610.1", 2024-04-20T01:55:25.020 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.020 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.020 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.020 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:38.305342Z", 2024-04-20T01:55:25.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.020 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.020 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.020 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.020 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.020 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.021 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.021 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.021 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.021 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.021 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.021 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.021 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.021 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.021 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.021 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.021 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.022 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.022 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:25.022 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:25.022 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.022 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:09:41Z" 2024-04-20T01:55:25.022 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.022 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.305342_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.305342Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:25.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.305342_610.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.305342_610.1 2024-04-20T01:55:25.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:46.677+0000 7f441d8e6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:46.678+0000 7f441d8e6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:25.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:25.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571597.910175_581.1", 2024-04-20T01:55:25.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:37.910175Z", 2024-04-20T01:55:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:25.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:25.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:09:46Z" 2024-04-20T01:55:25.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:25.032 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.032 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.032 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.032 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.032 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.032 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.033 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.033 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.033 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.033 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.033 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.033 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.033 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.033 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:25.033 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.033 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.034 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.034 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.034 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.034 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.034 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.034 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.034 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:25.034 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.034 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.034 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.035 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:25.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.035 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.035 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.035 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.035 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.035 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.035 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.035 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571597.910175_581.1", 2024-04-20T01:55:25.035 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.036 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:37.910175Z", 2024-04-20T01:55:25.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.036 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.036 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.036 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.036 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.036 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.036 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.037 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.037 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.037 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.037 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.037 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.037 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.037 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.037 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.037 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.038 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:25.038 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:25.038 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.038 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:09:46Z" 2024-04-20T01:55:25.038 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.038 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571597.910175_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:37.910175Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:25.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571597.910175_581.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571597.910175_581.1 2024-04-20T01:55:25.038 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.039 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:51.828+0000 7fbb26bca780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:51.830+0000 7fbb26bca780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571598.086361_594.1", 2024-04-20T01:55:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:38.086361Z", 2024-04-20T01:55:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:09:51Z" 2024-04-20T01:55:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:25.045 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.045 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.045 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.045 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.045 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.046 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.046 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.046 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.046 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.046 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.046 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.046 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.047 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:25.047 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.047 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.047 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.047 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.047 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.047 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.047 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.047 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.047 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:25.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.048 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.048 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:25.048 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.048 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.048 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.048 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.048 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.048 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.049 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.049 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571598.086361_594.1", 2024-04-20T01:55:25.049 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.049 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.049 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.049 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:38.086361Z", 2024-04-20T01:55:25.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.049 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.049 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.049 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.049 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.049 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.050 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.050 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.050 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.050 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.050 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.050 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.050 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.051 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.051 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:25.051 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:25.051 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.051 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:09:51Z" 2024-04-20T01:55:25.051 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.051 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.086361_594.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.086361Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:25.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.086361_594.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.086361_594.1 2024-04-20T01:55:25.052 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.052 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:56.981+0000 7f5878b87780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:09:56.983+0000 7f5878b87780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571598.962301_616.1", 2024-04-20T01:55:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:38.962301Z", 2024-04-20T01:55:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:09:56Z" 2024-04-20T01:55:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:25.059 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.059 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.059 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.059 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.059 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.060 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.060 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.060 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.060 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.060 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.060 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.060 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.060 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.060 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:25.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.061 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.061 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.061 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.061 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.061 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.062 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.062 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:25.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.062 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.062 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:25.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.063 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.063 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.063 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.063 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.063 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571598.962301_616.1", 2024-04-20T01:55:25.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:38.962301Z", 2024-04-20T01:55:25.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.064 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.064 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.064 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.064 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.064 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.065 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.065 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.065 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.065 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.065 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.065 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:25.065 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:25.065 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.066 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:09:56Z" 2024-04-20T01:55:25.066 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.066 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571598.962301_616.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:38.962301Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:25.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571598.962301_616.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713571601.228671_641.1 target=1_1713571598.962301_616.1 2024-04-20T01:55:25.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:10:02.118+0000 7f6f8c4b8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:10:02.119+0000 7f6f8c4b8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713571601.228671_641.1", 2024-04-20T01:55:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:06:41.228671Z", 2024-04-20T01:55:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:10:02Z" 2024-04-20T01:55:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:25.073 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.073 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.073 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.073 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.073 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.073 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.074 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.074 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.074 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.074 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.074 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:25.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.074 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.075 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.075 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.075 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.075 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.075 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.075 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.075 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:25.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.075 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:25.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.076 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.076 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.076 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.076 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.076 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.076 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.076 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571601.228671_641.1", 2024-04-20T01:55:25.076 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.076 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.076 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.077 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:41.228671Z", 2024-04-20T01:55:25.077 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.077 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.077 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.077 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.077 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.077 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.077 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.077 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.077 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.077 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.078 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.078 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.078 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.078 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.078 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.078 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.078 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.078 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.078 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:25.078 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:25.078 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.079 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:02Z" 2024-04-20T01:55:25.079 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.079 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571601.228671_641.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:41.228671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:25.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:25.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:25.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:10:02.239+0000 7f1323785780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:10:02.240+0000 7f1323785780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713571601.228671_641.1", 2024-04-20T01:55:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T00:06:41.228671Z", 2024-04-20T01:55:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T00:10:02Z" 2024-04-20T01:55:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:25.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:25.086 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.086 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.086 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.086 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.086 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.086 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.086 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.086 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.086 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.086 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.087 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.087 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.087 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.087 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:25.087 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.087 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.087 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.087 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.087 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.087 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.087 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.088 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.088 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.088 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:25.088 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.088 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.088 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.088 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:25.088 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.088 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.088 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.088 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.089 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.089 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.089 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.089 INFO:tasks.rgw_multisite_tests: "marker": "1_1713571601.228671_641.1", 2024-04-20T01:55:25.089 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.089 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.089 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.089 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:06:41.228671Z", 2024-04-20T01:55:25.089 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.089 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.089 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.090 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.090 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.090 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.090 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.090 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.090 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.090 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.090 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.090 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.090 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.090 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.091 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.091 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.091 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.091 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.091 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:25.091 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:25.091 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.091 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:02Z" 2024-04-20T01:55:25.091 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.091 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713571601.228671_641.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:06:41.228671Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:25.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713571601.228671_641.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:25.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'aktlji-55', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:25.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'aktlji-55', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket aktlji-55 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'aktlji-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:25.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'aktlji-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket aktlji-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'aktlji-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:25.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'aktlji-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket aktlji-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.093 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:6", 2024-04-20T01:55:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.29", 2024-04-20T01:55:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000078", 2024-04-20T01:55:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "meta.history", 2024-04-20T01:55:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:3", 2024-04-20T01:55:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:5", 2024-04-20T01:55:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10", 2024-04-20T01:55:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:6", 2024-04-20T01:55:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.23", 2024-04-20T01:55:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000070", 2024-04-20T01:55:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:1", 2024-04-20T01:55:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000104", 2024-04-20T01:55:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:8", 2024-04-20T01:55:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:5", 2024-04-20T01:55:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:8", 2024-04-20T01:55:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000026", 2024-04-20T01:55:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23", 2024-04-20T01:55:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:1", 2024-04-20T01:55:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:0", 2024-04-20T01:55:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26", 2024-04-20T01:55:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:3", 2024-04-20T01:55:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.8", 2024-04-20T01:55:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:7", 2024-04-20T01:55:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14", 2024-04-20T01:55:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000028", 2024-04-20T01:55:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:6", 2024-04-20T01:55:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:8", 2024-04-20T01:55:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:6", 2024-04-20T01:55:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20", 2024-04-20T01:55:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:10", 2024-04-20T01:55:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:2", 2024-04-20T01:55:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:3", 2024-04-20T01:55:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:1", 2024-04-20T01:55:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.1", 2024-04-20T01:55:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:0", 2024-04-20T01:55:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:4", 2024-04-20T01:55:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:2", 2024-04-20T01:55:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.30", 2024-04-20T01:55:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13", 2024-04-20T01:55:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:2", 2024-04-20T01:55:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:6", 2024-04-20T01:55:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:3", 2024-04-20T01:55:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000040", 2024-04-20T01:55:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:2", 2024-04-20T01:55:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.0", 2024-04-20T01:55:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.6", 2024-04-20T01:55:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19", 2024-04-20T01:55:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:6", 2024-04-20T01:55:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:5", 2024-04-20T01:55:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:1", 2024-04-20T01:55:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:9", 2024-04-20T01:55:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000015", 2024-04-20T01:55:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:3", 2024-04-20T01:55:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:2", 2024-04-20T01:55:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:2", 2024-04-20T01:55:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:7", 2024-04-20T01:55:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:1", 2024-04-20T01:55:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000069", 2024-04-20T01:55:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:10", 2024-04-20T01:55:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000095", 2024-04-20T01:55:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000003", 2024-04-20T01:55:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29", 2024-04-20T01:55:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000047", 2024-04-20T01:55:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:0", 2024-04-20T01:55:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:9", 2024-04-20T01:55:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:4", 2024-04-20T01:55:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:4", 2024-04-20T01:55:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:1", 2024-04-20T01:55:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:10", 2024-04-20T01:55:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000035", 2024-04-20T01:55:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:2", 2024-04-20T01:55:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000037", 2024-04-20T01:55:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:4", 2024-04-20T01:55:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:4", 2024-04-20T01:55:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.0.retry", 2024-04-20T01:55:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:3", 2024-04-20T01:55:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:4", 2024-04-20T01:55:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:6", 2024-04-20T01:55:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:3", 2024-04-20T01:55:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:2", 2024-04-20T01:55:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000024", 2024-04-20T01:55:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:1", 2024-04-20T01:55:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:8", 2024-04-20T01:55:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:10", 2024-04-20T01:55:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000000", 2024-04-20T01:55:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:4", 2024-04-20T01:55:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:1", 2024-04-20T01:55:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000031", 2024-04-20T01:55:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:1", 2024-04-20T01:55:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:1", 2024-04-20T01:55:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:2", 2024-04-20T01:55:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:0", 2024-04-20T01:55:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:2", 2024-04-20T01:55:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.3", 2024-04-20T01:55:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:6", 2024-04-20T01:55:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:10", 2024-04-20T01:55:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:10", 2024-04-20T01:55:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:7", 2024-04-20T01:55:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:3", 2024-04-20T01:55:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:6", 2024-04-20T01:55:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:5", 2024-04-20T01:55:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:9", 2024-04-20T01:55:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:0", 2024-04-20T01:55:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:7", 2024-04-20T01:55:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:0", 2024-04-20T01:55:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:7", 2024-04-20T01:55:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:7", 2024-04-20T01:55:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:1", 2024-04-20T01:55:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:10", 2024-04-20T01:55:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:4", 2024-04-20T01:55:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:2", 2024-04-20T01:55:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_loggenerations_metadata", 2024-04-20T01:55:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:3", 2024-04-20T01:55:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:10", 2024-04-20T01:55:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12", 2024-04-20T01:55:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:5", 2024-04-20T01:55:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:8", 2024-04-20T01:55:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:6", 2024-04-20T01:55:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:3", 2024-04-20T01:55:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:3", 2024-04-20T01:55:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:1", 2024-04-20T01:55:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.2.7JqVP+4zZdZ.0", 2024-04-20T01:55:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000076", 2024-04-20T01:55:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:7", 2024-04-20T01:55:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000113", 2024-04-20T01:55:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22", 2024-04-20T01:55:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000005", 2024-04-20T01:55:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:5", 2024-04-20T01:55:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000011", 2024-04-20T01:55:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:10", 2024-04-20T01:55:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000058", 2024-04-20T01:55:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000052", 2024-04-20T01:55:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:4", 2024-04-20T01:55:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:9", 2024-04-20T01:55:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.21", 2024-04-20T01:55:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:1", 2024-04-20T01:55:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:7", 2024-04-20T01:55:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:0", 2024-04-20T01:55:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:0", 2024-04-20T01:55:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:9", 2024-04-20T01:55:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:3", 2024-04-20T01:55:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:10", 2024-04-20T01:55:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25", 2024-04-20T01:55:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46", 2024-04-20T01:55:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:3", 2024-04-20T01:55:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:4", 2024-04-20T01:55:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18", 2024-04-20T01:55:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:9", 2024-04-20T01:55:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:9", 2024-04-20T01:55:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000088", 2024-04-20T01:55:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000080", 2024-04-20T01:55:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:4", 2024-04-20T01:55:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:3", 2024-04-20T01:55:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000090", 2024-04-20T01:55:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22", 2024-04-20T01:55:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:1", 2024-04-20T01:55:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:4", 2024-04-20T01:55:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000110", 2024-04-20T01:55:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:9", 2024-04-20T01:55:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:5", 2024-04-20T01:55:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.0", 2024-04-20T01:55:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:4", 2024-04-20T01:55:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000096", 2024-04-20T01:55:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:1", 2024-04-20T01:55:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:6", 2024-04-20T01:55:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:1", 2024-04-20T01:55:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:2", 2024-04-20T01:55:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000087", 2024-04-20T01:55:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000008", 2024-04-20T01:55:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:0", 2024-04-20T01:55:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:6", 2024-04-20T01:55:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:7", 2024-04-20T01:55:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:3", 2024-04-20T01:55:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:2", 2024-04-20T01:55:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:8", 2024-04-20T01:55:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:9", 2024-04-20T01:55:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000006", 2024-04-20T01:55:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:1", 2024-04-20T01:55:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000029", 2024-04-20T01:55:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:7", 2024-04-20T01:55:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:6", 2024-04-20T01:55:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000089", 2024-04-20T01:55:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000022", 2024-04-20T01:55:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:7", 2024-04-20T01:55:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:7", 2024-04-20T01:55:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:6", 2024-04-20T01:55:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:2", 2024-04-20T01:55:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:10", 2024-04-20T01:55:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bilog.trim", 2024-04-20T01:55:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:3", 2024-04-20T01:55:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:2", 2024-04-20T01:55:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000016", 2024-04-20T01:55:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:4", 2024-04-20T01:55:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:5", 2024-04-20T01:55:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:9", 2024-04-20T01:55:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000041", 2024-04-20T01:55:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10", 2024-04-20T01:55:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:8", 2024-04-20T01:55:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:7", 2024-04-20T01:55:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:1", 2024-04-20T01:55:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27", 2024-04-20T01:55:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25", 2024-04-20T01:55:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000018", 2024-04-20T01:55:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:6", 2024-04-20T01:55:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:6", 2024-04-20T01:55:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000092", 2024-04-20T01:55:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:9", 2024-04-20T01:55:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:6", 2024-04-20T01:55:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:2", 2024-04-20T01:55:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:8", 2024-04-20T01:55:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.27", 2024-04-20T01:55:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.4", 2024-04-20T01:55:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:8", 2024-04-20T01:55:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:3", 2024-04-20T01:55:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:8", 2024-04-20T01:55:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:4", 2024-04-20T01:55:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:8", 2024-04-20T01:55:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:6", 2024-04-20T01:55:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000014", 2024-04-20T01:55:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:4", 2024-04-20T01:55:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28", 2024-04-20T01:55:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000112", 2024-04-20T01:55:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:2", 2024-04-20T01:55:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:9", 2024-04-20T01:55:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000007", 2024-04-20T01:55:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000021", 2024-04-20T01:55:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:1", 2024-04-20T01:55:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:3", 2024-04-20T01:55:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:6", 2024-04-20T01:55:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.16", 2024-04-20T01:55:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000064", 2024-04-20T01:55:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16", 2024-04-20T01:55:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:8", 2024-04-20T01:55:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000071", 2024-04-20T01:55:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:10", 2024-04-20T01:55:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:3", 2024-04-20T01:55:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000074", 2024-04-20T01:55:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:5", 2024-04-20T01:55:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:8", 2024-04-20T01:55:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:6", 2024-04-20T01:55:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:8", 2024-04-20T01:55:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:6", 2024-04-20T01:55:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:7", 2024-04-20T01:55:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:8", 2024-04-20T01:55:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:10", 2024-04-20T01:55:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:7", 2024-04-20T01:55:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:0", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000081", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.3.retry", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:5", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000009", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:6", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000121", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:3", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:3", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.1", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:4", 2024-04-20T01:55:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:8", 2024-04-20T01:55:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.2", 2024-04-20T01:55:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:8", 2024-04-20T01:55:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:6", 2024-04-20T01:55:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.17", 2024-04-20T01:55:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000125", 2024-04-20T01:55:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000082", 2024-04-20T01:55:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:7", 2024-04-20T01:55:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:2", 2024-04-20T01:55:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:0", 2024-04-20T01:55:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:6", 2024-04-20T01:55:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:6", 2024-04-20T01:55:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24", 2024-04-20T01:55:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:6", 2024-04-20T01:55:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:3", 2024-04-20T01:55:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000105", 2024-04-20T01:55:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000059", 2024-04-20T01:55:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:8", 2024-04-20T01:55:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:10", 2024-04-20T01:55:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:7", 2024-04-20T01:55:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000077", 2024-04-20T01:55:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:8", 2024-04-20T01:55:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000032", 2024-04-20T01:55:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:8", 2024-04-20T01:55:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:2", 2024-04-20T01:55:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:7", 2024-04-20T01:55:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:1", 2024-04-20T01:55:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:7", 2024-04-20T01:55:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:9", 2024-04-20T01:55:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:0", 2024-04-20T01:55:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:3", 2024-04-20T01:55:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.24", 2024-04-20T01:55:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.2", 2024-04-20T01:55:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:10", 2024-04-20T01:55:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:1", 2024-04-20T01:55:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:4", 2024-04-20T01:55:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:3", 2024-04-20T01:55:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:5", 2024-04-20T01:55:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:4", 2024-04-20T01:55:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:1", 2024-04-20T01:55:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:7", 2024-04-20T01:55:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000053", 2024-04-20T01:55:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:1", 2024-04-20T01:55:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:9", 2024-04-20T01:55:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:7", 2024-04-20T01:55:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:6", 2024-04-20T01:55:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27", 2024-04-20T01:55:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:7", 2024-04-20T01:55:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000091", 2024-04-20T01:55:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:9", 2024-04-20T01:55:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:3", 2024-04-20T01:55:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:9", 2024-04-20T01:55:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:10", 2024-04-20T01:55:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:2", 2024-04-20T01:55:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000065", 2024-04-20T01:55:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:4", 2024-04-20T01:55:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.1.retry", 2024-04-20T01:55:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000083", 2024-04-20T01:55:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000010", 2024-04-20T01:55:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:9", 2024-04-20T01:55:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:3", 2024-04-20T01:55:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:9", 2024-04-20T01:55:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:8", 2024-04-20T01:55:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:5", 2024-04-20T01:55:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:0", 2024-04-20T01:55:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:10", 2024-04-20T01:55:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:3", 2024-04-20T01:55:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:0", 2024-04-20T01:55:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.7", 2024-04-20T01:55:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.3", 2024-04-20T01:55:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.3.retry", 2024-04-20T01:55:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:0", 2024-04-20T01:55:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:3", 2024-04-20T01:55:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:3", 2024-04-20T01:55:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:6", 2024-04-20T01:55:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:6", 2024-04-20T01:55:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.1.e8rVRID2eg2.0", 2024-04-20T01:55:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:0", 2024-04-20T01:55:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:8", 2024-04-20T01:55:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:3", 2024-04-20T01:55:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:2", 2024-04-20T01:55:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:1", 2024-04-20T01:55:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:6", 2024-04-20T01:55:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:0", 2024-04-20T01:55:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:0", 2024-04-20T01:55:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:5", 2024-04-20T01:55:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:4", 2024-04-20T01:55:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:5", 2024-04-20T01:55:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000045", 2024-04-20T01:55:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11", 2024-04-20T01:55:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000002", 2024-04-20T01:55:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:6", 2024-04-20T01:55:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.12", 2024-04-20T01:55:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:4", 2024-04-20T01:55:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:6", 2024-04-20T01:55:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:1", 2024-04-20T01:55:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:8", 2024-04-20T01:55:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000116", 2024-04-20T01:55:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:4", 2024-04-20T01:55:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:6", 2024-04-20T01:55:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.3", 2024-04-20T01:55:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:6", 2024-04-20T01:55:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.25", 2024-04-20T01:55:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:0", 2024-04-20T01:55:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000034", 2024-04-20T01:55:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:9", 2024-04-20T01:55:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:1", 2024-04-20T01:55:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:10", 2024-04-20T01:55:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:4", 2024-04-20T01:55:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:4", 2024-04-20T01:55:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:0", 2024-04-20T01:55:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:1", 2024-04-20T01:55:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000101", 2024-04-20T01:55:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:2", 2024-04-20T01:55:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:10", 2024-04-20T01:55:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:7", 2024-04-20T01:55:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:0", 2024-04-20T01:55:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45", 2024-04-20T01:55:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.13", 2024-04-20T01:55:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:1", 2024-04-20T01:55:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:10", 2024-04-20T01:55:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:2", 2024-04-20T01:55:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:7", 2024-04-20T01:55:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:3", 2024-04-20T01:55:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:8", 2024-04-20T01:55:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000079", 2024-04-20T01:55:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:0", 2024-04-20T01:55:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000049", 2024-04-20T01:55:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000117", 2024-04-20T01:55:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000044", 2024-04-20T01:55:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:5", 2024-04-20T01:55:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:7", 2024-04-20T01:55:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:10", 2024-04-20T01:55:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:9", 2024-04-20T01:55:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:5", 2024-04-20T01:55:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:4", 2024-04-20T01:55:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:5", 2024-04-20T01:55:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:8", 2024-04-20T01:55:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.14", 2024-04-20T01:55:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000066", 2024-04-20T01:55:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:10", 2024-04-20T01:55:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:0", 2024-04-20T01:55:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:1", 2024-04-20T01:55:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:7", 2024-04-20T01:55:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:2", 2024-04-20T01:55:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:4", 2024-04-20T01:55:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:2", 2024-04-20T01:55:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000068", 2024-04-20T01:55:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:8", 2024-04-20T01:55:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000085", 2024-04-20T01:55:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:9", 2024-04-20T01:55:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:10", 2024-04-20T01:55:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:3", 2024-04-20T01:55:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:3", 2024-04-20T01:55:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:5", 2024-04-20T01:55:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:1", 2024-04-20T01:55:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:9", 2024-04-20T01:55:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:8", 2024-04-20T01:55:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:7", 2024-04-20T01:55:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:5", 2024-04-20T01:55:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000073", 2024-04-20T01:55:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000038", 2024-04-20T01:55:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20", 2024-04-20T01:55:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19", 2024-04-20T01:55:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:8", 2024-04-20T01:55:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:1", 2024-04-20T01:55:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45", 2024-04-20T01:55:25.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:9", 2024-04-20T01:55:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:9", 2024-04-20T01:55:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.10", 2024-04-20T01:55:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000118", 2024-04-20T01:55:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:8", 2024-04-20T01:55:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:5", 2024-04-20T01:55:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:2", 2024-04-20T01:55:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:5", 2024-04-20T01:55:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000036", 2024-04-20T01:55:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:6", 2024-04-20T01:55:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:9", 2024-04-20T01:55:25.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:0", 2024-04-20T01:55:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:8", 2024-04-20T01:55:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:10", 2024-04-20T01:55:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000103", 2024-04-20T01:55:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:5", 2024-04-20T01:55:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:10", 2024-04-20T01:55:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23", 2024-04-20T01:55:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:0", 2024-04-20T01:55:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:5", 2024-04-20T01:55:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:10", 2024-04-20T01:55:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000119", 2024-04-20T01:55:25.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:4", 2024-04-20T01:55:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:5", 2024-04-20T01:55:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:7", 2024-04-20T01:55:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000098", 2024-04-20T01:55:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.0", 2024-04-20T01:55:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:5", 2024-04-20T01:55:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:4", 2024-04-20T01:55:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11", 2024-04-20T01:55:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.2.retry", 2024-04-20T01:55:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:4", 2024-04-20T01:55:25.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:5", 2024-04-20T01:55:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:4", 2024-04-20T01:55:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:5", 2024-04-20T01:55:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:2", 2024-04-20T01:55:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:2", 2024-04-20T01:55:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000027", 2024-04-20T01:55:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47", 2024-04-20T01:55:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:3", 2024-04-20T01:55:25.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:0", 2024-04-20T01:55:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:7", 2024-04-20T01:55:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000019", 2024-04-20T01:55:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:1", 2024-04-20T01:55:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:9", 2024-04-20T01:55:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000039", 2024-04-20T01:55:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:1", 2024-04-20T01:55:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:3", 2024-04-20T01:55:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:0", 2024-04-20T01:55:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:1", 2024-04-20T01:55:25.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:10", 2024-04-20T01:55:25.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.2.retry", 2024-04-20T01:55:25.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:1", 2024-04-20T01:55:25.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:8", 2024-04-20T01:55:25.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:0", 2024-04-20T01:55:25.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:10", 2024-04-20T01:55:25.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000100", 2024-04-20T01:55:25.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:0", 2024-04-20T01:55:25.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:8", 2024-04-20T01:55:25.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:10", 2024-04-20T01:55:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000093", 2024-04-20T01:55:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:5", 2024-04-20T01:55:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:6", 2024-04-20T01:55:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:8", 2024-04-20T01:55:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:5", 2024-04-20T01:55:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:10", 2024-04-20T01:55:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:8", 2024-04-20T01:55:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000004", 2024-04-20T01:55:25.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000063", 2024-04-20T01:55:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000122", 2024-04-20T01:55:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000057", 2024-04-20T01:55:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:1", 2024-04-20T01:55:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:8", 2024-04-20T01:55:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:3", 2024-04-20T01:55:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000054", 2024-04-20T01:55:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:3", 2024-04-20T01:55:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000114", 2024-04-20T01:55:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:0", 2024-04-20T01:55:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.2", 2024-04-20T01:55:25.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:5", 2024-04-20T01:55:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.0", 2024-04-20T01:55:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:0", 2024-04-20T01:55:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:4", 2024-04-20T01:55:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:3", 2024-04-20T01:55:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000012", 2024-04-20T01:55:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:7", 2024-04-20T01:55:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.3", 2024-04-20T01:55:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:9", 2024-04-20T01:55:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:0", 2024-04-20T01:55:25.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:6", 2024-04-20T01:55:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14", 2024-04-20T01:55:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:10", 2024-04-20T01:55:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000084", 2024-04-20T01:55:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000043", 2024-04-20T01:55:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:7", 2024-04-20T01:55:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:7", 2024-04-20T01:55:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.11", 2024-04-20T01:55:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:1", 2024-04-20T01:55:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:5", 2024-04-20T01:55:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:0", 2024-04-20T01:55:25.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.0.nl0urTltrWn.0", 2024-04-20T01:55:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:0", 2024-04-20T01:55:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:2", 2024-04-20T01:55:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:9", 2024-04-20T01:55:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000111", 2024-04-20T01:55:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:6", 2024-04-20T01:55:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000048", 2024-04-20T01:55:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46", 2024-04-20T01:55:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000020", 2024-04-20T01:55:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:9", 2024-04-20T01:55:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:2", 2024-04-20T01:55:25.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:2", 2024-04-20T01:55:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18", 2024-04-20T01:55:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:5", 2024-04-20T01:55:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:3", 2024-04-20T01:55:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000099", 2024-04-20T01:55:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000056", 2024-04-20T01:55:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:0", 2024-04-20T01:55:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:1", 2024-04-20T01:55:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:4", 2024-04-20T01:55:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:3", 2024-04-20T01:55:25.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:10", 2024-04-20T01:55:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.19", 2024-04-20T01:55:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:9", 2024-04-20T01:55:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.28", 2024-04-20T01:55:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:9", 2024-04-20T01:55:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.0.retry", 2024-04-20T01:55:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000072", 2024-04-20T01:55:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:7", 2024-04-20T01:55:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:2", 2024-04-20T01:55:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:0", 2024-04-20T01:55:25.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:2", 2024-04-20T01:55:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15", 2024-04-20T01:55:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:10", 2024-04-20T01:55:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:2", 2024-04-20T01:55:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000062", 2024-04-20T01:55:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.2", 2024-04-20T01:55:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000115", 2024-04-20T01:55:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:9", 2024-04-20T01:55:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.1", 2024-04-20T01:55:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:1", 2024-04-20T01:55:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:8", 2024-04-20T01:55:25.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000033", 2024-04-20T01:55:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:1", 2024-04-20T01:55:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:9", 2024-04-20T01:55:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:5", 2024-04-20T01:55:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:9", 2024-04-20T01:55:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:7", 2024-04-20T01:55:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:7", 2024-04-20T01:55:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:6", 2024-04-20T01:55:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:10", 2024-04-20T01:55:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1", 2024-04-20T01:55:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:10", 2024-04-20T01:55:25.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17", 2024-04-20T01:55:25.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:6", 2024-04-20T01:55:25.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:3", 2024-04-20T01:55:25.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000025", 2024-04-20T01:55:25.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000123", 2024-04-20T01:55:25.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:2", 2024-04-20T01:55:25.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.31", 2024-04-20T01:55:25.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:9", 2024-04-20T01:55:25.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:0", 2024-04-20T01:55:25.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.18", 2024-04-20T01:55:25.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:5", 2024-04-20T01:55:25.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:7", 2024-04-20T01:55:25.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000108", 2024-04-20T01:55:25.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000094", 2024-04-20T01:55:25.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:4", 2024-04-20T01:55:25.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:6", 2024-04-20T01:55:25.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000060", 2024-04-20T01:55:25.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:2", 2024-04-20T01:55:25.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:5", 2024-04-20T01:55:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:0", 2024-04-20T01:55:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13", 2024-04-20T01:55:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:0", 2024-04-20T01:55:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000109", 2024-04-20T01:55:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.15", 2024-04-20T01:55:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:10", 2024-04-20T01:55:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:7", 2024-04-20T01:55:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000013", 2024-04-20T01:55:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:9", 2024-04-20T01:55:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:5", 2024-04-20T01:55:25.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28", 2024-04-20T01:55:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:5", 2024-04-20T01:55:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:2", 2024-04-20T01:55:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.3", 2024-04-20T01:55:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.0", 2024-04-20T01:55:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:5", 2024-04-20T01:55:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000051", 2024-04-20T01:55:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:8", 2024-04-20T01:55:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:3", 2024-04-20T01:55:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:8", 2024-04-20T01:55:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000106", 2024-04-20T01:55:25.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:10", 2024-04-20T01:55:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:5", 2024-04-20T01:55:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:8", 2024-04-20T01:55:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:9", 2024-04-20T01:55:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000017", 2024-04-20T01:55:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:9", 2024-04-20T01:55:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16", 2024-04-20T01:55:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:4", 2024-04-20T01:55:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:9", 2024-04-20T01:55:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:0", 2024-04-20T01:55:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:5", 2024-04-20T01:55:25.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:10", 2024-04-20T01:55:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:7", 2024-04-20T01:55:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000023", 2024-04-20T01:55:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15", 2024-04-20T01:55:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:6", 2024-04-20T01:55:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:9", 2024-04-20T01:55:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:8", 2024-04-20T01:55:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:5", 2024-04-20T01:55:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.1.retry", 2024-04-20T01:55:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000107", 2024-04-20T01:55:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:8", 2024-04-20T01:55:25.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:4", 2024-04-20T01:55:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000046", 2024-04-20T01:55:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:0", 2024-04-20T01:55:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:1", 2024-04-20T01:55:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.26", 2024-04-20T01:55:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:8", 2024-04-20T01:55:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:4", 2024-04-20T01:55:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47", 2024-04-20T01:55:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:2", 2024-04-20T01:55:25.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.22", 2024-04-20T01:55:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:0", 2024-04-20T01:55:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:0", 2024-04-20T01:55:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.1", 2024-04-20T01:55:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:9", 2024-04-20T01:55:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:4", 2024-04-20T01:55:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.2", 2024-04-20T01:55:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:7", 2024-04-20T01:55:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.3.Ul7Ylj3XnqY.0", 2024-04-20T01:55:25.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000067", 2024-04-20T01:55:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:7", 2024-04-20T01:55:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:8", 2024-04-20T01:55:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:7", 2024-04-20T01:55:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:4", 2024-04-20T01:55:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:7", 2024-04-20T01:55:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29", 2024-04-20T01:55:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:8", 2024-04-20T01:55:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:2", 2024-04-20T01:55:25.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:8", 2024-04-20T01:55:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:2", 2024-04-20T01:55:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:5", 2024-04-20T01:55:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:4", 2024-04-20T01:55:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:0", 2024-04-20T01:55:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:2", 2024-04-20T01:55:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.9", 2024-04-20T01:55:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000086", 2024-04-20T01:55:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:8", 2024-04-20T01:55:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21", 2024-04-20T01:55:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.5", 2024-04-20T01:55:25.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21", 2024-04-20T01:55:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:4", 2024-04-20T01:55:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:1", 2024-04-20T01:55:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:9", 2024-04-20T01:55:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:0", 2024-04-20T01:55:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:6", 2024-04-20T01:55:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.20", 2024-04-20T01:55:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24", 2024-04-20T01:55:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:1", 2024-04-20T01:55:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:10", 2024-04-20T01:55:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:9", 2024-04-20T01:55:25.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:7", 2024-04-20T01:55:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:10", 2024-04-20T01:55:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:5", 2024-04-20T01:55:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:1", 2024-04-20T01:55:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:2", 2024-04-20T01:55:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:5", 2024-04-20T01:55:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:2", 2024-04-20T01:55:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000050", 2024-04-20T01:55:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000102", 2024-04-20T01:55:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:4", 2024-04-20T01:55:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:3", 2024-04-20T01:55:25.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:9", 2024-04-20T01:55:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:6", 2024-04-20T01:55:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:10", 2024-04-20T01:55:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12", 2024-04-20T01:55:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:4", 2024-04-20T01:55:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:9", 2024-04-20T01:55:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:4", 2024-04-20T01:55:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:1", 2024-04-20T01:55:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:0", 2024-04-20T01:55:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:5", 2024-04-20T01:55:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000124", 2024-04-20T01:55:25.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:10", 2024-04-20T01:55:25.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:5", 2024-04-20T01:55:25.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:3", 2024-04-20T01:55:25.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:9", 2024-04-20T01:55:25.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:2", 2024-04-20T01:55:25.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:6", 2024-04-20T01:55:25.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000001", 2024-04-20T01:55:25.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:1", 2024-04-20T01:55:25.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000126", 2024-04-20T01:55:25.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:8", 2024-04-20T01:55:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:5", 2024-04-20T01:55:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:7", 2024-04-20T01:55:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:3", 2024-04-20T01:55:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000055", 2024-04-20T01:55:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:10", 2024-04-20T01:55:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:7", 2024-04-20T01:55:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000061", 2024-04-20T01:55:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000097", 2024-04-20T01:55:25.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:4", 2024-04-20T01:55:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:10", 2024-04-20T01:55:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:4", 2024-04-20T01:55:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000042", 2024-04-20T01:55:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:2", 2024-04-20T01:55:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:1", 2024-04-20T01:55:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:9", 2024-04-20T01:55:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:7", 2024-04-20T01:55:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:10", 2024-04-20T01:55:25.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:2", 2024-04-20T01:55:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:6", 2024-04-20T01:55:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:7", 2024-04-20T01:55:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:6", 2024-04-20T01:55:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000075", 2024-04-20T01:55:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:2", 2024-04-20T01:55:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:3", 2024-04-20T01:55:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000120", 2024-04-20T01:55:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000030", 2024-04-20T01:55:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:10", 2024-04-20T01:55:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:3", 2024-04-20T01:55:25.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:0", 2024-04-20T01:55:25.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:4", 2024-04-20T01:55:25.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:10" 2024-04-20T01:55:25.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:25.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: 2024-04-20T01:55:25.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.165 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: [ 2024-04-20T01:55:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:6", 2024-04-20T01:55:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.29", 2024-04-20T01:55:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000078", 2024-04-20T01:55:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "meta.history", 2024-04-20T01:55:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:3", 2024-04-20T01:55:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:5", 2024-04-20T01:55:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10", 2024-04-20T01:55:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:9", 2024-04-20T01:55:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:6", 2024-04-20T01:55:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.23", 2024-04-20T01:55:25.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000070", 2024-04-20T01:55:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:1", 2024-04-20T01:55:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:1", 2024-04-20T01:55:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000104", 2024-04-20T01:55:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:8", 2024-04-20T01:55:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:9", 2024-04-20T01:55:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000026", 2024-04-20T01:55:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23", 2024-04-20T01:55:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:1", 2024-04-20T01:55:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:0", 2024-04-20T01:55:25.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.8", 2024-04-20T01:55:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:7", 2024-04-20T01:55:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000028", 2024-04-20T01:55:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:8", 2024-04-20T01:55:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:8", 2024-04-20T01:55:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:6", 2024-04-20T01:55:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:3", 2024-04-20T01:55:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:6", 2024-04-20T01:55:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:10", 2024-04-20T01:55:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19", 2024-04-20T01:55:25.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:4", 2024-04-20T01:55:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:2", 2024-04-20T01:55:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.0.retry", 2024-04-20T01:55:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:3", 2024-04-20T01:55:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.1", 2024-04-20T01:55:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.30", 2024-04-20T01:55:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13", 2024-04-20T01:55:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:6", 2024-04-20T01:55:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:7", 2024-04-20T01:55:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000040", 2024-04-20T01:55:25.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:7", 2024-04-20T01:55:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:2", 2024-04-20T01:55:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.0", 2024-04-20T01:55:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.6", 2024-04-20T01:55:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:1", 2024-04-20T01:55:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:8", 2024-04-20T01:55:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19", 2024-04-20T01:55:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:6", 2024-04-20T01:55:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:1", 2024-04-20T01:55:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:6", 2024-04-20T01:55:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:8", 2024-04-20T01:55:25.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:2", 2024-04-20T01:55:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000015", 2024-04-20T01:55:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:3", 2024-04-20T01:55:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000069", 2024-04-20T01:55:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000095", 2024-04-20T01:55:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:0", 2024-04-20T01:55:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000003", 2024-04-20T01:55:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29", 2024-04-20T01:55:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000047", 2024-04-20T01:55:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:0", 2024-04-20T01:55:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:9", 2024-04-20T01:55:25.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:5", 2024-04-20T01:55:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:9", 2024-04-20T01:55:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:10", 2024-04-20T01:55:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000035", 2024-04-20T01:55:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:8", 2024-04-20T01:55:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000037", 2024-04-20T01:55:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:4", 2024-04-20T01:55:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:3", 2024-04-20T01:55:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:1", 2024-04-20T01:55:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:9", 2024-04-20T01:55:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:5", 2024-04-20T01:55:25.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:3", 2024-04-20T01:55:25.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "data_log.2.DWMttCqmkiM.0", 2024-04-20T01:55:25.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:2", 2024-04-20T01:55:25.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:7", 2024-04-20T01:55:25.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:1", 2024-04-20T01:55:25.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:1", 2024-04-20T01:55:25.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000024", 2024-04-20T01:55:25.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:2", 2024-04-20T01:55:25.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:5", 2024-04-20T01:55:25.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:1", 2024-04-20T01:55:25.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:10", 2024-04-20T01:55:25.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:10", 2024-04-20T01:55:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000000", 2024-04-20T01:55:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:1", 2024-04-20T01:55:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000031", 2024-04-20T01:55:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:0", 2024-04-20T01:55:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "data_log.3", 2024-04-20T01:55:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:6", 2024-04-20T01:55:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:7", 2024-04-20T01:55:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:4", 2024-04-20T01:55:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:6", 2024-04-20T01:55:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:5", 2024-04-20T01:55:25.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:4", 2024-04-20T01:55:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:5", 2024-04-20T01:55:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:9", 2024-04-20T01:55:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:5", 2024-04-20T01:55:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:3", 2024-04-20T01:55:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:0", 2024-04-20T01:55:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:7", 2024-04-20T01:55:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:0", 2024-04-20T01:55:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:7", 2024-04-20T01:55:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:10", 2024-04-20T01:55:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:3", 2024-04-20T01:55:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:2", 2024-04-20T01:55:25.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "data_loggenerations_metadata", 2024-04-20T01:55:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:6", 2024-04-20T01:55:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:6", 2024-04-20T01:55:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12", 2024-04-20T01:55:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:6", 2024-04-20T01:55:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:9", 2024-04-20T01:55:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:1", 2024-04-20T01:55:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000076", 2024-04-20T01:55:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:4", 2024-04-20T01:55:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000113", 2024-04-20T01:55:25.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22", 2024-04-20T01:55:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000005", 2024-04-20T01:55:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000011", 2024-04-20T01:55:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:10", 2024-04-20T01:55:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000058", 2024-04-20T01:55:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000052", 2024-04-20T01:55:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.21", 2024-04-20T01:55:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:9", 2024-04-20T01:55:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:3", 2024-04-20T01:55:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:1", 2024-04-20T01:55:25.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:9", 2024-04-20T01:55:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:8", 2024-04-20T01:55:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:2", 2024-04-20T01:55:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:7", 2024-04-20T01:55:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:0", 2024-04-20T01:55:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:0", 2024-04-20T01:55:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:9", 2024-04-20T01:55:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:8", 2024-04-20T01:55:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27", 2024-04-20T01:55:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25", 2024-04-20T01:55:25.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:4", 2024-04-20T01:55:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:8", 2024-04-20T01:55:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:6", 2024-04-20T01:55:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:8", 2024-04-20T01:55:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:9", 2024-04-20T01:55:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:9", 2024-04-20T01:55:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000088", 2024-04-20T01:55:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000080", 2024-04-20T01:55:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:4", 2024-04-20T01:55:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000090", 2024-04-20T01:55:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:5", 2024-04-20T01:55:25.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:7", 2024-04-20T01:55:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000110", 2024-04-20T01:55:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:9", 2024-04-20T01:55:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:9", 2024-04-20T01:55:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:0", 2024-04-20T01:55:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.0", 2024-04-20T01:55:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:4", 2024-04-20T01:55:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:4", 2024-04-20T01:55:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000096", 2024-04-20T01:55:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:6", 2024-04-20T01:55:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:1", 2024-04-20T01:55:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:2", 2024-04-20T01:55:25.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000087", 2024-04-20T01:55:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000008", 2024-04-20T01:55:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:0", 2024-04-20T01:55:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:3", 2024-04-20T01:55:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:7", 2024-04-20T01:55:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:3", 2024-04-20T01:55:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:3", 2024-04-20T01:55:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:8", 2024-04-20T01:55:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:9", 2024-04-20T01:55:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:1", 2024-04-20T01:55:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000006", 2024-04-20T01:55:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29", 2024-04-20T01:55:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000029", 2024-04-20T01:55:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:7", 2024-04-20T01:55:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:9", 2024-04-20T01:55:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:10", 2024-04-20T01:55:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000089", 2024-04-20T01:55:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000022", 2024-04-20T01:55:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.0", 2024-04-20T01:55:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:9", 2024-04-20T01:55:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:7", 2024-04-20T01:55:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:7", 2024-04-20T01:55:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bilog.trim", 2024-04-20T01:55:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:6", 2024-04-20T01:55:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:2", 2024-04-20T01:55:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13", 2024-04-20T01:55:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000016", 2024-04-20T01:55:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:6", 2024-04-20T01:55:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:8", 2024-04-20T01:55:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:2", 2024-04-20T01:55:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000041", 2024-04-20T01:55:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:10", 2024-04-20T01:55:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:1", 2024-04-20T01:55:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:8", 2024-04-20T01:55:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000018", 2024-04-20T01:55:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:6", 2024-04-20T01:55:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000092", 2024-04-20T01:55:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:8", 2024-04-20T01:55:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:0", 2024-04-20T01:55:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:2", 2024-04-20T01:55:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:10", 2024-04-20T01:55:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:5", 2024-04-20T01:55:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.27", 2024-04-20T01:55:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.4", 2024-04-20T01:55:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:3", 2024-04-20T01:55:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:0", 2024-04-20T01:55:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:6", 2024-04-20T01:55:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:8", 2024-04-20T01:55:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000014", 2024-04-20T01:55:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:4", 2024-04-20T01:55:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:2", 2024-04-20T01:55:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000112", 2024-04-20T01:55:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:9", 2024-04-20T01:55:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000007", 2024-04-20T01:55:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000021", 2024-04-20T01:55:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:1", 2024-04-20T01:55:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:1", 2024-04-20T01:55:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:3", 2024-04-20T01:55:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.16", 2024-04-20T01:55:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:0", 2024-04-20T01:55:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000064", 2024-04-20T01:55:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:0", 2024-04-20T01:55:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000071", 2024-04-20T01:55:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:1", 2024-04-20T01:55:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:10", 2024-04-20T01:55:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:3", 2024-04-20T01:55:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000074", 2024-04-20T01:55:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:5", 2024-04-20T01:55:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:8", 2024-04-20T01:55:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:8", 2024-04-20T01:55:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:6", 2024-04-20T01:55:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:7", 2024-04-20T01:55:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:8", 2024-04-20T01:55:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26", 2024-04-20T01:55:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:1", 2024-04-20T01:55:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:6", 2024-04-20T01:55:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000081", 2024-04-20T01:55:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.3.retry", 2024-04-20T01:55:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000009", 2024-04-20T01:55:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:6", 2024-04-20T01:55:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000121", 2024-04-20T01:55:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:3", 2024-04-20T01:55:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "data_log.1", 2024-04-20T01:55:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17", 2024-04-20T01:55:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:9", 2024-04-20T01:55:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:4", 2024-04-20T01:55:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:6", 2024-04-20T01:55:25.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:8", 2024-04-20T01:55:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:2", 2024-04-20T01:55:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.2", 2024-04-20T01:55:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:6", 2024-04-20T01:55:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.17", 2024-04-20T01:55:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000125", 2024-04-20T01:55:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000082", 2024-04-20T01:55:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:2", 2024-04-20T01:55:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:8", 2024-04-20T01:55:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24", 2024-04-20T01:55:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:6", 2024-04-20T01:55:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:3", 2024-04-20T01:55:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000105", 2024-04-20T01:55:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000059", 2024-04-20T01:55:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:1", 2024-04-20T01:55:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:7", 2024-04-20T01:55:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:2", 2024-04-20T01:55:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:5", 2024-04-20T01:55:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000077", 2024-04-20T01:55:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:10", 2024-04-20T01:55:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000032", 2024-04-20T01:55:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:7", 2024-04-20T01:55:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:3", 2024-04-20T01:55:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:1", 2024-04-20T01:55:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:5", 2024-04-20T01:55:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:7", 2024-04-20T01:55:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:9", 2024-04-20T01:55:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:0", 2024-04-20T01:55:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.24", 2024-04-20T01:55:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:10", 2024-04-20T01:55:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.2", 2024-04-20T01:55:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:10", 2024-04-20T01:55:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:4", 2024-04-20T01:55:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:3", 2024-04-20T01:55:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:4", 2024-04-20T01:55:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:4", 2024-04-20T01:55:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:0", 2024-04-20T01:55:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:7", 2024-04-20T01:55:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000053", 2024-04-20T01:55:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:4", 2024-04-20T01:55:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:1", 2024-04-20T01:55:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:9", 2024-04-20T01:55:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27", 2024-04-20T01:55:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000091", 2024-04-20T01:55:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46", 2024-04-20T01:55:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:3", 2024-04-20T01:55:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:2", 2024-04-20T01:55:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:9", 2024-04-20T01:55:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000065", 2024-04-20T01:55:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.1.retry", 2024-04-20T01:55:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000083", 2024-04-20T01:55:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000010", 2024-04-20T01:55:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:8", 2024-04-20T01:55:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:3", 2024-04-20T01:55:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.7", 2024-04-20T01:55:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25", 2024-04-20T01:55:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.3", 2024-04-20T01:55:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:9", 2024-04-20T01:55:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:3", 2024-04-20T01:55:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:8", 2024-04-20T01:55:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:1", 2024-04-20T01:55:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:6", 2024-04-20T01:55:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:6", 2024-04-20T01:55:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:4", 2024-04-20T01:55:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:1", 2024-04-20T01:55:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:0", 2024-04-20T01:55:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:8", 2024-04-20T01:55:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:9", 2024-04-20T01:55:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:2", 2024-04-20T01:55:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:9", 2024-04-20T01:55:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:0", 2024-04-20T01:55:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:5", 2024-04-20T01:55:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "mdlog.sync-status.shard.2", 2024-04-20T01:55:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:10", 2024-04-20T01:55:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:4", 2024-04-20T01:55:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11", 2024-04-20T01:55:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000045", 2024-04-20T01:55:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:10", 2024-04-20T01:55:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000002", 2024-04-20T01:55:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:6", 2024-04-20T01:55:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.12", 2024-04-20T01:55:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:4", 2024-04-20T01:55:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:7", 2024-04-20T01:55:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:2", 2024-04-20T01:55:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:1", 2024-04-20T01:55:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000116", 2024-04-20T01:55:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:4", 2024-04-20T01:55:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:8", 2024-04-20T01:55:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.3", 2024-04-20T01:55:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:6", 2024-04-20T01:55:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.25", 2024-04-20T01:55:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000034", 2024-04-20T01:55:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:0", 2024-04-20T01:55:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:4", 2024-04-20T01:55:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:4", 2024-04-20T01:55:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:7", 2024-04-20T01:55:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:1", 2024-04-20T01:55:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000101", 2024-04-20T01:55:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:2", 2024-04-20T01:55:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:10", 2024-04-20T01:55:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:7", 2024-04-20T01:55:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45", 2024-04-20T01:55:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:0", 2024-04-20T01:55:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.13", 2024-04-20T01:55:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "mdlog.sync-status.shard.1", 2024-04-20T01:55:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17", 2024-04-20T01:55:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:3", 2024-04-20T01:55:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:3", 2024-04-20T01:55:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000079", 2024-04-20T01:55:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000049", 2024-04-20T01:55:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000117", 2024-04-20T01:55:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000044", 2024-04-20T01:55:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24", 2024-04-20T01:55:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:1", 2024-04-20T01:55:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:10", 2024-04-20T01:55:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:1", 2024-04-20T01:55:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:9", 2024-04-20T01:55:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:5", 2024-04-20T01:55:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:4", 2024-04-20T01:55:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:5", 2024-04-20T01:55:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:8", 2024-04-20T01:55:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.14", 2024-04-20T01:55:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000066", 2024-04-20T01:55:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:10", 2024-04-20T01:55:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:7", 2024-04-20T01:55:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:1", 2024-04-20T01:55:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:7", 2024-04-20T01:55:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000068", 2024-04-20T01:55:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:8", 2024-04-20T01:55:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000085", 2024-04-20T01:55:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:5", 2024-04-20T01:55:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:9", 2024-04-20T01:55:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:10", 2024-04-20T01:55:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:3", 2024-04-20T01:55:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:5", 2024-04-20T01:55:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:1", 2024-04-20T01:55:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:9", 2024-04-20T01:55:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:1", 2024-04-20T01:55:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:7", 2024-04-20T01:55:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:5", 2024-04-20T01:55:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000073", 2024-04-20T01:55:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:7", 2024-04-20T01:55:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:2", 2024-04-20T01:55:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28", 2024-04-20T01:55:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000038", 2024-04-20T01:55:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20", 2024-04-20T01:55:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:8", 2024-04-20T01:55:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:5", 2024-04-20T01:55:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.10", 2024-04-20T01:55:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.3", 2024-04-20T01:55:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000118", 2024-04-20T01:55:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:5", 2024-04-20T01:55:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:0", 2024-04-20T01:55:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000036", 2024-04-20T01:55:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:6", 2024-04-20T01:55:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:6", 2024-04-20T01:55:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:0", 2024-04-20T01:55:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:0", 2024-04-20T01:55:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:6", 2024-04-20T01:55:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:9", 2024-04-20T01:55:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000103", 2024-04-20T01:55:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:10", 2024-04-20T01:55:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18", 2024-04-20T01:55:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "mdlog.sync-status.shard.3", 2024-04-20T01:55:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:0", 2024-04-20T01:55:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:3", 2024-04-20T01:55:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000119", 2024-04-20T01:55:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:10", 2024-04-20T01:55:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:5", 2024-04-20T01:55:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000098", 2024-04-20T01:55:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:10", 2024-04-20T01:55:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:5", 2024-04-20T01:55:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:4", 2024-04-20T01:55:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11", 2024-04-20T01:55:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.2.retry", 2024-04-20T01:55:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:4", 2024-04-20T01:55:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "mdlog.sync-status.shard.0", 2024-04-20T01:55:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45", 2024-04-20T01:55:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:5", 2024-04-20T01:55:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:1", 2024-04-20T01:55:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:6", 2024-04-20T01:55:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:5", 2024-04-20T01:55:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:2", 2024-04-20T01:55:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000027", 2024-04-20T01:55:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47", 2024-04-20T01:55:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:3", 2024-04-20T01:55:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "mdlog.sync-status", 2024-04-20T01:55:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000019", 2024-04-20T01:55:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.2.retry", 2024-04-20T01:55:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:1", 2024-04-20T01:55:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000039", 2024-04-20T01:55:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:1", 2024-04-20T01:55:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:3", 2024-04-20T01:55:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10", 2024-04-20T01:55:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:3", 2024-04-20T01:55:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:8", 2024-04-20T01:55:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:3", 2024-04-20T01:55:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:1", 2024-04-20T01:55:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:7", 2024-04-20T01:55:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:0", 2024-04-20T01:55:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:10", 2024-04-20T01:55:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000100", 2024-04-20T01:55:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:6", 2024-04-20T01:55:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:7", 2024-04-20T01:55:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:10", 2024-04-20T01:55:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:1", 2024-04-20T01:55:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:10", 2024-04-20T01:55:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:10", 2024-04-20T01:55:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:4", 2024-04-20T01:55:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000093", 2024-04-20T01:55:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:5", 2024-04-20T01:55:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:3", 2024-04-20T01:55:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:1", 2024-04-20T01:55:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:6", 2024-04-20T01:55:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:10", 2024-04-20T01:55:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22", 2024-04-20T01:55:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:8", 2024-04-20T01:55:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000004", 2024-04-20T01:55:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000063", 2024-04-20T01:55:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000122", 2024-04-20T01:55:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:10", 2024-04-20T01:55:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000057", 2024-04-20T01:55:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:1", 2024-04-20T01:55:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:0", 2024-04-20T01:55:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:9", 2024-04-20T01:55:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:8", 2024-04-20T01:55:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:8", 2024-04-20T01:55:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.1.retry", 2024-04-20T01:55:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000054", 2024-04-20T01:55:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000114", 2024-04-20T01:55:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:2", 2024-04-20T01:55:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:0", 2024-04-20T01:55:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.2", 2024-04-20T01:55:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:5", 2024-04-20T01:55:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "data_log.0", 2024-04-20T01:55:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:4", 2024-04-20T01:55:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000012", 2024-04-20T01:55:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:3", 2024-04-20T01:55:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:7", 2024-04-20T01:55:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:6", 2024-04-20T01:55:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:9", 2024-04-20T01:55:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:0", 2024-04-20T01:55:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14", 2024-04-20T01:55:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:8", 2024-04-20T01:55:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000084", 2024-04-20T01:55:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000043", 2024-04-20T01:55:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47", 2024-04-20T01:55:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:7", 2024-04-20T01:55:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.11", 2024-04-20T01:55:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12", 2024-04-20T01:55:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:6", 2024-04-20T01:55:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:9", 2024-04-20T01:55:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:5", 2024-04-20T01:55:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:0", 2024-04-20T01:55:25.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:0", 2024-04-20T01:55:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:10", 2024-04-20T01:55:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:2", 2024-04-20T01:55:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:5", 2024-04-20T01:55:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000111", 2024-04-20T01:55:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:8", 2024-04-20T01:55:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:8", 2024-04-20T01:55:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000048", 2024-04-20T01:55:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:0", 2024-04-20T01:55:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46", 2024-04-20T01:55:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:2", 2024-04-20T01:55:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000020", 2024-04-20T01:55:25.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:9", 2024-04-20T01:55:25.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:3", 2024-04-20T01:55:25.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:2", 2024-04-20T01:55:25.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18", 2024-04-20T01:55:25.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:10", 2024-04-20T01:55:25.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:0", 2024-04-20T01:55:25.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:3", 2024-04-20T01:55:25.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000099", 2024-04-20T01:55:25.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000056", 2024-04-20T01:55:25.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:1", 2024-04-20T01:55:25.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:4", 2024-04-20T01:55:25.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "data_log.3.wJbFlsc9Vqh.0", 2024-04-20T01:55:25.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:10", 2024-04-20T01:55:25.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:7", 2024-04-20T01:55:25.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:10", 2024-04-20T01:55:25.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.19", 2024-04-20T01:55:25.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:4", 2024-04-20T01:55:25.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.28", 2024-04-20T01:55:25.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:2", 2024-04-20T01:55:25.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.0.retry", 2024-04-20T01:55:25.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:4", 2024-04-20T01:55:25.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000072", 2024-04-20T01:55:25.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:7", 2024-04-20T01:55:25.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:2", 2024-04-20T01:55:25.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:2", 2024-04-20T01:55:25.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000062", 2024-04-20T01:55:25.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:1", 2024-04-20T01:55:25.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:3", 2024-04-20T01:55:25.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000115", 2024-04-20T01:55:25.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:7", 2024-04-20T01:55:25.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:9", 2024-04-20T01:55:25.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.1", 2024-04-20T01:55:25.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:8", 2024-04-20T01:55:25.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000033", 2024-04-20T01:55:25.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21", 2024-04-20T01:55:25.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:1", 2024-04-20T01:55:25.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:9", 2024-04-20T01:55:25.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:5", 2024-04-20T01:55:25.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:6", 2024-04-20T01:55:25.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:4", 2024-04-20T01:55:25.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:3", 2024-04-20T01:55:25.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:6", 2024-04-20T01:55:25.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:9", 2024-04-20T01:55:25.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:2", 2024-04-20T01:55:25.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:5", 2024-04-20T01:55:25.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:2", 2024-04-20T01:55:25.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:5", 2024-04-20T01:55:25.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:3", 2024-04-20T01:55:25.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000025", 2024-04-20T01:55:25.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000123", 2024-04-20T01:55:25.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:2", 2024-04-20T01:55:25.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.31", 2024-04-20T01:55:25.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.18", 2024-04-20T01:55:25.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:8", 2024-04-20T01:55:25.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:8", 2024-04-20T01:55:25.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:7", 2024-04-20T01:55:25.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000108", 2024-04-20T01:55:25.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:2", 2024-04-20T01:55:25.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:5", 2024-04-20T01:55:25.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:0", 2024-04-20T01:55:25.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000094", 2024-04-20T01:55:25.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:4", 2024-04-20T01:55:25.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000060", 2024-04-20T01:55:25.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:2", 2024-04-20T01:55:25.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:7", 2024-04-20T01:55:25.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:0", 2024-04-20T01:55:25.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:8", 2024-04-20T01:55:25.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:0", 2024-04-20T01:55:25.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000109", 2024-04-20T01:55:25.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.15", 2024-04-20T01:55:25.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:10", 2024-04-20T01:55:25.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000013", 2024-04-20T01:55:25.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:9", 2024-04-20T01:55:25.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:0", 2024-04-20T01:55:25.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:5", 2024-04-20T01:55:25.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28", 2024-04-20T01:55:25.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:4", 2024-04-20T01:55:25.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:3", 2024-04-20T01:55:25.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:5", 2024-04-20T01:55:25.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:2", 2024-04-20T01:55:25.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26", 2024-04-20T01:55:25.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.3", 2024-04-20T01:55:25.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.bea5de7f-54a0-40b9-979f-37d0f728ff53.0", 2024-04-20T01:55:25.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:5", 2024-04-20T01:55:25.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000051", 2024-04-20T01:55:25.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:8", 2024-04-20T01:55:25.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:10", 2024-04-20T01:55:25.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "data_log.0.T0tGY/bKh7w.0", 2024-04-20T01:55:25.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:6", 2024-04-20T01:55:25.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000106", 2024-04-20T01:55:25.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:1", 2024-04-20T01:55:25.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14", 2024-04-20T01:55:25.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:8", 2024-04-20T01:55:25.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:9", 2024-04-20T01:55:25.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000017", 2024-04-20T01:55:25.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:0", 2024-04-20T01:55:25.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:2", 2024-04-20T01:55:25.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:9", 2024-04-20T01:55:25.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16", 2024-04-20T01:55:25.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:4", 2024-04-20T01:55:25.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:6", 2024-04-20T01:55:25.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:9", 2024-04-20T01:55:25.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:5", 2024-04-20T01:55:25.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:0", 2024-04-20T01:55:25.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:10", 2024-04-20T01:55:25.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:10", 2024-04-20T01:55:25.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:4", 2024-04-20T01:55:25.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:7", 2024-04-20T01:55:25.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000023", 2024-04-20T01:55:25.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:1", 2024-04-20T01:55:25.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:7", 2024-04-20T01:55:25.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15", 2024-04-20T01:55:25.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:6", 2024-04-20T01:55:25.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:9", 2024-04-20T01:55:25.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.2", 2024-04-20T01:55:25.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:8", 2024-04-20T01:55:25.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:0", 2024-04-20T01:55:25.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000107", 2024-04-20T01:55:25.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000046", 2024-04-20T01:55:25.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:5", 2024-04-20T01:55:25.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:1", 2024-04-20T01:55:25.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:0", 2024-04-20T01:55:25.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.26", 2024-04-20T01:55:25.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:5", 2024-04-20T01:55:25.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:3", 2024-04-20T01:55:25.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20", 2024-04-20T01:55:25.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.22", 2024-04-20T01:55:25.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:8", 2024-04-20T01:55:25.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:0", 2024-04-20T01:55:25.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:0", 2024-04-20T01:55:25.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:1", 2024-04-20T01:55:25.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:9", 2024-04-20T01:55:25.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:4", 2024-04-20T01:55:25.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:10", 2024-04-20T01:55:25.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "data_log.2", 2024-04-20T01:55:25.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:7", 2024-04-20T01:55:25.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000067", 2024-04-20T01:55:25.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:8", 2024-04-20T01:55:25.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:4", 2024-04-20T01:55:25.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:7", 2024-04-20T01:55:25.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:7", 2024-04-20T01:55:25.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:2", 2024-04-20T01:55:25.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:8", 2024-04-20T01:55:25.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "data_log.1.fX1+jKwmIGM.0", 2024-04-20T01:55:25.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:2", 2024-04-20T01:55:25.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:5", 2024-04-20T01:55:25.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:4", 2024-04-20T01:55:25.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:7", 2024-04-20T01:55:25.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:2", 2024-04-20T01:55:25.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.9", 2024-04-20T01:55:25.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000086", 2024-04-20T01:55:25.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:8", 2024-04-20T01:55:25.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:7", 2024-04-20T01:55:25.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:2", 2024-04-20T01:55:25.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.5", 2024-04-20T01:55:25.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21", 2024-04-20T01:55:25.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:5", 2024-04-20T01:55:25.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.3.retry", 2024-04-20T01:55:25.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:4", 2024-04-20T01:55:25.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:6", 2024-04-20T01:55:25.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:4", 2024-04-20T01:55:25.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:9", 2024-04-20T01:55:25.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:0", 2024-04-20T01:55:25.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:6", 2024-04-20T01:55:25.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync.error-log.20", 2024-04-20T01:55:25.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:3", 2024-04-20T01:55:25.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:1", 2024-04-20T01:55:25.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:9", 2024-04-20T01:55:25.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23", 2024-04-20T01:55:25.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16", 2024-04-20T01:55:25.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:7", 2024-04-20T01:55:25.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:5", 2024-04-20T01:55:25.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:10", 2024-04-20T01:55:25.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:5", 2024-04-20T01:55:25.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:10", 2024-04-20T01:55:25.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:5", 2024-04-20T01:55:25.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:2", 2024-04-20T01:55:25.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000050", 2024-04-20T01:55:25.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:2", 2024-04-20T01:55:25.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000102", 2024-04-20T01:55:25.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:3", 2024-04-20T01:55:25.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:10", 2024-04-20T01:55:25.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:10", 2024-04-20T01:55:25.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:2", 2024-04-20T01:55:25.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:7", 2024-04-20T01:55:25.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:0", 2024-04-20T01:55:25.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:5", 2024-04-20T01:55:25.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000124", 2024-04-20T01:55:25.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.1", 2024-04-20T01:55:25.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15", 2024-04-20T01:55:25.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:10", 2024-04-20T01:55:25.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:9", 2024-04-20T01:55:25.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:5", 2024-04-20T01:55:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:3", 2024-04-20T01:55:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:2", 2024-04-20T01:55:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:6", 2024-04-20T01:55:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000001", 2024-04-20T01:55:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000126", 2024-04-20T01:55:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:8", 2024-04-20T01:55:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:5", 2024-04-20T01:55:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000055", 2024-04-20T01:55:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:4", 2024-04-20T01:55:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000061", 2024-04-20T01:55:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:7", 2024-04-20T01:55:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:7", 2024-04-20T01:55:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:3", 2024-04-20T01:55:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000097", 2024-04-20T01:55:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:4", 2024-04-20T01:55:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:10", 2024-04-20T01:55:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:3", 2024-04-20T01:55:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:4", 2024-04-20T01:55:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:4", 2024-04-20T01:55:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000042", 2024-04-20T01:55:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:9", 2024-04-20T01:55:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:2", 2024-04-20T01:55:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:4", 2024-04-20T01:55:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "datalog.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:10", 2024-04-20T01:55:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:0", 2024-04-20T01:55:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:6", 2024-04-20T01:55:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:7", 2024-04-20T01:55:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:6", 2024-04-20T01:55:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000075", 2024-04-20T01:55:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:2", 2024-04-20T01:55:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:3", 2024-04-20T01:55:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:2", 2024-04-20T01:55:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000120", 2024-04-20T01:55:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "obj_delete_at_hint.0000000030", 2024-04-20T01:55:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:10", 2024-04-20T01:55:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:3", 2024-04-20T01:55:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:0", 2024-04-20T01:55:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.bea5de7f-54a0-40b9-979f-37d0f728ff53:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:4", 2024-04-20T01:55:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:10", 2024-04-20T01:55:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:4" 2024-04-20T01:55:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: 2024-04-20T01:55:25.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.29", 2024-04-20T01:55:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000078", 2024-04-20T01:55:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "meta.history", 2024-04-20T01:55:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:9", 2024-04-20T01:55:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.23", 2024-04-20T01:55:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000070", 2024-04-20T01:55:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:1", 2024-04-20T01:55:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000104", 2024-04-20T01:55:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:8", 2024-04-20T01:55:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:9", 2024-04-20T01:55:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:8", 2024-04-20T01:55:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:5", 2024-04-20T01:55:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:8", 2024-04-20T01:55:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000026", 2024-04-20T01:55:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26", 2024-04-20T01:55:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:3", 2024-04-20T01:55:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.8", 2024-04-20T01:55:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14", 2024-04-20T01:55:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000028", 2024-04-20T01:55:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:6", 2024-04-20T01:55:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:8", 2024-04-20T01:55:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:3", 2024-04-20T01:55:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.2.XuvZF2h1atf.0", 2024-04-20T01:55:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20", 2024-04-20T01:55:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:6", 2024-04-20T01:55:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.1.jWKpb7Z2XkW.0", 2024-04-20T01:55:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19", 2024-04-20T01:55:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:4", 2024-04-20T01:55:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.0.retry", 2024-04-20T01:55:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:1", 2024-04-20T01:55:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:0", 2024-04-20T01:55:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:4", 2024-04-20T01:55:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:2", 2024-04-20T01:55:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.30", 2024-04-20T01:55:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:2", 2024-04-20T01:55:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:6", 2024-04-20T01:55:25.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:6", 2024-04-20T01:55:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:3", 2024-04-20T01:55:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:7", 2024-04-20T01:55:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000040", 2024-04-20T01:55:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:7", 2024-04-20T01:55:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.0", 2024-04-20T01:55:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.6", 2024-04-20T01:55:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:1", 2024-04-20T01:55:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:8", 2024-04-20T01:55:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:5", 2024-04-20T01:55:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:6", 2024-04-20T01:55:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:9", 2024-04-20T01:55:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:8", 2024-04-20T01:55:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:2", 2024-04-20T01:55:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000015", 2024-04-20T01:55:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:2", 2024-04-20T01:55:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:2", 2024-04-20T01:55:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:7", 2024-04-20T01:55:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:1", 2024-04-20T01:55:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000069", 2024-04-20T01:55:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:10", 2024-04-20T01:55:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000095", 2024-04-20T01:55:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:0", 2024-04-20T01:55:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000003", 2024-04-20T01:55:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000047", 2024-04-20T01:55:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:5", 2024-04-20T01:55:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:9", 2024-04-20T01:55:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:4", 2024-04-20T01:55:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:1", 2024-04-20T01:55:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000035", 2024-04-20T01:55:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:8", 2024-04-20T01:55:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:2", 2024-04-20T01:55:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000037", 2024-04-20T01:55:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:4", 2024-04-20T01:55:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.0.retry", 2024-04-20T01:55:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:1", 2024-04-20T01:55:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:4", 2024-04-20T01:55:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:6", 2024-04-20T01:55:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:9", 2024-04-20T01:55:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:5", 2024-04-20T01:55:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:3", 2024-04-20T01:55:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:3", 2024-04-20T01:55:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:7", 2024-04-20T01:55:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:1", 2024-04-20T01:55:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:1", 2024-04-20T01:55:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000024", 2024-04-20T01:55:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:2", 2024-04-20T01:55:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:5", 2024-04-20T01:55:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:8", 2024-04-20T01:55:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:10", 2024-04-20T01:55:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000000", 2024-04-20T01:55:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:4", 2024-04-20T01:55:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000031", 2024-04-20T01:55:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:1", 2024-04-20T01:55:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:1", 2024-04-20T01:55:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:2", 2024-04-20T01:55:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:2", 2024-04-20T01:55:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.3", 2024-04-20T01:55:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:1", 2024-04-20T01:55:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:7", 2024-04-20T01:55:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:10", 2024-04-20T01:55:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:10", 2024-04-20T01:55:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:3", 2024-04-20T01:55:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:4", 2024-04-20T01:55:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:5", 2024-04-20T01:55:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:4", 2024-04-20T01:55:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:5", 2024-04-20T01:55:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:3", 2024-04-20T01:55:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:7", 2024-04-20T01:55:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:1", 2024-04-20T01:55:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:3", 2024-04-20T01:55:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:4", 2024-04-20T01:55:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:2", 2024-04-20T01:55:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:2", 2024-04-20T01:55:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_loggenerations_metadata", 2024-04-20T01:55:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:3", 2024-04-20T01:55:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:6", 2024-04-20T01:55:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:10", 2024-04-20T01:55:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:6", 2024-04-20T01:55:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:5", 2024-04-20T01:55:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:8", 2024-04-20T01:55:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:9", 2024-04-20T01:55:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:3", 2024-04-20T01:55:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:3", 2024-04-20T01:55:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000076", 2024-04-20T01:55:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:4", 2024-04-20T01:55:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:7", 2024-04-20T01:55:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000113", 2024-04-20T01:55:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000005", 2024-04-20T01:55:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:5", 2024-04-20T01:55:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:5", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000011", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000058", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000052", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:4", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:9", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.21", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:9", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:3", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:9", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:8", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:2", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:3", 2024-04-20T01:55:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:10", 2024-04-20T01:55:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:8", 2024-04-20T01:55:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27", 2024-04-20T01:55:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46", 2024-04-20T01:55:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:4", 2024-04-20T01:55:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:8", 2024-04-20T01:55:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:6", 2024-04-20T01:55:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:3", 2024-04-20T01:55:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:4", 2024-04-20T01:55:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:8", 2024-04-20T01:55:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18", 2024-04-20T01:55:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000088", 2024-04-20T01:55:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000080", 2024-04-20T01:55:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:3", 2024-04-20T01:55:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000090", 2024-04-20T01:55:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22", 2024-04-20T01:55:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:1", 2024-04-20T01:55:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:5", 2024-04-20T01:55:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:4", 2024-04-20T01:55:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:7", 2024-04-20T01:55:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000110", 2024-04-20T01:55:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:9", 2024-04-20T01:55:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:0", 2024-04-20T01:55:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.0", 2024-04-20T01:55:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:4", 2024-04-20T01:55:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000096", 2024-04-20T01:55:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:1", 2024-04-20T01:55:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000087", 2024-04-20T01:55:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000008", 2024-04-20T01:55:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:3", 2024-04-20T01:55:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:6", 2024-04-20T01:55:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:2", 2024-04-20T01:55:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:3", 2024-04-20T01:55:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:1", 2024-04-20T01:55:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.0.JXRWqBwqaXn.0", 2024-04-20T01:55:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000006", 2024-04-20T01:55:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:1", 2024-04-20T01:55:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29", 2024-04-20T01:55:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000029", 2024-04-20T01:55:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:7", 2024-04-20T01:55:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:9", 2024-04-20T01:55:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:7", 2024-04-20T01:55:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:6", 2024-04-20T01:55:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:10", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000089", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000022", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.0", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:9", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:7", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:7", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:6", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:2", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:10", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bilog.trim", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:6", 2024-04-20T01:55:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:3", 2024-04-20T01:55:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13", 2024-04-20T01:55:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000016", 2024-04-20T01:55:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:4", 2024-04-20T01:55:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:5", 2024-04-20T01:55:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:9", 2024-04-20T01:55:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:6", 2024-04-20T01:55:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:8", 2024-04-20T01:55:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:2", 2024-04-20T01:55:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000041", 2024-04-20T01:55:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10", 2024-04-20T01:55:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:10", 2024-04-20T01:55:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:1", 2024-04-20T01:55:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:7", 2024-04-20T01:55:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:1", 2024-04-20T01:55:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27", 2024-04-20T01:55:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25", 2024-04-20T01:55:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000018", 2024-04-20T01:55:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:6", 2024-04-20T01:55:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000092", 2024-04-20T01:55:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:9", 2024-04-20T01:55:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:6", 2024-04-20T01:55:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:2", 2024-04-20T01:55:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:0", 2024-04-20T01:55:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:2", 2024-04-20T01:55:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:10", 2024-04-20T01:55:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:5", 2024-04-20T01:55:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.27", 2024-04-20T01:55:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:0", 2024-04-20T01:55:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.4", 2024-04-20T01:55:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:8", 2024-04-20T01:55:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:0", 2024-04-20T01:55:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:6", 2024-04-20T01:55:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:8", 2024-04-20T01:55:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:4", 2024-04-20T01:55:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:6", 2024-04-20T01:55:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000014", 2024-04-20T01:55:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:2", 2024-04-20T01:55:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:3", 2024-04-20T01:55:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:6", 2024-04-20T01:55:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28", 2024-04-20T01:55:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000112", 2024-04-20T01:55:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:2", 2024-04-20T01:55:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000007", 2024-04-20T01:55:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000021", 2024-04-20T01:55:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:1", 2024-04-20T01:55:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:6", 2024-04-20T01:55:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.16", 2024-04-20T01:55:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:0", 2024-04-20T01:55:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000064", 2024-04-20T01:55:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:0", 2024-04-20T01:55:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16", 2024-04-20T01:55:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:8", 2024-04-20T01:55:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000071", 2024-04-20T01:55:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:1", 2024-04-20T01:55:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000074", 2024-04-20T01:55:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:8", 2024-04-20T01:55:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:8", 2024-04-20T01:55:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:6", 2024-04-20T01:55:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:10", 2024-04-20T01:55:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:7", 2024-04-20T01:55:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:1", 2024-04-20T01:55:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:0", 2024-04-20T01:55:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:6", 2024-04-20T01:55:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000081", 2024-04-20T01:55:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:5", 2024-04-20T01:55:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000009", 2024-04-20T01:55:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000121", 2024-04-20T01:55:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:3", 2024-04-20T01:55:25.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.1", 2024-04-20T01:55:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:9", 2024-04-20T01:55:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:6", 2024-04-20T01:55:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:2", 2024-04-20T01:55:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:8", 2024-04-20T01:55:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.17", 2024-04-20T01:55:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000125", 2024-04-20T01:55:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000082", 2024-04-20T01:55:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:7", 2024-04-20T01:55:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:0", 2024-04-20T01:55:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:8", 2024-04-20T01:55:25.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:7", 2024-04-20T01:55:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:6", 2024-04-20T01:55:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:6", 2024-04-20T01:55:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000105", 2024-04-20T01:55:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000059", 2024-04-20T01:55:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:8", 2024-04-20T01:55:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:1", 2024-04-20T01:55:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:10", 2024-04-20T01:55:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:2", 2024-04-20T01:55:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:5", 2024-04-20T01:55:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000077", 2024-04-20T01:55:25.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:10", 2024-04-20T01:55:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:8", 2024-04-20T01:55:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000032", 2024-04-20T01:55:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:8", 2024-04-20T01:55:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:2", 2024-04-20T01:55:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:3", 2024-04-20T01:55:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:5", 2024-04-20T01:55:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:3", 2024-04-20T01:55:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.24", 2024-04-20T01:55:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:10", 2024-04-20T01:55:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.2", 2024-04-20T01:55:25.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:1", 2024-04-20T01:55:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:4", 2024-04-20T01:55:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:5", 2024-04-20T01:55:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:0", 2024-04-20T01:55:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:1", 2024-04-20T01:55:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000053", 2024-04-20T01:55:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:4", 2024-04-20T01:55:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:7", 2024-04-20T01:55:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:6", 2024-04-20T01:55:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:7", 2024-04-20T01:55:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000091", 2024-04-20T01:55:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46", 2024-04-20T01:55:25.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:9", 2024-04-20T01:55:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:2", 2024-04-20T01:55:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:10", 2024-04-20T01:55:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:2", 2024-04-20T01:55:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000065", 2024-04-20T01:55:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:4", 2024-04-20T01:55:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000083", 2024-04-20T01:55:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000010", 2024-04-20T01:55:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:9", 2024-04-20T01:55:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:3", 2024-04-20T01:55:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:9", 2024-04-20T01:55:25.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:5", 2024-04-20T01:55:25.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:0", 2024-04-20T01:55:25.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:10", 2024-04-20T01:55:25.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:0", 2024-04-20T01:55:25.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.7", 2024-04-20T01:55:25.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25", 2024-04-20T01:55:25.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.3", 2024-04-20T01:55:25.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.3.retry", 2024-04-20T01:55:25.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:9", 2024-04-20T01:55:25.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:0", 2024-04-20T01:55:25.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:3", 2024-04-20T01:55:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:8", 2024-04-20T01:55:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:1", 2024-04-20T01:55:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:6", 2024-04-20T01:55:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:6", 2024-04-20T01:55:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:4", 2024-04-20T01:55:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:1", 2024-04-20T01:55:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:0", 2024-04-20T01:55:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:0", 2024-04-20T01:55:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:9", 2024-04-20T01:55:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:3", 2024-04-20T01:55:25.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:1", 2024-04-20T01:55:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:6", 2024-04-20T01:55:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:9", 2024-04-20T01:55:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:0", 2024-04-20T01:55:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mdlog.sync-status.shard.2", 2024-04-20T01:55:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:10", 2024-04-20T01:55:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:5", 2024-04-20T01:55:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11", 2024-04-20T01:55:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000045", 2024-04-20T01:55:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11", 2024-04-20T01:55:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:10", 2024-04-20T01:55:25.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:9", 2024-04-20T01:55:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000002", 2024-04-20T01:55:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.12", 2024-04-20T01:55:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:7", 2024-04-20T01:55:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:6", 2024-04-20T01:55:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:2", 2024-04-20T01:55:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:8", 2024-04-20T01:55:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000116", 2024-04-20T01:55:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:8", 2024-04-20T01:55:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:6", 2024-04-20T01:55:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "meta.log.f494ba41-e721-4411-8938-a297d95f7246.3", 2024-04-20T01:55:25.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.25", 2024-04-20T01:55:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:0", 2024-04-20T01:55:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000034", 2024-04-20T01:55:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:9", 2024-04-20T01:55:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:0", 2024-04-20T01:55:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.3.fX54X+2Z0es.0", 2024-04-20T01:55:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:4", 2024-04-20T01:55:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:1", 2024-04-20T01:55:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:10", 2024-04-20T01:55:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:7", 2024-04-20T01:55:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:4", 2024-04-20T01:55:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:0", 2024-04-20T01:55:25.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000101", 2024-04-20T01:55:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:0", 2024-04-20T01:55:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:0", 2024-04-20T01:55:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.13", 2024-04-20T01:55:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:1", 2024-04-20T01:55:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:10", 2024-04-20T01:55:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mdlog.sync-status.shard.1", 2024-04-20T01:55:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17", 2024-04-20T01:55:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:2", 2024-04-20T01:55:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:7", 2024-04-20T01:55:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:3", 2024-04-20T01:55:25.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:8", 2024-04-20T01:55:25.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000079", 2024-04-20T01:55:25.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:0", 2024-04-20T01:55:25.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000049", 2024-04-20T01:55:25.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000117", 2024-04-20T01:55:25.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000044", 2024-04-20T01:55:25.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:5", 2024-04-20T01:55:25.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24", 2024-04-20T01:55:25.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:1", 2024-04-20T01:55:25.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:7", 2024-04-20T01:55:25.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:1", 2024-04-20T01:55:25.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.14", 2024-04-20T01:55:25.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000066", 2024-04-20T01:55:25.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:7", 2024-04-20T01:55:25.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:0", 2024-04-20T01:55:25.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:2", 2024-04-20T01:55:25.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:4", 2024-04-20T01:55:25.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:2", 2024-04-20T01:55:25.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000068", 2024-04-20T01:55:25.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000085", 2024-04-20T01:55:25.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:5", 2024-04-20T01:55:25.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:9", 2024-04-20T01:55:25.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:9", 2024-04-20T01:55:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:3", 2024-04-20T01:55:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:8", 2024-04-20T01:55:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:1", 2024-04-20T01:55:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000073", 2024-04-20T01:55:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:7", 2024-04-20T01:55:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:2", 2024-04-20T01:55:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28", 2024-04-20T01:55:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000038", 2024-04-20T01:55:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19", 2024-04-20T01:55:25.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:1", 2024-04-20T01:55:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45", 2024-04-20T01:55:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:5", 2024-04-20T01:55:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:9", 2024-04-20T01:55:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.10", 2024-04-20T01:55:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.3", 2024-04-20T01:55:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000118", 2024-04-20T01:55:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:2", 2024-04-20T01:55:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:0", 2024-04-20T01:55:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:5", 2024-04-20T01:55:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000036", 2024-04-20T01:55:25.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:6", 2024-04-20T01:55:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:9", 2024-04-20T01:55:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:0", 2024-04-20T01:55:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:8", 2024-04-20T01:55:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:10", 2024-04-20T01:55:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:6", 2024-04-20T01:55:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:9", 2024-04-20T01:55:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000103", 2024-04-20T01:55:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:5", 2024-04-20T01:55:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:10", 2024-04-20T01:55:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18", 2024-04-20T01:55:25.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mdlog.sync-status.shard.3", 2024-04-20T01:55:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:10", 2024-04-20T01:55:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23", 2024-04-20T01:55:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:3", 2024-04-20T01:55:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:5", 2024-04-20T01:55:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:10", 2024-04-20T01:55:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000119", 2024-04-20T01:55:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:10", 2024-04-20T01:55:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:4", 2024-04-20T01:55:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:5", 2024-04-20T01:55:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:5", 2024-04-20T01:55:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:7", 2024-04-20T01:55:25.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000098", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:10", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:10", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.0", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mdlog.sync-status.shard.0", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:4", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:1", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:6", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000027", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:0", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:7", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mdlog.sync-status", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000019", 2024-04-20T01:55:25.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.2.retry", 2024-04-20T01:55:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:9", 2024-04-20T01:55:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000039", 2024-04-20T01:55:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:0", 2024-04-20T01:55:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10", 2024-04-20T01:55:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:3", 2024-04-20T01:55:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:1", 2024-04-20T01:55:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:8", 2024-04-20T01:55:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:10", 2024-04-20T01:55:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.2.retry", 2024-04-20T01:55:25.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:3", 2024-04-20T01:55:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:8", 2024-04-20T01:55:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:0", 2024-04-20T01:55:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:7", 2024-04-20T01:55:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:0", 2024-04-20T01:55:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000100", 2024-04-20T01:55:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:0", 2024-04-20T01:55:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:6", 2024-04-20T01:55:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:7", 2024-04-20T01:55:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:10", 2024-04-20T01:55:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:8", 2024-04-20T01:55:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:1", 2024-04-20T01:55:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:10", 2024-04-20T01:55:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:4", 2024-04-20T01:55:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000093", 2024-04-20T01:55:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:3", 2024-04-20T01:55:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:1", 2024-04-20T01:55:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:8", 2024-04-20T01:55:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:5", 2024-04-20T01:55:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22", 2024-04-20T01:55:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000004", 2024-04-20T01:55:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000063", 2024-04-20T01:55:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000122", 2024-04-20T01:55:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:10", 2024-04-20T01:55:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000057", 2024-04-20T01:55:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:0", 2024-04-20T01:55:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:9", 2024-04-20T01:55:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:8", 2024-04-20T01:55:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.1.retry", 2024-04-20T01:55:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:3", 2024-04-20T01:55:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000054", 2024-04-20T01:55:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:3", 2024-04-20T01:55:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000114", 2024-04-20T01:55:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:2", 2024-04-20T01:55:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.2", 2024-04-20T01:55:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.0", 2024-04-20T01:55:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:3", 2024-04-20T01:55:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000012", 2024-04-20T01:55:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:3", 2024-04-20T01:55:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.3", 2024-04-20T01:55:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:6", 2024-04-20T01:55:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:10", 2024-04-20T01:55:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:8", 2024-04-20T01:55:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000084", 2024-04-20T01:55:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000043", 2024-04-20T01:55:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47", 2024-04-20T01:55:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:7", 2024-04-20T01:55:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.11", 2024-04-20T01:55:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12", 2024-04-20T01:55:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:6", 2024-04-20T01:55:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:9", 2024-04-20T01:55:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:1", 2024-04-20T01:55:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:10", 2024-04-20T01:55:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:9", 2024-04-20T01:55:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:5", 2024-04-20T01:55:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000111", 2024-04-20T01:55:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:8", 2024-04-20T01:55:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:6", 2024-04-20T01:55:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:8", 2024-04-20T01:55:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000048", 2024-04-20T01:55:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:0", 2024-04-20T01:55:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:2", 2024-04-20T01:55:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000020", 2024-04-20T01:55:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:3", 2024-04-20T01:55:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:2", 2024-04-20T01:55:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:10", 2024-04-20T01:55:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:0", 2024-04-20T01:55:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:5", 2024-04-20T01:55:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:3", 2024-04-20T01:55:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:3", 2024-04-20T01:55:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000099", 2024-04-20T01:55:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000056", 2024-04-20T01:55:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:0", 2024-04-20T01:55:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:10", 2024-04-20T01:55:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:3", 2024-04-20T01:55:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:7", 2024-04-20T01:55:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.19", 2024-04-20T01:55:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:4", 2024-04-20T01:55:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:9", 2024-04-20T01:55:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.28", 2024-04-20T01:55:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:2", 2024-04-20T01:55:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:9", 2024-04-20T01:55:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:4", 2024-04-20T01:55:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000072", 2024-04-20T01:55:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:0", 2024-04-20T01:55:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15", 2024-04-20T01:55:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:10", 2024-04-20T01:55:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:2", 2024-04-20T01:55:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000062", 2024-04-20T01:55:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:1", 2024-04-20T01:55:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.2", 2024-04-20T01:55:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:3", 2024-04-20T01:55:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000115", 2024-04-20T01:55:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:7", 2024-04-20T01:55:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.1", 2024-04-20T01:55:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:1", 2024-04-20T01:55:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000033", 2024-04-20T01:55:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21", 2024-04-20T01:55:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:5", 2024-04-20T01:55:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:9", 2024-04-20T01:55:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:5", 2024-04-20T01:55:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:6", 2024-04-20T01:55:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:4", 2024-04-20T01:55:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:7", 2024-04-20T01:55:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:3", 2024-04-20T01:55:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:7", 2024-04-20T01:55:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:9", 2024-04-20T01:55:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:10", 2024-04-20T01:55:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:2", 2024-04-20T01:55:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:5", 2024-04-20T01:55:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:2", 2024-04-20T01:55:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:5", 2024-04-20T01:55:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17", 2024-04-20T01:55:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:6", 2024-04-20T01:55:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000025", 2024-04-20T01:55:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000123", 2024-04-20T01:55:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.31", 2024-04-20T01:55:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:9", 2024-04-20T01:55:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:0", 2024-04-20T01:55:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.18", 2024-04-20T01:55:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:8", 2024-04-20T01:55:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:5", 2024-04-20T01:55:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:8", 2024-04-20T01:55:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000108", 2024-04-20T01:55:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:2", 2024-04-20T01:55:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:5", 2024-04-20T01:55:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:0", 2024-04-20T01:55:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000094", 2024-04-20T01:55:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:6", 2024-04-20T01:55:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000060", 2024-04-20T01:55:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:7", 2024-04-20T01:55:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:5", 2024-04-20T01:55:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:8", 2024-04-20T01:55:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13", 2024-04-20T01:55:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000109", 2024-04-20T01:55:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.15", 2024-04-20T01:55:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:7", 2024-04-20T01:55:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000013", 2024-04-20T01:55:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:0", 2024-04-20T01:55:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:4", 2024-04-20T01:55:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:3", 2024-04-20T01:55:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26", 2024-04-20T01:55:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000051", 2024-04-20T01:55:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:3", 2024-04-20T01:55:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:8", 2024-04-20T01:55:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:10", 2024-04-20T01:55:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:6", 2024-04-20T01:55:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000106", 2024-04-20T01:55:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:1", 2024-04-20T01:55:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:10", 2024-04-20T01:55:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:5", 2024-04-20T01:55:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14", 2024-04-20T01:55:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000017", 2024-04-20T01:55:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-22:73f46f02-c7ab-4426-a267-2273af911da5.4354.22:0", 2024-04-20T01:55:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:2", 2024-04-20T01:55:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:6", 2024-04-20T01:55:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:0", 2024-04-20T01:55:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:0", 2024-04-20T01:55:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:10", 2024-04-20T01:55:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:4", 2024-04-20T01:55:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000023", 2024-04-20T01:55:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:1", 2024-04-20T01:55:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:7", 2024-04-20T01:55:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:9", 2024-04-20T01:55:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.2", 2024-04-20T01:55:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:9", 2024-04-20T01:55:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:4", 2024-04-20T01:55:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:5", 2024-04-20T01:55:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.1.retry", 2024-04-20T01:55:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:0", 2024-04-20T01:55:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000107", 2024-04-20T01:55:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:8", 2024-04-20T01:55:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:4", 2024-04-20T01:55:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000046", 2024-04-20T01:55:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:5", 2024-04-20T01:55:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:0", 2024-04-20T01:55:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:0", 2024-04-20T01:55:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.26", 2024-04-20T01:55:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:5", 2024-04-20T01:55:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:3", 2024-04-20T01:55:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20", 2024-04-20T01:55:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:8", 2024-04-20T01:55:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:4", 2024-04-20T01:55:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47", 2024-04-20T01:55:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:2", 2024-04-20T01:55:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.22", 2024-04-20T01:55:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:8", 2024-04-20T01:55:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:1", 2024-04-20T01:55:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.03463f3d-be9b-4035-b2ce-7cd216cfabda.1", 2024-04-20T01:55:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29:10", 2024-04-20T01:55:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_log.2", 2024-04-20T01:55:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:7", 2024-04-20T01:55:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:7", 2024-04-20T01:55:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000067", 2024-04-20T01:55:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:7", 2024-04-20T01:55:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:4", 2024-04-20T01:55:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:4", 2024-04-20T01:55:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-29:73f46f02-c7ab-4426-a267-2273af911da5.4354.29", 2024-04-20T01:55:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:8", 2024-04-20T01:55:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:7", 2024-04-20T01:55:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:0", 2024-04-20T01:55:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.9", 2024-04-20T01:55:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000086", 2024-04-20T01:55:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:7", 2024-04-20T01:55:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:2", 2024-04-20T01:55:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21", 2024-04-20T01:55:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.5", 2024-04-20T01:55:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:5", 2024-04-20T01:55:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.3.retry", 2024-04-20T01:55:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:6", 2024-04-20T01:55:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:4", 2024-04-20T01:55:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:1", 2024-04-20T01:55:25.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-19:73f46f02-c7ab-4426-a267-2273af911da5.4354.19:9", 2024-04-20T01:55:25.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:9", 2024-04-20T01:55:25.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync.error-log.20", 2024-04-20T01:55:25.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24", 2024-04-20T01:55:25.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:3", 2024-04-20T01:55:25.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:10", 2024-04-20T01:55:25.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23", 2024-04-20T01:55:25.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16", 2024-04-20T01:55:25.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:5", 2024-04-20T01:55:25.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:5", 2024-04-20T01:55:25.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:10", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:5", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-10:73f46f02-c7ab-4426-a267-2273af911da5.4354.10:2", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000050", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-25:73f46f02-c7ab-4426-a267-2273af911da5.4354.25:2", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000102", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-23:73f46f02-c7ab-4426-a267-2273af911da5.4354.23:4", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:9", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:6", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:10", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:4", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:9", 2024-04-20T01:55:25.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:2", 2024-04-20T01:55:25.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-12:73f46f02-c7ab-4426-a267-2273af911da5.4354.12:4", 2024-04-20T01:55:25.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:1", 2024-04-20T01:55:25.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:7", 2024-04-20T01:55:25.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000124", 2024-04-20T01:55:25.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "datalog.sync-status.shard.73f46f02-c7ab-4426-a267-2273af911da5.1", 2024-04-20T01:55:25.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.full-sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15", 2024-04-20T01:55:25.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-17:73f46f02-c7ab-4426-a267-2273af911da5.4354.17:9", 2024-04-20T01:55:25.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-45:73f46f02-c7ab-4426-a267-2273af911da5.4354.45:5", 2024-04-20T01:55:25.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:5", 2024-04-20T01:55:25.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:9", 2024-04-20T01:55:25.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:8", 2024-04-20T01:55:25.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000001", 2024-04-20T01:55:25.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:1", 2024-04-20T01:55:25.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000126", 2024-04-20T01:55:25.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:7", 2024-04-20T01:55:25.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000055", 2024-04-20T01:55:25.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:10", 2024-04-20T01:55:25.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:4", 2024-04-20T01:55:25.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-18:73f46f02-c7ab-4426-a267-2273af911da5.4354.18:7", 2024-04-20T01:55:25.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000061", 2024-04-20T01:55:25.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-21:73f46f02-c7ab-4426-a267-2273af911da5.4354.21:7", 2024-04-20T01:55:25.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:7", 2024-04-20T01:55:25.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:3", 2024-04-20T01:55:25.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000097", 2024-04-20T01:55:25.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:4", 2024-04-20T01:55:25.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-11:73f46f02-c7ab-4426-a267-2273af911da5.4354.11:4", 2024-04-20T01:55:25.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-26:73f46f02-c7ab-4426-a267-2273af911da5.4354.26:3", 2024-04-20T01:55:25.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-16:73f46f02-c7ab-4426-a267-2273af911da5.4354.16:4", 2024-04-20T01:55:25.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-48:73f46f02-c7ab-4426-a267-2273af911da5.5440.1:2", 2024-04-20T01:55:25.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000042", 2024-04-20T01:55:25.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:9", 2024-04-20T01:55:25.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-15:73f46f02-c7ab-4426-a267-2273af911da5.4354.15:1", 2024-04-20T01:55:25.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-46:73f46f02-c7ab-4426-a267-2273af911da5.4354.46:9", 2024-04-20T01:55:25.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:4", 2024-04-20T01:55:25.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-47:73f46f02-c7ab-4426-a267-2273af911da5.4354.47:7", 2024-04-20T01:55:25.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-28:73f46f02-c7ab-4426-a267-2273af911da5.4354.28:0", 2024-04-20T01:55:25.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:6", 2024-04-20T01:55:25.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:2", 2024-04-20T01:55:25.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-13:73f46f02-c7ab-4426-a267-2273af911da5.4354.13:6", 2024-04-20T01:55:25.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000075", 2024-04-20T01:55:25.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-27:73f46f02-c7ab-4426-a267-2273af911da5.4354.27:2", 2024-04-20T01:55:25.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000120", 2024-04-20T01:55:25.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj_delete_at_hint.0000000030", 2024-04-20T01:55:25.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-24:73f46f02-c7ab-4426-a267-2273af911da5.4354.24:10", 2024-04-20T01:55:25.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.73f46f02-c7ab-4426-a267-2273af911da5:aktlji-20:73f46f02-c7ab-4426-a267-2273af911da5.4354.20:4", 2024-04-20T01:55:25.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket.sync-status.03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-14:73f46f02-c7ab-4426-a267-2273af911da5.4354.14:10" 2024-04-20T01:55:25.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:25.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: 2024-04-20T01:55:25.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data checkpoint: source=test-zone1 target=test-zone2 2024-04-20T01:55:25.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:25.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000004737", 2024-04-20T01:55:25.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:10:02.651783Z" 2024-04-20T01:55:25.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:25.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000008823", 2024-04-20T01:55:25.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:10:02.653237Z" 2024-04-20T01:55:25.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:25.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000009809", 2024-04-20T01:55:25.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:10:02.654389Z" 2024-04-20T01:55:25.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:25.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000004178", 2024-04-20T01:55:25.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:10:02.655689Z" 2024-04-20T01:55:25.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:25.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:25.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:25.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:10:03Z" 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.312 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.312 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.312 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.312 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.312 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.312 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.312 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.312 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.312 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.312 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.312 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.313 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.313 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.313 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.313 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.313 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.313 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.313 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.313 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.313 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.313 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.314 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.314 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.314 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.314 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.314 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.314 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.314 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.314 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.314 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.314 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.315 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.315 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.315 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.315 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.315 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.315 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.315 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.315 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.315 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.315 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.315 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.316 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.316 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.316 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.316 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.316 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.316 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.316 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.316 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.316 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.316 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.316 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.316 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.317 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.317 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:03Z" 2024-04-20T01:55:25.317 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.317 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:10:08Z" 2024-04-20T01:55:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.323 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.323 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.323 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.323 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.323 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.323 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.323 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.323 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.323 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.323 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.323 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.323 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.324 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.324 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.324 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.324 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.324 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.324 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.324 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.324 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.325 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.325 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.325 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.325 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.325 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.325 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.325 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.325 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.325 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.326 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.326 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.326 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.326 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.326 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.326 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.326 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.326 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.326 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.326 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.326 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.327 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.327 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.327 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.327 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.327 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.327 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.327 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.327 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.327 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.327 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.327 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.328 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:08Z" 2024-04-20T01:55:25.328 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.328 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.328 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:10:13Z" 2024-04-20T01:55:25.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.334 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.334 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.334 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.334 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.334 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.334 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.334 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.334 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.335 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.335 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.335 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.335 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.335 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.335 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.335 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.335 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.335 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.335 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.335 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.335 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.336 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.336 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.336 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.336 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.336 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.336 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.336 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.336 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.336 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.336 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.336 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.336 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.337 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.337 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.337 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.337 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.337 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.337 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.337 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.337 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.337 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.337 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.337 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.338 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.338 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.338 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.338 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.338 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.338 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.338 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.338 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.338 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.338 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.339 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.339 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.339 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:13Z" 2024-04-20T01:55:25.339 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.339 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:10:18Z" 2024-04-20T01:55:25.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.345 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.345 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.345 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.346 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.346 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.346 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.346 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.346 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.346 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.346 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.346 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.346 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.346 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.346 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.347 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.347 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.347 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.347 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.347 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.347 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.347 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.347 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.347 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.347 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.348 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.348 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.348 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.348 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.348 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.348 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.348 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.348 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.348 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.348 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.348 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.349 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.349 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.349 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.349 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.349 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.349 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.349 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.349 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.349 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.349 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.349 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.349 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.350 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.350 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.350 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.350 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.350 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.350 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.350 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.350 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.350 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.351 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.351 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:18Z" 2024-04-20T01:55:25.351 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.351 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:10:24Z" 2024-04-20T01:55:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.357 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.357 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.357 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.358 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.358 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.358 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.358 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.358 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.358 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.358 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.358 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.358 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.358 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.358 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.359 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.359 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.359 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.359 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.359 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.359 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.359 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.359 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.359 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.360 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.360 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.360 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.360 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.360 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.360 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.360 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.360 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.360 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.360 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.360 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.360 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.361 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.361 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.361 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.361 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.361 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.361 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.361 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.361 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.361 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.361 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.361 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.362 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.362 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.362 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.362 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.362 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.362 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.362 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.362 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:24Z" 2024-04-20T01:55:25.362 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.362 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.362 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.363 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.363 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:10:30Z" 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.368 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.369 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.369 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.369 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.369 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.369 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.369 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.369 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.369 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.369 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.369 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.369 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.369 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.370 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.370 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.370 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.370 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.370 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.370 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.370 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.370 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.370 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.370 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.370 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.371 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.371 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.371 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.371 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.371 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.371 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.371 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.371 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.371 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.371 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.372 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.372 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.372 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.372 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.372 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.372 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.372 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.372 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.372 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.372 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.373 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.373 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.373 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.373 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.373 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.373 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.373 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.373 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.373 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.373 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.373 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.373 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.374 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:30Z" 2024-04-20T01:55:25.374 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.374 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:10:35Z" 2024-04-20T01:55:25.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.380 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.380 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.380 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.380 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.381 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.381 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.381 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.381 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.381 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.381 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.381 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.381 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.381 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.381 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.381 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.381 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.382 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.382 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.382 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.382 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.382 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.382 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.382 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.382 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.382 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.382 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.383 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.383 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.383 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.383 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.383 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.383 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.383 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.383 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.383 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.383 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.383 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.383 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.384 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.384 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.384 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.384 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.384 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.384 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.384 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.384 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.384 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.384 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.385 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.385 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.385 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.385 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.385 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.385 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.385 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.385 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.385 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:35Z" 2024-04-20T01:55:25.385 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.385 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:10:40Z" 2024-04-20T01:55:25.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.392 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.392 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.392 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.392 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.392 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.393 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.393 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.393 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.393 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.393 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.393 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.393 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.393 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.394 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.394 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.394 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.394 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.394 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.394 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.394 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.394 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.394 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.395 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.395 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.395 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.395 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.395 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.395 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.395 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.396 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.396 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.396 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.396 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.396 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.396 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.396 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.396 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.396 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.396 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.396 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.396 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.397 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.397 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.397 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.397 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.397 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.397 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.397 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.397 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:40Z" 2024-04-20T01:55:25.397 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.397 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:10:45Z" 2024-04-20T01:55:25.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.404 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.404 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.404 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.404 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.404 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.404 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.404 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.404 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.404 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.404 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.405 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.405 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.405 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.405 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.405 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.405 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.405 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.405 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.405 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.406 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.406 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.406 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.406 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.406 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.406 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.406 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.407 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.407 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.407 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.407 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.407 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.407 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.407 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.408 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.408 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.408 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.408 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.408 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.408 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.408 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.408 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.408 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.408 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.408 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.409 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.409 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.409 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:45Z" 2024-04-20T01:55:25.409 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.409 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:10:50Z" 2024-04-20T01:55:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.415 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.415 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.415 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.416 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.417 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.418 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.418 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.418 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.418 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.418 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.418 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.418 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.418 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.418 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.418 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.419 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.419 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.419 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.419 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.419 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.419 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.419 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.419 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.419 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.419 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.419 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.419 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.420 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.420 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.420 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.420 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.420 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.420 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:50Z" 2024-04-20T01:55:25.420 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.420 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.421 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.421 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:10:55Z" 2024-04-20T01:55:25.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.426 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.426 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.427 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.427 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.427 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.427 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.427 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.427 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.427 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.427 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.427 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.427 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.428 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.428 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.428 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.428 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.428 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.428 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.428 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.428 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.428 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.428 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.428 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.429 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.429 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.429 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.429 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.429 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.429 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.429 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.429 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.429 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.429 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.430 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.430 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.430 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.430 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.430 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.430 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.430 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.430 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.430 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.430 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.431 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.431 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.431 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.431 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.431 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.434 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.434 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.434 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.434 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.434 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.435 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:10:55Z" 2024-04-20T01:55:25.435 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.435 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:11:01Z" 2024-04-20T01:55:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.441 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.441 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.441 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.441 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.441 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.442 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.442 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.442 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.442 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.442 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.442 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.442 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.442 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.442 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.442 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.442 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.443 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.443 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.443 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.443 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.443 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.443 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.443 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.443 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.443 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.443 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.444 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.444 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.444 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.444 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.444 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.444 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.445 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.445 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.445 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.445 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.445 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.445 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.445 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.445 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.445 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.445 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.445 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.446 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.446 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.446 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.446 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.446 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.446 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:11:01Z" 2024-04-20T01:55:25.446 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.446 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:11:06Z" 2024-04-20T01:55:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.452 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.453 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.453 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.453 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.453 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.453 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.453 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.453 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.453 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.453 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.453 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.453 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.454 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.454 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.454 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.454 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.454 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.454 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.454 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.454 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.454 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.455 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.455 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.455 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.455 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.455 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.455 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.455 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.455 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.455 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.455 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.455 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.455 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.456 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.456 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.456 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.456 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.456 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.456 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.456 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.456 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.456 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.456 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.456 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.457 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.457 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.457 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.457 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.457 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.457 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.457 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.457 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.457 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.457 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.457 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.458 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:11:06Z" 2024-04-20T01:55:25.458 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.458 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.458 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.458 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:11:11Z" 2024-04-20T01:55:25.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.464 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.464 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.464 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.464 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.464 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.464 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.464 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.464 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.465 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.465 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.465 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.465 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.465 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.465 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.465 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.465 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.465 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.465 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.465 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.466 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.466 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.466 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.466 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.466 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.466 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.466 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.466 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.466 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.466 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.467 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.467 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.467 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.467 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.467 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.467 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.467 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.467 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.467 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.467 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.467 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.467 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.468 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.468 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.468 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.468 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.468 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.468 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.468 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.468 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.468 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.468 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.468 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.468 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:11:11Z" 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:11:16Z" 2024-04-20T01:55:25.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.475 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.475 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.475 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.475 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.475 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.475 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.476 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.476 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.476 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.476 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.476 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.476 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.476 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.476 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.476 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.476 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.476 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.476 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.477 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.477 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.477 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.477 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.477 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.477 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.477 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.477 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.477 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.477 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.477 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.478 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.478 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.478 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.478 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.478 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.478 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.478 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.478 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.478 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.478 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.478 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.479 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.479 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.479 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.479 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.479 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.479 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.479 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.479 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.479 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.479 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.479 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.480 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.480 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.480 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.480 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.480 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.480 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.480 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:11:16Z" 2024-04-20T01:55:25.480 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.480 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:11:21Z" 2024-04-20T01:55:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.487 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.487 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.487 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.487 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.487 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.487 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.487 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.487 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.487 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.487 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.488 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.488 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.488 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.488 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.488 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.488 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.488 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.488 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.488 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.488 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.488 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.489 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.489 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.489 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.489 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.489 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.489 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.489 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.489 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.489 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.489 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.489 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.489 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.490 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.490 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.490 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.490 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.490 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.490 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.490 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.490 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.490 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.490 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.490 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.491 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.491 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.491 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.491 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.491 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.491 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.491 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.491 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.491 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.491 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.491 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.491 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.492 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.492 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:11:21Z" 2024-04-20T01:55:25.492 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.492 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.492 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:11:26Z" 2024-04-20T01:55:25.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.498 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.498 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.498 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.498 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.498 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.499 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.499 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.499 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.499 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.499 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.499 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.499 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.499 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.499 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.499 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.499 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.500 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.500 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.500 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.500 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.500 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.500 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.500 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.500 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.501 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.501 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.501 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.501 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.501 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.501 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.501 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.501 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.501 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.502 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.502 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.502 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.502 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.502 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.502 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.502 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.502 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.502 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.502 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.502 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.503 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.503 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.503 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.503 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.503 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.503 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.503 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.503 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.503 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:11:26Z" 2024-04-20T01:55:25.503 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.503 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.504 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.504 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:11:31Z" 2024-04-20T01:55:25.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.510 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.510 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.510 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.510 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.510 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.510 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.510 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.511 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.511 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.511 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.511 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.511 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.511 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.511 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.511 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.511 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.511 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.511 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.512 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.513 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.513 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.513 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.513 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.513 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.513 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.513 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.513 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.513 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.513 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.513 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.514 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.514 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.514 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.514 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.514 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.514 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.514 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.514 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.514 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.514 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.514 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:11:31Z" 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:11:37Z" 2024-04-20T01:55:25.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.521 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.521 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.521 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.521 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.521 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.522 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.522 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.522 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.522 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.522 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.522 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.522 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.522 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.522 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.522 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.523 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.523 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.523 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.523 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.523 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.523 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.523 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.523 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.523 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.523 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.523 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.523 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.524 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.524 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.524 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.524 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.524 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.524 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.524 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.524 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.524 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.524 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.524 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.524 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.525 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.525 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.525 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.525 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.525 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.525 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.525 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.525 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.525 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.525 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.525 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.526 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.526 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.526 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.526 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.526 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.526 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.526 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:11:37Z" 2024-04-20T01:55:25.526 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.526 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.526 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.527 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.527 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:11:42Z" 2024-04-20T01:55:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.532 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.533 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.533 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.533 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.533 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.533 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.533 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.533 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.533 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.533 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.533 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.533 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.533 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.534 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.534 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.534 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.534 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.534 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.534 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.534 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.534 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.534 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.534 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.534 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.535 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.535 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.535 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.535 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.535 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.535 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.535 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.535 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.535 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.535 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.536 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.536 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.536 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.536 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.536 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.536 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.536 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.536 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.536 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.536 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.537 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.537 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.537 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.537 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.537 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.537 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.537 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.537 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.537 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.537 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.537 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:11:42Z" 2024-04-20T01:55:25.537 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.538 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:11:47Z" 2024-04-20T01:55:25.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.544 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.544 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.544 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.544 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.544 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.544 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.544 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.544 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.544 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.544 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.545 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.545 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.545 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.545 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.545 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.545 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.545 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.545 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.545 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.545 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.545 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.546 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.546 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.546 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.546 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.546 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.546 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.546 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.546 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.546 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.546 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.547 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.547 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.547 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.547 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.547 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.547 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.547 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.547 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.547 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.547 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.547 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.548 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.548 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.548 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.548 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.548 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.548 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.548 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.548 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.548 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.548 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.548 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.548 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.549 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.549 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:11:47Z" 2024-04-20T01:55:25.549 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.549 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.550 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.550 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:11:52Z" 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.556 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.557 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.557 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.557 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.557 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.557 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.557 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.557 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.557 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.557 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.557 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.557 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.558 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.558 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.558 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.558 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.558 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.558 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.558 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.559 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.559 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.559 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.559 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.559 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.559 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.559 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.559 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.559 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.559 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.559 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.560 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.560 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.560 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.560 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.560 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.560 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.560 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.560 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.560 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.561 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.561 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.561 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.561 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.561 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.561 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.561 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:11:52Z" 2024-04-20T01:55:25.561 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.561 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:11:57Z" 2024-04-20T01:55:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.569 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.569 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.569 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.569 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.569 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.569 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.569 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.569 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.569 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.569 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.570 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.570 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.570 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.570 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.570 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.570 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.570 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.570 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.570 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.570 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.570 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.571 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.571 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.571 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.571 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.571 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.571 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.571 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.571 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.571 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.571 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.571 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.572 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.572 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.572 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.572 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.572 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.572 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.572 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.572 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.572 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.572 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:11:57Z" 2024-04-20T01:55:25.573 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.573 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:12:02Z" 2024-04-20T01:55:25.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.580 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.580 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.580 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.580 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.580 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.580 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.580 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.580 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.580 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.580 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.581 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.581 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.581 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.581 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.581 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.581 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.581 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.581 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.581 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.582 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.582 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.582 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.582 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.582 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.582 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.582 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.582 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.582 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.582 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.582 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.583 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.583 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.583 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.583 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.583 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.583 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.583 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.583 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.583 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.583 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.584 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.584 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.584 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.584 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.584 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.584 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.584 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.584 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.584 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.584 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.584 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.585 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.585 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.585 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.585 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:12:02Z" 2024-04-20T01:55:25.585 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.585 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:12:07Z" 2024-04-20T01:55:25.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.592 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.592 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.592 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.592 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.592 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.592 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.592 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.592 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.592 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.592 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.592 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.593 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.593 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.593 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.593 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.593 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.593 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.593 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.593 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.593 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.594 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.594 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.594 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.594 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.594 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.594 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.594 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.594 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.594 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.594 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.594 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.595 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.595 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.595 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.595 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.595 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.595 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.595 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.595 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.595 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.595 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.595 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.596 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.597 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:12:07Z" 2024-04-20T01:55:25.597 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.597 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.597 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.597 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:12:13Z" 2024-04-20T01:55:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.603 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.603 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.603 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.603 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.603 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.603 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.604 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.604 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.604 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.604 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.604 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.604 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.604 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.604 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.604 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.604 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.604 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.605 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.605 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.605 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.605 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.605 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.605 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.605 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.605 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.605 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.605 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.605 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.605 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.606 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.606 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.606 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.606 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.606 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.606 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.606 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.606 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.607 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.607 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.607 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.607 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.607 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.607 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.607 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.607 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.607 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.607 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.607 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.607 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.608 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.608 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.608 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.608 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.608 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.608 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.608 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.608 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.608 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:12:13Z" 2024-04-20T01:55:25.608 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.609 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.609 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.609 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:12:18Z" 2024-04-20T01:55:25.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.615 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.615 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.615 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.615 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.615 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.615 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.615 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.615 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.616 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.616 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.616 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.616 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.616 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.616 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.616 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.616 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.616 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.616 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.616 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.616 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.617 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.617 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.617 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.617 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.617 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.617 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.617 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.617 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.617 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.617 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.617 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.617 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.618 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.618 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.618 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.618 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.618 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.618 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.618 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.618 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.618 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.618 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.618 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.619 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.619 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.619 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.619 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.619 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.619 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.619 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.619 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.619 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.619 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.619 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.619 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.620 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.620 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.620 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:12:18Z" 2024-04-20T01:55:25.620 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.620 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:12:23Z" 2024-04-20T01:55:25.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.626 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.626 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.626 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.627 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.627 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.627 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.627 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.627 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.627 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.627 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.627 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.627 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.627 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.627 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.627 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.628 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.628 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.628 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.628 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.628 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.628 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.628 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.628 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.628 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.628 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.628 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.629 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.629 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.629 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.629 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.629 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.629 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.629 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.629 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.629 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.630 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.630 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.630 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.630 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.630 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.630 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.630 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.630 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.630 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.630 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.630 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.631 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.631 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.631 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.631 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.631 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.631 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.631 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.631 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.631 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.631 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:12:23Z" 2024-04-20T01:55:25.631 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.631 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.632 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.632 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:12:28Z" 2024-04-20T01:55:25.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.638 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.638 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.638 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.638 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.638 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.639 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.639 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.639 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.639 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.639 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.639 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.639 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.639 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.639 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.639 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.639 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.641 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.642 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.642 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.642 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.642 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.642 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.642 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.642 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.642 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.642 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.643 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.644 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:12:28Z" 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:12:33Z" 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.650 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.651 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.653 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:12:33Z" 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:12:38Z" 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.659 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.660 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.661 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.662 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:12:38Z" 2024-04-20T01:55:25.663 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:12:43Z" 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.669 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.670 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.671 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.672 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:12:43Z" 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:12:49Z" 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.678 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.679 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.680 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.681 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:12:49Z" 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:12:54Z" 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.688 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.689 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.690 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.691 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:12:54Z" 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:12:59Z" 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.697 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.698 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.699 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.700 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:12:59Z" 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:13:04Z" 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.706 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.707 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.708 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.709 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.710 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:13:04Z" 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:13:09Z" 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.716 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.718 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.719 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:13:09Z" 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:13:14Z" 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.725 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.726 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.727 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:13:14Z" 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:13:20Z" 2024-04-20T01:55:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.736 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.737 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.738 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:13:20Z" 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:13:25Z" 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.744 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.745 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.746 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:13:25Z" 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:13:30Z" 2024-04-20T01:55:25.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.754 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.755 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.756 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.757 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:13:30Z" 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:13:35Z" 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.763 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.764 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.765 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:13:35Z" 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:13:40Z" 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.773 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.774 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.775 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.776 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:13:40Z" 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:13:45Z" 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.782 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.783 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.785 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:13:45Z" 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:13:50Z" 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.791 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.793 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.794 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.795 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:13:50Z" 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:13:56Z" 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.801 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.802 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.803 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.804 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.805 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.805 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.805 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.805 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.805 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.805 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:13:56Z" 2024-04-20T01:55:25.805 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.805 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:14:01Z" 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.811 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.812 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.813 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.814 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:14:01Z" 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:14:06Z" 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.820 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.821 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.822 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.823 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:14:06Z" 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:14:11Z" 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.829 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.830 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.831 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.832 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:14:11Z" 2024-04-20T01:55:25.833 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:14:16Z" 2024-04-20T01:55:25.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.839 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.839 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.839 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.839 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.839 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.839 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.839 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.839 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.839 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.843 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.843 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.843 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.843 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.843 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.843 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.844 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.845 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.846 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:14:16Z" 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:14:21Z" 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.852 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.853 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.854 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.855 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:14:21Z" 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:14:26Z" 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.861 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.862 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.863 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.864 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:14:26Z" 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:14:33Z" 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.871 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.872 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.873 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.874 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:14:33Z" 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:14:38Z" 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.880 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.881 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.882 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.883 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:14:38Z" 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:14:43Z" 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.889 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.890 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.891 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.892 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:14:43Z" 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.893 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:14:48Z" 2024-04-20T01:55:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.899 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.900 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.901 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.902 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:14:48Z" 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:14:53Z" 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.908 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.909 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.910 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.911 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:14:53Z" 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:14:58Z" 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.917 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.919 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.920 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.921 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:14:58Z" 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:15:03Z" 2024-04-20T01:55:25.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.927 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.927 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.927 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.927 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.927 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.927 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.927 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.927 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.928 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.929 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.930 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:15:03Z" 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 6925600971813139478 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 0, 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 0 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:15:09Z" 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.936 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: "instance_id": 6925600971813139478 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000003896", 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:33:20.691358Z" 2024-04-20T01:55:25.937 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007562", 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:54:37.707798Z" 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.938 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008408", 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:36:38.704801Z" 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:25.939 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000002914", 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:26:44.680751Z" 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:15:09Z" 2024-04-20T01:55:25.940 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:25.941 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003896', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:33:20.691358Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007562', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:54:37.707798Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008408', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:36:38.704801Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002914', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:26:44.680751Z'}}] 2024-04-20T01:55:25.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000004737', 1: '00000000000000000000:00000000000000008823', 2: '00000000000000000000:00000000000000009809', 3: '00000000000000000000:00000000000000004178'} 2024-04-20T01:55:25.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000003896', 1: '00000000000000000000:00000000000000007562', 2: '00000000000000000000:00000000000000008408', 3: '00000000000000000000:00000000000000002914'} 2024-04-20T01:55:25.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000004737 target=00000000000000000000:00000000000000003896, shard=1 master=00000000000000000000:00000000000000008823 target=00000000000000000000:00000000000000007562, shard=2 master=00000000000000000000:00000000000000009809 target=00000000000000000000:00000000000000008408, shard=3 master=00000000000000000000:00000000000000004178 target=00000000000000000000:00000000000000002914 2024-04-20T01:55:25.941 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:25.942 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.942 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:25.942 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2024-04-20T01:55:25.942 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:25.942 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:25.942 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1200, in test_bucket_sync_enable_right_after_disable 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket_name) 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-58 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-58/ 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-58/ 2024-04-20T01:55:25.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-58/ 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.944 INFO:tasks.rgw_multisite_tests:/aktlji-58/ 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Uw1N46Oi7wLiRVxDI2tTX4XvDIY= 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Uw1N46Oi7wLiRVxDI2tTX4XvDIY=', 'Content-Length': '0'} 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000507b64efe8535686-0066230912-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=aktlji-59 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-59/ 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-59/ 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-59/ 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests:/aktlji-59/ 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4/lrIrhRBl6B3Dj5e7mUWPuBnGA= 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4/lrIrhRBl6B3Dj5e7mUWPuBnGA=', 'Content-Length': '0'} 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000006a34385aebf0907c-0066230912-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=aktlji-60 2024-04-20T01:55:25.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-60/ 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-60/ 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-60/ 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.947 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:/aktlji-60/ 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:m893jTxAdDaNyLk47Ic1QlhqQOQ= 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:m893jTxAdDaNyLk47Ic1QlhqQOQ=', 'Content-Length': '0'} 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000008e735612b264b681-0066230912-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-58/ 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-58/ 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-58/ 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:25.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests:/aktlji-58/ 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU= 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU=', 'Content-Length': '0'} 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069d279ab4b1b6137-0066230912-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-58/obj1 2024-04-20T01:55:25.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-58/obj1 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-58/obj1 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.950 INFO:tasks.rgw_multisite_tests:/aktlji-58/obj1 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sJ3b+TWcUcWSkJyQx4jDQjJNMJw= 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sJ3b+TWcUcWSkJyQx4jDQjJNMJw='} 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af7857bfcdb35ead-0066230912-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-58/ 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-58/ 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-58/ 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests:/aktlji-58/ 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU= 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU=', 'Content-Length': '0'} 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092e18a1c5ee0e3f1-0066230912-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-58/obj2 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-58/obj2 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-58/obj2 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:/aktlji-58/obj2 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.953 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:FgfpwYYvfUFkYrXj7q9NKOEpJB0= 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FgfpwYYvfUFkYrXj7q9NKOEpJB0='} 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000570c3b0cebc218a4-0066230912-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-58/ 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-58/ 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-58/ 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.954 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests:/aktlji-58/ 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU= 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU=', 'Content-Length': '0'} 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ec78a372168cc2d-0066230912-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-58/obj3 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-58/obj3 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-58/obj3 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:/aktlji-58/obj3 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:kbB12d9lIrLPU8TWMs2RNHzo1eo= 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kbB12d9lIrLPU8TWMs2RNHzo1eo='} 2024-04-20T01:55:25.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f480254e4028d4e-0066230912-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-58/ 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-58/ 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-58/ 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.957 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:/aktlji-58/ 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU= 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cACAQBdgm0sInzm1vC71Sn7JZNU=', 'Content-Length': '0'} 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000188e84a987157055-0066230912-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-58/obj4 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-58/obj4 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-58/obj4 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:25.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.959 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.959 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:25.959 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:25.959 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.959 INFO:tasks.rgw_multisite_tests:/aktlji-58/obj4 2024-04-20T01:55:25.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.959 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8l4azwgngBPXs5a+64kq1SIe90E= 2024-04-20T01:55:25.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8l4azwgngBPXs5a+64kq1SIe90E='} 2024-04-20T01:55:25.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0a517c18ae3a4d0-0066230912-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-59/ 2024-04-20T01:55:25.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-59/ 2024-04-20T01:55:25.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:25.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-59/ 2024-04-20T01:55:25.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:25.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.961 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:25.961 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.961 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.961 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.961 INFO:tasks.rgw_multisite_tests:/aktlji-59/ 2024-04-20T01:55:25.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.961 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0= 2024-04-20T01:55:25.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0=', 'Content-Length': '0'} 2024-04-20T01:55:25.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ebd2cf450b1d602-0066230912-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-59/obj1 2024-04-20T01:55:25.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-59/obj1 2024-04-20T01:55:25.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-59/obj1 2024-04-20T01:55:25.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:25.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:25.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.962 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.962 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:25.962 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:25.962 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.963 INFO:tasks.rgw_multisite_tests:/aktlji-59/obj1 2024-04-20T01:55:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.963 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Q21Hu9dIGS8vlo9n1Hx4g4zHI28= 2024-04-20T01:55:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q21Hu9dIGS8vlo9n1Hx4g4zHI28='} 2024-04-20T01:55:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091d8749dec40f0d0-0066230912-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-59/ 2024-04-20T01:55:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-59/ 2024-04-20T01:55:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-59/ 2024-04-20T01:55:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:25.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.964 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:25.964 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.964 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.964 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.964 INFO:tasks.rgw_multisite_tests:/aktlji-59/ 2024-04-20T01:55:25.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.964 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0= 2024-04-20T01:55:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0=', 'Content-Length': '0'} 2024-04-20T01:55:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a838b0be22d70aa9-0066230912-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-59/obj2 2024-04-20T01:55:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-59/obj2 2024-04-20T01:55:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-59/obj2 2024-04-20T01:55:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.966 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.966 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:25.966 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:25.966 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.966 INFO:tasks.rgw_multisite_tests:/aktlji-59/obj2 2024-04-20T01:55:25.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.966 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:DZpeP8HndGLAoEcoKg21G8e8XIY= 2024-04-20T01:55:25.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DZpeP8HndGLAoEcoKg21G8e8XIY='} 2024-04-20T01:55:25.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f9f7efc2217011c-0066230912-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-59/ 2024-04-20T01:55:25.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-59/ 2024-04-20T01:55:25.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:25.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-59/ 2024-04-20T01:55:25.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:25.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.967 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:25.968 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.968 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.968 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.968 INFO:tasks.rgw_multisite_tests:/aktlji-59/ 2024-04-20T01:55:25.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.968 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0= 2024-04-20T01:55:25.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0=', 'Content-Length': '0'} 2024-04-20T01:55:25.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e0fa88933c781eb-0066230912-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-59/obj3 2024-04-20T01:55:25.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-59/obj3 2024-04-20T01:55:25.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-59/obj3 2024-04-20T01:55:25.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:25.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:25.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.969 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.969 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:25.969 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:25.969 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.969 INFO:tasks.rgw_multisite_tests:/aktlji-59/obj3 2024-04-20T01:55:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.970 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ULEgHV1bOeMrE52BEz529L17xsw= 2024-04-20T01:55:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ULEgHV1bOeMrE52BEz529L17xsw='} 2024-04-20T01:55:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009da038568d2a0d8-0066230912-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-59/ 2024-04-20T01:55:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-59/ 2024-04-20T01:55:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-59/ 2024-04-20T01:55:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:25.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.971 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:25.971 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.971 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.971 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.971 INFO:tasks.rgw_multisite_tests:/aktlji-59/ 2024-04-20T01:55:25.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.971 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0= 2024-04-20T01:55:25.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zuZl8vq2xFhKPr5WxAnF2fu4Gu0=', 'Content-Length': '0'} 2024-04-20T01:55:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029c879e4aa53d7b4-0066230912-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-59/obj4 2024-04-20T01:55:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-59/obj4 2024-04-20T01:55:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-59/obj4 2024-04-20T01:55:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.973 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.973 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:25.973 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:25.973 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.973 INFO:tasks.rgw_multisite_tests:/aktlji-59/obj4 2024-04-20T01:55:25.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.973 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:r+JfoDoW6G1xwR4BjxSftuM7MS8= 2024-04-20T01:55:25.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:r+JfoDoW6G1xwR4BjxSftuM7MS8='} 2024-04-20T01:55:25.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008af5bd50eaeb867e-0066230912-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-60/ 2024-04-20T01:55:25.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-60/ 2024-04-20T01:55:25.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:25.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-60/ 2024-04-20T01:55:25.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:25.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:25.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.974 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:25.974 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.975 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.975 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.975 INFO:tasks.rgw_multisite_tests:/aktlji-60/ 2024-04-20T01:55:25.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.975 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA= 2024-04-20T01:55:25.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA=', 'Content-Length': '0'} 2024-04-20T01:55:25.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000105d76e60edc62eb-0066230912-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-60/obj1 2024-04-20T01:55:25.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-60/obj1 2024-04-20T01:55:25.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-60/obj1 2024-04-20T01:55:25.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:25.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:25.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:25.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.976 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.976 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:25.976 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:25.976 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.976 INFO:tasks.rgw_multisite_tests:/aktlji-60/obj1 2024-04-20T01:55:25.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.977 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:C34MfmQDDYUoFQfbL91D1v217oE= 2024-04-20T01:55:25.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:C34MfmQDDYUoFQfbL91D1v217oE='} 2024-04-20T01:55:25.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001524945254796eaf-0066230912-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-60/ 2024-04-20T01:55:25.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-60/ 2024-04-20T01:55:25.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:25.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-60/ 2024-04-20T01:55:25.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:25.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:25.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.978 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:25.978 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.978 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.978 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.978 INFO:tasks.rgw_multisite_tests:/aktlji-60/ 2024-04-20T01:55:25.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.978 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA= 2024-04-20T01:55:25.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA=', 'Content-Length': '0'} 2024-04-20T01:55:25.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000077bc338feace2cd-0066230912-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-60/obj2 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-60/obj2 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-60/obj2 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.979 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:25.980 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:25.980 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.980 INFO:tasks.rgw_multisite_tests:/aktlji-60/obj2 2024-04-20T01:55:25.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.980 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:lCkXsPpkxknOP6hOIrCpzfeRN+Q= 2024-04-20T01:55:25.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lCkXsPpkxknOP6hOIrCpzfeRN+Q='} 2024-04-20T01:55:25.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c23ad40f3f88510d-0066230912-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-60/ 2024-04-20T01:55:25.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-60/ 2024-04-20T01:55:25.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:25.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-60/ 2024-04-20T01:55:25.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:25.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:25.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.981 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:25.981 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.981 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.981 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.982 INFO:tasks.rgw_multisite_tests:/aktlji-60/ 2024-04-20T01:55:25.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.982 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA= 2024-04-20T01:55:25.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA=', 'Content-Length': '0'} 2024-04-20T01:55:25.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007e47ce5b3f5d495-0066230912-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-60/obj3 2024-04-20T01:55:25.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-60/obj3 2024-04-20T01:55:25.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-60/obj3 2024-04-20T01:55:25.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:25.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:25.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:25.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.983 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.983 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:25.983 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:25.983 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.983 INFO:tasks.rgw_multisite_tests:/aktlji-60/obj3 2024-04-20T01:55:25.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.984 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rGAljMQNIAOotE46qWJxJWhlj80= 2024-04-20T01:55:25.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rGAljMQNIAOotE46qWJxJWhlj80='} 2024-04-20T01:55:25.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6caf0178f892c29-0066230912-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-60/ 2024-04-20T01:55:25.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-60/ 2024-04-20T01:55:25.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:25.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-60/ 2024-04-20T01:55:25.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:25.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:25.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:25.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.985 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:25.985 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.985 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:25.985 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.985 INFO:tasks.rgw_multisite_tests:/aktlji-60/ 2024-04-20T01:55:25.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.985 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA= 2024-04-20T01:55:25.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fw/BlgYtbr6giAa+iWsek1oH6AA=', 'Content-Length': '0'} 2024-04-20T01:55:25.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f197682550131549-0066230912-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-60/obj4 2024-04-20T01:55:25.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-60/obj4 2024-04-20T01:55:25.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:25.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-60/obj4 2024-04-20T01:55:25.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:25.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:25.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:25.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:25.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:25.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:25.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:25.987 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:25.987 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:25.987 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:25.987 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:15:14 GMT 2024-04-20T01:55:25.987 INFO:tasks.rgw_multisite_tests:/aktlji-60/obj4 2024-04-20T01:55:25.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:25.987 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:UxTm6skByjgL2sOKXu8AM48jQl0= 2024-04-20T01:55:25.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:15:14 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UxTm6skByjgL2sOKXu8AM48jQl0='} 2024-04-20T01:55:25.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a8f3aa5986a534a-0066230912-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:15:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:25.987 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.987 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:25.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713572114.281893_688.1", 2024-04-20T01:55:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:15:14.281893Z" 2024-04-20T01:55:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:25.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:25.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572114.281893_688.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:25.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:25.989 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.990 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:15:14.791+0000 7f40909a9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:15:14.792+0000 7f40909a9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:25.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:25.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713572114.281893_688.1", 2024-04-20T01:55:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:15:14.281893Z", 2024-04-20T01:55:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:25.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:15:14Z" 2024-04-20T01:55:25.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:25.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:25.996 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:25.996 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:25.996 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:25.996 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:25.996 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:25.996 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.996 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.996 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:25.996 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.996 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:25.996 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.997 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.997 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:25.997 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.997 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:25.997 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.997 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:25.997 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.997 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.997 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.997 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.997 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:25.997 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.998 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.998 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:25.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.998 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.998 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.998 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:25.998 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.998 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.998 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.998 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.998 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:25.998 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:25.999 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:25.999 INFO:tasks.rgw_multisite_tests: "marker": "1_1713572114.281893_688.1", 2024-04-20T01:55:25.999 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:25.999 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:25.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:25.999 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:15:14.281893Z", 2024-04-20T01:55:25.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:25.999 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:25.999 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:25.999 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:25.999 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:25.999 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.000 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.000 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.000 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.000 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.000 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.000 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.000 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.001 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.001 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.001 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.001 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:15:14Z" 2024-04-20T01:55:26.001 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.001 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572114.281893_688.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:15:14.281893Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572114.281893_688.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572114.281893_688.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.001 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:14.918+0000 7f0e9b20c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:14.919+0000 7f0e9b20c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713572114.281893_688.1", 2024-04-20T01:55:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T00:15:14.281893Z", 2024-04-20T01:55:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T00:15:14Z" 2024-04-20T01:55:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.008 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.008 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.008 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.008 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.008 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.008 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.008 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.008 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.008 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.008 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.008 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.009 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.009 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.009 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.009 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.009 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.009 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.009 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.009 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.009 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.009 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.009 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.010 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.010 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.010 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.010 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.010 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.010 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.010 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.010 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.010 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.010 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.010 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.011 INFO:tasks.rgw_multisite_tests: "marker": "1_1713572114.281893_688.1", 2024-04-20T01:55:26.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.011 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.011 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.011 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:15:14.281893Z", 2024-04-20T01:55:26.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.011 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.011 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.011 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.011 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.012 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.012 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.012 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.012 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.012 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.012 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.012 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.012 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.013 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.013 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.013 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.013 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:15:14Z" 2024-04-20T01:55:26.013 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.013 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572114.281893_688.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:15:14.281893Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572114.281893_688.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572114.281893_688.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-58 2024-04-20T01:55:26.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-58 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-58 2024-04-20T01:55:26.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-58 2024-04-20T01:55:26.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-58', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-58 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:15.191+0000 7fcd9462e780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:20.192+0000 7fcd9462e780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:25.194+0000 7fcd9462e780 1 bucket sync caught up with source: 2024-04-20T01:55:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [00000000001.25.6, , 00000000001.33.6, , 00000000001.35.6, , , 00000000001.50.6, , , ] 2024-04-20T01:55:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [00000000001.25.6, , 00000000001.33.6, , 00000000001.35.6, , , 00000000001.50.6, , , ] 2024-04-20T01:55:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:25.194+0000 7fcd9462e780 0 bucket checkpoint complete 2024-04-20T01:55:26.015 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-58 2024-04-20T01:55:26.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-58', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-58', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-58 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:25.321+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:30.324+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:35.325+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:40.328+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:45.328+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:50.329+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:15:55.330+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:16:00.330+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:16:05.331+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:16:10.332+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:16:15.332+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:16:20.333+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:16:25.334+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:16:30.336+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:16:35.336+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:16:40.336+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:16:45.337+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:16:50.337+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:16:55.338+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:17:00.340+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:17:05.341+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:17:10.341+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:17:15.342+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:17:20.342+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:17:25.343+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:17:30.345+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:17:35.345+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:17:40.345+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:17:45.346+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:17:50.347+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:17:55.348+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:18:00.349+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:18:05.349+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:18:10.350+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:18:15.351+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:18:20.351+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:18:25.352+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:18:30.352+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:18:35.352+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:18:40.353+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:18:45.354+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:18:50.356+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:18:55.357+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:19:00.357+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:19:05.358+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:19:10.358+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:19:15.359+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:19:20.360+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:19:25.361+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:19:30.361+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:19:35.362+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:19:40.363+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:19:45.364+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:19:50.365+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:19:55.365+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:00.366+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:05.366+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:10.367+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:15.366+0000 7fa062b01780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:20.367+0000 7fa062b01780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:20.367+0000 7fa062b01780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:20.367+0000 7fa062b01780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.021 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.021 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.022 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.022 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable_enable 2024-04-20T01:55:26.022 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.022 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.022 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.022 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.022 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1235, in test_bucket_sync_disable_enable 2024-04-20T01:55:26.022 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket_name) 2024-04-20T01:55:26.023 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.023 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.023 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.023 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.023 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.023 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.023 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.023 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-61 2024-04-20T01:55:26.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-61/ 2024-04-20T01:55:26.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-61/ 2024-04-20T01:55:26.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-61/ 2024-04-20T01:55:26.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.024 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.024 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.025 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.025 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.025 INFO:tasks.rgw_multisite_tests:/aktlji-61/ 2024-04-20T01:55:26.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.025 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zELhLL8pFAhZyy2zQmsdh4kFWbM= 2024-04-20T01:55:26.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zELhLL8pFAhZyy2zQmsdh4kFWbM=', 'Content-Length': '0'} 2024-04-20T01:55:26.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000007280f33b15bf3dbf-0066230a44-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=aktlji-62 2024-04-20T01:55:26.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-62/ 2024-04-20T01:55:26.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-62/ 2024-04-20T01:55:26.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-62/ 2024-04-20T01:55:26.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.026 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.026 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.026 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.027 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.027 INFO:tasks.rgw_multisite_tests:/aktlji-62/ 2024-04-20T01:55:26.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.027 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:bJu5FKskFP+s98QairhbeSVBcyk= 2024-04-20T01:55:26.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bJu5FKskFP+s98QairhbeSVBcyk=', 'Content-Length': '0'} 2024-04-20T01:55:26.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000006553eba53bd93cdf-0066230a44-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=aktlji-63 2024-04-20T01:55:26.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-63/ 2024-04-20T01:55:26.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-63/ 2024-04-20T01:55:26.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-63/ 2024-04-20T01:55:26.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:26.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:26.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T01:55:26.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.028 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.028 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.028 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.028 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.028 INFO:tasks.rgw_multisite_tests:/aktlji-63/ 2024-04-20T01:55:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.029 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sy5sRecQvBss291kAJ87TF8JQsw= 2024-04-20T01:55:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sy5sRecQvBss291kAJ87TF8JQsw=', 'Content-Length': '0'} 2024-04-20T01:55:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a29df66556c19c2f-0066230a44-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-61/ 2024-04-20T01:55:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-61/ 2024-04-20T01:55:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-61/ 2024-04-20T01:55:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.030 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.030 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.030 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.030 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.030 INFO:tasks.rgw_multisite_tests:/aktlji-61/ 2024-04-20T01:55:26.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.030 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c= 2024-04-20T01:55:26.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c=', 'Content-Length': '0'} 2024-04-20T01:55:26.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025bbf59016fe55a0-0066230a44-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-61/obj1 2024-04-20T01:55:26.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-61/obj1 2024-04-20T01:55:26.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-61/obj1 2024-04-20T01:55:26.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:26.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.032 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.032 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:26.032 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.032 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.032 INFO:tasks.rgw_multisite_tests:/aktlji-61/obj1 2024-04-20T01:55:26.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.032 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:B8g5HTe3ec0Cg+5yBwW3HwIrNb8= 2024-04-20T01:55:26.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:B8g5HTe3ec0Cg+5yBwW3HwIrNb8='} 2024-04-20T01:55:26.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000913bea91a45ca312-0066230a44-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-61/ 2024-04-20T01:55:26.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-61/ 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-61/ 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.033 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.034 INFO:tasks.rgw_multisite_tests:/aktlji-61/ 2024-04-20T01:55:26.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.034 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c= 2024-04-20T01:55:26.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c=', 'Content-Length': '0'} 2024-04-20T01:55:26.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000769f872b6c4a16fe-0066230a44-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-61/obj2 2024-04-20T01:55:26.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-61/obj2 2024-04-20T01:55:26.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-61/obj2 2024-04-20T01:55:26.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:26.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.035 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.035 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:26.035 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.035 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.035 INFO:tasks.rgw_multisite_tests:/aktlji-61/obj2 2024-04-20T01:55:26.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.035 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qnDAvb9cyiqAVAfIS/q9QTsDJAw= 2024-04-20T01:55:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qnDAvb9cyiqAVAfIS/q9QTsDJAw='} 2024-04-20T01:55:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009ed0cd32c15f9d2-0066230a44-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-61/ 2024-04-20T01:55:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-61/ 2024-04-20T01:55:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-61/ 2024-04-20T01:55:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.037 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.037 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.037 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.037 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.037 INFO:tasks.rgw_multisite_tests:/aktlji-61/ 2024-04-20T01:55:26.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.037 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c= 2024-04-20T01:55:26.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c=', 'Content-Length': '0'} 2024-04-20T01:55:26.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050ea4574c5444c6a-0066230a44-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-61/obj3 2024-04-20T01:55:26.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-61/obj3 2024-04-20T01:55:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-61/obj3 2024-04-20T01:55:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.038 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.038 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:26.038 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.039 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.039 INFO:tasks.rgw_multisite_tests:/aktlji-61/obj3 2024-04-20T01:55:26.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.039 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ujMVbv0LQR3hOXa7EXeA20YRKvs= 2024-04-20T01:55:26.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ujMVbv0LQR3hOXa7EXeA20YRKvs='} 2024-04-20T01:55:26.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da8edbf4c7dd647e-0066230a44-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-61/ 2024-04-20T01:55:26.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-61/ 2024-04-20T01:55:26.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-61/ 2024-04-20T01:55:26.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.040 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.040 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.040 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.040 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.040 INFO:tasks.rgw_multisite_tests:/aktlji-61/ 2024-04-20T01:55:26.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.041 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c= 2024-04-20T01:55:26.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zfpuMoV7w8CRGMeHbrXrtnJBq3c=', 'Content-Length': '0'} 2024-04-20T01:55:26.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008137e8ecf65ca9f-0066230a44-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-61/obj4 2024-04-20T01:55:26.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-61/obj4 2024-04-20T01:55:26.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-61/obj4 2024-04-20T01:55:26.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:26.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.042 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.042 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:26.042 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.042 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.042 INFO:tasks.rgw_multisite_tests:/aktlji-61/obj4 2024-04-20T01:55:26.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.042 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zMlrHofW2ZFCv0OJMaOkdGL6LvA= 2024-04-20T01:55:26.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zMlrHofW2ZFCv0OJMaOkdGL6LvA='} 2024-04-20T01:55:26.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f33c0af208f8cb95-0066230a44-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-62/ 2024-04-20T01:55:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-62/ 2024-04-20T01:55:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-62/ 2024-04-20T01:55:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.043 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.044 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.044 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.044 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.044 INFO:tasks.rgw_multisite_tests:/aktlji-62/ 2024-04-20T01:55:26.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.044 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4= 2024-04-20T01:55:26.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4=', 'Content-Length': '0'} 2024-04-20T01:55:26.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af7de60da7202999-0066230a44-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-62/obj1 2024-04-20T01:55:26.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-62/obj1 2024-04-20T01:55:26.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-62/obj1 2024-04-20T01:55:26.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:26.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.045 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.045 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:26.045 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.046 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.046 INFO:tasks.rgw_multisite_tests:/aktlji-62/obj1 2024-04-20T01:55:26.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.046 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hdG8QdSKDvfX3To0iOzswI9zdck= 2024-04-20T01:55:26.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hdG8QdSKDvfX3To0iOzswI9zdck='} 2024-04-20T01:55:26.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000055bf84b1ca46dd3-0066230a44-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-62/ 2024-04-20T01:55:26.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-62/ 2024-04-20T01:55:26.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-62/ 2024-04-20T01:55:26.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.047 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.047 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.047 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.047 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.047 INFO:tasks.rgw_multisite_tests:/aktlji-62/ 2024-04-20T01:55:26.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.048 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4= 2024-04-20T01:55:26.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4=', 'Content-Length': '0'} 2024-04-20T01:55:26.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2747f0692b0c5cc-0066230a44-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-62/obj2 2024-04-20T01:55:26.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-62/obj2 2024-04-20T01:55:26.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-62/obj2 2024-04-20T01:55:26.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:26.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.049 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.049 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:26.049 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.049 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.049 INFO:tasks.rgw_multisite_tests:/aktlji-62/obj2 2024-04-20T01:55:26.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.049 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8i0Tzo9GXSavfamHyrB2UNTkveM= 2024-04-20T01:55:26.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8i0Tzo9GXSavfamHyrB2UNTkveM='} 2024-04-20T01:55:26.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d08169187e5672c-0066230a44-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-62/ 2024-04-20T01:55:26.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-62/ 2024-04-20T01:55:26.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-62/ 2024-04-20T01:55:26.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.051 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.051 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.051 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.051 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.051 INFO:tasks.rgw_multisite_tests:/aktlji-62/ 2024-04-20T01:55:26.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.052 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4= 2024-04-20T01:55:26.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4=', 'Content-Length': '0'} 2024-04-20T01:55:26.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b288a45c879d9c8e-0066230a44-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-62/obj3 2024-04-20T01:55:26.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-62/obj3 2024-04-20T01:55:26.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-62/obj3 2024-04-20T01:55:26.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:26.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.053 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.053 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:26.053 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.053 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.053 INFO:tasks.rgw_multisite_tests:/aktlji-62/obj3 2024-04-20T01:55:26.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.053 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:DCf5raunn+HVOZUoJ34efX1+qps= 2024-04-20T01:55:26.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:DCf5raunn+HVOZUoJ34efX1+qps='} 2024-04-20T01:55:26.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f56ba6ad076f550-0066230a44-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-62/ 2024-04-20T01:55:26.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-62/ 2024-04-20T01:55:26.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-62/ 2024-04-20T01:55:26.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.055 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.055 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.055 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.055 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.055 INFO:tasks.rgw_multisite_tests:/aktlji-62/ 2024-04-20T01:55:26.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.055 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4= 2024-04-20T01:55:26.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:40G8YDwYzW+ACzu2ydzuxUwgbh4=', 'Content-Length': '0'} 2024-04-20T01:55:26.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032c0ac9ac405ed29-0066230a44-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-62/obj4 2024-04-20T01:55:26.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-62/obj4 2024-04-20T01:55:26.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-62/obj4 2024-04-20T01:55:26.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:26.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.056 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.056 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:26.057 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.057 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.057 INFO:tasks.rgw_multisite_tests:/aktlji-62/obj4 2024-04-20T01:55:26.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.057 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hLcjJ8Trc4EhT93SvuWoYzmCBoY= 2024-04-20T01:55:26.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hLcjJ8Trc4EhT93SvuWoYzmCBoY='} 2024-04-20T01:55:26.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d9b3a56fdb9c22a-0066230a44-1bfc-test-zone2'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-63/ 2024-04-20T01:55:26.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-63/ 2024-04-20T01:55:26.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-63/ 2024-04-20T01:55:26.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:26.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:26.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.058 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.058 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.058 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.058 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.058 INFO:tasks.rgw_multisite_tests:/aktlji-63/ 2024-04-20T01:55:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.059 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs= 2024-04-20T01:55:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs=', 'Content-Length': '0'} 2024-04-20T01:55:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2d52d2fa7c11271-0066230a44-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-63/obj1 2024-04-20T01:55:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-63/obj1 2024-04-20T01:55:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-63/obj1 2024-04-20T01:55:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:26.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:26.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.060 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.060 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:26.060 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.060 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.060 INFO:tasks.rgw_multisite_tests:/aktlji-63/obj1 2024-04-20T01:55:26.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.060 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:inhQ3CBVkAbuxBlccsbFkC3sPWo= 2024-04-20T01:55:26.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:inhQ3CBVkAbuxBlccsbFkC3sPWo='} 2024-04-20T01:55:26.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9deee732e798844-0066230a44-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-63/ 2024-04-20T01:55:26.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-63/ 2024-04-20T01:55:26.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-63/ 2024-04-20T01:55:26.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:26.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:26.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.062 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.062 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.062 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.062 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.062 INFO:tasks.rgw_multisite_tests:/aktlji-63/ 2024-04-20T01:55:26.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.062 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs= 2024-04-20T01:55:26.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs=', 'Content-Length': '0'} 2024-04-20T01:55:26.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ada6f1c8a021e1e-0066230a44-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-63/obj2 2024-04-20T01:55:26.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-63/obj2 2024-04-20T01:55:26.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-63/obj2 2024-04-20T01:55:26.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:26.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:26.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:26.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.063 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.063 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:26.064 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.064 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.064 INFO:tasks.rgw_multisite_tests:/aktlji-63/obj2 2024-04-20T01:55:26.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.064 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eNd/t8fOGs4X9kg06udtRv00+jI= 2024-04-20T01:55:26.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eNd/t8fOGs4X9kg06udtRv00+jI='} 2024-04-20T01:55:26.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059809a182e0876fc-0066230a44-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-63/ 2024-04-20T01:55:26.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-63/ 2024-04-20T01:55:26.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-63/ 2024-04-20T01:55:26.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:26.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:26.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.065 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.065 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.065 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.065 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.065 INFO:tasks.rgw_multisite_tests:/aktlji-63/ 2024-04-20T01:55:26.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.066 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs= 2024-04-20T01:55:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs=', 'Content-Length': '0'} 2024-04-20T01:55:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb2c723e79526275-0066230a44-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-63/obj3 2024-04-20T01:55:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-63/obj3 2024-04-20T01:55:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-63/obj3 2024-04-20T01:55:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:26.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:26.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:26.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.067 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.067 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:26.067 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.067 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.067 INFO:tasks.rgw_multisite_tests:/aktlji-63/obj3 2024-04-20T01:55:26.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.067 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:LsPEGFxoe+Hd3KIHJbrSgAIfX0U= 2024-04-20T01:55:26.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:LsPEGFxoe+Hd3KIHJbrSgAIfX0U='} 2024-04-20T01:55:26.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036185da51af73b7a-0066230a44-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-63/ 2024-04-20T01:55:26.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-63/ 2024-04-20T01:55:26.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-63/ 2024-04-20T01:55:26.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:26.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:26.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.069 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.069 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.069 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.069 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.069 INFO:tasks.rgw_multisite_tests:/aktlji-63/ 2024-04-20T01:55:26.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.069 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs= 2024-04-20T01:55:26.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+bqFjqDoRd2c8bjNhmJYRGQjWxs=', 'Content-Length': '0'} 2024-04-20T01:55:26.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb42a48cdb7b18bd-0066230a44-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-63/obj4 2024-04-20T01:55:26.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-63/obj4 2024-04-20T01:55:26.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-63/obj4 2024-04-20T01:55:26.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:26.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:26.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:26.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.070 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.070 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:26.070 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.071 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:20:20 GMT 2024-04-20T01:55:26.071 INFO:tasks.rgw_multisite_tests:/aktlji-63/obj4 2024-04-20T01:55:26.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.071 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:yu0Lf8/p6zOFSWzWWDDxi7sISCo= 2024-04-20T01:55:26.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:20:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yu0Lf8/p6zOFSWzWWDDxi7sISCo='} 2024-04-20T01:55:26.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041cbd7e5d46822ec-0066230a44-15ba-test-zone3'), ('Date', 'Sat, 20 Apr 2024 00:20:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713572420.485008_738.1", 2024-04-20T01:55:26.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:20:20.485008Z" 2024-04-20T01:55:26.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:26.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.485008_738.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:20:21.040+0000 7f173d1d2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:20:21.041+0000 7f173d1d2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713572420.485008_738.1", 2024-04-20T01:55:26.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:20:20.485008Z", 2024-04-20T01:55:26.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:20:21Z" 2024-04-20T01:55:26.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.080 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.080 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.080 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.080 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.080 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.080 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.080 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.080 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.080 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.081 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.081 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.081 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.081 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.081 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.081 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.081 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.081 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.081 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.081 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.081 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.082 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.082 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.082 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.082 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.082 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.082 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.082 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.082 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.082 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.082 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.082 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.083 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.083 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.083 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.083 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.083 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.083 INFO:tasks.rgw_multisite_tests: "marker": "1_1713572420.485008_738.1", 2024-04-20T01:55:26.083 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.083 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.083 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.083 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:20:20.485008Z", 2024-04-20T01:55:26.083 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.083 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.084 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.084 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.084 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.084 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.084 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.084 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.084 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.084 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.084 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.084 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.085 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.085 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.085 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.085 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.085 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.085 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.085 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.085 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:20:21Z" 2024-04-20T01:55:26.085 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.085 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572420.485008_738.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:20:20.485008Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.485008_738.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.485008_738.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:21.168+0000 7f1a3819e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:21.169+0000 7f1a3819e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713572420.432178_731.1", 2024-04-20T01:55:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T00:20:20.432178Z", 2024-04-20T01:55:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T00:20:21Z" 2024-04-20T01:55:26.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.092 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.092 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.092 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.092 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.092 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.092 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.093 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.093 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.093 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.093 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.093 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.093 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.093 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.093 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.094 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.094 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.094 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.094 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.094 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.094 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.094 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.094 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.094 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.094 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.095 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.095 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.095 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.095 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.095 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.095 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.095 INFO:tasks.rgw_multisite_tests: "marker": "1_1713572420.432178_731.1", 2024-04-20T01:55:26.095 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.095 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.095 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.096 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:20:20.432178Z", 2024-04-20T01:55:26.096 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.096 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.096 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.096 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.096 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.096 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.096 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.096 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.096 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.096 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.096 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.097 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.097 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.097 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.097 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.097 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.097 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.097 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.097 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.097 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.097 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.097 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:20:21Z" 2024-04-20T01:55:26.098 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.098 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572420.432178_731.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:20:20.432178Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.485008_738.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.432178_731.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1713572420.485008_738.1 target=1_1713572420.432178_731.1 2024-04-20T01:55:26.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:26.289+0000 7fe88f99f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:26.290+0000 7fe88f99f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713572420.485008_738.1", 2024-04-20T01:55:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T00:20:20.485008Z", 2024-04-20T01:55:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T00:20:26Z" 2024-04-20T01:55:26.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.104 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.104 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.105 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.105 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.105 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.105 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.105 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.105 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.105 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.105 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.105 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.105 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.105 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.106 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.106 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.106 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.106 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.106 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.106 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.106 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.106 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.106 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.107 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.107 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.107 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.107 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.107 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.107 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.107 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.107 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.107 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.107 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.107 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.108 INFO:tasks.rgw_multisite_tests: "marker": "1_1713572420.485008_738.1", 2024-04-20T01:55:26.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.108 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:20:20.485008Z", 2024-04-20T01:55:26.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.108 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.108 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.108 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.108 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.108 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.108 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.109 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.109 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.109 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.109 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.109 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.109 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.109 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.109 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.109 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.109 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.110 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.110 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.110 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.110 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:20:26Z" 2024-04-20T01:55:26.110 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.110 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572420.485008_738.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:20:20.485008Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.485008_738.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572420.485008_738.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-61 2024-04-20T01:55:26.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-61 2024-04-20T01:55:26.111 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-61 2024-04-20T01:55:26.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:26.565+0000 7f6d504a2780 1 bucket sync caught up with source: 2024-04-20T01:55:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [00000000001.48.6, , 00000000001.27.6, , 00000000001.159.6, , , 00000000001.39.6, , , ] 2024-04-20T01:55:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [00000000001.48.6, , 00000000001.27.6, , 00000000001.159.6, , , 00000000001.39.6, , , ] 2024-04-20T01:55:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:26.565+0000 7f6d504a2780 0 bucket checkpoint complete 2024-04-20T01:55:26.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-61 2024-04-20T01:55:26.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:26.685+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:31.686+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:36.686+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:41.687+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:46.687+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:51.687+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:20:56.688+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:21:01.689+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:21:06.689+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:21:11.689+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:21:16.690+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:21:21.691+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:21:26.692+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:21:31.693+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:21:36.693+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:21:41.694+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:21:46.694+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:21:51.695+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:21:56.695+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:22:01.696+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:22:06.696+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:22:11.696+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:22:16.697+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:22:21.697+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:22:26.698+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:22:31.698+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:22:36.699+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:22:41.700+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:22:46.701+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:22:51.701+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:22:56.702+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:23:01.702+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:23:06.703+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:23:11.704+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:23:16.704+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:23:21.705+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:23:26.706+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:23:31.706+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:23:36.706+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:23:41.707+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:23:46.707+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:23:51.707+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:23:56.707+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:24:01.708+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:24:06.709+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:24:11.709+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:24:16.710+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:24:21.710+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:24:26.711+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:24:31.711+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:24:36.712+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:24:41.712+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:24:46.713+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:24:51.714+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:24:56.714+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:01.715+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:06.716+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:11.717+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:16.717+0000 7f658ad94780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:21.718+0000 7f658ad94780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:21.718+0000 7f658ad94780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:21.718+0000 7f658ad94780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.118 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.118 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.118 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.118 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multipart_object_sync 2024-04-20T01:55:26.118 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.119 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.119 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.119 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.119 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1277, in test_multipart_object_sync 2024-04-20T01:55:26.119 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-20T01:55:26.119 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.119 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.119 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.119 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.119 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.120 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.120 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.120 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-64 2024-04-20T01:55:26.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/ 2024-04-20T01:55:26.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/ 2024-04-20T01:55:26.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-64/ 2024-04-20T01:55:26.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.121 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.121 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.121 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.121 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:25:21 GMT 2024-04-20T01:55:26.121 INFO:tasks.rgw_multisite_tests:/aktlji-64/ 2024-04-20T01:55:26.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.121 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:pSkwUN5rt1FWM6NayF4QxoVnL/4= 2024-04-20T01:55:26.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:25:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pSkwUN5rt1FWM6NayF4QxoVnL/4=', 'Content-Length': '0'} 2024-04-20T01:55:26.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000c7890efb504c22b6-0066230b71-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:25:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=aktlji-65 2024-04-20T01:55:26.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-65/ 2024-04-20T01:55:26.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-65/ 2024-04-20T01:55:26.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-65/ 2024-04-20T01:55:26.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.123 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.123 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.123 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.123 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:25:21 GMT 2024-04-20T01:55:26.123 INFO:tasks.rgw_multisite_tests:/aktlji-65/ 2024-04-20T01:55:26.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.123 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:J4QsIqWb6zxtHBxtns+nKDT4J6o= 2024-04-20T01:55:26.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:25:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:J4QsIqWb6zxtHBxtns+nKDT4J6o=', 'Content-Length': '0'} 2024-04-20T01:55:26.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000078c3ca9e19492fc9-0066230b71-1bfc-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:25:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone3 name=aktlji-66 2024-04-20T01:55:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-66/ 2024-04-20T01:55:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-66/ 2024-04-20T01:55:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-66/ 2024-04-20T01:55:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T01:55:26.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.125 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.125 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.125 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.125 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:25:21 GMT 2024-04-20T01:55:26.125 INFO:tasks.rgw_multisite_tests:/aktlji-66/ 2024-04-20T01:55:26.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.125 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:s+RUmNN45rI9DSMZub+k9zDrKnw= 2024-04-20T01:55:26.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:25:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:s+RUmNN45rI9DSMZub+k9zDrKnw=', 'Content-Length': '0'} 2024-04-20T01:55:26.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009f017945377e0d3b-0066230b71-15ba-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:25:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART?uploads 2024-04-20T01:55:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART?uploads 2024-04-20T01:55:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-04-20T01:55:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-64/MULTIPART?uploads 2024-04-20T01:55:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.127 INFO:tasks.rgw_multisite_tests:POST 2024-04-20T01:55:26.127 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.127 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.127 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:25:21 GMT 2024-04-20T01:55:26.127 INFO:tasks.rgw_multisite_tests:/aktlji-64/MULTIPART?uploads 2024-04-20T01:55:26.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.127 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1yrjRGj5hCmARrbew9bIftY+MJU= 2024-04-20T01:55:26.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:25:21 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1yrjRGj5hCmARrbew9bIftY+MJU=', 'Content-Length': '0'} 2024-04-20T01:55:26.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000056bbca11aab0fec4-0066230b71-15b6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'aktlji-64MULTIPART2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5' 2024-04-20T01:55:26.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=1 2024-04-20T01:55:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=1 2024-04-20T01:55:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=1 2024-04-20T01:55:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-20T01:55:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.129 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.129 INFO:tasks.rgw_multisite_tests:ebKBBg0ze5srhMzzkK3PdA== 2024-04-20T01:55:26.129 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.129 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:25:23 GMT 2024-04-20T01:55:26.129 INFO:tasks.rgw_multisite_tests:/aktlji-64/MULTIPART?partNumber=1&uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T01:55:26.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.129 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dTIbx+16/TUnJGVJG2HSCOjL7h0= 2024-04-20T01:55:26.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:25:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dTIbx+16/TUnJGVJG2HSCOjL7h0='} 2024-04-20T01:55:26.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000389fb27a967f0449-0066230b73-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=2 2024-04-20T01:55:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=2 2024-04-20T01:55:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=2 2024-04-20T01:55:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-20T01:55:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.130 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.131 INFO:tasks.rgw_multisite_tests:dIQ6OrGTo4m87YmUAtmdXw== 2024-04-20T01:55:26.131 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.131 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:25:23 GMT 2024-04-20T01:55:26.131 INFO:tasks.rgw_multisite_tests:/aktlji-64/MULTIPART?partNumber=2&uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T01:55:26.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.131 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:9W0y84ZR5gqH1rNoqgK+zZRCkdU= 2024-04-20T01:55:26.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:25:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:9W0y84ZR5gqH1rNoqgK+zZRCkdU='} 2024-04-20T01:55:26.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e7fc214f140a311-0066230b73-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=3 2024-04-20T01:55:26.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=3 2024-04-20T01:55:26.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=3 2024-04-20T01:55:26.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-20T01:55:26.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.132 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.132 INFO:tasks.rgw_multisite_tests:e4RW4edMN49Fhh9TYZ51tg== 2024-04-20T01:55:26.132 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.133 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:25:23 GMT 2024-04-20T01:55:26.133 INFO:tasks.rgw_multisite_tests:/aktlji-64/MULTIPART?partNumber=3&uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T01:55:26.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.133 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zZs0UaatU5T1LQuTlO1CcDYhzxk= 2024-04-20T01:55:26.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:25:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zZs0UaatU5T1LQuTlO1CcDYhzxk='} 2024-04-20T01:55:26.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f638f9ae544046a3-0066230b73-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=4 2024-04-20T01:55:26.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=4 2024-04-20T01:55:26.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5&partNumber=4 2024-04-20T01:55:26.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-20T01:55:26.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.134 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.134 INFO:tasks.rgw_multisite_tests:l+jDi4yPcIlVvEiQwDQZTQ== 2024-04-20T01:55:26.134 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.134 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:25:23 GMT 2024-04-20T01:55:26.134 INFO:tasks.rgw_multisite_tests:/aktlji-64/MULTIPART?partNumber=4&uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T01:55:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.135 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:c/Ga37AzTgI/Y1dL8JNc+GPgqnw= 2024-04-20T01:55:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:25:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:c/Ga37AzTgI/Y1dL8JNc+GPgqnw='} 2024-04-20T01:55:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb3a99e11bb2b384-0066230b73-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T01:55:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T01:55:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T01:55:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.136 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:26.136 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.136 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.136 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:25:23 GMT 2024-04-20T01:55:26.136 INFO:tasks.rgw_multisite_tests:/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T01:55:26.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.136 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:I144DsXgjw4tpR0TUgEA5/LR5UI= 2024-04-20T01:55:26.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:25:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:I144DsXgjw4tpR0TUgEA5/LR5UI=', 'Content-Length': '0'} 2024-04-20T01:55:26.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000094d361532fff48-0066230b73-15b6-test-zone1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART 2024-04-20T01:55:26.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T01:55:26.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T01:55:26.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-04-20T01:55:26.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T01:55:26.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.137 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.137 INFO:tasks.rgw_multisite_tests: 1 2024-04-20T01:55:26.138 INFO:tasks.rgw_multisite_tests: "79b281060d337b9b2b84ccf390adcf74" 2024-04-20T01:55:26.138 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.138 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.138 INFO:tasks.rgw_multisite_tests: 2 2024-04-20T01:55:26.138 INFO:tasks.rgw_multisite_tests: "74843a3ab193a389bced899402d99d5f" 2024-04-20T01:55:26.138 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.138 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.138 INFO:tasks.rgw_multisite_tests: 3 2024-04-20T01:55:26.138 INFO:tasks.rgw_multisite_tests: "7b8456e1e74c378f45861f53619e75b6" 2024-04-20T01:55:26.138 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.138 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.138 INFO:tasks.rgw_multisite_tests: 4 2024-04-20T01:55:26.139 INFO:tasks.rgw_multisite_tests: "97e8c38b8c8f708955bc4890c034194d" 2024-04-20T01:55:26.139 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.139 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'Content-Type': 'text/xml'} 2024-04-20T01:55:26.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.139 INFO:tasks.rgw_multisite_tests:POST 2024-04-20T01:55:26.139 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.140 INFO:tasks.rgw_multisite_tests:text/xml 2024-04-20T01:55:26.140 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:25:23 GMT 2024-04-20T01:55:26.140 INFO:tasks.rgw_multisite_tests:/aktlji-64/MULTIPART?uploadId=2~JMTz2ywG3wyVC8zBxApFjLfCp7SCFq5 2024-04-20T01:55:26.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.140 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:IsnPSZrFKEiMAzfnOViJXUeI0nk= 2024-04-20T01:55:26.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:25:23 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IsnPSZrFKEiMAzfnOViJXUeI0nk=', 'Content-Length': '460'} 2024-04-20T01:55:26.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a153a317eea4ddf5-0066230b73-15b6-test-zone1'), ('Content-Type', 'application/xml'), ('Content-Length', '325'), ('Date', 'Sat, 20 Apr 2024 00:25:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: http://smithi019.front.sepia.ceph.com:8000/aktlji-64/MULTIPARTaktlji-64MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-04-20T01:55:26.140 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.140 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713572721.843198_786.1", 2024-04-20T01:55:26.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:25:21.843198Z" 2024-04-20T01:55:26.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:26.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572721.843198_786.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:25:23.553+0000 7fbeca55c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:25:23.554+0000 7fbeca55c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713572721.843198_786.1", 2024-04-20T01:55:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:25:21.843198Z", 2024-04-20T01:55:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:25:23Z" 2024-04-20T01:55:26.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.149 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.149 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.149 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.149 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.149 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.149 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.149 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.149 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.150 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.150 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.150 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.150 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.150 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.150 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.150 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.150 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.150 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.150 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.151 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.151 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.151 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.151 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.151 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.151 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.151 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.151 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.151 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.151 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.152 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.152 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.152 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.152 INFO:tasks.rgw_multisite_tests: "marker": "1_1713572721.843198_786.1", 2024-04-20T01:55:26.152 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.152 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.152 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.152 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:25:21.843198Z", 2024-04-20T01:55:26.152 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.152 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.152 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.152 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.153 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.153 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.153 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.153 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.153 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.153 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.153 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.153 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.153 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.153 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.153 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.153 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.154 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.154 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.154 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.154 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.154 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.154 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:25:23Z" 2024-04-20T01:55:26.154 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.154 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572721.843198_786.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:25:21.843198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572721.843198_786.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572721.843198_786.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:23.683+0000 7f206f41b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:23.684+0000 7f206f41b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713572721.843198_786.1", 2024-04-20T01:55:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T00:25:21.843198Z", 2024-04-20T01:55:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T00:25:23Z" 2024-04-20T01:55:26.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.161 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.161 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.161 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.161 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.161 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.161 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.161 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.161 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.162 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.162 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.162 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.162 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.162 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.162 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.162 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.162 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.162 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.162 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.163 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.163 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.163 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.163 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.163 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.163 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.163 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.163 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.163 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.163 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.163 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: "marker": "1_1713572721.843198_786.1", 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:25:21.843198Z", 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.164 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.165 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.165 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.165 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.165 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.165 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.165 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.165 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.165 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.165 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.165 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.165 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.165 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.166 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.166 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.166 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.166 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.166 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.166 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.166 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:25:23Z" 2024-04-20T01:55:26.166 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.166 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713572721.843198_786.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:25:21.843198Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572721.843198_786.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713572721.843198_786.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-64 2024-04-20T01:55:26.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-64 2024-04-20T01:55:26.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-64 2024-04-20T01:55:26.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.167 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:24.973+0000 7fec6c4b7780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:29.974+0000 7fec6c4b7780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:34.974+0000 7fec6c4b7780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:39.974+0000 7fec6c4b7780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:44.976+0000 7fec6c4b7780 1 bucket sync caught up with source: 2024-04-20T01:55:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [, , , , , 00000000011.61.6, , , , , ] 2024-04-20T01:55:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [, , , , , 00000000011.61.6, , , , , ] 2024-04-20T01:55:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:44.976+0000 7fec6c4b7780 0 bucket checkpoint complete 2024-04-20T01:55:26.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-64 2024-04-20T01:55:26.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:45.101+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:50.101+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:25:55.102+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:26:00.103+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:26:05.103+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:26:10.104+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:26:15.104+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:26:20.105+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:26:25.106+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:26:30.106+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:26:35.107+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:26:40.108+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:26:45.109+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:26:50.109+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:26:55.110+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:27:00.111+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:27:05.111+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:27:10.112+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:27:15.113+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:27:20.113+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:27:25.114+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:27:30.113+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:27:35.114+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:27:40.114+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:27:45.115+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:27:50.116+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:27:55.116+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:28:00.117+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:28:05.118+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:28:10.118+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:28:15.119+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:28:20.119+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:28:25.120+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:28:30.121+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:28:35.121+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:28:40.122+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:28:45.123+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:28:50.123+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:28:55.124+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:29:00.124+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:29:05.125+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:29:10.126+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:29:15.126+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:29:20.125+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:29:25.126+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:29:30.127+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:29:35.127+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:29:40.128+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:29:45.128+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:29:50.129+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:29:55.129+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:00.130+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:05.131+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:10.132+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:15.132+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:20.133+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:25.133+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:30.134+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:35.135+0000 7fd83969d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:40.136+0000 7fd83969d780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:40.136+0000 7fd83969d780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:40.136+0000 7fd83969d780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.174 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.175 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.175 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.175 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_index_log_trim 2024-04-20T01:55:26.175 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.175 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.175 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.175 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.175 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1343, in test_bucket_index_log_trim 2024-04-20T01:55:26.175 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2024-04-20T01:55:26.176 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1339, in make_test_bucket 2024-04-20T01:55:26.176 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, name) 2024-04-20T01:55:26.176 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.176 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.176 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.176 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.176 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.176 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.176 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.176 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-68 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-68/ 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-68/ 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-68/ 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.177 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.178 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.178 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:26.178 INFO:tasks.rgw_multisite_tests:/aktlji-68/ 2024-04-20T01:55:26.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.178 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:iskKeeAKonkc+Cgmz0utnMli6dA= 2024-04-20T01:55:26.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iskKeeAKonkc+Cgmz0utnMli6dA=', 'Content-Length': '0'} 2024-04-20T01:55:26.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000061455e1a9d366f2d-0066230cb0-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-68/ 2024-04-20T01:55:26.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-68/ 2024-04-20T01:55:26.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-68/ 2024-04-20T01:55:26.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.179 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.179 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.179 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.179 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:26.179 INFO:tasks.rgw_multisite_tests:/aktlji-68/ 2024-04-20T01:55:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.180 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM= 2024-04-20T01:55:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM=', 'Content-Length': '0'} 2024-04-20T01:55:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e3be95405b92798-0066230cb0-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-68/a 2024-04-20T01:55:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-68/a 2024-04-20T01:55:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-68/a 2024-04-20T01:55:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.181 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.181 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.181 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.181 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:26.181 INFO:tasks.rgw_multisite_tests:/aktlji-68/a 2024-04-20T01:55:26.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.181 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:aUOd+DlhZn6zM5gahjG3E/pPbko= 2024-04-20T01:55:26.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aUOd+DlhZn6zM5gahjG3E/pPbko='} 2024-04-20T01:55:26.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000871e83ddac4eccaa-0066230cb0-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-68/ 2024-04-20T01:55:26.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-68/ 2024-04-20T01:55:26.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-68/ 2024-04-20T01:55:26.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.183 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.183 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.183 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.183 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:26.183 INFO:tasks.rgw_multisite_tests:/aktlji-68/ 2024-04-20T01:55:26.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.183 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM= 2024-04-20T01:55:26.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM=', 'Content-Length': '0'} 2024-04-20T01:55:26.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d4b3d96f8447258-0066230cb0-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-68/b 2024-04-20T01:55:26.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-68/b 2024-04-20T01:55:26.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-68/b 2024-04-20T01:55:26.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.184 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.184 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.185 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.185 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:26.185 INFO:tasks.rgw_multisite_tests:/aktlji-68/b 2024-04-20T01:55:26.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.185 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:98Nn0a8dfWOiZ6kq8KqNdvtrmc8= 2024-04-20T01:55:26.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:98Nn0a8dfWOiZ6kq8KqNdvtrmc8='} 2024-04-20T01:55:26.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057c9bef41d096f2c-0066230cb0-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-68/ 2024-04-20T01:55:26.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-68/ 2024-04-20T01:55:26.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-68/ 2024-04-20T01:55:26.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.186 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.186 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.186 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.186 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:26.186 INFO:tasks.rgw_multisite_tests:/aktlji-68/ 2024-04-20T01:55:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.187 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM= 2024-04-20T01:55:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM=', 'Content-Length': '0'} 2024-04-20T01:55:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4d129e40ce932b0-0066230cb0-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-68/c 2024-04-20T01:55:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-68/c 2024-04-20T01:55:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-68/c 2024-04-20T01:55:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.188 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.188 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.188 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.188 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:26.188 INFO:tasks.rgw_multisite_tests:/aktlji-68/c 2024-04-20T01:55:26.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.188 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:mDgtTHX2+XpbcUgMHS0xpH+ocD8= 2024-04-20T01:55:26.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mDgtTHX2+XpbcUgMHS0xpH+ocD8='} 2024-04-20T01:55:26.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e57c405a6adca730-0066230cb0-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-68/ 2024-04-20T01:55:26.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-68/ 2024-04-20T01:55:26.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-68/ 2024-04-20T01:55:26.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.189 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.190 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.190 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.190 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:26.190 INFO:tasks.rgw_multisite_tests:/aktlji-68/ 2024-04-20T01:55:26.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.190 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM= 2024-04-20T01:55:26.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6IWTgraCjwbhUxYzFc99DeWeBfM=', 'Content-Length': '0'} 2024-04-20T01:55:26.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000472872a5445ce247-0066230cb0-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-68/d 2024-04-20T01:55:26.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-68/d 2024-04-20T01:55:26.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-68/d 2024-04-20T01:55:26.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.191 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.191 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.191 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.191 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:30:40 GMT 2024-04-20T01:55:26.192 INFO:tasks.rgw_multisite_tests:/aktlji-68/d 2024-04-20T01:55:26.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.192 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:R43ssYu0Kjjdlqb0gJHub5g+esw= 2024-04-20T01:55:26.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:30:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R43ssYu0Kjjdlqb0gJHub5g+esw='} 2024-04-20T01:55:26.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d26458c82a96cd9-0066230cb0-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:30:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713573040.767133_830.1", 2024-04-20T01:55:26.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:30:40.767133Z" 2024-04-20T01:55:26.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:26.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.767133_830.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:30:41.134+0000 7ff6ac576780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:30:41.136+0000 7ff6ac576780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713573040.758847_827.1", 2024-04-20T01:55:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:30:40.758847Z", 2024-04-20T01:55:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:30:41Z" 2024-04-20T01:55:26.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.201 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.201 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.201 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.201 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.201 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.201 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.201 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.201 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.201 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.201 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.202 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.202 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.202 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.202 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.202 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.202 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.202 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.202 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.202 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.202 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.202 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.202 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.203 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.203 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.203 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.203 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.203 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.203 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.203 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.203 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.203 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.203 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.203 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.204 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.204 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.204 INFO:tasks.rgw_multisite_tests: "marker": "1_1713573040.758847_827.1", 2024-04-20T01:55:26.204 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.204 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.204 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:30:40.758847Z", 2024-04-20T01:55:26.204 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.204 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.204 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.204 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.204 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.205 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.205 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.205 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.205 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.205 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.205 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.205 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.205 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.205 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.205 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.205 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.206 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.206 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.206 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.206 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.206 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.206 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:30:41Z" 2024-04-20T01:55:26.206 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.206 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573040.758847_827.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:30:40.758847Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.767133_830.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.758847_827.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713573040.767133_830.1 target=1_1713573040.758847_827.1 2024-04-20T01:55:26.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:30:46.284+0000 7fd90b7a0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:30:46.285+0000 7fd90b7a0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713573040.767133_830.1", 2024-04-20T01:55:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:30:40.767133Z", 2024-04-20T01:55:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:30:46Z" 2024-04-20T01:55:26.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.213 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.213 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.213 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.213 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.213 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.213 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.213 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.214 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.214 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.214 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.214 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.214 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.214 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.214 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.214 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.214 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.215 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.215 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.215 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.215 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.215 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.215 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.215 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.215 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.215 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.216 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.216 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.216 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.216 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.216 INFO:tasks.rgw_multisite_tests: "marker": "1_1713573040.767133_830.1", 2024-04-20T01:55:26.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:30:40.767133Z", 2024-04-20T01:55:26.216 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.216 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.217 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.217 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.217 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.217 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.217 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.217 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.217 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.217 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.218 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.218 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.218 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.218 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.218 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.218 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.218 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.218 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.218 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:30:46Z" 2024-04-20T01:55:26.218 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.218 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573040.767133_830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:30:40.767133Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.767133_830.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.767133_830.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:46.410+0000 7f60f74b3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:46.412+0000 7f60f74b3780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713573040.767133_830.1", 2024-04-20T01:55:26.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T00:30:40.767133Z", 2024-04-20T01:55:26.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T00:30:46Z" 2024-04-20T01:55:26.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.226 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.226 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.226 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.226 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.226 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.226 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.226 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.226 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.226 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.226 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.226 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.227 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.227 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.227 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.227 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.227 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.227 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.227 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.227 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.227 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.227 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.227 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.228 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.228 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.228 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.228 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.228 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.228 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.228 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.228 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.228 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.228 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.229 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.229 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.229 INFO:tasks.rgw_multisite_tests: "marker": "1_1713573040.767133_830.1", 2024-04-20T01:55:26.229 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.229 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.229 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.229 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:30:40.767133Z", 2024-04-20T01:55:26.229 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.229 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.229 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.229 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.229 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.230 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.230 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.230 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.230 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.230 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.230 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.230 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.230 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.230 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.230 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.231 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.231 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.231 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.231 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.231 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.231 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.231 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:30:46Z" 2024-04-20T01:55:26.231 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.231 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573040.767133_830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:30:40.767133Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.767133_830.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573040.767133_830.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-68 2024-04-20T01:55:26.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-68 2024-04-20T01:55:26.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-68 2024-04-20T01:55:26.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.232 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:46.683+0000 7f964666e780 1 bucket sync caught up with source: 2024-04-20T01:55:26.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [, 00000000001.37.6, , , , 00000000001.25.6, , , 00000000001.59.6, , 00000000001.77.6] 2024-04-20T01:55:26.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [, 00000000001.37.6, , , , 00000000001.25.6, , , 00000000001.59.6, , 00000000001.77.6] 2024-04-20T01:55:26.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:46.683+0000 7f964666e780 0 bucket checkpoint complete 2024-04-20T01:55:26.233 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-68 2024-04-20T01:55:26.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-68', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-68', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-68 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:46.800+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:51.800+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:30:56.801+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:31:01.801+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:31:06.801+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:31:11.801+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:31:16.802+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:31:21.802+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:31:26.803+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:31:31.804+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:31:36.805+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:31:41.805+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:31:46.806+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:31:51.806+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:31:56.807+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:32:01.807+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:32:06.808+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:32:11.808+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:32:16.809+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:32:21.810+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:32:26.811+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:32:31.812+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:32:36.813+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:32:41.813+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:32:46.813+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:32:51.813+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:32:56.814+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:33:01.814+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:33:06.815+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:33:11.815+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:33:16.816+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:33:21.817+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:33:26.817+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:33:31.818+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:33:36.819+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:33:41.819+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:33:46.820+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:33:51.820+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:33:56.821+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:34:01.822+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:34:06.823+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:34:11.824+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:34:16.824+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:34:21.825+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:34:26.825+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:34:31.826+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:34:36.826+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:34:41.826+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:34:46.826+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:34:51.827+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:34:56.828+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:01.828+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:06.829+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:11.829+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:16.830+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:21.831+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:26.831+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:31.832+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:36.833+0000 7f8eda369780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:41.833+0000 7f8eda369780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:41.833+0000 7f8eda369780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:41.833+0000 7f8eda369780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.239 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.239 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.239 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.239 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2024-04-20T01:55:26.239 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.240 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.240 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.240 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.240 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1397, in test_bucket_reshard_index_log_trim 2024-04-20T01:55:26.240 INFO:tasks.rgw_multisite_tests: test_bucket = make_test_bucket() 2024-04-20T01:55:26.240 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1393, in make_test_bucket 2024-04-20T01:55:26.240 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, name) 2024-04-20T01:55:26.240 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.240 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.240 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.241 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.241 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.241 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.241 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.241 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-69 2024-04-20T01:55:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-69/ 2024-04-20T01:55:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-69/ 2024-04-20T01:55:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-69/ 2024-04-20T01:55:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.242 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.242 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.242 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.242 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:35:41 GMT 2024-04-20T01:55:26.242 INFO:tasks.rgw_multisite_tests:/aktlji-69/ 2024-04-20T01:55:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.243 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:RsZ9hlvpQ2/5qdu6H0K7ePZ5sb0= 2024-04-20T01:55:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RsZ9hlvpQ2/5qdu6H0K7ePZ5sb0=', 'Content-Length': '0'} 2024-04-20T01:55:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000819e2b0ae7c12107-0066230ddd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-69/ 2024-04-20T01:55:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-69/ 2024-04-20T01:55:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-69/ 2024-04-20T01:55:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:35:41 GMT 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests:/aktlji-69/ 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA= 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA=', 'Content-Length': '0'} 2024-04-20T01:55:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000831ed5edae64b130-0066230ddd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-69/a 2024-04-20T01:55:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-69/a 2024-04-20T01:55:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-69/a 2024-04-20T01:55:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.245 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.246 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.246 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.246 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:35:41 GMT 2024-04-20T01:55:26.246 INFO:tasks.rgw_multisite_tests:/aktlji-69/a 2024-04-20T01:55:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.246 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:JfHFy/OhA/rSbOjMPxM3p1D/A1I= 2024-04-20T01:55:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JfHFy/OhA/rSbOjMPxM3p1D/A1I='} 2024-04-20T01:55:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066b9056edf44cf0c-0066230ddd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-69/ 2024-04-20T01:55:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-69/ 2024-04-20T01:55:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-69/ 2024-04-20T01:55:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.247 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.247 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.247 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.247 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:35:41 GMT 2024-04-20T01:55:26.248 INFO:tasks.rgw_multisite_tests:/aktlji-69/ 2024-04-20T01:55:26.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.248 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA= 2024-04-20T01:55:26.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA=', 'Content-Length': '0'} 2024-04-20T01:55:26.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f59c59a60128626-0066230ddd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-69/b 2024-04-20T01:55:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-69/b 2024-04-20T01:55:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-69/b 2024-04-20T01:55:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.251 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.252 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.252 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.252 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:35:41 GMT 2024-04-20T01:55:26.252 INFO:tasks.rgw_multisite_tests:/aktlji-69/b 2024-04-20T01:55:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.252 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:E+AVTVhq93OAQMTTXYbu02nub7s= 2024-04-20T01:55:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:E+AVTVhq93OAQMTTXYbu02nub7s='} 2024-04-20T01:55:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1e525b38e8b7a3b-0066230ddd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-69/ 2024-04-20T01:55:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-69/ 2024-04-20T01:55:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-69/ 2024-04-20T01:55:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.253 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.253 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.253 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.253 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:35:41 GMT 2024-04-20T01:55:26.254 INFO:tasks.rgw_multisite_tests:/aktlji-69/ 2024-04-20T01:55:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.254 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA= 2024-04-20T01:55:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA=', 'Content-Length': '0'} 2024-04-20T01:55:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5edb605fadf1ba8-0066230ddd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-69/c 2024-04-20T01:55:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-69/c 2024-04-20T01:55:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-69/c 2024-04-20T01:55:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.255 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.255 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.255 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.255 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:35:41 GMT 2024-04-20T01:55:26.255 INFO:tasks.rgw_multisite_tests:/aktlji-69/c 2024-04-20T01:55:26.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.256 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:slCGFJ6lAshGu+0IH8RW6I5iSQ4= 2024-04-20T01:55:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:slCGFJ6lAshGu+0IH8RW6I5iSQ4='} 2024-04-20T01:55:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a249b1bc9f58d27-0066230ddd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-69/ 2024-04-20T01:55:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-69/ 2024-04-20T01:55:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-69/ 2024-04-20T01:55:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.257 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.257 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.257 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.257 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:35:41 GMT 2024-04-20T01:55:26.257 INFO:tasks.rgw_multisite_tests:/aktlji-69/ 2024-04-20T01:55:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.257 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA= 2024-04-20T01:55:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QSUcdbeFDgaIzVcQhyEbSSv/8cA=', 'Content-Length': '0'} 2024-04-20T01:55:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000919f02caf6671097-0066230ddd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-69/d 2024-04-20T01:55:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-69/d 2024-04-20T01:55:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-69/d 2024-04-20T01:55:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.258 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.259 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.259 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.259 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:35:41 GMT 2024-04-20T01:55:26.259 INFO:tasks.rgw_multisite_tests:/aktlji-69/d 2024-04-20T01:55:26.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.259 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:kCIiwyTcW6hugRqgjWlksA1CO3w= 2024-04-20T01:55:26.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:35:41 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kCIiwyTcW6hugRqgjWlksA1CO3w='} 2024-04-20T01:55:26.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000255cfcdb6245d1ec-0066230ddd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:35:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713573341.878158_866.1", 2024-04-20T01:55:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:35:41.878158Z" 2024-04-20T01:55:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573341.878158_866.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:35:42.198+0000 7fed1cee1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:35:42.199+0000 7fed1cee1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713573341.878158_866.1", 2024-04-20T01:55:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:35:41.878158Z", 2024-04-20T01:55:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:35:42Z" 2024-04-20T01:55:26.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.268 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.268 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.268 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.268 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.268 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.268 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.268 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.269 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.269 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.269 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.269 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.269 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.269 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.269 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.269 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.269 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.269 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.269 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.270 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.270 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.270 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.270 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.270 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.270 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.270 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.270 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.270 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.271 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.271 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.271 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.271 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.271 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.271 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.271 INFO:tasks.rgw_multisite_tests: "marker": "1_1713573341.878158_866.1", 2024-04-20T01:55:26.271 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.271 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:35:41.878158Z", 2024-04-20T01:55:26.271 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.272 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.272 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.272 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.272 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.272 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.272 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.272 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.272 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.272 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.272 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.272 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.273 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.273 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.273 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.273 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.273 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.273 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.273 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.273 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.273 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:35:42Z" 2024-04-20T01:55:26.273 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.273 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573341.878158_866.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:35:41.878158Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573341.878158_866.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573341.878158_866.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:42.322+0000 7f06a6fdb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:42.323+0000 7f06a6fdb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713573341.878158_866.1", 2024-04-20T01:55:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T00:35:41.878158Z", 2024-04-20T01:55:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T00:35:42Z" 2024-04-20T01:55:26.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.280 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.280 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.280 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.280 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.281 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.281 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.281 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.281 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.281 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.281 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.281 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.281 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.281 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.282 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.282 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.282 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.282 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.282 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.282 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.282 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.282 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.282 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.282 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.283 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.283 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.283 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.283 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.283 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.283 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.283 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.283 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.283 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.283 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.283 INFO:tasks.rgw_multisite_tests: "marker": "1_1713573341.878158_866.1", 2024-04-20T01:55:26.283 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.284 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.284 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.284 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:35:41.878158Z", 2024-04-20T01:55:26.284 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.284 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.284 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.284 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.284 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.284 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.284 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.284 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.284 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.285 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.285 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.285 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.285 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.285 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.285 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.285 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.285 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.285 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.285 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.285 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.285 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.286 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:35:42Z" 2024-04-20T01:55:26.286 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.286 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573341.878158_866.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:35:41.878158Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573341.878158_866.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573341.878158_866.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-69 2024-04-20T01:55:26.286 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.286 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-69 2024-04-20T01:55:26.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-69 2024-04-20T01:55:26.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:42.636+0000 7f7389d5d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:47.638+0000 7f7389d5d780 1 bucket sync caught up with source: 2024-04-20T01:55:26.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [, 00000000001.53.6, , , , 00000000001.51.6, , , 00000000001.41.6, , 00000000001.31.6] 2024-04-20T01:55:26.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [, 00000000001.53.6, , , , 00000000001.51.6, , , 00000000001.41.6, , 00000000001.31.6] 2024-04-20T01:55:26.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:47.638+0000 7f7389d5d780 0 bucket checkpoint complete 2024-04-20T01:55:26.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-69 2024-04-20T01:55:26.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:47.763+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:52.764+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:35:57.765+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:36:02.766+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:36:07.767+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:36:12.767+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:36:17.768+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:36:22.768+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:36:27.768+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:36:32.768+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:36:37.769+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:36:42.769+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:36:47.770+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:36:52.770+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:36:57.771+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:37:02.772+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:37:07.772+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:37:12.773+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:37:17.773+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:37:22.774+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:37:27.774+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:37:32.775+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:37:37.776+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:37:42.777+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:37:47.778+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:37:52.778+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:37:57.779+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:38:02.779+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:38:07.780+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:38:12.781+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:38:17.780+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:38:22.781+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:38:27.782+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:38:32.782+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:38:37.782+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:38:42.783+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:38:47.784+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:38:52.785+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:38:57.786+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:39:02.787+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:39:07.787+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:39:12.788+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:39:17.788+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:39:22.789+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:39:27.790+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:39:32.791+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:39:37.792+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:39:42.792+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:39:47.793+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:39:52.793+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:39:57.794+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:02.794+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:07.794+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:12.795+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:17.795+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:22.796+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:27.796+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:32.797+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:37.798+0000 7f5e6af92780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:42.798+0000 7f5e6af92780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:42.798+0000 7f5e6af92780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:42.798+0000 7f5e6af92780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.294 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.294 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.294 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.294 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_incremental 2024-04-20T01:55:26.294 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.295 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.295 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.295 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.295 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1480, in test_bucket_reshard_incremental 2024-04-20T01:55:26.295 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-20T01:55:26.295 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.295 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.295 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.295 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.295 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.295 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.296 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.296 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-70/ 2024-04-20T01:55:26.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-70/ 2024-04-20T01:55:26.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-70/ 2024-04-20T01:55:26.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.297 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.297 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.297 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.297 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:40:42 GMT 2024-04-20T01:55:26.297 INFO:tasks.rgw_multisite_tests:/aktlji-70/ 2024-04-20T01:55:26.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.297 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:q7m1pHNOKOcDvN8GwuzIzYFobO4= 2024-04-20T01:55:26.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:40:42 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:q7m1pHNOKOcDvN8GwuzIzYFobO4=', 'Content-Length': '0'} 2024-04-20T01:55:26.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000022e7b86caa5b7bf6-0066230f0a-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:40:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=aktlji-70 2024-04-20T01:55:26.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713573642.846158_911.1", 2024-04-20T01:55:26.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:40:42.846158Z" 2024-04-20T01:55:26.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:26.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573642.846158_911.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:40:43.081+0000 7f4f75025780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:40:43.082+0000 7f4f75025780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713573642.846158_911.1", 2024-04-20T01:55:26.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:40:42.846158Z", 2024-04-20T01:55:26.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:40:43Z" 2024-04-20T01:55:26.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.306 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.306 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.306 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.306 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.306 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.306 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.307 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.307 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.307 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.307 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.307 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.307 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.307 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.307 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.307 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.307 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.307 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.308 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.308 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.308 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.308 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.308 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.308 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.308 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.308 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.308 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.308 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.308 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.309 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.309 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.309 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.309 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.309 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.309 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.309 INFO:tasks.rgw_multisite_tests: "marker": "1_1713573642.846158_911.1", 2024-04-20T01:55:26.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.309 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.309 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:40:42.846158Z", 2024-04-20T01:55:26.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.310 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.310 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.310 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.310 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.310 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.310 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.310 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.310 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.310 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.310 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.310 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.311 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.311 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.311 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.311 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.311 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.311 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.311 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.311 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.311 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:40:43Z" 2024-04-20T01:55:26.311 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.311 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573642.846158_911.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:40:42.846158Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573642.846158_911.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573642.846158_911.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.312 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:43.204+0000 7f713f8ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:43.206+0000 7f713f8ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713573642.846158_911.1", 2024-04-20T01:55:26.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T00:40:42.846158Z", 2024-04-20T01:55:26.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T00:40:43Z" 2024-04-20T01:55:26.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.318 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.318 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.318 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.318 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.318 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.318 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.319 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.319 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.319 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.319 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.319 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.319 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.319 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.319 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.319 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.320 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.320 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.320 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.320 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.320 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.320 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.320 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.320 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.320 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.320 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.321 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.321 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.321 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.321 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.321 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.321 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.321 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.321 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.321 INFO:tasks.rgw_multisite_tests: "marker": "1_1713573642.846158_911.1", 2024-04-20T01:55:26.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.321 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.322 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:40:42.846158Z", 2024-04-20T01:55:26.322 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.322 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.322 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.322 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.322 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.322 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.322 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.322 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.322 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.322 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.322 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.323 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.323 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.323 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.323 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.323 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.323 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.323 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.323 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.323 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.323 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:40:43Z" 2024-04-20T01:55:26.323 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.324 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573642.846158_911.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:40:42.846158Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573642.846158_911.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573642.846158_911.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-70/ 2024-04-20T01:55:26.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-70/ 2024-04-20T01:55:26.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-70/ 2024-04-20T01:55:26.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.325 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.325 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.325 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.325 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:40:43 GMT 2024-04-20T01:55:26.325 INFO:tasks.rgw_multisite_tests:/aktlji-70/ 2024-04-20T01:55:26.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.325 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU= 2024-04-20T01:55:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU=', 'Content-Length': '0'} 2024-04-20T01:55:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096de8cc27e5e9e04-0066230f0b-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-70/a 2024-04-20T01:55:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-70/a 2024-04-20T01:55:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-70/a 2024-04-20T01:55:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.327 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.327 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.327 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.327 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:40:43 GMT 2024-04-20T01:55:26.327 INFO:tasks.rgw_multisite_tests:/aktlji-70/a 2024-04-20T01:55:26.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.327 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Y3RPz0A+Gk4dVFzdB2zg99+mACA= 2024-04-20T01:55:26.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Y3RPz0A+Gk4dVFzdB2zg99+mACA='} 2024-04-20T01:55:26.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fccb80b3b5b32578-0066230f0b-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-70/ 2024-04-20T01:55:26.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-70/ 2024-04-20T01:55:26.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-70/ 2024-04-20T01:55:26.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.328 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.328 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.329 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.329 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:40:43 GMT 2024-04-20T01:55:26.329 INFO:tasks.rgw_multisite_tests:/aktlji-70/ 2024-04-20T01:55:26.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.329 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU= 2024-04-20T01:55:26.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU=', 'Content-Length': '0'} 2024-04-20T01:55:26.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b62e71e86dccea6-0066230f0b-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-70/b 2024-04-20T01:55:26.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-70/b 2024-04-20T01:55:26.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-70/b 2024-04-20T01:55:26.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.330 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.330 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.330 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.330 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:40:43 GMT 2024-04-20T01:55:26.330 INFO:tasks.rgw_multisite_tests:/aktlji-70/b 2024-04-20T01:55:26.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.331 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:gkNcH1aGAP0gK0gK1hWBassHdfI= 2024-04-20T01:55:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gkNcH1aGAP0gK0gK1hWBassHdfI='} 2024-04-20T01:55:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014e3297547e1738a-0066230f0b-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-70/ 2024-04-20T01:55:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-70/ 2024-04-20T01:55:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-70/ 2024-04-20T01:55:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.332 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.332 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.332 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.332 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:40:43 GMT 2024-04-20T01:55:26.332 INFO:tasks.rgw_multisite_tests:/aktlji-70/ 2024-04-20T01:55:26.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.332 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU= 2024-04-20T01:55:26.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU=', 'Content-Length': '0'} 2024-04-20T01:55:26.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039c4eca17803c9bc-0066230f0b-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-70/c 2024-04-20T01:55:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-70/c 2024-04-20T01:55:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-70/c 2024-04-20T01:55:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.333 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.334 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.334 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.334 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:40:43 GMT 2024-04-20T01:55:26.334 INFO:tasks.rgw_multisite_tests:/aktlji-70/c 2024-04-20T01:55:26.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.334 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cEKCPFA8Cnw60001l3KY1AzfWb0= 2024-04-20T01:55:26.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cEKCPFA8Cnw60001l3KY1AzfWb0='} 2024-04-20T01:55:26.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e6aba41482bb90f-0066230f0b-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-70/ 2024-04-20T01:55:26.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-70/ 2024-04-20T01:55:26.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-70/ 2024-04-20T01:55:26.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.335 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.335 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.335 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.335 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:40:43 GMT 2024-04-20T01:55:26.335 INFO:tasks.rgw_multisite_tests:/aktlji-70/ 2024-04-20T01:55:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.336 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU= 2024-04-20T01:55:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KjKJsuowabJeo33ilGZbIUPymYU=', 'Content-Length': '0'} 2024-04-20T01:55:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d562664088c7fb82-0066230f0b-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-70/d 2024-04-20T01:55:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-70/d 2024-04-20T01:55:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-70/d 2024-04-20T01:55:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.337 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.337 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.337 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.337 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:40:43 GMT 2024-04-20T01:55:26.337 INFO:tasks.rgw_multisite_tests:/aktlji-70/d 2024-04-20T01:55:26.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.337 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eIkW0cm0lW6PGdheRPdFQhadDEw= 2024-04-20T01:55:26.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:40:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eIkW0cm0lW6PGdheRPdFQhadDEw='} 2024-04-20T01:55:26.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b80b6d2d0c43672-0066230f0b-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:40:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-70 2024-04-20T01:55:26.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-70 2024-04-20T01:55:26.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-70 2024-04-20T01:55:26.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:43.605+0000 7f912d553780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:48.608+0000 7f912d553780 1 bucket sync caught up with source: 2024-04-20T01:55:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [, 00000000001.36.6, , , , 00000000001.54.6, , , 00000000001.47.6, , 00000000001.62.6] 2024-04-20T01:55:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [, 00000000001.36.6, , , , 00000000001.54.6, , , 00000000001.47.6, , 00000000001.62.6] 2024-04-20T01:55:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:48.608+0000 7f912d553780 0 bucket checkpoint complete 2024-04-20T01:55:26.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-70 2024-04-20T01:55:26.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.339 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:48.733+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:53.734+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:40:58.734+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:41:03.735+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:41:08.736+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:41:13.736+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:41:18.737+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:41:23.738+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:41:28.739+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:41:33.740+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:41:38.740+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:41:43.741+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:41:48.741+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:41:53.743+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:41:58.743+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:42:03.744+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:42:08.744+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:42:13.745+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:42:18.745+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:42:23.746+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:42:28.746+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:42:33.747+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:42:38.747+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:42:43.748+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:42:48.749+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:42:53.752+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:42:58.753+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:43:03.753+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:43:08.755+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:43:13.756+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:43:18.756+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:43:23.757+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:43:28.758+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:43:33.758+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:43:38.759+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:43:43.759+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:43:48.759+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:43:53.760+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:43:58.760+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:44:03.761+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:44:08.762+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:44:13.762+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:44:18.762+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:44:23.763+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:44:28.764+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:44:33.764+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:44:38.765+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:44:43.766+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:44:48.767+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:44:53.767+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:44:58.768+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:03.769+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:08.769+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:13.770+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:18.770+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:23.771+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:28.771+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:33.771+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:38.772+0000 7f166948a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:43.772+0000 7f166948a780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:43.772+0000 7f166948a780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:43.772+0000 7f166948a780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.345 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.345 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.345 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.345 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_full 2024-04-20T01:55:26.346 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.346 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.346 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.346 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.346 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1531, in test_bucket_reshard_full 2024-04-20T01:55:26.346 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-20T01:55:26.346 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.346 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.347 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.347 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.347 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.347 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.347 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.347 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/ 2024-04-20T01:55:26.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/ 2024-04-20T01:55:26.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/ 2024-04-20T01:55:26.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.348 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.348 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.348 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.348 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:45:43 GMT 2024-04-20T01:55:26.348 INFO:tasks.rgw_multisite_tests:/aktlji-71/ 2024-04-20T01:55:26.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.349 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WId1r/HRr3QRORISmp0vCa5QP7I= 2024-04-20T01:55:26.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:45:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WId1r/HRr3QRORISmp0vCa5QP7I=', 'Content-Length': '0'} 2024-04-20T01:55:26.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b79229535b68761f-0066231037-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:45:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=aktlji-71 2024-04-20T01:55:26.349 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.349 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713573943.819700_950.1", 2024-04-20T01:55:26.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:45:43.819700Z" 2024-04-20T01:55:26.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:26.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.819700_950.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:45:44.050+0000 7fd8b71ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:45:44.052+0000 7fd8b71ae780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713573943.811171_947.1", 2024-04-20T01:55:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:45:43.811171Z", 2024-04-20T01:55:26.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:45:44Z" 2024-04-20T01:55:26.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.358 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.358 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.358 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.358 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.358 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.358 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.358 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.358 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.358 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.359 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.359 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.359 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.359 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.359 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.359 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.359 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.359 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.359 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.359 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.360 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.360 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.360 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.360 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.360 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.360 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.360 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.360 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.360 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.360 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.360 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.360 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.361 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.361 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.361 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.361 INFO:tasks.rgw_multisite_tests: "marker": "1_1713573943.811171_947.1", 2024-04-20T01:55:26.361 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.361 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.361 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:45:43.811171Z", 2024-04-20T01:55:26.361 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.361 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.361 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.361 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.362 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.362 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.362 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.362 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.362 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.362 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.362 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.362 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.362 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.362 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.362 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.363 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.363 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.363 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.363 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.363 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.363 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.363 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:45:44Z" 2024-04-20T01:55:26.363 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.363 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573943.811171_947.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:45:43.811171Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.819700_950.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.811171_947.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713573943.819700_950.1 target=1_1713573943.811171_947.1 2024-04-20T01:55:26.364 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.364 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:45:49.186+0000 7fc99df5f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:45:49.188+0000 7fc99df5f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713573943.819700_950.1", 2024-04-20T01:55:26.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:45:43.819700Z", 2024-04-20T01:55:26.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:45:49Z" 2024-04-20T01:55:26.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.370 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.370 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.370 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.371 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.371 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.371 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.371 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.371 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.371 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.371 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.371 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.371 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.371 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.371 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.371 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.372 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.372 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.372 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.372 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.372 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.372 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.372 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.372 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.372 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.372 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.372 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.372 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.373 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.373 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.373 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.373 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.373 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.373 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.373 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.373 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.373 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.373 INFO:tasks.rgw_multisite_tests: "marker": "1_1713573943.819700_950.1", 2024-04-20T01:55:26.373 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.373 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.374 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.374 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:45:43.819700Z", 2024-04-20T01:55:26.374 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.374 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.374 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.374 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.374 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.374 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.374 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.374 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.374 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.374 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.375 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.375 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.375 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.375 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.375 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.375 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.375 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.375 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.375 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.375 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.375 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.375 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:45:49Z" 2024-04-20T01:55:26.376 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.376 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573943.819700_950.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:45:43.819700Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.819700_950.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.819700_950.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:49.310+0000 7fddcba88780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:45:49.311+0000 7fddcba88780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713573943.819700_950.1", 2024-04-20T01:55:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T00:45:43.819700Z", 2024-04-20T01:55:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T00:45:49Z" 2024-04-20T01:55:26.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.382 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.382 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.382 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.382 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.383 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.383 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.383 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.383 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.383 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.383 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.383 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.383 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.383 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.383 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.383 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.383 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.384 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.384 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.384 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.384 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.384 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.384 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.384 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.384 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.384 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.384 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.384 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.385 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.385 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.385 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.385 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.385 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.385 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.385 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.385 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.385 INFO:tasks.rgw_multisite_tests: "marker": "1_1713573943.819700_950.1", 2024-04-20T01:55:26.385 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.385 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.385 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.386 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:45:43.819700Z", 2024-04-20T01:55:26.386 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.386 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.386 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.386 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.386 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.386 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.386 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.386 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.386 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.386 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.387 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.387 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.387 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.387 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.387 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.387 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.387 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.387 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.387 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.387 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.387 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.387 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:45:49Z" 2024-04-20T01:55:26.388 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.388 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713573943.819700_950.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:45:43.819700Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.819700_950.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713573943.819700_950.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.388 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: DEBUG: waiting for process to exit 2024-04-20T01:55:26.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-20T01:55:26.388 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi031.stdout: INFO: 2024-04-20T00:45:49.337+0000 7fe8dd09a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 48535) UID: 0 2024-04-20T01:55:26.388 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi031.stdout: INFO: 2024-04-20T00:45:49.337+0000 7fe8e226f7c0 -1 shutting down 2024-04-20T01:55:26.388 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2024-04-20T01:55:26.388 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Stopped 2024-04-20T01:55:26.389 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2024-04-20T01:55:26.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-20T01:55:26.389 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi019.stdout: INFO: 2024-04-20T00:45:55.440+0000 7f37d525f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 49475) UID: 0 2024-04-20T01:55:26.389 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi019.stdout: INFO: 2024-04-20T00:45:55.440+0000 7f37da4347c0 -1 shutting down 2024-04-20T01:55:26.389 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 has been restored 2024-04-20T01:55:26.389 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2024-04-20T01:55:26.389 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2024-04-20T01:55:26.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/ 2024-04-20T01:55:26.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/ 2024-04-20T01:55:26.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/ 2024-04-20T01:55:26.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.390 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.390 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.390 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.390 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.390 INFO:tasks.rgw_multisite_tests:/aktlji-71/ 2024-04-20T01:55:26.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.391 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T01:55:26.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T01:55:26.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009eb5c3b0e3b09280-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/a 2024-04-20T01:55:26.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/a 2024-04-20T01:55:26.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/a 2024-04-20T01:55:26.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.392 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.392 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.392 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.392 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.392 INFO:tasks.rgw_multisite_tests:/aktlji-71/a 2024-04-20T01:55:26.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.392 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:H6YcSYwXBr0vfVTRUJPVO6R4tJ8= 2024-04-20T01:55:26.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:H6YcSYwXBr0vfVTRUJPVO6R4tJ8='} 2024-04-20T01:55:26.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ec3b5e92aaa8a35-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/ 2024-04-20T01:55:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/ 2024-04-20T01:55:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/ 2024-04-20T01:55:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.393 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.394 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.394 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.394 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.394 INFO:tasks.rgw_multisite_tests:/aktlji-71/ 2024-04-20T01:55:26.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.394 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T01:55:26.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T01:55:26.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af281c41a8107049-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/b 2024-04-20T01:55:26.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/b 2024-04-20T01:55:26.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/b 2024-04-20T01:55:26.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.395 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.395 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.395 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.395 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.396 INFO:tasks.rgw_multisite_tests:/aktlji-71/b 2024-04-20T01:55:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.396 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:kvk+rS/pHLXR0LjLY77Za3O/Ezc= 2024-04-20T01:55:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kvk+rS/pHLXR0LjLY77Za3O/Ezc='} 2024-04-20T01:55:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098474d4a012b9d21-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/ 2024-04-20T01:55:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/ 2024-04-20T01:55:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/ 2024-04-20T01:55:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.397 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.397 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.397 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.397 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.397 INFO:tasks.rgw_multisite_tests:/aktlji-71/ 2024-04-20T01:55:26.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.397 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T01:55:26.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T01:55:26.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001691372bd535cf84-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/c 2024-04-20T01:55:26.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/c 2024-04-20T01:55:26.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/c 2024-04-20T01:55:26.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.399 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.399 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.399 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.399 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.399 INFO:tasks.rgw_multisite_tests:/aktlji-71/c 2024-04-20T01:55:26.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.399 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:L9Dly0WeXsCgOV6bKUPfSidhDKs= 2024-04-20T01:55:26.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:L9Dly0WeXsCgOV6bKUPfSidhDKs='} 2024-04-20T01:55:26.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b3c74b42f332bcc-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/ 2024-04-20T01:55:26.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/ 2024-04-20T01:55:26.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/ 2024-04-20T01:55:26.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.400 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.400 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.401 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.401 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.401 INFO:tasks.rgw_multisite_tests:/aktlji-71/ 2024-04-20T01:55:26.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.401 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T01:55:26.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T01:55:26.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f8c7f8ea3e4855a-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/d 2024-04-20T01:55:26.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/d 2024-04-20T01:55:26.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/d 2024-04-20T01:55:26.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.402 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.402 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.402 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.402 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.402 INFO:tasks.rgw_multisite_tests:/aktlji-71/d 2024-04-20T01:55:26.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.403 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1AozUUsFcJDkcwZ2WMZQs5bHRns= 2024-04-20T01:55:26.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1AozUUsFcJDkcwZ2WMZQs5bHRns='} 2024-04-20T01:55:26.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ed055b1a4134d1f-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'aktlji-71', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'aktlji-71', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket aktlji-71 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-71 2024-04-20T01:55:26.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 4 2024-04-20T01:55:26.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/ 2024-04-20T01:55:26.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/ 2024-04-20T01:55:26.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/ 2024-04-20T01:55:26.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.404 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.405 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.405 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.405 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.405 INFO:tasks.rgw_multisite_tests:/aktlji-71/ 2024-04-20T01:55:26.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.405 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T01:55:26.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T01:55:26.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b8368ff32b9836a-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/e 2024-04-20T01:55:26.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/e 2024-04-20T01:55:26.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/e 2024-04-20T01:55:26.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.406 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.406 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.406 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.407 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.407 INFO:tasks.rgw_multisite_tests:/aktlji-71/e 2024-04-20T01:55:26.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.407 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/wu48I6TFzwV5fJTrx9A4IsS7Vw= 2024-04-20T01:55:26.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/wu48I6TFzwV5fJTrx9A4IsS7Vw='} 2024-04-20T01:55:26.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ec9205171adbf58-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/ 2024-04-20T01:55:26.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/ 2024-04-20T01:55:26.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/ 2024-04-20T01:55:26.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.408 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.408 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.408 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.408 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.408 INFO:tasks.rgw_multisite_tests:/aktlji-71/ 2024-04-20T01:55:26.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.408 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8be2ac9f6e12b33-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/f 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/f 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/f 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.410 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.410 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.410 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.410 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.410 INFO:tasks.rgw_multisite_tests:/aktlji-71/f 2024-04-20T01:55:26.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.410 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:S9F20SUOIOASKEXZe/gztz7i4g0= 2024-04-20T01:55:26.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:S9F20SUOIOASKEXZe/gztz7i4g0='} 2024-04-20T01:55:26.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a85ca78514843554-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/ 2024-04-20T01:55:26.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/ 2024-04-20T01:55:26.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/ 2024-04-20T01:55:26.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.411 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.411 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.412 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.412 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.412 INFO:tasks.rgw_multisite_tests:/aktlji-71/ 2024-04-20T01:55:26.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.412 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T01:55:26.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T01:55:26.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098bd757019deaa96-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/g 2024-04-20T01:55:26.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/g 2024-04-20T01:55:26.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/g 2024-04-20T01:55:26.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.413 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.413 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.413 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.413 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.413 INFO:tasks.rgw_multisite_tests:/aktlji-71/g 2024-04-20T01:55:26.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.414 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rAq85E+DxbpGO8CbzcSboFk/JKY= 2024-04-20T01:55:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rAq85E+DxbpGO8CbzcSboFk/JKY='} 2024-04-20T01:55:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000980174ec2f433953-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/ 2024-04-20T01:55:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/ 2024-04-20T01:55:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/ 2024-04-20T01:55:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.415 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.415 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.415 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.415 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.415 INFO:tasks.rgw_multisite_tests:/aktlji-71/ 2024-04-20T01:55:26.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.415 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8= 2024-04-20T01:55:26.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o9o+5lCYtXyDiFNwBFPZa+NQj+8=', 'Content-Length': '0'} 2024-04-20T01:55:26.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bea185cfc2458d1-0066231049-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-71/h 2024-04-20T01:55:26.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-71/h 2024-04-20T01:55:26.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-71/h 2024-04-20T01:55:26.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.416 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.417 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.417 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.417 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:46:01 GMT 2024-04-20T01:55:26.417 INFO:tasks.rgw_multisite_tests:/aktlji-71/h 2024-04-20T01:55:26.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.417 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6elNbVD7uedbOGiSIPunpMqtDFo= 2024-04-20T01:55:26.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:46:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6elNbVD7uedbOGiSIPunpMqtDFo='} 2024-04-20T01:55:26.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ce0bf043ab89f33-0066231049-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:46:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.417 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Restarting daemon 2024-04-20T01:55:26.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-20T01:55:26.417 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Started 2024-04-20T01:55:26.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > curl http://smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-20T01:55:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T01:55:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-20T01:55:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi031.front.sepia.ceph.com port 8000: Connection refused 2024-04-20T01:55:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > curl http://smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T01:55:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-20T01:55:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-20T01:55:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: anonymous 2024-04-20T01:55:26.418 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2024-04-20T01:55:26.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-20T01:55:26.418 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2024-04-20T01:55:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > curl http://smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-20T01:55:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T01:55:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-20T01:55:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi019.front.sepia.ceph.com port 8001: Connection refused 2024-04-20T01:55:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > curl http://smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T01:55:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-20T01:55:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-20T01:55:26.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: anonymous 2024-04-20T01:55:26.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-71 2024-04-20T01:55:26.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-71 2024-04-20T01:55:26.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-71 2024-04-20T01:55:26.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-71', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-71 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:46:04.285+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.421 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2024-04-20T01:55:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:46:09.286+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:46:14.287+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:46:19.287+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:46:24.288+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:46:29.289+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:46:34.289+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:46:39.290+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:46:44.291+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:46:49.291+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:46:54.292+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:46:59.292+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:47:04.293+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:47:09.294+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:47:14.293+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:47:19.294+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:47:24.294+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:47:29.295+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:47:34.299+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:47:39.299+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:47:44.300+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:47:49.300+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:47:54.301+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:47:59.302+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:04.303+0000 7f162f19a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:09.304+0000 7f162f19a780 1 bucket sync caught up with source: 2024-04-20T01:55:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [00000000003.99.6, 00000000002.51.6, 00000000002.97.6] 2024-04-20T01:55:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [00000000003.99.6, 00000000002.51.6, 00000000002.97.6] 2024-04-20T01:55:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:09.304+0000 7f162f19a780 0 bucket checkpoint complete 2024-04-20T01:55:26.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-71 2024-04-20T01:55:26.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-71', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-71 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:09.433+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:14.434+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:19.435+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:24.436+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:29.436+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:34.437+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:39.437+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:44.438+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:49.439+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:54.439+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:48:59.439+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:49:04.439+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:49:09.440+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:49:14.441+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:49:19.441+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:49:24.442+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:49:29.443+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:49:34.443+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:49:39.444+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:49:44.444+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:49:49.445+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:49:54.445+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:49:59.446+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:50:04.446+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:50:09.448+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:50:14.449+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:50:19.449+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:50:24.450+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:50:29.451+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:50:34.451+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:50:39.452+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:50:44.453+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:50:49.452+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:50:54.453+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:50:59.453+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:51:04.454+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:51:09.455+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:51:14.456+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:51:19.456+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:51:24.457+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:51:29.457+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:51:34.458+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:51:39.458+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:51:44.459+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:51:49.460+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:51:54.460+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:51:59.462+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:52:04.463+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:52:09.464+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:52:14.464+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:52:19.465+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:52:24.465+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:52:29.466+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:52:34.466+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:52:39.466+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:52:44.467+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:52:49.468+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:52:54.468+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:52:59.469+0000 7fed53e78780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:04.470+0000 7fed53e78780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:04.470+0000 7fed53e78780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:04.470+0000 7fed53e78780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.429 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.429 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.429 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.429 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, then run bucket sync 2024-04-20T01:55:26.430 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.430 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.430 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.430 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.430 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1624, in test_bucket_sync_run_basic_incremental 2024-04-20T01:55:26.430 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-20T01:55:26.430 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.430 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.430 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.431 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.431 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.431 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.431 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.431 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/ 2024-04-20T01:55:26.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/ 2024-04-20T01:55:26.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/ 2024-04-20T01:55:26.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.432 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.432 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.432 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.432 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.432 INFO:tasks.rgw_multisite_tests:/aktlji-72/ 2024-04-20T01:55:26.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.432 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:UE9wsSVQjJMIjxzvVehNz2XM+ds= 2024-04-20T01:55:26.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UE9wsSVQjJMIjxzvVehNz2XM+ds=', 'Content-Length': '0'} 2024-04-20T01:55:26.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000bd3e8b13b5423f16-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=aktlji-72 2024-04-20T01:55:26.433 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.433 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713574385.072559_1017.1", 2024-04-20T01:55:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:53:05.072559Z" 2024-04-20T01:55:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574385.072559_1017.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:53:05.307+0000 7f580ca83780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:53:05.308+0000 7f580ca83780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713574385.072559_1017.1", 2024-04-20T01:55:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:53:05.072559Z", 2024-04-20T01:55:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:53:05Z" 2024-04-20T01:55:26.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.441 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.441 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.442 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.442 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.442 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.442 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.442 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.442 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.442 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.442 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.442 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.442 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.442 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.443 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.443 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.443 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.443 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.443 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.443 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.443 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.443 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.443 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.443 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.444 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.444 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.444 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.444 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.444 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.444 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.445 INFO:tasks.rgw_multisite_tests: "marker": "1_1713574385.072559_1017.1", 2024-04-20T01:55:26.445 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.445 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.445 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:53:05.072559Z", 2024-04-20T01:55:26.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.445 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.445 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.445 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.445 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.445 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.446 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.446 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.446 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.446 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.446 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.446 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.446 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.446 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.446 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.446 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.446 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.447 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.447 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.447 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.447 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:53:05Z" 2024-04-20T01:55:26.447 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.447 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713574385.072559_1017.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:53:05.072559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574385.072559_1017.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574385.072559_1017.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.448 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:05.438+0000 7f0267f51780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:05.439+0000 7f0267f51780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713574385.072559_1017.1", 2024-04-20T01:55:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T00:53:05.072559Z", 2024-04-20T01:55:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T00:53:05Z" 2024-04-20T01:55:26.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.454 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.454 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.454 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.454 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.454 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.454 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.454 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.454 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.454 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.455 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.455 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.455 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.455 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.455 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.455 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.455 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.455 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.455 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.455 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.455 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.455 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.456 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.456 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.456 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.456 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.456 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.456 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.456 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.456 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.456 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.456 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.456 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.456 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.457 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.457 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.457 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.457 INFO:tasks.rgw_multisite_tests: "marker": "1_1713574385.072559_1017.1", 2024-04-20T01:55:26.457 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.457 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.457 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.457 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:53:05.072559Z", 2024-04-20T01:55:26.457 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.457 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.457 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.457 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.458 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.458 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.458 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.458 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.458 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.458 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.458 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.459 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.459 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.459 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.459 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.459 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.459 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.459 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:53:05Z" 2024-04-20T01:55:26.459 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.459 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713574385.072559_1017.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:53:05.072559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574385.072559_1017.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.459 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574385.072559_1017.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.460 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.460 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "objs": [ 2024-04-20T01:55:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj00000000000000000001", 2024-04-20T01:55:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj00000000000000000005", 2024-04-20T01:55:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj00000000000000000012", 2024-04-20T01:55:26.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj00000000000000000004", 2024-04-20T01:55:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj00000000000000000011", 2024-04-20T01:55:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj00000000000000000000", 2024-04-20T01:55:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj00000000000000000010", 2024-04-20T01:55:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj00000000000000000003", 2024-04-20T01:55:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj00000000000000000013", 2024-04-20T01:55:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj00000000000000000002", 2024-04-20T01:55:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "obj00000000000000000006" 2024-04-20T01:55:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/ 2024-04-20T01:55:26.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/ 2024-04-20T01:55:26.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/ 2024-04-20T01:55:26.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.462 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.462 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.462 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.463 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.463 INFO:tasks.rgw_multisite_tests:/aktlji-72/ 2024-04-20T01:55:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.463 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T01:55:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T01:55:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094ecf7d64f54cbce-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/obj00000000000000000002 2024-04-20T01:55:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/obj00000000000000000002 2024-04-20T01:55:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/obj00000000000000000002 2024-04-20T01:55:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.464 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.464 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.464 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.464 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.464 INFO:tasks.rgw_multisite_tests:/aktlji-72/obj00000000000000000002 2024-04-20T01:55:26.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.464 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dH6Y8sH8w6WtD9T4iVZrSt3oddA= 2024-04-20T01:55:26.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dH6Y8sH8w6WtD9T4iVZrSt3oddA='} 2024-04-20T01:55:26.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0277dbba0c40c93-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/ 2024-04-20T01:55:26.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/ 2024-04-20T01:55:26.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/ 2024-04-20T01:55:26.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.466 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.466 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.466 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.466 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.466 INFO:tasks.rgw_multisite_tests:/aktlji-72/ 2024-04-20T01:55:26.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.466 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T01:55:26.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T01:55:26.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a2273292ee5f024-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/obj00000000000000000006 2024-04-20T01:55:26.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/obj00000000000000000006 2024-04-20T01:55:26.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/obj00000000000000000006 2024-04-20T01:55:26.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.467 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.467 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.468 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.468 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.468 INFO:tasks.rgw_multisite_tests:/aktlji-72/obj00000000000000000006 2024-04-20T01:55:26.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.468 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:80gAY/d4TySH/6kHDtojiYeIvXg= 2024-04-20T01:55:26.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:80gAY/d4TySH/6kHDtojiYeIvXg='} 2024-04-20T01:55:26.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4e23ad06d816eae-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/ 2024-04-20T01:55:26.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/ 2024-04-20T01:55:26.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/ 2024-04-20T01:55:26.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.469 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.469 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.469 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.469 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.469 INFO:tasks.rgw_multisite_tests:/aktlji-72/ 2024-04-20T01:55:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.470 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T01:55:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T01:55:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afcd7bb477b5e131-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/obj00000000000000000011 2024-04-20T01:55:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/obj00000000000000000011 2024-04-20T01:55:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/obj00000000000000000011 2024-04-20T01:55:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.471 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.471 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.471 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.471 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.471 INFO:tasks.rgw_multisite_tests:/aktlji-72/obj00000000000000000011 2024-04-20T01:55:26.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.471 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:OLu+qWqckjHhfExSIVZveY3cWu4= 2024-04-20T01:55:26.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OLu+qWqckjHhfExSIVZveY3cWu4='} 2024-04-20T01:55:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019a59cd629217576-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/ 2024-04-20T01:55:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/ 2024-04-20T01:55:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/ 2024-04-20T01:55:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.473 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.473 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.473 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.473 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.473 INFO:tasks.rgw_multisite_tests:/aktlji-72/ 2024-04-20T01:55:26.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.473 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T01:55:26.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T01:55:26.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f2a2cd577238e03-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/obj00000000000000000000 2024-04-20T01:55:26.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/obj00000000000000000000 2024-04-20T01:55:26.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/obj00000000000000000000 2024-04-20T01:55:26.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.474 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.474 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.474 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.475 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.475 INFO:tasks.rgw_multisite_tests:/aktlji-72/obj00000000000000000000 2024-04-20T01:55:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.475 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:63a60/G+QTrdnveC0q0lg1EDk+Y= 2024-04-20T01:55:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:63a60/G+QTrdnveC0q0lg1EDk+Y='} 2024-04-20T01:55:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000613885d886d6260c-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/ 2024-04-20T01:55:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/ 2024-04-20T01:55:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/ 2024-04-20T01:55:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.476 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.476 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.476 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.476 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.476 INFO:tasks.rgw_multisite_tests:/aktlji-72/ 2024-04-20T01:55:26.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.476 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T01:55:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T01:55:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000558c2bfff50a1658-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/obj00000000000000000012 2024-04-20T01:55:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/obj00000000000000000012 2024-04-20T01:55:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/obj00000000000000000012 2024-04-20T01:55:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.478 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.478 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.478 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.478 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.478 INFO:tasks.rgw_multisite_tests:/aktlji-72/obj00000000000000000012 2024-04-20T01:55:26.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.478 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6Ni3nCSz5x7HyYp5l2SQ+6+dCQQ= 2024-04-20T01:55:26.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6Ni3nCSz5x7HyYp5l2SQ+6+dCQQ='} 2024-04-20T01:55:26.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d19205184cd46a09-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/ 2024-04-20T01:55:26.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/ 2024-04-20T01:55:26.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/ 2024-04-20T01:55:26.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.479 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.480 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.480 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.480 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.480 INFO:tasks.rgw_multisite_tests:/aktlji-72/ 2024-04-20T01:55:26.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.480 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T01:55:26.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T01:55:26.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000733a79903b20643-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/obj00000000000000000010 2024-04-20T01:55:26.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/obj00000000000000000010 2024-04-20T01:55:26.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/obj00000000000000000010 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:/aktlji-72/obj00000000000000000010 2024-04-20T01:55:26.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.482 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:d8+ZPb+QbkVQbpGGqrBnD3VhZuA= 2024-04-20T01:55:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:d8+ZPb+QbkVQbpGGqrBnD3VhZuA='} 2024-04-20T01:55:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007738205c2eca72c5-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/ 2024-04-20T01:55:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/ 2024-04-20T01:55:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/ 2024-04-20T01:55:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.483 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.483 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.483 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.483 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.483 INFO:tasks.rgw_multisite_tests:/aktlji-72/ 2024-04-20T01:55:26.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.483 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T01:55:26.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T01:55:26.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002125699db9dcc552-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/obj00000000000000000001 2024-04-20T01:55:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/obj00000000000000000001 2024-04-20T01:55:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/obj00000000000000000001 2024-04-20T01:55:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.484 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.485 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.485 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.485 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.485 INFO:tasks.rgw_multisite_tests:/aktlji-72/obj00000000000000000001 2024-04-20T01:55:26.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.485 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cNXt7wwxMa9t4aC63R0/YeTdO18= 2024-04-20T01:55:26.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cNXt7wwxMa9t4aC63R0/YeTdO18='} 2024-04-20T01:55:26.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af56a54b05a3e9b8-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/ 2024-04-20T01:55:26.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/ 2024-04-20T01:55:26.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/ 2024-04-20T01:55:26.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.486 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.486 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.486 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.486 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.487 INFO:tasks.rgw_multisite_tests:/aktlji-72/ 2024-04-20T01:55:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.487 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T01:55:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T01:55:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f796983fa18d727-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/obj00000000000000000004 2024-04-20T01:55:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/obj00000000000000000004 2024-04-20T01:55:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/obj00000000000000000004 2024-04-20T01:55:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.488 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.488 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.488 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.488 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.488 INFO:tasks.rgw_multisite_tests:/aktlji-72/obj00000000000000000004 2024-04-20T01:55:26.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.488 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tZAR6oSHJwadb9/Oq+igxLm49LI= 2024-04-20T01:55:26.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tZAR6oSHJwadb9/Oq+igxLm49LI='} 2024-04-20T01:55:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ad50978b56daefca-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/ 2024-04-20T01:55:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/ 2024-04-20T01:55:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/ 2024-04-20T01:55:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.490 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.490 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.490 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.490 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.490 INFO:tasks.rgw_multisite_tests:/aktlji-72/ 2024-04-20T01:55:26.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.490 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T01:55:26.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T01:55:26.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075ceafba51acfc7e-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/obj00000000000000000013 2024-04-20T01:55:26.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/obj00000000000000000013 2024-04-20T01:55:26.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/obj00000000000000000013 2024-04-20T01:55:26.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.491 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.491 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.491 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.491 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.492 INFO:tasks.rgw_multisite_tests:/aktlji-72/obj00000000000000000013 2024-04-20T01:55:26.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.492 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:s2wvGBF6qkJrWk43B88+XYd5hy0= 2024-04-20T01:55:26.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:s2wvGBF6qkJrWk43B88+XYd5hy0='} 2024-04-20T01:55:26.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f38f318e2dc3c902-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/ 2024-04-20T01:55:26.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/ 2024-04-20T01:55:26.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/ 2024-04-20T01:55:26.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.493 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.493 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.493 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.493 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.493 INFO:tasks.rgw_multisite_tests:/aktlji-72/ 2024-04-20T01:55:26.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.493 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U= 2024-04-20T01:55:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4ekeiKJ+NqHoYyD0RlBOtD5JM6U=', 'Content-Length': '0'} 2024-04-20T01:55:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ccbee340dd8cc66-00662311f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-72/obj00000000000000000003 2024-04-20T01:55:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-72/obj00000000000000000003 2024-04-20T01:55:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-72/obj00000000000000000003 2024-04-20T01:55:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.495 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.495 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.495 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.495 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:53:05 GMT 2024-04-20T01:55:26.495 INFO:tasks.rgw_multisite_tests:/aktlji-72/obj00000000000000000003 2024-04-20T01:55:26.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.495 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:xlFIQhx//WARG+nNsGXh5v6YTDs= 2024-04-20T01:55:26.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:53:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xlFIQhx//WARG+nNsGXh5v6YTDs='} 2024-04-20T01:55:26.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005feede02f6a91edd-00662311f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:53:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-72 2024-04-20T01:55:26.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-72 2024-04-20T01:55:26.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-72 2024-04-20T01:55:26.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:06.112+0000 7f242a0fd780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:11.112+0000 7f242a0fd780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:16.114+0000 7f242a0fd780 1 bucket sync caught up with source: 2024-04-20T01:55:26.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [00000000001.41.6, , 00000000001.41.6, 00000000001.84.6, 00000000001.21.6, 00000000001.39.6, 00000000001.65.6, 00000000001.45.6, 00000000001.67.6, 00000000001.59.6, 00000000001.37.6] 2024-04-20T01:55:26.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [00000000001.41.6, , 00000000001.41.6, 00000000001.84.6, 00000000001.21.6, 00000000001.39.6, 00000000001.65.6, 00000000001.45.6, 00000000001.67.6, 00000000001.59.6, 00000000001.37.6] 2024-04-20T01:55:26.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:16.114+0000 7f242a0fd780 0 bucket checkpoint complete 2024-04-20T01:55:26.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-72 2024-04-20T01:55:26.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-72', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-72 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:16.241+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:21.242+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:26.243+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:31.243+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:36.244+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:41.244+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:46.245+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:51.246+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:53:56.247+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:54:01.248+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:54:06.248+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:54:11.249+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:54:16.249+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:54:21.249+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:54:26.250+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:54:31.250+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:54:36.251+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:54:41.252+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:54:46.252+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:54:51.252+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:54:56.253+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:55:01.254+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:55:06.254+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:55:11.255+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:55:16.255+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:55:21.256+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:55:26.256+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:55:31.257+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:55:36.259+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:55:41.259+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:55:46.260+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:55:51.261+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:55:56.261+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:56:01.262+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:56:06.263+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:56:11.262+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:56:16.263+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:56:21.263+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:56:26.264+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:56:31.265+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:56:36.265+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:56:41.266+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:56:46.266+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:56:51.267+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:56:56.268+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:57:01.269+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:57:06.269+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:57:11.270+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:57:16.270+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:57:21.271+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:57:26.272+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:57:31.273+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:57:36.274+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:57:41.274+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:57:46.275+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:57:51.275+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:57:56.275+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:01.275+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:06.276+0000 7fa4f484a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:11.277+0000 7fa4f484a780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:11.277+0000 7fa4f484a780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:11.277+0000 7fa4f484a780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.503 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.503 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.503 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.503 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, trash them, then run bucket sync init 2024-04-20T01:55:26.503 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.504 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.504 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.504 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.504 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1691, in test_zap_init_bucket_sync_run 2024-04-20T01:55:26.504 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-20T01:55:26.504 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.504 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.504 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.504 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.504 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.504 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.505 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.505 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-73/ 2024-04-20T01:55:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-73/ 2024-04-20T01:55:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-73/ 2024-04-20T01:55:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.506 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.506 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.506 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.506 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:58:11 GMT 2024-04-20T01:55:26.506 INFO:tasks.rgw_multisite_tests:/aktlji-73/ 2024-04-20T01:55:26.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.506 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sFSXiTUhXSSuUu4NuPNnOv2HgqQ= 2024-04-20T01:55:26.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sFSXiTUhXSSuUu4NuPNnOv2HgqQ=', 'Content-Length': '0'} 2024-04-20T01:55:26.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000012f45b2f854bff52-0066231323-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=aktlji-73 2024-04-20T01:55:26.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713574691.323977_1063.1", 2024-04-20T01:55:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T00:58:11.323977Z" 2024-04-20T01:55:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:23:08.397150Z" 2024-04-20T01:55:26.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:58:11.610+0000 7fc20ab82780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T00:58:11.611+0000 7fc20ab82780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713574691.323977_1063.1", 2024-04-20T01:55:26.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T00:58:11.323977Z", 2024-04-20T01:55:26.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T00:58:11Z" 2024-04-20T01:55:26.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.515 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.515 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.515 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.515 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.516 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.516 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.516 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.516 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.516 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.516 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.516 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.516 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.516 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.517 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.517 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.517 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.517 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.517 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.517 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.517 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.517 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.517 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.517 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.517 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.518 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.518 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.518 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.518 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.518 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.518 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.518 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.518 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.518 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.518 INFO:tasks.rgw_multisite_tests: "marker": "1_1713574691.323977_1063.1", 2024-04-20T01:55:26.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.518 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.519 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.519 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:58:11.323977Z", 2024-04-20T01:55:26.519 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.519 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.519 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.519 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.519 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.519 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.519 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.519 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.519 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.519 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.520 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.520 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.520 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.520 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.520 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.520 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.520 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.520 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.520 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.520 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.520 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.520 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:58:11Z" 2024-04-20T01:55:26.521 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.521 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713574691.323977_1063.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:58:11.323977Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.521 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:11.735+0000 7f34c3423780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:11.736+0000 7f34c3423780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713574691.323977_1063.1", 2024-04-20T01:55:26.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T00:58:11.323977Z", 2024-04-20T01:55:26.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T00:58:11Z" 2024-04-20T01:55:26.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.527 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.527 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.528 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.528 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.528 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.528 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.528 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.528 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.528 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.528 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.528 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.528 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.528 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.528 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.529 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.529 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.529 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.529 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.529 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.529 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.529 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.529 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.529 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.529 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.529 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.530 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.530 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.530 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.530 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.530 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.530 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.530 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.530 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.530 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.530 INFO:tasks.rgw_multisite_tests: "marker": "1_1713574691.323977_1063.1", 2024-04-20T01:55:26.530 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.531 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.531 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.531 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T00:58:11.323977Z", 2024-04-20T01:55:26.531 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.531 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.531 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.531 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.531 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.531 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.531 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.531 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568988.397150_80.1", 2024-04-20T01:55:26.531 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.532 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.532 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.532 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:23:08.397150Z", 2024-04-20T01:55:26.532 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.532 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.532 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.532 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.532 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.532 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.532 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.532 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.532 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.533 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T00:58:11Z" 2024-04-20T01:55:26.533 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.533 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713574691.323977_1063.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T00:58:11.323977Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713568988.397150_80.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-19T23:23:08.397150Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713574691.323977_1063.1', 3: '1_1713568988.397150_80.1'} 2024-04-20T01:55:26.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-73/ 2024-04-20T01:55:26.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-73/ 2024-04-20T01:55:26.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-73/ 2024-04-20T01:55:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.534 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.534 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.534 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.534 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:58:11 GMT 2024-04-20T01:55:26.534 INFO:tasks.rgw_multisite_tests:/aktlji-73/ 2024-04-20T01:55:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.535 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY= 2024-04-20T01:55:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY=', 'Content-Length': '0'} 2024-04-20T01:55:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f91c7c5995e2135-0066231323-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-73/obj11 2024-04-20T01:55:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-73/obj11 2024-04-20T01:55:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-73/obj11 2024-04-20T01:55:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.536 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.536 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.536 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.536 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:58:11 GMT 2024-04-20T01:55:26.536 INFO:tasks.rgw_multisite_tests:/aktlji-73/obj11 2024-04-20T01:55:26.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.536 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:uAuX9VCnFp3QMTYx963x5kCUI3I= 2024-04-20T01:55:26.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:uAuX9VCnFp3QMTYx963x5kCUI3I='} 2024-04-20T01:55:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca89e6169c9d663e-0066231323-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-73/ 2024-04-20T01:55:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-73/ 2024-04-20T01:55:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-73/ 2024-04-20T01:55:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.538 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.538 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.538 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.538 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:58:11 GMT 2024-04-20T01:55:26.538 INFO:tasks.rgw_multisite_tests:/aktlji-73/ 2024-04-20T01:55:26.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.538 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY= 2024-04-20T01:55:26.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY=', 'Content-Length': '0'} 2024-04-20T01:55:26.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069ece72e58c953d6-0066231323-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-73/obj22 2024-04-20T01:55:26.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-73/obj22 2024-04-20T01:55:26.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-73/obj22 2024-04-20T01:55:26.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.539 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.539 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.539 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.540 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:58:11 GMT 2024-04-20T01:55:26.540 INFO:tasks.rgw_multisite_tests:/aktlji-73/obj22 2024-04-20T01:55:26.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.540 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QUK90/CMs8q3alpCf9hT0aWhjSE= 2024-04-20T01:55:26.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QUK90/CMs8q3alpCf9hT0aWhjSE='} 2024-04-20T01:55:26.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000566a42085a82cf31-0066231323-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-73/ 2024-04-20T01:55:26.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-73/ 2024-04-20T01:55:26.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-73/ 2024-04-20T01:55:26.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.541 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.541 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.541 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.541 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:58:11 GMT 2024-04-20T01:55:26.541 INFO:tasks.rgw_multisite_tests:/aktlji-73/ 2024-04-20T01:55:26.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.541 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY= 2024-04-20T01:55:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY=', 'Content-Length': '0'} 2024-04-20T01:55:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eacc4384e59c566c-0066231323-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-73/obj33 2024-04-20T01:55:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-73/obj33 2024-04-20T01:55:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-73/obj33 2024-04-20T01:55:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.543 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.543 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.543 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.543 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:58:11 GMT 2024-04-20T01:55:26.543 INFO:tasks.rgw_multisite_tests:/aktlji-73/obj33 2024-04-20T01:55:26.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.543 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zApvcus8FnxB1IJnVBJg7QCUrUg= 2024-04-20T01:55:26.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zApvcus8FnxB1IJnVBJg7QCUrUg='} 2024-04-20T01:55:26.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000592ecdfabaf05da1-0066231323-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-73/ 2024-04-20T01:55:26.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-73/ 2024-04-20T01:55:26.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-73/ 2024-04-20T01:55:26.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.544 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.544 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.545 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.545 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:58:11 GMT 2024-04-20T01:55:26.545 INFO:tasks.rgw_multisite_tests:/aktlji-73/ 2024-04-20T01:55:26.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.545 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY= 2024-04-20T01:55:26.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY=', 'Content-Length': '0'} 2024-04-20T01:55:26.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008161af0b62d18a6e-0066231323-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-73/obj44 2024-04-20T01:55:26.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-73/obj44 2024-04-20T01:55:26.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-73/obj44 2024-04-20T01:55:26.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.546 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.546 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.546 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.546 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:58:11 GMT 2024-04-20T01:55:26.546 INFO:tasks.rgw_multisite_tests:/aktlji-73/obj44 2024-04-20T01:55:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.547 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:faRgBH7uYpjXRW5wVhZjkPYq9wE= 2024-04-20T01:55:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:faRgBH7uYpjXRW5wVhZjkPYq9wE='} 2024-04-20T01:55:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c5ded47a39488b0-0066231323-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-73/ 2024-04-20T01:55:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-73/ 2024-04-20T01:55:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-73/ 2024-04-20T01:55:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.548 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.548 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.548 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.548 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:58:11 GMT 2024-04-20T01:55:26.548 INFO:tasks.rgw_multisite_tests:/aktlji-73/ 2024-04-20T01:55:26.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.548 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY= 2024-04-20T01:55:26.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HpkRsyKoE7R1KtEboiYsCfx+oHY=', 'Content-Length': '0'} 2024-04-20T01:55:26.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000456ea47d3028d0f2-0066231323-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-73/obj55 2024-04-20T01:55:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-73/obj55 2024-04-20T01:55:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-73/obj55 2024-04-20T01:55:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.549 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.550 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.550 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.550 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 00:58:11 GMT 2024-04-20T01:55:26.550 INFO:tasks.rgw_multisite_tests:/aktlji-73/obj55 2024-04-20T01:55:26.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.550 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3ms3zpQEpOL0dr2tFPoMu1AjFD0= 2024-04-20T01:55:26.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 00:58:11 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3ms3zpQEpOL0dr2tFPoMu1AjFD0='} 2024-04-20T01:55:26.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000766102c2dc765709-0066231323-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 00:58:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-73 2024-04-20T01:55:26.550 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.550 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-73 2024-04-20T01:55:26.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-73 2024-04-20T01:55:26.551 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.551 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-73', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-73 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:12.097+0000 7f869acb9780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:17.097+0000 7f869acb9780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:22.098+0000 7f869acb9780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:27.100+0000 7f869acb9780 1 bucket sync caught up with source: 2024-04-20T01:55:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [, , , , , , , , 00000000001.24.6, 00000000002.89.6, 00000000002.46.6] 2024-04-20T01:55:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [, , , , , , , , 00000000001.24.6, 00000000002.89.6, 00000000002.46.6] 2024-04-20T01:55:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:27.100+0000 7f869acb9780 0 bucket checkpoint complete 2024-04-20T01:55:26.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-73 2024-04-20T01:55:26.552 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.552 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-73', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-73 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:27.226+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:32.226+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:37.227+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:42.228+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:47.228+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:52.229+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:58:57.230+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:59:02.230+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:59:07.231+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:59:12.232+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:59:17.233+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:59:22.233+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:59:27.234+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:59:32.234+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:59:37.235+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:59:42.236+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:59:47.235+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:59:52.236+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T00:59:57.236+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:00:02.237+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:00:07.238+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:00:12.238+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:00:17.239+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:00:22.239+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:00:27.240+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:00:32.240+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:00:37.241+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:00:42.241+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:00:47.241+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:00:52.242+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:00:57.243+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:01:02.244+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:01:07.244+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:01:12.245+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:01:17.246+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:01:22.246+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:01:27.247+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:01:32.246+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:01:37.247+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:01:42.247+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:01:47.248+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:01:52.248+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:01:57.249+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:02:02.250+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:02:07.250+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:02:12.251+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:02:17.252+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:02:22.252+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:02:27.253+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:02:32.254+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:02:37.254+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:02:42.255+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:02:47.256+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:02:52.257+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:02:57.257+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:02.258+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:07.259+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:12.259+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:17.260+0000 7fca338b5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:22.259+0000 7fca338b5780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:22.259+0000 7fca338b5780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:22.260+0000 7fca338b5780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.558 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.558 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.558 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.558 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_full_sync_after_data_sync_init 2024-04-20T01:55:26.558 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.559 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.559 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.559 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.559 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1769, in test_bucket_full_sync_after_data_sync_init 2024-04-20T01:55:26.559 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-20T01:55:26.559 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.559 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.559 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.559 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.559 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.560 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.560 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.560 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-74/ 2024-04-20T01:55:26.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-74/ 2024-04-20T01:55:26.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-74/ 2024-04-20T01:55:26.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.561 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.561 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.561 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.561 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:03:28 GMT 2024-04-20T01:55:26.561 INFO:tasks.rgw_multisite_tests:/aktlji-74/ 2024-04-20T01:55:26.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.561 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:gre5QuW480JFWxNer0oVfpqYdAY= 2024-04-20T01:55:26.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:03:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gre5QuW480JFWxNer0oVfpqYdAY=', 'Content-Length': '0'} 2024-04-20T01:55:26.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000fc0802ff73dd7102-0066231460-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:03:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=aktlji-74 2024-04-20T01:55:26.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575008.904609_1102.1", 2024-04-20T01:55:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:28.904609Z" 2024-04-20T01:55:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:26.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.904609_1102.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:03:29.143+0000 7f79e2d06780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:03:29.144+0000 7f79e2d06780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575008.904609_1102.1", 2024-04-20T01:55:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:28.904609Z", 2024-04-20T01:55:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:03:29Z" 2024-04-20T01:55:26.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.570 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.570 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.570 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.570 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.570 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.571 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.571 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.571 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.571 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.571 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.571 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.571 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.571 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.571 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.571 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.571 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.572 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.572 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.572 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.572 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.572 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.572 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.572 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.572 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.572 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.572 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.573 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.573 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.573 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.573 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.573 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.573 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.573 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.573 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575008.904609_1102.1", 2024-04-20T01:55:26.573 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.573 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.573 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.573 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:28.904609Z", 2024-04-20T01:55:26.574 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.574 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.574 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.574 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.574 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.574 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.574 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.574 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.574 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.574 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.574 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.574 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.575 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.575 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.575 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.575 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.575 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.575 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.575 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.575 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.575 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.575 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:03:29Z" 2024-04-20T01:55:26.575 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.575 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575008.904609_1102.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:28.904609Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.904609_1102.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.904609_1102.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.576 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.576 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:29.272+0000 7fe1f8c4f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:29.273+0000 7fe1f8c4f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575008.896558_1099.1", 2024-04-20T01:55:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:28.896558Z", 2024-04-20T01:55:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:03:29Z" 2024-04-20T01:55:26.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.582 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.582 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.582 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.582 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.582 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.583 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.583 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.583 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.583 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.583 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.583 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.583 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.583 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.583 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.583 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.583 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.584 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.584 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.584 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.584 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.584 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.584 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.584 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.584 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.584 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.584 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.584 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.584 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.585 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.585 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.585 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.585 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.585 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.585 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.585 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.585 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575008.896558_1099.1", 2024-04-20T01:55:26.585 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.585 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.585 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.585 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:28.896558Z", 2024-04-20T01:55:26.586 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.586 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.586 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.586 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.586 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.586 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.586 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.586 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.586 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.586 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.586 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.587 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.587 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.587 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.587 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.587 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.587 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.587 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.587 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.587 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.587 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.587 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:03:29Z" 2024-04-20T01:55:26.587 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.588 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575008.896558_1099.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:28.896558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.904609_1102.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.896558_1099.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1713575008.904609_1102.1 target=1_1713575008.896558_1099.1 2024-04-20T01:55:26.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:34.394+0000 7f3834791780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:34.395+0000 7f3834791780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575008.904609_1102.1", 2024-04-20T01:55:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:28.904609Z", 2024-04-20T01:55:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:03:34Z" 2024-04-20T01:55:26.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.594 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.594 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.594 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.594 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.595 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.595 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.595 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.595 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.595 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.595 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.595 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.595 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.595 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.595 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.595 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.595 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.596 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.596 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.596 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.596 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.596 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.596 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.596 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.596 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.596 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.596 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.596 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.597 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.597 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.597 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.597 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.597 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.597 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.597 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.597 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.597 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.597 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575008.904609_1102.1", 2024-04-20T01:55:26.597 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.597 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.598 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.598 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:28.904609Z", 2024-04-20T01:55:26.598 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.598 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.598 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.598 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.598 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.598 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.598 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.598 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.598 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.598 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.599 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.599 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.599 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.599 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.599 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.599 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.599 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.599 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.599 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.599 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.599 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.599 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:03:34Z" 2024-04-20T01:55:26.600 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.600 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575008.904609_1102.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:28.904609Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.904609_1102.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575008.904609_1102.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.600 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.600 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: DEBUG: waiting for process to exit 2024-04-20T01:55:26.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-20T01:55:26.600 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi031.stdout: INFO: 2024-04-20T01:03:34.421+0000 7faf8cbd8700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 100462) UID: 0 2024-04-20T01:55:26.600 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi031.stdout: INFO: 2024-04-20T01:03:34.421+0000 7faf91dad7c0 -1 shutting down 2024-04-20T01:55:26.600 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2024-04-20T01:55:26.601 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Stopped 2024-04-20T01:55:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-74/ 2024-04-20T01:55:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-74/ 2024-04-20T01:55:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-74/ 2024-04-20T01:55:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.602 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.602 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.602 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.602 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:03:40 GMT 2024-04-20T01:55:26.602 INFO:tasks.rgw_multisite_tests:/aktlji-74/ 2024-04-20T01:55:26.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.602 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4= 2024-04-20T01:55:26.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4=', 'Content-Length': '0'} 2024-04-20T01:55:26.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b82c76c0fda1e0e4-006623146c-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-74/obj11 2024-04-20T01:55:26.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-74/obj11 2024-04-20T01:55:26.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-74/obj11 2024-04-20T01:55:26.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.603 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.603 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.603 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.604 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:03:40 GMT 2024-04-20T01:55:26.604 INFO:tasks.rgw_multisite_tests:/aktlji-74/obj11 2024-04-20T01:55:26.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.604 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KLLYFcx31SY8zsPwNo9txJNSW6Q= 2024-04-20T01:55:26.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KLLYFcx31SY8zsPwNo9txJNSW6Q='} 2024-04-20T01:55:26.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bff0ff75ce660c2e-006623146c-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-74/ 2024-04-20T01:55:26.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-74/ 2024-04-20T01:55:26.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-74/ 2024-04-20T01:55:26.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.605 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.605 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.605 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.605 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:03:40 GMT 2024-04-20T01:55:26.605 INFO:tasks.rgw_multisite_tests:/aktlji-74/ 2024-04-20T01:55:26.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.605 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4= 2024-04-20T01:55:26.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4=', 'Content-Length': '0'} 2024-04-20T01:55:26.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d246e66b906b05c0-006623146c-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-74/obj22 2024-04-20T01:55:26.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-74/obj22 2024-04-20T01:55:26.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-74/obj22 2024-04-20T01:55:26.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.607 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.607 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.607 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.607 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:03:40 GMT 2024-04-20T01:55:26.607 INFO:tasks.rgw_multisite_tests:/aktlji-74/obj22 2024-04-20T01:55:26.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.607 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:t6Vhg/2Pa3d1dh7qUIDt+77jmjI= 2024-04-20T01:55:26.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:t6Vhg/2Pa3d1dh7qUIDt+77jmjI='} 2024-04-20T01:55:26.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfa0f255808b4316-006623146c-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-74/ 2024-04-20T01:55:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-74/ 2024-04-20T01:55:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-74/ 2024-04-20T01:55:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.608 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.609 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.609 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.609 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:03:40 GMT 2024-04-20T01:55:26.609 INFO:tasks.rgw_multisite_tests:/aktlji-74/ 2024-04-20T01:55:26.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.609 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4= 2024-04-20T01:55:26.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4=', 'Content-Length': '0'} 2024-04-20T01:55:26.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e20c1de30acf869e-006623146c-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-74/obj33 2024-04-20T01:55:26.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-74/obj33 2024-04-20T01:55:26.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-74/obj33 2024-04-20T01:55:26.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.610 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.610 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.610 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.610 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:03:40 GMT 2024-04-20T01:55:26.611 INFO:tasks.rgw_multisite_tests:/aktlji-74/obj33 2024-04-20T01:55:26.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.611 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:feHLlioOVlgAb1sKs2YoxrLIJCU= 2024-04-20T01:55:26.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:feHLlioOVlgAb1sKs2YoxrLIJCU='} 2024-04-20T01:55:26.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094ee7789141c0ce8-006623146c-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-74/ 2024-04-20T01:55:26.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-74/ 2024-04-20T01:55:26.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-74/ 2024-04-20T01:55:26.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.612 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.612 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.612 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.612 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:03:40 GMT 2024-04-20T01:55:26.612 INFO:tasks.rgw_multisite_tests:/aktlji-74/ 2024-04-20T01:55:26.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.612 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4= 2024-04-20T01:55:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4=', 'Content-Length': '0'} 2024-04-20T01:55:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b360b6d8578b3b6-006623146c-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-74/obj44 2024-04-20T01:55:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-74/obj44 2024-04-20T01:55:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-74/obj44 2024-04-20T01:55:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.614 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.614 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.614 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.614 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:03:40 GMT 2024-04-20T01:55:26.614 INFO:tasks.rgw_multisite_tests:/aktlji-74/obj44 2024-04-20T01:55:26.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.614 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2vkRpRUOOq+s/XUHCrJHn4MA7VI= 2024-04-20T01:55:26.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2vkRpRUOOq+s/XUHCrJHn4MA7VI='} 2024-04-20T01:55:26.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e621aef8424939e0-006623146c-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-74/ 2024-04-20T01:55:26.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-74/ 2024-04-20T01:55:26.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-74/ 2024-04-20T01:55:26.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.615 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.615 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.616 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.616 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:03:40 GMT 2024-04-20T01:55:26.616 INFO:tasks.rgw_multisite_tests:/aktlji-74/ 2024-04-20T01:55:26.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.616 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4= 2024-04-20T01:55:26.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:4YpO5WrWDYyVkm0kkmMg+TiElI4=', 'Content-Length': '0'} 2024-04-20T01:55:26.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cf768cd7a25d343-006623146c-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-74/obj55 2024-04-20T01:55:26.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-74/obj55 2024-04-20T01:55:26.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-74/obj55 2024-04-20T01:55:26.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.617 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.617 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.617 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.617 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:03:40 GMT 2024-04-20T01:55:26.617 INFO:tasks.rgw_multisite_tests:/aktlji-74/obj55 2024-04-20T01:55:26.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.618 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:lF9REuFNEudtFC9RA55Q8kP2KxA= 2024-04-20T01:55:26.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:03:40 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:lF9REuFNEudtFC9RA55Q8kP2KxA='} 2024-04-20T01:55:26.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048c0a2ab1aedd7a6-006623146c-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:03:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.618 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.618 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:03:41.778+0000 7fd370546780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-20T01:55:26.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:03:41.780+0000 7fd370546780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-20T01:55:26.618 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Restarting daemon 2024-04-20T01:55:26.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-20T01:55:26.619 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Started 2024-04-20T01:55:26.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > curl http://smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T01:55:26.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-20T01:55:26.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-20T01:55:26.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi031.front.sepia.ceph.com port 8000: Connection refused 2024-04-20T01:55:26.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > curl http://smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T01:55:26.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-20T01:55:26.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-20T01:55:26.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: anonymous 2024-04-20T01:55:26.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-74 2024-04-20T01:55:26.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-74 2024-04-20T01:55:26.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-74 2024-04-20T01:55:26.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-74', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-74 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:43.197+0000 7fbdb4f20780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.621 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 has been restored 2024-04-20T01:55:26.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:48.199+0000 7fbdb4f20780 1 bucket sync caught up with source: 2024-04-20T01:55:26.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [, , , , , , , , 00000000001.43.6, 00000000002.42.6, 00000000002.64.6] 2024-04-20T01:55:26.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [, , , , , , , , 00000000001.43.6, 00000000002.42.6, 00000000002.64.6] 2024-04-20T01:55:26.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:48.199+0000 7fbdb4f20780 0 bucket checkpoint complete 2024-04-20T01:55:26.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-74 2024-04-20T01:55:26.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.621 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-74', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-74 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:48.325+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:53.326+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:03:58.326+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:04:03.327+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:04:08.328+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:04:13.328+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:04:18.329+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:04:23.329+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:04:28.330+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:04:33.331+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:04:38.332+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:04:43.332+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:04:48.332+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:04:53.333+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:04:58.333+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:05:03.334+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:05:08.334+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:05:13.334+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:05:18.335+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:05:23.335+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:05:28.336+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:05:33.337+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:05:38.337+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:05:43.338+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:05:48.338+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:05:53.339+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:05:58.340+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:06:03.340+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:06:08.341+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:06:13.342+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:06:18.343+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:06:23.344+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:06:28.344+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:06:33.345+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:06:38.346+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:06:43.346+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:06:48.347+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:06:53.346+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:06:58.347+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:07:03.347+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:07:08.348+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:07:13.349+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:07:18.349+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:07:23.350+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:07:28.351+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:07:33.351+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:07:38.352+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:07:43.353+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:07:48.354+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:07:53.354+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:07:58.355+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:03.356+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:08.357+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:13.357+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:18.358+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:23.359+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:28.359+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:33.360+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:38.361+0000 7f0e26770780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:43.360+0000 7f0e26770780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:43.360+0000 7f0e26770780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:43.360+0000 7f0e26770780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.627 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.627 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.627 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.628 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2024-04-20T01:55:26.628 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.628 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.628 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.628 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.628 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1809, in test_resharded_bucket_full_sync_after_data_sync_init 2024-04-20T01:55:26.629 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-20T01:55:26.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.629 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.629 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.629 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.629 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.629 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.629 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.630 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.630 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.631 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.631 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:08:43 GMT 2024-04-20T01:55:26.631 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.631 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EVL4Q1Jwt8qEdV5UXxclgOFVQdI= 2024-04-20T01:55:26.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:08:43 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EVL4Q1Jwt8qEdV5UXxclgOFVQdI=', 'Content-Length': '0'} 2024-04-20T01:55:26.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000099b191ea4b029d1-006623159b-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:08:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=aktlji-75 2024-04-20T01:55:26.631 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.631 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575323.409191_1143.1", 2024-04-20T01:55:26.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:08:43.409191Z" 2024-04-20T01:55:26.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:26.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.633 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:08:43.648+0000 7f5e97a47780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:08:43.649+0000 7f5e97a47780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575323.401070_1140.1", 2024-04-20T01:55:26.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:08:43.401070Z", 2024-04-20T01:55:26.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:08:43Z" 2024-04-20T01:55:26.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.640 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.640 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.640 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.640 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.640 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.640 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.640 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.640 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.640 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.641 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.641 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.641 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.641 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.641 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.641 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.641 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.641 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.642 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.642 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.642 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.642 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.642 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.642 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.642 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.642 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.642 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.643 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.643 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.643 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.643 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575323.401070_1140.1", 2024-04-20T01:55:26.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:08:43.401070Z", 2024-04-20T01:55:26.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.643 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.643 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.643 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.644 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.644 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.644 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.644 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.644 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.644 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.644 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.644 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.644 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.645 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.645 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.645 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.645 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.645 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.645 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:08:43Z" 2024-04-20T01:55:26.645 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.645 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575323.401070_1140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:08:43.401070Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.401070_1140.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713575323.409191_1143.1 target=1_1713575323.401070_1140.1 2024-04-20T01:55:26.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:08:48.788+0000 7f4ebb339780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:08:48.789+0000 7f4ebb339780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575323.401070_1140.1", 2024-04-20T01:55:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:08:43.401070Z", 2024-04-20T01:55:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:08:48Z" 2024-04-20T01:55:26.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.652 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.652 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.652 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.652 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.652 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.652 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.652 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.653 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.653 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.653 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.653 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.653 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.653 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.653 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.654 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.654 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.654 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.654 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.654 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.654 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.658 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.658 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.658 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.658 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.658 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.658 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.658 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575323.401070_1140.1", 2024-04-20T01:55:26.658 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.658 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.658 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.658 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:08:43.401070Z", 2024-04-20T01:55:26.658 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.659 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:08:48Z" 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575323.401070_1140.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:08:43.401070Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.401070_1140.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713575323.409191_1143.1 target=1_1713575323.401070_1140.1 2024-04-20T01:55:26.660 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.661 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:08:53.930+0000 7f2e9d8ee780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:08:53.931+0000 7f2e9d8ee780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575323.409191_1143.1", 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:08:43.409191Z", 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:08:53Z" 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.666 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.667 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.668 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575323.409191_1143.1", 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:08:43.409191Z", 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.669 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.670 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:08:53Z" 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575323.409191_1143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:08:43.409191Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:54.055+0000 7f1fdd42c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:08:54.057+0000 7f1fdd42c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575323.409191_1143.1", 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:08:43.409191Z", 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:08:54Z" 2024-04-20T01:55:26.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.677 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.678 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575323.409191_1143.1", 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.679 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:08:43.409191Z", 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.680 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:08:54Z" 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575323.409191_1143.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:08:43.409191Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575323.409191_1143.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: DEBUG: waiting for process to exit 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi031.stdout: INFO: 2024-04-20T01:08:54.083+0000 7f1203990700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 102262) UID: 0 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi031.stdout: INFO: 2024-04-20T01:08:54.083+0000 7f1208b657c0 -1 shutting down 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 is failed for ~0s 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Stopped 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048a6f22aee2f528c-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj11 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj11 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj11 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.684 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj11 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WV1BymXys4gH3jySusyNnRgIf/E= 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WV1BymXys4gH3jySusyNnRgIf/E='} 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d595c1055f2c6ad-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bb5a7bcd758814b-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj22 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj22 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj22 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.687 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj22 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WEK5K+wr/s1DRi/jdrBD0T36gsk= 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WEK5K+wr/s1DRi/jdrBD0T36gsk='} 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019ac8039b17b1ac0-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c07b1e72a4458d3-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj33 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj33 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj33 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.690 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj33 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:PVl+CFd41SJDje+CUqOTygr4QmE= 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PVl+CFd41SJDje+CUqOTygr4QmE='} 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e8c04067a89e4f4-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2b6198fd7bad820-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj44 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj44 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj44 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.693 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj44 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:999mwR8QWdn51Bpep/BbK7MNl6U= 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:999mwR8QWdn51Bpep/BbK7MNl6U='} 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002efe01e910677bd9-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031cea4ef2422b078-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj55 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj55 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj55 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.696 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj55 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/lIqI/fALGpkKGzn3xvboh+duL0= 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/lIqI/fALGpkKGzn3xvboh+duL0='} 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066911843e55d3f75-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-75 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-75 2024-04-20T01:55:26.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 5 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.698 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009fb81f61634beff-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj17 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj17 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj17 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj17 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:xbDQZg76Np/yYNqSn1lsRooNno8= 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:xbDQZg76Np/yYNqSn1lsRooNno8='} 2024-04-20T01:55:26.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000329cbbf6557e2ff7-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.701 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001017aab0432ffaad-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj34 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj34 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj34 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj34 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:a7uKIATiH145ODg/4rnFX+1HoCU= 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a7uKIATiH145ODg/4rnFX+1HoCU='} 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e49d47cc1137e19a-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.704 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073b879c059a27216-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj51 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj51 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj51 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj51 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:gKXAVTODbSpVj0GNWC45VHZ1134= 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gKXAVTODbSpVj0GNWC45VHZ1134='} 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df95ed9cd9f17c36-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.707 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025d74bfe9c1fdd2d-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj68 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj68 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj68 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj68 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:whdlUghEvMJC16mi2NXeSkRhSu4= 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:whdlUghEvMJC16mi2NXeSkRhSu4='} 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc69b8698ff05b18-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.710 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ab19876b85b64d7-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj85 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj85 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj85 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj85 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Uu4qftHaZwCs99H9OIiKSTbUC+8= 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Uu4qftHaZwCs99H9OIiKSTbUC+8='} 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e460cb838c287f2-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-75 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-75 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 10 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007664418d86881d15-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj19 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj19 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj19 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.715 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj19 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6uFT/n0xpqL0LaY8AtJSvhPKCBI= 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6uFT/n0xpqL0LaY8AtJSvhPKCBI='} 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efbbb305d8e199c6-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089f2ab260dbc4406-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj38 2024-04-20T01:55:26.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj38 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj38 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.718 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj38 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:yhenCCdq2XqZDnL2IL+J5S8wleU= 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:yhenCCdq2XqZDnL2IL+J5S8wleU='} 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c43602ffdb28f61a-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f236d88c94ae48b2-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj57 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj57 2024-04-20T01:55:26.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj57 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj57 2024-04-20T01:55:26.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:CzUv7i/o/wczp+Hel/XILDmmRw0= 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CzUv7i/o/wczp+Hel/XILDmmRw0='} 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7bc4f4d79076526-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c364915ef18c78a-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj76 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj76 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj76 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj76 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.724 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GktW9uRGWuuCa9+BvcpKabYqKbY= 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GktW9uRGWuuCa9+BvcpKabYqKbY='} 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000757a555d0f0d04a0-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E= 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QJPaTKLyH2UQIJocaJsuQnFU+/E=', 'Content-Length': '0'} 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c57ea330e4320a5-00662315ac-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj95 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj95 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj95 2024-04-20T01:55:26.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:00 GMT 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj95 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.727 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:5IsZ4qjDDDYXAnIBihP3vJEKO9c= 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5IsZ4qjDDDYXAnIBihP3vJEKO9c='} 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dac7a58be76fef5e-00662315ac-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-75 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-75 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 15 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.729 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009591f2a76b5e3fe1-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj23 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj23 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj23 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj23 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:U+fOwGptBGIrbz7uqrdRADLdpFg= 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:U+fOwGptBGIrbz7uqrdRADLdpFg='} 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001de860ac0ea497b1-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001d40b3e890c7f17-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj46 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj46 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj46 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj46 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ny2fjq+NelNW0gyJrl6VB0mDR00= 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ny2fjq+NelNW0gyJrl6VB0mDR00='} 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003cfae39382fb04fb-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000864cddb5a7ae4c43-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj69 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj69 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj69 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj69 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QeVwxr5KRg0/WmLykzU7uddMwz4= 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QeVwxr5KRg0/WmLykzU7uddMwz4='} 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a92295291655d387-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086ea014fbe526568-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj92 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj92 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj92 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj92 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:5zMtUnpe0syEnLX5ZILWwieZg6w= 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:5zMtUnpe0syEnLX5ZILWwieZg6w='} 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008bada654bc51acbd-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.741 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000427e2511c56ff65b-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj115 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj115 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj115 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj115 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:PRJ8J2IfWSiu9tPEt07LQX7JUdU= 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PRJ8J2IfWSiu9tPEt07LQX7JUdU='} 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007783402eb5ca5d6b-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-75 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-75 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 20 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e9b3fc4ca83a595-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj29 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj29 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj29 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj29 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.746 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1LfbGEiCzmYEO+tzx7ajy+jrj+Q= 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1LfbGEiCzmYEO+tzx7ajy+jrj+Q='} 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a50808eed0e756e8-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df6a114c5b869b44-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj58 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj58 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj58 2024-04-20T01:55:26.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj58 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:bb+beoF78n9O0bZp4P4mLHtR7cY= 2024-04-20T01:55:26.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bb+beoF78n9O0bZp4P4mLHtR7cY='} 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017e29029d743844a-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.750 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f45db7fd184172ec-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj87 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj87 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj87 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj87 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:e4I5hXIhSK79sKBNV779ReXqtWc= 2024-04-20T01:55:26.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:e4I5hXIhSK79sKBNV779ReXqtWc='} 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053bba21bb8328c94-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.753 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef4cf2e38c8c67db-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj116 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj116 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj116 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj116 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:104rpxWBODYh6YyInDcK5k6jQzA= 2024-04-20T01:55:26.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:104rpxWBODYh6YyInDcK5k6jQzA='} 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d15704370a52c41d-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.756 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df75eea371381b76-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj145 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj145 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj145 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj145 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nBHNpF+gWIQ5RnzpTZ7evscXCJ0= 2024-04-20T01:55:26.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nBHNpF+gWIQ5RnzpTZ7evscXCJ0='} 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ea8244a072e1523-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-75 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-75 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 25 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.760 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000354c5342f6ac1455-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj31 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj31 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj31 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.761 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj31 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:+C6X35vKwb/d6PoRgEqVDz9Aw3E= 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:+C6X35vKwb/d6PoRgEqVDz9Aw3E='} 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a4d9a04b0412c34-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000639cc67704e1f1d1-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj62 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj62 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj62 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.764 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj62 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8kWgJT0flyuD/R3qMnilAP488tk= 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8kWgJT0flyuD/R3qMnilAP488tk='} 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddae6ed9d7290ce4-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2b227adb1cff4b9-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj93 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj93 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj93 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.767 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj93 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:kfmnLOJcxx5EwJVcd+dCAJrWo0c= 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kfmnLOJcxx5EwJVcd+dCAJrWo0c='} 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce92c16a4f68ccaa-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000385650e4745a54e-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj124 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj124 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj124 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.770 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj124 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8z6sQGEwY/DKpnd692K39GHxqGQ= 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8z6sQGEwY/DKpnd692K39GHxqGQ='} 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1fbd9df934dc0fb-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg= 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cIIKQoYpMWXkHlRSOCW07h3vEOg=', 'Content-Length': '0'} 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aef16d5f9cc43cbd-00662315ad-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj155 2024-04-20T01:55:26.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj155 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj155 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:01 GMT 2024-04-20T01:55:26.773 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj155 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:akifC3vRJ++nWyrP0yYp7Y6CeoE= 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:akifC3vRJ++nWyrP0yYp7Y6CeoE='} 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b71853ca5ff28d8-00662315ad-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-75', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-75 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-75 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 30 2024-04-20T01:55:26.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.775 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:02 GMT 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI= 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI=', 'Content-Length': '0'} 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2add120f8461130-00662315ae-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj37 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj37 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj37 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:02 GMT 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj37 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cRgsQzUWjbdbqbpRsnZAQLBJ75g= 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cRgsQzUWjbdbqbpRsnZAQLBJ75g='} 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8c1649ebe67f232-00662315ae-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.778 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:02 GMT 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI= 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI=', 'Content-Length': '0'} 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3d8d8e833b1167f-00662315ae-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj74 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj74 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj74 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:02 GMT 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj74 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:anNSQSPZMuvxPB/kJA+vXlyTkAw= 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:anNSQSPZMuvxPB/kJA+vXlyTkAw='} 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edf2f2641a47031d-00662315ae-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:02 GMT 2024-04-20T01:55:26.781 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI= 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI=', 'Content-Length': '0'} 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f6e9661ecec62b0-00662315ae-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj111 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj111 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj111 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:02 GMT 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj111 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:SzsUBaVw9k+uN2W4d8CM8oM9aXg= 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SzsUBaVw9k+uN2W4d8CM8oM9aXg='} 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fcc0d277bb85c5d-00662315ae-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:02 GMT 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI= 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI=', 'Content-Length': '0'} 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019c87e5b0c9d7366-00662315ae-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj148 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj148 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj148 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:02 GMT 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj148 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:TZaJteZAeZlaItxrkaaLy0mtspA= 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:TZaJteZAeZlaItxrkaaLy0mtspA='} 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c336fb6a908f8503-00662315ae-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/ 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/ 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/ 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:02 GMT 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests:/aktlji-75/ 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.787 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI= 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GQxp/ThdHdQO4wXOz5jXnNxMjrI=', 'Content-Length': '0'} 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c13850dd2babcd90-00662315ae-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-75/obj185 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-75/obj185 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-75/obj185 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:09:02 GMT 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:/aktlji-75/obj185 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:KdZdjTqwgqdH8eKertDSZw0+2Uc= 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:09:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:KdZdjTqwgqdH8eKertDSZw0+2Uc='} 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000934c0543faadb449-00662315ae-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:09:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'init', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync init --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:09:02.473+0000 7f0c06bee780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:09:02.475+0000 7f0c06bee780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Restarting daemon 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0: INFO: Started 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > curl http://smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi031.front.sepia.ceph.com port 8000: Connection refused 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > curl http://smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T01:55:26.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: anonymous 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-75 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-75 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-75 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:09:03.882+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c2.rgw.client.0 has been restored 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:09:08.883+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:09:13.884+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:09:18.885+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:09:23.886+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:09:28.891+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:09:33.892+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:09:38.893+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:09:43.893+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:09:48.897+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:09:53.898+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:09:58.899+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:10:03.900+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:10:08.902+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:10:13.903+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:10:18.903+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:10:23.904+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:10:28.905+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:10:33.904+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:10:38.905+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:10:43.906+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:10:48.906+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:10:53.907+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:10:58.908+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:03.909+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:08.909+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:13.910+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:18.911+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:23.912+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:28.912+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:33.913+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:38.914+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:43.915+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:48.917+0000 7f95a6056780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:53.921+0000 7f95a6056780 1 bucket sync caught up with source: 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [, , , , , , 00000000002.77.6, , , , , , 00000000002.52.6, , , , , 00000000002.64.6, , , , , , , , , , , , , , , 00000000001.81.6, , , , 00000000001.53.6] 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [, , , , , , 00000000002.77.6, , , , , , 00000000002.52.6, , , , , 00000000002.64.6, , , , , , , , , , , , , , , 00000000001.81.6, , , , 00000000001.53.6] 2024-04-20T01:55:26.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:53.921+0000 7f95a6056780 0 bucket checkpoint complete 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-75 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-75', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-75 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:54.048+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:11:59.049+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:12:04.049+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:12:09.050+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:12:14.050+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:12:19.051+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:12:24.051+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:12:29.051+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:12:34.052+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:12:39.052+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:12:44.052+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:12:49.053+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:12:54.053+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:12:59.054+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:13:04.055+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:13:09.055+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:13:14.056+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:13:19.057+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:13:24.057+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:13:29.058+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:13:34.059+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:13:39.060+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:13:44.061+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:13:49.061+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:13:54.062+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:13:59.062+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:14:04.063+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:14:09.063+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:14:14.063+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:14:19.064+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:14:24.064+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:14:29.065+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:14:34.065+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:14:39.066+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:14:44.066+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:14:49.067+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:14:54.067+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:14:59.068+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:15:04.068+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:15:09.069+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:15:14.069+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:15:19.070+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:15:24.071+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:15:29.072+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:15:34.072+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:15:39.072+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:15:44.073+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:15:49.073+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:15:54.074+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:15:59.074+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:04.074+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:09.074+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:14.075+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:19.076+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:24.076+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:29.077+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:34.077+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:39.078+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:44.078+0000 7f2f97a4a780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:49.079+0000 7f2f97a4a780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:49.079+0000 7f2f97a4a780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:49.079+0000 7f2f97a4a780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.800 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.800 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.800 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.800 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init 2024-04-20T01:55:26.800 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.800 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.800 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1827, in test_bucket_incremental_sync_after_data_sync_init 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-76/ 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-76/ 2024-04-20T01:55:26.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-76/ 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.802 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:16:49 GMT 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:/aktlji-76/ 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rh7KF+lmFeIVwr1Gg0fFCQwmMzM= 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:16:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rh7KF+lmFeIVwr1Gg0fFCQwmMzM=', 'Content-Length': '0'} 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000dcf1db16a1098cc5-0066231781-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:16:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=aktlji-76 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575809.125650_1219.1", 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:16:49.125650Z" 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575809.125650_1219.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:16:49.372+0000 7f51df366780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:16:49.373+0000 7f51df366780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575809.125650_1219.1", 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:16:49.125650Z", 2024-04-20T01:55:26.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:16:49Z" 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.810 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.811 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.812 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575809.125650_1219.1", 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:16:49.125650Z", 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.814 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:16:49Z" 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575809.125650_1219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:16:49.125650Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575809.125650_1219.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575809.125650_1219.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:49.507+0000 7f32f3689780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:49.508+0000 7f32f3689780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575809.125650_1219.1", 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:16:49.125650Z", 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:16:49Z" 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.821 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575809.125650_1219.1", 2024-04-20T01:55:26.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:16:49.125650Z", 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:16:49Z" 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.825 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713575809.125650_1219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:16:49.125650Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575809.125650_1219.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713575809.125650_1219.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-76/ 2024-04-20T01:55:26.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-76/ 2024-04-20T01:55:26.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-76/ 2024-04-20T01:55:26.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:16:49 GMT 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests:/aktlji-76/ 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Ny9xGTiw1HqwFFVcElPpQi77kX8= 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:16:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Ny9xGTiw1HqwFFVcElPpQi77kX8=', 'Content-Length': '0'} 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041495a2ee85b2680-0066231781-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:16:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-76/dummy 2024-04-20T01:55:26.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-76/dummy 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-76/dummy 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.828 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:16:49 GMT 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:/aktlji-76/dummy 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:CXEax3Mtb5vqsqbNgyUQctMqzPY= 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:16:49 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CXEax3Mtb5vqsqbNgyUQctMqzPY='} 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000941bccd289db4fcf-0066231781-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:16:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-76 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-76 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-76 2024-04-20T01:55:26.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-76 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-76', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-76 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:49.804+0000 7f0928a5d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:54.804+0000 7f0928a5d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:16:59.805+0000 7f0928a5d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:04.806+0000 7f0928a5d780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:09.807+0000 7f0928a5d780 1 bucket sync caught up with source: 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [, , , , , 00000000001.50.6, , , , , ] 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [, , , , , 00000000001.50.6, , , , , ] 2024-04-20T01:55:26.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:09.807+0000 7f0928a5d780 0 bucket checkpoint complete 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-76 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-76', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-76 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:09.950+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:14.951+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:19.952+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:24.953+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:29.953+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:34.954+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:39.954+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:44.955+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:49.955+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:54.955+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:17:59.956+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:18:04.956+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:18:09.957+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:18:14.957+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:18:19.958+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:18:24.958+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:18:29.959+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:18:34.960+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:18:39.961+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:18:44.961+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:18:49.962+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:18:54.962+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:18:59.963+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:19:04.964+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:19:09.965+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:19:14.966+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:19:19.966+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:19:24.967+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:19:29.967+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:19:34.968+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:19:39.969+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:19:44.968+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:19:49.969+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:19:54.970+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:19:59.970+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:20:04.971+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:20:09.971+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:20:14.972+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:20:19.972+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:20:24.973+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:20:29.974+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:20:34.975+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:20:39.975+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:20:44.976+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:20:49.977+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:20:54.977+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:20:59.978+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:21:04.979+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:21:09.980+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:21:14.980+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:21:19.981+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:21:24.981+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:21:29.982+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:21:34.982+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:21:39.982+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:21:44.983+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:21:49.983+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:21:54.984+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:21:59.985+0000 7fa9579d5780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:04.985+0000 7fa9579d5780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:04.985+0000 7fa9579d5780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:04.985+0000 7fa9579d5780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.836 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.836 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.836 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.836 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-04-20T01:55:26.836 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1875, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.837 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.838 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:W5z/SgdUCd8HfLoin/gHpxcstJw= 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W5z/SgdUCd8HfLoin/gHpxcstJw=', 'Content-Length': '0'} 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000053bffdba745723e9-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=aktlji-77 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713576125.034276_1247.1", 2024-04-20T01:55:26.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:22:05.034276Z" 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576125.034276_1247.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:22:05.300+0000 7f45e5c28780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:22:05.301+0000 7f45e5c28780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576125.034276_1247.1", 2024-04-20T01:55:26.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:22:05.034276Z", 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:22:05Z" 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.846 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.848 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576125.034276_1247.1", 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.849 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:22:05.034276Z", 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.850 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:22:05Z" 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576125.034276_1247.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:22:05.034276Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576125.034276_1247.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.851 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576125.034276_1247.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:05.425+0000 7f5d37391780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:05.426+0000 7f5d37391780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713576125.034276_1247.1", 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:22:05.034276Z", 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:22:05Z" 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.857 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.858 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.859 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576125.034276_1247.1", 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:22:05.034276Z", 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.860 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.861 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:22:05Z" 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576125.034276_1247.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:22:05.034276Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576125.034276_1247.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576125.034276_1247.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:55:26.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007498be42dd850835-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj11 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj11 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj11 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.864 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj11 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ouNMp4NlmUgSOpH6pyAV1mAIueg= 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ouNMp4NlmUgSOpH6pyAV1mAIueg='} 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b39e294fdaf223d-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:55:26.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047acfdb13dd43847-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj22 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj22 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj22 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.867 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj22 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:iTY3mIfQN80DYEoFgKiGYodhGRc= 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iTY3mIfQN80DYEoFgKiGYodhGRc='} 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008157ac1bc06c22a9-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fec0975f3125778d-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj33 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj33 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj33 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.870 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj33 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ii6UHlN0w6hs323xg5ixAe81K+o= 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ii6UHlN0w6hs323xg5ixAe81K+o='} 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fcceb188d8b2917-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c689bb725ebee116-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj44 2024-04-20T01:55:26.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj44 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj44 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.873 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj44 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:T7wR2ecN303BsFjok9mtuWd/1rY= 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:T7wR2ecN303BsFjok9mtuWd/1rY='} 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001036aee4ab4dcdd4-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af711cf498863d52-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj55 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj55 2024-04-20T01:55:26.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj55 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj55 2024-04-20T01:55:26.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:R+ujjuo+5K5FfUbfpPIl0FIwGmM= 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:R+ujjuo+5K5FfUbfpPIl0FIwGmM='} 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5acb03d421de65a-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-77 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-77 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 5 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.878 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aafcde9f53067389-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj17 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj17 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj17 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj17 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:n+4ekqLfX12nxL+Y2AmfFP/9gRU= 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:n+4ekqLfX12nxL+Y2AmfFP/9gRU='} 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f1d24b7a8378939-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005d48b13729b75c4-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj34 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj34 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj34 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj34 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:pFImoluXp0un0ham5wLxZZHa65U= 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:pFImoluXp0un0ham5wLxZZHa65U='} 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000932ed2420122ef97-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.884 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d39ee21979ba20d-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj51 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj51 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj51 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.885 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj51 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2RsQ0TCVn69+zBqMKm+kgA46FDI= 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2RsQ0TCVn69+zBqMKm+kgA46FDI='} 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a7c6c6a5b71ed59f-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:55:26.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d91d52aec3d61e1-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj68 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj68 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj68 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.888 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj68 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Q9bGpwtlvzHDQLOJfuev9Se2iTw= 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Q9bGpwtlvzHDQLOJfuev9Se2iTw='} 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a1beab8d52ee0e8-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A= 2024-04-20T01:55:26.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:tGB6GvxlPx1ObQU1eyfAaFluo8A=', 'Content-Length': '0'} 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dedfb2996511a1f8-00662318bd-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj85 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj85 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj85 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.891 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:05 GMT 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj85 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:CkE9SgEa5K2ATNh6mgr+pWhDBNY= 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:CkE9SgEa5K2ATNh6mgr+pWhDBNY='} 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc68576d5ae91c99-00662318bd-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-77 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-77 2024-04-20T01:55:26.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 10 2024-04-20T01:55:26.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.894 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.894 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.894 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.894 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.894 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.894 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0d10758f3311b39-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj19 2024-04-20T01:55:26.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj19 2024-04-20T01:55:26.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj19 2024-04-20T01:55:26.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.896 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.896 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.896 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.896 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.896 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj19 2024-04-20T01:55:26.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sYJWYO+YpEssECkjJyXyE6ELfRY= 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sYJWYO+YpEssECkjJyXyE6ELfRY='} 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009774f7f4a2b472b8-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a8945892a453c2e-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj38 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj38 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj38 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj38 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.899 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Bk5bcJrPgArD639If8FAWafZqHc= 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Bk5bcJrPgArD639If8FAWafZqHc='} 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d61c87957da41dd3-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.900 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e9445b5f6890c18-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj57 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj57 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj57 2024-04-20T01:55:26.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj57 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.902 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8+5gxWt5sWXqDbODXZkbuPLrqMQ= 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8+5gxWt5sWXqDbODXZkbuPLrqMQ='} 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2404ddf6e73abba-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.903 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000648858bad4d2c4c5-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj76 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj76 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj76 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj76 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:x3+7SrPn5/b+ozDyxy7i1aj7nUg= 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:x3+7SrPn5/b+ozDyxy7i1aj7nUg='} 2024-04-20T01:55:26.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000106106536248e116-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.906 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006781492c13d25d21-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj95 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj95 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj95 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj95 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:3NIucZo6xpHJzK5SySKbO1EIWEA= 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:3NIucZo6xpHJzK5SySKbO1EIWEA='} 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2eefe48653813b8-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-77 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-77 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 15 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046c4a1fe13bd1c9b-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj23 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj23 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj23 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.911 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj23 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:JoY9ZlIk0pAnC/Duzbr2tBLb6Mw= 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JoY9ZlIk0pAnC/Duzbr2tBLb6Mw='} 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd2a1b745a3bf01c-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba7fe11ed92cbc73-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj46 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj46 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj46 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.914 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj46 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:68ni2RjKPeecMy//tE4PJHdvp/Q= 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:68ni2RjKPeecMy//tE4PJHdvp/Q='} 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008032d50011b2e10a-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aae973c35dbb3d83-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj69 2024-04-20T01:55:26.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj69 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj69 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.917 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj69 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eXXWr8Wus+ucvtXw2kw71fBFuCE= 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eXXWr8Wus+ucvtXw2kw71fBFuCE='} 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003577b5d2b6a64222-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7d8a7360233c160-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj92 2024-04-20T01:55:26.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj92 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj92 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.920 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj92 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:egv4Y6GyyEPafTkalrYW3HdfzfQ= 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:egv4Y6GyyEPafTkalrYW3HdfzfQ='} 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b4cf12743a756c2-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1a584cd8cd04f4f-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj115 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj115 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj115 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj115 2024-04-20T01:55:26.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:dm5QwWzqkVo/Ku/3bInNeQZLoqk= 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:dm5QwWzqkVo/Ku/3bInNeQZLoqk='} 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cdbf9a508564a01-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-77 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-77 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 20 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000633aa38930a40ea0-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj29 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj29 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj29 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj29 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:me/ELblCV3EVtEGcTU/F0kLOkMY= 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:me/ELblCV3EVtEGcTU/F0kLOkMY='} 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a19700d2fb2bfd4-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.928 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070570624814e2f86-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj58 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj58 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj58 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.929 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj58 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:273MvvHPtzZJ+cKSpM2IWi9gxL8= 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:273MvvHPtzZJ+cKSpM2IWi9gxL8='} 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009fddde84bf028f5-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ad52cc5f788f277-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj87 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj87 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj87 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.932 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj87 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Z2pLyUu1yzS4b38Q90IIX6Wk0hY= 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Z2pLyUu1yzS4b38Q90IIX6Wk0hY='} 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a44d740a20c75e9c-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cda15536ff443e1-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj116 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj116 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj116 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.935 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj116 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cev/2EYsoAificmW9hLNBhLIZAs= 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cev/2EYsoAificmW9hLNBhLIZAs='} 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000656eda1f0baa9a3f-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8= 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:rll5tXJgaxs/pUDtm5mHkBo+Rz8=', 'Content-Length': '0'} 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf22fb66a6c53205-00662318be-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj145 2024-04-20T01:55:26.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj145 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj145 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:06 GMT 2024-04-20T01:55:26.938 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj145 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:athWNQtFoCHcCZctAGgoYy+/c3A= 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:athWNQtFoCHcCZctAGgoYy+/c3A='} 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c62dba12af93b14-00662318be-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-77 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-77 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 25 2024-04-20T01:55:26.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.940 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050d7828d261b3d46-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj31 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj31 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj31 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj31 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:nGMmORNhMjkneyLFVsvyapQj+FA= 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:nGMmORNhMjkneyLFVsvyapQj+FA='} 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002755e69c53d8b2a-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.943 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000749a33231cbf70f6-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj62 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj62 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj62 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj62 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:0Csf2+CcYtSamZ0s9Fhe8KYPCe4= 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:0Csf2+CcYtSamZ0s9Fhe8KYPCe4='} 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4341f1e43006059-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d432265f003dbcd-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj93 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj93 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj93 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj93 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6ICjO7vvH418IxxsJArrV2pIewY= 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6ICjO7vvH418IxxsJArrV2pIewY='} 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbeb1c11e156c545-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:55:26.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068968a0d442a4247-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj124 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj124 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj124 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.950 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj124 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WLPZeRBlQD9/p8WT6WLNV93FHNY= 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WLPZeRBlQD9/p8WT6WLNV93FHNY='} 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f504494284be022e-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:55:26.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c0bc78182814b20-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj155 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj155 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj155 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.953 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj155 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:z83ovdzPz95akeex35wVIFpa2hQ= 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:z83ovdzPz95akeex35wVIFpa2hQ='} 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd1cd3568a7dbd62-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'aktlji-77', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket aktlji-77 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: tenant: 2024-04-20T01:55:26.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: bucket name: aktlji-77 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: total entries: 30 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.955 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff9f4f2ae993bbc4-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj37 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj37 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj37 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj37 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:mpyUizAZHDnGRlllniIK8tLIaLU= 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mpyUizAZHDnGRlllniIK8tLIaLU='} 2024-04-20T01:55:26.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000766c155b0a2d7f04-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.958 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b594fe27bb352db-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj74 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj74 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj74 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj74 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WahL01DHylbkLPNECsYb7lhsP0U= 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WahL01DHylbkLPNECsYb7lhsP0U='} 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4fa4b01c4bf2ff0-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.961 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fbcc92f07874180-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj111 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj111 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj111 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj111 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:XFVLOkyXaeNCWt7c3iyza0UoqvI= 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:XFVLOkyXaeNCWt7c3iyza0UoqvI='} 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0eca5fe5d5c4bcc-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.964 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ec46cef8886beff-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj148 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj148 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj148 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj148 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:IzUU5PEnHUi12I0/I+fjsNO0cok= 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IzUU5PEnHUi12I0/I+fjsNO0cok='} 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e379fdfb2e53dfd-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/ 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/ 2024-04-20T01:55:26.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/ 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.967 INFO:tasks.rgw_multisite_tests:/aktlji-77/ 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak= 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QVKV2wT1ud+dKuZGRCdP9V2zOak=', 'Content-Length': '0'} 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085ea693e299f7ad3-00662318bf-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-77/obj185 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-77/obj185 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-77/obj185 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:22:07 GMT 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:/aktlji-77/obj185 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ROmZMhld595jsC1yZgbhWyv0Uzc= 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:22:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ROmZMhld595jsC1yZgbhWyv0Uzc='} 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000fa00ce4eadd3adf-00662318bf-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:22:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-77 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-77 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-77 2024-04-20T01:55:26.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-77 2024-04-20T01:55:26.970 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.970 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-77', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-77 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:08.054+0000 7f6b17288780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:13.058+0000 7f6b17288780 1 bucket sync caught up with source: 2024-04-20T01:55:26.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [, , , , , , 00000000002.71.6, , , , , , 00000000002.84.6, , , , , 00000000002.123.6, , , , , , , , , , , , , , , 00000000001.68.6, , , , 00000000001.114.6] 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [, , , , , , 00000000002.71.6, , , , , , 00000000002.84.6, , , , , 00000000002.123.6, , , , , , , , , , , , , , , 00000000001.68.6, , , , 00000000001.114.6] 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:13.058+0000 7f6b17288780 0 bucket checkpoint complete 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-77 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-77', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-77', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-77 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:13.207+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:18.208+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:23.209+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:28.209+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:33.210+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:38.211+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:43.211+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:48.212+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:53.214+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:22:58.214+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:23:03.215+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:23:08.216+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:23:13.216+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:23:18.217+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:23:23.217+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:23:28.217+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:23:33.217+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:23:38.218+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:23:43.219+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:23:48.219+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:23:53.220+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:23:58.220+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:24:03.221+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:24:08.221+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:24:13.222+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:24:18.225+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:24:23.226+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:24:28.226+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:24:33.227+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:24:38.227+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:24:43.229+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:24:48.229+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:24:53.230+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:24:58.231+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:25:03.231+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:25:08.232+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:25:13.232+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:25:18.232+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:25:23.232+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:25:28.233+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:25:33.234+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:25:38.234+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:25:43.235+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:25:48.235+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:25:53.236+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:25:58.237+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:26:03.237+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:26:08.238+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:26:13.239+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:26:18.239+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:26:23.240+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:26:28.240+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:26:33.242+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:26:38.242+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:26:43.243+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:26:48.244+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:26:53.244+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:26:58.245+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:03.245+0000 7f24e241b780 1 waiting to reach incremental sync.. 2024-04-20T01:55:26.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:08.245+0000 7f24e241b780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:26.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:08.245+0000 7f24e241b780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:08.245+0000 7f24e241b780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:26.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:26.976 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:26.976 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.976 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:26.976 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-04-20T01:55:26.976 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:26.976 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 1915, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:26.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-78/ 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-78/ 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-78/ 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.978 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:27:08 GMT 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:/aktlji-78/ 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qx7VaSXgTN+wdroLMo6uWANwbuc= 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:27:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qx7VaSXgTN+wdroLMo6uWANwbuc=', 'Content-Length': '0'} 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000471c281676a8698a-00662319ec-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:27:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=aktlji-78 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:27:08.292227Z" 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:26.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.981 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.981 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:27:08.539+0000 7fbebd8b0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:27:08.540+0000 7fbebd8b0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:26.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:27:08Z" 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.986 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.987 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:55:26.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.990 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:27:08Z" 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:08.665+0000 7ff4b7b73780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:08.666+0000 7ff4b7b73780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:26.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713576127.522556_1271.1", 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:22:07.522556Z", 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:27:08Z" 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.997 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.998 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:26.999 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576127.522556_1271.1", 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:22:07.522556Z", 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.001 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:27:08Z" 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576127.522556_1271.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:22:07.522556Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576127.522556_1271.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1713576428.292227_1338.1 target=1_1713576127.522556_1271.1 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:13.799+0000 7f8009551780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:13.800+0000 7f8009551780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:27:13Z" 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.007 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.008 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576428.292227_1338.1", 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:08.292227Z", 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.010 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.011 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:27:13Z" 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576428.292227_1338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:08.292227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576428.292227_1338.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:27.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-78/ 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-78/ 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-78/ 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.013 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:27:13 GMT 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:/aktlji-78/ 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w= 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w=', 'Content-Length': '0'} 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079be9952398f4e78-00662319f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-78/obj11 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-78/obj11 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-78/obj11 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:27:13 GMT 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:/aktlji-78/obj11 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:MlmUsjsDdWXFO2oTQi4+/3Kw5VM= 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:MlmUsjsDdWXFO2oTQi4+/3Kw5VM='} 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af8877fd92dc2904-00662319f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-78/ 2024-04-20T01:55:27.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-78/ 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-78/ 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:27:13 GMT 2024-04-20T01:55:27.016 INFO:tasks.rgw_multisite_tests:/aktlji-78/ 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w= 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w=', 'Content-Length': '0'} 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7b32aff48cdf5f9-00662319f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-78/obj22 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-78/obj22 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-78/obj22 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:27:13 GMT 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:/aktlji-78/obj22 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AVFijVeEeD8RI7+K2TYHYlEIUiA= 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AVFijVeEeD8RI7+K2TYHYlEIUiA='} 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000123cabd8d17a0c15-00662319f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-78/ 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-78/ 2024-04-20T01:55:27.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-78/ 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:27:13 GMT 2024-04-20T01:55:27.019 INFO:tasks.rgw_multisite_tests:/aktlji-78/ 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w= 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w=', 'Content-Length': '0'} 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c1c64c127c58983-00662319f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-78/obj33 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-78/obj33 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-78/obj33 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:27:13 GMT 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:/aktlji-78/obj33 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:2oD0iX6ZTBKJhmLBfbgiHIH6apc= 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:2oD0iX6ZTBKJhmLBfbgiHIH6apc='} 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068ba0ab5f943ee46-00662319f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-78/ 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-78/ 2024-04-20T01:55:27.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-78/ 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:27:13 GMT 2024-04-20T01:55:27.022 INFO:tasks.rgw_multisite_tests:/aktlji-78/ 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w= 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w=', 'Content-Length': '0'} 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e96a1c3b45221b9c-00662319f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-78/obj44 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-78/obj44 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-78/obj44 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:27:13 GMT 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:/aktlji-78/obj44 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Pcz/KceXnp9ZkbzhogxskMt+9sk= 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Pcz/KceXnp9ZkbzhogxskMt+9sk='} 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c35bf85808ae2a50-00662319f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-78/ 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-78/ 2024-04-20T01:55:27.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-78/ 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:27:13 GMT 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests:/aktlji-78/ 2024-04-20T01:55:27.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w= 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:AqVkBUuEDQIJdazTW3XCDgkh87w=', 'Content-Length': '0'} 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f737f7f19a121006-00662319f1-15b6-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-78/obj55 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-78/obj55 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-78/obj55 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:27:13 GMT 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:/aktlji-78/obj55 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:JVhb7AQ6x/S4mq4iDRhpvKRHieQ= 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:27:13 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:JVhb7AQ6x/S4mq4iDRhpvKRHieQ='} 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ed9a86c8c1c40de-00662319f1-15b6-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:27:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone2 bucket=aktlji-78 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:27.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket aktlji-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: Sync is disabled for bucket aktlji-78 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone1 target=test-zone3 bucket=aktlji-78 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-78', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-78 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:14.205+0000 7f35a78ef780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:19.205+0000 7f35a78ef780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:24.206+0000 7f35a78ef780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:29.206+0000 7f35a78ef780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:34.208+0000 7f35a78ef780 1 bucket sync caught up with source: 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: local status: [, , , , , , , , 00000000001.77.6, 00000000002.53.6, 00000000002.110.6] 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: remote markers: [, , , , , , , , 00000000001.77.6, 00000000002.53.6, 00000000002.110.6] 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:34.208+0000 7f35a78ef780 0 bucket checkpoint complete 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=test-zone2 target=test-zone1 bucket=aktlji-78 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'aktlji-78', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket aktlji-78 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:34.345+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:39.346+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:44.347+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:49.347+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:54.348+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:27:59.349+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:28:04.349+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:28:09.350+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:28:14.350+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:28:19.351+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:28:24.352+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:28:29.353+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:28:34.354+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:28:39.354+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:28:44.355+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:28:49.355+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:28:54.356+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:28:59.356+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:29:04.356+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:29:09.357+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:29:14.357+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:29:19.358+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:29:24.359+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:29:29.359+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:29:34.360+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:29:39.365+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:29:44.365+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:29:49.366+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:29:54.367+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:29:59.367+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:30:04.368+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:30:09.369+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:30:14.370+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:30:19.370+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:30:24.371+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:30:29.372+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:30:34.372+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:30:39.375+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:30:44.374+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:30:49.375+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:30:54.376+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:30:59.376+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:31:04.377+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:31:09.377+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:31:14.378+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:31:19.378+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:31:24.379+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:31:29.379+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:31:34.380+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:31:39.381+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:31:44.382+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:31:49.383+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:31:54.383+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:31:59.385+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:32:04.386+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:32:09.387+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:32:14.387+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:32:19.388+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:32:24.388+0000 7f9301bd2780 1 waiting to reach incremental sync.. 2024-04-20T01:55:27.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:32:29.389+0000 7f9301bd2780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-20T01:55:27.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:32:29.389+0000 7f9301bd2780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:27.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:32:29.389+0000 7f9301bd2780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-20T01:55:27.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-20T01:55:27.034 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:27.034 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.034 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:27.034 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2024-04-20T01:55:27.035 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:27.035 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:27.036 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:27.036 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:27.036 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2213, in test_sync_flow_symmetrical_zonegroup_all 2024-04-20T01:55:27.036 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objnames[1], content) 2024-04-20T01:55:27.036 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-04-20T01:55:27.036 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-04-20T01:55:27.036 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-04-20T01:55:27.036 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:27.036 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.036 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:34:42.281271Z" 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:34:59.257+0000 7ff71a661780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:34:59.260+0000 7ff71a661780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:34:59Z" 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.046 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.046 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.046 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.046 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.046 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.046 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.046 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.046 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.046 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.046 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.047 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:34:59Z" 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.048 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:27.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:27.049 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.049 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:34:59.639+0000 7f892875d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:34:59.643+0000 7f892875d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:34:59Z" 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.055 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.056 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.058 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.058 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.058 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.058 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.058 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.058 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.058 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.058 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.058 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.058 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.058 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.058 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:34:59Z" 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.059 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[]} 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:27.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.062 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:27.062 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.062 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "epoch": 11, 2024-04-20T01:55:27.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:55:27.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": [], 2024-04-20T01:55:27.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period_map": { 2024-04-20T01:55:27.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zonegroups": [ 2024-04-20T01:55:27.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "test-zonegroup", 2024-04-20T01:55:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zonegroup", 2024-04-20T01:55:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "api_name": "", 2024-04-20T01:55:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "is_master": true, 2024-04-20T01:55:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:55:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "hostnames": [], 2024-04-20T01:55:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "hostnames_s3website": [], 2024-04-20T01:55:27.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone2", 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": false, 2024-04-20T01:55:27.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone1", 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": true, 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone3", 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": true, 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:27.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "placement_targets": [ 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "default-placement", 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [], 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "storage_classes": [ 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "STANDARD" 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "default_placement": "default-placement", 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_policy": { 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "groups": [ 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-group", 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_flow": { 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "symmetrical": [ 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-flow1", 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pipes": [ 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-pipe", 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "*", 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": { 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "*", 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "params": { 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "filter": { 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [] 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": {}, 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "priority": 0, 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mode": "system", 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user": "" 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "enabled" 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled_features": [ 2024-04-20T01:55:27.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "short_zone_ids": [ 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 2010863401 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 1221149675 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 1478903101 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zonegroup": "test-zonegroup", 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period_config": { 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_quota": { 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false, 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "check_on_raw": false, 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size": -1, 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size_kb": 0, 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_objects": -1 2024-04-20T01:55:27.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user_quota": { 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false, 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "check_on_raw": false, 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size": -1, 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size_kb": 0, 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_objects": -1 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user_ratelimit": { 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:27.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_ratelimit": { 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "anonymous_ratelimit": { 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:27.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_name": "test-realm", 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:27.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-80 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-80/ 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-80/ 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-80/ 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.081 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:35:00 GMT 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests:/aktlji-80/ 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zHY8sgPC+DI1hitUnsdcb57Lo1w= 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zHY8sgPC+DI1hitUnsdcb57Lo1w=', 'Content-Length': '0'} 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f59b5f2b80a09323-0066231bc4-1a14-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-80/ 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-80/ 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-80/ 2024-04-20T01:55:27.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:35:00 GMT 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests:/aktlji-80/ 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fAFCTFM4J34pdIUZpc84TECl3iw= 2024-04-20T01:55:27.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fAFCTFM4J34pdIUZpc84TECl3iw=', 'Content-Length': '0'} 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000485dba4c616e5070-0066231bc4-1a14-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-80/obj1 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-80/obj1 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-80/obj1 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.084 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:35:00 GMT 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:/aktlji-80/obj1 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:a6277L6S+qIjU7QVC4K1REkSw2A= 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a6277L6S+qIjU7QVC4K1REkSw2A='} 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c554f60784784f91-0066231bc4-1a14-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:35:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=aktlji-81 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-81/ 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-81/ 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-81/ 2024-04-20T01:55:27.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:35:00 GMT 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests:/aktlji-81/ 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.086 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:IRVvqCy6B104OvfM8472GTwhsSc= 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:35:00 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:IRVvqCy6B104OvfM8472GTwhsSc=', 'Content-Length': '0'} 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000000f010139d9eb59ce-0066231bc4-214b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:35:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-81/ 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-81/ 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-81/ 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.087 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:35:01 GMT 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests:/aktlji-81/ 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fdpx45D434IPtURLvtz2oxM3lqs= 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:35:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fdpx45D434IPtURLvtz2oxM3lqs=', 'Content-Length': '0'} 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084b2fd1df345d854-0066231bc5-214b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:35:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-81/obj2 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-81/obj2 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-81/obj2 2024-04-20T01:55:27.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:35:01 GMT 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:/aktlji-81/obj2 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:o6lT7M6/Il+i86f+6ub2qoh4a1U= 2024-04-20T01:55:27.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:35:01 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o6lT7M6/Il+i86f+6ub2qoh4a1U='} 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000677ee53ed9fef1f7-0066231bc5-214b-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:35:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:35:01.005366Z" 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:01.431+0000 7fef99dbf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:01.432+0000 7fef99dbf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:35:01Z" 2024-04-20T01:55:27.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.097 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.097 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.097 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.097 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.097 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.098 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.098 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.098 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.098 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.098 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.098 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.098 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.098 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.099 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.099 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.099 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.099 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.099 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.099 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.099 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.099 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.099 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.099 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.100 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.101 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:35:01Z" 2024-04-20T01:55:27.102 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:06.588+0000 7f10575ac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:06.589+0000 7f10575ac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:35:06Z" 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.108 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.109 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.110 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.111 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.112 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:35:06Z" 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:11.734+0000 7f7853221780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:11.735+0000 7f7853221780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:35:11Z" 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.119 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.120 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.121 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.122 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:35:11Z" 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.123 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:16.874+0000 7f2b88432780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:16.875+0000 7f2b88432780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:35:16Z" 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.129 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.130 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.131 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.132 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.133 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:35:16Z" 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:22.015+0000 7f8f0446e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:22.016+0000 7f8f0446e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:35:22Z" 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.140 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.141 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.142 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.143 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:35:22Z" 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.144 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:27.152+0000 7f0d7c734780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:27.154+0000 7f0d7c734780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:35:27Z" 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.151 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.152 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.153 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.154 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:35:27Z" 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:32.292+0000 7fbd8f54f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:32.293+0000 7fbd8f54f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:35:32Z" 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.161 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.162 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.163 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.164 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:35:32Z" 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:37.432+0000 7f0524492780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:37.434+0000 7f0524492780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:35:37Z" 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.171 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.172 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.173 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.174 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.175 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:35:37Z" 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:42.568+0000 7fae5901b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:42.569+0000 7fae5901b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:35:42Z" 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.183 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.184 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.185 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:35:42Z" 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:47.716+0000 7f221ed32780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:47.717+0000 7f221ed32780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:35:47Z" 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.192 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.193 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.194 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.195 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.196 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:35:47Z" 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:52.851+0000 7f94a7271780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:52.852+0000 7f94a7271780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:35:52Z" 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.203 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.204 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.205 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:35:52Z" 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:57.991+0000 7f0825f98780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:35:57.992+0000 7f0825f98780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:35:57Z" 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.214 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.215 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.215 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.215 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.215 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.215 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.215 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.215 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.215 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.215 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.216 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.217 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:35:57Z" 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:03.133+0000 7f799a9fa780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:03.135+0000 7f799a9fa780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:36:03Z" 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.224 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.225 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.226 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:03Z" 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:08.272+0000 7f2c78fac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:08.274+0000 7f2c78fac780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:36:08Z" 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.234 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.235 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.236 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.237 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.238 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:08Z" 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:13.410+0000 7f7abf470780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:13.411+0000 7f7abf470780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:36:13Z" 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.244 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.245 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.248 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:13Z" 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:18.562+0000 7f5759bf9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:18.563+0000 7f5759bf9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:36:18Z" 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.255 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.256 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.258 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:18Z" 2024-04-20T01:55:27.259 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:23.696+0000 7f925dd53780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:23.697+0000 7f925dd53780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:36:23Z" 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.265 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.266 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.267 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.267 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.267 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.267 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.267 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.267 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.267 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.267 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.267 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.267 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.267 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.267 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.270 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.270 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.270 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.270 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.270 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.270 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.270 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.270 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.271 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:23Z" 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.272 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:28.831+0000 7ff907d2d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:28.832+0000 7ff907d2d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:36:28Z" 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.278 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.279 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.280 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.281 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.282 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:28Z" 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:33.971+0000 7f3ec43d8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:33.972+0000 7f3ec43d8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:36:33Z" 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.289 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.290 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.291 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.292 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:33Z" 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.294 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.294 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:39.111+0000 7f9c5d388780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:39.113+0000 7f9c5d388780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:36:39Z" 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.299 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.301 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.302 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.303 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:39Z" 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:44.254+0000 7f369a4b6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:44.255+0000 7f369a4b6780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:36:44Z" 2024-04-20T01:55:27.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.310 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.312 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.313 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:44Z" 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:49.405+0000 7faa9e96d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:49.406+0000 7faa9e96d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:36:49Z" 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.320 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.321 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.322 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576882.281271_1460.1", 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:34:42.281271Z", 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.324 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:49Z" 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576882.281271_1460.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:34:42.281271Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576882.281271_1460.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713576901.005366_1501.1 target=1_1713576882.281271_1460.1 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:54.555+0000 7fcdafd30780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:36:54.556+0000 7fcdafd30780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:36:54Z" 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.330 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.331 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.333 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.334 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:54Z" 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:27.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:36:54.691+0000 7f37d73b4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:36:54.692+0000 7f37d73b4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:36:54Z" 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.341 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.342 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.343 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.344 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.345 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:54Z" 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000026070", 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:35:00.951408Z" 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:55:27.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000025929", 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:27.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:13.914481Z" 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:36:54Z" 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.353 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.354 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000025929", 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:13.914481Z" 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.355 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:36:54Z" 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014922', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:22:09.862093Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025929', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:27:13.914481Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.871854Z'}}] 2024-04-20T01:55:27.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:55:27.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000025929', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:55:27.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026070 target=00000000000000000000:00000000000000025929 2024-04-20T01:55:27.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000025929", 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:13.914481Z" 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:00Z" 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.363 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000025929", 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.364 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:13.914481Z" 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.365 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.366 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:00Z" 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014922', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:22:09.862093Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025929', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:27:13.914481Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.871854Z'}}] 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000025929', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000026070 target=00000000000000000000:00000000000000025929 2024-04-20T01:55:27.367 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000026070", 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:00.951408Z" 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:55:27.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:05Z" 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.372 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.373 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000026070", 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:00.951408Z" 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.374 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.375 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.376 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:05Z" 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014922', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:22:09.862093Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026070', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:35:00.951408Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.871854Z'}}] 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-80/ 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-80/ 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-80/ 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:05 GMT 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests:/aktlji-80/ 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:oFQfFoM/QoG6tZRnSKSucYeFfVk= 2024-04-20T01:55:27.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:oFQfFoM/QoG6tZRnSKSucYeFfVk=', 'Content-Length': '0'} 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052c52808507b4774-0066231c41-214b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-80/obj1 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-80/obj1 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-80/obj1 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.379 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:05 GMT 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests:/aktlji-80/obj1 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:W66jrqJ3cFuGMlA6GELsnxV77wY= 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W66jrqJ3cFuGMlA6GELsnxV77wY=', 'Content-Length': '0'} 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:35:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-80:73f46f02-c7ab-4426-a267-2273af911da5.6678.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000798d6acc008907e4-0066231c41-214b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:37:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-80/obj1 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-80/obj1 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-80/obj1 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:05 GMT 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests:/aktlji-80/obj1 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:f0kgrI8agX9jhDEhF0cls0mHl+Q= 2024-04-20T01:55:27.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:f0kgrI8agX9jhDEhF0cls0mHl+Q=', 'Content-Length': '0'} 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:35:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-80:73f46f02-c7ab-4426-a267-2273af911da5.6678.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bf59d6e04909330-0066231c41-214b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:37:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-81/ 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-81/ 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-81/ 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:05 GMT 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests:/aktlji-81/ 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cfSw4saMrlsfyUzZIWzITsBdp2g= 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cfSw4saMrlsfyUzZIWzITsBdp2g=', 'Content-Length': '0'} 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035832d1b9800d6a2-0066231c41-1a14-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-81/obj2 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-81/obj2 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-81/obj2 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:05 GMT 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests:/aktlji-81/obj2 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.384 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:M2KWg98T3TQa8SKdr+K5trydlOk= 2024-04-20T01:55:27.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:05 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:M2KWg98T3TQa8SKdr+K5trydlOk=', 'Content-Length': '0'} 2024-04-20T01:55:27.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d82829a508eee6e7-0066231c41-1a14-test-zone1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:37:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.385 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:27.385 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.385 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:27.385 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_select: 2024-04-20T01:55:27.385 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:27.386 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:27.386 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:27.386 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2271, in test_sync_flow_symmetrical_zonegroup_select 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesA, content) 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2080, in check_objects_exist 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:35:01.005366Z" 2024-04-20T01:55:27.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:05.567+0000 7f55d1841780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:05.568+0000 7f55d1841780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:05Z" 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.396 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.397 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.398 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:05Z" 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:37:05.698+0000 7f8499774780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:37:05.699+0000 7f8499774780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:27.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:37:05Z" 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.405 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.406 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.408 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.409 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:05Z" 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1,test-zone2', '--dest-zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1,test-zone2 --dest-zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "epoch": 12, 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": [], 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period_map": { 2024-04-20T01:55:27.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zonegroups": [ 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "test-zonegroup", 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zonegroup", 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "api_name": "", 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "is_master": true, 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "hostnames": [], 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "hostnames_s3website": [], 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone2", 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": false, 2024-04-20T01:55:27.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone1", 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:27.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": true, 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:27.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone3", 2024-04-20T01:55:27.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:27.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:55:27.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:27.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:27.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:27.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:27.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:27.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": true, 2024-04-20T01:55:27.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:27.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:27.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "placement_targets": [ 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "default-placement", 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [], 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "storage_classes": [ 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "STANDARD" 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "default_placement": "default-placement", 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_policy": { 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "groups": [ 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-group", 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_flow": { 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "symmetrical": [ 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-flow1", 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-flow", 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pipes": [ 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-pipe", 2024-04-20T01:55:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "*", 2024-04-20T01:55:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": { 2024-04-20T01:55:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "*", 2024-04-20T01:55:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "params": { 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "filter": { 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [] 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": {}, 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "priority": 0, 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mode": "system", 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user": "" 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "enabled" 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled_features": [ 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "short_zone_ids": [ 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 2010863401 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 1221149675 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 1478903101 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zonegroup": "test-zonegroup", 2024-04-20T01:55:27.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period_config": { 2024-04-20T01:55:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_quota": { 2024-04-20T01:55:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false, 2024-04-20T01:55:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "check_on_raw": false, 2024-04-20T01:55:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size": -1, 2024-04-20T01:55:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size_kb": 0, 2024-04-20T01:55:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_objects": -1 2024-04-20T01:55:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user_quota": { 2024-04-20T01:55:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false, 2024-04-20T01:55:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "check_on_raw": false, 2024-04-20T01:55:27.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size": -1, 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size_kb": 0, 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_objects": -1 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user_ratelimit": { 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_ratelimit": { 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:27.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "anonymous_ratelimit": { 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:27.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_name": "test-realm", 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-82 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/ 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/ 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/ 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:06 GMT 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests:/aktlji-82/ 2024-04-20T01:55:27.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:PvV6lZimwM3UDDQJ5+vQ41zKrcw= 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PvV6lZimwM3UDDQJ5+vQ41zKrcw=', 'Content-Length': '0'} 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000000eea9ec4df1104e5-0066231c42-1a85-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/ 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/ 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/ 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:06 GMT 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests:/aktlji-82/ 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WyLsm31kewzKEDbXYC63OakHPuE= 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WyLsm31kewzKEDbXYC63OakHPuE=', 'Content-Length': '0'} 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f398f607b044242-0066231c42-1a85-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/obj1 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/obj1 2024-04-20T01:55:27.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/obj1 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:06 GMT 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:/aktlji-82/obj1 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.433 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:EhNkmXd/AhCe96ROE0B2NdByLog= 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:EhNkmXd/AhCe96ROE0B2NdByLog='} 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096cba567cc846e48-0066231c42-1a85-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/ 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/ 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/ 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:06 GMT 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests:/aktlji-82/ 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WyLsm31kewzKEDbXYC63OakHPuE= 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WyLsm31kewzKEDbXYC63OakHPuE=', 'Content-Length': '0'} 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008214e85accf3caa1-0066231c42-1a85-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/obj2 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/obj2 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/obj2 2024-04-20T01:55:27.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:06 GMT 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:/aktlji-82/obj2 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.436 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/sp3wPG7jXdhf1rR5YyhvW2Q/II= 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/sp3wPG7jXdhf1rR5YyhvW2Q/II='} 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7edf268edd79dcc-0066231c42-1a85-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/ 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/ 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/ 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:06 GMT 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests:/aktlji-82/ 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WyLsm31kewzKEDbXYC63OakHPuE= 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WyLsm31kewzKEDbXYC63OakHPuE=', 'Content-Length': '0'} 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009554b0e201012c0b-0066231c42-1a85-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/obj3 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/obj3 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/obj3 2024-04-20T01:55:27.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.439 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.439 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.439 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.439 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:06 GMT 2024-04-20T01:55:27.439 INFO:tasks.rgw_multisite_tests:/aktlji-82/obj3 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BlXPxbgEjI8rQ9s0OAHLSUiu1GQ= 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BlXPxbgEjI8rQ9s0OAHLSUiu1GQ='} 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfc93fe32be1584c-0066231c42-1a85-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=aktlji-83 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-83/ 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-83/ 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-83/ 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:06 GMT 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests:/aktlji-83/ 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ifjdD3+ka7ykU8NHMnrFep1TMAI= 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:06 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ifjdD3+ka7ykU8NHMnrFep1TMAI=', 'Content-Length': '0'} 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000004d3a7f76c223d190-0066231c42-2227-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-83/ 2024-04-20T01:55:27.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-83/ 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-83/ 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.442 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:07 GMT 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:/aktlji-83/ 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:mTNBRxj2ItidhuAaSB6piL04svs= 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mTNBRxj2ItidhuAaSB6piL04svs=', 'Content-Length': '0'} 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003500419773a435f4-0066231c43-2227-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-83/obj4 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-83/obj4 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-83/obj4 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:07 GMT 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:/aktlji-83/obj4 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:8FYp9nEIhRT8/6bg4kpvDoixZMg= 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:37:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:8FYp9nEIhRT8/6bg4kpvDoixZMg='} 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c01d2380719a188b-0066231c43-2227-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-83/ 2024-04-20T01:55:27.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-83/ 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-83/ 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:07 GMT 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:/aktlji-83/ 2024-04-20T01:55:27.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.446 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:mTNBRxj2ItidhuAaSB6piL04svs= 2024-04-20T01:55:27.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mTNBRxj2ItidhuAaSB6piL04svs=', 'Content-Length': '0'} 2024-04-20T01:55:27.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068106d727cc2ed95-0066231c43-2227-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-83/obj5 2024-04-20T01:55:27.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-83/obj5 2024-04-20T01:55:27.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-83/obj5 2024-04-20T01:55:27.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:07 GMT 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:/aktlji-83/obj5 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BAtkieSUp9xADodimn09ZYBlZcM= 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:37:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BAtkieSUp9xADodimn09ZYBlZcM='} 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea0435e9ccd1d42b-0066231c43-2227-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-83/ 2024-04-20T01:55:27.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-83/ 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-83/ 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:07 GMT 2024-04-20T01:55:27.448 INFO:tasks.rgw_multisite_tests:/aktlji-83/ 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:mTNBRxj2ItidhuAaSB6piL04svs= 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:37:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mTNBRxj2ItidhuAaSB6piL04svs=', 'Content-Length': '0'} 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab86712260c4189a-0066231c43-2227-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-83/obj6 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-83/obj6 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-83/obj6 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:37:07 GMT 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:/aktlji-83/obj6 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:SuaB4VyEfPVj9zzzGzUwSiYhpnc= 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:37:07 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:SuaB4VyEfPVj9zzzGzUwSiYhpnc='} 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002cba9e03ecce6a55-0066231c43-2227-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:37:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:27.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:37:07.057559Z" 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.452 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:07.513+0000 7f5a99788780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:07.514+0000 7f5a99788780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:07Z" 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.458 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.459 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.460 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.461 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.462 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:07Z" 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:12.664+0000 7fba0a682780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:12.665+0000 7fba0a682780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:12Z" 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.468 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.469 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.470 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.471 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.472 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.472 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.472 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.472 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.472 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.472 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.472 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:12Z" 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.474 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:17.806+0000 7f18ceb8d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:17.807+0000 7f18ceb8d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:17Z" 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.482 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.483 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.484 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.485 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.486 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:17Z" 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:22.949+0000 7f969a957780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:22.950+0000 7f969a957780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:22Z" 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.493 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.494 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.495 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.496 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.497 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.498 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:22Z" 2024-04-20T01:55:27.498 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.498 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.498 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.498 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:28.088+0000 7f3cc0634780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:28.089+0000 7f3cc0634780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:28Z" 2024-04-20T01:55:27.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.504 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.504 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.504 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.504 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.504 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.504 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.504 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.504 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.504 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.506 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.507 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:28Z" 2024-04-20T01:55:27.508 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:33.222+0000 7f0c9b749780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:33.224+0000 7f0c9b749780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:33Z" 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.514 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.515 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.516 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.517 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.518 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:33Z" 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:38.368+0000 7f66dce6e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:38.369+0000 7f66dce6e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:38Z" 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.525 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.526 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.527 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.528 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.529 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:38Z" 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:43.509+0000 7fe131f3d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:43.510+0000 7fe131f3d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:43Z" 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.536 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.537 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.538 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.539 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:43Z" 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.540 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:48.643+0000 7fe44ae04780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:48.644+0000 7fe44ae04780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:48Z" 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.546 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.548 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.549 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.550 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:48Z" 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:53.802+0000 7fc436448780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:53.803+0000 7fc436448780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:53Z" 2024-04-20T01:55:27.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.557 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.558 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.559 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.560 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:53Z" 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:58.946+0000 7f27db4ec780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:37:58.948+0000 7f27db4ec780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:37:58Z" 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.567 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.568 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.569 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.570 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.571 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:37:58Z" 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:04.077+0000 7f4f1d356780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:04.078+0000 7f4f1d356780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:38:04Z" 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.577 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.578 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.579 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.581 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:38:04Z" 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:09.226+0000 7fba4fe30780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:09.226+0000 7fba4fe30780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:38:09Z" 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.588 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.589 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.590 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.591 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.592 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:38:09Z" 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:14.353+0000 7f44fd931780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:14.355+0000 7f44fd931780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:38:14Z" 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.599 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.600 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.600 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.600 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.600 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.600 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.600 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.600 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.600 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.600 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.600 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.601 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.602 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.603 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:38:14Z" 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:19.499+0000 7fe60cd3a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:19.500+0000 7fe60cd3a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:38:19Z" 2024-04-20T01:55:27.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.610 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.610 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.610 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.610 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.610 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.610 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.610 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.610 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.610 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.610 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.613 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:38:19Z" 2024-04-20T01:55:27.614 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:24.629+0000 7f7dfb766780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:24.630+0000 7f7dfb766780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:38:24Z" 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.620 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.621 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.622 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.623 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.624 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:38:24Z" 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:29.757+0000 7f9117763780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:29.758+0000 7f9117763780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:38:29Z" 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.631 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.632 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:38:29Z" 2024-04-20T01:55:27.635 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:34.897+0000 7f37474a0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:34.898+0000 7f37474a0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:38:34Z" 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.641 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.642 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.643 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.644 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.645 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:38:34Z" 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:40.041+0000 7f3d55bc1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:40.042+0000 7f3d55bc1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:38:40Z" 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.652 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.655 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:38:40Z" 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.656 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.657 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:45.190+0000 7f1acd257780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:45.191+0000 7f1acd257780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:38:45Z" 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.666 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:38:45Z" 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:50.328+0000 7fc39b76c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:50.329+0000 7fc39b76c780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:38:50Z" 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.673 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.674 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.675 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.675 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.675 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.675 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.675 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.675 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.675 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.675 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.678 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.678 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.678 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.678 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.678 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.678 INFO:tasks.rgw_multisite_tests: "marker": "1_1713576901.005366_1501.1", 2024-04-20T01:55:27.678 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.678 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.678 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:01.005366Z", 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.679 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:38:50Z" 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713576901.005366_1501.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:35:01.005366Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.680 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713576901.005366_1501.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713576901.005366_1501.1 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:55.466+0000 7f4b64680780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:38:55.467+0000 7f4b64680780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577026.786661_1526.1", 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:06.786661Z", 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:38:55Z" 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.686 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.687 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.688 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577026.786661_1526.1", 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:06.786661Z", 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.690 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:38:55Z" 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577026.786661_1526.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:06.786661Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577026.786661_1526.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577027.057559_1543.1 target=1_1713577026.786661_1526.1 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:00.595+0000 7f22e0f1b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:00.596+0000 7f22e0f1b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:39:00Z" 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.697 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.699 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.700 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:00Z" 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:39:00.722+0000 7fe27e7f8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:39:00.723+0000 7fe27e7f8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:39:00Z" 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.707 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.708 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.709 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.710 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.711 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:00Z" 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000020458", 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:38:37.839807Z" 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020458'} 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000026070", 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:00.951408Z" 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:27.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:39:01Z" 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.719 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000026070", 2024-04-20T01:55:27.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:00.951408Z" 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.721 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.722 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:01Z" 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014922', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:22:09.862093Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026070', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:35:00.951408Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.871854Z'}}] 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020458'} 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015063 target=00000000000000000000:00000000000000014922, shard=1 master=00000000000000000000:00000000000000026490 target=00000000000000000000:00000000000000026070, shard=3 master=00000000000000000000:00000000000000020458 target=00000000000000000000:00000000000000020176 2024-04-20T01:55:27.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000026070", 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:35:00.951408Z" 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:39:06Z" 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.728 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000014922", 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.729 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:22:09.862093Z" 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000026070", 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:35:00.951408Z" 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.730 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.731 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000020176", 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.871854Z" 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:27.732 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:06Z" 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014922', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:22:09.862093Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026070', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:35:00.951408Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.871854Z'}}] 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020458'} 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000014922', 1: '00000000000000000000:00000000000000026070', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020176'} 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015063 target=00000000000000000000:00000000000000014922, shard=1 master=00000000000000000000:00000000000000026490 target=00000000000000000000:00000000000000026070, shard=3 master=00000000000000000000:00000000000000020458 target=00000000000000000000:00000000000000020176 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000020458", 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.839807Z" 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:39:11Z" 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.738 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.739 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.740 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000020458", 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.839807Z" 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:11Z" 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015063', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.851769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026490', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.842528Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020458', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.839807Z'}}] 2024-04-20T01:55:27.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020458'} 2024-04-20T01:55:27.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020458'} 2024-04-20T01:55:27.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:27.743 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.743 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: [ 2024-04-20T01:55:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:55:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "last_update": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:55:27.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000012909", 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "last_update": "2024-04-20T01:39:08.004155Z" 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "last_update": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000008844", 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "last_update": "2024-04-20T01:39:08.044239Z" 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=test-zone2 markers={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008844'} 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "instance_id": 11239922052735367740 2024-04-20T01:55:27.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:37:07.140694Z" 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000008704", 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:37:07.190175Z" 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 0, 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 0 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:39:11Z" 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:27.749 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: "instance_id": 11239922052735367740 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.140694Z" 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.751 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.752 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008704", 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.190175Z" 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:27.753 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:11Z" 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.116484Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012769', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.140694Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012769', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:53.230970Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008704', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.190175Z'}}] 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008844'} 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012769', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008704'} 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone1 behind zone test-zone2: shard=1 master=00000000000000000000:00000000000000012909 target=00000000000000000000:00000000000000012769, shard=3 master=00000000000000000000:00000000000000008844 target=00000000000000000000:00000000000000008704 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "instance_id": 11239922052735367740 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:55:27.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:37:07.140694Z" 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000008704", 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:37:07.190175Z" 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 0, 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 0 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:39:16Z" 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests: "instance_id": 11239922052735367740 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.759 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.760 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.140694Z" 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008704", 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.190175Z" 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:16Z" 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.116484Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012769', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.140694Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012769', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:53.230970Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008704', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.190175Z'}}] 2024-04-20T01:55:27.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008844'} 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012769', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008704'} 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone1 behind zone test-zone2: shard=1 master=00000000000000000000:00000000000000012909 target=00000000000000000000:00000000000000012769, shard=3 master=00000000000000000000:00000000000000008844 target=00000000000000000000:00000000000000008704 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "instance_id": 11239922052735367740 2024-04-20T01:55:27.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000012909", 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:39:08.004155Z" 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000008704", 2024-04-20T01:55:27.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:37:07.190175Z" 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 0, 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 0 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:39:22Z" 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: "instance_id": 11239922052735367740 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:55:27.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000012909", 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:39:08.004155Z" 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.771 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008704", 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.190175Z" 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.772 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:22Z" 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.116484Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:08.004155Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012769', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:53.230970Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008704', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.190175Z'}}] 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008844'} 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008704'} 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone1 behind zone test-zone2: shard=3 master=00000000000000000000:00000000000000008844 target=00000000000000000000:00000000000000008704 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "instance_id": 11239922052735367740 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000012909", 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:39:08.004155Z" 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:55:27.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000008844", 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:39:08.044239Z" 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 0, 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 0 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:39:28Z" 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.778 INFO:tasks.rgw_multisite_tests: "instance_id": 11239922052735367740 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000007274", 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.116484Z" 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.779 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000012909", 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:39:08.004155Z" 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.780 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000012769", 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:32:53.230970Z" 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000008844", 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.781 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:39:08.044239Z" 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests: "total": 0, 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests: "complete": 0 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:28Z" 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.782 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007274', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.116484Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012909', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:08.004155Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012769', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:32:53.230970Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008844', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:08.044239Z'}}] 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008844'} 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000007274', 1: '00000000000000000000:00000000000000012909', 2: '00000000000000000000:00000000000000012769', 3: '00000000000000000000:00000000000000008844'} 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/ 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/ 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/ 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:28 GMT 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests:/aktlji-82/ 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:eovWTalBBgr6i/AWulGbvpwwm18= 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:eovWTalBBgr6i/AWulGbvpwwm18=', 'Content-Length': '0'} 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c5d3a7447dc8fdc-0066231cd0-2227-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/obj1 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/obj1 2024-04-20T01:55:27.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/obj1 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:28 GMT 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests:/aktlji-82/obj1 2024-04-20T01:55:27.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:wLd8AqGjDbuTv+1f9M2IBcGJYrg= 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wLd8AqGjDbuTv+1f9M2IBcGJYrg=', 'Content-Length': '0'} 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-82:73f46f02-c7ab-4426-a267-2273af911da5.6793.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098d87b99f5d5f0d6-0066231cd0-2227-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/obj1 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/obj1 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/obj1 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:28 GMT 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests:/aktlji-82/obj1 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:1tO/c646dPtzLxbcMWXh17HNUTI= 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:1tO/c646dPtzLxbcMWXh17HNUTI=', 'Content-Length': '0'} 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-82:73f46f02-c7ab-4426-a267-2273af911da5.6793.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007625753151cdf63d-0066231cd0-2227-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/obj2 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/obj2 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/obj2 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:28 GMT 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests:/aktlji-82/obj2 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.788 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/hX3pqwLf/yjrwXKCwBer7NhSvk= 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/hX3pqwLf/yjrwXKCwBer7NhSvk=', 'Content-Length': '0'} 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-82:73f46f02-c7ab-4426-a267-2273af911da5.6793.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d1900ceb4a58eed-0066231cd0-2227-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/obj2 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/obj2 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/obj2 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.789 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:28 GMT 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests:/aktlji-82/obj2 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:N7vF+RL++FlekOi9yOlLDrOrs7M= 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:N7vF+RL++FlekOi9yOlLDrOrs7M=', 'Content-Length': '0'} 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:37:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-82:73f46f02-c7ab-4426-a267-2273af911da5.6793.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef26ed010c8bccbb-0066231cd0-2227-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-82/obj3 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-82/obj3 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-82/obj3 2024-04-20T01:55:27.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:28 GMT 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests:/aktlji-82/obj3 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6hQMOL8+4wP3cXbr9AHvOb3qdrQ= 2024-04-20T01:55:27.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:28 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6hQMOL8+4wP3cXbr9AHvOb3qdrQ=', 'Content-Length': '0'} 2024-04-20T01:55:27.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000015d3e789bebfb61-0066231cd0-2227-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:39:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.792 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:27.792 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.792 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:27.792 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_directional_zonegroup_select: 2024-04-20T01:55:27.792 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:27.793 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:27.793 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:27.793 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:27.793 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2347, in test_sync_flow_directional_zonegroup_select 2024-04-20T01:55:27.793 INFO:tasks.rgw_multisite_tests: check_objects_not_exist(bucket, objnamesB) 2024-04-20T01:55:27.793 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2088, in check_objects_not_exist 2024-04-20T01:55:27.793 INFO:tasks.rgw_multisite_tests: check_object_not_exists(bucket, objname) 2024-04-20T01:55:27.793 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2084, in check_object_not_exists 2024-04-20T01:55:27.793 INFO:tasks.rgw_multisite_tests: assert_equal(k, None) 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:AssertionError: != None 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:27.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:37:07.057559Z" 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:28.272+0000 7f14416e0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:28.273+0000 7f14416e0780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:39:28Z" 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.801 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.802 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.803 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:28Z" 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.805 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:39:28.403+0000 7fba96bfd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:39:28.404+0000 7fba96bfd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:39:28Z" 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.811 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.812 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.813 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.814 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.815 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:28Z" 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "epoch": 13, 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": [], 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period_map": { 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zonegroups": [ 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "test-zonegroup", 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zonegroup", 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "api_name": "", 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "is_master": true, 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "hostnames": [], 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "hostnames_s3website": [], 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone2", 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": false, 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:27.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone1", 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:27.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": true, 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone3", 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:27.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": true, 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "placement_targets": [ 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "default-placement", 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [], 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "storage_classes": [ 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "STANDARD" 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "default_placement": "default-placement", 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_policy": { 2024-04-20T01:55:27.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "groups": [ 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-group", 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_flow": { 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "symmetrical": [ 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-flow1", 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-flow", 2024-04-20T01:55:27.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "directional": [ 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest_zone": "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pipes": [ 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-pipe", 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "*", 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": { 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "*", 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "params": { 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "filter": { 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [] 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": {}, 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "priority": 0, 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mode": "system", 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user": "" 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "enabled" 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled_features": [ 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "short_zone_ids": [ 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:27.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 2010863401 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 1221149675 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 1478903101 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zonegroup": "test-zonegroup", 2024-04-20T01:55:27.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period_config": { 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_quota": { 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false, 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "check_on_raw": false, 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size": -1, 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size_kb": 0, 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_objects": -1 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user_quota": { 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false, 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "check_on_raw": false, 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size": -1, 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size_kb": 0, 2024-04-20T01:55:27.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_objects": -1 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user_ratelimit": { 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_ratelimit": { 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:27.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "anonymous_ratelimit": { 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_name": "test-realm", 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]},{"id":"sync-flow","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["test-zone2","test-zone1"]},"dest":{"bucket":"*","zones":["test-zone2","test-zone1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-84 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/ 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/ 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/ 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests:/aktlji-84/ 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:raIT75XfteKyAnPqirknLmovY/I= 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:raIT75XfteKyAnPqirknLmovY/I=', 'Content-Length': '0'} 2024-04-20T01:55:27.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000c1a1ba9a3b40159e-0066231cd1-1afa-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/ 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/ 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/ 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.837 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:/aktlji-84/ 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:PTXe08CBU/rEonEezwlIT/DoYF0= 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PTXe08CBU/rEonEezwlIT/DoYF0=', 'Content-Length': '0'} 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff18b47de1847e38-0066231cd1-1afa-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/obj1 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/obj1 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/obj1 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:/aktlji-84/obj1 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:bN9WfzkIjsC3fzqVU7pZrdMKzwA= 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bN9WfzkIjsC3fzqVU7pZrdMKzwA='} 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099b6cd1479cb2ea3-0066231cd1-1afa-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/ 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/ 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/ 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.840 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:/aktlji-84/ 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:PTXe08CBU/rEonEezwlIT/DoYF0= 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PTXe08CBU/rEonEezwlIT/DoYF0=', 'Content-Length': '0'} 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f602b1c77dc448a5-0066231cd1-1afa-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/obj2 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/obj2 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/obj2 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:/aktlji-84/obj2 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:gXBIRZ8az4J5ggdSWtQS7P1Cs4M= 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:gXBIRZ8az4J5ggdSWtQS7P1Cs4M='} 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1f7412ed7d75cdb-0066231cd1-1afa-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/ 2024-04-20T01:55:27.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/ 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/ 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.843 INFO:tasks.rgw_multisite_tests:/aktlji-84/ 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:PTXe08CBU/rEonEezwlIT/DoYF0= 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PTXe08CBU/rEonEezwlIT/DoYF0=', 'Content-Length': '0'} 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b01ec849296601f1-0066231cd1-1afa-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/obj3 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/obj3 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/obj3 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:/aktlji-84/obj3 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/RDNyHoRPPCz6MXdGMVdcRWCCAU= 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/RDNyHoRPPCz6MXdGMVdcRWCCAU='} 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005dcba42ed1e2e964-0066231cd1-1afa-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone2 name=aktlji-85 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-85/ 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-85/ 2024-04-20T01:55:27.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-85/ 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.846 INFO:tasks.rgw_multisite_tests:/aktlji-85/ 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:VseUGqI3SdX/5C45LGSfHMewHvg= 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:VseUGqI3SdX/5C45LGSfHMewHvg=', 'Content-Length': '0'} 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000083346015ba5118d6-0066231cd1-230b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-85/ 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-85/ 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-85/ 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests:/aktlji-85/ 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WFH3enhYO3U6bq0T6oJEaghkNKY= 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WFH3enhYO3U6bq0T6oJEaghkNKY=', 'Content-Length': '0'} 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044a83feb71594998-0066231cd1-230b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-85/obj4 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-85/obj4 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-85/obj4 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:/aktlji-85/obj4 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.849 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:FoRwJa2TRFdFg+zFcIDXUlSr6GM= 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:FoRwJa2TRFdFg+zFcIDXUlSr6GM='} 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8ab8bb44a16dc5a-0066231cd1-230b-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-85/ 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-85/ 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-85/ 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.850 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests:/aktlji-85/ 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WFH3enhYO3U6bq0T6oJEaghkNKY= 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WFH3enhYO3U6bq0T6oJEaghkNKY=', 'Content-Length': '0'} 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4f86f3df0c0af55-0066231cd1-230b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-85/obj5 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-85/obj5 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-85/obj5 2024-04-20T01:55:27.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:/aktlji-85/obj5 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:kMoUfM+7EiuTlwTX7tjG7+0bOSg= 2024-04-20T01:55:27.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:kMoUfM+7EiuTlwTX7tjG7+0bOSg='} 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fdd482859e3c233c-0066231cd1-230b-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-85/ 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-85/ 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-85/ 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:27.853 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests:/aktlji-85/ 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WFH3enhYO3U6bq0T6oJEaghkNKY= 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WFH3enhYO3U6bq0T6oJEaghkNKY=', 'Content-Length': '0'} 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003efbf367a26357f7-0066231cd1-230b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-85/obj6 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-85/obj6 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-85/obj6 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:27.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:39:29 GMT 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:/aktlji-85/obj6 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:mJ8Mwq4tdqMTI0MrQE40kl+yqmE= 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:39:29 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mJ8Mwq4tdqMTI0MrQE40kl+yqmE='} 2024-04-20T01:55:27.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1d9af974dcb1281-0066231cd1-230b-test-zone2'), ('Date', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:27.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577169.777139_1578.1", 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:39:29.777139Z" 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:27.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:30.279+0000 7f9680690780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:30.280+0000 7f9680690780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:39:30Z" 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.863 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.865 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:30Z" 2024-04-20T01:55:27.867 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:35.457+0000 7fadbc265780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:35.458+0000 7fadbc265780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:39:35Z" 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.873 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.874 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.875 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.876 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.877 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:35Z" 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:40.594+0000 7f26b03dd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:40.596+0000 7f26b03dd780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:39:40Z" 2024-04-20T01:55:27.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.886 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:40Z" 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:45.737+0000 7fc4d4c06780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:45.738+0000 7fc4d4c06780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:39:45Z" 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.894 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.895 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.896 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:45Z" 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:50.873+0000 7fde2af5d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:50.874+0000 7fde2af5d780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:39:50Z" 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.904 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.906 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.907 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.908 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:50Z" 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:56.011+0000 7fd74586e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:39:56.012+0000 7fd74586e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:39:56Z" 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.915 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.916 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.917 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:39:56Z" 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.919 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:01.146+0000 7f41ddab2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:01.147+0000 7f41ddab2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:40:01Z" 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.925 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.926 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.927 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.928 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.929 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:40:01Z" 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:06.293+0000 7f1e99c5e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:06.294+0000 7f1e99c5e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:40:06Z" 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.936 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.937 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.938 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.939 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:40:06Z" 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.940 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:11.438+0000 7f1d560dc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:11.439+0000 7f1d560dc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:40:11Z" 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.946 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.949 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.950 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:40:11Z" 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:16.580+0000 7f7b55fc8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:16.582+0000 7f7b55fc8780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:40:16Z" 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.957 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.958 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.959 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.960 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:40:16Z" 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.962 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.962 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:21.716+0000 7f07ec19e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:21.717+0000 7f07ec19e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:40:21Z" 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.967 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.968 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.969 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.970 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.971 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.972 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.972 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.972 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:40:21Z" 2024-04-20T01:55:27.972 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.972 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.972 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.972 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.972 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.972 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.972 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:26.879+0000 7f537c1cf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:26.880+0000 7f537c1cf780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:40:26Z" 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.978 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.979 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.980 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.981 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:40:26Z" 2024-04-20T01:55:27.982 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:32.025+0000 7f80226d9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:32.026+0000 7f80226d9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:40:32Z" 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.988 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.990 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.991 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.992 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:40:32Z" 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:27.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:37.173+0000 7f6d59dde780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:37.174+0000 7f6d59dde780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:27.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:27.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:27.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:27.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:40:37Z" 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:27.999 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.001 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.002 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:40:37Z" 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.003 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:42.303+0000 7f24bb7f2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:42.305+0000 7f24bb7f2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:40:42Z" 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.010 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.011 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.012 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.013 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:40:42Z" 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:47.437+0000 7f7b966e1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:47.439+0000 7f7b966e1780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:40:47Z" 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.020 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.021 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.022 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:40:47Z" 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:52.575+0000 7f2e8ea8a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:52.577+0000 7f2e8ea8a780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:40:52Z" 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.030 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.031 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.032 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.033 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:40:52Z" 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:57.722+0000 7ff6e21dc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:40:57.723+0000 7ff6e21dc780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:40:57Z" 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.040 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.041 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.042 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.043 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.044 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:40:57Z" 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:41:02.862+0000 7f40c878b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:41:02.863+0000 7f40c878b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:41:02Z" 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.051 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.052 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.053 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577027.057559_1543.1", 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:37:07.057559Z", 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.054 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:41:02Z" 2024-04-20T01:55:28.055 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577027.057559_1543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:37:07.057559Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577027.057559_1543.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577169.777139_1578.1 target=1_1713577027.057559_1543.1 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:41:08.005+0000 7fcd07992780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:41:08.006+0000 7fcd07992780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577169.777139_1578.1", 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:39:29.777139Z", 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:41:08Z" 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.061 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.063 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577169.777139_1578.1", 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:39:29.777139Z", 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.064 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.065 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:41:08Z" 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577169.777139_1578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:29.777139Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:41:08.134+0000 7fceb74b2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:41:08.135+0000 7fceb74b2780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:28.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577169.777139_1578.1", 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:39:29.777139Z", 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:28.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:41:08Z" 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.072 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577169.777139_1578.1", 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:39:29.777139Z", 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:41:08Z" 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.076 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577169.777139_1578.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:29.777139Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577169.777139_1578.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:55:28.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000020738", 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:39:29.514702Z" 2024-04-20T01:55:28.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:55:28.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:55:28.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:41:08Z" 2024-04-20T01:55:28.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.087 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:28.088 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.089 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.090 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:41:08Z" 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015063', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.851769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026490', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.842528Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020598', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:07.899475Z'}}] 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020598'} 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015203 target=00000000000000000000:00000000000000015063, shard=1 master=00000000000000000000:00000000000000026910 target=00000000000000000000:00000000000000026490, shard=3 master=00000000000000000000:00000000000000020738 target=00000000000000000000:00000000000000020598 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:41:13Z" 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.096 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.097 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.098 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.099 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:41:13Z" 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.100 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015063', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.851769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026490', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.842528Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020598', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:07.899475Z'}}] 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020598'} 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015203 target=00000000000000000000:00000000000000015063, shard=1 master=00000000000000000000:00000000000000026910 target=00000000000000000000:00000000000000026490, shard=3 master=00000000000000000000:00000000000000020738 target=00000000000000000000:00000000000000020598 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:28.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:41:18Z" 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.106 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.107 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.108 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.110 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.110 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.110 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.110 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.110 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:41:18Z" 2024-04-20T01:55:28.110 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.110 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015063', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.851769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026490', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.842528Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020598', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:07.899475Z'}}] 2024-04-20T01:55:28.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:55:28.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020598'} 2024-04-20T01:55:28.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015203 target=00000000000000000000:00000000000000015063, shard=1 master=00000000000000000000:00000000000000026910 target=00000000000000000000:00000000000000026490, shard=3 master=00000000000000000000:00000000000000020738 target=00000000000000000000:00000000000000020598 2024-04-20T01:55:28.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:55:28.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:41:23Z" 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.115 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.116 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.117 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:55:28.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:41:23Z" 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.119 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015063', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.851769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026490', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.842528Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020598', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:07.899475Z'}}] 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020598'} 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015203 target=00000000000000000000:00000000000000015063, shard=1 master=00000000000000000000:00000000000000026910 target=00000000000000000000:00000000000000026490, shard=3 master=00000000000000000000:00000000000000020738 target=00000000000000000000:00000000000000020598 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:28.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:41:29Z" 2024-04-20T01:55:28.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015063", 2024-04-20T01:55:28.125 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.851769Z" 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000026490", 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.126 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:38:37.842528Z" 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.127 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000020598", 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:39:07.899475Z" 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.128 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:41:29Z" 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015063', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.851769Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026490', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:38:37.842528Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020598', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:39:07.899475Z'}}] 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015063', 1: '00000000000000000000:00000000000000026490', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020598'} 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015203 target=00000000000000000000:00000000000000015063, shard=1 master=00000000000000000000:00000000000000026910 target=00000000000000000000:00000000000000026490, shard=3 master=00000000000000000000:00000000000000020738 target=00000000000000000000:00000000000000020598 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:41:34Z" 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.134 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015203", 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:41:00.554653Z" 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.135 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000026910", 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:41:00.551471Z" 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.136 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024649", 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:27:17.869908Z" 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000020878", 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.137 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:41:30.495154Z" 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:41:34Z" 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015203', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.554653Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026910', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:41:00.551471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024649', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:27:17.869908Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020878', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:41:30.495154Z'}}] 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020738'} 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015203', 1: '00000000000000000000:00000000000000026910', 2: '00000000000000000000:00000000000000024649', 3: '00000000000000000000:00000000000000020878'} 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/ 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/ 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/ 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests:/aktlji-84/ 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QPaT0up6+kbSILGegmJVSkTDTvU= 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QPaT0up6+kbSILGegmJVSkTDTvU=', 'Content-Length': '0'} 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007da55f99f1bb39a3-0066231d4e-230b-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/obj1 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/obj1 2024-04-20T01:55:28.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/obj1 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests:/aktlji-84/obj1 2024-04-20T01:55:28.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:poQpk1qw+5z95rLoZlO+Tu+adX8= 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:poQpk1qw+5z95rLoZlO+Tu+adX8=', 'Content-Length': '0'} 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-84:73f46f02-c7ab-4426-a267-2273af911da5.6914.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d291140d1b59c20-0066231d4e-230b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/obj1 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/obj1 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/obj1 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests:/aktlji-84/obj1 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Dc/wGOb5x9otFFEpQVEPEvCa9QM= 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Dc/wGOb5x9otFFEpQVEPEvCa9QM=', 'Content-Length': '0'} 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-84:73f46f02-c7ab-4426-a267-2273af911da5.6914.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047064a1abfbe4a9b-0066231d4e-230b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/obj2 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/obj2 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/obj2 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests:/aktlji-84/obj2 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.144 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:bK8+vwThoLlSHLsQ1fgk+RVrx3Y= 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bK8+vwThoLlSHLsQ1fgk+RVrx3Y=', 'Content-Length': '0'} 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-84:73f46f02-c7ab-4426-a267-2273af911da5.6914.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e2e40b8247de1e9-0066231d4e-230b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/obj2 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/obj2 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/obj2 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.145 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests:/aktlji-84/obj2 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:UqnSCJ48ids3GsErTKl+IOrHDrE= 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:UqnSCJ48ids3GsErTKl+IOrHDrE=', 'Content-Length': '0'} 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-84:73f46f02-c7ab-4426-a267-2273af911da5.6914.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec54cf3d3693e32e-0066231d4e-230b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/obj3 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/obj3 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/obj3 2024-04-20T01:55:28.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests:/aktlji-84/obj3 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BqVQT/4X0Xki93F45B9F3o7f9/k= 2024-04-20T01:55:28.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BqVQT/4X0Xki93F45B9F3o7f9/k=', 'Content-Length': '0'} 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-84:73f46f02-c7ab-4426-a267-2273af911da5.6914.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fc1a25ea40f9ca0-0066231d4e-230b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/obj3 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/obj3 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/obj3 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:28.148 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:/aktlji-84/obj3 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:PBbi+VHM1CVk3qlLWVKH/KT5ECs= 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:PBbi+VHM1CVk3qlLWVKH/KT5ECs=', 'Content-Length': '0'} 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-84:73f46f02-c7ab-4426-a267-2273af911da5.6914.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e1b8bd06c83d4f4-0066231d4e-230b-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/ 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/ 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/ 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests:/aktlji-84/ 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:QPaT0up6+kbSILGegmJVSkTDTvU= 2024-04-20T01:55:28.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:QPaT0up6+kbSILGegmJVSkTDTvU=', 'Content-Length': '0'} 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099d2466410cfee3a-0066231d4e-1b06-test-zone3'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/obj1 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/obj1 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/obj1 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.151 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:/aktlji-84/obj1 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:poQpk1qw+5z95rLoZlO+Tu+adX8= 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:poQpk1qw+5z95rLoZlO+Tu+adX8=', 'Content-Length': '0'} 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx000008a905d5fed85776c-0066231d4e-1b06-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/obj2 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/obj2 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/obj2 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests:/aktlji-84/obj2 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:bK8+vwThoLlSHLsQ1fgk+RVrx3Y= 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:bK8+vwThoLlSHLsQ1fgk+RVrx3Y=', 'Content-Length': '0'} 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000031caa2a795cf7149-0066231d4e-1b06-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-84/obj3 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-84/obj3 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-84/obj3 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8001 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.154 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:/aktlji-84/obj3 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:BqVQT/4X0Xki93F45B9F3o7f9/k= 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:BqVQT/4X0Xki93F45B9F3o7f9/k=', 'Content-Length': '0'} 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000021f2526c5039bd55-0066231d4e-1b06-test-zone3'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-85/ 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-85/ 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-85/ 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests:/aktlji-85/ 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:6hVfn4Sz82C8PlLrMowU9W0g2sw= 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:6hVfn4Sz82C8PlLrMowU9W0g2sw=', 'Content-Length': '0'} 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072ad049e2f61ed8a-0066231d4e-1afa-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-85/obj4 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-85/obj4 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-85/obj4 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.157 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:41:34 GMT 2024-04-20T01:55:28.158 INFO:tasks.rgw_multisite_tests:/aktlji-85/obj4 2024-04-20T01:55:28.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.158 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:GSrekYlei2FJN/yI+5MPjvjd1ec= 2024-04-20T01:55:28.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:41:34 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:GSrekYlei2FJN/yI+5MPjvjd1ec=', 'Content-Length': '0'} 2024-04-20T01:55:28.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:39:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '03463f3d-be9b-4035-b2ce-7cd216cfabda:aktlji-85:73f46f02-c7ab-4426-a267-2273af911da5.6914.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041d507968a777f20-0066231d4e-1afa-test-zone1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:41:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.158 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:28.158 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.158 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:28.158 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2024-04-20T01:55:28.159 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:28.159 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2730, in test_sync_multiple_buckets_to_single 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesB, content) 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2080, in check_objects_exist 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577834.466139_1718.1", 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:50:34.466139Z" 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:50:34.834+0000 7f6b1d63f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:50:34.835+0000 7f6b1d63f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:50:34Z" 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.168 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.171 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:50:34Z" 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577812.772163_1716.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.772163Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577834.466139_1718.1 target=1_1713577812.772163_1716.1 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:50:39.983+0000 7f6224b6b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:50:39.985+0000 7f6224b6b780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:50:39Z" 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.178 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.180 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.181 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.181 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.181 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.181 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.181 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.181 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:55:28.181 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.181 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.181 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.181 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:55:28.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.181 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.182 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:50:39Z" 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577812.772163_1716.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.772163Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577834.466139_1718.1 target=1_1713577812.772163_1716.1 2024-04-20T01:55:28.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:50:45.117+0000 7f01be384780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:50:45.118+0000 7f01be384780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:50:45Z" 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.189 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.190 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.191 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.192 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:50:45Z" 2024-04-20T01:55:28.193 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577812.772163_1716.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.772163Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577834.466139_1718.1 target=1_1713577812.772163_1716.1 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:50:50.249+0000 7f8bd843e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:50:50.250+0000 7f8bd843e780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:50:50Z" 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.199 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.200 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.201 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577812.772163_1716.1", 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:12.772163Z", 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.202 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.203 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:50:50Z" 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577812.772163_1716.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:12.772163Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577812.772163_1716.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577834.466139_1718.1 target=1_1713577812.772163_1716.1 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:50:55.387+0000 7fe6b0d75780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:50:55.388+0000 7fe6b0d75780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577834.466139_1718.1", 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:34.466139Z", 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:50:55Z" 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.210 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.211 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577834.466139_1718.1", 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.212 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:34.466139Z", 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.213 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:50:55Z" 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577834.466139_1718.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:34.466139Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:50:55.520+0000 7f1620fb9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:50:55.521+0000 7f1620fb9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577834.466139_1718.1", 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:50:34.466139Z", 2024-04-20T01:55:28.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:50:55Z" 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.220 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.221 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.222 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577834.466139_1718.1", 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:34.466139Z", 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.223 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.224 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:50:55Z" 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577834.466139_1718.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:34.466139Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577834.466139_1718.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[]} 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-90 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-90/ 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-90/ 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-90/ 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:50:55 GMT 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests:/aktlji-90/ 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:16i0W4/ROY1NXCxd4KXb7xFNncA= 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:55 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:16i0W4/ROY1NXCxd4KXb7xFNncA=', 'Content-Length': '0'} 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000022682f4dc252fb7d-0066231f7f-1cbe-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:50:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-91 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/ 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/ 2024-04-20T01:55:28.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/ 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:50:55 GMT 2024-04-20T01:55:28.228 INFO:tasks.rgw_multisite_tests:/aktlji-91/ 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Iq7tDJv7utXl0B/jsJCOkNWEY/w= 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:50:55 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Iq7tDJv7utXl0B/jsJCOkNWEY/w=', 'Content-Length': '0'} 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000001f3c088f3fbc49ab-0066231f7f-1cbe-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:50:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:50:55.729112Z" 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:28.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:50:55.971+0000 7fc49cdad780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:50:55.972+0000 7fc49cdad780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577855.686103_1719.1", 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:55.686103Z", 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:50:55Z" 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.236 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.237 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.238 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577855.686103_1719.1", 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:55.686103Z", 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.239 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.240 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:50:55Z" 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.686103_1719.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.686103Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.686103_1719.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone2 behind master: shard=2 master=1_1713577855.729112_1726.1 target=1_1713577855.686103_1719.1 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:51:01.107+0000 7f08ed7d4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:51:01.108+0000 7f08ed7d4780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:01Z" 2024-04-20T01:55:28.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:01Z" 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.729112_1726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.729112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:51:01.252+0000 7f3e439c9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:51:01.253+0000 7f3e439c9780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:28.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:51:01Z" 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.257 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.258 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.259 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.260 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.261 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:01Z" 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.729112_1726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.729112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=test-zone1', '--dest-zone=test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=test-zone1 --dest-zone=test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[],"status":"allowed"}]} 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-90', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-90', '--dest-bucket=aktlji-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-90', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-90', '--dest-bucket=aktlji-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-aktlji-90 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=aktlji-90 --dest-bucket=aktlji-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:28.263 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-91', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-91', '--dest-bucket=aktlji-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-91', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-91', '--dest-bucket=aktlji-91', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-aktlji-91 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=aktlji-91 --dest-bucket=aktlji-91 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "epoch": 18, 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": [], 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period_map": { 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zonegroups": [ 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "test-zonegroup", 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zonegroup", 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "api_name": "", 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "is_master": true, 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "hostnames": [], 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "hostnames_s3website": [], 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone2", 2024-04-20T01:55:28.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": false, 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:28.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone1", 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:28.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": true, 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone3", 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": true, 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "placement_targets": [ 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "default-placement", 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [], 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "storage_classes": [ 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "STANDARD" 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "default_placement": "default-placement", 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_policy": { 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "groups": [ 2024-04-20T01:55:28.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-group", 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_flow": { 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "directional": [ 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest_zone": "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pipes": [ 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-pipe-aktlji-90", 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-90", 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": { 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-91", 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "params": { 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "filter": { 2024-04-20T01:55:28.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [] 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": {}, 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "priority": 0, 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mode": "system", 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user": "" 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-pipe-aktlji-91", 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-91", 2024-04-20T01:55:28.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": { 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-91", 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "params": { 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "filter": { 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [] 2024-04-20T01:55:28.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": {}, 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "priority": 0, 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mode": "system", 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user": "" 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "enabled" 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled_features": [ 2024-04-20T01:55:28.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "short_zone_ids": [ 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 2010863401 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 1221149675 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 1478903101 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zonegroup": "test-zonegroup", 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period_config": { 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_quota": { 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false, 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "check_on_raw": false, 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size": -1, 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size_kb": 0, 2024-04-20T01:55:28.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_objects": -1 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user_quota": { 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false, 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "check_on_raw": false, 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size": -1, 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size_kb": 0, 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_objects": -1 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user_ratelimit": { 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:28.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_ratelimit": { 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "anonymous_ratelimit": { 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:28.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_name": "test-realm", 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:28.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-90/ 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-90/ 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-90/ 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.282 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:51:02 GMT 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:/aktlji-90/ 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sIvuZQ+J3ndacA6H5KRx3dpQ/XE= 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sIvuZQ+J3ndacA6H5KRx3dpQ/XE=', 'Content-Length': '0'} 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1d8a182a3483104-0066231f86-1dec-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-90/obj1 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-90/obj1 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-90/obj1 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:51:02 GMT 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:/aktlji-90/obj1 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Dc7brSe5PRN6JD8Lm1xM70anyuM= 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Dc7brSe5PRN6JD8Lm1xM70anyuM='} 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe40ebbf376afaca-0066231f86-1dec-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-90/ 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-90/ 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-90/ 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.285 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:51:02 GMT 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:/aktlji-90/ 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sIvuZQ+J3ndacA6H5KRx3dpQ/XE= 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sIvuZQ+J3ndacA6H5KRx3dpQ/XE=', 'Content-Length': '0'} 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d2f663d110720fe-0066231f86-1dec-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-90/obj2 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-90/obj2 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-90/obj2 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.290 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:28.290 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:28.290 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:28.290 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:51:02 GMT 2024-04-20T01:55:28.290 INFO:tasks.rgw_multisite_tests:/aktlji-90/obj2 2024-04-20T01:55:28.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.290 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:/v/R/wUicGqWo0CjagyAm+ExVPY= 2024-04-20T01:55:28.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:/v/R/wUicGqWo0CjagyAm+ExVPY='} 2024-04-20T01:55:28.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000296e79f502584d9d-0066231f86-1dec-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/ 2024-04-20T01:55:28.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/ 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/ 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:51:02 GMT 2024-04-20T01:55:28.291 INFO:tasks.rgw_multisite_tests:/aktlji-91/ 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:mXfbVpFiuIYYSG2SmI3/TLGNM2E= 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mXfbVpFiuIYYSG2SmI3/TLGNM2E=', 'Content-Length': '0'} 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5a9f8ff9d901398-0066231f86-1dec-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/obj3 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/obj3 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/obj3 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:51:02 GMT 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:/aktlji-91/obj3 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:43IifLKJSfFLPGqjCJ05TNnPwng= 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:43IifLKJSfFLPGqjCJ05TNnPwng='} 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b71f8e87951152b1-0066231f86-1dec-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/ 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/ 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/ 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:51:02 GMT 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests:/aktlji-91/ 2024-04-20T01:55:28.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:mXfbVpFiuIYYSG2SmI3/TLGNM2E= 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:mXfbVpFiuIYYSG2SmI3/TLGNM2E=', 'Content-Length': '0'} 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000596dd50c0dc61bcb-0066231f86-1dec-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/obj4 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/obj4 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/obj4 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:51:02 GMT 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:/aktlji-91/obj4 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:hIAxjr2JMDYqiumSro12g5sph3E= 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:51:02 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:hIAxjr2JMDYqiumSro12g5sph3E='} 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6c81492b860fac3-0066231f86-1dec-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:50:55.729112Z" 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:51:02.959+0000 7fbb89b18780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:51:02.961+0000 7fbb89b18780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:02Z" 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.304 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.305 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.306 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.307 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:02Z" 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.729112_1726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.729112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:51:03.182+0000 7f897ab27780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:51:03.184+0000 7f897ab27780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:51:03Z" 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.314 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.315 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.316 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.317 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.318 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:03Z" 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.729112_1726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.729112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:03Z" 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.326 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.327 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.328 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.329 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:03Z" 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:08Z" 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.335 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.336 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.337 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.338 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.339 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:08Z" 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:13Z" 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.345 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.346 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.347 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.348 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:13Z" 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:18Z" 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.354 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.355 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.356 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.357 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.358 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:18Z" 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:24Z" 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.364 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.365 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.366 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.366 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.366 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.366 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.366 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.366 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.366 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.366 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.366 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.366 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.366 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.367 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:24Z" 2024-04-20T01:55:28.368 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:29Z" 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.374 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.375 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.376 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.377 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:29Z" 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.378 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:34Z" 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.383 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.384 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.385 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.386 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.386 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.386 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.386 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.386 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.386 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.386 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.386 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.386 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.386 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.386 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.386 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.387 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:34Z" 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:39Z" 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.393 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.394 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.394 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.394 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.394 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.394 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.394 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.394 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.394 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.394 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.394 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.395 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.396 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:39Z" 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.397 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:44Z" 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.403 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.404 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.405 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.406 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.406 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.406 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.406 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.406 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.406 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.406 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.406 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:44Z" 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:49Z" 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.413 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.414 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.415 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.416 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:49Z" 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:51:54Z" 2024-04-20T01:55:28.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.422 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.425 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.426 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:51:54Z" 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:52:00Z" 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.432 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.433 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.434 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.435 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:52:00Z" 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.436 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:52:05Z" 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.442 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.445 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:52:05Z" 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.446 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:52:10Z" 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.451 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.452 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.453 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.454 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.455 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:52:10Z" 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:52:15Z" 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.461 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.462 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:52:15Z" 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:52:20Z" 2024-04-20T01:55:28.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.471 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.472 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.473 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.474 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:52:20Z" 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:52:25Z" 2024-04-20T01:55:28.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.481 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.481 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.481 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.481 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.482 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.482 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.482 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.482 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.482 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.482 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.483 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.483 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.483 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.483 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.483 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.483 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.483 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.483 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.484 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.485 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.486 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.486 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.486 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.486 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.486 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.486 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.486 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.486 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.486 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.486 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:52:25Z" 2024-04-20T01:55:28.486 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:52:32Z" 2024-04-20T01:55:28.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.495 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.497 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.498 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:52:32Z" 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:52:37Z" 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.504 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.505 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.506 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.507 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.508 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:52:37Z" 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:52:42Z" 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.514 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.515 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.517 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:52:42Z" 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:52:47Z" 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.523 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.524 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.525 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.526 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.527 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:52:47Z" 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:52:52Z" 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.533 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.534 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.536 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:52:52Z" 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:52:57Z" 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.542 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.543 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.544 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.545 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.546 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:52:57Z" 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:02Z" 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.552 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015343", 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891072Z" 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.553 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000027750", 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.957440Z" 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.554 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021298", 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.555 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:48:27.971508Z" 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:02Z" 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027750', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.957440Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021298', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:48:27.971508Z'}}] 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015343', 1: '00000000000000000000:00000000000000027750', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021298'} 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000015483 target=00000000000000000000:00000000000000015343, shard=1 master=00000000000000000000:00000000000000028030 target=00000000000000000000:00000000000000027750, shard=3 master=00000000000000000000:00000000000000021438 target=00000000000000000000:00000000000000021298 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:08Z" 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.561 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.562 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.563 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.564 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.565 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:08Z" 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-90/ 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-90/ 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-90/ 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests:/aktlji-90/ 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:sBy4ske2mPxpy2n6OfhXLinH78s= 2024-04-20T01:55:28.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:sBy4ske2mPxpy2n6OfhXLinH78s=', 'Content-Length': '0'} 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000169c6b0493fa95a4-0066232004-27d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-90/obj1 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-90/obj1 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-90/obj1 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.568 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:/aktlji-90/obj1 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:wp+yaJ7FJjBtGLXUJeJ+dFlUQEA= 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:wp+yaJ7FJjBtGLXUJeJ+dFlUQEA=', 'Content-Length': '0'} 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000187977713b3bf610-0066232004-27d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-90/obj2 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-90/obj2 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-90/obj2 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests:/aktlji-90/obj2 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:a5O7uZCabDpLzti9+67bTSiY7cc= 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a5O7uZCabDpLzti9+67bTSiY7cc=', 'Content-Length': '0'} 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000d0cf673e7dcc3a3d-0066232004-27d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-90/obj3 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-90/obj3 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-90/obj3 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.571 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:/aktlji-90/obj3 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ywIkfox+hkrZCJ9S9M/R2vfuDQs= 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ywIkfox+hkrZCJ9S9M/R2vfuDQs=', 'Content-Length': '0'} 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000eed9210ca6b0ba3e-0066232004-27d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-90/obj4 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-90/obj4 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-90/obj4 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests:/aktlji-90/obj4 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:U/LONsj0hFtM4Ph2zUcJdUn7zww= 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:U/LONsj0hFtM4Ph2zUcJdUn7zww=', 'Content-Length': '0'} 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000ba79d1693d625176-0066232004-27d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/ 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/ 2024-04-20T01:55:28.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/ 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.574 INFO:tasks.rgw_multisite_tests:/aktlji-91/ 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:54EqijFpEwVF2ELHPaocF/em7Mk= 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:54EqijFpEwVF2ELHPaocF/em7Mk=', 'Content-Length': '0'} 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016ba72d0faa67c60-0066232004-27d7-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/obj1 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/obj1 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/obj1 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests:/aktlji-91/obj1 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:A6uaoZun8/dAJUKAte+dYFbbOlU= 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:A6uaoZun8/dAJUKAte+dYFbbOlU=', 'Content-Length': '0'} 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-90:73f46f02-c7ab-4426-a267-2273af911da5.7366.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5b4ba39e18cfedf-0066232004-27d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/obj1 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/obj1 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:28.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/obj1 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests:/aktlji-91/obj1 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.577 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:fBLmDHKIkRTdpYTMbClXoqCc/B0= 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:fBLmDHKIkRTdpYTMbClXoqCc/B0=', 'Content-Length': '0'} 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-90:73f46f02-c7ab-4426-a267-2273af911da5.7366.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000016f315d50671f00-0066232004-27d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/obj2 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/obj2 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/obj2 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests:/aktlji-91/obj2 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:vpOrk73glIWeBwOSkMyEf+y2lP8= 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:vpOrk73glIWeBwOSkMyEf+y2lP8=', 'Content-Length': '0'} 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-90:73f46f02-c7ab-4426-a267-2273af911da5.7366.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006105163525d73614-0066232004-27d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/obj2 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/obj2 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/obj2 2024-04-20T01:55:28.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests:/aktlji-91/obj2 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:OeFqJxBaMdKDmRvBM48z2XbaWtI= 2024-04-20T01:55:28.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:OeFqJxBaMdKDmRvBM48z2XbaWtI=', 'Content-Length': '0'} 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-90:73f46f02-c7ab-4426-a267-2273af911da5.7366.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f8c32d83c8c92a0-0066232004-27d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/obj3 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/obj3 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/obj3 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.581 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests:/aktlji-91/obj3 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:WtovSonjPNYCUfwljF7Y+iOPx94= 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:WtovSonjPNYCUfwljF7Y+iOPx94=', 'Content-Length': '0'} 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-91:73f46f02-c7ab-4426-a267-2273af911da5.7366.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004029042808fcf3f8-0066232004-27d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/obj3 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/obj3 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/obj3 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests:/aktlji-91/obj3 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:aCH/mUlOojNePGs6rpb2MjsOw7M= 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:aCH/mUlOojNePGs6rpb2MjsOw7M=', 'Content-Length': '0'} 2024-04-20T01:55:28.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:51:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-91:73f46f02-c7ab-4426-a267-2273af911da5.7366.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069fa83dd53e5e9d4-0066232004-27d7-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-91/obj4 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-91/obj4 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-91/obj4 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.584 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.585 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.585 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.585 INFO:tasks.rgw_multisite_tests:/aktlji-91/obj4 2024-04-20T01:55:28.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.585 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:HokDBMfYIxhDp8IuUua6TyNkltU= 2024-04-20T01:55:28.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:HokDBMfYIxhDp8IuUua6TyNkltU=', 'Content-Length': '0'} 2024-04-20T01:55:28.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx0000077f5d438f8aa46c4-0066232004-27d7-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.585 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:28.585 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.585 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-20T01:55:28.586 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket_to_multiple: 2024-04-20T01:55:28.586 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:28.586 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-20T01:55:28.586 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-20T01:55:28.586 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2849, in test_sync_single_bucket_to_multiple 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesA, content) 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2080, in check_objects_exist 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:50:55.729112Z" 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:53:08.426+0000 7f3563500780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:53:08.427+0000 7f3563500780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:08Z" 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.594 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.595 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.596 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.597 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.598 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:08Z" 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.729112_1726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.729112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.599 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:53:08.565+0000 7f3211255780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:53:08.566+0000 7f3211255780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:28.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:53:08Z" 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.605 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.606 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.607 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577855.729112_1726.1", 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:55.729112Z", 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.608 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:08Z" 2024-04-20T01:55:28.609 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577855.729112_1726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:55.729112Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577855.729112_1726.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-92 2024-04-20T01:55:28.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-92/ 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-92/ 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-92/ 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.611 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:/aktlji-92/ 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:Vj8QkI9ZRSfpRlUPzWfQtTij3VU= 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:Vj8QkI9ZRSfpRlUPzWfQtTij3VU=', 'Content-Length': '0'} 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000dda16759288e2032-0066232004-1dec-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=test-zone1 name=aktlji-93 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-93/ 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-93/ 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-93/ 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:08 GMT 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests:/aktlji-93/ 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:cfwTYpc3YiQ6c3bkh6ydoO4kXwE= 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:08 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:cfwTYpc3YiQ6c3bkh6ydoO4kXwE=', 'Content-Length': '0'} 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000003b4812be046d6506-0066232004-1dec-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:53:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.613 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:53:08.757488Z" 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:53:09.024+0000 7f1bd85ee780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:53:09.025+0000 7f1bd85ee780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:53:08.757488Z", 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:09Z" 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.622 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:55:28.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:53:08.757488Z", 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.624 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:09Z" 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.625 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577988.757488_1755.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:53:08.757488Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:53:09.152+0000 7f0877c32780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:53:09.153+0000 7f0877c32780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577988.723277_1748.1", 2024-04-20T01:55:28.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:53:08.723277Z", 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:53:09Z" 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577988.723277_1748.1", 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.634 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:53:08.723277Z", 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.635 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:09Z" 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577988.723277_1748.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:53:08.723277Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.723277_1748.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone test-zone3 behind master: shard=2 master=1_1713577988.757488_1755.1 target=1_1713577988.723277_1748.1 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:53:14.284+0000 7fbe55b96780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:53:14.286+0000 7fbe55b96780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:53:08.757488Z", 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:28.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:28.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:53:14Z" 2024-04-20T01:55:28.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.642 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.642 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.642 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.642 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.642 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.642 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.643 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.644 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:53:08.757488Z", 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.645 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.646 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:14Z" 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577988.757488_1755.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:53:08.757488Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=test-zone1,test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=test-zone1,test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-92', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-92', '--dest-bucket=aktlji-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-92', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-92', '--dest-bucket=aktlji-92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-aktlji-92 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=aktlji-92 --dest-bucket=aktlji-92 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-92","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:28.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-93', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-92', '--dest-bucket=aktlji-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-aktlji-93', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-92', '--dest-bucket=aktlji-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-aktlji-93 --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=aktlji-92 --dest-bucket=aktlji-93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-92","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-93","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:28.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-92', '--dest-bucket=aktlji-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=test-zone1', '--dest-zones=test-zone2', '--source-bucket=aktlji-92', '--dest-bucket=aktlji-93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=test-zone1 --dest-zones=test-zone2 --source-bucket=aktlji-92 --dest-bucket=aktlji-93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-92","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-93","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-92","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-93","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "epoch": 19, 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "predecessor_uuid": "958711cb-8a73-4fca-9bb6-5b0d38627b0b", 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": [], 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period_map": { 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zonegroups": [ 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "test-zonegroup", 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zonegroup", 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "api_name": "", 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "is_master": true, 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:28.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "hostnames": [], 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "hostnames_s3website": [], 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone2", 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi031.front.sepia.ceph.com:8000" 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:28.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": false, 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:28.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone1", 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8000" 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": true, 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:28.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "test-zone3", 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "endpoints": [ 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "http://smithi019.front.sepia.ceph.com:8001" 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_meta": false, 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "log_data": true, 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-20T01:55:28.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "read_only": false, 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tier_type": "", 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from_all": true, 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_from": [], 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "redirect_zone": "", 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "supported_features": [ 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "compress-encrypted", 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "placement_targets": [ 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "name": "default-placement", 2024-04-20T01:55:28.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [], 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "storage_classes": [ 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "STANDARD" 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "default_placement": "default-placement", 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_policy": { 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "groups": [ 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-group", 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "data_flow": { 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "symmetrical": [ 2024-04-20T01:55:28.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-flow1", 2024-04-20T01:55:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "directional": [ 2024-04-20T01:55:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest_zone": "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:55:28.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pipes": [ 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-pipe-aktlji-90", 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-90", 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": { 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-91", 2024-04-20T01:55:28.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "params": { 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "filter": { 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [] 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": {}, 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "priority": 0, 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mode": "system", 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user": "" 2024-04-20T01:55:28.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-pipe-aktlji-91", 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-91", 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": { 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-91", 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "params": { 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "filter": { 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [] 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": {}, 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "priority": 0, 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mode": "system", 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user": "" 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-pipe-aktlji-92", 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-92", 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": { 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-92", 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:55:28.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "params": { 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "filter": { 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [] 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": {}, 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "priority": 0, 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mode": "system", 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user": "" 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-pipe-aktlji-93", 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-92", 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": { 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-93", 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "params": { 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "filter": { 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [] 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": {}, 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "priority": 0, 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mode": "system", 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user": "" 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "id": "sync-pipe", 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-92", 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "73f46f02-c7ab-4426-a267-2273af911da5" 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": { 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket": "aktlji-93", 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "zones": [ 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "03463f3d-be9b-4035-b2ce-7cd216cfabda" 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "params": { 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "source": { 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "filter": { 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "tags": [] 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "dest": {}, 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "priority": 0, 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "mode": "system", 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user": "" 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "enabled" 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled_features": [ 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "resharding" 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ], 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "short_zone_ids": [ 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "03463f3d-be9b-4035-b2ce-7cd216cfabda", 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 2010863401 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 1221149675 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": "bea5de7f-54a0-40b9-979f-37d0f728ff53", 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": 1478903101 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zonegroup": "test-zonegroup", 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "master_zone": "73f46f02-c7ab-4426-a267-2273af911da5", 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period_config": { 2024-04-20T01:55:28.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_quota": { 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false, 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "check_on_raw": false, 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size": -1, 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size_kb": 0, 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_objects": -1 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user_quota": { 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false, 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "check_on_raw": false, 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size": -1, 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_size_kb": 0, 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_objects": -1 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "user_ratelimit": { 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "bucket_ratelimit": { 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "anonymous_ratelimit": { 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_ops": 0, 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_ops": 0, 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_read_bytes": 0, 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "max_write_bytes": 0, 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "enabled": false 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_id": "0542eeed-b7db-454d-a499-a81b016de7e3", 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_name": "test-realm", 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-20T01:55:28.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["test-zone2","test-zone1"]}],"directional":[{"source_zone":"test-zone1","dest_zone":"test-zone2"}]},"pipes":[{"id":"sync-pipe-aktlji-90","source":{"bucket":"aktlji-90","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-91","source":{"bucket":"aktlji-91","zones":["test-zone1"]},"dest":{"bucket":"aktlji-91","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-92","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-92","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-aktlji-93","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"aktlji-92","zones":["test-zone1"]},"dest":{"bucket":"aktlji-93","zones":["test-zone2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-92/ 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-92/ 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-92/ 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:15 GMT 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests:/aktlji-92/ 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qVmR/AVK/5r/GZ2c9pbUywXmzic= 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qVmR/AVK/5r/GZ2c9pbUywXmzic=', 'Content-Length': '0'} 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fdf581878a43dff2-006623200b-1e80-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-92/obj1 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-92/obj1 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:28.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-92/obj1 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:15 GMT 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:/aktlji-92/obj1 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.676 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:zII3qIqFsLNeRxYV11N/xxImEio= 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:53:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:zII3qIqFsLNeRxYV11N/xxImEio='} 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f57f74ddafa93020-006623200b-1e80-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-92/ 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-92/ 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-92/ 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.677 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:15 GMT 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests:/aktlji-92/ 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:qVmR/AVK/5r/GZ2c9pbUywXmzic= 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:53:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:qVmR/AVK/5r/GZ2c9pbUywXmzic=', 'Content-Length': '0'} 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022773a0ff92d9f46-006623200b-1e80-test-zone1'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-92/obj2 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-92/obj2 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-92/obj2 2024-04-20T01:55:28.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi019.front.sepia.ceph.com:8000 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:PUT 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:53:15 GMT 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:/aktlji-92/obj2 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.679 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:a0maXniRS1UJN3PGkWpm6hb5ldA= 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 01:53:15 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:a0maXniRS1UJN3PGkWpm6hb5ldA='} 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000601dd7760de354f1-006623200b-1e80-test-zone1'), ('Date', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: No --period given, using current period=f494ba41-e721-4411-8938-a297d95f7246 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-19T23:20:37.840174Z" 2024-04-20T01:55:28.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "0.000000" 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:53:08.757488Z" 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:03:23.344070Z" 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:53:16.264+0000 7fbe351fb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: 2024-04-20T01:53:16.266+0000 7fbe351fb780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "", 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:53:08.757488Z", 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "state": 1, 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 1, 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 1 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:16Z" 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.687 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.688 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.689 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:53:08.757488Z", 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.690 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.691 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:16Z" 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577988.757488_1755.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:53:08.757488Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone2 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:53:16.471+0000 7fd75af0f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: 2024-04-20T01:53:16.473+0000 7fd75af0f780 20 RGW-SYNC:meta: read sync status 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "info": { 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "markers": [ 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 0, 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 1, 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 1, 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "", 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "0.000000", 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 2, 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:53:08.757488Z", 2024-04-20T01:55:28.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "key": 3, 2024-04-20T01:55:28.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "val": { 2024-04-20T01:55:28.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "state": 1, 2024-04-20T01:55:28.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "realm_epoch": 2 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "total": 1, 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "complete": 1 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "current_time": "2024-04-20T01:53:16Z" 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: "period": "f494ba41-e721-4411-8938-a297d95f7246", 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.698 INFO:tasks.rgw_multisite_tests: "marker": "1_1713568837.840174_14.1", 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-19T23:20:37.840174Z", 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-20T01:55:28.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: "marker": "1_1713577988.757488_1755.1", 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:53:08.757488Z", 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: "marker": "1_1713575003.344070_1445.1", 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:03:23.344070Z", 2024-04-20T01:55:28.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-20T01:55:28.702 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.702 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.702 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.702 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.702 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.702 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-20T01:55:28.702 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-20T01:55:28.702 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.702 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:16Z" 2024-04-20T01:55:28.702 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.702 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.702 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1713568837.840174_14.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-19T23:20:37.840174Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1713577988.757488_1755.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:53:08.757488Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1713575003.344070_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:03:23.344070Z', 'realm_epoch': 2}}] 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1713568837.840174_14.1', 1: '', 2: '1_1713577988.757488_1755.1', 3: '1_1713575003.344070_1445.1'} 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=test-zone3 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: [ 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000028170", 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:53:15.700250Z" 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: }, 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: { 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "marker": "00000000000000000000:00000000000000021578", 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: "last_update": "2024-04-20T01:53:15.749553Z" 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: } 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi019.stdout: INFO: ] 2024-04-20T01:55:28.704 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=test-zone1 markers={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:16Z" 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.709 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.711 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.712 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.713 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:16Z" 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:21Z" 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.719 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.720 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.721 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.722 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:21Z" 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:27Z" 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.729 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.731 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.732 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:27Z" 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:32Z" 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.738 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.739 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.740 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:32Z" 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.742 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:37Z" 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.748 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.749 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.751 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:37Z" 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:42Z" 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.758 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.759 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:42Z" 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:47Z" 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.767 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.768 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.769 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.770 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.770 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.770 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.770 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.770 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.770 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.770 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.770 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.771 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:47Z" 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:52Z" 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.777 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.778 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.779 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.780 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:52Z" 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:53:57Z" 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.786 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.787 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.788 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.789 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.790 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:53:57Z" 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:54:03Z" 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.796 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.797 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.798 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.799 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:54:03Z" 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:54:08Z" 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.807 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.808 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.809 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:54:08Z" 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:54:13Z" 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.815 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.816 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.817 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.818 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:54:13Z" 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:54:18Z" 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.826 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.827 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.828 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:54:18Z" 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:54:23Z" 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.834 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.835 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.836 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.837 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:54:23Z" 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:54:28Z" 2024-04-20T01:55:28.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.844 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.845 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.846 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.847 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:54:28Z" 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:54:33Z" 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.853 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.854 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.856 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:54:33Z" 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:54:39Z" 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.863 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.865 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.866 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:54:39Z" 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:54:44Z" 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.872 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.873 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.874 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.874 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.874 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.874 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.874 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.874 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.874 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.874 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.874 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.875 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.875 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.875 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.875 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.875 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.875 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.875 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.876 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.876 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.876 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.876 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.876 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.876 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.876 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.876 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.876 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.877 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.877 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.877 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.877 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.877 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.877 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.877 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.878 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.878 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:54:44Z" 2024-04-20T01:55:28.878 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.878 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:54:49Z" 2024-04-20T01:55:28.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.884 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.884 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.884 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.885 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.885 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.885 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.885 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.885 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.885 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.885 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.885 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.885 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.885 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.886 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.886 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.886 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.886 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.886 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.886 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.886 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.886 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.886 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.886 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.886 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.887 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.887 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.887 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.887 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.887 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.887 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.887 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.887 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.888 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.888 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.888 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.888 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.888 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.888 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.888 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.888 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.888 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.888 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.888 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.889 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.889 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.889 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.889 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.889 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.889 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.889 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.889 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:54:49Z" 2024-04-20T01:55:28.889 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.889 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:54:54Z" 2024-04-20T01:55:28.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.895 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.896 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.896 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.896 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.896 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.896 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.896 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.896 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.896 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.896 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.896 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.896 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.897 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.897 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.897 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.897 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.897 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.897 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.897 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.897 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.898 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.898 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.898 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.898 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.898 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.898 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.898 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.898 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.898 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.898 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.898 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.899 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.899 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.899 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.899 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.899 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.899 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.899 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.899 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.899 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.899 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.899 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.900 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.900 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.900 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.900 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.900 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.900 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.900 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.900 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.900 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.900 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.900 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:54:54Z" 2024-04-20T01:55:28.901 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.901 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:54:59Z" 2024-04-20T01:55:28.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.907 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.907 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.910 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.910 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.910 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.910 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.910 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.910 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.910 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.910 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.910 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.910 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.911 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.911 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.911 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.911 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.911 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.911 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.911 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.911 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.911 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.911 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.912 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.912 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.912 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.912 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.912 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.912 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.912 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.912 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.912 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.912 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.912 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.913 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.913 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.913 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.913 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.913 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.913 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.913 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.913 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.913 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.913 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.913 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.913 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.914 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.914 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.914 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.914 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.914 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.914 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.914 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.914 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.914 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.915 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:54:59Z" 2024-04-20T01:55:28.915 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.915 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.915 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.915 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:55:04Z" 2024-04-20T01:55:28.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.921 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.921 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.921 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.921 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.922 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.922 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.922 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.922 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.922 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.922 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.922 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.922 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.922 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.922 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.922 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.923 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.923 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.923 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.923 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.923 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.923 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.923 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.923 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.923 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.923 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.923 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.924 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.924 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.924 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.924 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.924 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.924 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.924 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.924 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.924 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.924 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.924 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.925 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.925 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.925 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.925 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.925 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.925 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.925 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.925 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.925 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.925 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.925 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.926 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.926 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.926 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.926 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.926 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.926 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.926 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.926 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.926 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.926 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:55:04Z" 2024-04-20T01:55:28.926 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.926 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:55:09Z" 2024-04-20T01:55:28.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.933 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.933 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.933 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.933 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.933 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.933 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.933 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.933 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.933 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.933 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.934 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.934 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.934 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.934 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.934 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.934 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.934 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.934 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.934 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.934 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.934 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.935 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.935 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.935 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.935 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.935 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.935 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.935 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.935 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.935 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.935 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.935 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.935 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.936 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.936 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.936 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.936 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.936 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.936 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.936 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.936 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.936 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.936 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.936 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.936 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.937 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.937 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.937 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.937 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.937 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.937 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.937 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.937 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.937 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.938 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:55:09Z" 2024-04-20T01:55:28.938 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.938 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:55:15Z" 2024-04-20T01:55:28.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.944 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.944 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.944 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.944 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.944 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.944 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.944 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.944 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.944 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.945 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.945 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.945 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.945 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.945 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.945 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.945 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.945 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.945 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.945 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.945 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.946 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.946 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028030", 2024-04-20T01:55:28.946 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.946 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.946 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.946 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.453429Z" 2024-04-20T01:55:28.946 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.946 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.946 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.946 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.946 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.947 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.947 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.947 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.947 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.947 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.947 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.947 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.947 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.947 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.947 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.947 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.948 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021438", 2024-04-20T01:55:28.948 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.948 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.948 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.403172Z" 2024-04-20T01:55:28.948 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.948 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.948 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.948 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.948 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.948 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.949 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.949 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:55:15Z" 2024-04-20T01:55:28.949 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.949 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028030', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.453429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021438', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.403172Z'}}] 2024-04-20T01:55:28.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028030', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021438'} 2024-04-20T01:55:28.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone test-zone2 behind zone test-zone1: shard=1 master=00000000000000000000:00000000000000028170 target=00000000000000000000:00000000000000028030, shard=3 master=00000000000000000000:00000000000000021578 target=00000000000000000000:00000000000000021438 2024-04-20T01:55:28.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-20T01:55:28.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-20T01:55:28.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-20T01:55:28.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-20T01:55:28.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "sync_status": { 2024-04-20T01:55:28.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "info": { 2024-04-20T01:55:28.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "sync", 2024-04-20T01:55:28.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "num_shards": 4, 2024-04-20T01:55:28.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "instance_id": 17812922715370875371 2024-04-20T01:55:28.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "markers": [ 2024-04-20T01:55:28.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 0, 2024-04-20T01:55:28.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 9, 2024-04-20T01:55:28.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 1, 2024-04-20T01:55:28.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000028170", 2024-04-20T01:55:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 70, 2024-04-20T01:55:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:53:15.700250Z" 2024-04-20T01:55:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 2, 2024-04-20T01:55:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: { 2024-04-20T01:55:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "key": 3, 2024-04-20T01:55:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "val": { 2024-04-20T01:55:28.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "status": "incremental-sync", 2024-04-20T01:55:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "marker": "00000000000000000000:00000000000000021578", 2024-04-20T01:55:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "next_step_marker": "", 2024-04-20T01:55:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total_entries": 0, 2024-04-20T01:55:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "pos": 0, 2024-04-20T01:55:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "timestamp": "2024-04-20T01:53:15.749553Z" 2024-04-20T01:55:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: ] 2024-04-20T01:55:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "full_sync": { 2024-04-20T01:55:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "total": 79, 2024-04-20T01:55:28.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "complete": 79 2024-04-20T01:55:28.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: }, 2024-04-20T01:55:28.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: "current_time": "2024-04-20T01:55:20Z" 2024-04-20T01:55:28.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi031.stdout: INFO: } 2024-04-20T01:55:28.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-20T01:55:28.955 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-20T01:55:28.955 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-20T01:55:28.955 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-20T01:55:28.955 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-20T01:55:28.955 INFO:tasks.rgw_multisite_tests: "instance_id": 17812922715370875371 2024-04-20T01:55:28.955 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.955 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-20T01:55:28.955 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.956 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-20T01:55:28.956 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.956 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.956 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000015483", 2024-04-20T01:55:28.956 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.956 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-20T01:55:28.956 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.956 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:51:02.485059Z" 2024-04-20T01:55:28.956 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.956 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.956 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.957 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-20T01:55:28.957 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.957 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.957 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000028170", 2024-04-20T01:55:28.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.957 INFO:tasks.rgw_multisite_tests: "total_entries": 70, 2024-04-20T01:55:28.957 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.957 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:53:15.700250Z" 2024-04-20T01:55:28.957 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.957 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.957 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.957 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-20T01:55:28.958 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.958 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.958 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000024930", 2024-04-20T01:55:28.958 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.958 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.958 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:50:15.891278Z" 2024-04-20T01:55:28.958 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.958 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.958 INFO:tasks.rgw_multisite_tests: { 2024-04-20T01:55:28.958 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-20T01:55:28.959 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-20T01:55:28.959 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-20T01:55:28.959 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000021578", 2024-04-20T01:55:28.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-20T01:55:28.959 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-20T01:55:28.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-20T01:55:28.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-20T01:53:15.749553Z" 2024-04-20T01:55:28.959 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.959 INFO:tasks.rgw_multisite_tests: } 2024-04-20T01:55:28.959 INFO:tasks.rgw_multisite_tests: ] 2024-04-20T01:55:28.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.959 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-20T01:55:28.960 INFO:tasks.rgw_multisite_tests: "total": 79, 2024-04-20T01:55:28.960 INFO:tasks.rgw_multisite_tests: "complete": 79 2024-04-20T01:55:28.960 INFO:tasks.rgw_multisite_tests: }, 2024-04-20T01:55:28.960 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-20T01:55:20Z" 2024-04-20T01:55:28.960 INFO:tasks.rgw_multisite_tests:} 2024-04-20T01:55:28.960 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015483', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-20T01:51:02.485059Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028170', 'next_step_marker': '', 'total_entries': 70, 'pos': 0, 'timestamp': '2024-04-20T01:53:15.700250Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024930', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:50:15.891278Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000021578', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-20T01:53:15.749553Z'}}] 2024-04-20T01:55:28.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000015483', 1: '00000000000000000000:00000000000000028170', 2: '00000000000000000000:00000000000000024930', 3: '00000000000000000000:00000000000000021578'} 2024-04-20T01:55:28.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-04-20T01:55:28.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-92/ 2024-04-20T01:55:28.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-92/ 2024-04-20T01:55:28.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-92/ 2024-04-20T01:55:28.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-20T01:55:28.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.961 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.962 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.962 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.962 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:55:20 GMT 2024-04-20T01:55:28.962 INFO:tasks.rgw_multisite_tests:/aktlji-92/ 2024-04-20T01:55:28.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.962 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:slSwALBBm9Fj87g8ENESDuXrTYg= 2024-04-20T01:55:28.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:slSwALBBm9Fj87g8ENESDuXrTYg=', 'Content-Length': '0'} 2024-04-20T01:55:28.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd5784dea44da6ee-0066232088-28bb-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-92/obj1 2024-04-20T01:55:28.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-92/obj1 2024-04-20T01:55:28.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-92/obj1 2024-04-20T01:55:28.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.963 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.963 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.963 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.963 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:55:20 GMT 2024-04-20T01:55:28.963 INFO:tasks.rgw_multisite_tests:/aktlji-92/obj1 2024-04-20T01:55:28.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.964 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ol8bukn6m9vlt1gGZe+7Fd1fLHo= 2024-04-20T01:55:28.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ol8bukn6m9vlt1gGZe+7Fd1fLHo=', 'Content-Length': '0'} 2024-04-20T01:55:28.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-92:73f46f02-c7ab-4426-a267-2273af911da5.7664.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005351028d89ba8175-0066232088-28bb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-92/obj1 2024-04-20T01:55:28.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-92/obj1 2024-04-20T01:55:28.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:28.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-92/obj1 2024-04-20T01:55:28.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.965 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:28.965 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.965 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.965 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:55:20 GMT 2024-04-20T01:55:28.965 INFO:tasks.rgw_multisite_tests:/aktlji-92/obj1 2024-04-20T01:55:28.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.965 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:RntvL+cK2GCjd+EyEwe5kxi2j08= 2024-04-20T01:55:28.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:RntvL+cK2GCjd+EyEwe5kxi2j08=', 'Content-Length': '0'} 2024-04-20T01:55:28.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-92:73f46f02-c7ab-4426-a267-2273af911da5.7664.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052e28215de0c37f9-0066232088-28bb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-92/obj2 2024-04-20T01:55:28.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-92/obj2 2024-04-20T01:55:28.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-92/obj2 2024-04-20T01:55:28.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.967 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.967 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.967 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.967 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:55:20 GMT 2024-04-20T01:55:28.967 INFO:tasks.rgw_multisite_tests:/aktlji-92/obj2 2024-04-20T01:55:28.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.967 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:iW34LuqJ9whOAY32a0/JeYjQnkc= 2024-04-20T01:55:28.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:iW34LuqJ9whOAY32a0/JeYjQnkc=', 'Content-Length': '0'} 2024-04-20T01:55:28.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-92:73f46f02-c7ab-4426-a267-2273af911da5.7664.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006364a2d23a6a34e5-0066232088-28bb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-92/obj2 2024-04-20T01:55:28.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-92/obj2 2024-04-20T01:55:28.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:28.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-92/obj2 2024-04-20T01:55:28.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.968 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:28.969 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.969 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.969 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:55:20 GMT 2024-04-20T01:55:28.969 INFO:tasks.rgw_multisite_tests:/aktlji-92/obj2 2024-04-20T01:55:28.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.969 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:W+kENkDcMnesPdNTQIEK8+U27hk= 2024-04-20T01:55:28.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:W+kENkDcMnesPdNTQIEK8+U27hk=', 'Content-Length': '0'} 2024-04-20T01:55:28.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-92:73f46f02-c7ab-4426-a267-2273af911da5.7664.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdd251937e8de5a0-0066232088-28bb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-93/ 2024-04-20T01:55:28.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-93/ 2024-04-20T01:55:28.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-93/ 2024-04-20T01:55:28.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.970 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.970 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.970 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.971 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:55:20 GMT 2024-04-20T01:55:28.971 INFO:tasks.rgw_multisite_tests:/aktlji-93/ 2024-04-20T01:55:28.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.971 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:ShlfjV73ipt3eZY8+uPx45eEFsA= 2024-04-20T01:55:28.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:ShlfjV73ipt3eZY8+uPx45eEFsA=', 'Content-Length': '0'} 2024-04-20T01:55:28.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c75a49d653e536d4-0066232088-28bb-test-zone2'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-93/obj1 2024-04-20T01:55:28.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-93/obj1 2024-04-20T01:55:28.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-93/obj1 2024-04-20T01:55:28.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.972 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.972 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.972 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.972 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:55:20 GMT 2024-04-20T01:55:28.972 INFO:tasks.rgw_multisite_tests:/aktlji-93/obj1 2024-04-20T01:55:28.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.972 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:w4ww0G51T4OEnuTKu9p+zMWyrp4= 2024-04-20T01:55:28.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:w4ww0G51T4OEnuTKu9p+zMWyrp4=', 'Content-Length': '0'} 2024-04-20T01:55:28.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-92:73f46f02-c7ab-4426-a267-2273af911da5.7664.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000723edad7b0294638-0066232088-28bb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-93/obj1 2024-04-20T01:55:28.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-93/obj1 2024-04-20T01:55:28.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-20T01:55:28.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-93/obj1 2024-04-20T01:55:28.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.974 INFO:tasks.rgw_multisite_tests:GET 2024-04-20T01:55:28.974 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.974 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.974 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:55:20 GMT 2024-04-20T01:55:28.974 INFO:tasks.rgw_multisite_tests:/aktlji-93/obj1 2024-04-20T01:55:28.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.974 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:o7eHfSBAxxy//YB94p0L/JOOCxg= 2024-04-20T01:55:28.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:o7eHfSBAxxy//YB94p0L/JOOCxg=', 'Content-Length': '0'} 2024-04-20T01:55:28.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 20 Apr 2024 01:53:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '73f46f02-c7ab-4426-a267-2273af911da5:aktlji-92:73f46f02-c7ab-4426-a267-2273af911da5.7664.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099360a3a70dd4d1b-0066232088-28bb-test-zone2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/aktlji-93/obj2 2024-04-20T01:55:28.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/aktlji-93/obj2 2024-04-20T01:55:28.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-20T01:55:28.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /aktlji-93/obj2 2024-04-20T01:55:28.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-20T01:55:28.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-20T01:55:28.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi031.front.sepia.ceph.com:8000 2024-04-20T01:55:28.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-20T01:55:28.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-20T01:55:28.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-20T01:55:28.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-20T01:55:28.975 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-20T01:55:28.976 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.976 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.976 INFO:tasks.rgw_multisite_tests:Sat, 20 Apr 2024 01:55:20 GMT 2024-04-20T01:55:28.976 INFO:tasks.rgw_multisite_tests:/aktlji-93/obj2 2024-04-20T01:55:28.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-20T01:55:28.976 INFO:tasks.rgw_multisite_tests:AWS T4PF8H1DJQT67HZSKX5F:J/v5zu5qmYipXULkPQvod0NFn+M= 2024-04-20T01:55:28.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Sat, 20 Apr 2024 01:55:20 GMT', 'Authorization': 'AWS T4PF8H1DJQT67HZSKX5F:J/v5zu5qmYipXULkPQvod0NFn+M=', 'Content-Length': '0'} 2024-04-20T01:55:28.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '248'), ('x-amz-request-id', 'tx00000a7624b452ef0c9d3-0066232088-28bb-test-zone2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 20 Apr 2024 01:55:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T01:55:28.976 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-20T01:55:28.977 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.977 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-20T01:55:28.977 INFO:tasks.rgw_multisite_tests:Ran 48 tests in 9164.111s 2024-04-20T01:55:28.977 INFO:tasks.rgw_multisite_tests: 2024-04-20T01:55:28.977 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=1, errors=1, failures=25) 2024-04-20T01:55:28.986 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-04-20T01:55:29.223 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fc8c3678e76f4188b0c5c0efc71f8d25 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph_646b8c7d293d8ec31a40c69f7b3237b19c1be24d/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-04-20T01:55:29.225 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-04-20T01:55:29.236 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-04-20T01:55:29.246 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-04-20T01:55:29.257 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-04-20T01:55:29.257 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:29.261 INFO:tasks.rgw.c1.client.0.smithi019.stdout:2024-04-20T01:55:29.269+0000 7fa0156ee700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 48099) UID: 0 2024-04-20T01:55:29.261 INFO:tasks.rgw.c1.client.0.smithi019.stdout:2024-04-20T01:55:29.269+0000 7fa01a8c37c0 -1 shutting down 2024-04-20T01:55:29.864 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-04-20T01:55:35.359 INFO:tasks.rgw.c1.client.0:Stopped 2024-04-20T01:55:35.360 DEBUG:teuthology.orchestra.run.smithi019:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2024-04-20T01:55:35.383 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f /etc/ceph/vault-root-token 2024-04-20T01:55:35.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2024-04-20T01:55:35.476 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-20T01:55:35.477 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:35.478 INFO:tasks.rgw.c1.client.1.smithi019.stdout:2024-04-20T01:55:35.488+0000 7f83c2186700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 72710) UID: 0 2024-04-20T01:55:35.478 INFO:tasks.rgw.c1.client.1.smithi019.stdout:2024-04-20T01:55:35.488+0000 7f83c735b7c0 -1 shutting down 2024-04-20T01:55:40.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-20T01:55:40.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-04-20T01:55:41.579 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-20T01:55:41.580 DEBUG:teuthology.orchestra.run.smithi019:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2024-04-20T01:55:41.602 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f /etc/ceph/vault-root-token 2024-04-20T01:55:41.677 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-20T01:55:41.678 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:41.702 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-20T01:55:41.713+0000 7f0a2ddee700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 103520) UID: 0 2024-04-20T01:55:41.702 INFO:tasks.rgw.c2.client.0.smithi031.stdout:2024-04-20T01:55:41.713+0000 7f0a32fc37c0 -1 shutting down 2024-04-20T01:55:46.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-20T01:55:46.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-20T01:55:47.779 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-20T01:55:47.780 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2024-04-20T01:55:47.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/vault-root-token 2024-04-20T01:55:47.901 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-04-20T01:55:47.902 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:47.902 INFO:tasks.rgw.c2.client.1.smithi031.stdout:2024-04-20T01:55:47.913+0000 7f4aa7207700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 46726) UID: 0 2024-04-20T01:55:47.902 INFO:tasks.rgw.c2.client.1.smithi031.stdout:2024-04-20T01:55:47.913+0000 7f4aac3dc7c0 -1 shutting down 2024-04-20T01:55:52.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-04-20T01:55:52.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-20T01:55:54.003 INFO:tasks.rgw.c2.client.1:Stopped 2024-04-20T01:55:54.004 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2024-04-20T01:55:54.027 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/vault-root-token 2024-04-20T01:55:54.102 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-20T01:55:54.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2024-04-20T01:55:54.674 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-20T01:55:54.674 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-20T01:55:54.675 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-04-20T01:55:54.675 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:54.743 INFO:tasks.ceph.c2.osd.0:Stopped 2024-04-20T01:55:54.744 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-04-20T01:55:54.744 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:54.823 INFO:tasks.ceph.c2.osd.1:Stopped 2024-04-20T01:55:54.824 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-04-20T01:55:54.824 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:54.895 INFO:tasks.ceph.c2.osd.2:Stopped 2024-04-20T01:55:54.895 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-20T01:55:54.896 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-04-20T01:55:54.896 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:54.924 INFO:tasks.ceph.c2.mgr.x:Stopped 2024-04-20T01:55:54.925 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-20T01:55:54.926 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-04-20T01:55:54.926 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:54.989 INFO:tasks.ceph.c2.mon.a:Stopped 2024-04-20T01:55:54.989 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-20T01:55:54.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-20T01:55:55.089 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi031.front.sepia.ceph.com 2024-04-20T01:55:55.090 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2024-04-20T01:55:55.484 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi031.front.sepia.ceph.com 2024-04-20T01:55:55.484 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2024-04-20T01:55:55.601 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi031.front.sepia.ceph.com 2024-04-20T01:55:55.602 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2024-04-20T01:55:55.698 INFO:tasks.ceph:Archiving mon data... 2024-04-20T01:55:55.699 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001/data/c2.mon.a.tgz 2024-04-20T01:55:55.700 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-04-20T01:55:55.720 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.038SaaDlEB 2024-04-20T01:55:55.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.038SaaDlEB 2024-04-20T01:55:55.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.038SaaDlEB 2024-04-20T01:55:56.201 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.038SaaDlEB is 434KB 2024-04-20T01:55:56.281 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.038SaaDlEB 2024-04-20T01:55:56.303 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-20T01:55:56.303 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-20T01:55:56.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-20T01:55:56.376 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-20T01:55:56.395 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2024-04-20T01:55:56.789 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-20T01:55:56.790 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-20T01:55:56.790 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-04-20T01:55:56.790 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:56.836 INFO:tasks.ceph.c1.osd.0:Stopped 2024-04-20T01:55:56.837 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-04-20T01:55:56.837 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:56.892 INFO:tasks.ceph.c1.osd.1:Stopped 2024-04-20T01:55:56.892 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-04-20T01:55:56.892 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:56.950 INFO:tasks.ceph.c1.osd.2:Stopped 2024-04-20T01:55:56.950 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-20T01:55:56.950 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-04-20T01:55:56.951 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:56.978 INFO:tasks.ceph.c1.mgr.x:Stopped 2024-04-20T01:55:56.979 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-20T01:55:56.979 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-04-20T01:55:56.979 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T01:55:57.044 INFO:tasks.ceph.c1.mon.a:Stopped 2024-04-20T01:55:57.044 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-20T01:55:57.044 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-20T01:55:57.045 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~0s 2024-04-20T01:55:57.120 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi019.front.sepia.ceph.com 2024-04-20T01:55:57.120 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2024-04-20T01:55:57.482 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi019.front.sepia.ceph.com 2024-04-20T01:55:57.483 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2024-04-20T01:55:57.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-04-20T01:55:57.590 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi019.front.sepia.ceph.com 2024-04-20T01:55:57.590 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2024-04-20T01:55:57.690 INFO:tasks.ceph:Archiving mon data... 2024-04-20T01:55:57.690 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001/data/c1.mon.a.tgz 2024-04-20T01:55:57.690 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-04-20T01:55:57.711 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.LXpAB176Rp 2024-04-20T01:55:57.711 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.LXpAB176Rp 2024-04-20T01:55:57.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.LXpAB176Rp 2024-04-20T01:55:58.019 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.LXpAB176Rp is 489KB 2024-04-20T01:55:58.118 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.LXpAB176Rp 2024-04-20T01:55:58.140 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-20T01:55:58.141 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-20T01:55:58.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-20T01:55:58.217 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-20T01:55:58.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-20T01:55:58.256 DEBUG:teuthology.orchestra.run.smithi031:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-20T01:55:58.289 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-20T01:55:58.292 INFO:teuthology.orchestra.run.smithi031.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-20T01:55:58.299 INFO:tasks.ceph:Archiving crash dumps... 2024-04-20T01:55:58.301 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001/remote/smithi019/crash 2024-04-20T01:55:58.302 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-20T01:55:58.373 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001/remote/smithi031/crash 2024-04-20T01:55:58.373 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-20T01:55:58.408 INFO:tasks.ceph:Compressing logs... 2024-04-20T01:55:58.408 DEBUG:teuthology.orchestra.run.smithi019:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T01:55:58.417 DEBUG:teuthology.orchestra.run.smithi031:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T01:56:02.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a has been restored 2024-04-20T02:01:13.575 INFO:tasks.ceph:Archiving logs... 2024-04-20T02:01:13.575 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001/remote/smithi019/log 2024-04-20T02:01:13.577 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-20T02:01:41.313 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001/remote/smithi031/log 2024-04-20T02:01:41.314 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-20T02:02:19.204 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-20T02:02:19.217 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-20T02:02:19.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-20T02:02:19.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-20T02:02:19.305 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-20T02:02:19.320 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-20T02:02:19.320 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T02:02:19.323 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T02:02:19.342 INFO:teuthology.orchestra.run.smithi019.stderr:bash: ntpq: command not found 2024-04-20T02:02:19.344 INFO:teuthology.orchestra.run.smithi031.stderr:bash: ntpq: command not found 2024-04-20T02:02:19.347 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T02:02:19.348 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-04-20T02:02:19.348 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 21 -475us[ -475us] +/- 81ms 2024-04-20T02:02:19.348 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -1255us[-1235us] +/- 65ms 2024-04-20T02:02:19.348 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +2299us[+2299us] +/- 81ms 2024-04-20T02:02:19.348 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T02:02:19.376 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T02:02:19.376 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-04-20T02:02:19.376 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 25 -466us[ -457us] +/- 81ms 2024-04-20T02:02:19.377 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -1332us[-1323us] +/- 65ms 2024-04-20T02:02:19.377 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +2255us[+2255us] +/- 80ms 2024-04-20T02:02:19.377 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T02:02:19.378 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-20T02:02:19.387 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-20T02:02:19.388 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-20T02:02:19.416 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-20T02:02:19.495 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-20T02:02:19.594 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-20T02:02:19.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T02:02:19.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-20T02:02:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T02:02:19.669 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-20T02:02:19.679 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-20T02:02:19.688 INFO:teuthology.task.internal:Duration was 10769.869666 seconds 2024-04-20T02:02:19.689 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-20T02:02:19.700 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-20T02:02:19.700 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-20T02:02:19.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-20T02:02:19.760 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T02:02:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T02:02:20.048 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-20T02:02:20.048 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2024-04-20T02:02:20.048 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-20T02:02:20.078 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-04-20T02:02:20.078 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-20T02:02:20.107 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-20T02:02:20.107 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T02:02:20.120 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T02:02:20.201 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-20T02:02:20.201 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-20T02:02:20.204 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-20T02:02:20.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-20T02:02:20.330 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-20T02:02:20.331 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-20T02:02:20.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-20T02:02:20.419 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-20T02:02:20.429 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-20T02:02:20.432 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-20T02:02:20.468 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2024-04-20T02:02:20.480 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-04-20T02:02:20.512 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-20T02:02:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T02:02:20.546 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-20T02:02:20.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T02:02:20.568 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-20T02:02:20.583 INFO:teuthology.task.internal:Transferring archived files... 2024-04-20T02:02:20.583 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001/remote/smithi019 2024-04-20T02:02:20.584 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-20T02:02:20.635 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001/remote/smithi031 2024-04-20T02:02:20.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-20T02:02:20.682 INFO:teuthology.task.internal:Removing archive directory... 2024-04-20T02:02:20.682 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-20T02:02:20.685 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-20T02:02:20.739 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-20T02:02:20.749 INFO:teuthology.task.internal:Not uploading archives. 2024-04-20T02:02:20.750 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-20T02:02:20.760 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-20T02:02:20.761 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-20T02:02:20.763 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-20T02:02:20.782 INFO:teuthology.orchestra.run.smithi019.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 20 02:02 /home/ubuntu/cephtest 2024-04-20T02:02:20.800 INFO:teuthology.orchestra.run.smithi031.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 20 02:02 /home/ubuntu/cephtest 2024-04-20T02:02:20.801 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-20T02:02:20.812 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-20T02:02:20.874 INFO:teuthology.nuke:Checking targets against current locks 2024-04-20T02:02:20.899 DEBUG:teuthology.nuke:shortname: smithi019 2024-04-20T02:02:20.900 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T02:02:20.926 DEBUG:teuthology.nuke:shortname: smithi031 2024-04-20T02:02:20.927 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T02:02:20.932 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 22:57:05.745051', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIcvqhGL0N9TBJPBbHqO/Z9Admuo3j1MrM0n8mryoZRaN5gAbxH+wX247Y6GP/hxWVAgp2QfcTB35iT/FQ1H4a4='} 2024-04-20T02:02:20.957 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-19_22:40:02-rgw-reef-distro-default-smithi/7664001', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 22:57:05.746628', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLPGn4JYphtVVEoq1g7lxNACnaZmW1uujquDNRq/8bGYczh4L4ye1SfOdTGMj2UeOhZiDkldB65GKI6v8+/ZkUU='} 2024-04-20T02:02:20.988 INFO:teuthology.orchestra.console.smithi019:Power off 2024-04-20T02:02:20.989 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-20T02:02:21.018 DEBUG:teuthology.orchestra.console.smithi019:power off output: Chassis Power Control: Down/Off 2024-04-20T02:02:21.018 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T02:02:21.043 INFO:teuthology.orchestra.console.smithi031:Power off 2024-04-20T02:02:21.044 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-20T02:02:21.058 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-04-20T02:02:21.073 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2024-04-20T02:02:21.074 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T02:02:21.105 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-04-20T02:02:25.059 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T02:02:25.107 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T02:02:25.174 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-04-20T02:02:25.222 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-04-20T02:02:29.177 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T02:02:29.223 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T02:02:29.292 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-04-20T02:02:29.336 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-04-20T02:02:33.294 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T02:02:33.338 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T02:02:33.412 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is off 2024-04-20T02:02:33.455 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2024-04-20T02:02:33.514 INFO:teuthology.orchestra.console.smithi019:Power off completed 2024-04-20T02:02:33.557 INFO:teuthology.orchestra.console.smithi031:Power off completed 2024-04-20T02:02:33.662 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi} duration: 10769.869665622711 failure_reason: rgw multisite test failures flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fc8c3678e76f4188b0c5c0efc71f8d25 status: fail success: false 2024-04-20T02:02:33.662 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-20T02:02:33.742 INFO:teuthology.run:FAIL